A pipelined data correlation and automated management system, method, device, and medium

Abstract

The application provides a pipeline data association and automatic management system, method, device and medium. The system comprises: a visual binding module, which is used for completing visual binding of iterations or versions and code baseline Commit ID submission identification, and triggering data acquisition; a pipeline module, which is used for automatically processing multi-source data related to iterations or versions and establishing an association relationship with a pipeline and an artifact; an artifact metadata module, which is used for managing associated metadata of the artifact and constructing a relationship graph; and a query and traceback module, which is used for realizing full-link data retrieval and visualized tracing. The application solves problems such as multi-plugin dependence and inaccurate data association, realizes automatic archiving and full-link tracing of artifact metadata, and improves the DevOps collaboration efficiency of an enterprise.

Description

Technical Field

[0001] The present application relates to the field of software development and DevOps technology, and in particular to a pipeline data association and automation management system, method, device and medium. Background Art

[0002] In the DevOps continuous integration and continuous delivery process, as software development scales and iterations accelerate, R&D faces challenges such as inaccurate data associations and chaotic artifact metadata management caused by multiple plugin dependencies. Traditional technical solutions rely on linking code commit records to work items, which only captures incremental code changes and cannot restore complete data from the entire branch lifecycle. Existing technologies often rely on the coordinated operation of multiple plugins (such as requirements plugins, deployment plugins, and archiving plugins), resulting in cumbersome configuration processes and a high risk of data inconsistencies due to manual manipulation. Furthermore, the lack of an automated mechanism to link test reports with code commits and artifact metadata results in a broken quality traceability chain. This makes it difficult to efficiently link and automatically manage data such as iteration or version information, code changes, test reports, and work items. This leads to inefficiencies in tracing issues and verifying the impact of changes, increasing collaboration costs and the risk of errors. Therefore, a technical solution that can link pipeline data and automate artifact metadata management is urgently needed to improve the efficiency and reliability of DevOps processes. Summary of the Invention

[0003] In view of this, the embodiments of the present application provide a pipeline data association and automated management system, method, device and medium, which integrates R&D data such as code baselines, work items, and test reports through iteration or version dimensions to achieve automated archiving and full-link traceability of product metadata, which is suitable for improving the efficiency of enterprise DevOps collaboration, and is particularly suitable for iteration or version management under the microservice architecture of the financial industry, such as the grayscale release scenario of the core system of a bank. The iteration or version generally refers to the work unit divided by cycle in software development, including elements such as code baselines and test plans. The product metadata generally refers to the descriptive data of the software build product, including the Commit ID submission identifier, the test report URL uniform resource locator, etc.

[0004] The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides a pipeline data association and automation management system, which includes a visualization binding module, a pipeline module, a product metadata module, and a query and tracing module; wherein,

[0006] The visual binding module is used to complete the visual binding of the iteration or version with the code baseline Commit ID submission identifier and trigger data acquisition;

[0007] The pipeline module is used to automatically process multi-source data related to iterations or versions and establish associations with pipelines and products;

[0008] The product metadata module is used to manage the associated metadata of the products and build a relationship map;

[0009] The query and tracing module is used to realize full-link data retrieval and visual tracing.

[0010] In one embodiment, the visual binding module is based on a Vue.js plug-in interface, supports CommitID submission identification input and automatic filling of product information, and is deeply integrated with the Gitlab code library to complete the visual binding of iterations or versions with code baselines and trigger data acquisition operations.

[0011] In one embodiment, the pipeline module includes: a multi-source data acquisition unit, a difference analysis unit, a log formatting unit, and a load balancing unit;

[0012] The multi-source data acquisition unit integrates the requirements management platform, test reporting system and code library multi-source API application programming interface to achieve comprehensive acquisition of work items, test reports and code change data;

[0013] The difference analysis unit uses Java 8 Stream API technology to implement an efficient comparison algorithm between work items and code submission information, accurately analyze differences and generate reports;

[0014] The log formatting unit records and formats the various data processing operations during the pipeline operation according to the standard format of the JSON mode;

[0015] When the number of concurrent compilation requests exceeds a threshold, the load balancing unit automatically triggers the Kubernetes HPA mechanism to expand plug-in instances and ensure system performance. The HPA refers to the horizontal pod autoscaler.

[0016] In one embodiment, the product metadata module includes: a metadata writing API application programming interface, a graph data storage unit, a visualization rendering engine, and a CMDB configuration management database docking interface;

[0017] The metadata writing API application programming interface supports multiple mainstream product repositories such as Nexus and Artifactory to achieve accurate metadata writing;

[0018] The graph data storage unit, based on the Neo4j graph database, maintains complex relationships between data and achieves data integrity and traceability;

[0019] The visualization rendering engine uses Grafana and ECharts tools to generate weighted association topology maps to intuitively display data associations.

[0020] The CMDB configuration management database docking interface transmits the relationship between the application programming interface iteration or version and the configuration item through the REST API expressive state transmission. The relationship includes the iteration ID, configuration item ID, and association timestamp. When the CMDB configuration management database system data changes, the timestamp field of the relationship in Neo4j is automatically updated.

[0021] In one embodiment, the query tracing module includes: Elasticsearch retrieval service and data visualization service;

[0022] The Elasticsearch search service is built on Elasticsearch and supports fuzzy matching search of key information of work item numbers to quickly locate relevant data;

[0023] The data visualization service uses Grafana and ECharts to implement chart rendering, and displays the search results and data association analysis results in an intuitive manner.

[0024] In a second aspect, an embodiment of the present application provides a pipeline data association and automated management method, including:

[0025] Iteration or version binding with code baseline and data acquisition: Enter the code baseline CommitID corresponding to the iteration or version, and the system automatically triggers the code base API application programming interface to obtain the complete code change set corresponding to the CommitID, and associates the artifact information, including the build results and test report URL uniform resource locator;

[0026] Pipeline multi-source data processing: The pipeline plug-in obtains the code baseline commit ID, work items, test reports, and difference reports associated with the iteration or version, and writes artifact information. The test report includes the test case ID, execution status, and time consumption information. The difference report includes a comparison of the work items within the iteration or version with the work items in the code commit information. Writing artifact information includes writing metadata to the target artifact and establishing data associations.

[0027] Artifact metadata management and traceability: Store iteration or version IDs, code baseline commit IDs, test reports, difference reports, visualization tools to render work items, and associated topology maps of code baselines and pipelines; use Elasticsearch indexes to build real-time search services, support fuzzy matching searches by work item number, and achieve full-link traceability.

[0028] In a third aspect, embodiments of the present application provide an electronic device comprising: a memory and a processor. The memory stores instructions, which are loaded and executed by the processor to implement the steps of the aforementioned pipeline data association and automated management method. The memory and the processor communicate with each other via an internal connection path.

[0029] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed, the steps of the above-mentioned pipeline data association and automated management method are implemented.

[0030] The advantages or beneficial effects of the above technical solution include at least:

[0031] 1. Significantly improved efficiency: A single plug-in integrates multi-dimensional data, shortening pipeline configuration time and improving work item matching accuracy;

[0032] 2. Metadata Accuracy Guarantee: The automated archiving mechanism further improves the accuracy of artifact metadata, avoiding release failures due to inconsistent metadata and improving software delivery quality.

[0033] 3. CMDB configuration management database integration efficiency: The automated synchronization mechanism improves the efficiency of configuration item association and reduces manual maintenance costs;

[0034] 4. Data security: Reduce the risk of data leakage through Elasticsearch permission control and desensitization processing;

[0035] 5. Exception handling capability: The exponential backoff retry mechanism improves the success rate of API application programming interface calls;

[0036] 6. Enhanced business adaptability: Supports the iterative or version development management model of mainstream customers, significantly improving business agility;

[0037] 7. Optimized resource utilization: The pipeline load balancing mechanism enables dynamic resource adjustment, automatically expands plug-in instances in concurrent scenarios, and reduces infrastructure costs.

[0038] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. In the accompanying drawings, unless otherwise specified, the same reference numerals throughout multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that the following drawings only illustrate certain embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can also be obtained based on these drawings without paying creative work.

[0040] Figure 1 This is a structural diagram of a pipeline data association and automation management system according to an embodiment of the present application;

[0041] Figure 2 This is a flow chart of a pipeline data association and automated management method according to an embodiment of the present application;

[0042] Figure 3 This is a diagram showing the execution principle of the pipeline data association and automation management system according to one embodiment of the present application. DETAILED DESCRIPTION

[0043] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0044] The terms "first," "second," "third," and "fourth," etc., in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, rather than to describe a specific order. In addition, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0045] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0046] As attached Figure 1 As shown, the embodiment of the present application provides a pipeline data association and automation management system, including a visual binding module, a pipeline module, a product metadata module, and a query and traceability module. Figure 3 , a detailed introduction to each module and its execution principle:

[0047] 1. Visual binding module: binds the code baseline Commit ID of the iteration or version through the visual interface, triggers the code library API application programming interface to obtain the complete code change set and related artifact information, and generates a structured data list containing file change records, build results, and test report URL uniform resource locators.

[0048] 2. The pipeline module automatically acquires and analyzes multi-dimensional data. It uses the pipeline plug-in to parse the test plan API application programming interface associated with the iteration or version. It uses the / api / v2 / get_runs_for_plan interface of the TestRail API to recursively obtain all test reports under the multi-level test plan, store them in the distributed file system by test suite classification, and obtain a JSON data structure containing the test case ID, execution status, and time consumption. It calls the requirements management system API application programming interface to obtain the iteration or version work item collection, performs regular expression matching on the work item list and the work item number in the code submission information, identifies the differences for unmatched work items, and generates a difference report CSV file. It writes metadata such as work items, test reports, difference reports, code baseline commit ID submission identifier, and code baseline latest commit ID submission identifier into the target artifact. Specifically, it updates the artifact metadata through the / v1 / components interface of the Nexus Repository REST API representational state transfer application programming interface, and submits Cypher statements to the Neo4j database to create triple relationships. For example:

[0049] MATCH (i:Iteration {id: 'ITER-123'})

[0050] MATCH (p:Product {name: 'order-service:1.0.0'})

[0051] CREATE (i)-[:ASSOCIATED_WITH {commit_id: '7c258a3'}]->(p)

[0052] CREATE (p)-[:HAS_TEST_REPORT {report_url: 'https: / / testrail / report1'}]->(t:TestReport)

[0053] When a TestRail API call fails, the system executes the following exception handling process:

[0054] 1. Automatically record error logs, including error code, call time, and request parameters;

[0055] 2. Enable the exponential backoff retry mechanism, with an initial retry interval of 100ms, a maximum retry interval of 5s, and a maximum of 3 retries (the 100ms, 5s, and 3 times are examples and can be adjusted based on actual conditions).

[0056] 3. If the retry fails, an alert notification is sent to the operation and maintenance personnel, including the API application programming interface endpoint and error details;

[0057] 4. If the 3 retries fail, the system automatically switches to the backup TestRail node and records the switch log;

[0058] 5. Mark the test report acquisition failure status in the work-in-progress metadata and re-trigger the acquisition process after manual intervention.

[0059] 3. Artifact metadata module, used to manage artifact-related metadata and build relationship graphs, including metadata writing API application programming interface, supporting Nexus, Artifactory and other mainstream artifact repositories, and realizing accurate metadata writing; visual rendering engine, using Grafana and ECharts tools, to generate weighted association topology diagrams to intuitively display data associations; CMDB configuration management database docking interface, through REST API expressive state transmission application programming interface to synchronize iterations or versions with configuration items, including iteration ID, configuration item ID, and association timestamp; when CMDB configuration management database system data changes, such as when the configuration item ID changes, the timestamp field of the association in Neo4j is automatically updated.

[0060] 4. The query and traceability module is used to implement full-link data retrieval and visual traceability. The Elasticsearch retrieval service is built on Elasticsearch and supports fuzzy matching retrieval of key information of work item numbers to quickly locate relevant data; the data visualization service uses Grafana and ECharts to implement chart rendering, which displays the search results and data association analysis results in an intuitive way.

[0061] 5. Implementation Principle: Enter the code baseline commit ID corresponding to an iteration or version. The system automatically triggers the code repository API to retrieve the complete code change set and associated artifact information such as build results and test reports, generating structured data. It then parses the test plan API to retrieve multi-level test reports, categorizes and stores them, and extracts key information. It then calls the requirements management system to retrieve work items, which are then matched against the code commit records using regular expressions to generate diff reports. It then writes metadata to artifacts using the Nexus API and creates a triple association in Neo4j: "iteration or version, artifact, test report." It then creates a dedicated field in the artifact to store this association information. The full chain of relationships is visualized using Grafana / ECharts, and Elasticsearch is used to quickly retrieve work items. When concurrent compile requests exceed 100, the Kubernetes HPA mechanism is automatically triggered to scale the number of plugin instances from 2 to 5, automatically triggering Kubernetes HPA to scale up plugin instances to optimize resource utilization. (The numbers 100, 2, and 5 are examples and can be adjusted based on actual needs.)

[0062] Taking the e-commerce order service iteration ITER-123 as an example, when the code baseline commit ID is 7c258a3:

[0063] 1. The system automatically obtains the code change set of the order-service:1.0.0 product, which contains 15 file modification records;

[0064] 2. Recursively retrieve 56 test reports from the three-tier test plan through the TestRail API. When storing test reports by payment module and order module, hash the report URL using the MD5 algorithm and store them in the distributed file system in the format of / iteration ID / module type / first two digits of the hash value / hash value.

[0065] 3. Compare 20 work items in the requirements management system, find 3 mismatches and generate a difference report;

[0066] 4. Create a triple relationship between iteration or version, artifact, and test report in Neo4j, and use Grafana to visualize the relationship topology.

[0067] As another embodiment, the present application provides a pipeline data association and automated management method, such as Figure 2 As shown, the following steps are included:

[0068] S101: Visualize the code baseline Commit ID that binds the iteration or version;

[0069] S102: Parse the test plan API application programming interface to obtain a test report;

[0070] S103: Compare work item differences and generate a difference report;

[0071] S104: Writing metadata to the target product;

[0072] S105: Create devops_relation field to store the association relationship;

[0073] S106: Visually present the relationship between work items, test reports, pipelines, and difference reports.

[0074] In addition, an embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the memory stores a program, and the processor is used to run the program. When the program runs, the steps of the pipeline data association and automated management method provided by the above invention are executed.

[0075] The device may also preferably include a communication interface, which is used for communicating with external devices and transmitting data interactively.

[0076] It should be noted that the memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory.

[0077] Furthermore, an embodiment of the present application also provides another computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the pipeline data association and automated management method provided by the above invention.

[0078] It should be understood that the computer-readable storage medium is any data storage device that can store data or programs that can be subsequently read by a computer system. Examples of computer-readable storage media include read-only memory, random access memory, CD ROMs, HDDs, DVDs, magnetic tapes, and optical data storage devices. Computer-readable storage media can also be distributed among network-coupled computer systems so that computer-readable code can be stored and executed in a distributed manner.

[0079] The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination thereof.

[0080] In some embodiments, the computer-readable storage medium may be non-transitory.

[0081] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0082] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the scope of the present application, and such modifications and improvements are all within the scope of protection of the present application.

Claims

1. A pipeline data association and automation management system, characterized in that: The pipeline data association and automation management system includes a visual binding module, a pipeline module, a product metadata module, and a query and tracing module; wherein, The visual binding module is used to complete the visual binding of the iteration or version with the code baseline Commit ID submission identifier and trigger data acquisition; The visual binding module is based on a Vue.js plug-in interface. It supports the input of commit IDs and the automatic filling of artifact information. It is deeply integrated with the GitLab code base and is used to complete the visual binding of iterations or versions to code baselines and trigger data acquisition operations. The pipeline module is used to automatically process multi-source data related to iterations or versions and establish associations with pipelines and products; The pipeline module includes: a multi-source data acquisition unit, a difference analysis unit, a log formatting unit, and a load balancing unit; The multi-source data acquisition unit integrates the requirements management platform, test reporting system and code library multi-source API application programming interface to achieve comprehensive acquisition of work items, test reports and code change data; The difference analysis unit uses Java 8 Stream API technology to implement an efficient comparison algorithm between work items and code submission information, accurately analyze differences and generate reports; The log formatting unit records and formats the various data processing operations during the pipeline operation according to the standard format of the JSON mode; The load balancing unit automatically triggers the Kubernetes HPA mechanism when concurrent compilation requests exceed a threshold, scaling plugin instances to ensure system performance. The HPA refers to the horizontal pod autoscaler. The product metadata module is used to manage the associated metadata of the products and build a relationship map; The product metadata module includes: a metadata writing API application programming interface, a graph data storage unit, a visualization rendering engine, and a CMDB configuration management database docking interface; The metadata writing API application programming interface supports multiple mainstream product repositories such as Nexus and Artifactory to achieve accurate metadata writing; The graph data storage unit, based on the Neo4j graph database, maintains complex relationships between data and achieves data integrity and traceability; The visualization rendering engine uses Grafana and ECharts tools to generate weighted association topology maps to intuitively display data associations. The CMDB configuration management database docking interface transmits the relationship between the iteration or version and the configuration item through the REST API expressive state transmission application programming interface synchronization, and the relationship includes the iteration ID, configuration item ID, and the relationship timestamp; when the CMDB configuration management database system data changes, the timestamp field of the relationship in Neo4j is automatically updated; The query and tracing module is used to realize full-link data retrieval and visual tracing; The query and tracing module includes: Elasticsearch retrieval service and data visualization service; The Elasticsearch search service is built on Elasticsearch and supports fuzzy matching search of key information of work item numbers to quickly locate relevant data; The data visualization service uses Grafana and ECharts to implement chart rendering, and displays the search results and data association analysis results in an intuitive manner.

2. A pipeline data association and automated management method, characterized in that: The pipeline data association and automated management method is applied to the pipeline data association and automated management system according to claim 1, and the pipeline data association and automated management method includes: Iteration or version binding with code baseline and data acquisition: Enter the code baseline Commit ID corresponding to the iteration or version, and the system automatically triggers the code base API application programming interface to obtain the complete code change set corresponding to the Commit ID, and associates the artifact information, including the build results and test report URL uniform resource locator; Pipeline multi-source data processing: The pipeline plug-in obtains the code baseline commit ID, work items, test reports, and difference reports associated with the iteration or version, and writes artifact information. The test report includes the test case ID, execution status, and time consumption information. The difference report includes a comparison of the work items within the iteration or version with the work items in the code commit information. Writing artifact information includes writing metadata to the target artifact and establishing data associations. Artifact metadata management and traceability: Store iteration or version IDs, code baseline commit IDs, test reports, difference reports, visualization tools to render work items, and associated topology maps of code baselines and pipelines; use Elasticsearch indexes to build real-time search services, support fuzzy matching searches by work item number, and achieve full-link traceability.

3. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the steps of the pipeline data association and automatic management method as claimed in claim 2.

4. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed, the steps of the pipeline data association and automatic management method according to claim 2 are implemented.

Images