A pipeline process adaptive configuration method and system

Abstract

The application relates to the technical field of software engineering, and discloses a pipeline process adaptive configuration method and system. The application generates structured language description data by acquiring source code data of a software project and performing static analysis and feature extraction; performs query and matching in a preset template library based on the structured language description data to generate a task template set; performs task dependency relationship analysis and process topology assembly on the task template set to generate initial pipeline data; and performs dynamic strategy adjustment on the execution process of the initial pipeline data by using real-time context data corresponding to the software project to generate target pipeline data. The application realizes automatic configuration of a continuous integration process through automatic language recognition and intelligent process generation, and solves the technical problem of low automation level caused by the lack of automatic recognition and adaptation capability of the project development language environment in the project continuous integration and continuous deployment technology.

Description

Technical Field

[0001] This invention relates to the field of software engineering technology, and in particular to a pipeline process adaptive configuration method and system. Background Technology

[0002] Currently, with the development of software engineering and the widespread application of microservice architecture, software projects in enterprise scenarios are increasingly adopting multi-language development models. For example, Java is used for backend main logic, Node.js for frontend interfaces, Python for data processing, and Go for high-concurrency modules. Different programming languages ​​have their own build toolchains and deployment processes. During the continuous integration and continuous deployment (CI / CD) process, it is necessary to configure build, testing, and release steps that match the characteristics of the language.

[0003] Existing mainstream CI / CD platforms, such as Jenkins, GitLab CI, and GitHub Actions, typically provide general pipeline orchestration capabilities. However, they suffer from the following issues when configuring workflows: they cannot automatically identify the programming language used in the project, cannot dynamically recommend suitable workflow templates based on the identification results, and the configuration process relies entirely on manual intervention. This forces developers to master toolchain knowledge in multiple languages ​​and repeatedly perform complex manual configurations in each new project, resulting in high operational barriers, low configuration efficiency, and poor workflow reusability. Summary of the Invention

[0004] This invention provides a pipeline process adaptive configuration method and system, which solves the technical problem that existing continuous integration and continuous deployment technologies lack the ability to automatically identify and adapt to the project development language environment, resulting in low automation levels.

[0005] The first aspect of this invention provides a pipeline process adaptive configuration method, comprising:

[0006] Obtain the source code data of the software project, perform static analysis and feature extraction on the source code data, and generate structured language description data;

[0007] The structured language is used to describe the data, which is then queried and matched in a preset template library to generate a set of task templates;

[0008] Based on the set of task templates, task dependency analysis and process topology assembly are performed to generate initial pipeline data.

[0009] The execution process of the initial pipeline data is dynamically adjusted using the real-time context data corresponding to the software project to generate the target pipeline data.

[0010] Optionally, the step of performing static analysis and feature extraction on the source code data to generate structured language description data includes:

[0011] Extract and parse predefined language configuration files from the source code data to generate preliminary language description data;

[0012] Statistical and weighted calculations are performed on the source code files in the source code data to generate weighted distribution data;

[0013] When the source code data contains multiple sub-modules, the directory of each sub-module is recursively traversed and language recognition is performed separately to generate a multi-module language mapping table.

[0014] Structured language description data is generated by aggregating at least one of the preliminary language description data, the weight distribution data, and the multi-module language mapping table.

[0015] Optionally, the step of using the structured language to describe the data and querying and matching it in a preset template library to generate a task template set includes:

[0016] Using the programming language type in the structured language description data as the query keyword, a basic task template group is constructed by querying a preset template library.

[0017] The basic task template group is filtered using the language version information and / or user project type in the structured language description data to generate a precise task template set;

[0018] When the structured language description data contains multiple programming languages, the task template sets corresponding to each programming language are aggregated to generate a unified multilingual project pipeline template.

[0019] A task template set is constructed based on one of the basic task template group, the precise task template set, and the multilingual project pipeline template.

[0020] Optionally, the step of performing task dependency analysis and process topology assembly based on the task template set to generate initial pipeline data includes:

[0021] The task template set is analyzed for input-output dependencies between task nodes to generate a task dependency topology graph.

[0022] The task dependency topology graph is sorted topologically to generate an initial flow control structure.

[0023] When the structured language description data contains multiple programming languages, the initial flow control structure is subjected to parallel branch aggregation processing to generate the target flow control structure.

[0024] The target flow control structure is combined with the task configuration parameters in the task template set to construct the initial pipeline data.

[0025] Optionally, the step of dynamically adjusting the execution process of the initial pipeline data using real-time context data corresponding to the software project to generate target pipeline data includes:

[0026] The project directory structure in the real-time context data corresponding to the software project is determined by rules to dynamically start and stop task nodes in the initial pipeline data, and task start and stop decision data is generated.

[0027] The task dependencies in the initial pipeline data are analyzed, and task nodes without dependencies are scheduled to be executed in parallel, generating task parallel execution data;

[0028] According to the pre-configured strategy, task nodes that fail or time out during the execution of the initial pipeline data are retried or interrupted, and task exception handling data is generated.

[0029] The initial pipeline data is comprehensively corrected using the task start / stop decision data, the task parallel execution data, and the task exception handling data to generate the target pipeline data.

[0030] Optionally, the method further includes:

[0031] The initial pipeline data or the target pipeline data is associated and stored with the corresponding structured language description data to generate a custom template;

[0032] Based on the configuration request of the new project and its structured language description data, query and match in the stored custom templates to generate a template recommendation list;

[0033] The template selected by the user from the template recommendation list is applied to the new project, generating the initialization pipeline data for the new project.

[0034] Optionally, the method further includes:

[0035] Register an external plugin that includes new programming language recognition rules to generate a language recognition rule set;

[0036] Register external plugins that include task templates for new programming languages ​​or build tools to generate an expanded collection of task templates;

[0037] The language recognition rule set and the task template extension set are integrated and hot-loaded to generate an extended system that supports automatic recognition and process configuration of new programming languages.

[0038] A second aspect of the present invention provides an adaptive configuration system for a pipeline process, comprising:

[0039] The description data generation module is used to acquire source code data of software projects, perform static analysis and feature extraction on the source code data, and generate structured language description data.

[0040] The template set generation module is used to query and match the data described by the structured language in a preset template library to generate a task template set;

[0041] The initial pipeline data generation module is used to perform task dependency analysis and process topology assembly based on the task template set to generate initial pipeline data;

[0042] The target pipeline data generation module is used to dynamically adjust the execution process of the initial pipeline data using real-time context data corresponding to the software project, and generate target pipeline data.

[0043] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the pipeline process adaptive configuration method as described in any of the preceding claims.

[0044] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the pipeline process adaptive configuration method as described in any of the preceding claims.

[0045] As can be seen from the above technical solutions, the present invention has the following advantages:

[0046] This invention first acquires the source code data of a software project and generates structured language description data through static analysis and feature extraction. Then, based on the structured language description data, it performs intelligent matching within a pre-set template library to generate a set of task templates. Next, it generates initial pipeline data through task dependency analysis and process topology assembly. Finally, it uses real-time context data to dynamically adjust the pipeline execution process, generating target pipeline data. This invention solves the problem of traditional CI / CD platforms' inability to automatically identify the project's technology stack through automatic language recognition technology. Through an intelligent template matching mechanism, it achieves precise adaptation of the build process. Through automatic process topology assembly, it optimizes the task execution order and parallelism; through dynamic strategy adjustment, it improves the efficiency and stability of pipeline execution. The implementation of this invention fully automates the configuration of CI / CD pipelines, lowers the technical threshold, improves software delivery efficiency, and ensures the compatibility and scalability of multi-language projects. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a flowchart of the steps of an adaptive configuration method for a pipeline process provided in Embodiment 1 of the present invention;

[0049] Figure 2 This is a structural block diagram of an enterprise-level, platform-based CI / CD system provided in Embodiment 1 of the present invention;

[0050] Figure 3 This is a flowchart of the steps of an adaptive configuration method for a pipeline process provided in Embodiment 2 of the present invention;

[0051] Figure 4 This is a structural block diagram of a pipeline process generator provided in Embodiment 2 of the present invention;

[0052] Figure 5 This is a structural block diagram of a pipeline process adaptive configuration system provided in Embodiment 3 of the present invention;

[0053] Figure 6 This is a structural block diagram of a computer device provided in Embodiment 4 of the present invention. Detailed Implementation

[0054] In current DevOps (Development and Operations) practices, pipeline platforms generally suffer from problems such as lack of language recognition capabilities, incompatible build process templates, and poor reusability of task modules when building software projects using different programming languages. These issues make it difficult to meet the requirements for flexibility and automation of continuous integration and deployment processes in a multi-language environment.

[0055] Specifically, the following core issues in existing technologies urgently need to be addressed: 1) Inability to automatically identify project languages: Existing pipeline tools lack the ability to structurally analyze code repository content, making it impossible to accurately determine the main programming language of a project based on configuration files or file structures. This results in the platform being unable to recommend and adapt build processes accordingly. 2) Poor universality of build process templates: Different language projects require specific compilation, packaging, testing, and deployment toolchains. Traditional CI platforms can only use static templates or manual scripts for configuration, making it difficult to automate and reuse in multi-project, multi-language scenarios. 3) Lack of intelligence and adaptability in process generation: Even if a unified interface can be used to configure the process, existing systems cannot dynamically skip irrelevant tasks, adjust the execution order, or determine task concurrency based on language characteristics, leading to rigid processes and low execution efficiency. 4) Weak platform scalability, unable to cover the evolving language ecosystem: When introducing new languages ​​or build methods, traditional systems need to define process templates from scratch, lacking a unified template registration interface mechanism, resulting in high expansion costs and long cycles. 5) High barrier to entry, affecting the user experience for beginners: For most users of the platform, manually configuring the pipeline is not only time-consuming and tedious, but also prone to errors, which greatly restricts the promotion and popularization of the platform.

[0056] Therefore, embodiments of the present invention provide an adaptive pipeline configuration method and system, an adaptive pipeline configuration method that can automatically identify and match the corresponding build process based on the programming language used in the project, aiming to achieve the following objectives:

[0057] 1) Automatically completes language recognition and template calling, improving the intelligence level of configuration;

[0058] 2) Provides the ability to automatically combine and visually arrange process modules;

[0059] 3) Achieve adaptive adjustment and execution optimization of process tasks;

[0060] 4) Reduce configuration difficulty and usage threshold;

[0061] 5) Support the platform to expand to new language types and build processes in the future.

[0062] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0063] Example 1

[0064] Please see Figure 1 and Figure 2 , Figure 1 The flowchart illustrates the steps of an adaptive configuration method for a pipeline process provided in Embodiment 1 of the present invention.

[0065] Figure 2 As an enterprise-level, platform-based CI / CD system, this system architecture consists of three core layers: a management platform, a user platform, and CI / CD component adapters, collectively supporting the complete enterprise-level software development process. The management platform focuses on global control, encompassing user management, access control, audit management, template management, online review, and data services, ensuring platform security, compliance, and standardization. The user platform serves as the operational interface for project delivery, providing team management, project management, member management, role management, environment management, agent management, quality control management, build management, deployment management, and pipeline functions, supporting team collaboration and full project lifecycle management. The pipeline refers to the automated execution process within the CI / CD system, composed of phased tasks, used to ensure the continuity and stability of software delivery.

[0066] The CI / CD component adapter is the system's automated execution engine. Its core components include a scheduler, log tracking, data services, and message queues, responsible for process scheduling, data acquisition, and inter-service communication. This engine directly manipulates underlying resources and repositories, including code repositories, component repositories, and disk repositories, and drives the quality inspection and build engines to perform specific tasks. The entire process runs on computing resources and processes data streams from business data and logs, ultimately outputting business results through the CI / CD component adapter.

[0067] The CI / CD component adapter core scheduling center drives the entire process through a pipeline engine. Specifically: build tasks are distributed by the scheduling center to build agents that poll the tasks, and build artifacts are stored in Build Storage; deployment tasks are dispatched by the Deploy Master to deployment agents, which retrieve artifacts from Deploy Storage and deploy them to the target environment. This engine relies on a powerful build farm, where agents are grouped by tags such as ALI-RELEASE / TEST / DEV, LINUX-RELEASE / TEST / DEV, and MAC-RELEASE / TEST / DEV to support cross-platform and multi-environment builds. Ultimately, deployment targets cover various terminals such as Alibaba Cloud, self-built servers, application service environments, Android Market, and the App Store, achieving a complete automated closed loop from code to multi-platform delivery.

[0068] This invention provides an adaptive configuration method for pipeline processes, comprising:

[0069] Step 101: Obtain the source code data of the software project, perform static analysis and feature extraction on the source code data, and generate structured language description data.

[0070] In this embodiment of the invention, source code data refers to the original data such as source code files, configuration files, and project structure files contained in the software project.

[0071] Structured language description data refers to standardized description information generated after analyzing source code, which includes the main programming language type, version information, and multi-module structure of the project.

[0072] Specifically, the project root directory and key subdirectories are scanned to extract and parse predefined language configuration files (such as pom.xml, package.json, go.mod, etc.). Simultaneously, source code files (such as .java, .py, .js, .go, etc.) are analyzed, and the primary programming language in a multi-language project is determined through weighted calculations. When the project contains multiple submodules, each submodule directory is recursively traversed and language recognition is performed separately. Finally, based on the configuration file parsing results, file weight distribution data, and at least one of the multi-module language mapping tables, structured language description data is generated.

[0073] Step 102: Use structured language to describe the data and perform queries and matching in the preset template library to generate a set of task templates.

[0074] In this embodiment of the invention, the preset template library refers to a set of standardized task templates pre-built and stored by the system, categorized by programming language and technology stack. Each template encapsulates the atomic task units required by a specific language project in the continuous integration / continuous deployment process, including but not limited to standardized operation instructions and environment configurations for stages such as compilation and building, dependency installation, code quality inspection, automated testing, security scanning, container packaging, and deployment.

[0075] Specifically, the programming language type in the structured language description data is used as the query keyword to perform a first-level match in a preset template library, constructing a basic task template group. Then, the basic task template group is filtered second-level using language version information and / or user project type to generate a precise task template set. When a project contains multiple programming languages, the task template sets corresponding to each language are aggregated, and parallel execution branches are created for task subsets of different languages ​​to generate a unified multi-language project pipeline template.

[0076] Step 103: Perform task dependency analysis and process topology assembly based on the task template set to generate initial pipeline data.

[0077] In this embodiment of the invention, the input-output dependencies between task nodes in the task template set are analyzed to generate a task dependency topology graph. The topology graph is then sorted, i.e., task nodes are organized into a flow structure with execution order based on the dependencies, generating an initial flow control structure. When the project is a multi-language project, the initial flow control structure undergoes parallel branch aggregation processing to generate a target flow control structure. Finally, the flow control structure is combined with the task configuration parameters in the task template set to construct initial pipeline data that has completed topology assembly but has not yet undergone runtime adjustments.

[0078] Step 104: Use the real-time context data corresponding to the software project to dynamically adjust the execution process of the initial pipeline data and generate the target pipeline data.

[0079] In this embodiment of the invention, real-time context data refers to the environmental state information of a project during pipeline execution, including directory structure, execution status, etc.

[0080] Specifically, the project directory structure in the real-time context data is used to determine rules, dynamically starting and stopping task nodes in the initial pipeline data, generating task start / stop decision data. Task dependencies in the initial pipeline data are analyzed, and task nodes without dependencies are scheduled for parallel execution, generating task parallel execution data. Failed or timed-out task nodes are retried or interrupted according to pre-configured strategies, generating task exception handling data. Finally, the initial pipeline data is comprehensively corrected based on the above data to generate the target pipeline data.

[0081] Example 2

[0082] Please see Figure 3 , Figure 3 This is a flowchart illustrating the steps of an adaptive configuration method for a pipeline process provided in Embodiment 2 of the present invention.

[0083] This invention provides an adaptive configuration method for pipeline processes, comprising:

[0084] Step 201: Obtain the source code data of the software project, perform static analysis and feature extraction on the source code data, and generate structured language description data.

[0085] Further, step 201 may include the following sub-steps:

[0086] S11. Extract and parse predefined language configuration files from the source code data to generate preliminary language description data.

[0087] In this embodiment of the invention, the predefined type of language configuration file refers to the standardized project configuration and dependency management files in the ecosystem of each programming language. Specifically, a file system scanner traverses the project directory structure to locate and read predefined type configuration files (including pom.xml, build.gradle, package.json, go.mod, requirements.txt, etc.). Then, the corresponding configuration file parser analyzes the file content structure to extract key information such as the explicitly declared programming language type, dependency list, and build tool configuration. Finally, based on the extracted information and the comments in the configuration file, the specific language version information (such as Python 3.9, Java 17, etc.) is further identified to generate preliminary language description data containing a complete technology stack description.

[0088] S12. Perform statistical and weighted calculations on the source code files in the source code data to generate weighted distribution data.

[0089] In this embodiment of the invention, source code files refer to files ending with a specific programming language extension. Specifically, a file type identifier scans all source code files in the project, classifies and counts them according to their file extensions (including .java, .py, .js, .go, etc.), and obtains the number of files in each language category. Then, a weight calculator applies predefined weighting rules (including file type weighting coefficients, file path depth weighting, file size weighting, etc.) to perform weighted calculations on the statistical results. Finally, a heuristic analyzer determines the primary programming language in the multi-language mixed project based on the weighted results, generating quantified weight distribution data.

[0090] S13. When the source code data contains multiple sub-modules, recursively traverse the directories of each sub-module and perform language recognition separately to generate a multi-module language mapping table.

[0091] In this embodiment of the invention, the multi-module language mapping table refers to a data structure that records the correspondence between each sub-module in a project and its primary programming language. Specifically, a directory traversaler detects the modular structure of the project. When multiple sub-modules are identified, a recursive analyzer is initiated to sequentially access the directories of each sub-module. For each sub-module, an independent language recognition process (including configuration file parsing and file statistical analysis) is performed. Then, a mapping builder establishes a correspondence between the paths of each sub-module and its identified primary programming language. Finally, a structured multi-module language mapping table is generated.

[0092] S14. Based on at least one of the preliminary language description data, weight distribution data, and multi-module language mapping table, aggregate to generate structured language description data.

[0093] In this embodiment of the invention, structured language description data refers to a standardized project technology stack description generated by integrating multiple analysis results. Specifically, a data aggregator receives preliminary language description data, weight distribution data, and a multi-module language mapping table as input sources. Then, a cross-validator performs consistency checks and conflict resolution on the analysis results from different data sources. Next, a decision fusion unit performs fusion processing on the multi-source data based on predefined priority rules (e.g., configuration file parsing results take precedence over statistical analysis results). Finally, unified and standardized structured language description data is generated, providing accurate input for downstream template matching and process generation modules.

[0094] Step 202: Use structured language to describe the data and perform queries and matching in the preset template library to generate a set of task templates.

[0095] Furthermore, step 202 may include the following sub-steps:

[0096] S21. Using the programming language type in the structured language description data as the query keyword, perform a query in the preset template library to construct a basic task template group.

[0097] In this embodiment of the invention, a template query engine receives structured language description data and extracts the programming language type as the main query keyword. Then, it accesses the index system of a preset template library and performs precise matching based on the language name (e.g., Java, Python, Go, etc.). Next, it loads the corresponding language's basic task sequence from the template repository, such as a standardized task chain for Java: Maven build → unit testing → SonarQube quality inspection → Docker image packaging. Simultaneously, a version controller ensures that all loaded templates meet version management requirements, maintaining template traceability, and ultimately constructs a complete set of basic task templates.

[0098] S22. Use a structured language to describe the language version information and / or user project type in the data, filter the basic task template group, and generate a precise task template set.

[0099] In this embodiment of the invention, the precise task template set refers to task templates that are precisely matched to the specific characteristics of the project after multi-level filtering. Specifically, a conditional filter receives a basic task template group as input, while simultaneously extracting language version information and user project type (such as Web services, Function Compute, front-end applications, etc.) from structured language description data. Then, a second-level filtering process is performed to select applicable build instructions and tool versions based on language version numbers. A third-level filtering process is then performed to exclude inapplicable task steps based on project type characteristics. Finally, a precise task template set highly matched to the specific project environment is generated.

[0100] S23. When the structured language description data contains multiple programming languages, the task template sets corresponding to each programming language are aggregated to generate a unified multilingual project pipeline template.

[0101] In this embodiment of the invention, the multilingual project pipeline template refers to a unified template that integrates task flows from multiple languages. Specifically, a multilingual analyzer detects multilingual features in structured language description data. Then, a template aggregator loads the task template sets corresponding to each programming language. Next, referring to the rules preset by the platform administrator through the language-task mapping table, dependency analysis is performed on different language task subsets. Finally, a process orchestrator creates independent parallel execution branches for each language task set and performs unified coordination at the process level to generate a complete cross-language project pipeline template.

[0102] S24. Construct a task template set based on one of the following: basic task template group, precise task template set, and multilingual project pipeline template.

[0103] In this embodiment of the invention, a template selector intelligently determines the type of project based on its specific characteristics (including the number of languages, project complexity, environmental requirements, etc.): for simple projects using only one language, a basic task template set is directly used; for complex projects using only one language requiring precise configuration, a precise task template set is selected; and for multilingual projects, a multilingual project pipeline template is used. The selected templates, after being standardized, serve as the basic output material for process assembly, providing a complete set of task templates for subsequent pipeline generation modules.

[0104] Step 203: Perform task dependency analysis and process topology assembly based on the task template set to generate initial pipeline data.

[0105] Furthermore, step 203 may include the following sub-steps:

[0106] S31. Analyze the input-output dependencies between task nodes in the task template set and generate a task dependency topology graph.

[0107] In embodiments of the present invention, such as Figure 4 The diagram shown illustrates the structure of the pipeline generator. The task dependency topology graph refers to a directed graph structure reflecting the dependencies between tasks. By parsing the input and output definitions of each task node in the task template set, a dependency matrix between tasks is established. A graph builder uses task nodes as vertices and dependencies as directed edges to construct a directed acyclic graph (DAG) structure. The generation-consumption relationships of task artifacts are identified; for example, the output artifact of a compilation task is an input dependency of a test task. Finally, a task dependency topology graph containing complete dependency information is generated. This topology graph transmits data through the API service layer, providing foundational data support for subsequent process scheduling.

[0108] S32. Sort the task dependency topology graph to generate the initial flow control structure.

[0109] In this embodiment of the invention, the initial flow control structure refers to the flow framework after topological sorting. Specifically, the task dependency topology graph is processed by a topology sorting engine using Kahn or DFS sorting algorithms to determine the linear execution sequence of task nodes. Independent task nodes without path dependencies in the graph are identified by a parallelization analyzer and marked as executable in parallel. Based on the sorting and parallelization analysis results, a flow control structure including serial and parallel stages is constructed. This control structure is visualized through a web front-end, constructing a flowchart showing each task node and its dependencies. The system retains a user interaction window when generating the flowchart, allowing users to customize and modify the task order, insert additional steps, or set intermediate trigger conditions.

[0110] S33. When the structured language describes data containing multiple programming languages, the initial flow control structure is processed by parallel branch aggregation to generate the target flow control structure.

[0111] In this embodiment of the invention, the target flow control structure refers to the flow framework optimized for multiple languages. Specifically, a language branch recognizer analyzes the multilingual information in the structured language description data to identify task subsets belonging to different programming languages. An independent execution branch is created for each language subset, and the dependencies between branches are analyzed. For language branches without cross-dependencies, the parallelization optimizer sets them to parallel execution mode. The execution timing and resource allocation of each branch are coordinated to ensure the correctness of parallel execution, ultimately generating a unified and coordinated multilingual target flow control structure.

[0112] S34. Combine the target flow control structure with the task configuration parameters in the task template set to construct the initial pipeline data.

[0113] In this embodiment of the invention, the initial pipeline data refers to the pipeline definition that has been assembled but not yet adjusted. Specifically, pre-configured task parameters, including build commands, timeout controls, image environments, and resource requirements, are extracted from the task template set. The target flow control structure is bound to the specific task configuration to generate an executable pipeline instance. The pipeline instance is converted into a standardized YAML or JSON format. Pipeline metadata is stored in MySQL, execution status is cached in Redis, and pipeline event messages are processed in Kafka. The final generated initial pipeline data can be published to a PaaS platform or application store, supporting integration with other third-party systems.

[0114] Step 204: Use the real-time context data corresponding to the software project to dynamically adjust the execution process of the initial pipeline data and generate the target pipeline data.

[0115] Furthermore, step 204 may include the following sub-steps:

[0116] S41. Perform rule-based determination on the project directory structure in the real-time context data corresponding to the software project, and generate task start / stop decision data based on the task nodes in the initial pipeline data that are dynamically started / stopped.

[0117] In this embodiment of the invention, task start / stop decision data refers to task execution control information obtained based on project structure analysis. Based on conditions such as whether a module contains a test directory, whether Sonar scanning is configured, and whether container deployment is enabled, corresponding tasks are automatically started or skipped, generating task start / stop decision data. Specifically, the project's directory structure information, including the source code directory, test directory, and configuration file directory, is acquired in real time. Then, a predefined start / stop rule library is loaded through a rule engine, and a condition matcher matches the directory structure with the rule library to determine the specific task nodes that need to be started or skipped. Finally, task start / stop decision data containing task start / stop status information is generated.

[0118] S42. Analyze the task dependencies in the initial pipeline data, schedule task nodes without dependencies to execute in parallel, and generate task parallel execution data.

[0119] In this embodiment of the invention, parallel task execution data refers to task scheduling information after sorting and parallelization optimization. Specifically, the task dependency graph in the initial pipeline data is parsed to identify the input-output dependency chains between task nodes. Tasks are grouped according to task type (e.g., compilation, testing, packaging), and the node arrangement order is adjusted based on the task classification results. Independent task nodes without dependencies are detected and marked as executable in parallel, ultimately generating parallel task execution data containing the task execution order and parallelization scheme.

[0120] S43. According to the pre-configured strategy, retry or interrupt task nodes that fail or time out during the execution of the initial pipeline data, and generate task exception handling data.

[0121] In this embodiment of the invention, task exception handling data refers to task handling information for execution exceptions. Specifically, task nodes that encounter exceptions during execution are processed according to configurable failure retry strategies, timeout interrupt mechanisms, and failure alarm logic to generate task exception handling data.

[0122] S44. Using task start / stop decision data, task parallel execution data, and task exception handling data, the initial pipeline data is comprehensively corrected to generate the target pipeline data.

[0123] In this embodiment of the invention, the target pipeline data refers to the final pipeline after dynamic adjustment and optimization. Specifically, task start / stop decision data, task parallel execution data, and task exception handling data are received as the basis for correction. Based on this data, the initial pipeline is optimized in multiple dimensions, including applying task start / stop status, adjusting execution order and parallel strategies, and integrating exception handling mechanisms. Execution status monitoring points are configured for each task node to support real-time tracking of the execution process and post-execution log playback. Finally, target pipeline data with dynamic adjustment capabilities and fault tolerance mechanisms is generated to ensure that the pipeline can execute efficiently and stably under various operating environments.

[0124] Step 205: Associate and store the initial pipeline data or target pipeline data with the corresponding structured language description data to generate a custom template.

[0125] In this embodiment of the invention, a custom template refers to a reusable template that associates and saves successfully running pipelines with technology stack information. Specifically, this invention allows users to save the currently generated pipeline process as a named private template or upload it to the platform's public template library. Templates are categorized and named according to project type, language, and deployment method through a naming and grouping mechanism, and a version management mechanism is used to retain historical versions each time they are saved, allowing for comparison of differences and rollback. Simultaneously, a permission management mechanism supports three types of permissions: private template, team shared template, and platform public template, generating custom templates that can be quickly referenced by team members or other users.

[0126] Step 206: Based on the configuration request of the new project and its structured language description data, query and match in the stored custom templates to generate a template recommendation list.

[0127] In this embodiment of the invention, the template recommendation list refers to a set of applicable templates matched based on project characteristics. Specifically, a quick reference entry allows users to directly select an existing template from the template list to generate a process when creating a new project. It also supports flowchart preview and structure cloning functions for all templates. The template recommendation list is generated by intelligently matching stored templates based on the technology stack characteristics of the new project.

[0128] Step 207: Apply the template selected by the user from the template recommendation list to the new project and generate the initialization pipeline data for the new project.

[0129] In this embodiment of the invention, the user-selected template configuration parameters and process structure are applied to a new project. This mechanism significantly improves the reusability of multiple projects, reduces the operational burden of repetitive configuration, and generates initialization pipeline data for the new project.

[0130] Step 208: Register an external plugin that contains new programming language recognition rules to generate a language recognition rule set.

[0131] In this embodiment of the invention, the language recognition rule set refers to the set of language recognition capabilities added through plugin extensions. Specifically, by parsing the JSON configuration file or script in the external plugin, the recognition rules and configuration file structure features of the new programming language are extracted, and these rules are transformed into a data structure that the system can recognize, generating a standardized language recognition rule set.

[0132] Step 209: Register external plugins that include task templates for new programming languages ​​or build tools to generate an extended collection of task templates.

[0133] In this embodiment of the invention, the task template extension set refers to the task template resources added through plugin extensions. Specifically, by parsing the build commands, test commands, and their configuration parameters defined in external plugins, these command sets are classified and organized according to specific language types to form system-callable task templates, generating a complete task template extension set.

[0134] Step 210: Integrate and hot-load the language recognition rule set and task template extension set into a system to generate an extended system that supports automatic recognition and process configuration of new programming languages.

[0135] In this embodiment of the invention, the extended system refers to an enhanced system with new language support capabilities. Specifically, a hot-plug mechanism is used to inject the language recognition rule set into the rule base of the language recognition module, while the task template extension set is registered in the template engine's index library. During the integration process, the compatibility detection module automatically verifies the syntax and structural legality of the plugins to ensure data integrity and consistency. The system automatically binds documentation and example projects to each plugin, establishing complete metadata information. Finally, an extended system supporting automatic recognition and process configuration of new programming languages ​​is generated through a dynamic loading mechanism, enabling the platform to quickly adapt to emerging languages ​​and toolchains.

[0136] Example 3

[0137] Please see Figure 5 , Figure 5 This is a structural block diagram of an adaptive configuration system for a pipeline process provided in Embodiment 3 of the present invention.

[0138] This invention provides an adaptive configuration system for pipeline processes, comprising:

[0139] The description data generation module 501 is used to acquire source code data of software projects, perform static analysis and feature extraction on the source code data, and generate structured language description data.

[0140] The template set generation module 502 is used to query and match data described in a structured language in a preset template library to generate a task template set.

[0141] The initial pipeline data generation module 503 is used to perform task dependency analysis and process topology assembly based on the task template set to generate initial pipeline data.

[0142] The target pipeline data generation module 504 is used to dynamically adjust the execution process of the initial pipeline data using real-time context data corresponding to the software project, and generate target pipeline data.

[0143] Furthermore, the description data generation module 501 can perform the following steps:

[0144] Extract and parse predefined language configuration files from the source code data to generate preliminary language description data;

[0145] Perform statistical and weighted calculations on the source code files in the source code data to generate weighted distribution data;

[0146] When the source code data contains multiple sub-modules, recursively traverse the directory of each sub-module and perform language recognition separately to generate a multi-module language mapping table.

[0147] Structured language description data is generated by aggregating at least one of the following: preliminary language description data, weight distribution data, and multi-module language mapping table.

[0148] Furthermore, the template set generation module 502 can perform the following steps:

[0149] Using the programming language type in the structured language description data as the query keyword, a basic task template group is constructed by querying the preset template library.

[0150] The basic task template set is filtered by using a structured language to describe the language version information and / or user project type in the data, and then a precise task template set is generated.

[0151] When the structured language description data contains multiple programming languages, the task template sets corresponding to each programming language are aggregated to generate a unified multilingual project pipeline template.

[0152] Construct a task template set based on one of the following: a basic task template group, a precise task template set, and a multilingual project pipeline template.

[0153] Furthermore, the initial pipeline data generation module 503 can perform the following steps:

[0154] Analyze the input-output dependencies between task nodes in the task template set to generate a task dependency topology graph;

[0155] The task dependency topology graph is sorted topologically to generate the initial flow control structure.

[0156] When the structured language describes data containing multiple programming languages, the initial flow control structure is processed by parallel branch aggregation to generate the target flow control structure.

[0157] The target flow control structure is combined with the task configuration parameters in the task template set to construct the initial pipeline data.

[0158] Furthermore, the target pipeline data generation module 504 can perform the following steps:

[0159] The project directory structure in the real-time context data corresponding to the software project is judged according to rules, and task start and stop decision data is generated by dynamically starting and stopping task nodes in the initial pipeline data.

[0160] Analyze the task dependencies in the initial pipeline data, schedule task nodes without dependencies to execute in parallel, and generate task parallel execution data;

[0161] According to the pre-configured strategy, task nodes that fail or time out during the execution of the initial pipeline data are retried or interrupted, and task exception handling data is generated.

[0162] The initial pipeline data is comprehensively corrected by using task start / stop decision data, task parallel execution data, and task exception handling data to generate target pipeline data.

[0163] Furthermore, the system also includes:

[0164] The custom template generation module is used to associate and store initial pipeline data or target pipeline data with corresponding structured language description data to generate custom templates;

[0165] The template recommendation list generation module is used to query and match stored custom templates based on the configuration request of a new project and its structured language description data, and generate a template recommendation list.

[0166] The pipeline data generation module is used to apply the template selected by the user from the template recommendation list to the new project and generate the initial pipeline data for the new project.

[0167] Furthermore, the system also includes:

[0168] The language recognition rule set generation module is used to register external plugins containing new programming language recognition rules and generate language recognition rule sets.

[0169] The task template extension collection generation module is used to register external plugins that contain task templates for new programming languages ​​or build tools, and generate task template extension collections.

[0170] The extended system generation module is used to integrate and hot-load the language recognition rule set and task template extension set into a system, generating an extended system that supports automatic recognition and process configuration of new programming languages.

[0171] Please see Figure 6 , Figure 6 This is a structural block diagram of a computer device provided in Embodiment 4 of the present invention.

[0172] An electronic device according to an embodiment of the present invention includes: a memory 601 and a processor 602. The memory 601 stores a computer program. When the computer program is executed by the processor 602, the processor 602 executes the pipeline process adaptive configuration method as described in any of the above embodiments.

[0173] Memory 601 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Memory 601 has storage space 603 for program code 613 for performing any of the method steps described above. For example, storage space 603 for program code may include various program codes 613 for implementing the various steps in the methods described above. This program code may be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The program code may be compressed, for example, in a suitable form. When run by a computing processing device, this code causes the computing processing device to perform the various steps in the methods described above. This program code may be read from or written to one or more computer program products. These computer program products include program code carriers such as hard disks, CDs, memory cards, or floppy disks. The program code may be compressed, for example, in a suitable form. When this code is run by a computing device, it causes the computing device to execute the various steps in the pipeline process adaptive configuration method described above.

[0174] Embodiment 5 of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pipeline process adaptive configuration method as described in any of the above embodiments.

[0175] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0179] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for adaptive configuration of a pipelined process, characterized by, The method comprises the following steps: acquiring source code data of a software project, performing static analysis and feature extraction on the source code data, and generating structured language description data; querying and matching in a preset template library by using the structured language description data, and generating a task template set; performing task dependency relationship analysis and process topology assembly based on the task template set, and generating initial pipeline data; performing dynamic strategy adjustment on an execution process of the initial pipeline data by using real-time context data corresponding to the software project, and generating target pipeline data; the step of performing task dependency relationship analysis and process topology assembly based on the task template set, and generating initial pipeline data, comprises: performing input-output dependency relationship analysis between task nodes of the task template set, and generating a task dependency topology graph; topologically sorting the task dependency topology graph, and generating an initial flow control structure; when the structured language description data contains multiple programming languages, performing parallel branch aggregation processing on the initial flow control structure, and generating a target flow control structure; combining the target flow control structure and task configuration parameters in the task template set, and constructing initial pipeline data; the step of performing dynamic strategy adjustment on an execution process of the initial pipeline data by using real-time context data corresponding to the software project, and generating target pipeline data, comprises: performing rule judgment on a project directory structure in the real-time context data corresponding to the software project, to dynamically start and stop task nodes in the initial pipeline data, and generating task start-stop decision data; analyzing task dependency relationships in the initial pipeline data, and scheduling task nodes without dependency relationships for parallel execution, and generating task parallel execution data; performing retry or interruption processing on task nodes that fail to execute or exceed a timeout in the execution process of the initial pipeline data according to a preconfigured strategy, and generating task exception handling data; comprehensively modifying the initial pipeline data by using the task start-stop decision data, the task parallel execution data, and the task exception handling data, and generating target pipeline data.

2. The method of claim 1, wherein, The step of performing static analysis and feature extraction on the source code data, and generating structured language description data, comprises: extracting and parsing a predefined type of language configuration file in the source code data, and generating preliminary language description data; performing statistics and weighted calculation on source code files in the source code data, and generating weight distribution data; when the source code data contains multiple sub-modules, recursively traversing sub-module directories and respectively performing language recognition, and generating a multi-module language mapping table; based on at least one of the preliminary language description data, the weight distribution data, and the multi-module language mapping table, aggregating to generate structured language description data.

3. The method of claim 1, wherein, The step of querying and matching in a preset template library by using the structured language description data, and generating a task template set, comprises: taking a programming language type in the structured language description data as a query keyword, querying in a preset template library, and constructing a basic task template group; Filter the basic task template set according to the language version information and / or user project type in the structured language description data, to generate an accurate task template set; When the structured language description data contains multiple programming languages, aggregate the task template sets corresponding to each programming language to generate a unified multilingual project pipeline template; Construct a task template set based on one of the basic task template set, the accurate task template set, and the multilingual project pipeline template.

4. The method of claim 1, wherein, The method further comprises: Store the initial pipeline data or the target pipeline data in association with the corresponding structured language description data to generate a custom template; Based on the configuration request of a new project and its structured language description data, query and match in the stored custom template to generate a template recommendation list; Apply the template selected by the user from the template recommendation list to the new project to generate the initial pipeline data of the new project.

5. The method of claim 1 or 4, wherein, The method further comprises: Register an external plug-in containing new programming language recognition rules to generate a language recognition rule set; Register an external plug-in containing a new programming language or a build tool task template to generate a task template expansion set; Systematically integrate and hot load the language recognition rule set and the task template expansion set to generate an extension system that supports automatic recognition and process configuration of new programming languages.

6. A system for adaptive configuration of a pipelined process according to any of the methods of claims 1-5, characterized in that, Comprise: A description data generation module for obtaining source code data of a software project, performing static analysis and feature extraction on the source code data, and generating structured language description data; A template set generation module for querying and matching in a preset template library using the structured language description data to generate a task template set; An initial pipeline data generation module for performing task dependency analysis and process topology assembly based on the task template set to generate initial pipeline data; A target pipeline data generation module for performing dynamic strategy adjustment on the execution process of the initial pipeline data using real-time context data corresponding to the software project to generate target pipeline data.

7. An electronic device, comprising: A memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the pipeline process adaptive configuration method according to any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed to implement the pipeline process adaptive configuration method according to any one of claims 1-5. The computer program is executed to implement the pipeline process adaptive configuration method according to any one of claims 1-5.

Images