DevOps-oriented containerized test environment deployment system and method

Abstract

The invention relates to a DevOps-oriented containerized test environment deployment system and method, and belongs to the technical field of Internet. A multi-service test environment is defined through a YAML file, environment variables are managed in combination with an. Env configuration file, dependency is optimized through dependency layering and a cache library, a container mirror image is constructed, and a container cluster is started; managing multiple tasks through a centralized scheduling architecture, and realizing resource isolation of a container cluster by using Cgroup; adjusting the number of container instances in real time based on a capacity expansion and contraction mechanism; a bridging network is used for automatically distributing a container IP, a built-in domain name system analyzes a service name dynamic discovery service, and rapid reconstruction of a containerized test environment is achieved. According to the method, a YAML and environment variable separation mechanism is adopted, real-time loading of development, testing and production environment configuration is supported, the deployment time is shortened to the second level, the resource utilization rate is increased to 90% from 60%, the multi-service communication difference is eliminated, and efficient interconnection between containers is ensured.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and in particular to a DevOps-oriented containerized test environment deployment system and method. Background Art

[0002] In today's software development practice, Development & Operations (DevOps) is a collective term for a set of processes, methods and systems that are used to promote communication and collaboration between application / software engineering development, technical operations and quality assurance personnel. The core of DevOps is to connect development and operations so that they can communicate and collaborate efficiently to shorten the software development cycle and deliver speed and quality. A complete DevOps platform that integrates development and operations can effectively improve software development efficiency.

[0003] Traditional DevOps has the following problems: lack of unified standardization in development, testing and production environments; heterogeneity of the underlying basic environment, and the diversification of the basic environment have caused resistance in the process from Dev to Ops; it is not easy to build, migrate and deploy. In order to solve the problems of traditional DevOps, a variety of containerized test environment deployment methods are used to improve it, such as test environments deployed on physical servers, test environments built based on virtualization technology, and test environments built using IaaS resources provided by cloud service providers, but there are still limitations.

[0004] For the traditional way of deploying the test environment on physical servers, from the implementation process point of view, it is necessary to directly install the operating system, middleware and application software on the physical hardware to build the target environment. Its core defects are reflected in the following dimensions: Deployment efficiency bottleneck: From hardware preparation to system configuration, manual operations are required. The whole process is time-consuming and long, and a single deployment often takes several hours or even longer; High complexity of operation and maintenance: Professional teams are required to continuously invest in equipment maintenance, system upgrades and troubleshooting, and at the same time, they need to bear continuous expenses such as computer room rental and power supply; Insufficient resource utilization: Hardware resources are in an inefficient operating state for a long time (for example, CPU utilization is usually less than 20%), and a single server can usually only support a single test environment, lacking virtualization technology support; Poor expansion flexibility: The fixed nature of hardware configuration requires the purchase of new equipment for environment expansion, and complex network adaptation and system debugging are required, making it difficult to achieve flexible resource elastic scaling; High initial investment pressure: A large amount of capital is required in the early stage to purchase hardware facilities such as servers and storage devices, and at the same time, there is the risk of asset depreciation caused by rapid depreciation of equipment.

[0005] With the widespread application of cloud computing and containerization technologies, the shortcomings of traditional physical deployment models in agile delivery, cost control and resource optimization have become increasingly obvious, and it is difficult to meet the dual requirements of modern software testing for efficiency and economy. The test environment built based on virtualization technology (such as VMware, Hyper-V or KVM) has obvious optimization compared with pure physical deployment solutions by creating multiple isolated virtual instances on physical servers. However, there are still significant limitations: each virtual instance needs to run an independent operating system kernel in full, resulting in repeated occupation of memory resources and a long system initialization time, which is difficult to adapt to high-frequency iteration test scenarios. In terms of operation and maintenance management, multiple independent systems need to continuously perform patch updates, dependency version alignment and security maintenance. Operation differences can easily cause "configuration drift" problems, resulting in a disconnect between the test environment and the production environment standards. Although virtual machine snapshots and template cloning technologies can partially achieve rapid replication of the environment, the generated image files are huge (often reaching GB level), and there are still efficiency constraints in storage transmission and version iteration. In the microservice architecture and continuous integration scenarios, its resource density and agility have gradually been surpassed by containerization solutions.

[0006] Use IaaS resources provided by cloud service providers (such as AWS, Azure, etc.) to build a test environment, and quickly build and manage the environment by dynamically allocating infrastructure. Although this solution has the advantages of high flexibility and reduced operation and maintenance complexity, there are still significant challenges: differences in resource configuration in different regions can easily lead to environmental baseline deviations, cross-platform system integration may affect test efficiency due to network transmission delays, and the complexity of cloud service configuration strategies increases the management burden. In addition, although the pay-as-you-go model reduces initial investment, the long-term operating costs increase significantly, and the user's ability to control the underlying resources is limited.

[0007] To sum up, although there are containerized test environment deployment methods to solve the problems of traditional DevOps, there are still problems such as low deployment efficiency, insufficient resource utilization, and poor environmental consistency. Summary of the invention

[0008] To solve the above technical problems, the present invention provides a DevOps-oriented containerized test environment deployment system and method, which realizes rapid environment reconstruction, improves DevOps resource utilization and reduces delayed communication of multiple services through multi-service environment definition, elastic resource scheduling and automated bridging network configuration.

[0009] To achieve the above-mentioned purpose of the invention, an embodiment provides a DevOps-oriented containerized test environment deployment system, including: a containerized environment definition module, which is used to define a multi-service test environment through a YAML file of Docker Compose, use an environment variable configuration file to uniformly manage environment variables, and then manage dependencies based on dependency layering and dependency cache library to build a container image; The elastic resource scheduling module is used to start container clusters based on container images, manage multi-task execution through a centralized scheduling architecture in the test environment, use Cgroups to achieve resource isolation of container clusters, and trigger the expansion and contraction mechanism according to real-time load indicators to dynamically adjust the number of container instances; The automated network configuration module is used to define the multi-service test environment in the containerized environment definition module, automatically allocate container IP addresses using a bridge network, and establish a domain name system to resolve service names, provide service discovery capabilities for container instances in the elastic resource scheduling module, and implement dynamic service discovery and communication links between containers.

[0010] In one embodiment, the use of an environment variable configuration file to uniformly manage environment variables includes: Integrate environment variables that were originally scattered in multiple configuration files into a single environment variable configuration file .env, where the environment variable configuration file .env contains database connection, service port or authentication key parameters for storing sensitive information or common configuration parameters; Automatically load sensitive information or common configuration parameters in the environment variable configuration file .env and pass them to the service to complete dynamic adaptation and injection of environment variables; By modifying sensitive information or common configuration parameters in the environment variable configuration file .env, the development, testing, and production environments can be switched.

[0011] In one embodiment, the dependency management is based on dependency layering and dependency cache library, including: dividing the dependencies into a basic layer, an intermediate layer and an application layer, storing the dependencies and metadata in all layers by building a central cache library, and when the dependencies are pulled, the dependencies are obtained from the local or proximal library first, and when the dependency pull does not hit, it is returned to the central cache library to achieve dependency management.

[0012] In one embodiment, the base layer is used to form a base image including an operating system and a language runtime. The operating system provides the underlying environment required for the container cluster runtime; the language runtime provides the code execution environment for the container cluster runtime; The intermediate layer is used to integrate the runtime dependencies and basic functional components of the container cluster based on the basic layer; The application layer is used to load the code required for running based on the basic layer and the intermediate layer, and set the command to execute the application when the container cluster is started.

[0013] Furthermore, the underlying environment includes hardware, file system structure, operating system library and basic tool set; the code execution environment includes standard class library, bytecode interpreter, thread scheduling and concurrency control.

[0014] In one embodiment, the metadata includes dependent versions, hash values, and compatibility tags.

[0015] In one embodiment, an incremental update mechanism is also established in the constructed central cache library. During the dependency management process, when a change is detected in the content of the dependency layer, resulting in a hash value that is inconsistent with the hash value recorded in the central cache library, a dependency upload or download operation is initiated; if no change occurs, the existing dependency cache is directly reused.

[0016] In one embodiment, the management of multi-task execution through a centralized scheduling architecture in a test environment includes: when multiple tasks are submitted, the master node sorts the tasks by priority and stores them in a task queue, the scheduler obtains pending tasks by periodically polling the queue, and screens available nodes based on three dimensions: label matching, real-time status, and resource load, and then selects the optimal node to execute the task based on a preset strategy; wherein the preset strategy includes load balancing priority, label matching priority, or physical location proximity allocation.

[0017] In one embodiment, the resource isolation of containers using Cgroups includes two control strategies at the container orchestration definition level and the underlying resource control level; The container orchestration definition layer is used to set static resource boundaries for each service using the Docker Compose YAML file to prevent a single container from over-occupying host resources and causing system instability. At the underlying resource control level, dynamic allocation of CPU resources and memory resources is achieved through Cgroups, wherein CPU dynamic resource allocation uses parameters to set relative weights between containers, and dynamic memory resource allocation directly sets absolute value boundaries through configuration files.

[0018] In one embodiment, the CPU dynamic resource allocation utilizes parameters to set relative weights between containers, including: defining CPU resource competition priorities by configuring container parameters, and when Cgroups detects CPU resource contention, it will forcibly limit the occupied time slices, so that the processes in the container enter a waiting state until the next scheduling cycle obtains the allocated CPU time slice, so that each container dynamically allocates CPU resources according to the weight value ratio; The dynamic allocation of memory resources directly sets the absolute value boundary through the configuration file, including: setting the maximum memory resource of the container through the configuration file, when the container memory resource usage reaches the threshold, Cgroups forcibly triggers the memory resource recovery operation and adjusts the threshold.

[0019] In one embodiment, the triggering of the expansion and contraction mechanism according to the real-time load index includes: Integrate the time series database and container monitoring components during data collection, collect container resource usage indicators in real time, and build a container resource consumption model to monitor container resources. The resource usage indicators include CPU resource utilization, memory usage, and network throughput. Define resource usage thresholds based on business needs and container resource monitoring results, declare alarm conditions through structured rule configuration files, write logical expressions in combination with query language, set duration constraints to exclude instantaneous interference, and monitor alarm events. The threshold is when the container CPU resource usage rate continues to exceed 80% or the memory usage exceeds 90%; The triggered alarm events are aggregated, deduplicated and prioritized, and multi-channel distribution is achieved through the alarm management module, while duplicate alarms are suppressed and the processing flow of alarms of different priorities is matched.

[0020] In one embodiment, the expansion and contraction mechanism includes: Horizontal expansion is used to create a new container instance, update the load balancing configuration, automatically add the new container instance IP to the backend list, and use the built-in domain name system service to automatically register the service name for the new container instance when the CPU resource usage of a service container instance exceeds 80% or the memory resource usage exceeds 90%. Scaling and exiting are used to mark idle container instances that are not assigned new requests when the CPU resource usage of a service container instance is continuously below 20% and the idle time exceeds the predefined threshold. A termination signal is sent to the target instance to remove the terminated instance and automatically update the service discovery record so that traffic is no longer routed to the terminated instance.

[0021] The present invention also provides a DevOps-oriented containerized test environment deployment method, and the system for DevOps-oriented containerized test environment deployment is applied, comprising the following steps: Developers define multi-service test environments through Docker Compose's YAML files, use .env files to uniformly manage environment variables, and then manage dependencies based on dependency layering and dependency cache libraries to build container images. Start the container cluster based on the built container image and orchestration configuration. In the test environment, manage multi-task execution through a centralized scheduling architecture, use Cgroups to achieve resource isolation of the container cluster, and trigger the expansion and contraction mechanism according to real-time load indicators to dynamically adjust the number of container instances. Based on the multi-service definition in the containerized environment definition module, the bridge network is used to automatically allocate container IPs, and a domain name system is established to resolve the service name, providing service discovery capabilities for container instances in the elastic resource scheduling module, realizing dynamic service discovery and communication links between containers, and ensuring service operation.

[0022] Compared with the prior art, the present invention has the following beneficial effects: (1) A multi-service environment definition method based on Docker Compose is used, and the .env file is used to uniformly manage environment variables, so that version changes can be loaded in real time during the development, testing, and production processes, and the test environment can be optimized from hundreds of seconds to seconds, and the test environment can be quickly rebuilt.

[0023] (2) Combining dynamic expansion and contraction with Cgroups resource isolation, we can avoid resource utilization contention and increase resource utilization from 60% to 90%.

[0024] (3) Introduce a bridge network for automated network configuration, with built-in domain name system resolution and service discovery to ensure low-latency communication among multiple services and eliminate differences in containerized test environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0026] Figure 1 It is a structural diagram of the DevOps-oriented containerized test environment deployment system provided by the present invention.

[0027] Figure 2 This is a flow chart of dependency cache optimization in dynamic deployment provided by the present invention.

[0028] Figure 3 This is a schematic diagram of the principle of implementing dynamic expansion and contraction of Agent any provided by the present invention.

[0029] Figure 4 This is a flow chart of real-time monitoring and indicator collection provided by the present invention.

[0030] Figure 5 This is a schematic diagram of the structure of the automated network configuration module provided by the present invention.

[0031] Figure 6This is a schematic diagram of the bridge network provided by the present invention.

[0032] Figure 7 A flow chart of the DevOps-oriented containerized test environment deployment method provided by the present invention.

[0033] Figure 8 A schematic diagram of a fully automated CI / CD pipeline based on Jenkins provided by the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the following is a The present invention is further described in detail in the following examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0035] In order to realize the rapid reconstruction of the DevOps containerized test environment, improve the DevOps resource utilization and reduce the delayed communication of multiple services, the embodiment provides a DevOps-oriented containerized test environment deployment system, such as Figure 1 As shown, it includes: a containerized environment definition module, an elastic resource scheduling module and an automated network configuration module.

[0036] Containerized environment definition module: In the field of automated test environment configuration management, traditional environment variable management methods have significant systemic defects: traditional technologies store environment variables in heterogeneous configuration files such as properties, yaml, and json, forming three independent entities: "development environment configuration class", "test environment configuration class", and "production environment configuration class". This architecture leads to: 1) a high rate of repeated definition of configuration items; 2) a long time-consuming cross-environment configuration synchronization; 3) a high error rate in manual maintenance.

[0037] The present invention defines a multi-service test environment through the YAML file of Docker Compose, establishes an "environment configuration centralization" class structure, uses the .env file to uniformly manage environment variables, and integrates the environment variables originally scattered in multiple configuration files into a single .env file entity. The .env file entity attributes include core parameters such as database connection, service port, and authentication key; the DockerCompose class realizes automatic loading of configuration through the load_env() method, and completes dynamic injection of environment variables in combination with the construct() method of service classes such as Nginx and MySQL; three subclasses, DevelopmentConfig (development), TestingConfig (testing), and ProductionConfig (production), are derived through the inheritance mechanism, and the environment switching can be completed by modifying the ENV_MODE variable.

[0038] Then, based on dependency layering and dependency cache library management, lightweight base images such as Alpine Linux are built to reduce the image size to hundreds of MB. Specifically, based on traditional containerized packaging, layered dependency packaging and intelligent cache library construction technology are proposed, such as Figure 2 As shown in the figure, in the containerized application deployment scenario, after the developer submits the code to the dependency version repository, the CI / CD system automatically triggers dependency analysis: if the dependency layer is not cached, the system builds a new dependency layer and pushes it to the image repository; if it is cached, the cache layer is directly pulled and assembled with the business code layer into a complete image. Subsequently, the operation and maintenance personnel inject dynamic environment variables through the configuration management tool, and finally the container orchestration platform automatically starts the container instance.

[0039] Among them, by dividing the dependencies into the base layer (operating system, runtime environment), the middle layer (framework, shared library), and the application layer (business code), each layer is independently built and generates a unique hash identifier, and modular management of software dependencies is achieved based on the layered architecture and containerized construction strategy. For example, in the multi-stage Docker build process, the base layer (base) is first established to install the OpenJDK 17 runtime environment based on the Alpine Linux 3.18 image to form a basic image containing the operating system and language runtime. The operating system provides the underlying environment required for the container cluster runtime, including hardware, file system structure, runtime system library and basic tool set; the language runtime provides the code execution environment for the container cluster runtime, including standard class libraries, bytecode interpreters, thread scheduling and concurrency control. The middle layer (libs) inherits the base layer environment, and deploys the JAR package in the project shared library directory to the / app / libs path through file copy operations, completing the integration of framework dependencies and public components when the container cluster is running. The final application layer (app) loads the business source code based on the first two layers, copies the project source code directory completely to the / app / src path, and sets the command to execute the Java application when the container starts.

[0040] Through the dynamic layering strategy, an independent image layer with a unique hash value is generated in each build stage. This layering mechanism ensures that when the content of a certain layer has not changed, the downstream build can directly reuse the cache layer, significantly improving the efficiency of continuous integration. The introduction of hash identifiers not only enables accurate tracking of image versions, but also ensures the repeatability and security of the build process through layer verification. This design pattern effectively separates the build process of infrastructure, common components, and business logic, which not only reduces the coupling between different layers, but also provides a standardized environmental benchmark for containerized deployment. It is particularly suitable for cloud-native applications and microservice architecture scenarios that require rapid iteration. Dependency layers with the same hash can be reused across projects, reducing 90% of repeated build overhead.

[0041] In addition, during the build process, dependency tree analysis tools (such as Maven Dependency Plugin or Python's pipdeptree) automatically detect version conflicts and recommend compatible versions. If the conflict cannot be automatically resolved, an alarm is triggered and a visual report is generated to prompt the developer to intervene.

[0042] For dependency cache libraries, build an enterprise-level central cache repository (such as Nexus or a custom HTTP server) to store compressed packages and metadata (versions, hashes, compatibility tags) of all layered dependencies. When pulling dependencies, first obtain them from local or near-end repositories. When dependency pulls miss, return to the central cache library to manage dependencies and reduce external network dependencies. And establish an incremental update mechanism. Only when it is detected that the content in the dependency layer has changed, resulting in a hash value that is inconsistent with the hash value recorded in the central cache library, will the dependency upload or download operation be initiated; if there is no change, the existing dependency cache will be reused directly.

[0043] The containerized environment definition module implements cross-environment dependency management capabilities and provides an environment-aware dependency loading layer. Its core goal is to automatically adapt differentiated dependency configurations according to different deployment stages. The environment-aware dependency loading layer identifies the running environment identifier (such as development, test, and production) and intelligently selects the corresponding dependency resources - for example, injecting a Mock service library in the test environment to isolate external dependencies, and loading performance-optimized and security-hardened component packages in the production environment, thereby achieving the best match between environmental characteristics and dependency versions. This dynamic switching mechanism is usually implemented in conjunction with environment variable configuration files (such as .env files), in which dependency parameters specific to each environment are pre-defined (such as the development environment uses a JDK image with integrated debugging tools, and the production environment uses a smaller Alpine streamlined image). The build tool dynamically selects the base image or dependency version by reading the variable value during the execution phase to ensure strong consistency between environmental characteristics and resource configuration.

[0044] In terms of container runtime maintenance, hot replacement technology is also introduced to improve service availability. By mounting dynamic link libraries (such as Python's .so files) to writable storage volumes in the container, or using Sidecar containers to synchronously update dependency files, version changes of key dependency libraries can be loaded in real time without interrupting services. It effectively avoids the operational bottleneck of traditional dependency updates that require restarting containers, which not only ensures business continuity, but also provides flexible technical support for scenarios such as grayscale releases and A / B testing. The overall design uses environment-sensitive configuration management and non-intrusive update strategies to enhance the system's adaptability to continuous delivery processes while improving deployment flexibility. It is particularly suitable for agile development and operation and maintenance systems that require multiple environments to operate in parallel.

[0045] Through the layered dependency architecture, intelligent cache library and dynamic environment adaptation technology, revolutionary optimization of dependency management is achieved, solving the core pain points of traditional solutions such as bloated images, inefficient construction, and inconsistent environments, providing an efficient, stable and scalable dependency management paradigm for containerized test environment deployment.

[0046] Elastic resource scheduling module: Utilizes the lightweight isolation mechanism of Docker container to start the test environment within 5 seconds. In the actual configuration scenario, the complete process of building Nginx image through Dockerfile is shown. The Dockerfile class defines all the configuration information required for container construction, including basic image settings, maintainer label declaration, working directory specification, static file copy operation, configuration file override settings, port exposure declaration and container startup command definition, and includes a run command method for installing dependencies. The Nginx image class represents the final build product, which indicates its association with Dockerfile through the built-from-Dockerfile attribute. The two arrows indicate that Dockerfile generates the dependency relationship of Nginx image by executing the defined build instructions, which fully presents the generation process from build configuration to the final image.

[0047] In addition, the number of container instances is automatically adjusted according to the test load, and resource utilization is increased to 90%. For example, based on the Master-Agent mechanism of Jenkins, Agent any is used to achieve dynamic expansion and contraction. Figure 3 The figure shows the schematic diagram of Agent any, which realizes efficient execution of distributed tasks through centralized scheduling architecture and dynamic resource allocation strategy, mainly including four core stages: task submission, queue management, node scheduling and result processing. The system adopts a master-slave design. When users or continuous integration systems submit tasks, they carry metadata such as execution priority, hardware label constraints, and number of failed retries. The master node (Master) sorts the tasks by priority and stores them in the job queue (JobQueue) for scheduling. The scheduler (Scheduler) obtains pending tasks by periodically polling the queue, and selects available computing nodes (Agents) based on three dimensions: label matching (such as filtering nodes with "linux" and "docker" features), real-time status (idle / busy) and resource load (CPU / memory utilization), and then selects the optimal node to execute the task based on the preset strategy of load balancing priority, label matching priority or physical location allocation.

[0048] When the task is then assigned, the master node marks the target Agent as "busy" to prevent duplicate assignments. The task execution results trigger different response paths: if the task is successfully completed, the Agent node automatically resets to "idle" and reports the latest resource load data, while clearing the local task cache to free up storage space; if the task fails, the system requeues the task according to the preset retry strategy (such as retrying up to N times). When the maximum number of retries is exceeded, an alarm notification is sent to the administrator and the final status of the task is marked as "failed". Based on this mechanism, it not only ensures the rapid response of high-priority tasks, but also avoids node overload through dynamic load evaluation. At the same time, the retry strategy and status tracking functions enhance the system's tolerance for temporary failures, forming a closed-loop task lifecycle management system, which is suitable for batch job scenarios that require elastic expansion and high availability.

[0049] In the embodiment, Cgroups is also used to limit the CPU resource and memory resource usage of the container to avoid resource contention, specifically: Refined resource management of containerized services is achieved through resource quota constraints and dynamic regulation mechanisms, which mainly include control strategies at two levels.

[0050] (1) At the container orchestration definition level, use the `docker-compose.yml` configuration file to set static resource boundaries for each service. For example, limit the web service to a maximum of 0.5 CPU cores and 512 MB of memory, or set a stricter runtime resource limit (such as 1 CPU core and 1 GB of memory) through the `deploy.resources.limits` field in Swarm cluster deployment mode. This prevents a single container from excessively occupying host resources and causing system instability. It also ensures that the service runs stably within the predetermined resource range, while providing a clear allocation basis for the resource scheduler to avoid resource contention among multiple services.

[0051] (2) At the underlying resource control level, dynamic resource adjustment capabilities are implemented through the Cgroups subsystem of the Linux kernel. CPU resource allocation uses the `cpu.shares` parameter to set the relative weights between containers, and memory resource limits are directly set through the `memory.limit_in_bytes` file to set absolute value boundaries.

[0052] For example, when a container attempts to use more CPU resources than the limit, the kernel will forcibly limit its occupied time slice, causing the process in the container to enter a waiting state until the next scheduling cycle obtains the allocated CPU resource time. The CPU resource utilization of the process will be "suppressed", which is manifested as slower application response or delayed task processing, but the container will not be terminated or restarted.

[0053] When the container memory usage exceeds the hard limit, the Linux kernel's OOM Killer is triggered and processes in the container are terminated to free up memory. By default, processes that occupy the most memory and have the lowest priority are terminated first (the main process of the container may be terminated directly, causing the container to exit). If the main process of the container is terminated, the container status will change to Exited and an error code 137 will be generated (indicating that the process was forcibly terminated by SIGKILL). You can view the container exit status through docker ps -a.

[0054] This setting allows operators to modify quota parameters in real time during container operation, such as temporarily increasing the CPU resource weight of computing-intensive tasks based on business load fluctuations, or implementing emergency capacity restrictions on memory leak services. The dynamic adjustment function is triggered through the file system interface or container management tools (such as `docker update`), and can take effect without interrupting the service process, significantly enhancing the system's resilience to sudden traffic and failure scenarios. This combination of dynamic and static resource management strategies not only ensures the reliability of basic resource isolation, but also gives the runtime environment full flexibility.

[0055] The resource usage status visualization and abnormal response capabilities of containerized services are achieved through the monitoring system and automated alarm mechanism, which mainly includes three core links.

[0056] First, at the data collection layer, by integrating the Prometheus time series database and the cAdvisor container monitoring component, we can obtain the container's fine-grained CPU resource utilization, memory resource usage, network throughput and other key performance indicators in real time to form a fine-grained resource consumption portrait of the container. As a monitoring agent embedded in the container, cAdvisor continuously exposes a standardized measurement data interface to Prometheus, providing a stable data source for subsequent analysis.

[0057] At the rule definition level, threshold policies are set based on business needs and system capacity planning. For example, an alarm condition is triggered when the container CPU resource usage exceeds 80% or the memory usage exceeds 90%. Structured declarations are made through Prometheus alarm rule configuration files (such as the `container_alerts` group), and logical expressions are written using the PromQL query language (such as counting the average CPU usage of Web services within 5 minutes), and duration constraints are set, such as exceeding the limit for 2 consecutive minutes to filter out instantaneous fluctuations. To ensure the accuracy and effectiveness of the alarm, the rule engine periodically evaluates indicator data, generates alarm events when the threshold is exceeded, and pushes them to the Alertmanager module.

[0058] In the alarm processing phase, the Alertmanager module aggregates, deduplicates and routes alarm events, supports notifying the operation and maintenance team through multiple channels such as email, Slack, Webhook, etc., and configures silent rules or hierarchical response strategies to avoid alarm storms. This monitoring and alarm linkage system not only realizes a closed loop from data collection to abnormal perception, but also combines historical data for trend analysis, provides decision-making basis for capacity planning and performance optimization, and significantly improves the observability and fault handling efficiency of distributed systems in complex load scenarios.

[0059] In the embodiment, Figure 4 As shown in the figure, through real-time monitoring and indicator collection, the expansion and contraction mechanism is triggered, including: horizontal expansion process, contraction process and graceful shutdown.

[0060] The trigger condition for horizontal scaling is when the CPU resource usage of a service instance exceeds 80% or the memory exceeds 90% for 5 consecutive minutes. When the trigger condition is met, call the Docker API to dynamically create a new instance: `docker-compose up -d--scale web=3`. Then update the load balancing configuration (such as Nginx Upstream) and use the automated network configuration module to add the new instance IP to the backend list. Finally, register the new instance service name (such as `web-service.test-net`) through the built-in domain name system of Docker to ensure that service discovery takes effect in real time.

[0061] The triggering conditions for the scaling-down process and graceful shutdown are that the instance CPU usage remains below 20% and the idle time exceeds 5 minutes. When the triggering conditions are met, the idle instance is marked and a SIGTERM signal is sent to notify the container to prepare to stop. Wait for 30 seconds (to process unfinished requests). If it does not exit automatically, send SIGKILL to force termination. Remove the instance from the Docker Compose configuration: `docker-compose down --scale web=1`, and use the automated network configuration module to update the service discovery record to ensure that traffic is no longer routed to the removed instance.

[0062] In the embodiment, the resource limit of the running container is also modified in real time through the `docker update` command, such as docker update --cpus 1.0 --memory 2G web_container_id; resource allocation is automatically optimized in combination with monitoring data, such as reducing CPU quota to save resources when the load is low, for dynamic resource adjustment.

[0063] In the implementation example, in terms of resource utilization optimization, elastic sharding is used to split high-load services into multiple lightweight instances, and requests are distributed through load balancing; hot and cold instance pools are used to maintain pre-started standby instance pools, which are directly activated during capacity expansion to reduce startup delays; priority scheduling is used to allocate higher CPU shares (`cpu.shares`) to critical services to ensure priority processing when resources are contended.

[0064] Automatic network configuration module: Figure 5 As shown in the figure, the container IP is automatically allocated based on the Docker bridge network, and the service name (such as db-service) is resolved through the built-in domain name system. Different from the traditional network, it can establish an automated domain name system resolution to realize the automation of domain names and IPs. Specifically, multi-service collaboration is realized based on Docker Compose container orchestration, and a dynamic communication link is built through the built-in domain name system service discovery mechanism: after the user triggers `docker-compose up`, service A / B containerization starts and automatically registers the `.test-net` domain name with the domain name system; when the user requests `serviceB.test-net`, the domain name system resolves and returns the target IP in real time, triggering the low-latency direct communication between service B and service A based on the container network, and finally forming a full closed-loop process of "user request → domain name system query → service collaboration → response return", realizing service registration automation, zero configuration of domain name system resolution, and efficient cross-container communication, and the communication delay between containers is less than 2ms, supporting multi-service collaborative testing; each deployment is generated from the same Compose file, eliminating manual configuration differences and ensuring environmental consistency.

[0065] In order to achieve multi-service collaboration, Figure 6 As shown in the figure, inter-service communication is achieved based on the Docker bridge network, and the service name (such as "Nginx" / "MySQL" / "Redis") is automatically resolved through the built-in domain name system service. Web services, database services, and cache services all directly access the target instance through the service name, forming a closed-loop process of "request→dynamic resolution→service routing→functional response". The core features are lightweight service discovery without IP hardcoding, efficient communication across container networks, and real-time effectiveness of domain name system resolution results.

[0066] After deployment, the service status is verified through the health check interface, and the test load (such as the number of concurrent requests) is monitored in real time. The number of container instances is dynamically adjusted (such as expanding from 2 API instances to 5); when scaling down, idle container resources are automatically released to avoid resource waste. Ultimately, a closed loop is formed from environment definition → resource allocation → network communication to support the efficient execution of test tasks.

[0067] The embodiment also provides a DevOps-oriented containerized test environment deployment method, such as Figure 7 As shown, the following steps are included: S1. Developers define multi-service test environments through Docker Compose's YAML files, use environment variable configuration files to uniformly manage environment variables, and then manage dependencies based on dependency layering and dependency cache libraries to build container images. S2. Start the container cluster based on the built container image and orchestration configuration. In the test environment, manage multi-task execution through a centralized scheduling architecture, use Cgroups to achieve resource isolation of the container cluster, and trigger the expansion and contraction mechanism according to real-time load indicators to dynamically adjust the number of container instances. S3. Based on the multi-service test environment defined in the containerized environment definition module, use the bridge network to automatically allocate container IPs, establish a domain name system to resolve service names, provide service discovery capabilities for container instances in the elastic resource scheduling module, implement dynamic service discovery and communication links between containers, and ensure service operation.

[0068] In the embodiment, the developer writes a docker-compose.yml file, defines the dependent test environment such as MySQL, Nginx, and API services through the YAML of Docker Compose, and configures environment variables in the .env file, such as DB_PASSWORD=test123, to separate sensitive information from code.

[0069] like Figure 8 As shown in the figure, the full-link configuration automation of CI / CD is realized through the deep integration of Jenkins Pipeline and Docker Compose. When the code is submitted to the Git repository, the Pipeline is triggered. Jenkins automatically parses the CI / CD pipeline, dynamically loads the docker-compose.yml and .env files, and injects multiple environment variables, such as database connection parameters for development / test / production modes, to achieve zero-delay environment switching. The corresponding configuration branch is dynamically selected through the ENV_MODE variable value without rebuilding the image; Docker builds the container image and uses Compose to orchestrate the Nginx and MySQL container clusters, while monitoring the test load (such as the number of concurrent requests) and dynamically adjusting the number of container instances (such as expanding from 2 API instances to 5); automatically releases idle container resources when scaling down to avoid resource waste.

[0070] When the container is started, a bridge network is automatically created, an IP is assigned to the container, and the service name (such as mysql-service.test-net) is automatically registered with the built-in domain name system. Jenkins obtains the service status in real time and updates the load balancing strategy through the REST API. Web services, database services, and cache services all directly access the target instance through the service name, forming a closed-loop process of "request→dynamic resolution→service routing→functional response", realizing lightweight service discovery without IP hard coding, efficient communication across container networks, and real-time effectiveness of domain name system resolution results; after the containerized deployment is completed, the health check interface Newman is automatically triggered to perform API test verification. If the health check fails after deployment, it automatically rolls back to the previous stable version of the Compose file. Through the deep integration of Jenkins Pipeline and Docker Compose, a complete link of "code change→image building→container orchestration→automated testing→feedback closed loop" is finally formed, realizing containerized isolated deployment, dynamic expansion of service orchestration, and seamless integration of test processes, reducing the deployment time of multiple service environments from several minutes of traditional solutions to seconds, and ensuring strict consistency of cross-environment configuration.

[0071] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A DevOps-oriented containerized test environment deployment system, characterized in that: include: The containerized environment definition module is used to define the multi-service test environment through the YAML file of Docker Compose, use the environment variable configuration file to uniformly manage the environment variables, and then manage the dependencies based on the dependency layering and dependency cache library to build the container image; The elastic resource scheduling module is used to start container clusters based on container images, manage multi-task execution through a centralized scheduling architecture in the test environment, use Cgroups to achieve resource isolation of container clusters, and trigger the expansion and contraction mechanism according to real-time load indicators to dynamically adjust the number of container instances; The automated network configuration module is used to automatically allocate container IP addresses based on the multi-service test environment defined in the containerized environment definition module using a bridge network, and to establish a domain name system to resolve service names, provide service discovery capabilities for container instances in the elastic resource scheduling module, and implement dynamic service discovery and communication links between containers.

2. The containerized test environment deployment system according to claim 1, characterized in that: The use of an environment variable configuration file to uniformly manage environment variables includes: Integrate environment variables that were originally scattered in multiple configuration files into a single environment variable configuration file .env, where the environment variable configuration file .env contains database connection, service port or authentication key parameters for storing sensitive information or common configuration parameters; Automatically load sensitive information or common configuration parameters in the environment variable configuration file .env and pass them to the service to complete dynamic adaptation and injection of environment variables; By modifying sensitive information or common configuration parameters in the environment variable configuration file .env, the development, testing, and production environments can be switched.

3. The containerized test environment deployment system according to claim 1, characterized in that: The described dependency management based on dependency layering and dependency cache library includes: dividing the dependencies into a basic layer, an intermediate layer and an application layer, storing the dependencies and metadata in all layers by building a central cache library, and when the dependencies are pulled, obtaining the dependencies from the local or proximal library first, and returning to the central cache library when the dependency pull misses, thereby realizing dependency management.

4. The containerized test environment deployment system according to claim 3, characterized in that: The basic layer is used to form a basic image including an operating system and a language runtime. The operating system provides the underlying environment required for the container cluster runtime; the language runtime provides the code execution environment for the container cluster runtime. The intermediate layer is used to integrate the runtime dependencies and basic functional components of the container cluster based on the basic layer; The application layer is used to load the code required for running based on the basic layer and the intermediate layer, and set the command to execute the application when the container cluster is started.

5. The containerized test environment deployment system according to claim 1, characterized in that: The management of multi-task execution through a centralized scheduling architecture in a test environment includes: when multiple tasks are submitted, the master node sorts the tasks by priority and stores them in the task queue, the scheduler obtains the tasks to be processed by periodically polling the queue, and screens the available nodes based on the three dimensions of label matching, real-time status and resource load, and then selects the optimal node to execute the task based on the preset strategy; wherein the preset strategy includes load balancing priority, label matching priority or physical location allocation.

6. The containerized test environment deployment system according to claim 1, characterized in that: The resource isolation of containers using Cgroups includes two control strategies at the container orchestration definition level and the underlying resource control level; The container orchestration definition layer is used to set static resource boundaries for each service using the Docker Compose YAML file to prevent a single container from over-occupying host resources and causing system instability. At the underlying resource control level, dynamic allocation of CPU resources and memory resources is achieved through Cgroups, wherein CPU dynamic resource allocation uses parameters to set relative weights between containers, and dynamic memory resource allocation directly sets absolute value boundaries through configuration files.

7. The containerized test environment deployment system according to claim 6, characterized in that: The CPU dynamic resource allocation utilizes parameters to set relative weights between containers, including: defining CPU resource competition priorities by configuring container parameters, and when Cgroups detects CPU resource contention, it will forcibly limit the occupied time slices, so that the processes in the container enter a waiting state until the next scheduling cycle obtains the allocated CPU time slice, so that each container dynamically allocates CPU resources according to the weight value ratio; The dynamic allocation of memory resources directly sets the absolute value boundary through the configuration file, including: setting the maximum memory resource of the container through the configuration file, when the container memory resource usage reaches the threshold, Cgroups forcibly triggers the memory resource recovery operation and adjusts the threshold.

8. The containerized test environment deployment system according to claim 1, characterized in that: The capacity expansion and contraction mechanism is triggered according to the real-time load index, including: Integrate the time series database and container monitoring components during data collection, collect container resource usage indicators in real time, and build a container resource consumption model to monitor container resources. The resource usage indicators include CPU resource utilization, memory usage, and network throughput. Define resource usage thresholds based on business needs and container resource monitoring results, declare alarm conditions through structured rule configuration files, write logical expressions in combination with query language, set duration constraints to exclude instantaneous interference, and monitor alarm events. The threshold is when the container CPU resource usage rate continues to exceed 80% or the memory usage exceeds 90%; The triggered alarm events are aggregated, deduplicated and prioritized, and multi-channel distribution is achieved through the alarm management module, while duplicate alarms are suppressed and the processing flow of alarms of different priorities is matched.

9. The containerized test environment deployment system according to claim 8, characterized in that: The expansion and contraction mechanism includes: Horizontal expansion is used to create a new container instance, update the load balancing configuration, automatically add the new container instance IP to the backend list, and use the built-in domain name system service to automatically register the service name for the new container instance when the CPU resource usage of a service container instance exceeds 80% or the memory resource usage exceeds 90%. Scaling and exiting are used to mark idle container instances that are not assigned new requests when the CPU resource usage of a service container instance is continuously below 20% and the idle time exceeds the predefined threshold. A termination signal is sent to the target instance to remove the terminated instance and automatically update the service discovery record so that traffic is no longer routed to the terminated instance.

10. A DevOps-oriented containerized test environment deployment method, characterized in that: A system for deploying a DevOps-oriented containerized test environment according to any one of claims 1 to 9, comprising the following steps: Developers define multi-service test environments through Docker Compose's YAML files, use environment variable configuration files to uniformly manage environment variables, and then manage dependencies based on dependency layering and dependency cache libraries to build container images. Start the container cluster based on the built container image and orchestration configuration. In the test environment, manage multi-task execution through a centralized scheduling architecture. Use Cgroups to isolate the resources of the container cluster. At the same time, trigger the expansion and contraction mechanism according to real-time load indicators to dynamically adjust the number of container instances. Based on the multi-service test environment defined in the containerized environment definition module, the bridge network is used to automatically allocate container IPs, and a domain name system is established to resolve service names. Service discovery capabilities are provided for container instances in the elastic resource scheduling module, and dynamic service discovery and communication links between containers are implemented to ensure service operation.

Images