System and method for realizing multi-terminal code construction

Abstract

The invention discloses a system and a method for realizing multi-terminal code construction, which realize automatic selection of matched server hardware resources according to the type of a construction project. The system for realizing multi-terminal code construction comprises a service layer for providing construction types of a plurality of development languages; the link layer is used for providing matched virtual server nodes according to environmental resource characteristics required by the construction types, and performing one-to-one mapping on the construction types and the virtual server nodes; the component layer is used for respectively providing assembly line components according to different construction types and constructing assembly line script nodes; and the hardware layer is used for providing a hardware virtualization environment for code execution compiling and packaging. By introducing a high-availability mechanism, the problem of single-point failure of a traditional single dispatching center is thoroughly solved. The virtualization technology allows a plurality of applications to share the same physical resource, hardware resources are utilized to the greatest extent through dynamic resource allocation and adjustment, and the cost is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of software development, and particularly relates to a system and method for realizing multi-terminal code construction. Background Art

[0002] In the process of software development, in order to improve development efficiency and enhance team collaboration, a series of steps from code writing to deployment of software are generally organized into a continuous process through automated tools and technologies. This process usually involves multiple stages, and each stage performs specific tasks, such as code construction, testing, packaging, deployment, etc.

[0003] Code construction is an important link in the software development process, which refers to the process of converting source code into an executable file or a deployable application. This process usually includes the following key steps: First, check out the latest source code from a version control system (such as Git, SVN, etc.), then convert the source code into a language or intermediate language that can be understood by a machine (such as Java bytecode), and finally combine the compiled binary files, configuration files, dependency libraries, etc. into a deployable package. For example, JAR files, WAR files, or Docker images. The purpose of construction is to ensure the correctness, consistency, and reliability of the software, and to ensure that the same process is followed for each build, thereby reducing human errors.

[0004] However, traditional code construction environments often focus on specific programming languages or technology stacks, such as Java, which leads to their limitations in dealing with multi-language projects. For example, if a CI tool is mainly built around Maven or Gradle, then the support for Python, Vue.js, React Native, or other mobile projects may be relatively limited or require additional configuration and plugins, which increases the difficulty of configuration and the potential error rate. In addition, if the construction environment is optimized only for one language, then when dealing with projects in different languages, it may not be able to efficiently utilize hardware resources. For example, Java projects may require a large amount of memory, while Python or JavaScript projects are more dependent on CPU computing power.

[0005] In addition, traditional construction solutions have the problem of single-point failure of the scheduling center. Once the scheduling service node fails, all component tasks cannot be executed. And the single-node method cannot make better use of resources, often resulting in insufficient bandwidth or IO read / write performance bottlenecks, leading to the situation where construction tasks queue up. Summary of the Invention

[0006] The present invention mainly focuses on the technical field of code construction. In view of the above technical problems, based on the centralized construction mode, the present invention proposes a system and method for realizing multi-terminal code construction, which can automatically select matching server hardware resources according to the type of construction project. At the same time, it supports multiple development languages such as Java, Python, Vue, Android, iOS, and Harmony.

[0007] To achieve the above object, the technical solution adopted by the present invention is: a system for realizing multi-terminal code construction, the overall architecture of which includes a service layer, a link layer, a component layer, and a hardware layer.

[0008] Service layer: Provide construction types of several development languages; Link layer: Provide matching virtual server nodes according to the environmental resource characteristics required by the construction type, and map the construction type to the virtual server node one by one; Component layer: Provide pipeline components respectively according to different construction types, and build pipeline script nodes; mainly including code checkout nodes, code scanning nodes, compilation nodes, and packaging nodes; the pipeline components are visual task orchestration components, which maintain a one-to-one correspondence with the pipeline script nodes. Users can freely build the pipeline components by dragging the components, and then automatically generate the pipeline script.

[0009] Hardware layer: Used to provide a virtualized server environment for code execution, compilation, and packaging. It includes operating systems such as Linux and MacOS. Virtualization technology allows multiple applications to share the same physical resources. Through dynamic resource allocation and adjustment, the hardware resources are utilized to the maximum extent, reducing costs.

[0010] The hardware layer includes physical servers for creating a virtual machine cluster, and virtual server nodes are created on the physical servers; Install ESXi on each physical server, and create a virtual switch vSwitch on each ESXi host for connecting virtual machines. The virtual server nodes on different physical servers are connected through virtual network communication; Use vSphere Client to connect to the ESXi host; create several virtual machines on each ESXi host, and configure the name, storage location, CPU, memory, and disk resources of the virtual machines for each development language on each ESXi host; Install the Linux operating system on the ESXi host; upload the ISO file of Linux to the data storage of the ESXi host; Virtual machine cluster configuration: Install the reverse proxy server Nginx and the high-availability service Keepalived on the physical server nodes; Keepalived forwards the build requests of the request side of the build interface to Nginx, and Nginx is used to proxy all build requests of the build type to ensure that requests accessing the virtual address VIP are correctly processed.

[0011] Since the build environments of each programming language are independent of each other, two nodes, Node1 and Node2, are created for each of the 6 programming languages to ensure resource isolation and high availability.

[0012] Furthermore, the physical server includes the first server Server1 and the second server Server2; create the first node Node1 on the first server Server1 and create the second node Node2 on the second server Server2; Node1 and Node2 each contain multiple of each build type. For example, there are nodes of 6 languages on Server1, and they are connected through virtual network communication; configure Keepalived as the master node on Server1 and configure Keepalived as the backup node on Server2.

[0013] Furthermore, the link layer maps the build type to the virtual server node one by one. The specific steps are as follows: First, maintain the IP and port of all open-source software project Jenkins service nodes, and then in the pipeline editing interface, select the corresponding Jenkins service address and save the mapping relationship to the database. When calling the service API, it is called in the form of VIP (virtual address) + Jenkins port. Here, the service API refers to the capability interface provided by the Jenkins service, mainly including the build script execution interface, the pipeline node execution result query interface, the build queue, and the server resource information acquisition interface.

[0014] Furthermore, the server environment is configured as follows: Install the open-source software project Jenkins on each node of each physical server for building, testing, and deploying code; install the Git plugin on each node of each physical server for code checkout; Build types include: Java, Python, iOS, Android, Vue, Harmony; JAVA node: Install the Java runtime environment JDK, and then install the dependency package management and compilation tool Maven; Python node: First install the Python runtime environment, and then install the setuptools tool for building, installing, uploading, and managing Python packages; Android Node: Install the Android running environment JDK, and then install the dependency package management and compilation tool Gradle; VUE Node: Install Node.js to provide an environment for running JavaScript code; install pnpm to manage the dependencies of JavaScript projects; Harmony Node: Install the Command Line Tools; this command line tool collection includes a series of tools used for HarmonyOS application development, including code linter codelinter, package management of third-party libraries ohpm, command line parsing hstack, and compilation and build hvigorw.

[0015] iOS Node: Install the Xcode integrated development environment (IDE) for compiling Swift and Objective-C code.

[0016] The present invention also discloses a method for implementing multi-terminal code construction, using the above-mentioned system for implementing multi-terminal code construction, including the following steps: Step 1, select the build type: Initialize several development language build types, including Java, Python, iOS, Android, Vue, and Harmony, and select the type to be built; Step 2, match the build environment: Provide a virtualized server environment, with 2 virtualized server environments corresponding to each development language. Provide different environment configurations according to different development languages. The general configurations include the operating system type, integrated tools, code checkout tools, compilation and packaging tools, CPU and memory configurations; each development language matches one virtualized server environment to establish a one-to-one mapping relationship; Step 3, select the build pipeline: The build pipeline includes a code checkout node, a code scanning node, a compilation node, and a packaging node; Generate a unique identifier binding the development language type when creating the build pipeline to implement matching the corresponding build pipeline through the build type; Step 4, execute the build pipeline: Push the corresponding build pipeline to the matching server environment through the Jenkins interface, trigger the build script, complete the packaging operation, and output the final product.

[0017] Furthermore, during the Jenkins interface call process, high availability and load balancing of requests are achieved through Keepalived. This method includes the following steps: Configure the virtual IP (VIP) rule in Keepalived to identify client requests; Store the correspondence between the virtualized server environment address and the Nginx port through the port mapping management module; During the Jenkins interface call process, first receive the build request from the client; Secondly, according to the preset virtual IP (VIP) rule, identify the target Nginx server to which the build request should be forwarded; Then, according to the configured listening port information in the Nginx server, forward the build request traffic to the corresponding virtualized server environment; Each virtualized server environment address is mapped to a different port in Nginx to achieve precise traffic allocation.

[0018] Furthermore, in terms of dynamic scheduling and allocation of hardware resources, an extended method based on the Jenkins build queue and resource usage is provided, which specifically includes the following implementation steps: 1. Build queue monitoring: The system regularly (every minute) obtains the build queue information from the Jenkins server, including the number of queued tasks, the estimated build time and average waiting time of each task. If it is found that the queue length exceeds the preset threshold (10 tasks) or the average waiting time exceeds the preset duration (10 minutes), an extended decision is triggered; 2. Resource usage monitoring: The system simultaneously monitors the resource usage of virtual server nodes, including the usage rates of CPU, memory, storage, and network bandwidth. If it is found that the resource usage rate of a certain node exceeds 80%, it is considered that the node is overloaded and a new virtual machine needs to be extended to share the pressure; 3. Dynamic extension decision: The system combines the build queue and resource usage, and comprehensively evaluates whether to extend the virtual machine according to 1 and 2. If it is judged that an extension is needed, one virtual machine is extended each time according to the preset virtual machine extension policy. (Administrators pre-create and configure multiple template virtual machines according to different code language build types. These template virtual machines have already installed and configured all necessary build tools and dependencies); 4. Virtual machine extension: When it is detected that the build queue is too long or resources are insufficient (referring to the monitoring data in point 2), the system automatically creates a new virtual machine instance through the cloning API of ESXi and adds it to the specified virtual machine cluster to immediately participate in the execution of build tasks; 5. Automatic contraction: When the build queue length is lower than the preset threshold (5 tasks) and the CPU and memory resource usage rates of the virtual machine are lower than 70% for more than the preset duration, the system will trigger an automatic contraction operation to recycle the virtual machines that are not currently executing build tasks to avoid resource waste.

[0019] The present invention has the following beneficial effects: 1. Solve the single point of failure problem of the dispatching center: By introducing a high-availability mechanism, the single point of failure problem of the traditional single dispatching center is completely solved. Using virtualization technology, load balancing and automatic failure switching can be achieved to ensure the continuous and stable operation of the system.

[0020] By solving the single point of failure problem of the dispatching center, the system can quickly switch to the standby node in the face of node failures, ensuring the business continuity of development and construction and the user experience, and improving the system stability; 2. Efficiently utilize physical resources: Adopting virtualization technology to abstract and pool physical resources enables multiple applications to run independently on the same physical platform, greatly improving the utilization rate and flexibility of resources.

[0021] Virtualization technology enables multiple applications to efficiently share physical resources, optimize resource utilization, avoid resource waste, and reduce the hardware procurement and maintenance costs; 3. Dynamic resource allocation and adjustment: The virtualization platform provides dynamic resource management capabilities, which can adjust the resource allocation of virtual machines in real time according to actual needs, avoid resource waste, and ensure application performance at the same time.

[0022] The dynamic resource allocation and adjustment mechanism ensures that each application can obtain the best performance under different load conditions, improving user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the system architecture diagram for realizing multi-terminal code construction in the embodiment of the present invention.

[0024] Figure 2 It is the schematic diagram of the system server environment configuration for realizing multi-terminal code construction in the embodiment of the present invention.

[0025] Figure 3 It is the method flow diagram for realizing multi-terminal code construction in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] For the convenience of those skilled in the art to understand, the present invention will be further described below in conjunction with the embodiments and the drawings.

[0027] The system for realizing multi-terminal code construction in this embodiment, as Figure 1 shown, the overall architecture includes a service layer, a link layer, a component layer, and a hardware layer.

[0028] Service layer: Provide build types for several development languages. The build types include: Java, Python, iOS, Android, Vue, Harmony.

[0029] Link layer: Provide matching virtual server nodes according to the environmental resource characteristics required by the build type, and make a one-to-one mapping between the build type and the virtual server node. The specific steps are as follows: First, maintain the IP and port of all Jenkins service nodes, then in the pipeline editing interface, select the service address of the corresponding Jenkins, and save the mapping relationship to the database. When calling the service API, it is called in the form of VIP (virtual address) + Jenkins port.

[0030] Component layer: Provide pipeline components according to different build types respectively, and build pipeline scripts; mainly including code checkout nodes, code scanning nodes, compilation nodes, and packaging nodes.

[0031] Hardware layer: Used to provide a hardware virtualization environment for code execution, compilation, and packaging. It includes operating systems such as Linux and MacOS. Virtualization technology allows multiple applications to share the same physical resources. Through dynamic resource allocation and adjustment, hardware resources are utilized to the maximum extent, reducing costs.

[0032] The detailed steps for server environment configuration are as follows.

[0033] 1. Prepare two physical servers for creating a cluster. Create the first node Node1 on the first server Server1 and the second node Node2 on the second server Server2 for building a Linux cluster. The servers meet the requirements of the host system ESXi, including sufficient CPU, memory, and storage resources. According to the different resource utilization rates of each language during compilation, the main resource configurations of each physical server are as follows in the table:

[0034] 2. Install ESXi on each physical server, create a virtual switch (vSwitch) on each ESXi host for connecting virtual machines, and configure the virtual network.

[0035] 3. Use vSphere Client to connect to the ESXi host. Create multiple virtual machines on each ESXi host, and configure the name, storage location, CPU, memory, and disk resources of the virtual machines according to the above table for each language.

[0036] 4. Install the Linux operating system. Upload the ISO file of Linux to the data storage of the ESXi host. Start Node1 and Node2, and select the installation medium (ISO file). After installation, configure the network settings to ensure that Node1 and Node2 can communicate with each other. Note that when configuring the iOS build environment, the MacOS operating system needs to be installed.

[0037] 5. Virtual machine cluster configuration. Install Nginx and Keepalived on Node1 and Node2. Configure Keepalived on Node1 as the master node with state BACKUP, and configure Keepalived on Node2 as the backup node with state BACKUP. Nginx is used to proxy all requests on the node to ensure that requests accessing the virtual network VIP are correctly processed.

[0038] 6. Since the build environments for each language are independent of each other, two nodes, node1 and node2, need to be created for each of the 6 languages to ensure resource isolation and high availability. As Figure 2 shown.

[0039] The necessary software to be installed will be described separately for each language below.

[0040] First, install the general-purpose software Jenkins on each node for automatic code building, testing, and deployment. Then install the Git plugin for code checkout.

[0041] JAVA node: Install the Java Runtime Environment JDK, and then install the dependency package management and compilation tool Maven.

[0042] Python node: First, install the Python runtime environment, and then install the setuptools tool for building, installing, uploading, and managing Python packages.

[0043] Android node: Install the Android Runtime Environment JDK, and then install the dependency package management and compilation tool Gradle.

[0044] VUE node: Install Node.js to provide an environment for running JavaScript code. Install pnpm to manage the dependencies of JavaScript projects.

[0045] HarmonyOS Node: Install the Command Line Tools. This command line toolset includes a series of tools used for HarmonyOS application development, such as the code linter codelinter, the package manager ohpm for third-party libraries, the command line parser hstack, and the compilation and build tool hvigorw.

[0046] iOS Node: Install the Xcode integrated development environment (IDE) for compiling Swift and Objective-C code.

[0047] As Figure 3 shown, the method for implementing multi-terminal code construction using the above-mentioned multi-terminal code construction system includes the following steps: Step 1, Select the build type: Initialize several development languages and select the type to be built. Step 2, Match the build environment: Provide a virtualized server environment, with 2 virtual machine clusters corresponding to each development language. Different environment configurations are provided according to different development languages. The general configurations include the operating system type, integrated tools, code checkout tools, compilation and packaging tools, CPU and memory configurations; each development language is matched with a virtual machine cluster environment to establish a one-to-one mapping relationship. Step 3, Select the build pipeline: The build pipeline includes a code checkout node, a code scanning node, a compilation node, and a packaging node; a unique identifier binding the development language type is generated when creating the build pipeline to achieve matching the corresponding build pipeline through the build type. Step 4, Execute the build pipeline: Push the corresponding build pipeline to the matching server environment through the Jenkins interface, trigger the build script, complete the packaging operation, and output the final product.

[0048] During the Jenkins interface call, high availability and load balancing of requests are achieved through Keepalived. This method includes the following steps: Configure the virtual IP (VIP) rules in Keepalived to identify client requests; store the correspondence between the virtualized server environment address and the Nginx port through the port mapping management module; during the Jenkins interface call, first receive the build request from the client; secondly, identify the target Nginx server to which the build request should be forwarded according to the preset virtual IP (VIP) rules; then forward the build request traffic to the corresponding virtualized server environment according to the listening port information configured in the Nginx server; each virtualized server environment address is mapped to a different port in Nginx to achieve precise traffic distribution.

[0049] Furthermore, in terms of dynamic scheduling and allocation of hardware resources, an extended method based on Jenkins build queue and resource usage is provided, which specifically includes the following implementation steps: 1. Build queue monitoring: The system regularly (every minute) obtains build queue information from the Jenkins server, including the number of queued tasks, the estimated build time and average waiting time of each task. If it is found that the queue length exceeds the preset threshold (10 tasks) or the average waiting time exceeds the preset duration (10 minutes), an extended decision is triggered; 2. Resource usage monitoring: The system simultaneously monitors the resource usage of virtual server nodes, including the usage rates of CPU, memory, storage, and network bandwidth. If it is found that the resource usage rate of a certain node exceeds 80%, it is considered that the node is overloaded and new virtual machines need to be extended to share the pressure; 3. Dynamic extension decision: The system combines the build queue and resource usage, and comprehensively evaluates whether to extend virtual machines based on 1 and 2. If it is determined that an extension is needed, one virtual machine is extended each time according to the preset virtual machine extension policy. (Administrators pre-create and configure multiple template virtual machines according to different code language build types. These template virtual machines have already been installed and configured with all necessary build tools and dependencies); 4. Virtual machine extension: When it is detected that the build queue is too long or the resources are insufficient (referring to the monitoring data in point 2), the system automatically creates new virtual machine instances through the ESXi cloning API and adds them to the specified virtual machine cluster to immediately participate in the execution of build tasks; 5. Automatic shrinkage: When the build queue length is below the preset threshold (5 tasks) and the CPU and memory resource usage rates of the virtual machine are below 70% for more than the preset duration, the system will trigger an automatic shrinkage operation to recycle the virtual machines that are not currently executing build tasks to avoid resource waste.

[0050] The above embodiments are only used to illustrate the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. Any changes made on the basis of the technical solution according to the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A system for implementing multi-terminal code construction, characterized in that: Includes service layer, link layer, component layer and hardware layer; Service layer: provides construction types in several development languages; Link layer: Provide matching virtual server nodes according to the environmental resource characteristics required by the build type, and make a one-to-one mapping between the build type and the virtual server node; Component layer: Provide pipeline components according to different build types and build pipeline script nodes; Including code checkout node, code scanning node, compilation node, and packaging node; Hardware layer: used to provide a virtualized server environment for code execution, compilation and packaging; The hardware layer includes physical servers used to create virtual machine clusters, and virtual server nodes are created on physical servers; Install ESXi on each physical server and create a virtual switch vSwitch on each ESXi host to connect virtual machines. Virtual server nodes on different physical servers are connected through virtual network communication. Use vSphere Client to connect to the ESXi host; create several virtual machines on each ESXi host, and configure the name, storage location, CPU, memory, and disk resources of the virtual machine for each development language on each ESXi host; Install the Linux operating system on the ESXi host; upload the Linux ISO file to the data storage of the ESXi host; Virtual machine cluster configuration: Install the reverse proxy server Nginx and the high-availability service Keepalived on the physical server nodes; Keepalived forwards the client's build request to Nginx, which is used to proxy all build requests of the build type.

2. The system for implementing multi-terminal code construction according to claim 1, characterized in that: The physical server includes a first server Server1 and a second server Server2; a first node Node1 is created on Server1, and a second node Node2 is created on Server2; Node1 and Node2 are connected through a virtual network communication; Keepalived is configured on Server1 as a primary node, and Keepalived is configured on Server2 as a backup node.

3. The system for implementing multi-terminal code construction according to claim 1 or 2, characterized in that: The link layer makes a one-to-one mapping between the build type and the virtual server node, including: first maintaining the IP and port of the Jenkins service node of all open source software projects, then selecting the corresponding Jenkins service address in the pipeline editing interface, and saving the mapping relationship to the database; when calling the service API, it is called in the form of virtual address VIP+Jenkins port.

4. The system for implementing multi-terminal code construction according to claim 3, characterized in that: The nodes on each physical server are installed with the open source software project Jenkins for building, testing, and deploying code; the nodes on each physical server are installed with the Git plug-in for code checkout; Build types include: Java, Python, iOS, Android, Vue, Harmony; JAVA node: Install the Java runtime environment JDK, and then install the dependency package management and compilation tool Maven; Python node: First install the Python runtime environment, then install the setuptools tool to build, install, upload, and manage Python packages; Android node: Install the Android runtime environment JDK, and then install the dependency package management and compilation tool Gradle; VUE node: Install Node.js to provide an environment for running JavaScript code; install pnpm to manage the dependencies of JavaScript projects; Harmony node: Install the command line tools Command Line Tools; iOS node: Install the Xcode integrated development environment IDE to compile Swift and Objective-C code.

5. The system for implementing multi-terminal code construction according to claim 1, characterized in that: The physical server's CPU is ≥ 20C, memory is ≥ 80G, and SSD hard disk is ≥ 6T.

6. A method for implementing multi-terminal code construction, characterized in that: A system for implementing multi-terminal code construction using any one of claims 1 to 5 comprises the following steps: Step 1: Select the build type: 6 language options are provided: Java, Python, Vue, Android, iOS, and Harmony. Developers can select the type they want to build. Step 2: Match the build environment: Provide a virtualized server environment. Each development language corresponds to two virtualized server environments. Provide different environment configurations according to different development languages. Common configurations include operating system type, integration tools, code checkout tools, compilation and packaging tools, CPU and memory configuration. Each development language matches a virtualized server environment to establish a one-to-one mapping relationship. Step 3: Select the build pipeline: The build pipeline includes code checkout node, code scanning node, compilation node, and packaging node. When creating a build pipeline, a unique identifier bound to the development language type is generated to match the corresponding build pipeline by the build type. Step 4: Execute the build pipeline: Push the corresponding build pipeline to the matching server environment through the Jenkins interface, trigger the build script, complete the packaging operation, and output the final product.

7. The method for implementing multi-terminal code construction according to claim 6, characterized in that: Configure virtual address VIP rules in Keepalived to identify client requests; store the correspondence between the virtualized server environment address and the Nginx port through the port mapping management module; During the Jenkins interface call process, the build request from the client is first received; secondly, the target Nginx to which the build request should be forwarded is identified according to the preset virtual address VIP rule; then, according to the listening port information configured in Nginx, the build request traffic is forwarded to the corresponding virtualized server environment to achieve accurate traffic distribution.

8. The method for implementing multi-terminal code construction according to claim 6 or 7, characterized in that: S1, build queue monitoring: The system regularly obtains build queue information from the Jenkins server, including the number of queued tasks, the estimated build time of each task, and the average waiting time; if the queue length exceeds the preset threshold or the average waiting time exceeds the preset duration, the expansion decision is triggered; S2, resource usage monitoring: The system monitors the resource usage of the virtual server construction node, including the usage of CPU, memory, storage and network bandwidth; if the resource usage of the node exceeds 80%, it is considered that the node load is too high and new virtual machines need to be expanded to share the pressure; S3, virtual machine expansion: When it is detected that the build queue is too long or the resources are insufficient, a new virtual machine is automatically created through the ESXi clone API and added to the specified virtual machine cluster to immediately participate in the execution of the build task; S4, automatic shrinkage: When the length of the build queue is lower than the preset threshold and the resource usage of the virtual machine is lower than 70% for more than the preset time, the shrinkage operation is triggered to recycle the virtual machines that are not currently executing build tasks.

Images