A method, system, electronic device, and storage medium for multi-host parallel compilation

Abstract

The present application discloses a method, system, electronic device and storage medium for multi-host parallel compilation, which is used to improve the utilization rate of compilation resources and compilation efficiency. The method for multi-host parallel compilation in the present application includes: starting and running compilation containers on multiple target hosts, the compilation containers being unified compilation containers pre-created based on pre-configured target Docker images, and the target Docker images including a complete source code compilation environment; dividing the target source code according to the compilation dependency relationship between logical modules to obtain multiple target source code modules, and there is a corresponding relationship between the target source code and the source code compilation environment; respectively allocating the multiple target source code modules to the compilation containers on multiple target hosts for parallel compilation to obtain multiple compilation results; aggregating the multiple compilation results to a target aggregation host to obtain an aggregation result; and based on the aggregation result, performing system image packaging to generate a system image file.

Description

Technical Field

[0001] This application relates to the field of code compilation technology, and in particular to a method, system, electronic device and storage medium for multi-host parallel compilation. Background Technology

[0002] The Android system source code is massive, with numerous modules, and its compilation process demands extremely high computing resources and is very time-consuming. During system development and testing, developers need to frequently perform full or incremental compilations to verify code changes. However, in single-machine compilation mode, the long waiting time severely restricts the improvement of development efficiency. To alleviate this contradiction, existing technologies typically adopt two mainstream solutions: one is based on distributed compilation tools such as compilation server sets or clusters, such as Distcc and Incredibuild, which improves speed by distributing compilation tasks to multiple machines for parallel processing; the other is to rely on a single high-performance server for centralized compilation, which shortens compilation time by improving the hardware configuration of a single machine.

[0003] However, the aforementioned existing technologies have significant drawbacks in compilation efficiency. Specifically, while distributed compilation tools and cluster solutions possess a certain degree of parallelism, their deployment and maintenance costs are high—each participating node requires uniform configuration of a complex compilation environment, dependency libraries, and network sharing settings. Inconsistent environments can easily lead to compilation failures, resulting in a heavy management burden. On the other hand, while a single high-performance server solution can avoid the problem of multi-machine environment collaboration, hardware procurement costs are high, and its performance improvement is limited by the hardware capacity of a single machine, making it impossible to flexibly utilize the resources of multiple hosts within an organization. Therefore, the problem of low compilation efficiency in existing technologies urgently needs to be addressed. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a method, system, electronic device, and storage medium for multi-host parallel compilation, which improves compilation resource utilization and compilation efficiency.

[0005] The technical solution provided in this application is described below: The first aspect of this application provides a method for multi-host parallel compilation, including: Start and run compilation containers on multiple target hosts. The compilation containers are unified compilation containers pre-created based on pre-configured target Docker images, which include a complete source code compilation environment. The target source code is divided according to the compilation dependencies between logical modules to obtain multiple target source code modules, and the target source code has a corresponding relationship with the source code compilation environment; The multiple target source code modules are respectively assigned to compilation containers on the multiple target hosts for parallel compilation to obtain multiple compilation results; The multiple compilation results are aggregated to the target aggregation host to obtain the aggregation result; Based on the aggregation results, the system image is packaged to generate a system image file.

[0006] Optionally, the plurality of target source code modules includes at least one composite module and / or one independent module, and the step of dividing the target source code according to the compilation dependencies between logical modules to obtain multiple target source code modules includes: The build configuration file of the target source code is parsed, and multiple logical modules in the build configuration file and the metadata corresponding to each logical module are identified. Based on the metadata, analyze whether there are compilation dependencies between each logical module; Merge logical modules that have direct or indirect compilation dependencies into a single composite module; Each logical module that has no compilation dependencies is treated as an independent module.

[0007] Optionally, after assigning the plurality of target source code modules to compilation containers on the plurality of target hosts for parallel compilation and obtaining multiple compilation results, the method further includes: The multiple compilation results are stored in a pre-defined unified output directory within each compilation container. The path of the unified output directory remains consistent within each compilation container. Each compilation result includes the generated target file, intermediate file, and compilation log.

[0008] Optionally, aggregating the multiple compilation results to a target aggregation host to obtain the aggregation result includes: Start and run the build container on the target aggregation host, wherein the build container on the target aggregation host is the same as the build containers on the multiple target hosts; Through a pre-configured network file subsystem, the preset unified output directory in each compilation container is mounted to multiple different mount points in the compilation container of the target aggregation host; By using the overlay file subsystem pre-configured within the compilation container of the target aggregation host, the output directory within the compilation container of the target aggregation host is used as the upper-level directory, and multiple different mount points within the compilation container of the target aggregation host are used as the lower-level directories, to merge the multiple compilation results and generate an aggregation result.

[0009] Optionally, after assigning the plurality of target source code modules to compilation containers on the plurality of target hosts for parallel compilation, the method further includes: When the compilation time of the compilation container on any target host exceeds the preset time, or when a compilation error alarm is returned, the compilation task of the compilation container of the currently detected target host is released, and the compilation task is reassigned to the compilation container of the target host that is currently idle.

[0010] Optionally, before packaging the system image based on the aggregation result and generating the system image file, the method further includes: Verify whether the aggregation result contains a compilation result with the same name and content. If so, retain one of the compilation results with the same name and content. If not, trigger an alarm and pause the system image packaging. And / or verify whether the aggregation result contains the compilation results corresponding to all target source code modules. If yes, then package the system image; if not, then recompile the target source code modules corresponding to the missing compilation results.

[0011] Optionally, the plurality of target hosts are idle hosts within the target local area network. Before distributing the plurality of target source code modules to compilation containers on the plurality of target hosts for parallel compilation, the method further includes: According to preset filtering conditions, idle hosts in the target local area network are filtered out. The filtering conditions are that the CPU utilization rate is less than a preset utilization rate threshold and the memory occupancy rate is less than a preset occupancy threshold within a preset continuous time period. The step of distributing the multiple target source code modules to compilation containers on the multiple target hosts for parallel compilation includes: The multiple target source code modules are distributed to compilation containers on multiple idle hosts for parallel compilation.

[0012] A second aspect of this application provides a system for multi-host parallel compilation, comprising: A startup unit is used to start and run compilation containers on multiple target hosts. The compilation containers are unified compilation containers pre-created based on pre-configured target Docker images, and the target Docker images include a complete source code compilation environment. A partitioning unit is used to divide the target source code according to the compilation dependencies between logical modules to obtain multiple target source code modules, wherein the target source code has a corresponding relationship with the source code compilation environment; A compilation unit is used to allocate the multiple target source code modules to compilation containers on the multiple target hosts for parallel compilation, thereby obtaining multiple compilation results; An aggregation unit is used to aggregate the multiple compilation results to a target aggregation host to obtain an aggregation result; The packaging unit is used to package the system image based on the aggregation result and generate a system image file.

[0013] A third aspect of this application provides an electronic device, the electronic device comprising: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to execute the first aspect and any optional method of the multi-host parallel compilation method.

[0014] A fourth aspect of this application provides a computer-readable storage medium on which a program is stored, which, when executed on a computer, performs the multi-host parallel compilation method of the first aspect and any one of the first aspects.

[0015] As can be seen from the above technical solutions, this application has the following beneficial effects: This application provides a method for multi-host parallel compilation. First, compilation containers on multiple target hosts are started and run. These compilation containers are unified compilation containers pre-created based on pre-configured target Docker images, which include a complete source code compilation environment. Next, the target source code is divided according to the compilation dependencies between logical modules to obtain multiple target source code modules, with a correspondence between the target source code and the source code compilation environment. Then, the multiple target source code modules are distributed to the compilation containers on multiple target hosts for parallel compilation, resulting in multiple compilation results. Next, the multiple compilation results are aggregated to a target aggregation host to obtain an aggregated result. Finally, based on the aggregated result, a system image is packaged to generate a system image file. The multi-host parallel compilation method of this application, on the one hand, pre-creates a unified compilation container based on a pre-configured target Docker image to obtain a unified compilation environment. That is, it utilizes the environment encapsulation characteristics of the target Docker image to ensure that the compilation containers on all participating target hosts have a complete and consistent compilation environment with the target source code. Even in the case of multiple machines with distributed compilation tools, a unified compilation environment can be quickly achieved, improving compilation efficiency. On the other hand, by dividing the target source code according to the compilation dependencies between logical modules, multiple target source code modules are obtained. This fully utilizes the computing resources of multiple hosts to achieve task parallelism, breaking through the hardware performance limit of a single high-performance server, shortening the overall compilation time, and ensuring the integrity and correctness of the compilation results through the isolation and unified aggregation mechanism of the compilation containers. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic flowchart of an embodiment of the multi-host parallel compilation method provided in this application; Figure 2 A flowchart illustrating another embodiment of the multi-host parallel compilation method provided in this application; Figure 3 A flowchart illustrating another embodiment of the multi-host parallel compilation method provided in this application; Figure 4 A flowchart illustrating another embodiment of the multi-host parallel compilation method provided in this application; Figure 5 A schematic diagram of an embodiment of the multi-host parallel compilation system provided in this application; Figure 6 A schematic diagram of an embodiment of the electronic device provided in this application. Detailed Implementation

[0018] This application provides a method, system, electronic device, and storage medium for multi-host parallel compilation, which can improve the utilization of compilation resources and compilation efficiency.

[0019] It should be noted that the multi-host parallel compilation method provided in this application can be applied to terminals, systems, and servers. For example, the terminal can be a computer, a portable computer terminal, or a desktop computer or other fixed terminal.

[0020] It should also be noted that the terms "first," "second," etc., in the specification and drawings are used to distinguish similar objects and are not necessarily used to describe a specific order, sequence, or importance.

[0021] Please see Figure 1 This application first provides an embodiment of a method for multi-host parallel compilation, the method comprising: Step S101: Start and run compilation containers on multiple target hosts. The compilation containers are unified compilation containers pre-created based on pre-configured target Docker images. The target Docker images include a complete source code compilation environment. Step S102: Divide the target source code according to the compilation dependencies between logical modules to obtain multiple target source code modules. The target source code and the source code compilation environment have a corresponding relationship. Step S103: Distribute multiple target source code modules to compilation containers on multiple target hosts for parallel compilation to obtain multiple compilation results; Step S104: Aggregate multiple compilation results to the target aggregation host to obtain the aggregation result; Step S105: Based on the aggregation results, package the system image to generate a system image file.

[0022] The multi-host parallel compilation method of this application first starts and runs compilation containers on multiple target hosts. The compilation containers are unified compilation containers pre-created based on pre-configured target Docker images, and the target Docker images include complete source code compilation environments. Then, the target source code is divided according to the compilation dependencies between logical modules to obtain multiple target source code modules, and the target source code and the source code compilation environment have a corresponding relationship. Then, the multiple target source code modules are respectively assigned to the compilation containers on multiple target hosts for parallel compilation to obtain multiple compilation results. Next, the multiple compilation results are aggregated to the target aggregation host to obtain the aggregation result. Finally, based on the aggregation result, the system image is packaged to generate a system image file. The multi-host parallel compilation method of this application, on the one hand, pre-creates a unified compilation container based on a pre-configured target Docker image to obtain a unified compilation environment. That is, it utilizes the environment encapsulation characteristics of the target Docker image to ensure that the compilation containers on all participating target hosts have a complete and consistent compilation environment with the target source code. Even in the case of multiple machines with distributed compilation tools, a unified compilation environment can be quickly achieved, improving compilation efficiency. On the other hand, by dividing the target source code according to the compilation dependencies between logical modules, multiple target source code modules are obtained. This fully utilizes the computing resources of multiple hosts to achieve task parallelism, breaking through the hardware performance limit of a single high-performance server, shortening the overall compilation time, and ensuring the integrity and correctness of the compilation results through the isolation and unified aggregation mechanism of the compilation containers.

[0023] In step S101 of this embodiment, compilation containers on multiple target hosts are started and run. These compilation containers are unified compilation containers pre-created based on a pre-configured target Docker image, which includes a complete source code compilation environment. First, multiple target hosts that will participate in the compilation are configured within the target local area network. Then, the Docker engine is installed on these target hosts. A pre-built base image, the target Docker image, is introduced. Specifically, this target Docker image configures a complete compilation environment adapted to the target source code to be compiled, i.e., a complete source code compilation environment. This complete source code compilation environment includes the corresponding version of the compilation toolchain, dependent toolsets, and dynamic libraries, ensuring compatibility between the source code compilation environment and the target source code, thus forming a unified target Docker image. Using Docker commands in the Docker engine, compilation containers are created on each target host based on the aforementioned target Docker image. These compilation containers are then started and kept running, ensuring that the compilation containers on all target hosts have a consistent compilation environment and avoiding compilation problems caused by environmental differences.

[0024] In step S102 of this embodiment, the target source code is divided according to the compilation dependencies between logical modules to obtain multiple target source code modules. The target source code and its compilation environment have a corresponding relationship. First, relying on the inherent module definition files in the target source code, the target source code is logically analyzed using parsing scripts or tools to identify all compilable logical modules, such as core logical modules like linuxkernel, qssi, mssi, and vendor, while simultaneously clarifying the dependencies between these logical modules. Then, the target source code is modularized according to these dependencies. The entire partitioning process must ensure that the target source code modules strictly correspond to the compilation environment in the target Docker image; that is, the toolchain, dependency libraries, and other configurations of the compilation environment must fully match the version and hardware manufacturer requirements of the target source code.

[0025] In step S103 of this embodiment, multiple target source code modules are distributed to compilation containers on multiple target hosts for parallel compilation, resulting in multiple compilation results. A scheduler is used for this distribution; this scheduler can be a centralized control node or a distributed script. Specifically, the scheduler, based on the multiple target source code modules generated in step S102, treats each target source code module as a compilation task and distributes it to compilation containers on different target hosts via remote execution commands or orchestration tools. Each compilation container, upon receiving a compilation task, independently executes the corresponding compilation commands in its isolated environment and outputs the compiled content or files to a unified designated directory within the container; this directory represents one compilation result.

[0026] Optionally, multiple target hosts are idle hosts within the target local area network. Therefore, before step S103, the hosts should be screened, which may include, but is not limited to, the following: Based on preset screening criteria, idle hosts within the target local area network are selected. The screening criteria are that hosts with CPU utilization less than a preset utilization threshold and memory usage less than a preset utilization threshold within a preset continuous time period are considered idle hosts. Then, multiple target source code modules are assigned to compilation containers on multiple idle hosts for parallel compilation. Before task allocation, a background monitoring service continuously collects performance metrics of all hosts within the target local area network, such as obtaining CPU and memory utilization through cgroups or system APIs. Then, screening is performed based on the performance metrics. The screening logic is as follows: first, a utilization threshold and a memory usage threshold are set, such as CPU utilization < 20% and memory usage < 30%; then, data is collected continuously for a period of time, such as 30 minutes. Hosts that meet the screening criteria are marked as "idle" and added to the resource pool, becoming target hosts. Subsequently, the scheduler only selects hosts from this pool to create or wake up compilation containers, thereby intelligently and elastically utilizing idle resources.

[0027] In step S104 of this embodiment, multiple compilation results are aggregated to a target aggregation host to obtain an aggregation result. This step can be achieved by using a network file subsystem, an overlay file subsystem, and a joint mounting technique. Specifically, multiple compilation results are mounted to the same target aggregation host by using a network file subsystem and a joint mounting technique, achieving distributed collection and mounting. Then, the overlay file subsystem is used to aggregate the compilation results of all mount points on the target aggregation host to obtain the aggregation result.

[0028] Optionally, when the compilation time of a compilation container on any target host exceeds a preset time, or an alarm message indicating a compilation error is returned, the compilation task of the currently detected compilation container on the target host is released, and the compilation task is reassigned to a compilation container on a target host that is currently idle. The scheduler in step S103 monitors the status of each compilation container in real time. Specifically, it sets a timeout threshold, i.e., a preset time, for each task; simultaneously, the compilation script is designed to report errors to the scheduler through a specific channel. Once the compilation time exceeds the preset time or a compilation error is returned, the scheduler immediately sends a termination signal to the problematic container to release the task, then re-enters the task queue and prioritizes its allocation to compilation containers on other hosts in the current resource pool that are lightly loaded or idle, thereby automatically achieving failover and task retries.

[0029] In step S105 of this embodiment, based on the aggregation result, a system image is packaged to generate a system image file. The system image packaging script is configured to read all compiled target files from the aggregation result generated in step S104. Then, the system image packaging script packages the discrete target files, library files, and configuration files into corresponding, directly writable image files according to a predetermined file system structure. If the aggregation result already contains complete and correct compilation artifacts, the final packaging process is no different from compiling the entire target source code on a single machine, ensuring the integrity of the final system image.

[0030] Reference Figure 2 According to some embodiments of this application, the multiple target source code modules include at least one combined module and / or one independent module. The step S102 of dividing the target source code according to the compilation dependencies between logical modules to obtain multiple target source code modules may specifically include, but is not limited to, the following: Step S201: Parse the build configuration file of the target source code and identify multiple logical modules in the build configuration file and the metadata corresponding to each logical module; Step S202: Based on metadata, analyze whether there are compilation dependencies between each logical module; Step S203: Merge logical modules that have direct or indirect compilation dependencies into a single composite module; Step S204: Treat each logical module that has no compilation dependencies as an independent module.

[0031] In steps S201 to S204 of the embodiment, the build configuration file of the target source code is first parsed, and multiple logical modules in the build configuration file and the metadata corresponding to each logical module are identified. Based on the metadata, it is analyzed whether there are compilation dependencies between each logical module. Then, logical modules that have direct or indirect compilation dependencies are merged into a combined module. Then, logical modules that do not have compilation dependencies are each treated as independent modules. The build configuration file of the target source code typically includes files such as Makefile, Android.bp, and Blueprint, which define the compilation rules and module structure of the source code. Through a preset configuration file parsing tool, the above-mentioned build configuration file is parsed and its content is extracted to identify multiple logical modules declared in the build configuration file, and the metadata corresponding to each logical module is extracted. The metadata includes, but is not limited to, module name, input source code path, output artifact type, toolchain version required for compilation, and explicitly declared dependent module identifiers. Then, the focus is on analyzing the "dependent module identifier" field in the metadata of each logical module. For example, if the metadata of module A contains the identifier of module B, then A directly depends on B, meaning that the compilation of A requires the compilation output of B as input. Indirect dependencies can also be identified by tracing several consecutive compilation dependencies. For example, if A depends on B and B depends on C, then A indirectly depends on C. At the same time, the rationality of dependencies is verified by combining the output output type. Finally, a dependency graph containing all logical modules is constructed, clearly showing the direct or indirect compilation dependencies between modules.

[0032] Based on the dependency graph generated above, a set of logical modules with direct or indirect compilation dependencies is selected. For example, if module X depends on module Y, and module Y depends on module Z, then X, Y, and Z constitute an associated set. All logical modules in this set are merged into a composite module, ensuring that the composite module contains a complete dependency chain. During merging, the metadata relationships of each module are preserved so that subtasks within the composite module are executed in dependency order during subsequent compilation, avoiding dependency conflicts caused by parallel compilation.

[0033] Besides the logical modules with compilation dependencies mentioned above, we filter out logical modules that are not depended on by any other module and do not depend on any other module themselves. These are modules whose metadata does not contain a "dependent module identifier," and which are also not identified in the metadata of other modules. Since these modules do not have compilation order constraints, they are treated as independent modules. Independent modules can be assigned to different compilation containers for parallel compilation without considering dependency coordination, maximizing the utilization of the parallel computing capabilities of multiple hosts.

[0034] Reference Figure 3 According to some embodiments of this application, the aggregation of multiple compilation results to the target aggregation host in step S104 to obtain the aggregation result may specifically include, but is not limited to, the following: Step S301: Store the multiple compilation results in a pre-defined unified output directory within each compilation container. The path of the unified output directory remains consistent within each compilation container. All compilation results include the generated target file, intermediate file, and compilation log. Step S302: Start and run the compilation container on the target aggregation host. The compilation container on the target aggregation host is the same as the compilation containers on multiple target hosts. Step S303: Using the pre-configured network file subsystem, the preset unified output directory in each compilation container is mounted to multiple different mount points in the compilation container of the target aggregation host. Step S304: Using the overlay file subsystem pre-configured within the compilation container of the target aggregation host, the output directory within the compilation container of the target aggregation host is used as the upper-level directory, and multiple different mount points within the compilation container of the target aggregation host are used as the lower-level directories, to merge multiple compilation results and generate an aggregation result.

[0035] In steps S301 to S304 of the embodiment, firstly, multiple compilation results are stored in a pre-defined unified output directory within each compilation container. The path of the unified output directory remains consistent within each compilation container. Each compilation result includes generated target files, intermediate files, and compilation logs. Then, the compilation container on the target aggregation host is started and run. The compilation container on the target aggregation host is unified with the compilation containers on multiple target hosts. Through a pre-configured network file subsystem, the pre-defined unified output directory within each compilation container is mounted to multiple different mount points within the compilation container on the target aggregation host. Then, through an overlay file subsystem pre-configured within the compilation container on the target aggregation host, using the output directory within the compilation container on the target aggregation host as the upper-level directory and the multiple different mount points within the compilation container on the target aggregation host as lower-level directories, the multiple compilation results are merged to generate an aggregated result. A unified output directory is pre-defined within each compilation container to ensure that the path of this directory is completely consistent across the compilation containers on all target hosts. Once each target container has completed the compilation of the target source code module, it automatically stores three types of files from the generated compilation results in this directory. These three types of files can include target files, such as binary programs, library files, and other final compilation artifacts; intermediate files, such as temporary target files and preprocessed files generated during the compilation process; and compilation logs, such as process data recording compilation time, warning messages, and progress nodes.

[0036] You can select a target aggregation host from multiple target hosts, or choose a new target aggregation host outside of multiple target hosts. The target aggregation host should start a build container that is completely identical to other target hosts. This container is also created based on the pre-configured target Docker image, thus having the same source code compilation environment as other build containers, ensuring compatibility between the processing of compilation results and the compilation phase during aggregation. The startup process is executed via Docker commands. After the build container starts, it remains running, waiting to receive and mount the compilation results from other target hosts.

[0037] A network file subsystem is pre-configured within the compilation container of the target aggregation host, and corresponding subsystem servers are configured in the compilation containers of other target hosts, granting read permissions to the unified output directory. Multiple independent mount point directories are created within the aggregation host container. Each mount point establishes a mapping with the unified output directory within a target host's compilation container via the subsystem server protocol, enabling remote access to the compilation results of that target host. The independent naming of different mount points ensures that compilation results from different sources are distinguishable within the aggregation host container. Thus, within the aggregation container, compilation results on all remote hosts can be accessed transparently, just like local files.

[0038] An overlay file subsystem is pre-configured within the compilation container of the target aggregation host. During merging, the unified output directory of the aggregation host's own compilation container is set as the parent directory, and the multiple mount point directories created in step S303 are set as the lower directories. The overlay file subsystem merges the files of the parent directory with all the lower directories according to the parent directory priority rule. The final merged directory is the aggregation result containing the compilation results of all target hosts, achieving seamless integration of multi-machine compilation products and preparing for subsequent system image packaging.

[0039] Reference Figure 4 According to some embodiments of this application, before packaging the system image, the aggregation results should also be checked for repeatability and integrity, which may include, but is not limited to, the following: Step S401: Verify whether the aggregation result contains a compilation result with the same name and content. If yes, retain one of the compilation results with the same name and content. If no, trigger an alarm and pause the system image packaging. Step S402, and / or verify whether the aggregation result contains the compilation results corresponding to all target source code modules. If yes, then package the system image; if not, then recompile the target source code modules corresponding to the missing compilation results.

[0040] In steps S401 to S402 of the embodiment, firstly, it is verified whether the aggregation result contains a compilation result with the same name and content. If so, one of the compilation results with the same name and content is retained; otherwise, an alarm is triggered and system image packaging is paused. Then, it is verified whether the aggregation result contains compilation results corresponding to all target source code modules. If so, system image packaging is performed; otherwise, the target source code modules corresponding to the missing compilation results are recompiled. The directory of the aggregation result is traversed by a preset script, i.e., the directory merged by overwriting the file subsystem in step S304. All files are recursively scanned and filenames and paths are recorded. Files with the same name are filtered out, regardless of whether the paths are completely identical. For each group of files with the same name, the content is checked for consistency by calculating the hash value of the files: if the hash values ​​are the same, the content is considered to be consistent, and one of them is kept to reduce redundancy, usually the parent directory or the first file scanned; if the hash values ​​are different, the content is considered to be conflicting, and an alarm mechanism is immediately triggered, such as sending an error log to the control node, displaying a pop-up prompt in the management interface, etc. At the same time, the system image packaging process is automatically paused, and the validity of the conflicting files is confirmed by manual verification before deciding which file to keep or whether to recompile the relevant modules.

[0041] To verify the completeness of the aggregation results, a target source code module list can be pre-built. This list records all combined modules and independent modules identified in steps S203 and S204, and associates them with the characteristic compilation artifacts of each target source code module. During verification, a script traverses the aggregation result directory, extracts the identification information of all compilation artifacts, such as filenames and module-specific tags, and compares them with the target source code module list to check for any unmatched modules—modules recorded in the list but not present in the aggregation results. If the compilation results of all modules are present, the aggregation result is considered complete, and the system image packaging script continues to execute. If a module is missing, its name is automatically recorded, and its corresponding target source code module is reassigned to a currently idle compilation container for recompilation. After recompilation, the aggregation steps continue until the aggregation result contains the compilation results of all target source code modules.

[0042] The above embodiments illustrate the multi-host parallel compilation method provided in this application. The following describes the multi-host parallel compilation system, electronic device, and storage medium provided in this application: Please see Figure 5 This application provides an embodiment of a multi-host parallel compilation system, the system comprising: The startup unit 501 is used to start and run compilation containers on multiple target hosts. The compilation containers are unified compilation containers pre-created based on pre-configured target Docker images. The target Docker images include a complete source code compilation environment. The partitioning unit 502 is used to divide the target source code according to the compilation dependency relationship between logical modules to obtain multiple target source code modules, and the target source code has a corresponding relationship with the source code compilation environment; Compilation unit 503 is used to distribute multiple target source code modules to compilation containers on multiple target hosts for parallel compilation, thereby obtaining multiple compilation results; Aggregation unit 504 is used to aggregate multiple compilation results to the target aggregation host to obtain the aggregation result; Packaging unit 505 is used to package the system image based on the aggregation results and generate a system image file.

[0043] Optionally, the multiple target source code modules may include at least one composite module and / or one independent module, and partitioning unit 502 is specifically used for: Parse the build configuration file of the target source code and identify multiple logical modules in the build configuration file and the metadata corresponding to each logical module; Based on metadata, analyze whether there are compilation dependencies between each logical module; Merge logical modules that have direct or indirect compilation dependencies into a single composite module; Each logical module that has no compilation dependencies is treated as an independent module.

[0044] Optionally, following compilation unit 503, the following may also be included: Storage unit 506 is used to store multiple compilation results to a pre-defined unified output directory in each compilation container. The path of the unified output directory remains consistent in each compilation container. All compilation results include generated target files, intermediate files, and compilation logs.

[0045] Optionally, the aggregation unit 504 is specifically used for: Start and run the build container on the target aggregation host. The build container on the target aggregation host is the same as the build containers on multiple target hosts. By using a pre-configured network file subsystem, the pre-defined unified output directory within each compilation container is mounted to multiple different mount points within the compilation container of the target aggregation host; By using the overlay file subsystem pre-configured within the compilation container of the target aggregation host, the output directory within the compilation container of the target aggregation host is used as the upper-level directory, and multiple different mount points within the compilation container of the target aggregation host are used as the lower-level directories, to merge multiple compilation results and generate an aggregated result.

[0046] Optionally, following compilation unit 503, the following may also be included: The reallocation unit 507 is used to release the compilation task of the currently detected compilation container of the target host and reallocate the compilation task to the compilation container of the target host that is currently idle when the compilation time of the compilation container on any target host exceeds a preset time or a compilation error alarm message is returned.

[0047] Optionally, prior to compilation unit 503, the following may also be included: The first verification unit 508 is used to verify whether the aggregation result contains a compilation result with the same name and content. If so, one of the compilation results with the same name and content is retained. If not, an alarm is triggered and the system image packaging is paused. And / or the second verification unit 509, used to verify whether the aggregation result contains the compilation results corresponding to all target source code modules. If yes, the system image is packaged; if not, the target source code modules corresponding to the missing compilation results are recompiled.

[0048] Optionally, multiple target hosts are idle hosts within the target local area network. Prior to compilation unit 503, the following are also included: The filtering unit 510 is used to filter out idle hosts in the target local area network according to preset filtering conditions. The filtering conditions are that the host whose CPU utilization rate is less than a preset utilization rate threshold and whose memory utilization rate is less than a preset utilization rate threshold within a preset continuous time period is an idle host. Compiler unit 503 is specifically used for: Multiple target source code modules are distributed to compilation containers on multiple idle hosts for parallel compilation.

[0049] Please see Figure 6 This application also provides an electronic device, including: Processor 601, memory 602, input / output unit 603, bus 604; The processor 601 is connected to the memory 602, the input / output unit 603, and the bus 604; The memory 602 stores a program, and the processor 601 calls the program to execute it, such as... Figures 1 to 4 The method of multi-host parallel compilation shown in any of the embodiments.

[0050] This application also relates to a computer-readable storage medium on which a program is stored, which, when run on a computer, causes the computer to perform actions such as... Figures 1 to 4 The method of multi-host parallel compilation shown in any of the embodiments.

[0051] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0052] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0053] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0054] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0055] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for multi-host parallel compilation, characterized in that, The method includes: Start and run compilation containers on multiple target hosts. The compilation containers are unified compilation containers pre-created based on pre-configured target Docker images, which include a complete source code compilation environment. The target source code is divided according to the compilation dependencies between logical modules to obtain multiple target source code modules, and the target source code has a corresponding relationship with the source code compilation environment; The multiple target source code modules are respectively assigned to compilation containers on the multiple target hosts for parallel compilation to obtain multiple compilation results; The multiple compilation results are aggregated to the target aggregation host to obtain the aggregation result; Based on the aggregation results, the system image is packaged to generate a system image file.

2. The method according to claim 1, characterized in that, The plurality of target source code modules includes at least one combined module and / or one independent module, and the step of dividing the target source code according to the compilation dependencies between logical modules to obtain the plurality of target source code modules includes: The build configuration file of the target source code is parsed, and multiple logical modules in the build configuration file and the metadata corresponding to each logical module are identified. Based on the metadata, analyze whether there are compilation dependencies between each logical module; Merge logical modules that have direct or indirect compilation dependencies into a single composite module; Each logical module that has no compilation dependencies is treated as an independent module.

3. The method according to claim 1, characterized in that, After assigning the multiple target source code modules to compilation containers on the multiple target hosts for parallel compilation to obtain multiple compilation results, the method further includes: The multiple compilation results are stored in a pre-defined unified output directory within each compilation container. The path of the unified output directory remains consistent within each compilation container. Each compilation result includes the generated target file, intermediate file, and compilation log.

4. The method according to claim 3, characterized in that, The step of aggregating the multiple compilation results to the target aggregation host to obtain the aggregation result includes: Start and run the build container on the target aggregation host, wherein the build container on the target aggregation host is the same as the build containers on the multiple target hosts; Through a pre-configured network file subsystem, the preset unified output directory in each compilation container is mounted to multiple different mount points in the compilation container of the target aggregation host; By using the overlay file subsystem pre-configured within the compilation container of the target aggregation host, the output directory within the compilation container of the target aggregation host is used as the upper-level directory, and multiple different mount points within the compilation container of the target aggregation host are used as the lower-level directories, to merge the multiple compilation results and generate an aggregation result.

5. The method according to any one of claims 1-4, characterized in that, After the process of distributing the plurality of target source code modules to compilation containers on the plurality of target hosts for parallel compilation, the method further includes: When the compilation time of the compilation container on any target host exceeds the preset time, or when a compilation error alarm is returned, the compilation task of the compilation container of the currently detected target host is released, and the compilation task is reassigned to the compilation container of the target host that is currently idle.

6. The method according to any one of claims 1-4, characterized in that, Before packaging the system image based on the aggregation result and generating the system image file, the method further includes: Verify whether the aggregation result contains a compilation result with the same name and content. If so, retain one of the compilation results with the same name and content. If not, trigger an alarm and suspend system image packaging. And / or verify whether the aggregation result contains the compilation results corresponding to all target source code modules. If yes, then package the system image; if not, then recompile the target source code modules corresponding to the missing compilation results.

7. The method according to any one of claims 1-4, characterized in that, The multiple target hosts are idle hosts within the target local area network. Before the multiple target source code modules are respectively allocated to the compilation containers on the multiple target hosts for parallel compilation, the following steps are also included: According to preset filtering conditions, idle hosts in the target local area network are filtered out. The filtering conditions are that the CPU utilization rate is less than a preset utilization rate threshold and the memory occupancy rate is less than a preset occupancy threshold within a preset continuous time period. The step of distributing the multiple target source code modules to compilation containers on the multiple target hosts for parallel compilation includes: The multiple target source code modules are distributed to compilation containers on multiple idle hosts for parallel compilation.

8. A system for multi-host parallel compilation, characterized in that, The system includes: A startup unit is used to start and run compilation containers on multiple target hosts. The compilation containers are unified compilation containers pre-created based on pre-configured target Docker images, and the target Docker images include a complete source code compilation environment. A partitioning unit is used to divide the target source code according to the compilation dependencies between logical modules to obtain multiple target source code modules, wherein the target source code has a corresponding relationship with the source code compilation environment; A compilation unit is used to allocate the multiple target source code modules to compilation containers on the multiple target hosts for parallel compilation, thereby obtaining multiple compilation results; An aggregation unit is used to aggregate the multiple compilation results to a target aggregation host to obtain an aggregation result; The packaging unit is used to package the system image based on the aggregation result and generate a system image file.

9. An electronic device, characterized in that, The electronic device includes: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to execute the multi-host parallel compilation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a program that, when executed on a computer, performs the multi-host parallel compilation method as described in any one of claims 1 to 7.

Images