Static website generator scheduling method and device for building tasks in parallel and electronic equipment

Abstract

The application discloses a static website generator parallel construction task scheduling method and device and electronic equipment. The method comprises the following steps: determining the load state of a server for executing a static website generator (SWG) parallel construction task; determining the task type of a subtask in the SWG parallel construction task, wherein the task type comprises a computation-intensive task and an input-output-intensive task; analyzing the load state, the task type and the initial concurrency of the server based on a task scheduling model to generate a scheduling strategy corresponding to the SWG parallel construction task, wherein the initial concurrency of the server is calculated based on a multi-dimensional mapping model. The application solves the technical problem that the SWG parallel construction in the related art generally adopts a fixed concurrency or a simple equal allocation strategy, it is difficult to realize dynamic perception of system load and task characteristics, and the SWG parallel construction efficiency is low and the stability is insufficient.

Description

Technical Field

[0001] This application relates to the field of task allocation, and more specifically, to a scheduling method, apparatus, and electronic device for parallel construction tasks of a static website generator. Background Technology

[0002] Static Website Generator (SWG) effectively reduces the time required to build large-scale static websites by breaking down the processes of source file parsing, template rendering, resource compression, and file output into multiple independent subtasks and using multithreading or coroutines to execute them in parallel.

[0003] However, in related technologies, SWG's parallel build process generally adopts a fixed concurrency strategy, such as setting the number of concurrent tasks to an equal value or a fixed multiple of the number of server CPU cores, or using an equal task allocation method to distribute all build tasks evenly to each execution thread. This approach fails to perceive and adapt to the real-time load status of the system or the resource characteristics of the tasks themselves, resulting in significant resource waste and difficulty in adapting to different server configurations and large-scale content build requirements. For example, when the server is under light load, the fixed concurrency number results in a large number of CPU cores being idle, failing to increase task concurrency and leading to significantly low build efficiency; while under heavy load, excessive concurrent tasks cause the CPU utilization to surge to 100%, easily causing process blocking, memory overflow, disk I / O contention, and ultimately leading to build interruption.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a scheduling method, apparatus, and electronic device for parallel construction tasks of a static website generator, which at least solves the technical problem that SWG parallel construction in related technologies generally adopts a fixed number of concurrent users or a simple equal distribution strategy, making it difficult to achieve dynamic perception of system load and task characteristics, resulting in low efficiency and insufficient stability of SWG parallel construction.

[0006] According to one aspect of the embodiments of this application, a scheduling method for a parallel construction task of a static website generator (SWG) is provided, comprising: determining the load state of the server executing the SWG parallel construction task; determining the task type of subtasks in the SWG parallel construction task, wherein the task type includes computationally intensive tasks and input / output intensive tasks, wherein computationally intensive tasks represent subtasks that consume central processing unit (CPU) computing resources, and input / output intensive tasks represent subtasks that wait for input and output devices to complete read and write operations; analyzing the load state, task type, and initial concurrency of the server based on a task scheduling model, and generating a scheduling strategy corresponding to the SWG parallel construction task, wherein the task scheduling model is used to allocate subtasks of different task types in the SWG parallel construction task to corresponding processors for processing, and the initial concurrency is calculated based on a multidimensional mapping model and is used to represent the maximum number of initial subtasks that the server is allowed to execute simultaneously.

[0007] Optionally, determining the load status of the server executing the parallel building task of the static website generator SWG includes: acquiring server indicator data according to a preset sampling period, wherein the indicator data includes at least the server's CPU utilization, context switching frequency, and process occupancy rate; preprocessing the indicator data; and dividing the preprocessed indicator data according to a preset load threshold to obtain the server's load level, wherein the load level is used to reflect the server's load status.

[0008] Optionally, the method further includes: determining the initial concurrency of the server; scaling up the initial concurrency according to a first preset ratio when the server is at a first load level; maintaining the initial concurrency when the server is at a second load level, wherein the load level corresponding to the second load level is higher than the load level corresponding to the first load level; and scaling down the initial concurrency according to a second preset ratio when the server is at a third load level, wherein the second preset ratio is less than the first preset ratio, and the load level corresponding to the third load level is higher than the load level corresponding to the second load level.

[0009] Optionally, the initial concurrency is determined as follows: Task data corresponding to the SWG parallel construction task is obtained, and a first feature vector corresponding to the task data is determined; server resource data is obtained, and a second feature vector corresponding to the server resource data is determined; a third feature vector corresponding to the server's indicator data is determined; the first, second, and third feature vectors are processed through a multidimensional mapping model to output the initial concurrency of the server, wherein the multidimensional mapping model includes a mapping function for establishing the mapping relationship between the server's resource features, indicator features, and the task features of the SWG parallel construction task.

[0010] Optionally, after determining the task type of the subtask in the SWG parallel construction task, the method further includes: classifying and parsing the SWG parallel construction task according to a preset parsing method to obtain the task type corresponding to each subtask in the SWG parallel construction task; obtaining auxiliary features of each subtask, wherein the auxiliary features include at least the execution time of each subtask and the dependency relationship between each subtask; and generating a task classification list corresponding to the SWG parallel construction task based on the task type and auxiliary features.

[0011] Optionally, the method further includes: determining the task priority of each subtask in the SWG parallel construction task based on the server's load status, the server's initial concurrency, the task types in the task classification list, and the task dependencies, wherein the task priority of computationally intensive tasks is negatively correlated with the server's load level, and the task priority of input / output intensive tasks is positively correlated with the server's load level.

[0012] Optionally, based on the task scheduling model, the load status, task type, and initial concurrency of the server are analyzed to generate a scheduling strategy corresponding to the SWG parallel construction task. This includes: analyzing the server load status, task type, initial concurrency of the server, and task priority based on the task scheduling model to generate a scheduling strategy corresponding to the SWG parallel construction task. The task scheduling model includes a task execution unit pool, a priority-aware channel, and a processor isolation strategy. The task execution unit pool includes a first unit pool for executing computationally intensive tasks and a second unit pool for executing input / output intensive tasks. The priority-aware channel includes a first communication channel for sensing high-priority tasks with a priority greater than or equal to a preset threshold and a second communication channel for sensing low-priority tasks with a priority less than a preset threshold. The processor isolation strategy includes a first processor corresponding to computationally intensive tasks and a second processor corresponding to input / output intensive tasks.

[0013] Optionally, the method further includes: obtaining the hardware configuration parameters of the target server, wherein the target server is any server other than the server; determining the server configuration template corresponding to the hardware configuration parameters according to preset configuration rules; and determining the target scheduling strategy corresponding to the target server when executing the SWG parallel construction task according to the server configuration template.

[0014] According to another aspect of the embodiments of this application, a scheduling device for a parallel construction task of a static website generator (SWG) is also provided, comprising: a first determining module, configured to determine the load state of the server executing the parallel construction task of the SWG static website generator; a second determining module, configured to determine the task type of the subtasks in the SWG parallel construction task, wherein the task type includes computationally intensive tasks and input / output intensive tasks, wherein computationally intensive tasks represent subtasks that consume central processing unit (CPU) computing resources, and input / output intensive tasks represent subtasks that wait for input and output devices to complete read / write operations; and an analysis module, configured to analyze the load state, task type, and initial concurrency of the server based on a task scheduling model, and generate a scheduling strategy corresponding to the SWG parallel construction task, wherein the task scheduling model is used to allocate subtasks of different task types in the SWG parallel construction task to corresponding processors for processing, and the initial concurrency is calculated based on a multidimensional mapping model, which represents the maximum number of initial subtasks that the server is allowed to execute simultaneously.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and used to execute the scheduling method for implementing the above-described parallel construction task of a static website generator.

[0016] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the scheduling method for the parallel construction task of the static website generator by running the computer program.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the scheduling method for the parallel construction task of the static website generator described above.

[0018] In this embodiment, the load status of the server executing the SWG parallel construction task is determined; the task type of the subtasks in the SWG parallel construction task is determined, wherein the task type includes computationally intensive tasks and input / output intensive tasks. Computationally intensive tasks represent subtasks that consume CPU computing resources, and input / output intensive tasks represent subtasks that wait for input and output devices to complete read and write operations; based on the task scheduling model, the load status, task type, and initial concurrency of the server are analyzed to generate a scheduling strategy corresponding to the SWG parallel construction task. The task scheduling model is used to classify different task types in the SWG parallel construction task. Subtasks of a certain type are assigned to corresponding processors for processing. The initial concurrency is calculated based on a multidimensional mapping model and is used to represent the maximum number of initial subtasks that the server is allowed to execute simultaneously. This achieves the goal of generating the optimal scheduling strategy based on the server's real-time load status and the subtask type of the SWG parallel construction task. This improves SWG construction efficiency, enhances system stability, and seamlessly adapts to heterogeneous server environments. Furthermore, it solves the technical problem of low efficiency and instability in SWG parallel construction, which is caused by the fact that related technologies generally adopt fixed concurrency or simple equal allocation strategies, making it difficult to dynamically perceive system load and task characteristics. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a hardware structure diagram of a computer terminal for implementing a scheduling method for parallel construction tasks of a static website generator according to an embodiment of this application;

[0021] Figure 2 This is a flowchart of a method for scheduling parallel construction tasks of a static website generator according to an embodiment of this application;

[0022] Figure 3 This is a flowchart illustrating a process that integrates real-time CPU load sensing and dynamic concurrency control according to an embodiment of this application.

[0023] Figure 4 This is a schematic diagram of a process that integrates task feature classification and load linkage allocation according to an embodiment of this application;

[0024] Figure 5 This is a flowchart illustrating a process that integrates automatic detection and adaptation strategies for server configurations according to an embodiment of this application.

[0025] Figure 6 This is a structural diagram of a scheduling device for parallel construction tasks of a static website generator according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] First, some nouns or terms that appear in the explanation of the embodiments of this application shall be interpreted as follows:

[0029] SWG (Static Website Generator): A tool that automatically compiles source files (such as Markdown, templates, and resources) into static HTML, CSS, and JS files for deploying websites that do not require backend services.

[0030] CPU (Central Processing Unit): The core hardware of a computer that executes program instructions. In this application, the system load status is determined by real-time monitoring of its utilization rate, context switching frequency, and other indicators, which serves as the basis for dynamic scheduling.

[0031] IO (Input / Output): refers to data read and write operations between the system and external devices (such as disks and networks). In this application, IO-intensive tasks refer to tasks that mainly consume time waiting for disk or network responses (such as file output and resource loading), forming a key distinction from CPU-intensive tasks (such as template rendering) in terms of resource consumption characteristics.

[0032] Goroutines are lightweight concurrent execution units built into the Go language. They are scheduled by the Go runtime and are lighter and less overhead than operating system threads. They are the basic carrier for implementing parallel build task execution in this application, with each build subtask being carried by a Goroutine.

[0033] Worker: In this application, it may be referred to as a "work unit" or "work cell". It refers to the Goroutine instance that actually executes the construction subtask. It is the basic execution body for concurrent task scheduling. Instead of using a fixed number of Workers, it is dynamically created or destroyed according to the load and task backlog, so as to realize the elastic allocation of resources.

[0034] GMP (Goroutine-Processor-Machine): A concurrent scheduling architecture based on the Go language, in which G (task), M (thread), and P (logic processor) work together.

[0035] From a technological development perspective, with the continuous expansion of static website application scenarios, SWG technology has evolved from an early single-file compilation tool into a complex build system supporting massive amounts of content, multiple template nesting, and multi-resource linkage. Early SWGs (such as the first-generation Jekyll and Hugo) were limited by hardware resources and technical concepts, mostly adopting a serial build mode, which was inefficient and could only meet the build needs of small static sites (dozens to hundreds of pages). With the increasing use of multi-core CPUs, diversified server configurations, and the widespread adoption of large-scale content sites (thousands or even tens of thousands of pages), parallel build has become the core direction for SWG performance optimization. Various SWG tools have gradually introduced concurrency mechanisms, but due to limitations in technological maturity, most solutions remain at a rudimentary stage of fixed concurrency (such as setting a fixed number of concurrent tasks equal to the number of CPU cores) or simple equal distribution (splitting tasks evenly across concurrent nodes), making it difficult to adapt to the current diverse server configurations and large-scale content build scenarios. Specific shortcomings are as follows:

[0036] 1. Lack of load awareness leads to an imbalance between build efficiency and stability. In related technologies, SWG parallel builds do not incorporate CPU load awareness technology, failing to capture real-time changes in key metrics such as CPU utilization and context switching frequency. Under light loads, idle CPU resources are not fully utilized, resulting in insufficient concurrent tasks and significantly lengthening the build cycle for large-scale content. Even when the server is idle, build efficiency cannot be improved. Under heavy loads, concurrent tasks are not reduced in time, leading to excessive tasks vying for CPU resources, easily causing process blocking and memory overflows, ultimately resulting in build interruptions and severely impacting SWG build stability.

[0037] 2. Unreasonable task allocation and serious resource waste. Task allocation was not optimized in conjunction with the core logic of SWG parallel build. Instead, simple equal allocation or fixed concurrency mode was used, ignoring the characteristics and differences of different build tasks. For example, template rendering is a CPU-intensive task and file output is an I / O-intensive task. Allocating resources for both according to the same standard resulted in a mismatch between CPU resources and task requirements, leading to uneven workloads. This not only caused ineffective consumption of hardware resources such as CPU and memory, but also further aggravated the loss of build efficiency.

[0038] 3. Poor server adaptability and insufficient scenario coverage. The system lacks server configuration adaptation technology, failing to automatically adapt to hardware differences between servers with varying configurations. Whether it's a low-end (2 cores, 4GB RAM), mid-range (4-8 cores, 8-16GB RAM), or high-end (16 cores and above) server, a uniform concurrency strategy is used. This results in the high-end server's hardware advantages being underutilized, leading to severe resource waste, while low-end servers frequently experience build anomalies due to resource overload. This makes it difficult to adapt to the actual needs of large-scale content creation and diverse server deployments.

[0039] To address the aforementioned issues, this application provides a scheduling method for parallel construction tasks of a static website generator, which can run on... Figure 1 The computer terminal shown is described below.

[0040] The scheduling method for parallel construction tasks of a static website generator provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for scheduling a method to implement parallel construction tasks of a static website generator is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0041] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0042] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the scheduling method for the parallel construction task of the static website generator in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the scheduling method for the parallel construction task of the static website generator described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0043] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for wireless communication with the Internet.

[0044] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0045] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0046] In the above operating environment, this application provides an embodiment of a scheduling method for parallel construction tasks of a static website generator. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] Figure 2 This is a flowchart of a scheduling method for parallel construction tasks of a static website generator according to an embodiment of this application, as shown below. Figure 2 As shown, the method includes the following steps:

[0048] Step S202: Determine the load status of the server executing the static website generator SWG parallel build task.

[0049] In step S202 above, the current operating status of the server executing the SWG parallel build task is first quantitatively evaluated by collecting three core indicators in real time: CPU utilization, context switching frequency, and process occupancy. Simultaneously, the collected data is compared and judged against preset three-level load thresholds (e.g., light load: CPU utilization < 50%; normal load: 50% ≤ CPU utilization ≤ 80%; heavy load: CPU utilization > 80%) to determine the server's current load level. This process does not rely on manual intervention or fixed configurations; instead, it automatically and accurately identifies the load status based on real-time collected system operating data, providing a direct and objective decision-making basis for subsequent dynamic adjustments to the concurrency.

[0050] Step S204: Determine the task type of the subtask in the SWG parallel construction task. The task type includes computationally intensive tasks and input / output intensive tasks. Computationally intensive tasks are subtasks that consume CPU computing resources, and input / output intensive tasks are subtasks that wait for input and output devices to complete read and write operations.

[0051] In step S204 above, it is necessary to clearly define the type classification of each subtask in the SWG parallel construction task. Specifically, this is reflected in classifying each subtask into compute-intensive (CPU-intensive) tasks and input / output-intensive (I / O-intensive) tasks based on the resource consumption characteristics of the SWG parallel construction task. CPU-intensive tasks specifically refer to subtasks that primarily consume CPU computing resources during execution, characterized by frequent arithmetic operations, logical judgments, or data processing, while relying less on external device read / write waiting. I / O-intensive tasks, on the other hand, specifically refer to subtasks that primarily rely on waiting for input and output devices to complete read / write operations during execution, characterized by execution time mainly consumed in I / O waiting phases such as file reading, disk writing, or network transmission, rather than continuous CPU computation. This classification is based solely on the direct dependency of the task on CPU computing resources or input / output device read / write operations, and does not involve additional characteristics such as task content, file format, or call chain.

[0052] Step S206: Analyze the load status, task type, and initial concurrency of the server based on the task scheduling model, and generate a scheduling strategy corresponding to the SWG parallel construction task. The task scheduling model is used to allocate subtasks of different task types in the SWG parallel construction task to the corresponding processors for processing. The initial concurrency is calculated based on the multidimensional mapping model and is used to represent the maximum number of initial subtasks that the server is allowed to execute simultaneously.

[0053] In step S206 above, the load status, task type, and initial concurrency of the server are analyzed based on a task scheduling model (such as the GMP model). This mainly involves simultaneously evaluating the current load status of computing resources, the type characteristics of each subtask in the SWG parallel construction task, and the initial concurrency calculated based on the multidimensional mapping model. The load status is used to determine the stress level of system resources, and each subtask is identified as either CPU-intensive or I / O-intensive. The maximum number of initial subtasks output by the multidimensional mapping model is used as a dynamic benchmark, providing a basis for subsequent task allocation decisions. Generating the scheduling strategy corresponding to the SWG parallel construction task involves directly outputting a set of explicit instructions based on the above analysis results. This specifies which processors should be assigned to subtasks of different task types, ensuring a load-type matching relationship between tasks and processor resources. This enables targeted task distribution in a multiprocessor environment, which does not rely on fixed rules or equal allocation but is dynamically determined by the scheduling model based on real-time load and task type.

[0054] Through steps S202 to S206, the goal of generating an optimal scheduling strategy based on the server's real-time load status and the sub-task types of the SWG parallel construction task is achieved. This improves SWG construction efficiency, enhances system stability, and seamlessly adapts to heterogeneous server environments. Furthermore, it solves the technical problem of low efficiency and instability in SWG parallel construction, which is often achieved through fixed concurrency or simple equal allocation strategies in related technologies, making it difficult to dynamically perceive system load and task characteristics. The following is a detailed explanation.

[0055] In step S202 above, determining the load status of the server executing the parallel construction task of the static website generator SWG includes: obtaining server indicator data according to a preset sampling period, wherein the indicator data includes at least the server's CPU utilization, context switching frequency, and process occupancy rate; preprocessing the indicator data; and dividing the preprocessed indicator data according to a preset load threshold to obtain the server's load level, wherein the load level is used to reflect the server's load status.

[0056] In this embodiment, addressing the shortcomings of "lack of load awareness and imbalance between construction efficiency and stability" in related technologies, a technical approach of "real-time CPU load awareness + dynamic concurrency control" is adopted, which differs from fixed concurrency or simple allocation strategies: ① An integrated real-time CPU load acquisition module is used, combining system call interfaces (such as the Linux system top command, Go language runtime package) with third-party monitoring libraries to periodically collect three core indicators: CPU utilization, context switching frequency, and process occupancy. The sampling period is set to 100ms to ensure the real-time nature of load awareness; ② Three load thresholds are preset (light load: CPU utilization < 50%; normal load: 50% ≤ CPU utilization ≤ 80%; heavy load: CPU utilization > 80%), and the load level is automatically determined based on the collected indicators; ③ The concurrency level is dynamically adjusted based on load level. Under light load, the concurrency is gradually increased by 20% of the current concurrency (not exceeding 3 times the number of CPU cores). Under heavy load, the concurrency is gradually decreased by 10% of the current concurrency (not less than the number of CPU cores). Under normal load, the current concurrency is maintained to prevent idleness under light load and overload under heavy load. The overall process is as follows: Figure 3 As shown, the specific steps include the following:

[0057] 1. Configure the acquisition parameters.

[0058] Start the real-time CPU load acquisition module with a preset sampling period of 100ms (balancing real-time acquisition with system resource consumption, avoiding excessive CPU resource consumption from high-frequency sampling). Define the three core indicators to be acquired and their acquisition methods: CPU utilization (acquired through the Linux system top command and the Go language runtime package to ensure data accuracy), context switching frequency (obtained through the system call interface syscall), and process utilization (collected by combining the third-party monitoring library gopsutil to collect the CPU utilization percentage of the current SWG build process). At the same time, set an abnormal threshold for indicators. When an indicator fails to be acquired or the data is abnormal (such as CPU utilization > 100%), the backup acquisition method will be automatically activated to ensure that the acquisition process is not interrupted.

[0059] 2. Periodically collect indicator data and perform preprocessing.

[0060] Based on a preset sampling period (100ms), three types of core indicator data are continuously collected. Simultaneously, the collected raw indicator data is preprocessed: outliers are removed (using 3... The principle is to remove data that deviates from the mean by 3 times the standard deviation, and to normalize the data (map the data to the 0-100 range to eliminate the difference in the units of different indicators) to obtain standardized indicator data, which provides reliable support for subsequent load level determination and dynamic mapping model calculation, and avoids judgment errors caused by the deviation of the original data.

[0061] 3. Determine the load level and output the result.

[0062] The system receives preprocessed metric data and makes a comprehensive judgment based on preset three-level load thresholds (light load: CPU utilization < 50%; normal load: 50% ≤ CPU utilization ≤ 80%; heavy load: CPU utilization > 80%), combined with context switching frequency and process occupancy (e.g., if CPU utilization is 45% but context switching frequency is too high, combined with process occupancy, if it is judged as light load, the expansion range is appropriately reduced). Finally, it outputs the current system load level (e.g., first load level (light load), second load level (normal load), third load level (heavy load)) and records the judgment timestamp to form a load level log.

[0063] 4. Dynamically adjust the number of concurrent connections based on the load level.

[0064] (1) Determine the initial concurrency through a multidimensional mapping model (dynamic mapping model), including: obtaining task data corresponding to the SWG parallel construction task and determining the first feature vector corresponding to the task data; obtaining server resource data and determining the second feature vector corresponding to the server resource data; determining the third feature vector corresponding to the server's indicator data; processing the first feature vector, the second feature vector and the third feature vector through the multidimensional mapping model, and outputting the initial concurrency of the server, wherein the multidimensional mapping model contains a mapping function for establishing the mapping relationship between the server's resource features, indicator features and the task features of the SWG parallel construction task.

[0065] Specifically, this dynamic mapping model is used to establish a mapping relationship between server resource characteristics, CPU metric characteristics, and task characteristics, and outputs the optimal concurrency as the benchmark value for the number of Workers. The specific expression is as follows:

[0066]

[0067] In the formula, This represents the output value of the dynamic mapping model, which is the optimal number of concurrent connections under the current server resources, load status, and task characteristics (and also serves as the initial baseline value for the number of Workers). This represents the first feature vector corresponding to the task data (dimension: CPU usage). IO ratio Execution time Dependency number ); This represents the second feature vector corresponding to the server resource data (dimension: number of CPU cores). Memory size Disk IOPS ); This represents the third feature vector corresponding to the indicator data (dimension: CPU utilization). Context switching frequency ); This represents the weights corresponding to each single-dimensional mapping function (obtained through training, with weights representing contributions from CPU-intensive tasks). =0.6, weight of contribution for I / O intensive tasks =0.5, the weight of task duration =0.3, task dependency impact weight =0.2); This represents the bias term (baseline value, default is 0.1); This represents a one-dimensional mapping function. The upper bound of the summation sign, 4, indicates the number of one-dimensional mapping functions, including... The specific expression is as follows:

[0068]

[0069]

[0070]

[0071]

[0072] In the formula, This indicates the concurrency potential of CPU-intensive tasks under the current server and load conditions. This indicates the percentage of CPU-intensive tasks (0-1). Indicates the number of CPU cores in the server. Indicates CPU utilization;

[0073] This indicates the concurrency potential of I / O intensive tasks under current disk and scheduling pressure. This indicates the percentage of I / O intensive tasks (0-1). Indicates disk IOPS, Indicates the frequency of context switching;

[0074] This indicates the CPU's parallel processing capabilities required for long-running tasks. Indicates the execution time of the task. Indicates the server's memory capacity;

[0075] This indicates the CPU's parallel processing capability requirements based on the task's dependency depth. Indicates the task dependency depth.

[0076] In the above process, by acquiring the task data of the SWG parallel construction task and extracting its first feature vector, the resource data and operation index data of the server are collected simultaneously to generate the second and third feature vectors respectively. Then, by using the mapping function built into the multidimensional mapping model, the dynamic correlation between task features, resource features and index features is comprehensively modeled, thereby outputting an initial concurrency number that matches the actual carrying capacity and task characteristics of the current system. This ensures that subsequent expansion, maintenance or reduction operations based on load level are based on scientifically quantified and multidimensionally perceived benchmark parameters, effectively avoiding the resource waste or performance bottlenecks caused by setting the concurrency number based on experience or static configuration in traditional methods, and significantly improving the accuracy of scheduling strategies, response speed and system stability.

[0077] (2) Dynamically adjust the number of concurrent users based on the initial number of concurrent users and the load level, including: determining the initial number of concurrent users of the server; expanding the initial number of concurrent users according to a first preset ratio when the server is at the first load level; maintaining the initial number of concurrent users when the server is at the second load level, wherein the load level corresponding to the second load level is higher than the load level corresponding to the first load level; and reducing the initial number of concurrent users according to a second preset ratio when the server is at the third load level, wherein the second preset ratio is less than the first preset ratio, and the load level corresponding to the third load level is higher than the load level corresponding to the second load level.

[0078] Specifically, when combining the baseline value (initial concurrency) output by the dynamic mapping model and performing concurrency control according to load level, the concurrency control rules involved are as follows:

[0079] Light load (CPU utilization < 50%): Gradually expand the capacity by 20% of the initial concurrency (i.e., the first preset ratio mentioned above) (maximum not exceeding 3 times the number of CPU cores and not exceeding 1.2 times the model output baseline value). After each expansion, wait 500ms and collect load data again to determine the load level. If it is still light load, continue to expand the capacity. If the load level becomes normal during the expansion process, stop the expansion and maintain the current concurrency.

[0080] Normal load (50%≤CPU utilization≤80%): Maintain the initial concurrency level, continuously monitor load data, and immediately trigger the corresponding control logic if the load level changes (becomes light or heavy).

[0081] Heavy load (CPU utilization > 80%): Gradually reduce the load by 10% of the initial concurrency (i.e., the second preset ratio mentioned above) (the minimum should not be less than the number of CPU cores and not less than 0.8 times the model output baseline value). After each reduction, wait 300ms and collect load data again to determine the load level. If it is still a heavy load, continue to reduce the load. If the load level becomes normal during the reduction process, stop reducing the load and maintain the current concurrency.

[0082] In the above process, the maximum number of initial subtasks that the server can execute simultaneously (the current optimal concurrency) is determined as the baseline value for concurrent scheduling. Combined with the acquired server load status, this initial concurrency is dynamically adjusted according to the load level, effectively avoiding a sharp drop in task processing capacity due to excessive scaling down. Furthermore, the rationality of the adjustment logic is ensured by setting a progressive relationship between load levels. This mechanism establishes a closed-loop response relationship between concurrent execution capacity and real-time system load, effectively alleviating the problem of resource idleness under light load and task backlog under heavy load with a fixed concurrency, achieving synergistic optimization of resource utilization and task processing efficiency.

[0083] In this embodiment, addressing the shortcomings of "unreasonable task allocation and serious resource waste" in related technologies, a "task feature classification + load-linked allocation" technique is adopted, differing from the inherent equal allocation mode: ① First, the entire SWG construction process tasks are classified and analyzed. Through task metadata tags and static analysis (file extensions, call chains), tasks are divided into two core types: CPU-intensive (template rendering) and IO-intensive (file output), clarifying the resource consumption characteristics of each type of task; ② A task priority mechanism is established: the priority of CPU-intensive tasks is negatively correlated with CPU load (priority decreases under heavy load), while the priority of IO-intensive tasks is positively correlated with CPU load (priority increases under light load); ③ Resources are dynamically allocated based on load level and task type. CPU-intensive tasks are bound to odd-numbered P (processors), and IO-intensive tasks are bound to even-numbered P, avoiding resource competition between different types of tasks, achieving load-adaptive task allocation and resource matching needs, and reducing ineffective resource consumption. The overall process is as follows: Figure 4 As shown, the specific steps include the following:

[0084] 1. Initialize task classification, clarifying classification criteria and parsing methods.

[0085] The classification criteria for the entire SWG parallel build process are pre-defined. Based on the resource consumption characteristics of the tasks, all subtasks are divided into two core types: ① CPU-intensive tasks (mainly template rendering and resource compression, characterized by high CPU utilization, low IO utilization, and stable execution time); ② IO-intensive tasks (mainly source file parsing and static file output, characterized by high IO utilization, low CPU utilization, and large fluctuations in execution time).

[0086] Subsequently, the classification and parsing method was clarified, adopting a dual parsing approach of "task metadata tags + static analysis" to ensure classification accuracy: ① Task metadata tags: When each build task is generated, a task-type tag is automatically added (e.g., task-type:render represents CPU-intensive, task-type:output represents IO-intensive); ② Static analysis: By parsing the task's file extension (e.g., .md, .html template files correspond to parsing / rendering tasks) and call chain (e.g., calling rendering functions corresponds to CPU-intensive tasks), the metadata tags are verified and corrected to avoid classification deviations caused by tag errors.

[0087] 2. Task classification analysis, output classification results.

[0088] After determining the task type of the subtask in the SWG parallel construction task, the process also includes: classifying and parsing the SWG parallel construction task according to a preset parsing method to obtain the task type corresponding to each subtask in the SWG parallel construction task; obtaining auxiliary features of each subtask, wherein the auxiliary features include at least the execution time of each subtask and the dependency relationship between each subtask; and generating a task classification list corresponding to the SWG parallel construction task based on the task type and auxiliary features.

[0089] Specifically, by performing fine-grained classification and analysis of SWG parallel construction tasks through a preset analysis method, the CPU-intensive or I / O-intensive attributes of each subtask can be accurately identified. At the same time, combined with auxiliary features such as the obtained execution time and inter-task dependencies, a structured task classification list is constructed. This allows the generation of subsequent scheduling strategies to no longer rely solely on coarse-grained load status and task type, but to be dynamically optimized based on the specific execution characteristics and collaborative relationships of each subtask. As a result, CPU-intensive tasks can be prioritized for allocation to high-performance cores, and I / O-intensive tasks can be reasonably scheduled to processors with sufficient I / O resources. This effectively avoids resource contention and idle waste, and solves the technical problems of insufficient scheduling basis and unbalanced resource allocation caused by fuzzy task feature identification.

[0090] 3. Establish a task priority mechanism that links CPU load with worker priority.

[0091] Based on the server's load status, initial concurrency, task types in the task classification list, and task dependencies, the task priorities of each subtask in the SWG parallel construction task are determined. Among them, the task priorities of computationally intensive tasks are negatively correlated with the server's load level, while the task priorities of input / output intensive tasks are positively correlated with the server's load level.

[0092] Specifically, a 5-level task priority system (1-5, with 5 being the highest) is preset. Based on task type and server load status, a priority linkage mechanism is established, which is also linked to Worker priority. This lays the foundation for subsequent priority preemption and concurrency optimization. The core rules are as follows:

[0093] (1) CPU-intensive tasks: The priority is negatively correlated with the CPU load level. Under heavy load (CPU > 80%), the priority is set to level 1, under normal load, it is set to level 3, and under light load, it is set to level 5.

[0094] (2) I / O intensive tasks: The priority is positively correlated with the CPU load level. It is set to level 5 under heavy load, level 3 under normal load, and level 1 under light load.

[0095] (3) Special rules: For subtasks with dependencies, the dependent task has a higher priority than the dependent task (e.g., the source file parsing task has a higher priority than the template rendering task) to ensure the continuity of task execution;

[0096] (4) Worker priority binding: High-priority tasks are assigned to high-priority workers, and low-priority tasks are assigned to low-priority workers to avoid low-priority workers from taking over high-priority task resources.

[0097] In the above process, by considering the server's load status, initial concurrency, task types in the task classification list, and their dependencies, the task priority of each subtask in the SWG parallel construction task can be dynamically determined. Based on the adjustment mechanism that CPU-intensive tasks are negatively correlated with load level and I / O-intensive tasks are positively correlated with load level, a differentiated scheduling strategy for different types of tasks is implemented: when the server load increases, the priority of CPU-intensive tasks is reduced to alleviate CPU contention pressure, while the priority of I / O-intensive tasks is increased to make full use of idle computing resources; when the server load decreases, the opposite adjustment is made, prioritizing the scheduling of CPU-intensive tasks to accelerate the overall construction process, while taking into account the resource adaptability of I / O-intensive tasks. This effectively solves the problem of resource allocation imbalance caused by statically fixed task priorities, and improves the responsiveness of task scheduling to changes in system load and resource utilization efficiency.

[0098] 4. Load-linked task allocation achieves resource isolation and optimizes the use of Go's concurrency features.

[0099] Optionally, based on the task scheduling model, the load status, task type, and initial concurrency of the server are analyzed to generate a scheduling strategy corresponding to the SWG parallel construction task. This includes: analyzing the server load status, task type, initial concurrency of the server, and task priority based on the task scheduling model to generate a scheduling strategy corresponding to the SWG parallel construction task. The task scheduling model includes a task execution unit pool, a priority-aware channel, and a processor isolation strategy. The task execution unit pool includes a first unit pool for executing computationally intensive tasks and a second unit pool for executing input / output intensive tasks. The priority-aware channel includes a first communication channel for sensing high-priority tasks with a priority greater than or equal to a preset threshold and a second communication channel for sensing low-priority tasks with a priority less than a preset threshold. The processor isolation strategy includes a first processor corresponding to computationally intensive tasks and a second processor corresponding to input / output intensive tasks.

[0100] In this embodiment, the GMP scheduling model based on Go language features is adopted, and further optimizations are made for task allocation scenarios as follows:

[0101] (1) Further optimization of Go's concurrency features: ① Task execution unit pool (i.e., task-specific Goroutine pool): Dedicated Goroutine pools are established for CPU-intensive and IO-intensive tasks respectively. The pool size can be adjusted based on the output value (initial concurrency) of the dynamic mapping model to avoid competition caused by different types of tasks sharing the Goroutine pool; ② Priority-aware channel (i.e., directed channel communication): High-priority tasks use independent directed channels (first communication channel), and low-priority tasks use shared channels (second communication channel) to avoid low-priority tasks blocking the communication of high-priority tasks; ③ Processor isolation strategy: A P (processor) group isolation strategy is adopted. Odd-numbered P are bound to the CPU-intensive task Goroutine pool, and even-numbered P are bound to the IO-intensive task Goroutine pool. At the same time, a task queue priority is set for each P, and high-priority tasks are executed first to reduce context switching caused by cross-priority task switching.

[0102] (2) Resource Isolation and Task Allocation Logic: ① Resource Isolation: Based on the Go language GMP scheduling model, the CPU processors are divided into two groups. Odd-numbered P groups are bound to CPU-intensive tasks, and even-numbered P groups are bound to I / O-intensive tasks. This avoids different types of tasks competing for the same P resource, thereby reducing context switching. At the same time, independent memory caches are allocated to the two groups of P to avoid cache contention. ② Allocation Order: High-priority tasks are allocated first, followed by medium and low-priority tasks. Under the same priority, tasks with short execution time and no dependencies are allocated first to improve task execution efficiency. ③ Load-Linked Adjustment: The allocation ratio of different types of tasks is dynamically adjusted according to the current CPU load level. Under light load (CPU < 50%), the allocation ratio of CPU-intensive tasks is increased (60%) to make full use of CPU resources. Under heavy load (CPU > 80%), the allocation ratio of I / O-intensive tasks is increased (60%) to reduce CPU usage pressure. Under normal load, the allocation ratio of the two types of tasks is balanced (50% each). ④ Worker allocation: Based on task priority and type, tasks are assigned to corresponding Workers. High-priority tasks take the lead in occupying idle Workers, thus achieving precise scheduling of Workers.

[0103] Based on the above optimization rules, the GMP scheduling model is used to analyze multi-dimensional characteristics such as server load status, task type, initial server concurrency, and task priority to generate a scheduling strategy corresponding to the SWG parallel construction task. The specific process analysis is as follows:

[0104] First, the task type (including task type label and task feature vector) is processed through a dedicated Goroutine pool (task execution unit pool). The tasks to be executed are injected into two isolated Goroutine pools. The first unit pool is dedicated to executing CPU-intensive tasks such as template rendering and resource compression, while the second unit pool is dedicated to executing I / O-intensive tasks such as file output and resource loading. The output is assigned to the queue of tasks to be executed in the corresponding execution pool, which are physically separated task buffers, each bound to its own Goroutine set.

[0105] It should be noted that the execution module does not actively decide on task allocation, but it can dynamically adjust the size of the two pools based on task allocation instructions. For example, when the load is light, the first unit pool expands to more than 60% of the model baseline value, and the second unit pool shrinks; when the load is heavy, the second unit pool expands and the first unit pool shrinks.

[0106] Secondly, based on the task priority score of each subtask through the Channel (priority-aware channel), the subtasks are classified according to a preset threshold (e.g., priority ≥ 4 is high priority). The subtasks are then injected into two independent communication channels (the first communication channel only receives submission requests from high-priority tasks, using a dedicated channel with large capacity, low latency, and no blocking; the second communication channel receives low-priority tasks, using a shared channel, allowing queuing and waiting) for communication. Subsequently, a task submission stream isolated by priority is output to ensure that high-priority tasks can enter the execution pool with the shortest path and lowest latency, avoiding communication blockage caused by the backlog of low-priority tasks, thereby ensuring the timeliness of core tasks.

[0107] Finally, based on the processor isolation strategy, according to the list of currently available CPU logical processors (P) and the binding status of the task execution unit pool, as well as the aforementioned hard rule of "odd P bound to CPU tasks, even P bound to I / O tasks", all P are divided into two groups: the first processor group (odd P) only obtains tasks from the first unit pool and executes them, and the second processor group (even P) only obtains tasks from the second unit pool and executes them. Then, a physically isolated execution environment is output, that is, I / O tasks are never executed on odd P, and CPU-intensive tasks are never executed on even P, thereby eliminating cache pollution, scheduling disturbances and context switching caused by mixed task types.

[0108] It should be noted that this isolation strategy does not perform task scheduling, but enforces resource binding to ensure that the context environment of each P is pure and only handles one type of task. Its execution result directly supports the safe activation of the work stealing mechanism in the GMP scheduling model: idle P can only steal the same type of task from busy P in the same group, eliminating cross-type scheduling.

[0109] In the above process, by combining the server's load status, task type, initial concurrency, and task priority, and using a task scheduling model that includes task execution unit pools, priority-aware channels, and processor isolation strategies, the optimal scheduling strategy corresponding to the parallel build task of the SWG can be effectively output. For example, when the static website generation system starts a parallel build task on a high-configuration server with 16 cores and 32GB of memory, the task scheduling model, based on the server's current CPU utilization of 75% load status, identifies that the tasks to be processed include CPU-intensive tasks with a large number of template renderings and IO-intensive tasks with a large number of static file outputs. Combining the initial concurrency set to 32 and the task priority distribution, the following scheduling strategy is generated: CPU-intensive tasks are assigned to the first unit pool for execution, IO-intensive tasks are assigned to the second unit pool for execution, high-priority tasks (priority ≥ 4) are scheduled first through the first communication channel, and low-priority tasks are queued through the second communication channel. At the same time, a processor isolation strategy is adopted, binding odd-numbered processors to the first unit pool to handle computationally intensive tasks, and even-numbered processors to the second unit pool to handle input / output intensive tasks. This technical solution ensures full-link priority for high-priority tasks in terms of channel awareness, execution unit allocation, and physical processor resources. It effectively prevents high-priority tasks from being blocked or delayed due to sharing resources with low-priority tasks, and significantly improves the response timeliness of core construction tasks and the overall scheduling efficiency of the system.

[0110] 5. Task allocation monitoring and optimization, supplementing concurrent blocking detection and avoidance.

[0111] The specific detection rules and avoidance logic involved in the continuous monitoring of task allocation and execution status are as follows:

[0112] (1) Concurrency blocking detection: ① For task allocation scenarios, focus on detecting Channel blocking (task submission blocking) and Worker blocking (task execution blocking); ② By parsing task submission logs and Worker execution logs, identify blocked tasks and their corresponding allocation nodes, and record the blocking duration and blocking reason; ③ If the task submission blocking duration is >100ms and the Worker execution blocking duration is >200ms, it is determined to be an abnormal blocking and triggers the avoidance mechanism.

[0113] (2) Blocking avoidance logic: ① Task submission blocking (Channel blocking): Immediately expand the capacity of the Channel for the corresponding priority and type of task, and temporarily increase the number of Goroutines / Workers for this type of task to speed up task consumption and alleviate blocking; ② Worker execution blocking: If the Worker is blocked for a CPU-intensive task, adjust the P bound to the Worker and transfer it to an idle P for execution; if the Worker is blocked for an IO-intensive task, trigger asynchronous IO switching and restart the blocked Worker; ③ Allocation ratio optimization: If a certain type of task frequently blocks, adjust the task allocation ratio to reduce the number of concurrent allocations for this type of task and avoid the spread of blocking.

[0114] 6. Closed-loop feedback calibration optimizes control accuracy.

[0115] Based on task execution efficiency and blocking avoidance effectiveness, the task classification criteria, priority rules, allocation ratio, and Go concurrency optimization parameters are dynamically adjusted and fed back to the dynamic mapping model to optimize model weights. This ensures that task allocation always matches the load state and task characteristics, reduces ineffective resource consumption, and lowers the probability of concurrent blocking.

[0116] Optionally, the above method further includes: obtaining the hardware configuration parameters of the target server, wherein the target server is any server other than the server; determining the server configuration template corresponding to the hardware configuration parameters according to preset configuration rules; and determining the target scheduling strategy corresponding to the target server when executing the SWG parallel construction task according to the server configuration template.

[0117] In this embodiment, addressing the shortcomings of "poor server adaptability and insufficient scenario coverage" in related technologies, a technical approach of "automatic server configuration detection + dynamic generation of adaptation strategies" is adopted, which differs from the inherent unified concurrency strategy: ① A server configuration detection module is added, which automatically collects hardware parameters such as the number of CPU cores, memory capacity, disk IOPS, and disk type (HDD / SSD) of the server through the gopsutil third-party library, eliminating the need for manual configuration; ② Three types of server configuration templates are preset (low-end: ≤2 cores / 4G; mid-end: 4-8 cores / 8-16G; high-end: ≥16 cores / 32G), and the corresponding template is automatically matched based on the detected parameters; ③ A concurrency scheduling strategy is dynamically generated based on the matched template, including core parameters such as basic concurrency, task queue capacity, cache size, and resource allocation weight. For example, high-end servers enable parallel rendering + asynchronous IO, while low-end servers limit the concurrency of CPU-intensive tasks, achieving adaptive adaptation of servers with different configurations and covering large-scale content building scenarios. The overall process is as follows: Figure 5 As shown, the specific steps include the following:

[0118] 1. Configure detection parameters.

[0119] First, the core hardware configuration parameters and detection methods for the probe are clearly defined. The gopsutil third-party library (cross-platform and with high detection accuracy) is used as the core detection tool to comprehensively cover the parameters required for server adaptation. Specific hardware configuration parameters include: number of CPU cores, CPU frequency, memory capacity (total memory, available memory), disk IOPS, disk type (HDD / SSD), cache size, and network bandwidth. At the same time, a probe timeout is set (default 5s). If the probe times out, it will automatically retry the probe (up to 3 times). If the probe still fails, the default configuration (parameters for a medium-sized server) will be output to ensure that the probe process does not affect the start of the SWG build task.

[0120] 2. Automatically detect server configuration and generate detection reports.

[0121] Automatically collect various hardware configuration parameters of the target server, organize and verify the collected hardware configuration parameters (e.g., if the number of CPU cores is 0, it is judged as a detection anomaly and the default value is used; if the memory capacity collection deviation is >10%, it is re-detected), and generate a standardized server configuration detection report, including server configuration type (preliminary judgment), specific values ​​of various hardware parameters, detection timestamp and detection status (success / failure), etc.

[0122] 3. Set up preset configuration rules and perform server configuration template matching.

[0123] Three types of server configuration templates are preset (covering all mainstream server configurations and are scalable), clearly defining the parameter range for each type of template, and also preset basic adaptation parameters for each type of template. The specific templates are as follows:

[0124] (1) Low-configuration server template: CPU ≤ 2 cores, CPU frequency ≤ 2.0GHz, memory ≤ 4G, disk IOPS < 1000, disk type is HDD, cache size ≤ 128MB, network bandwidth ≤ 100Mbps; preset basic adaptation parameters: maximum concurrency = number of CPU cores × 2, cache ratio = memory × 5%;

[0125] (2) Medium-configuration server template: CPU 4-8 cores, CPU frequency 2.0-3.0GHz, memory 8-16G, disk IOPS 1000-3000, disk type HDD / SSD, cache size 128-512MB, network bandwidth 100-1000Mbps; preset basic adaptation parameters: maximum concurrency = number of CPU cores × 1.5, cache ratio = memory × 10%;

[0126] (3) High-configuration server template: CPU ≥ 16 cores, CPU frequency ≥ 3.0 GHz, memory ≥ 32 G, disk IOPS ≥ 3000, disk type is SSD, cache size ≥ 512 MB, network bandwidth ≥ 1000 Mbps; preset basic adaptation parameters: maximum concurrency = number of CPU cores × 3, cache ratio = memory × 15%.

[0127] Based on the detection report, the detected hardware configuration parameters are compared with the preset server configuration templates. The core parameter priority matching principle is adopted (prioritizing the matching of three core parameters: CPU core count, memory capacity, and disk IOPS, with weights of 0.4, 0.3, and 0.3 respectively), and the corresponding server configuration template is automatically matched. If the detected parameters are between two types of templates, the closer template is matched (e.g., 3 CPU cores, 6GB memory, matching the mid-range template and fine-tuning the basic adaptation parameters). At the same time, the matching results and template deviation values ​​are recorded.

[0128] 4. Generate the corresponding scheduling strategy based on the matched server configuration template.

[0129] Based on the preset template-policy mapping rules and combined with dynamic mapping model optimization, a parallel build scheduling policy for SWG is generated for the current server configuration. The specific mapping rules are as follows:

[0130] (1) Low-spec server template adaptation strategy: ① Basic concurrency = number of CPU cores (to avoid overload), maximum concurrency = number of CPU cores × 2 (based on template preset parameters); ② Task queue capacity = number of CPU cores × 10 (to reduce queue backlog and adapt to low-spec memory); ③ Cache optimization: cache size = memory capacity × 5% (to save memory), enable cache eviction mechanism (LRU algorithm), prioritize caching frequently accessed template files; ④ Resource allocation weight: 60% for IO-intensive tasks, 40% for CPU-intensive tasks (to adapt to the weak IO performance of HDD disks); ⑤ Task execution mode: CPU-intensive tasks are executed serially, IO-intensive tasks are executed in parallel, and the number of CPU-intensive tasks executed simultaneously is limited to ≤2; ⑥ Dynamic mapping model optimization: adjust model weights and increase the weight of the IO dimension ( =0.6), reduce the CPU dimension weight ( =0.3), adapted to low-end CPU performance; ⑦ Other adaptations: disable high-resource compression (such as reducing the compression level from 9 to 6) to reduce CPU and memory usage.

[0131] (2) Mid-range server template adaptation strategy: ① Basic concurrency = number of CPU cores × 1.5, maximum concurrency = number of CPU cores × 2.5; ② Task queue capacity = number of CPU cores × 15; ③ Cache optimization: cache size = memory capacity × 10%, enable second-level cache (memory cache + disk cache), balance cache performance and memory usage; ④ Resource allocation weight: 50% for CPU-intensive tasks, 50% for IO-intensive tasks, balance the use of CPU and IO resources; ⑤ Task execution mode: both types of tasks are executed in parallel, enable batch task processing (reduce task switching); ⑥ Dynamic mapping model optimization: maintain default weight ( =0.6, =0.5), fine-tune the bias term b=0.15 to adapt to the balanced performance of medium-sized servers; ⑦ Other adaptations: compression level=7, dynamically adjust the compression speed in combination with the CPU frequency.

[0132] (3) High-configuration server template adaptation strategy: ① Basic concurrency = number of CPU cores × 2, maximum concurrency = number of CPU cores × 3; ② Task queue capacity = number of CPU cores × 20; ③ Cache optimization: cache size = memory capacity × 15% (fully utilize memory), enable cache preloading mechanism, preload commonly used templates and source files to improve build efficiency; ④ Resource allocation weight: 60% for CPU-intensive tasks, 40% for IO-intensive tasks (fully utilize high-configuration CPU performance); ⑤ Task execution mode: enable parallel rendering + asynchronous IO, support parallel processing of multiple batches of tasks, fully utilize CPU and SSD disk performance, adapt to large-scale content construction (tens of thousands of pages); ⑥ Dynamic mapping model optimization: increase CPU dimension weight ( =0.7), adding memory dimension features and weights ( =0.3), adapted to high-end memory and CPU performance; ⑦ Other adaptations: compression level=9, enable multi-threaded compression, and dynamically adjust resource upload speed based on network bandwidth.

[0133] 5. Strategy deployment and dynamic updates to continuously adapt to changes in server configuration.

[0134] The adapted scheduling strategy is applied to the SWG parallel build task, automatically deployed and activated; simultaneously, the scheduling strategy is continuously updated and the dynamic mapping model is optimized.

[0135] (1) Server configuration monitoring: The server core parameters (number of CPU cores, memory, disk IOPS) are detected every 30 seconds. If a configuration change is detected (such as expanding memory, replacing SSD disk, increasing the number of CPU cores), the server configuration template is automatically updated and a new adaptation strategy is generated. The update is seamless and requires no manual intervention.

[0136] (2) Build efficiency optimization: Continuously monitor build efficiency. If the build time for large-scale content is too long or the resource utilization is too low, automatically fine-tune the strategy parameters (such as increasing the cache size, adjusting the basic concurrency, and optimizing the dynamic mapping model weights).

[0137] (3) Model synchronization optimization: Synchronize and dynamically map the construction data and adaptation parameters under different server configurations to realize cross-configuration iteration of the model and improve the adaptation accuracy of the model on different servers.

[0138] 6. Adaptation verification and exception handling.

[0139] After the scheduling strategy is deployed, a set of test tasks (containing different types and numbers of build tasks) are automatically executed to verify the effectiveness of the adaptation strategy. If problems such as resource overload, build interruption, or frequent blocking occur, the strategy is automatically rolled back to the previous version. At the same time, the strategy parameters are fine-tuned based on the template deviation value, and the strategy is redeployed for verification to ensure that the adaptation strategy is stable and reliable. If abnormal server configuration is detected (such as hardware failure causing a sharp drop in CPU utilization), the strategy is automatically switched to the emergency adaptation strategy (reducing the number of concurrent connections and disabling unnecessary caches) to ensure that the build tasks proceed normally.

[0140] In the above process, by acquiring the hardware configuration parameters of the target server, identifying its key hardware capabilities such as the number of processor cores, memory capacity, storage type, and I / O bandwidth, and matching the corresponding server configuration template according to preset configuration rules, a target scheduling strategy adapted to the hardware environment can be effectively generated. This allows CPU-intensive subtasks to be preferentially allocated to high-core, high-frequency processor resources, while I / O-intensive subtasks are preferentially scheduled to devices with high-speed storage or high-throughput network interfaces. At the same time, by combining the server's current load status and task type analysis results, the scheduling strategy can be refined and adaptively adjusted, avoiding resource idleness or overload bottlenecks caused by hardware capability mismatch. This solves the technical problem of rigid scheduling strategies that cannot adapt to heterogeneous server environments.

[0141] Overall, this application constructs an intelligent scheduling system that integrates dynamic load perception, task feature-driven approach, and server self-adaptation, overcoming the triple limitations of fixed concurrency or equal allocation strategies in terms of efficiency, stability, and versatility. By determining the server load status of the SWG parallel build task in real time and accurately identifying the task type (including CPU-intensive or I / O-intensive) of its subtasks, the system then performs collaborative analysis of load status and task type based on a task scheduling model to dynamically generate an appropriate scheduling strategy. This strategy allocates different types of subtasks to processors that match their resource requirements, effectively avoiding the technical problems of processor resource mismatch and load imbalance caused by fixed or equal allocation strategies.

[0142] In this embodiment, the three modules of real-time CPU load acquisition, intelligent task type classification, and automatic server hardware detection are deeply coupled with the Go language GMP scheduling model. This innovatively realizes full-link dynamic closed-loop control of concurrency, task allocation, resource isolation, and scheduling strategy. This enables the SWG system to automatically adjust its execution strategy based on real-time load, task characteristics, and underlying hardware. This not only significantly improves build efficiency and system stability but also achieves a qualitative leap from adapting a single strategy to all servers to automatically generating the optimal scheduling scheme for a single server. This fills the technical gap in the industry for adaptive scheduling in large-scale, multi-configuration, and high-concurrency static build scenarios.

[0143] According to embodiments of this application, a scheduling device for parallel building tasks of a static website generator is provided. It should be noted that the scheduling device for parallel building tasks of a static website generator in this application can be used to execute the scheduling method for parallel building tasks of a static website generator provided in this application. The scheduling device for parallel building tasks of a static website generator provided in this application is described below.

[0144] Figure 6 This is a structural diagram of a scheduling device for parallel construction tasks of a static website generator, provided according to an embodiment of this application. Figure 6 As shown, the device includes:

[0145] The first determining module 60 is used to determine the load status of the server executing the static website generator SWG parallel build task;

[0146] The second determining module 62 is used to determine the task type of the subtask in the SWG parallel construction task. The task type includes computationally intensive tasks and input / output intensive tasks. Computationally intensive tasks refer to subtasks that consume central processing unit computing resources, and input / output intensive tasks refer to subtasks that wait for input and output devices to complete read and write operations.

[0147] Analysis module 64 is used to analyze the load status, task type and initial concurrency of the server based on the task scheduling model, and generate a scheduling strategy corresponding to the SWG parallel construction task. The task scheduling model is used to allocate subtasks of different task types in the SWG parallel construction task to the corresponding processors for processing. The initial concurrency is calculated based on the multidimensional mapping model and is used to represent the maximum number of initial subtasks that the server is allowed to execute simultaneously.

[0148] By using the first determining module, the second determining module, and the analysis module in the scheduling device of the parallel construction task of the static website generator, the goal of generating the optimal scheduling strategy based on the real-time load status of the server and the sub-task type of the SWG parallel construction task is achieved. This results in improving SWG construction efficiency, enhancing system stability, and seamlessly adapting to heterogeneous server environments. Furthermore, it solves the technical problem of low efficiency and instability in SWG parallel construction, which is caused by the fact that related technologies generally adopt fixed concurrency or simple equal distribution strategies, making it difficult to dynamically perceive system load and task characteristics.

[0149] In the scheduling device for parallel construction tasks of static website generator provided in this application embodiment, the first determining module is further used to obtain the server's indicator data according to a preset sampling period, wherein the indicator data includes at least the server's CPU utilization, context switching frequency and process occupancy rate; preprocess the indicator data; and divide the preprocessed indicator data according to a preset load threshold to obtain the server's load level, wherein the load level is used to reflect the server's load status.

[0150] In the scheduling device for parallel construction tasks of a static website generator provided in this application embodiment, the first determining module is further configured to determine the initial concurrency of the server; when the server is at a first load level, the initial concurrency is expanded according to a first preset ratio; when the server is at a second load level, the initial concurrency is maintained, wherein the load level corresponding to the second load level is higher than the load level corresponding to the first load level; when the server is at a third load level, the initial concurrency is reduced according to a second preset ratio, wherein the second preset ratio is less than the first preset ratio, and the load level corresponding to the third load level is higher than the load level corresponding to the second load level.

[0151] In the scheduling device for parallel construction tasks of a static website generator provided in this application embodiment, the first determining module is further configured to acquire task data corresponding to the SWG parallel construction task and determine a first feature vector corresponding to the task data; acquire server resource data and determine a second feature vector corresponding to the server resource data; determine a third feature vector corresponding to the server's indicator data; process the first feature vector, the second feature vector, and the third feature vector through a multi-dimensional mapping model, and output the initial concurrency of the server, wherein the multi-dimensional mapping model includes a mapping function for establishing the mapping relationship between the server's resource features, indicator features, and the task features of the SWG parallel construction task.

[0152] In the scheduling device for parallel construction tasks of static website generator provided in this application embodiment, the second determining module is further configured to classify and parse the SWG parallel construction task according to a preset parsing method to obtain the task type corresponding to each subtask in the SWG parallel construction task; obtain auxiliary features of each subtask, wherein the auxiliary features include at least the execution time of each subtask and the dependency relationship between each subtask; and generate a task classification list corresponding to the SWG parallel construction task according to the task type and auxiliary features.

[0153] In the scheduling device for the parallel construction task of the static website generator provided in this application embodiment, the analysis module is further used to determine the task priority of each subtask in the SWG parallel construction task based on the server load status, the initial concurrency of the server, the task type in the task classification list and the task dependency relationship. Among them, the task priority of computationally intensive tasks is negatively correlated with the server load level, and the task priority of input-output intensive tasks is positively correlated with the server load level.

[0154] In the scheduling device for the parallel construction task of the static website generator provided in this application embodiment, the analysis module is further used to analyze the server load status, task type, initial concurrency of the server and task priority based on the task scheduling model, and generate a scheduling strategy corresponding to the SWG parallel construction task. The task scheduling model includes a task execution unit pool, a priority-aware channel and a processor isolation strategy. The task execution unit pool includes a first unit pool for executing computationally intensive tasks and a second unit pool for executing input / output intensive tasks. The priority-aware channel includes a first communication channel for sensing high-priority tasks whose task priority is greater than or equal to a preset threshold and a second communication channel for sensing low-priority tasks whose task priority is less than a preset threshold. The processor isolation strategy includes a first processor corresponding to computationally intensive tasks and a second processor corresponding to input / output intensive tasks.

[0155] In the scheduling device for parallel building tasks of static website generator provided in this application embodiment, the analysis module is further used to obtain the hardware configuration parameters of the target server, wherein the target server is any server other than the server; determine the server configuration template corresponding to the hardware configuration parameters according to the preset configuration rules; and determine the target scheduling strategy corresponding to the target server when executing the SWG parallel building task according to the server configuration template.

[0156] This application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and used to execute the scheduling method for implementing the above-described parallel construction task of the static website generator.

[0157] It should be noted that the aforementioned electronic equipment is used to perform Figure 2 The scheduling method for the parallel construction task of the static website generator shown above also applies to this electronic device, and will not be repeated here.

[0158] This application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device containing the non-volatile storage medium executes the scheduling method for the parallel construction task of the static website generator by running the computer program.

[0159] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The scheduling method for the parallel construction task of the static website generator shown above also applies to this non-volatile storage medium, and will not be repeated here.

[0160] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the scheduling method for the parallel construction tasks of the static website generator described above.

[0161] It should be noted that the above-mentioned computer program product is used to execute... Figure 2 The scheduling method for the parallel construction tasks of the static website generator shown above also applies to this computer program product, and will not be repeated here.

[0162] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0163] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0165] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0167] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0168] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A scheduling method for parallel construction tasks of a static website generator, characterized in that, include: Determine the load status of the server executing the static website generator SWG parallel build task; The task type of the subtask in the SWG parallel construction task is determined, wherein the task type includes computationally intensive tasks and input / output intensive tasks. The computationally intensive task is a subtask that consumes central processing unit computing resources, and the input / output intensive task is a subtask that waits for input and output devices to complete read and write operations. Based on the task scheduling model, the load status, the task type, and the initial concurrency of the server are analyzed to generate a scheduling strategy corresponding to the SWG parallel construction task. The task scheduling model is used to allocate subtasks of different task types in the SWG parallel construction task to the corresponding processors for processing. The initial concurrency is calculated based on the multidimensional mapping model and is used to represent the maximum number of initial subtasks that the server is allowed to execute simultaneously. The initial concurrency count is calculated based on a multidimensional mapping model, including: acquiring task data corresponding to the SWG parallel construction task and determining a first feature vector corresponding to the task data; acquiring server resource data and determining a second feature vector corresponding to the server resource data; determining a third feature vector corresponding to the server's indicator data; processing the first feature vector, the second feature vector, and the third feature vector through the multidimensional mapping model to output the initial concurrency count of the server, wherein the multidimensional mapping model includes a mapping function for establishing the mapping relationship between the server's resource features, indicator features, and the task features of the SWG parallel construction task, the specific expression of which is as follows: ； In the formula, This represents the initial concurrency level. This represents the first feature vector. This represents the second feature vector. This represents the third feature vector; This represents a one-dimensional mapping function. The upper bound of the summation sign, 4, indicates the number of such one-dimensional mapping functions, including the first mapping function corresponding to the computationally intensive task. The second mapping function corresponding to the input-output intensive task. The third mapping function corresponding to the task duration And the fourth mapping function corresponding to task dependencies. ; This represents the weights corresponding to each single-dimensional mapping function, including the contribution weights of computationally intensive tasks. Contribution weight of input-output intensive tasks Task duration affects weight. And the weight of task dependency ; This indicates the bias term.

2. The method according to claim 1, characterized in that, Determine the load status of the server executing the static website generator SWG parallel build task, including: The server's metrics data are obtained according to a preset sampling period, wherein the metrics data include at least the server's CPU utilization, context switching frequency, and process occupancy rate. The indicator data is preprocessed; Based on a preset load threshold, the preprocessed indicator data is divided to obtain the load level of the server, wherein the load level is used to reflect the load status of the server.

3. The method according to claim 2, characterized in that, The method further includes: Determine the initial concurrency of the server; When the server is at the first load level, the initial concurrency is increased according to a first preset ratio. When the server is at the second load level, the initial concurrency is maintained, wherein the load level corresponding to the second load level is higher than the load level corresponding to the first load level; When the server is at the third load level, the initial concurrency is reduced according to a second preset ratio, wherein the second preset ratio is less than the first preset ratio, and the load level corresponding to the third load level is higher than the load level corresponding to the second load level.

4. The method according to claim 1, characterized in that, After determining the task type of the subtask in the SWG parallel construction task, the method further includes: The SWG parallel construction task is classified and parsed according to a preset parsing method to obtain the task type corresponding to each subtask in the SWG parallel construction task. Obtain auxiliary features for each subtask, wherein the auxiliary features include at least the execution time of each subtask and the dependencies between the subtasks; Based on the task type and the auxiliary features, a task classification list corresponding to the SWG parallel construction task is generated.

5. The method according to claim 4, characterized in that, The method further includes: Based on the server's load status, the server's initial concurrency, the task types and task dependencies in the task classification list, the task priorities of each subtask in the SWG parallel construction task are determined. The task priorities of computationally intensive tasks are negatively correlated with the server's load level, while the task priorities of input / output intensive tasks are positively correlated with the server's load level.

6. The method according to claim 5, characterized in that, Based on the task scheduling model, the load status, task type, and initial concurrency of the server are analyzed to generate a scheduling strategy corresponding to the SWG parallel construction task, including: Based on the task scheduling model, the server load status, task type, initial concurrency of the server, and task priority are analyzed to generate a scheduling strategy corresponding to the parallel construction task of the SWG. The task scheduling model includes a task execution unit pool, a priority-aware channel, and a processor isolation strategy. The task execution unit pool includes a first unit pool for executing computationally intensive tasks and a second unit pool for executing input / output intensive tasks. The priority-aware channel includes a first communication channel for sensing high-priority tasks with priorities greater than or equal to a preset threshold and a second communication channel for sensing low-priority tasks with priorities less than the preset threshold. The processor isolation strategy includes a first processor corresponding to the computationally intensive task and a second processor corresponding to the input / output intensive task.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the hardware configuration parameters of the target server, wherein the target server is any server other than the server mentioned above; Based on preset configuration rules, determine the server configuration template corresponding to the hardware configuration parameters; The target scheduling strategy corresponding to the target server when executing the SWG parallel construction task is determined based on the server configuration template.

8. A scheduling device for parallel construction tasks of a static website generator, characterized in that, include: The first determination module is used to determine the load status of the server executing the parallel build task of the static website generator SWG; The second determining module is used to determine the task type of the subtask in the SWG parallel construction task, wherein the task type includes computationally intensive tasks and input / output intensive tasks, the computationally intensive tasks represent subtasks that consume central processing unit computing resources, and the input / output intensive tasks represent subtasks that wait for input and output devices to complete read and write operations; The analysis module is used to analyze the load status, task type, and initial concurrency of the server based on a task scheduling model, and generate a scheduling strategy corresponding to the SWG parallel construction task. The task scheduling model allocates subtasks of different task types within the SWG parallel construction task to corresponding processors for processing. The initial concurrency is calculated based on a multidimensional mapping model and represents the maximum number of initial subtasks that the server is allowed to execute simultaneously. The calculation of the initial concurrency based on the multidimensional mapping model includes: acquiring task data corresponding to the SWG parallel construction task and determining a first feature vector corresponding to the task data; acquiring server resource data and determining a second feature vector corresponding to the server resource data; determining a third feature vector corresponding to the server's indicator data; and processing the first, second, and third feature vectors through the multidimensional mapping model to output the initial concurrency of the server. The multidimensional mapping model includes a mapping function for establishing the mapping relationship between the server's resource features, indicator features, and task features of the SWG parallel construction task, with the specific expression as follows: ； In the formula, This represents the initial concurrency level. This represents the first feature vector. This represents the second feature vector. This represents the third feature vector; This represents a one-dimensional mapping function. The upper bound of the summation sign, 4, indicates the number of such one-dimensional mapping functions, including the first mapping function corresponding to the computationally intensive task. The second mapping function corresponding to the input-output intensive task. The third mapping function corresponding to the task duration And the fourth mapping function corresponding to task dependencies. ; This represents the weights corresponding to each single-dimensional mapping function, including the contribution weights of computationally intensive tasks. Contribution weight of input-output intensive tasks Task duration affects weight. And the weight of task dependency ; This indicates the bias term.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store program instructions; and the processor, connected to the memory, is used to execute a scheduling method for implementing the parallel construction task of a static website generator according to any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the scheduling method for parallel construction tasks of the static website generator according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the scheduling method for parallel construction tasks of the static website generator as described in any one of claims 1 to 7.

Images