Heterogeneous computing resource pooling and scheduling method and device of intelligent computing cloud platform

Abstract

The application provides a heterogeneous computing power resource pooling scheduling method and device of an intelligent computing cloud platform, and relates to the technical fields of intelligent computing centers, intelligent computing centers, computing power infrastructure and intelligent computing cloud, which comprises the following steps: S1, generating, in a reinforcement learning training task, sample generation and gradient update subtasks as heterogeneous subtasks and respectively deploying them to first and second computing power resource pools; S2, generating training samples by the sample generation subtask and caching them to a sample buffer area, and dynamically adjusting the number of computing power resources in the first computing power resource pool according to the water level of the sample buffer area; S3, calculating the version difference between a new weight version generated by the gradient update subtask and an old weight version currently used by the sample generation subtask; and S4, when the version difference reaches a version lag tolerance, updating the weight version used by the sample generation subtask to the new weight version based on a target weight fragment in which parameter changes occur between the new and old weight versions, so as to realize on-demand scheduling of computing power resources.

Description

Technical Field

[0001] This invention relates to the fields of intelligent computing centers, smart computing centers, computing infrastructure, and smart cloud technologies, specifically to a method and apparatus for pooling and scheduling heterogeneous computing resources on an intelligent computing cloud platform. Background Technology

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "smart computing centers" have emerged.

[0003] An "intelligent computing center" refers to a facility that provides the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models) by utilizing large-scale heterogeneous computing resources, including general-purpose and intelligent computing power. Intelligent computing centers encompass facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0004] "Intelligent computing center" includes, but is not limited to, "intelligent computing center".

[0005] "Intelligent computing center" or artificial intelligence computing center is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting artificial intelligence computing architecture.

[0006] "Computing power" is the core of "intelligent computing center" and "smart computing center". It is the ability of computer equipment or computing / data center to process parameters. It is the ability of computer hardware and software to work together to execute a certain computing requirement. It is the computing power to achieve the target result output by processing parameter data. It is a new type of productivity that integrates parameter computing power, network carrying capacity and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] In intelligent computing cloud platforms, reinforcement learning algorithms are typically used for model training. Reinforcement learning training mainly consists of two core iterative stages: sample generation and gradient update. In a conventional training process, multiple inference operations are first performed based on the current model weights to generate a batch of training samples. Then, gradient calculations and model parameter updates are performed using these batch training samples. Subsequently, a new round of sample generation is performed based on the updated model weights. Through this two-stage iterative cycle, model training eventually converges. Related technologies typically deploy these two heterogeneous training stages—sample generation and gradient update—within a homogeneous hardware computing resource pool. However, sample generation focuses on model inference operations, requiring high hardware memory and operator adaptability; gradient update focuses on ensemble communication and batch iterative calculations, requiring high hardware card interconnect bandwidth and parallel computing capabilities. The computing power requirements for these two stages differ significantly. The traditional method of uniformly deploying homogeneous computing resources cannot adapt to the differentiated computing power requirements of the two training stages, easily leading to periodic idleness and waste of computing resources.

[0008] It is evident that since the emergence of intelligent computing centers, the two core tasks of sample generation and gradient update in reinforcement learning training have adopted a unified deployment of homogeneous computing resources, which easily leads to the problem of periodic idleness and low utilization of computing resources. This problem has always been an urgent issue to be solved in this field. Summary of the Invention

[0009] This invention provides a heterogeneous computing resource pooling scheduling method and device for an intelligent computing cloud platform, in order to solve the problem that in the existing technology, the two core tasks of sample generation and gradient update are deployed in a unified manner using homogeneous computing resources during reinforcement learning training, which easily leads to the phased idleness and low utilization of computing resources.

[0010] To solve the above problems, the present invention is implemented as follows: In a first aspect, the present invention provides a method for pooling and scheduling heterogeneous computing resources in an intelligent computing cloud platform, comprising: Step S1: The sample generation subtask and gradient update subtask in the reinforcement learning training task are treated as heterogeneous subtasks and deployed to the first computing resource pool and the second computing resource pool, which are divided from the computing resource pool of the intelligent computing cloud platform, respectively. The computing resources configured in the first computing resource pool and the second computing resource pool are heterogeneous. Step S2: Training samples are continuously generated and cached in the sample buffer through the sample generation subtask, and the amount of computing resources in the first computing resource pool is dynamically adjusted according to the water level of the sample buffer. The sample buffer is used to continuously supply training samples for the gradient update subtask. Step S3: After the gradient update subtask iteratively generates a new weight version, calculate the version difference between the new weight version and the old weight version currently used by the sample generation subtask. Step S4: When the version difference reaches the preset version lag tolerance, based on the target weight slice where the parameters have changed between the old and new weight versions, update the weight version of the model used in the sample generation subtask from the old weight version to the new weight version.

[0011] In one embodiment, step S2 includes: Step S2.1: When the water level in the sample buffer is detected to be lower than the preset lower limit, the first computing power resource pool is expanded; and when the water level in the sample buffer is detected to be higher than the preset upper limit, the first computing power resource pool is reduced. During the expansion and contraction process, the number of computing resources in the first computing resource pool always remains within the preset range.

[0012] In one embodiment, after step S2.1, the method further includes: Step S2.2: Mark the computing resources released after the reduction as recyclable and return them to the computing resource pool for use by other computing power task scheduling.

[0013] In one embodiment, step S2.1 includes: Step S2.1.1: Configure the water level range, proportional gain coefficient, control period, and upper and lower limits of the number of computing resources in the first computing resource pool for the sample buffer. Step S2.1.2: Determine the target center value of the water level based on the water level range, and obtain the water level deviation by combining it with the current number of samples in the sample buffer. Step S2.1.3: Combine the current number of computing resources in the first computing resource pool, the proportional gain coefficient, and the water level deviation to calculate the target number of computing resources for the next control cycle; Step S2.1.4: Limit the target computing power resource quantity within the upper and lower limits to obtain the actual effective computing power resource quantity; Step S2.1.5: Expand or shrink the first computing power resource pool according to the actual number of effective computing power resources.

[0014] In one embodiment, step S4 includes: Step S4.1: Divide the weight parameters corresponding to the new weight version and the old weight version into multiple weight fragments according to the same partitioning rule, and calculate the hash fingerprint of each weight fragment. Step S4.2: Compare the hash fingerprints of the weight slices corresponding to the old and new weight versions, and identify the weight slices with inconsistent hash fingerprints as the target weight slices that have undergone parameter changes. Step S4.3: Push the target weight shard to the first computing power resource pool to perform a hot weight update on the model used by the sample generation subtask. Weight shards with consistent hash fingerprints are not pushed or transmitted.

[0015] In one embodiment, step S4.3 includes: During the weight hot update process, the sample generation subtask continuously generates training samples based on the old weight version, and switches to the new weight version after the target weight slice is loaded.

[0016] In one embodiment, the heterogeneous computing resource pooling scheduling method of the intelligent computing cloud platform further includes: Step S5: Obtain the task stage of the reinforcement learning training task; Step S6: Dynamically adjust the version lag tolerance according to the task stage, wherein the version lag tolerance corresponding to the later stage of the task is greater than the version lag tolerance corresponding to the earlier stage of the task.

[0017] In one embodiment, step S1 includes: Step S1.1: Obtain hardware capability profile information of each candidate computing resource in the intelligent computing cloud platform; Step S1.2: Based on the hardware capability profile information, determine a preset number of computing resources from the candidate computing resources whose inter-card interconnection bandwidth meets the requirements of aggregate communication, and construct the second computing resource pool. Step S1.3: Based on the hardware capability profile information, select computing resources from the candidate computing resources that meet the inference execution requirements in terms of video memory, computing precision, and operator capabilities, and whose inter-card interconnect bandwidth is lower than that of the computing resources in the second computing resource pool, and define them as the available resource range of the first computing resource pool.

[0018] Secondly, the present invention also provides a heterogeneous computing power resource pooling scheduling device for an intelligent computing cloud platform, comprising: The deployment module is used to treat the sample generation subtask and gradient update subtask in the reinforcement learning training task as heterogeneous subtasks and deploy them to the first computing resource pool and the second computing resource pool, which are divided from the computing resource pool of the intelligent computing cloud platform, respectively. The computing resources configured in the first computing resource pool and the second computing resource pool are heterogeneous. The adjustment module is used to continuously generate training samples through the sample generation subtask and cache them in the sample buffer, and dynamically adjust the amount of computing resources in the first computing resource pool according to the water level of the sample buffer. The sample buffer continuously supplies training samples to the gradient update subtask. The calculation module is used to calculate the version difference between the new weight version and the old weight version currently used by the sample generation subtask after the gradient update subtask iteratively generates a new weight version. The update module is used to update the weight version of the model used in the sample generation subtask from the old weight version to the new weight version when the version difference reaches the preset version lag tolerance, based on the target weight slice where the parameters have changed between the old and new weight versions.

[0019] In one embodiment, the adjustment module is further configured to: expand the capacity of the first computing power resource pool when the water level of the sample buffer is detected to be lower than the preset lower limit, and shrink the capacity of the first computing power resource pool when the water level of the sample buffer is detected to be higher than the preset upper limit; wherein, during the expansion and shrinkage process, the number of computing power resources in the first computing power resource pool is always within a preset number range.

[0020] In one embodiment, the adjustment module is further configured to: mark the computing resources released after scaling down as recyclable and return them to the computing resource pool for use by other computing power task scheduling.

[0021] In one embodiment, the adjustment module is further configured to: configure the water level range, proportional gain coefficient, control period, and upper and lower limits of the number of computing resources in the first computing resource pool; determine the target center value of the water level based on the water level range, and obtain the water level deviation by combining it with the current number of samples in the sample buffer; calculate the target number of computing resources for the next control period by combining the current number of computing resources in the first computing resource pool, the proportional gain coefficient, and the water level deviation; limit the target number of computing resources within the upper and lower limits to obtain the actual effective number of computing resources; and expand or shrink the first computing resource pool based on the actual effective number of computing resources.

[0022] In one embodiment, the update module is further configured to: divide the weight parameters corresponding to the new weight version and the old weight version into multiple weight fragments according to the same segmentation rule, and calculate the hash fingerprint of each weight fragment; compare the hash fingerprints of the weight fragments corresponding to the new and old weight versions, and determine the weight fragments with inconsistent hash fingerprints as the target weight fragments where parameter changes have occurred; push the target weight fragments to the first computing power resource pool to perform hot weight updates on the model used by the sample generation subtask, wherein weight fragments with consistent hash fingerprints are not pushed and transmitted.

[0023] In one embodiment, the update module is further configured to: during the weight hot update process, the sample generation subtask continuously generates training samples based on the old weight version, and switches to the new weight version after the target weight slice is loaded.

[0024] In one embodiment, the heterogeneous computing power resource pooling scheduling device of the intelligent computing cloud platform further includes a tolerance adjustment module, used to: obtain the task stage of the reinforcement learning training task; and dynamically adjust the version lag tolerance according to the task stage, wherein the version lag tolerance corresponding to the later stage of the task is greater than the version lag tolerance corresponding to the earlier stage of the task.

[0025] In one embodiment, the deployment module is further configured to: acquire hardware capability profile information of each candidate computing power resource in the intelligent computing cloud platform; determine a preset number of computing power resources whose inter-card interconnect bandwidth meets the requirements of aggregate communication from the candidate computing power resources based on the hardware capability profile information, and construct a second computing power resource pool; and select computing power resources from the candidate computing power resources whose video memory, computing precision, and operator capabilities meet the requirements of inference execution and whose inter-card interconnect bandwidth is lower than that of the computing power resources in the second computing power resource pool based on the hardware capability profile information, and define them as the available resource range of the first computing power resource pool.

[0026] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the heterogeneous computing power resource pooling scheduling method of the intelligent computing cloud platform described in the first aspect above.

[0027] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the heterogeneous computing power resource pooling scheduling method of the intelligent computing cloud platform described in the first aspect above.

[0028] Fifthly, the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the heterogeneous computing power resource pooling scheduling method of the intelligent computing cloud platform as described in the first aspect above.

[0029] In this invention, step S1 involves treating the sample generation subtask and gradient update subtask in the reinforcement learning training task as heterogeneous subtasks, and deploying them to a first computing resource pool and a second computing resource pool, which are partitioned from the computing resource pool of the intelligent computing cloud platform, respectively. The computing resources configured in the first computing resource pool and the second computing resource pool are heterogeneous. Step S2 involves continuously generating training samples through the sample generation subtask and caching them in the sample buffer. The number of computing resources in the first computing resource pool is dynamically adjusted according to the level of the sample buffer. The sample buffer continuously supplies training samples to the gradient update subtask. Step S3 involves calculating the version difference between the new weight version and the old weight version currently used by the sample generation subtask after the gradient update subtask iteratively generates a new weight version. Step S4 involves updating the weight version of the model used by the sample generation subtask from the old weight version to the new weight version based on the target weight slice where parameter changes have occurred between the old and new weight versions. Therefore, the sample generation and gradient update subtasks of the reinforcement learning training task are treated as heterogeneous subtasks and deployed to the first and second heterogeneous computing resource pools, respectively, to achieve decoupling and adaptation of hardware resources for the two types of subtasks. For the sample generation side, the computing resource scale of the first computing resource pool is dynamically and elastically adjusted based on the sample buffer level to match the actual needs of sample production and rationally allocate computing resource supply for the sample generation stage. The weight update stage does not employ a real-time full synchronization mechanism; instead, when the version difference between the old and new weight versions reaches the version lag tolerance, only the changed target weight fragment is pushed to complete the hot update, reducing a large amount of invalid network transmission. This solution improves the overall utilization efficiency of heterogeneous computing resources while ensuring the normal progress of training. Attached Figure Description

[0030] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart of a heterogeneous computing power resource pooling scheduling method for an intelligent computing cloud platform provided by the present invention; Figure 2 This is a schematic diagram of a curve illustrating the elastic scaling of the first computing power resource pool driven by the water level of a sample buffer, as provided by the present invention. Figure 3 This is a diagram of the heterogeneous computing power resource pooling scheduling architecture of the intelligent computing cloud platform provided by the present invention; Figure 4 This is a data interaction diagram of the heterogeneous computing power resource pooling scheduling provided by the present invention; Figure 5 This is an exemplary schematic diagram of a heterogeneous computing power resource pooling scheduling device for an intelligent computing cloud platform provided by the present invention.

[0032] Figure 6 This is a structural diagram of an electronic device provided by the present invention. Detailed Implementation

[0033] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0034] The “computing power” mentioned in this invention refers to: the ability of computer equipment or computing / data center to process information; the ability of computer hardware and software to work together to perform a certain computing requirement; the computing power to achieve the target result output by processing information data; and a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity, mainly providing services to society through computing power infrastructure.

[0035] The "computational power" (CP) described in this invention refers to the ability of a data center server to process data and output results. It is a comprehensive indicator of a data center's computing power, encompassing general computing power, supercomputing power, and intelligent computing power. The commonly used unit of measurement is floating-point operations per second (FLOPS, 1 EFLOPS = 10^18 FLOPS), with higher values ​​indicating stronger overall computing power. It is estimated that 1 EFLOPS is approximately the computing power output of 5 Tianhe-2A supercomputers, 500,000 mainstream server CPUs, or 2 million mainstream laptops. The calculation formula is: CP = CP 通用 +CP 智能 +CP 超级 .

[0036] The "Network Power" (NP) mentioned in this invention refers to the performance of data transmission capability of computing facilities, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, and involves network transmission within and between data centers. It is a comprehensive indicator for measuring network transmission scheduling capability.

[0037] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in four aspects: data storage capacity, performance, security and reliability, and green and low-carbon operation. It is a comprehensive indicator for measuring the data storage capacity of a data center, including external storage devices such as storage arrays and internal storage devices within servers. The commonly used unit of measurement for storage capacity is exabytes (EB, 1EB = 2^60 bytes), while the commonly used unit of measurement for performance is the number of read / write operations per second (IOPS / TB). Disaster recovery ratio is an important indicator of security and reliability.

[0038] The "computing infrastructure" mentioned in this invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, enabling centralized computing, storage, transmission, and application of information.

[0039] The "new information infrastructure" mentioned in this invention refers to network infrastructure such as 5G networks, fiber optic broadband networks, backbone networks, international communication networks, and satellite internet; computing infrastructure such as data centers, general computing centers, intelligent computing centers, and supercomputing centers; and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0040] The “computing power” mentioned in this invention includes: general computing power, intelligent computing power, and supercomputing power.

[0041] The "general computing power" mentioned in this invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0042] The "intelligent computing power" mentioned in this invention refers to: a computing platform deployed on a large scale based on dedicated chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various artificial intelligence innovative applications, such as natural language processing and machine vision.

[0043] The “supercomputing power” mentioned in this invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for computing in cutting-edge scientific fields, such as planetary simulation, drug molecule design, and gene analysis.

[0044] The "intelligent computing center" described in this invention refers to a facility that utilizes large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.), to primarily provide the necessary computing power, data, and algorithms for artificial intelligence applications (such as the development, training, and inference of deep learning models). The intelligent computing center encompasses facilities, hardware, and software, and can provide full-stack capabilities from underlying computing power to top-level application enablement.

[0045] The "intelligent computing cloud platform" mentioned in this invention, abbreviated as "intelligent computing cloud", refers to a cloud computing platform that integrates hardware and software resources based on an intelligent computing center.

[0046] The "intelligent computing center" mentioned in this invention includes, but is not limited to, "smart computing center".

[0047] The "intelligent computing center" mentioned in this invention, also known as an artificial intelligence computing center, is a type of computing infrastructure that provides computing power services, data services, and algorithm services required for artificial intelligence applications, based on artificial intelligence theory and adopting an artificial intelligence computing architecture.

[0048] The "computing center" mentioned in this invention refers to a facility that is mainly composed of infrastructure such as wind, thermal, hydro, and electricity, and IT hardware and software equipment, and has computing power, carrying capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0049] The "supercomputing center" mentioned in this invention refers to a supercomputing data center, which is a data center based on supercomputers or large-scale computing clusters. It can provide large-scale computing, storage and network services and is widely used in aerospace, defense, oil exploration, climate modeling and genome sequencing and other application scenarios.

[0050] The “computing resources” mentioned in this invention refer to the technologies and facilities required for the development of the digital society that have the ability to compute, transmit, store and apply information, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guaranteeing resources such as wind, fire, water and electricity.

[0051] The "model" mentioned in this invention includes, but is not limited to, "large language model" and "multimodal large model".

[0052] The "large language model" mentioned in this invention refers to a large-scale language model (LLM), which is a language model with a large number of parameters. It is designed to understand and generate human language, and is trained with a large amount of text data. It can perform a wide range of tasks, including text summarization, translation, and sentiment analysis.

[0053] The “Multimodal Large Models” mentioned in this invention refer to models that combine multimodal information such as text, images, videos, and audio for training, including but not limited to multimodal large language models.

[0054] The “computing power running task” mentioned in this invention refers to a specific workload or job executed on computing power resources that requires a certain amount of computing power support, usually involving complex data processing, numerical calculation, model training or simulation scenarios.

[0055] The "sample generation subtask" described in this invention refers to the inference execution subtask deployed in the first computing resource pool during the reinforcement learning training process, which performs inference operations based on model weights to generate training samples for model training.

[0056] The "gradient update subtask" described in this invention refers to a training execution subtask deployed in the second computing resource pool during the reinforcement learning training process. This subtask receives training samples supplied by the sample buffer, completes gradient calculation, and updates model parameters.

[0057] The "weight" mentioned in this invention refers to the learnable model parameters within the model trained by reinforcement learning, which is the set of parameters that the model relies on to perform inference operations and output training samples.

[0058] The "weight version" mentioned in this invention refers to the model weights in different states formed after the model has undergone one or more gradient updates. A new weight version is generated after each round of gradient update to characterize the version state after the model parameters have been iterated. The weight version generated in each round of gradient update can be marked by a version number, which increments sequentially with the gradient update round.

[0059] The “sample buffer” mentioned in this invention refers to a data buffer area used to cache the training samples output by the sample generation subtask; the sample buffer supplies training samples to the gradient update subtask, and the level of the sample buffer is represented by the number of training samples cached in the buffer.

[0060] It's important to note that conventional computing resource scheduling typically targets homogeneous computing clusters with similar hardware specifications. The hardware capabilities and interconnection conditions of each computing resource are largely similar, and the scheduling logic focuses on allocating and mapping tasks to computing resources. However, in heterogeneous computing resource scenarios, the hardware specifications of different computing resources vary significantly, including memory capacity, computational precision, operator support capabilities, and inter-card interconnection bandwidth. Different business sub-tasks have different hardware requirements; some tasks have stringent requirements for inter-card communication bandwidth, while others prioritize inference throughput performance. Conducting heterogeneous computing resource scheduling requires identifying and matching the multi-dimensional hardware capabilities of computing resources. The diverse hardware combinations and complex constraints significantly increase the overall scheduling complexity, making heterogeneous computing resource scheduling far more difficult to implement than conventional computing resource scheduling.

[0061] Please see Figure 1 , Figure 1 This is a flowchart of a heterogeneous computing power resource pooling scheduling method for an intelligent computing cloud platform provided by the present invention, such as... Figure 1 As shown, the heterogeneous computing power resource pooling scheduling method of the intelligent computing cloud platform includes: Step S1: The sample generation subtask and gradient update subtask in the reinforcement learning training task are treated as heterogeneous subtasks and deployed to the first computing resource pool and the second computing resource pool, which are divided from the computing resource pool of the intelligent computing cloud platform, respectively. The computing resources configured in the first computing resource pool and the second computing resource pool are heterogeneous.

[0062] Specifically, the intelligent computing cloud platform possesses a unified computing resource pool, which integrates various types of hardware computing resources to support the operation of various model training tasks within the platform. During reinforcement learning training, this invention, based on the difference in computing power requirements between the sample generation subtask and the gradient update subtask, divides the unified computing resource pool into two independent computing resource pools, each dedicated to supporting one of the two different subtasks.

[0063] Among them, the computing resources configured in the first computing resource pool and the second computing resource pool are heterogeneous, that is, the hardware computing power carried by the two computing resource pools are different, respectively corresponding to the actual operation requirements of the two types of sub-tasks, solving the problem that the traditional uniform deployment of homogeneous computing resources cannot take into account the differentiated needs of the two tasks.

[0064] The sample generation subtask is primarily responsible for model inference and batch generation of training samples. It leans towards inference-related tasks, with low requirements for interconnect bandwidth between graphics cards, placing greater emphasis on the memory and operator inference capabilities of a single card. Therefore, this invention deploys the sample generation subtask in the first computing resource pool, which mainly uses accelerator cards with higher performance-to-price ratios for inference, such as the L40S graphics card or domestically produced inference accelerator cards, capable of stably and efficiently completing the sample generation work.

[0065] The gradient update subtask is primarily responsible for calculating gradients and updating model parameters based on training samples, and belongs to the training iteration task. During training, frequent gradient synchronization and data communication between multiple GPUs are required, placing extremely high demands on inter-GPU interconnect bandwidth and communication latency. Therefore, this invention deploys the gradient update subtask in a second computing resource pool, which is equipped with high-performance training accelerator cards such as H100 and H800 with high-speed interconnect capabilities, effectively meeting the communication and computational needs of gradient iterative training.

[0066] It should be noted that the reinforcement learning training in this invention can be conventional reinforcement learning (RL) training, or reinforcement learning from human feedback (RLHF) training commonly used for large models.

[0067] It should be noted that the reinforcement learning training task of the model in this invention can be one of the computing power running tasks.

[0068] The beneficial technical effects achievable by step S1 are as follows: the two types of heterogeneous subtasks are deployed to the first and second computing resource pools configured with heterogeneous computing resources, respectively. The sample generation subtask uses inference computing resources with better cost performance, while the gradient update subtask uses training computing resources with high interconnect performance. This avoids the high cost of training computing resources being idle and wasted during the sample generation stage, thereby reducing the overall computing cost of reinforcement learning training. At the same time, the hardware of the two types of subtasks is isolated from each other. When the computing resources on the sample generation side are elastically scaled later, it will not interfere with the training communication process of gradient update, ensuring the overall stable operation of the training task.

[0069] Step S2 involves continuously generating training samples through the sample generation subtask and caching them in the sample buffer. The amount of computing resources in the first computing resource pool is dynamically adjusted according to the water level in the sample buffer. The sample buffer is used to continuously supply training samples for the gradient update subtask.

[0070] Specifically, the training samples generated by the sample generation subtask are stored in the sample buffer, and the sample buffer continuously outputs training samples to the gradient update subtask. This step monitors the water level of the sample buffer in real time and dynamically adjusts the amount of computing resources in the first computing resource pool according to the changes in the water level: when the number of samples in the sample buffer is insufficient, the first computing resource pool is expanded to improve the sample generation speed; when the sample buffer accumulates too many samples, the first computing resource pool is shrunk to reduce the throughput of sample generation.

[0071] The beneficial technical effects that can be achieved by step S2 are: through the above dynamic adjustment, the number of samples in the sample buffer is kept within a reasonable range, which can ensure that the gradient update subtask can obtain enough training samples for training, and prevent excessive backlog of training samples, thereby realizing on-demand scheduling of computing resources on the sample generation side.

[0072] Step S3: After the gradient update subtask iteratively generates a new weight version, calculate the version difference between the new weight version and the old weight version currently used by the sample generation subtask.

[0073] In this invention, the gradient update subtask completes one round of parameter iteration using training samples from the sample buffer, generating a new weight version. Meanwhile, the sample generation subtask continues to use the old weight version from the previous iteration. This step compares the new and old weight versions and calculates the version difference between them. In this embodiment, the version difference is the iteration interval between the new and old weight versions, representing the number of iterations the model's weights used in the sample generation subtask lag behind the gradient update subtask's.

[0074] For example, after the gradient update subtask completes multiple rounds of training iterations, it obtains a new weight version with the version number V8, while the sample generation subtask is still using the old weight version V4; comparing V8 and V4, the version difference is 8-4=4.

[0075] The beneficial technical effects that step S3 can achieve are: by calculating the version difference between the old and new weight versions, the number of iterations that the weights of the sample generation subtask side lag behind the gradient update subtask side can be quantified. This provides an objective quantitative basis for determining whether to trigger the weight update of the model on the sample generation subtask side, avoiding blind and frequent weight updates, and also preventing the weight version difference between the two sides from being too large, thus reducing the situation where the sample generation results deviate significantly from the training objective.

[0076] Step S4: When the version difference reaches the preset version lag tolerance, based on the target weight slice where the parameters have changed between the old and new weight versions, update the weight version of the model used in the sample generation subtask from the old weight version to the new weight version.

[0077] The version lag tolerance represents the maximum number of versions that the weight version of the sample generation subtask model is allowed to lag behind the weight version of the gradient update subtask model. This version lag tolerance is denoted as K. Only when the weight version of the sample generation subtask model lags behind the gradient update subtask model by K versions will a weight version update be triggered, completing the switch from the old weight version to the new weight version. The update process does not distribute the complete model weights; instead, it selects the parts of the model whose parameters have changed between the old and new weight versions as target weight slices. Only these slices are synchronized to complete the switch of the model used in the sample generation subtask from the old weight version to the new weight version.

[0078] For example, if K is 4, it means that the weight version of the sample generation sub-task model is allowed to lag the gradient update sub-task model's weight version by a maximum of 4 versions; once the difference between the two versions reaches 4, the weight update operation is triggered.

[0079] The value of K has two effects: the larger K is, the larger the allowed version lag span, the lower the frequency of weight updates triggered by the model on the sample generation subtask side, and the lower the transmission overhead, but the matching degree between the output samples of the sample generation subtask and the current policy on the training side will decrease; the smaller K is, the freshness of the generated samples is higher, but the weight updates of the model on the sample generation subtask side will be more frequent, and the corresponding transmission overhead will also increase.

[0080] It should be noted that the role of K in this invention is different from the staleness control mechanism in the existing open-source system AReaL. AReaL's mechanism is merely an algorithm-level condition used to select whether to accept or discard lagging samples, and it only applies to the sample selection stage. In contrast, the K in this invention controls the timing of weight updates, directly participating in scheduling-level decisions and enabling linkage between algorithm parameters and scheduling decisions.

[0081] Specifically, K mainly affects two aspects: first, the frequency of hot updates, where synchronization is only performed when the version difference reaches K; and second, network transmission overhead, determining the network resources used in the weight synchronization process. In other words, K not only controls how often weights are synchronized, but also affects the overall transmission bandwidth consumption. Depending on the value of K, a trade-off can be made between sample freshness and network transmission overhead.

[0082] The beneficial technical effects achievable by step S4 are as follows: when the weights of the model used in the sample generation subtask are updated, only the target weight slices whose parameters have changed are synchronized, without having to distribute the complete model weights, thus reducing the amount of data transmission during weight synchronization; at the same time, by controlling the timing of weight updates through the version lag tolerance K, a balance can be struck between sample freshness and network transmission overhead, avoiding network pressure caused by overly frequent updates and preventing excessive differences in weight versions on both sides, thus ensuring the quality of sample generation.

[0083] Furthermore, it should be noted that this invention deploys the sample generation subtask and the gradient update subtask in separate pools. Although this appears similar in form to the well-known Prefill-Decode (PD) separation deployment in the inference field (such as DistServe and vLLM's decoupled prefill services) in that it "splits the computation stages into different hardware pools," the application scenarios, the objects to be split, and the technical problems to be solved are actually quite different. PD separation deployment is mainly aimed at online inference service scenarios where model weights remain unchanged.

[0084] In one embodiment, step S2 includes: Step S2.1: When the water level in the sample buffer is detected to be lower than the preset lower limit, the first computing power resource pool is expanded; when the water level in the sample buffer is detected to be higher than the preset upper limit, the first computing power resource pool is reduced.

[0085] In some implementations, when the water level in the sample buffer is detected to be lower than the preset lower limit, the first computing power resource pool is expanded according to the preset proportional feedback control law to increase the amount of computing power resources, thereby accelerating the generation speed of training samples.

[0086] In some implementations, when the water level in the sample buffer is detected to be higher than the preset upper limit, the first computing power resource pool is reduced in size according to the preset proportional feedback control law, thereby reducing the amount of computing power resources and lowering the sample generation rate.

[0087] During the expansion and contraction process, the number of computing resources in the first computing resource pool is always within the preset range to avoid over-scheduling of computing resources.

[0088] The beneficial technical effects achievable by step S2.1 are as follows: This step monitors the sample buffer level in real time and dynamically expands or shrinks the first computing power resource pool. When the buffer level is low, it expands in time to accelerate sample production, ensure that the gradient update subtask can obtain enough training samples, and prevent training lag. When the level is high, it shrinks to avoid a large accumulation of samples. It can adapt to the changes in sample consumption at different training stages. The computing power resources in the first computing power resource pool are adjusted on demand, no longer maintaining a fixed number of computing power resources, reducing the waste of computing power resources, improving computing power utilization, and reducing the computing power cost of reinforcement learning training.

[0089] In one embodiment, after step S2.1, the method further includes: Step S2.2: Mark the computing resources released after the reduction as recyclable and return them to the computing resource pool for use by other computing power task scheduling.

[0090] For example, if a computing resource in the first computing resource pool is released, it can be marked as recyclable and returned to the computing resource pool. In this way, other computing tasks on the intelligent computing cloud platform can call upon the computing resource to execute the corresponding task when needed.

[0091] The beneficial technical effects that can be achieved by step S2.2 are: This step can release computing resources that are no longer used on the sample generation side in a timely manner, avoid the long-term monopoly of computing resources, and make them available for reuse by other tasks on the platform, thereby further improving the overall utilization efficiency of computing resources.

[0092] In one embodiment, during step S2.1, when the water level in the sample buffer is detected to be lower than the preset lower limit, the process of expanding the first computing power resource pool, and when the water level in the sample buffer is detected to be higher than the preset upper limit, the process of shrinking the first computing power resource pool, both involve calculating the amount of computing power resources required for the first computing power resource pool after expansion or shrinkage. The following will describe the calculation method for the amount of computing power resources required for the first computing power resource pool after expansion or shrinkage, based on steps S2.1.1 to S2.1.5. That is, step S2.1 can be understood as including: Step S2.1.1: Configure the water level range, proportional gain coefficient, control period, and upper and lower limits of the number of computing resources in the first computing resource pool for the sample buffer.

[0093] In this invention, the water level range of the sample buffer is denoted as [Q_low, Q_high]. Here, Q_low is the preset lower limit of the water level, and Q_high is the preset upper limit of the water level.

[0094] In this invention, the preset proportional gain coefficient is denoted as k_p.

[0095] In this invention, the preset control period is denoted as Δt.

[0096] In this invention, the lower limit of the number of computing resources in the first computing resource pool is denoted as N_min, and the upper limit of the number of computing resources in the first computing resource pool is denoted as N_max.

[0097] Step S2.1.2: Determine the target center value of the water level based on the water level range, and obtain the water level deviation by combining it with the current number of samples in the sample buffer.

[0098] In this invention, the target center value of the water level is the average of the preset lower limit and the preset upper limit of the water level. If the target center value of the water level is denoted as Q_target, then Q_target = (Q_low + Q_high) / 2.

[0099] In this invention, the current number of samples in the sample buffer is denoted as Q(t).

[0100] In this invention, the water level deviation refers to the difference between the target center value of the water level and the current number of samples in the sample buffer, which can be expressed as Q_target-Q(t).

[0101] Step S2.1.3: Combine the current number of computing resources in the first computing resource pool, the proportional gain coefficient, and the water level deviation to calculate the target number of computing resources for the next control cycle.

[0102] Step S2.1.4: Limit the target computing power resources to the upper and lower limits to obtain the actual effective computing power resources.

[0103] In this invention, steps S2.1.3 to S2.1.4 can be represented based on the following formula: N_rollout(t+Δt) = clip( N_rollout(t) + k_p·(Q_target - Q(t)), N_min, N_max ) In the above formula, N_rollout(t+Δt) represents the target number of computing resources in the first computing resource pool for the next control period; N_rollout(t) represents the current number of computing resources in the first computing resource pool; k_p represents the proportional gain coefficient; Q_target-Q(t) represents the water level deviation; where Q_target represents the target center value of the water level; Q(t) represents the current number of samples in the sample buffer; the clip(·, N_min, N_max) function is used to limit the target number of computing resources in the next control period to the upper and lower limits of the number of computing resources in the first computing resource pool.

[0104] The above calculation formula is enabled only when the water level of the sample buffer exceeds the [Q_low, Q_high] water level interval; when the water level is within this interval, the expansion or contraction of the first computing power resource pool is not performed.

[0105] Step S2.1.5, performing expansion or contraction on the first computing power resource pool according to the actual effective number of computing power resources.

[0106] In this embodiment, the above proportional feedback control law is enabled to adjust the computing power resources only when the current number of samples in the sample buffer exceeds the water level interval [Q_low, Q_high]; when the current number of samples Q(t) in the sample buffer is within the [Q_low, Q_high] interval, the current number of computing power resources in the first computing power resource pool remains unchanged, and no computing power resource adjustment is performed.

[0107] Specifically, when Q(t)<Q_low, the proportional feedback calculation is triggered, at this time Q_target - Q(t)>0, the control law drives N_rollout to increase, that is, the first computing power resource pool is expanded according to the actual effective number of computing power resources, so as to increase the number of generated training samples.

[0108] Specifically, when Q(t)>Q_high, the proportional feedback calculation is triggered, at this time Q_target - Q(t)<0, the control law drives N_rollout to decrease, that is, the first computing power resource pool is contracted according to the actual effective computing power resource pool, so as to reduce the number of generated training samples.

[0109] In a specific numerical embodiment, the water level interval of the sample buffer is configured as Q_low=2000, Q_high=8000, thus the calculated water level target center value Q_target=(Q_low+Q_high) / 2=5000; the proportional gain coefficient k_p=0.002 is configured; the control period Δt is set as the time consumption of one sample generation batch; the lower limit N_min=2 and the upper limit N_max=12 of the number of computing power resources in the first computing power resource pool are set. At the set initial moment, the current number of samples in the sample buffer Q(0)=1500, and the current number of computing power resources in the first computing power resource pool N_rollout(0)=4.

[0110] According to the aforementioned control calculation formula: N_rollout (t+Δt) = clip ( N_rollout (t) + k_p·(Q_target Q (t)), N_min, N_max ), calculate the actual effective computing power resources in the first control cycle: N_rollout (Δt) = clip (4 + 0.002×(5000-1500), 2, 12)=clip (4+7, 2, 12)=11.

[0111] It is evident that when the sample buffer level is significantly lower than the preset lower limit Q_low, this control law can drive the number of computing resources in the first computing resource pool to approach the upper limit within one control cycle, rapidly improving the training sample generation throughput. As the number of computing resources in the first computing resource pool increases, the sample generation rate increases, and the current number of samples Q(t) in the sample buffer gradually recovers; the water level deviation Q_target... The absolute value of Q(t) decreases accordingly, and the adjustment step size of the computing power resource quantity in each subsequent control cycle is narrowed accordingly until the current sample quantity Q(t) in the sample buffer enters the water level range [Q_low,Q_high] and tends to stabilize near the center value of the water level target Q_target. At this time, the computing power resource quantity of the first computing power resource pool stabilizes and no longer frequently performs expansion and contraction actions.

[0112] Figure 2 This is a schematic diagram illustrating the elastic scaling of the first computing power resource pool driven by the water level of a sample buffer, as shown in this invention. The values ​​shown are illustrative examples. Figure 2 As shown in the attached figure, the causal relationship between the sample buffer level Q(t) and the number of computing resources N_rollout(t) in the first computing resource pool is illustrated using schematic numerical values ​​during this control process. Figure 2 The curves in the diagram are all illustrative values ​​and are only used to explain the logic of the control law: When the current number of samples Q(t) in the sample buffer is lower than the preset lower limit Q_low, the number of computing resources N_rollout(t) in the first computing resource pool increases in a stepwise manner; when the water level rises back into the water level range, the number of computing resources stabilizes; if the current number of samples Q(t) in the sample buffer exceeds the preset upper limit Q_high due to factors such as sample consumption lag, the number of computing resources in the first computing resource pool decreases in a stepwise manner, and the computing resources released by shrinking are returned to the computing resource pool of the intelligent computing cloud platform.

[0113] The beneficial technical effects achievable by steps S2.1.1 to S2.1.5 are as follows: By adopting a proportional feedback control law, computing power resource adjustment is only initiated when the water level in the sample buffer exceeds the preset range, maintaining computing power stability within the range and avoiding frequent fluctuations in computing power resources; it can quickly increase sample generation throughput when samples are insufficient and promptly reduce the generation scale when samples are backlogged, avoiding unnecessary expansion and contraction actions within the normal water level range, reducing scheduling processing overhead, achieving stable and controllable elastic scaling of the first computing power resource pool, ensuring the stability of the training process, and reducing the waste of computing power resources.

[0114] In one embodiment, step S4 includes: Step S4.1: Divide the weight parameters corresponding to the new weight version and the old weight version into multiple weight fragments according to the same splitting rule, and calculate the hash fingerprint of each weight fragment.

[0115] Specifically, for the complete model weight parameters, a unified partitioning rule is pre-defined. Following this rule, the weight parameters corresponding to the new weight version and the old weight version are partitioned, resulting in multiple sets of weight partitions with one-to-one correspondence. The weight parameters corresponding to the two sets of weight versions use completely consistent partition boundaries to ensure that subsequent pairwise comparisons can be made based on partition position. For each partitioned weight partition, a corresponding hash fingerprint is calculated. The hash fingerprint is obtained by calculating the weight parameters within the partition; if any parameter within a weight partition changes, its corresponding hash fingerprint will change.

[0116] Step S4.2: Compare the hash fingerprints of the weight segments corresponding to the old and new weight versions, and identify the weight segments with inconsistent hash fingerprints as the target weight segments that have undergone parameter changes.

[0117] Specifically, based on the corresponding position of each weight shard, the hash fingerprints of the new weight version and the old weight version are compared pairwise. If the hash fingerprints of shards at the same position are the same, it means that the weight parameters within that weight shard have not changed; if the hash fingerprints are different, it means that the parameters within that weight shard have been updated, and that shard is selected as the target weight shard. This allows for precise identification of the weight shards whose parameters have actually changed, filtering out weight shard data whose parameters have not been modified.

[0118] Step S4.3: Push the target weight shard to the first computing power resource pool to perform a hot weight update on the model used by the sample generation subtask. Weight shards with consistent hash fingerprints are not pushed or transmitted.

[0119] Specifically, after selecting the target weight fragments, only the selected target weight fragments are pushed to the first computing resource pool via the network. The sample generation subtask uses the received target weight fragments to update the local model weights. For weight fragments with consistent hash fingerprints and unchanged parameters, network transmission is not performed; the fragment data already saved locally by the sample generation subtask is directly reused, without needing to be re-sent. This method eliminates the need to transmit the complete model weights, reducing the amount of data transmitted during hot updates.

[0120] The beneficial technical effects achievable by steps S4.1 to S4.3 are as follows: by comparing the hash fingerprints calculated on the weight shards, only the target weight shards that have changed are transmitted, without having to distribute the complete model weights, which significantly reduces the network transmission overhead of hot weight updates; the shard-hash comparison method can quickly locate the changed parts and improve the efficiency of hot update execution.

[0121] In one embodiment, step S4.3 includes: During the weight hot update process, the sample generation subtask continuously generates training samples based on the old weight version, and switches to the new weight version after the target weight slice is loaded.

[0122] Specifically, during the entire process of pushing the target weight shards to the first computing power resource pool and performing hot weight updates, the sample generation subtask will not interrupt sample generation. It will continue to use the existing old weight version to output training samples, and sample generation will not be paused due to weight synchronization operations. After all target weight shards have been transmitted and loaded and taken effect on the sample generation subtask side, the weight version switch will be completed. Subsequent sample generation will then be based on the updated new weight version, avoiding sample generation interruptions during the weight synchronization phase and ensuring the continuous operation of the training process.

[0123] The beneficial technical effects that can be achieved by using the above hot update method in step S4.3 are as follows: This step does not interrupt sample generation during the segment update process. The weight version is switched only after all segments have been loaded, avoiding the interruption of sample generation caused by the update operation and ensuring the continuous and stable operation of reinforcement learning training.

[0124] In one embodiment, the heterogeneous computing resource pooling scheduling method of the intelligent computing cloud platform further includes: Step S5: Obtain the task stage of the reinforcement learning training task.

[0125] In this invention, the reinforcement learning training task can be divided into stages based on the training progress status. For example, the task stage of the reinforcement learning training task can be divided into a pre-convergence stage and a post-convergence stage according to the model convergence status.

[0126] It should be noted that the present invention does not limit the number of training stages or the criteria for dividing them, and can be flexibly set according to actual business needs. For example, it can be divided into two stages, three stages or more training stages.

[0127] Step S6: Dynamically adjust the version lag tolerance according to the task stage, wherein the version lag tolerance corresponding to the later stage of the task is greater than the version lag tolerance corresponding to the earlier stage of the task.

[0128] This invention dynamically adjusts the version lag tolerance according to the task stage, so that the version lag tolerance configured in the later stage of the task is greater than the version lag tolerance in the earlier stage.

[0129] For example, suppose the task phase of reinforcement learning training is divided into a pre-convergence phase and a post-convergence phase. The version lag tolerance corresponding to the post-convergence phase is greater than that corresponding to the pre-convergence phase. That is, in the training phase after the model has converged, considering that the model has converged, the sample generation subtask is allowed to generate samples using lagging weight versions over a larger weight version span, adapting to the iterative features of different training phases.

[0130] The beneficial technical effects achievable through steps S5-S6 are as follows: By dynamically adjusting the version lag tolerance according to different training stages of the reinforcement learning training task, a smaller version lag tolerance is adopted in the early stages of training when model parameter updates fluctuate greatly and sample freshness requirements are high, ensuring that samples are generated using newer weight versions and guaranteeing training convergence. In the later stages, when the model tends to converge and parameter updates become more gradual, increasing the version lag tolerance can appropriately relax the weight version update interval, reduce the synchronization frequency of weight partitions, effectively reduce the network transmission pressure and scheduling overhead of the platform, and further improve the overall training performance and resource utilization while ensuring that training accuracy is not affected.

[0131] In one embodiment, step S1 includes: Step S1.1: Obtain hardware capability profile information of each candidate computing resource in the intelligent computing cloud platform.

[0132] Specifically, the hardware capability profile information corresponding to each candidate computing power resource in the intelligent computing cloud platform is read. This hardware capability profile information includes dimensions such as video memory capacity, supported computing precision set, inter-card interconnection bandwidth, and supported operator set, which are used to perform differentiated screening of candidate computing power resources.

[0133] Step S1.2: Based on the hardware capability profile information, determine a preset number of computing resources from the candidate computing resources whose inter-card interconnection bandwidth meets the requirements of aggregate communication, and construct the second computing resource pool.

[0134] Specifically, based on the card-to-card interconnection bandwidth field in the hardware capability profile information, computing resources whose card-to-card interconnection bandwidth can meet the requirements of aggregated communication services such as gradient synchronization are selected, and a preset number of such computing resources are selected to form a second computing resource pool, thereby ensuring the performance requirements of training aggregated communication operations such as gradient aggregation and parameter synchronization.

[0135] Step S1.3: Based on the hardware capability profile information, select computing resources from the candidate computing resources that meet the inference execution requirements in terms of video memory, computing precision, and operator capabilities, and whose inter-card interconnect bandwidth is lower than that of the computing resources in the second computing resource pool, and define them as the available resource range of the first computing resource pool.

[0136] Specifically, the selection process continues based on hardware capability profile information, choosing computing resources whose memory capacity, supported computational precision, and operator capabilities are all suitable for sample generation subtasks. At the same time, the inter-card interconnect bandwidth of this portion of computing resources is lower than that of the computing resources in the second computing resource pool. This portion of computing resources is then designated as the available resource range of the first computing resource pool, allowing the subsequent scaling-up and scaling-down scheduling mechanism to complete the application and reclamation of computing resources within this resource range.

[0137] The beneficial technical effects achievable by steps S1.1 to S1.3 are as follows: Candidate computing power resources on the platform are differentiated and pooled using hardware capability profile information; computing power resources with high inter-card interconnect bandwidth are allocated to the gradient update subtask to meet the performance requirements of gradient synchronization set communication; computing power resources with adequate inference capabilities and relatively low interconnect bandwidth are assigned to the sample generation subtask to achieve matching between heterogeneous hardware and business scenarios, fully leverage the capabilities of different specifications of computing power, and improve the overall resource utilization of the platform.

[0138] Figure 3 This is a diagram illustrating the heterogeneous computing power resource pooling scheduling architecture of the intelligent computing cloud platform provided in this embodiment of the invention. Figure 3 As shown, the scheduling architecture is divided into four layers from top to bottom: algorithm layer, scheduling layer, generation pool, and training pool.

[0139] The algorithm layer carries the reinforcement learning training task, which explicitly declares the version lag tolerance K to constrain the weight version lag span that the sample generation subtask can tolerate.

[0140] The scheduling layer comprises three functional components: a pool scheduler, a level controller, and a weight version pipeline. The pool scheduler is used to perform pool selection and resource management for heterogeneous computing resources; the level controller implements elastic scaling control of the first computing resource pool based on the water level of the sample buffer; and the weight version pipeline is responsible for scheduling the comparison of old and new weight shards, hash verification, and hot weight updates. When the version lag tolerance K increases, the weight version pipeline will reduce the trigger frequency of hot weight updates, thereby reducing network transmission bandwidth consumption.

[0141] The generation pool includes a first computing power resource pool and a sample buffer. The first computing power resource pool supports elastic scaling and is used to deploy sample generation subtasks. The sample buffer is used to cache the training samples produced by the sample generation subtasks.

[0142] The training pool includes a second computing power resource pool. The number of computing power resources in the second computing power resource pool is fixed. This part of the computing power uses accelerator cards with high-speed inter-card interconnection bandwidth to undertake the gradient update subtask of reinforcement learning.

[0143] This architecture forms a hierarchical driving relationship from top to bottom: "task declaration - scheduling decision - resource execution". At the same time, the generation pool and the training pool form a complete runtime closed loop through the transfer of training samples and the distribution and updating of weight versions, supporting the entire heterogeneous computing power resource pooling reinforcement learning training process.

[0144] Figure 4 This is a data interaction diagram illustrating the pooled scheduling of heterogeneous computing resources provided in an embodiment of the present invention. For example... Figure 4 As shown, the architecture includes a second computing resource pool, a first computing resource pool, and a sample buffer.

[0145] The second computing resource pool has a fixed amount of computing resources and is used to run gradient update subtasks to complete model gradient calculation and weight version iteration updates.

[0146] The first computing resource pool supports elastic scaling of computing resources and is used to run the sample generation subtask. The computing resources in the dashed boxes in the diagram represent dynamically expandable and scalable computing resources. The first computing resource pool performs sample generation and continuously writes the generated training samples into the sample buffer.

[0147] The sample buffer serves as an intermediate cache unit, receiving training samples output from the first computing power resource pool. The gradient update subtask of the second computing power resource pool continuously reads and consumes training samples from the sample buffer to carry out model training iterations.

[0148] In the weight synchronization link, weight data is not sent out for every weight update; the target weight shard with parameter changes is pushed to the first computing power resource pool only when the weight version of the model used by the sample generation subtask lags behind the weight version of the gradient update subtask by a version lag tolerance K. This completes the hot update of the model without having to send out the full weights, thus reducing network transmission overhead.

[0149] Thus, a closed-loop operation process of sample production-caching consumption and on-demand weighted sharding is formed among the first computing resource pool, the sample buffer, and the second computing resource pool, realizing the pooled scheduling of heterogeneous computing resources for reinforcement learning training.

[0150] Figure 5 This is an exemplary schematic diagram of a heterogeneous computing power resource pooling scheduling device for an intelligent computing cloud platform proposed in this invention, as shown below. Figure 5 The heterogeneous computing resource pooling scheduling device 500 of the intelligent computing cloud platform shown includes a deployment module 501, an adjustment module 502, a computing module 503, and an update module 504, wherein: The deployment module 501 is used to deploy the sample generation subtask and gradient update subtask in the reinforcement learning training task as heterogeneous subtasks to the first computing resource pool and the second computing resource pool, which are divided from the computing resource pool of the intelligent computing cloud platform, respectively. The computing resources configured in the first computing resource pool and the second computing resource pool are heterogeneous. The adjustment module 502 is used to continuously generate training samples through the sample generation subtask and cache them in the sample buffer, and dynamically adjust the amount of computing resources in the first computing resource pool according to the water level of the sample buffer. The sample buffer is used to continuously supply training samples for the gradient update subtask. The calculation module 503 is used to calculate the version difference between the new weight version and the old weight version currently used by the sample generation subtask after the gradient update subtask iteratively generates a new weight version. The update module 504 is used to update the weight version of the model used in the sample generation subtask from the old weight version to the new weight version when the version difference reaches the preset version lag tolerance, based on the target weight slice where the parameters have changed between the old and new weight versions.

[0151] In one embodiment, the adjustment module 502 is further configured to: expand the capacity of the first computing power resource pool when the water level of the sample buffer is detected to be lower than the preset lower limit, and shrink the capacity of the first computing power resource pool when the water level of the sample buffer is detected to be higher than the preset upper limit; wherein, during the expansion and shrinkage process, the number of computing power resources in the first computing power resource pool is always within the preset number range.

[0152] In one embodiment, the adjustment module 502 is further configured to: mark the computing resources released after the reduction as recyclable and return them to the computing resource pool for use by other computing power task scheduling.

[0153] In one embodiment, the adjustment module 502 is further configured to: configure the water level range, proportional gain coefficient, control period, and upper and lower limits of the number of computing resources in the first computing resource pool of the sample buffer; determine the target center value of the water level based on the water level range, and obtain the water level deviation by combining the current number of samples in the sample buffer; calculate the target number of computing resources in the next control period by combining the current number of computing resources in the first computing resource pool, the proportional gain coefficient, and the water level deviation; limit the target number of computing resources within the upper and lower limits to obtain the actual effective number of computing resources; and expand or shrink the first computing resource pool according to the actual effective number of computing resources.

[0154] In one embodiment, the update module 504 is further configured to: divide the weight parameters corresponding to the new weight version and the old weight version into multiple weight fragments according to the same segmentation rule, and calculate the hash fingerprint of each weight fragment; compare the hash fingerprints of the weight fragments corresponding to the new and old weight versions, and determine the weight fragments with inconsistent hash fingerprints as the target weight fragments where parameter changes have occurred; push the target weight fragments to the first computing power resource pool to perform hot weight updates on the model used by the sample generation subtask, wherein weight fragments with consistent hash fingerprints are not pushed and transmitted.

[0155] In one embodiment, the update module 504 is further configured to: during the weight hot update process, the sample generation subtask continuously generates training samples based on the old weight version, and switches to the new weight version after the target weight slice is loaded.

[0156] In one embodiment, the heterogeneous computing power resource pooling scheduling device 500 of the intelligent computing cloud platform further includes a tolerance adjustment module, used to: obtain the task stage of the reinforcement learning training task; and dynamically adjust the version lag tolerance according to the task stage, wherein the version lag tolerance corresponding to the later stage of the task is greater than the version lag tolerance corresponding to the earlier stage of the task.

[0157] In one embodiment, the deployment module 501 is further configured to: acquire hardware capability profile information of each candidate computing power resource in the intelligent computing cloud platform; determine a preset number of computing power resources whose inter-card interconnect bandwidth meets the requirements of aggregate communication from the candidate computing power resources based on the hardware capability profile information, and construct a second computing power resource pool; and select computing power resources from the candidate computing power resources whose video memory, computing precision, and operator capabilities meet the requirements of inference execution and whose inter-card interconnect bandwidth is lower than that of the computing power resources in the second computing power resource pool based on the hardware capability profile information, and define them as the available resource range of the first computing power resource pool.

[0158] The heterogeneous computing power resource pooling scheduling device for the intelligent computing cloud platform provided by this invention can realize the various processes of the above-mentioned heterogeneous computing power resource pooling scheduling method for the intelligent computing cloud platform. The technical features are one-to-one and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0159] It should be noted that the heterogeneous computing power resource pooling scheduling device of the intelligent computing cloud platform in this invention can be a device, or it can be a component, integrated circuit, or chip in an electronic device.

[0160] The present invention also provides an electronic device, see below. Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory 601, a processor 602, and a program or instructions stored in the memory 601 that run on the processor 602. When the program or instructions are executed by the processor 602, they can achieve the following: Figure 1 The steps in the corresponding implementation of the heterogeneous computing power resource pooling scheduling method for the intelligent computing cloud platform, and the achievement of the same beneficial effects, will not be elaborated here.

[0161] The processor 602 can be a CPU, ASIC, FPGA, or GPU.

[0162] Those skilled in the art will understand that all or part of the steps of the above-described embodiments of the heterogeneous computing power resource pooling scheduling method for computing cloud platforms can be implemented by hardware related to program instructions, and the program can be stored in a readable medium.

[0163] The present invention also provides a readable storage medium on which a computer program is stored, and which, when executed by a processor, can perform the above-described functions. Figure 1 The corresponding implementation of the heterogeneous computing power resource pooling scheduling method in the cloud platform can achieve any step and achieve the same technical effect. To avoid repetition, it will not be described again here. The storage medium mentioned is such as read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0164] The present invention also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described... Figure 1 The corresponding implementation of the heterogeneous computing power resource pooling scheduling method for the cloud platform can achieve the same technical effect, and will not be described in detail here to avoid repetition.

[0165] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0166] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0167] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of this application.

[0168] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for pooling and scheduling heterogeneous computing resources in an intelligent computing cloud platform, characterized in that, include: Step S1: The sample generation subtask and gradient update subtask in the reinforcement learning training task are treated as heterogeneous subtasks and deployed to the first computing resource pool and the second computing resource pool, which are divided from the computing resource pool of the intelligent computing cloud platform, respectively. The computing resources configured in the first computing resource pool and the second computing resource pool are heterogeneous. Step S2: Training samples are continuously generated and cached in the sample buffer through the sample generation subtask, and the amount of computing resources in the first computing resource pool is dynamically adjusted according to the water level of the sample buffer. The sample buffer continuously supplies training samples to the gradient update subtask. Step S3: After the gradient update subtask iteratively generates a new weight version, calculate the version difference between the new weight version and the old weight version currently used by the sample generation subtask. Step S4: When the version difference reaches the preset version lag tolerance, based on the target weight slice where the parameters have changed between the old and new weight versions, the weight version of the model used in the sample generation subtask is updated from the old weight version to the new weight version.

2. The method according to claim 1, characterized in that, Step S2 includes: Step S2.1: When the water level of the sample buffer is detected to be lower than the preset lower limit, the first computing power resource pool is expanded; and when the water level of the sample buffer is detected to be higher than the preset upper limit, the first computing power resource pool is reduced. During the expansion and contraction process, the number of computing resources in the first computing resource pool remains within a preset range.

3. The method according to claim 2, characterized in that, Following step S2.1, the method further includes: Step S2.2: Mark the computing resources released after the reduction as recyclable and return them to the computing resource pool for use by other computing power task scheduling.

4. The method according to claim 2, characterized in that, Step S2.1 includes: Step S2.1.1: Configure the water level range, proportional gain coefficient, control period, and upper and lower limits of the number of computing resources in the first computing resource pool for the sample buffer. Step S2.1.2: Determine the target center value of the water level based on the water level range, and obtain the water level deviation by combining it with the current number of samples in the sample buffer. Step S2.1.3: Combine the current number of computing resources in the first computing resource pool, the proportional gain coefficient, and the water level deviation to calculate the target number of computing resources for the next control cycle; Step S2.1.4: Limit the target computing power resource quantity within the upper and lower limits of the quantity to obtain the actual effective computing power resource quantity; Step S2.1.5: Expand or shrink the first computing power resource pool according to the actual number of effective computing power resources.

5. The method according to any one of claims 1-4, characterized in that, Step S4 includes: Step S4.1: Divide the weight parameters corresponding to the new weight version and the old weight version into multiple weight fragments according to the same partitioning rule, and calculate the hash fingerprint of each weight fragment. Step S4.2: Compare the hash fingerprints of the weight segments corresponding to the old and new weight versions, and determine the weight segments with inconsistent hash fingerprints as the target weight segments that have undergone parameter changes. Step S4.3: Push the target weight shard to the first computing power resource pool to perform a hot weight update on the model used by the sample generation subtask. Weight shards with consistent hash fingerprints are not pushed or transmitted.

6. The method according to claim 5, characterized in that, Step S4.3 includes: During the hot weight update process, the sample generation subtask continuously generates training samples based on the old weight version. After the target weight slice is loaded, it switches to the new weight version.

7. The method according to claim 1, characterized in that, The method further includes: Step S5: Obtain the task stage of the reinforcement learning training task; Step S6: Dynamically adjust the version lag tolerance according to the task stage, wherein the version lag tolerance corresponding to the later stage of the task is greater than the version lag tolerance corresponding to the earlier stage of the task.

8. The method according to claim 1, characterized in that, Step S1 includes: Step S1.1: Obtain hardware capability profile information of each candidate computing resource in the intelligent computing cloud platform; Step S1.2: Based on the hardware capability profile information, determine a preset number of computing resources from the candidate computing resources whose inter-card interconnection bandwidth meets the requirements of aggregate communication, and construct the second computing resource pool. Step S1.3: Based on the hardware capability profile information, select computing resources from the candidate computing resources that meet the inference execution requirements in terms of video memory, computing precision, and operator capability, and whose inter-card interconnect bandwidth is lower than that of the computing resources in the second computing resource pool, and define them as the available resource range of the first computing resource pool.

9. A heterogeneous computing power resource pooling scheduling device for an intelligent computing cloud platform, characterized in that, include: The deployment module is used to treat the sample generation subtask and gradient update subtask in the reinforcement learning training task as heterogeneous subtasks and deploy them to the first computing resource pool and the second computing resource pool, which are divided from the computing resource pool of the intelligent computing cloud platform, respectively. The computing resources configured in the first computing resource pool and the second computing resource pool are heterogeneous. The adjustment module is used to continuously generate training samples through the sample generation subtask and cache them in the sample buffer, and dynamically adjust the amount of computing resources in the first computing resource pool according to the water level of the sample buffer, wherein the sample buffer continuously supplies training samples to the gradient update subtask. The calculation module is used to calculate the version difference between the new weight version and the old weight version currently used by the sample generation subtask after the gradient update subtask iteratively generates a new weight version; The update module is used to update the weight version of the model used in the sample generation subtask from the old weight version to the new weight version when the version difference reaches a preset version lag tolerance, based on the target weight slice where the parameters have changed between the old and new weight versions.

10. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the heterogeneous computing power resource pooling scheduling method of the intelligent computing cloud platform as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the heterogeneous computing power resource pooling scheduling method of the intelligent computing cloud platform as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, It includes computer instructions, which, when executed by a processor, implement the steps of the heterogeneous computing power resource pooling scheduling method of the intelligent computing cloud platform as described in any one of claims 1 to 8.

Images