Computing power resource dynamic adjustment-based reasoning acceleration method and device for intelligent computing center cloud platform

Abstract

The invention provides an intelligent computing center cloud platform reasoning acceleration method and device based on computing power resource dynamic adjustment, and relates to the technical field of intelligent computing centers, intelligent computing centers and computing power infrastructure. The method comprises the following steps: S1, obtaining data reading parameters corresponding to each batch in a plurality of batches, the data reading parameters comprise a data sequence length parameter and / or a batch size parameter, and the batches are the number of times of reading the data file; s2, sequentially reading data files of a first computing power operation task based on the data reading parameters to obtain data corresponding to each batch; s3, reasoning the data corresponding to each batch in sequence to obtain a plurality of intermediate results; and S4, combining the plurality of intermediate results to obtain a reasoning result. The calculation efficiency of the intelligent calculation center is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent computing centers, smart computing centers and computing power infrastructure, and specifically to an inference acceleration method and device for an intelligent computing center cloud platform based on dynamic adjustment of computing power resources. Background Art

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "intelligent computing centers" have emerged.

[0003] An "Intelligent Computing Center" is a facility that uses large-scale heterogeneous computing resources, including general-purpose and intelligent computing power, to provide the computing power, data, and algorithms required for AI applications (such as AI deep learning model development, model training, and model inference). The Intelligent Computing Center encompasses facilities, hardware, and software, and provides a full stack of capabilities, from bottom-level computing power to top-level application enablement.

[0004] “Intelligent Computing Center” includes but is not limited to “Smart Computing Center”.

[0005] "Intelligent Computing Center" refers to an artificial intelligence computing center. It is a type of computing power infrastructure that is based on artificial intelligence theory, adopts artificial intelligence computing architecture, and provides computing power services, data services, and algorithm services required for artificial intelligence applications.

[0006] "Computing power" is the core of "intelligent computing center" and "intelligent computing center". It is the ability of computer equipment or computing / data center to process parameters. It is the ability of computer hardware and software to work together to perform certain computing needs. It is the computing power to achieve target result output by processing parameter data. It is a new type of productivity that integrates parameter computing power, network carrying capacity, and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] In intelligent computing centers, computing resources are usually provided by accelerator cards to execute computing operation tasks. In the prior art, the full amount of data to be processed is usually loaded into the memory, and the accelerator card is used to infer the full amount of data to obtain the inference results. However, in the prior art, there are situations where there is a large amount of full data to be processed, such as inference and prediction of long sequence protein data. At this time, directly loading the full amount of data into the memory and inferring may cause a memory overflow (Out Of Memory, OOM) problem. When this problem occurs, the computing operation task will be forced to be interrupted, resulting in very low computing efficiency.

[0008] It can be seen that the existing technology has the problem of low computing efficiency of the intelligent computing center. Summary of the Invention

[0009] The embodiment of the present invention provides an inference acceleration method and device for an intelligent computing center cloud platform based on dynamic adjustment of computing resources, so as to solve the problem of low computing efficiency caused by excessive load of the acceleration card of the intelligent computing center in the prior art.

[0010] To solve the above problems, the present invention is achieved as follows:

[0011] In a first aspect, an embodiment of the present invention provides an inference acceleration method based on dynamic adjustment of computing resources on an intelligent computing center cloud platform, comprising:

[0012] Step S1, obtaining data reading parameters corresponding to each batch in a plurality of batches, wherein the data reading parameters include a data sequence length parameter and / or a batch size parameter, and the plurality of batches is the number of times the data file is read;

[0013] Step S2: Read the data files of the first computing power running tasks in sequence based on the data reading parameters to obtain data corresponding to each batch;

[0014] Step S3: inferring the data corresponding to each batch in turn to obtain multiple intermediate results;

[0015] Step S4: Merge the multiple intermediate results to obtain an inference result.

[0016] In one embodiment, step S1 includes:

[0017] Step S11: When it is necessary to obtain data reading parameters corresponding to the Nth batch, obtain the GPU utilization and video memory utilization corresponding to the first GPU when performing inference on the data corresponding to the N-1th batch, where the first GPU is the GPU that executes the first computing power operation task, and N is a positive integer greater than 1;

[0018] Step S12: Calculate the data sequence length parameter based on the GPU utilization rate corresponding to the first GPU;

[0019] Step S13: Calculate the batch size parameter based on the video memory utilization corresponding to the first GPU and the data sequence length parameter.

[0020] In one embodiment, the video memory utilization includes available video memory utilization corresponding to the first GPU, a first occupied video memory utilization, and a second occupied video memory utilization, wherein the first occupied video memory utilization is used to characterize the occupancy of model parameters in the first computing power running task, and the second occupied video memory utilization is used to characterize the system occupied video memory;

[0021] The step S13 includes:

[0022] Step S131: Obtain the available video memory utilization, the first occupied video memory utilization, the second occupied video memory utilization and inference accuracy, as well as the data embedding dimension and the number of attention heads of the task running with the first computing power;

[0023] Step S132: Subtract the first occupied video memory utilization and the second occupied video memory utilization from the available video memory utilization to obtain an intermediate occupied video memory utilization;

[0024] Step S133: Calculate a first sum value, where the first sum value is the sum of a first product and a second product, where the first product is the product of the data sequence length parameter, the number of attention heads, and the inference accuracy, and the second product is the product of the data sequence length parameter, the data embedding dimension, and the inference accuracy;

[0025] Step S134: Set the quotient of the intermediate occupied video memory utilization and the first sum value as the batch size parameter.

[0026] In one embodiment, the inference precision is a half-precision floating point number.

[0027] In one embodiment, step S1 includes:

[0028] Step S11 ′: when it is necessary to obtain the data reading parameters corresponding to the first batch, the preset length parameter is set as the data sequence length parameter, and / or the preset size parameter is set as the batch size parameter.

[0029] In one embodiment, step S3 includes:

[0030] Step S31: setting an input dimension parameter based on the data sequence length parameter corresponding to the first batch, where the first batch is one of the multiple batches;

[0031] Step S32: Infer the first batch of data based on input dimension parameters through a preset inference model to obtain a first intermediate result, where the preset inference model is a model trained to infer data, and the first intermediate result is one of the multiple intermediate results.

[0032] In one embodiment, after step S32, the method further includes:

[0033] Step S33: After obtaining the first intermediate result and before inferring the data corresponding to the second batch, delete the intermediate variables and cache generated during the inference process of the first batch of data, and the second batch is a batch after the first batch.

[0034] In a second aspect, an embodiment of the present invention further provides an inference acceleration device based on dynamic adjustment of computing resources on an intelligent computing center cloud platform, comprising:

[0035] an acquisition module, configured to acquire data reading parameters corresponding to each of a plurality of batches, wherein the data reading parameters include a data sequence length parameter and / or a batch size parameter, and the plurality of batches are the number of times the data file is read;

[0036] A reading module, configured to sequentially read the data files of the first computing power running tasks based on the data reading parameters to obtain data corresponding to each batch;

[0037] An inference module is used to sequentially infer the data corresponding to each batch to obtain multiple intermediate results;

[0038] A merging module is used to merge the multiple intermediate results to obtain an inference result.

[0039] In the third aspect, the present invention also provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and runnable on the processor. When the computer program is executed by the processor, the steps of the inference acceleration method based on dynamic adjustment of computing resources of the intelligent computing center cloud platform as described in the first aspect above are implemented.

[0040] In a fourth aspect, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the inference acceleration method based on dynamic adjustment of computing power resources of the intelligent computing center cloud platform as described in the first aspect above are implemented.

[0041] In a fifth aspect, the present invention also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the inference acceleration method based on dynamic adjustment of computing power resources of the intelligent computing center cloud platform as described in the first aspect above.

[0042] In the present invention, step S1, obtains the data reading parameters corresponding to each batch in a plurality of batches, the data reading parameters include a data sequence length parameter and / or a batch size parameter, and the plurality of batches are the number of times the data file is read; step S2, based on the data reading parameters, sequentially reads the data file of the first computing power running task to obtain the data corresponding to each batch; step S3, sequentially infers the data corresponding to each batch to obtain a plurality of intermediate results; step S4, merges the plurality of intermediate results to obtain an inference result. In this way, by sequentially reading the data file of the first computing power running task based on the data reading parameters, the data corresponding to each batch is obtained, thereby avoiding reading all the data at once; then sequentially inferring the data corresponding to each batch to obtain a plurality of intermediate results, and merging the plurality of intermediate results to obtain an inference result, thereby avoiding the problem of video memory overflow caused by reading all the data at once and the computing power task being forced to be interrupted, thereby greatly improving computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 This is a flowchart of an inference acceleration method based on dynamic adjustment of computing resources of an intelligent computing center cloud platform provided by an embodiment of the present invention;

[0045] Figure 2 This is a structural diagram of an inference acceleration device based on dynamic adjustment of computing resources for an intelligent computing center cloud platform provided by an embodiment of the present invention;

[0046] Figure 3 This is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0048] The "computing power" mentioned in the present invention refers to: the ability of computer equipment or computing / data centers to process information, the ability of computer hardware and software to work together to execute certain computing requirements, and the computing power to achieve target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity, and mainly provides services to society through computing power infrastructure.

[0049] The "computing power" (Computational Power, CP) mentioned in the present invention refers to: the ability of a data center server to process data and output results. It is a comprehensive indicator to measure the computing power of a data center, including general computing power, super computing power and intelligent computing power. The commonly used unit of measurement is the number of floating-point operations performed per second (FLOPS, 1EFLOPS=10^18FLOPS). The larger the value, the stronger the comprehensive computing power. According to calculations, 1EFLOPS is approximately the computing power output of 5 Tianhe-2A or 500,000 mainstream server CPUs or 2 million mainstream notebooks. The calculation formula is: CP=CP 通用 +CP 智能 +CP 超级 .

[0050] The "carrying capacity" (Network Power, NP) mentioned in the present invention refers to: it is the performance of the data transmission capability of the computing power facility, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, etc. It involves network transmission within and between data centers, and is a comprehensive indicator for measuring network transmission scheduling capabilities.

[0051] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in terms of data storage capacity, performance, security and reliability, and environmental friendliness. It is a comprehensive indicator for measuring a data center's data storage capacity, encompassing both external storage devices such as storage arrays and internal server storage. Storage capacity is commonly measured in exabytes (EB, 1EB = 2^60 bytes), while performance is commonly measured in IOPS / TB (Input / Output Operations Per Second / TB). Disaster recovery ratio is a key indicator of security and reliability.

[0052] The "computing power infrastructure" mentioned in the present invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, and can realize the centralized calculation, storage, transmission and application of information.

[0053] The "new information infrastructure" mentioned in the present invention refers to: mainly including network infrastructure such as 5G networks, fiber-optic broadband networks, backbone networks, international communication networks, satellite Internet, computing power infrastructure such as data centers, general computing power centers, intelligent computing centers, supercomputing centers, and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0054] The "computing power" mentioned in the present invention includes: general computing power, intelligent computing power and super computing power.

[0055] The "general computing power" mentioned in the present invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0056] The "intelligent computing power" mentioned in this invention refers to: a computing platform based on specialized chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various innovative artificial intelligence applications, such as natural language processing and machine vision.

[0057] The "supercomputing power" mentioned in the present invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for calculations in cutting-edge scientific fields, such as planetary simulation, drug molecule design, genetic analysis, etc.

[0058] The "intelligent computing center" described in this article refers to a facility that provides the computing power, data, and algorithms required for artificial intelligence applications (such as AI deep learning model development, model training, and model inference) by utilizing large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.). The intelligent computing center encompasses facilities, hardware, and software, and can provide a full stack of capabilities, from bottom-level computing power to top-level application enablement.

[0059] The "intelligent computing center cloud platform" mentioned in the present invention refers to: a cloud computing platform that provides comprehensive services based on the hardware resources and software resources of the intelligent computing center.

[0060] The "intelligent computing center" mentioned in the present invention includes but is not limited to the "intelligent computing center".

[0061] The "intelligent computing center" mentioned in the present invention is an artificial intelligence computing center, which is a type of computing power infrastructure based on artificial intelligence theory, adopts artificial intelligence computing architecture, and provides computing power services, data services and algorithm services required for artificial intelligence applications.

[0062] The "computing power center" mentioned in the present invention refers to: a facility that is mainly composed of infrastructure such as wind, fire, water, electricity, and IT hardware and software equipment, and has computing power, transportation capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0063] The "supercomputing center" mentioned in the present invention refers to: a supercomputing data center, which is a data center based on a supercomputer or a large-scale computing cluster, which can provide large-scale computing, storage and network services and other functions, and is widely used in application scenarios such as aerospace, national defense, oil exploration, climate modeling and genome sequencing.

[0064] The "computing resources" mentioned in the present invention refer to: technologies and facilities with information calculation, transmission, storage and application capabilities required for the development of a digital society, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guarantee resources such as wind, fire, water, and electricity.

[0065] The “model” mentioned in the present invention includes but is not limited to a “large language model” and a “multimodal large model”.

[0066] The "large language model" mentioned in the present invention refers to a large-scale language model (LLM), which is a language model with a large parameter scale. It is designed to understand and generate human language. It is trained with a large amount of text data and can perform a wide range of tasks including text summarization, translation, sentiment analysis, etc.

[0067] The "Multimodal Large Models" mentioned in the present invention refer to models that combine multimodal information such as text, images, video, and audio for training, including but not limited to multimodal large language models.

[0068] The "computing power operation task" mentioned in the present invention refers to: a specific workload or job executed on computing power resources that requires a certain amount of computing power support, usually involving complex data processing, numerical calculation, model reasoning or simulation scenarios.

[0069] See Figure 1 , Figure 1 This is a flowchart of an inference acceleration method based on dynamic adjustment of computing resources on an intelligent computing center cloud platform provided by an embodiment of the present invention. Figure 1 As shown, the following steps are included:

[0070] Step S1: Obtain data reading parameters corresponding to each batch in a plurality of batches, wherein the data reading parameters include a data sequence length parameter and / or a batch size parameter, and the plurality of batches are the number of times the data file is read.

[0071] The above-mentioned multiple batches can be a pre-set number of times to read the data file, or can be a batch reading of the data file based on data reading parameters until all the data in the data file is read. The data reading parameters are used to obtain the data corresponding to each batch, and the data reading parameters include a data sequence length parameter and / or a batch size parameter, so that when reading data from the data file, the data read into the memory is limited based on the data sequence length parameter and / or the batch size parameter, so as to avoid the problem of out of memory (OOM) caused by the length of the data sequence read into the memory being too long or the batch size being too large.

[0072] Step S2: Based on the data reading parameters, the data files of the first computing power running tasks are read in sequence to obtain the data corresponding to each batch.

[0073] The above-mentioned first computing power operation task is a computing power operation task that needs to call the computing power resources of the intelligent computing center to execute. The data file of the first computing power operation task is used to provide the full amount of data to be read. By reading the data file of the first computing power operation task in sequence, the data corresponding to each batch is obtained, avoiding reading the full amount of data at one time, so that the load of the accelerator card is at a reasonable level when reasoning about the data corresponding to each batch, thereby greatly improving the computing efficiency. In some embodiments, the first computing power operation task is a task with a large amount of data, such as a protein three-dimensional structure prediction task or a large model training task. It should be noted that for the protein three-dimensional structure prediction task, its video memory consumption increases quadratically with the length of the data sequence. The video memory utilization rate of an amino acid sequence of 1200 length is 4 times that of an amino acid sequence of 600 length, so it is necessary to read the data file in batches to overcome the OOM problem.

[0074] Among them, reading the data files of the first computing power running task in sequence based on the data reading parameters can be implemented through the data stream processing module of the intelligent computing center. The data stream processing module can perform streaming loading and dynamic blocking on the data files to avoid full data memory occupation. Specifically, the generator of the data stream processing module is used to read the data files in batches to avoid loading the full amount of data into the memory at one time, and generate (sequence_id, sequence) tuples (that is, the data corresponding to each batch) for each batch to support high-throughput processing. Among them, reading the data files of the first computing power running task in sequence can be implemented by the following code:

[0075]

[0076] The above code is used to read the data files in sequence to obtain the data corresponding to each batch.

[0077] Step S3: Inferring the data corresponding to each batch in turn to obtain multiple intermediate results.

[0078] The aforementioned sequential reasoning of the data corresponding to each batch involves the accelerator card in the intelligent computing center providing computing resources to sequentially reason about the data corresponding to each batch. The sequential reasoning of the data corresponding to each batch involves reasoning about the data corresponding to the currently acquired batch before obtaining and reasoning about the data corresponding to the next batch. This ensures that only one batch of data is stored in memory at a time, avoiding the need to read all the data at once. This keeps the accelerator card's load at a reasonable level when reasoning about the data corresponding to each batch, thereby significantly improving computing efficiency.

[0079] Step S4: Merge the multiple intermediate results to obtain an inference result.

[0080] The above-mentioned merging of the multiple intermediate results to obtain the inference result can be performed by first saving the multiple intermediate results in batches, and after completing the inference on the data corresponding to all batches, merging all the intermediate results to obtain the inference result.

[0081] For example, the storage and merging of intermediate results can be achieved through the result storage module in the intelligent computing center. Among them, the result storage module can achieve efficient storage of inference results. In some embodiments, the intermediate results can be stored using incremental writing technology, for example, using comma-separated values ​​(CSV) append mode to save intermediate results batch by batch, which can be specifically implemented by the following code:

[0082]

[0083] In this way, the above code is used to store and merge the intermediate results to obtain the inference results.

[0084] In the present invention, step S1, obtains the data reading parameters corresponding to each batch in a plurality of batches, the data reading parameters include a data sequence length parameter and / or a batch size parameter, and the plurality of batches are the number of times the data file is read; step S2, based on the data reading parameters, sequentially reads the data file of the first computing power running task to obtain the data corresponding to each batch; step S3, sequentially infers the data corresponding to each batch to obtain a plurality of intermediate results; step S4, merges the plurality of intermediate results to obtain an inference result. In this way, by sequentially reading the data file of the first computing power running task based on the data reading parameters, the data corresponding to each batch is obtained, thereby avoiding reading all the data at once; then sequentially inferring the data corresponding to each batch to obtain a plurality of intermediate results, and merging the plurality of intermediate results to obtain an inference result, reading all the data at once causes a problem of video memory overflow and forced interruption of the computing power task, thereby greatly improving computing efficiency.

[0085] In one embodiment, step S1 includes:

[0086] Step S11: When it is necessary to obtain data reading parameters corresponding to the Nth batch, obtain the GPU utilization and video memory utilization corresponding to the first GPU when performing inference on the data corresponding to the N-1th batch, where the first GPU is the GPU that executes the first computing power operation task, and N is a positive integer greater than 1;

[0087] Step S12: Calculate the data sequence length parameter based on the GPU utilization rate corresponding to the first GPU;

[0088] Step S13: Calculate the batch size parameter based on the video memory utilization corresponding to the first GPU and the data sequence length parameter.

[0089] It's important to note that existing technologies often use fixed data loading strategies (e.g., fixed input dimensions, fixed batch sizes, etc.), which fail to dynamically adjust resource usage based on the current batch data (especially changes in sequence length). When batches contain extremely long sequences or have vastly different sequence lengths, using fixed data loading strategies can lead to low computing resource utilization or even insufficient computing resources for model inference.

[0090] To solve the above problems, in the present invention, the data sequence length parameter of the Nth batch is determined by the GPU utilization when the first GPU performs inference on the N-1th batch, and the data sequence length parameter of the Nth batch is determined by the video memory utilization and the data sequence length parameter when the first GPU performs inference on the N-1th batch, thereby correcting the data corresponding to each obtained batch, so that the video memory utilization and GPU utilization of the first GPU are maintained at a reasonable level when inferring the data corresponding to the Nth batch, avoiding the situation where the GPU utilization or video memory utilization is too low.

[0091] In some implementations, the data sequence length parameter and batch size parameter can be adjusted by the model dynamic scheduling module of the intelligent computing center. The model dynamic scheduling module can dynamically adapt the data sequence length parameter and batch size parameter of the model to maximize GPU utilization and video memory utilization.

[0092] The data sequence length parameter can be directly determined by GPU utilization. Specifically, if GPU utilization is high when inferring the data corresponding to batch N-1, the data sequence length parameter obtained for batch N needs to be reduced. If GPU utilization is low when inferring the data corresponding to batch N-1, the data sequence length parameter obtained for batch N needs to be increased.

[0093] In some implementations, the data sequence length parameter may be adjusted using the following code:

[0094]

[0095] The above code is used to construct the input layer for the data sequence length parameter of the data corresponding to the current batch, avoiding the waste of video memory caused by fixed dimensions.

[0096] In some other implementations, the batch size parameter can be adjusted by the following code:

[0097]

[0098] The above code determines whether the batch size parameter needs to be adjusted under different video memory utilization conditions. The batch size parameter needs to be calculated based on the memory utilization and the data sequence length parameter.

[0099] Specifically, in one embodiment, the video memory utilization includes available video memory utilization corresponding to the first GPU, a first occupied video memory utilization, and a second occupied video memory utilization, wherein the first occupied video memory utilization is used to characterize the occupancy of model parameters in the first computing power running task, and the second occupied video memory utilization is used to characterize the system occupied video memory;

[0100] The step S13 includes:

[0101] Step S131: Obtain the available video memory utilization, the first occupied video memory utilization, the second occupied video memory utilization and inference accuracy, as well as the data embedding dimension and the number of attention heads of the task running with the first computing power;

[0102] Step S132: Subtract the first occupied video memory utilization and the second occupied video memory utilization from the available video memory utilization to obtain an intermediate occupied video memory utilization;

[0103] Step S133: Calculate a first sum value, where the first sum value is the sum of a first product and a second product, where the first product is the product of the data sequence length parameter, the number of attention heads, and the inference accuracy, and the second product is the product of the data sequence length parameter, the data embedding dimension, and the inference accuracy;

[0104] Step S134: Set the quotient of the intermediate occupied video memory utilization and the first sum value as the batch size parameter.

[0105] It's important to note that to avoid OOM, existing technologies typically use smaller batch sizes for processing. However, this approach significantly reduces GPU utilization, resulting in lower overall inference throughput and longer task completion times. Furthermore, the retry of computing tasks caused by memory overflow in existing technologies further exacerbates GPU computing resource loss.

[0106] To address the above issues, the present invention obtains the available memory utilization, the first occupied memory utilization, the second occupied memory utilization, and the inference accuracy, as well as the data embedding dimension and number of attention heads for the first computing power running task; subtracts the first occupied memory utilization and the second occupied memory utilization from the available memory utilization to obtain the intermediate occupied memory utilization; calculates a first sum, which is the sum of the first product and the second product, where the first product is the product of the data sequence length parameter, the number of attention heads, and the inference accuracy, and the second product is the product of the data sequence length parameter, the data embedding dimension, and the inference accuracy; and sets the quotient of the intermediate occupied memory utilization and the first sum as the batch size parameter. In this way, the batch size parameter is calculated using the memory utilization and the data sequence length parameter, and model inference using this batch size parameter can effectively improve GPU utilization.

[0107] It should be noted that the memory utilization of the first GPU is mainly expressed by the following formula:

[0108] M total =M model +M activation +M data +M base ;

[0109] In the formula, M total is the total memory utilization; M model is the memory utilization of the model parameters, which is a static value; M activation For the forward calculation of intermediate activation values, M activation is a dynamically changing value related to the data sequence length parameter; M data is the memory utilization rate of input / output data, M data is a dynamically changing value related to the batch size parameter; M base It is the system's video memory utilization, which is a static value.

[0110] Among them, the memory utilization of the model parameters is usually M model =4×P / 10 9 (GB), P is the number of model parameters, the coefficient 4 means that one parameter is 4 bytes, and the default precision is FP32.

[0111] Forward calculation of intermediate activation value M activation Calculated by the following formula:

[0112] M activation =2×L 2 ×H×B×k / 10 9 (GB);

[0113] In the formula, L is the data sequence length parameter, H is the number of attention heads, B is the batch size parameter, k is the precision coefficient, and the coefficient 2 represents the space of the key-value pair matrix storing the data.

[0114] Input / output data memory utilization M data Calculated by the following formula:

[0115] M data =4×L×B×E×k / 10 9 (GB);

[0116] In the formula, E is the embedding dimension.

[0117] In this way, the batch size parameter batch_size can be inferred from the above formula. The specific calculation process can be expressed as follows:

[0118]

[0119] In the formula, M available The available video memory utilization.

[0120] In one embodiment, the inference precision is a half-precision floating point number.

[0121] It should be noted that during inference, higher inference precision leads to higher accuracy results, but this also places a greater load on the accelerator card and can cause gradient overflow. To balance accuracy and accelerator card load and prevent gradient overflow, this invention uses half-precision floating-point numbers (FP16) for inference precision. Compared to full-precision floating-point numbers (FP32), this prevents gradient overflow, reduces accelerator card load, and maintains higher accuracy.

[0122] Specifically, the following code can be used to express reasoning using FP16 precision:

[0123] tf.keras.mixed_precision.set_global_policy('mixed_float16')#Enable FP16 globally

[0124] opt=tf.keras.optimizers.Adam(learning_rate=1e-3)

[0125] opt = tf.keras.mixed_precision.LossScaleOptimizer(opt) #Add loss scaling

[0126] In this way, the above code enables FP16 precision for inference and obtains the inference result.

[0127] In one embodiment, step S1 includes:

[0128] Step S11 ′: when it is necessary to obtain the data reading parameters corresponding to the first batch, the preset length parameter is set as the data sequence length parameter, and / or the preset size parameter is set as the batch size parameter.

[0129] In the present invention, when reading the data corresponding to the first batch, since there is no video memory utilization and GPU utilization corresponding to the previous batch, the data reading parameters cannot be calculated through the video memory utilization and GPU utilization. Therefore, the preset values ​​are used as the data sequence length parameters and / or batch size parameters to realize the reading of the first batch of data.

[0130] In one embodiment, step S3 includes:

[0131] Step S31: setting an input dimension parameter based on the data sequence length parameter corresponding to the first batch, where the first batch is one of the multiple batches;

[0132] Step S32: Infer the first batch of data based on input dimension parameters through a preset inference model to obtain a first intermediate result, where the preset inference model is a model trained to infer data, and the first intermediate result is one of the multiple intermediate results.

[0133] It should be noted that when performing inference using a preset inference model, memory must be allocated for the preset inference model before inference can be performed. When performing inference on data corresponding to different batches, the memory required by the preset inference model may vary due to differences in data sequence length and batch size.

[0134] In the present invention, to reduce excess memory allocated to a preset inference model, an input dimension parameter is set based on the data sequence length parameter corresponding to a first batch, where the first batch is one of the multiple batches; the preset inference model is used to infer the data of the first batch based on the input dimension parameter to obtain a first intermediate result, where the preset inference model is a model trained to infer data. In this way, the input dimension parameter is determined based on the data of the first batch, and the required memory can be determined based on the input dimension parameter. This reduces memory waste when the preset inference model infers the data of the first batch based on the input dimension parameter, prevents OOM problems, and improves the utilization of computing resources.

[0135] Among them, adjusting the input dimension parameters can be achieved through the following code:

[0136] max_batch_len=max(len(seq)for seq in batch_seqs)

[0137] dynamic_seq_len=min(512,max_batch_len+2)

[0138] In the code, the upper limit of the input dimension parameter is set through the data sequence length parameter.

[0139] In one embodiment, after step S32, the method further includes:

[0140] Step S33: After obtaining the first intermediate result and before inferring the data corresponding to the second batch, delete the intermediate variables and cache generated during the inference process of the first batch of data, and the second batch is a batch after the first batch.

[0141] It should be noted that after processing the data corresponding to each batch, there will be data such as intermediate variables and caches left over. The intermediate variables and caches will occupy a certain amount of computing resources, but they will not be needed later. In order to improve the utilization of computing resources, in the present invention, after obtaining the first intermediate result and before reasoning the data corresponding to the second batch, the intermediate variables and caches generated during the reasoning process of the first batch of data are deleted. In this way, by deleting the intermediate variables and caches after processing the data corresponding to each batch, the occupation of computing resources is avoided, thereby improving the utilization of computing resources.

[0142] Among them, deleting intermediate variables and caches can be achieved through the following code:

[0143] # Clear intermediate variables

[0144] del encoded_x,global_representations,model

[0145] #Release the video memory occupied by TensorFlow

[0146] tf.keras.backend.clear_session()

[0147] #Forced garbage collection

[0148] gc.collect()

[0149] The above code can be executed by the intelligent computing center's memory optimization module, which manages the center's memory resources. After processing each batch of data, the module deletes intermediate variables and caches, avoiding the use of computing resources and improving their utilization.

[0150] In some implementations, the batch size parameter, data sequence length parameter, and inference accuracy are updated to improve computing resource utilization, as shown in the following table:

[0151]

[0152]

[0153] The table shows the computing resource utilization after updating the batch size parameter. Compared to the traditional method (which resulted in memory overflow when the batch size parameter was 500), processing efficiency increased by 4.2 times, computing resource utilization reached 81.5%, and the cosine similarity between the output result and the full-precision (FP32) prediction was ≥99.7%. This demonstrates that this method can effectively improve computing resource utilization.

[0154] See Figure 2 , Figure 2 This is a structural diagram of an inference acceleration device based on dynamic adjustment of computing resources on an intelligent computing center cloud platform provided by an embodiment of the present invention. Figure 2 As shown, the inference acceleration device 200 of the intelligent computing center cloud platform based on dynamic adjustment of computing resources includes:

[0155] An acquisition module 201 is configured to acquire data reading parameters corresponding to each of a plurality of batches, wherein the data reading parameters include a data sequence length parameter and / or a batch size parameter, and the plurality of batches are the number of times the data file is read;

[0156] A reading module 202 is configured to sequentially read the data files of the first computing power running tasks based on the data reading parameters to obtain data corresponding to each batch;

[0157] The inference module 203 is used to sequentially infer the data corresponding to each batch to obtain multiple intermediate results;

[0158] The merging module 204 is configured to merge the multiple intermediate results to obtain an inference result.

[0159] In one embodiment, the acquisition module 201 includes:

[0160] an acquiring unit, configured to, when data reading parameters corresponding to the Nth batch need to be acquired, acquire a GPU utilization rate and a video memory utilization rate corresponding to a first GPU when inference is performed on data corresponding to the N-1th batch, where the first GPU is a GPU that executes the first computing power operation task, and N is a positive integer greater than 1;

[0161] A first calculation unit is configured to calculate the data sequence length parameter based on a GPU utilization rate corresponding to the first GPU;

[0162] The second computing unit is configured to calculate the batch size parameter based on a video memory utilization rate corresponding to the first GPU and the data sequence length parameter.

[0163] In one embodiment, the video memory utilization includes available video memory utilization corresponding to the first GPU, a first occupied video memory utilization, and a second occupied video memory utilization, wherein the first occupied video memory utilization is used to characterize the occupancy of model parameters in the first computing power running task, and the second occupied video memory utilization is used to characterize the system occupied video memory;

[0164] The second calculation unit includes:

[0165] an acquisition subunit, configured to acquire the available video memory utilization, the first occupied video memory utilization, the second occupied video memory utilization, and the inference accuracy, as well as the data embedding dimension and the number of attention heads of the task executed by the first computing power;

[0166] a first calculation subunit, configured to subtract the first occupied video memory utilization and the second occupied video memory utilization from the available video memory utilization to obtain an intermediate occupied video memory utilization;

[0167] a second computing subunit, configured to calculate a first sum value, where the first sum value is the sum of a first product and a second product, where the first product is the product of the data sequence length parameter, the number of attention heads, and the inference accuracy, and the second product is the product of the data sequence length parameter, the data embedding dimension, and the inference accuracy;

[0168] The third calculation subunit is configured to set a quotient of the intermediate occupied video memory utilization and the first sum value as the batch size parameter.

[0169] In one embodiment, the inference precision is a half-precision floating point number.

[0170] In one embodiment, the acquisition module 201 includes:

[0171] The first setting unit is used to set the preset length parameter to the data sequence length parameter and / or set the preset size parameter to the batch size parameter when it is necessary to obtain the data reading parameter corresponding to the first batch.

[0172] In one embodiment, the reasoning module 203 includes:

[0173] a second setting unit, configured to set an input dimension parameter based on the data sequence length parameter corresponding to a first batch, where the first batch is one of the multiple batches;

[0174] An inference unit is used to infer the first batch of data based on input dimension parameters through a preset inference model to obtain a first intermediate result, wherein the preset inference model is a model trained to infer data, and the first intermediate result is one of the multiple intermediate results.

[0175] In one embodiment, the reasoning module 203 further includes:

[0176] A deletion unit is used to delete the intermediate variables and cache generated during the reasoning process of the first batch of data after obtaining the first intermediate result and before reasoning on the data corresponding to the second batch, where the second batch is a batch after the first batch.

[0177] The inference acceleration device based on dynamic adjustment of computing power resources for the intelligent computing center cloud platform provided in the embodiment of the present invention can realize the various processes of each embodiment of the inference acceleration method based on dynamic adjustment of computing power resources for the above-mentioned intelligent computing center cloud platform. The technical features correspond one to one and can achieve the same technical effects. To avoid repetition, they will not be described here.

[0178] It should be noted that the inference acceleration device of the intelligent computing center cloud platform of the present invention based on dynamic adjustment of computing resources can be a device, or a component, integrated circuit, or chip in an electronic device.

[0179] The present invention also provides an electronic device, see Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided by the present invention. The electronic device includes a memory 301, a processor 302, and a program or instruction stored in the memory 301. When the program or instruction is executed by the processor 302, Figure 1 Any steps in the corresponding embodiment of the inference acceleration method based on dynamic adjustment of computing resources of the intelligent computing center cloud platform and the same beneficial effects are achieved will not be repeated here.

[0180] The processor 302 may be a CPU, an ASIC, an FPGA or a GPU.

[0181] Those skilled in the art will understand that all or part of the steps of the embodiment of the inference acceleration method for implementing the above-mentioned intelligent computing center cloud platform based on dynamic adjustment of computing resources can be completed through hardware related to program instructions, and the program can be stored in a readable medium.

[0182] The present invention also provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above Figure 1 The corresponding steps in the embodiment of the inference acceleration method based on dynamic adjustment of computing resources of the intelligent computing center cloud platform can achieve the same technical effect. To avoid repetition, they are not repeated here. The storage medium is such as read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0183] The present invention also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the above Figure 1 The corresponding intelligent computing center cloud platform has various processes of the inference acceleration method embodiment based on dynamic adjustment of computing resources, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0184] The terms "first", "second" and the like in the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. In addition, the terms "comprise" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or devices. In addition, "and / or" is used in this application to represent at least one of the connected objects, for example A and / or B and / or C, which means comprising seven situations including single A, single B, single C, and both A and B exist, both B and C exist, both A and C exist, and both A, B and C exist.

[0185] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0186] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of each embodiment of the present application.

[0187] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An inference acceleration method based on dynamic adjustment of computing resources in an intelligent computing center cloud platform, characterized in that: include: Step S1, obtaining data reading parameters corresponding to each batch in a plurality of batches, wherein the data reading parameters include a data sequence length parameter and / or a batch size parameter, and the plurality of batches is the number of times the data file is read; Step S2: Read the data files of the first computing power running tasks in sequence based on the data reading parameters to obtain data corresponding to each batch; Step S3: inferring the data corresponding to each batch in turn to obtain multiple intermediate results; Step S4: Merge the multiple intermediate results to obtain an inference result.

2. The method according to claim 1, wherein The step S1 comprises: Step S11: When it is necessary to obtain data reading parameters corresponding to the Nth batch, obtain the GPU utilization and video memory utilization corresponding to the first GPU when performing inference on the data corresponding to the N-1th batch, where the first GPU is the GPU that executes the first computing power operation task, and N is a positive integer greater than 1; Step S12: Calculate the data sequence length parameter based on the GPU utilization rate corresponding to the first GPU; Step S13: Calculate the batch size parameter based on the video memory utilization corresponding to the first GPU and the data sequence length parameter.

3. The method according to claim 2, wherein The video memory utilization includes the available video memory utilization corresponding to the first GPU, a first occupied video memory utilization, and a second occupied video memory utilization, wherein the first occupied video memory utilization is used to characterize the occupancy of model parameters in the first computing power running task, and the second occupied video memory utilization is used to characterize the system occupied video memory; The step S13 includes: Step S131: Obtain the available video memory utilization, the first occupied video memory utilization, the second occupied video memory utilization and inference accuracy, as well as the data embedding dimension and the number of attention heads of the task running with the first computing power; Step S132: Subtract the first occupied video memory utilization and the second occupied video memory utilization from the available video memory utilization to obtain an intermediate occupied video memory utilization; Step S133: Calculate a first sum value, where the first sum value is the sum of a first product and a second product, where the first product is the product of the data sequence length parameter, the number of attention heads, and the inference accuracy, and the second product is the product of the data sequence length parameter, the data embedding dimension, and the inference accuracy; Step S134: Set the quotient of the intermediate occupied video memory utilization and the first sum value as the batch size parameter.

4. The method according to claim 3, wherein The inference precision is half-precision floating point number.

5. The method according to claim 1, wherein The step S1 comprises: Step S11 ′: when it is necessary to obtain the data reading parameters corresponding to the first batch, the preset length parameter is set as the data sequence length parameter, and / or the preset size parameter is set as the batch size parameter.

6. The method according to any one of claims 1 to 5, characterized in that The step S3 comprises: Step S31: setting an input dimension parameter based on the data sequence length parameter corresponding to the first batch, where the first batch is one of the multiple batches; Step S32: Infer the first batch of data based on input dimension parameters through a preset inference model to obtain a first intermediate result, where the preset inference model is a model trained to infer data, and the first intermediate result is one of the multiple intermediate results.

7. The method according to claim 6, wherein After step S32, the method further includes: Step S33: After obtaining the first intermediate result and before inferring the data corresponding to the second batch, delete the intermediate variables and cache generated during the inference process of the first batch of data, and the second batch is a batch after the first batch.

8. An inference acceleration device based on dynamic adjustment of computing resources on an intelligent computing center cloud platform, characterized in that: include: an acquisition module, configured to acquire data reading parameters corresponding to each of a plurality of batches, wherein the data reading parameters include a data sequence length parameter and / or a batch size parameter, and the plurality of batches are the number of times the data file is read; A reading module, configured to sequentially read the data files of the first computing power running tasks based on the data reading parameters to obtain data corresponding to each batch; An inference module is used to sequentially infer the data corresponding to each batch to obtain multiple intermediate results; A merging module is used to merge the multiple intermediate results to obtain an inference result.

9. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the inference acceleration method based on dynamic adjustment of computing resources of an intelligent computing center cloud platform as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the inference acceleration method based on dynamic adjustment of computing power resources of an intelligent computing center cloud platform as described in any one of claims 1 to 7.

11. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the inference acceleration method based on dynamic adjustment of computing power resources of an intelligent computing center cloud platform as described in any one of claims 1 to 7.

Images