Method and device for realizing model training through computing power of intelligent computing center

Abstract

The invention provides a method and a device for realizing model training through computing power of an intelligent computing center, and relates to the technical field of computing power infrastructure, the method comprises the following steps: asynchronously filling a first buffer area with first training data obtained by preprocessing original data; under the condition that the second training data in the second buffer area is filled into a training thread, at least part of the first training data in the first buffer area is filled into the second buffer area, the training thread is a thread used for conducting model training on the training data in the GPU, and the capacity of the first buffer area is larger than that of the second buffer area; the second training data is training data of a batch before the first training data. The bottleneck of the preprocessing process caused by capacity limitation of the second buffer area is eliminated, the preprocessing process and the subsequent training process are ensured to be continuously carried out in parallel, the time for waiting for data loading in training is remarkably shortened, and the computing power resource utilization rate and the model training efficiency of an intelligent computing center are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent computing centers, smart computing centers and computing power infrastructure, and specifically to a method and device for implementing model training through the computing power of an intelligent computing center. Background Art

[0002] With the rapid development of artificial intelligence technology, "intelligent computing centers" and "intelligent computing centers" have emerged.

[0003] An "Intelligent Computing Center" is a facility that uses large-scale heterogeneous computing resources, including general-purpose and intelligent computing power, to provide the computing power, data, and algorithms required for AI applications (such as AI deep learning model development, model training, and model inference). The Intelligent Computing Center encompasses facilities, hardware, and software, and provides a full stack of capabilities, from bottom-level computing power to top-level application enablement.

[0004] “Intelligent Computing Center” includes but is not limited to “Smart Computing Center”.

[0005] "Intelligent Computing Center" refers to an artificial intelligence computing center. It is a type of computing power infrastructure that is based on artificial intelligence theory, adopts artificial intelligence computing architecture, and provides computing power services, data services, and algorithm services required for artificial intelligence applications.

[0006] "Computing power" is the core of "intelligent computing center" and "intelligent computing center". It is the ability of computer equipment or computing / data center to process information. It is the ability of computer hardware and software to work together to perform certain computing needs. It is the computing power to achieve target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity. It mainly provides services to society through computing power infrastructure.

[0007] During the training of deep learning models, efficient utilization of computing resources remains a key challenge in improving training efficiency and reducing costs. Currently, while data transmission efficiency can be improved to a certain extent through methods such as memory caching, data preprocessing (such as image decoding, cropping, and normalization) still relies on the CPU, and data transmission between the CPU and GPU has high latency. This leads to the problem of training waiting for data to load, resulting in wasted computing resources.

[0008] It can be seen that since the emergence of intelligent computing centers, how to reduce the time waiting for data loading during training to improve the utilization of computing resources in intelligent computing centers has become an urgent problem that needs to be solved. Summary of the Invention

[0009] The embodiments of the present invention provide a method and device for implementing model training through the computing power of an intelligent computing center to solve the problem of how to reduce the time waiting for data loading during training and improve the computing power resource utilization of the intelligent computing center since the emergence of the intelligent computing center.

[0010] To solve the above problems, the present invention is achieved as follows:

[0011] In a first aspect, an embodiment of the present invention provides a method for implementing model training by using the computing power of an intelligent computing center, including:

[0012] Step S1: asynchronously filling first training data obtained by preprocessing the original data into a first buffer;

[0013] Step S2: When the second training data in the second buffer is filled into the training thread, at least part of the first training data in the first buffer is filled into the second buffer. The training thread is a thread in the GPU used to perform model training on the training data. The capacity of the first buffer is greater than the capacity of the second buffer. The second training data is the training data of the batch before the first training data.

[0014] In one embodiment, before step S1, the method further includes:

[0015] Step S3: preprocess the original data through DALIPipeline to obtain multiple batches of training data, where the multiple batches of training data include the first training data and the second training data. The preprocessing includes decoding processing, image size processing, and label processing, and the preprocessing is performed in the GPU.

[0016] In one embodiment, step S1 includes:

[0017] Step S11: creating a double-buffer queue in the first buffer, the double-buffer queue including a producer thread and a consumer thread, the producer thread being used to obtain the multiple batches of training data from the DALIPipeline based on a time sequence, and the consumer thread being used to fill at least part of the multiple batches of training data into the second buffer, the producer thread and the consumer thread being parallel processing threads;

[0018] Step S12: When the producer thread obtains the first training data in the multiple batches of training data, the first training data is buffered in a queue corresponding to the consumer thread.

[0019] In one embodiment, before step S2, the method further includes:

[0020] Step S4: monitoring the status information of the training thread in real time, where the status information is used to indicate the training progress of the GPU;

[0021] Step S5: When the status information indicates that the GPU has completed training of the third training data, fill the second training data in the second buffer into the training thread, where the third training data is training data of a batch before the second training data.

[0022] In one embodiment, step S2 includes:

[0023] When the second training data in the second buffer is filled into the training thread, and when the capacity corresponding to the first training data is greater than the remaining capacity of the second buffer, first sub-training data in the first buffer is filled into the second buffer, where the capacity corresponding to the first sub-training data is less than or equal to the remaining capacity of the second buffer, and the first sub-training data is part of the first training data;

[0024] When the second training data in the second buffer is filled into the training thread, and when the capacity corresponding to the first training data is less than or equal to the remaining capacity in the second buffer, the first training data in the first buffer is filled into the second buffer.

[0025] In one embodiment, the first buffer is a memory buffer, and the second buffer is a video memory buffer.

[0026] In a second aspect, an embodiment of the present invention further provides a device for implementing model training through the computing power of an intelligent computing center, comprising:

[0027] A first filling module is used to asynchronously fill the first training data obtained after preprocessing the original data into the first buffer;

[0028] The second filling module is used to fill at least part of the first training data in the first buffer into the second buffer when the second training data in the second buffer is filled into the training thread. The training thread is a thread in the GPU used to perform model training on the training data. The capacity of the first buffer is greater than the capacity of the second buffer. The second training data is the training data of the batch before the first training data.

[0029] In a third aspect, the present invention also provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the model training method implemented by the computing power of an intelligent computing center as described in the first aspect above are implemented.

[0030] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the model training method implemented by the computing power of an intelligent computing center as described in the first aspect above.

[0031] In a fifth aspect, the present invention also provides a computer program product comprising computer instructions, which, when executed by a processor, implement the steps in the model training method using the computing power of an intelligent computing center as described in the first aspect above.

[0032] In an embodiment of the present invention, first, the first training data after preprocessing of the original data is asynchronously filled into the first buffer, and the preprocessing and training processes are processed in parallel, thereby reducing the situation where the GPU waits for preprocessing, resulting in a long training data loading time; then, when the second training data in the second buffer is used for training, at least part of the first training data in the first buffer is filled into the second buffer, and the large capacity of the first buffer is utilized to ensure that there is a continuous supply of training data in the second buffer, eliminating the bottleneck of the preprocessing process caused by the capacity limitation of the second buffer, thereby ensuring that the preprocessing process and the subsequent training process are continuously carried out in parallel, significantly shortening the time for training waiting for data loading, and improving the computing resource utilization rate and model training efficiency of the intelligent computing center. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0034] Figure 1 This is one of the flow charts of a method for implementing model training by using the computing power of an intelligent computing center, provided by an embodiment of the present invention;

[0035] Figure 2 This is the second flowchart of a method for implementing model training by using the computing power of an intelligent computing center, provided by an embodiment of the present invention;

[0036] Figure 3 This is a structural diagram of a device for implementing model training through the computing power of an intelligent computing center, provided by an embodiment of the present invention;

[0037] Figure 4 This is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0039] The "computing power" mentioned in the present invention refers to: the ability of computer equipment or computing / data centers to process information, the ability of computer hardware and software to work together to execute certain computing requirements, and the computing power to achieve target result output by processing information data. It is a new type of productivity that integrates information computing power, network carrying capacity, and data storage capacity, and mainly provides services to society through computing power infrastructure.

[0040] The "computing power" (CP) mentioned in the present invention refers to: the ability of a data center server to process data and output results. It is a comprehensive indicator to measure the computing power of a data center, including general computing power, super computing power and intelligent computing power. The commonly used unit of measurement is the number of floating-point operations performed per second (FLOPS, 1EFLOPS=10^18FLOPS). The larger the value, the stronger the comprehensive computing power. According to calculations, 1EFLOPS is approximately the computing power output of 5 Tianhe-2A or 500,000 mainstream server CPUs or 2 million mainstream notebooks. The calculation formula is: CP=CP 通用 +CP 智能 +CP 超级 .

[0041] The "carrying capacity" (Network Power, NP) mentioned in the present invention refers to: it is the performance of the data transmission capability of the computing power facility, which includes comprehensive capabilities such as network architecture, network bandwidth, transmission latency, intelligent management and scheduling, etc. It involves network transmission within and between data centers, and is a comprehensive indicator for measuring network transmission scheduling capabilities.

[0042] The "Storage Power" (SP) described in this invention refers to the comprehensive capabilities of a data center in terms of data storage capacity, performance, security and reliability, and environmental friendliness. It is a comprehensive indicator for measuring a data center's data storage capacity, encompassing both external storage devices such as storage arrays and internal server storage. Storage capacity is commonly measured in exabytes (EB, 1EB = 2^60 bytes), while performance is commonly measured in IOPS / TB (Input / Output Operations Per Second / TB). Disaster recovery ratio is a key indicator of security and reliability.

[0043] The "computing power infrastructure" mentioned in the present invention refers to a new type of information infrastructure that integrates information computing power, network carrying capacity, and data storage capacity, and can realize the centralized calculation, storage, transmission and application of information.

[0044] The "new information infrastructure" mentioned in the present invention refers to: mainly including network infrastructure such as 5G networks, fiber-optic broadband networks, backbone networks, international communication networks, satellite Internet, computing power infrastructure such as data centers, general computing power centers, intelligent computing centers, supercomputing centers, and new technology facilities such as artificial intelligence, blockchain, and quantum computing.

[0045] The "computing power" mentioned in the present invention includes: general computing power, intelligent computing power and super computing power.

[0046] The "general computing power" mentioned in the present invention refers to the computing power provided by servers based on CPU (Central Processing Unit) chips, which is used to support basic general computing such as cloud computing and edge computing.

[0047] The "intelligent computing power" mentioned in this invention refers to: a computing platform based on specialized chips such as GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), and ASIC (Application Specific Integrated Circuit) for various innovative artificial intelligence applications, such as natural language processing and machine vision.

[0048] The "supercomputing power" mentioned in the present invention refers to the computing power provided by high-performance computing clusters such as supercomputers. It utilizes the centralized computing resources of multiple computer systems working in parallel and uses a dedicated operating system to handle extremely complex or data-intensive problems. It is mainly used for calculations in cutting-edge scientific fields, such as planetary simulation, drug molecule design, genetic analysis, etc.

[0049] The "intelligent computing center" described in this article refers to a facility that provides the computing power, data, and algorithms required for artificial intelligence applications (such as AI deep learning model development, model training, and model inference) by utilizing large-scale heterogeneous computing resources, including general-purpose computing power (CPU) and intelligent computing power (GPU, FPGA, ASIC, etc.). The intelligent computing center encompasses facilities, hardware, and software, and can provide a full stack of capabilities, from bottom-level computing power to top-level application enablement.

[0050] The "intelligent computing center" mentioned in the present invention includes but is not limited to the "intelligent computing center".

[0051] The "intelligent computing center" mentioned in the present invention is an artificial intelligence computing center, which is a type of computing power infrastructure based on artificial intelligence theory, adopts artificial intelligence computing architecture, and provides computing power services, data services and algorithm services required for artificial intelligence applications.

[0052] The "computing power center" mentioned in the present invention refers to: a facility that is mainly composed of infrastructure such as wind, fire, water, electricity, and IT hardware and software equipment, and has computing power, transportation capacity, and storage capacity, including general data centers, intelligent computing centers, supercomputing centers, etc.

[0053] The "supercomputing center" mentioned in the present invention refers to: a supercomputing data center, which is a data center based on a supercomputer or a large-scale computing cluster, which can provide large-scale computing, storage and network services and other functions, and is widely used in application scenarios such as aerospace, national defense, oil exploration, climate modeling and genome sequencing.

[0054] The "computing resources" mentioned in the present invention refer to: technologies and facilities with information calculation, transmission, storage and application capabilities required for the development of a digital society, including but not limited to computing resources such as CPUs and GPUs, network resources such as switches and routers, storage resources such as storage arrays and distributed storage, security resources such as firewalls and intrusion detection systems, and supporting and guarantee resources such as wind, fire, water, and electricity.

[0055] The “model” mentioned in the present invention includes but is not limited to a “large language model” and a “multimodal large model”.

[0056] The "large language model" mentioned in the present invention refers to a large language model (LLM), which is a language model with a large parameter scale. It is designed to understand and generate human language. It is trained with a large amount of text data and can perform a wide range of tasks including text summarization, translation, sentiment analysis, etc.

[0057] The "Multimodal Large Models" mentioned in the present invention refer to models that combine multimodal information such as text, images, video, and audio for training, including but not limited to multimodal large language models.

[0058] See Figure 1 , Figure 1 This is one of the flow charts of a method for implementing model training through the computing power of an intelligent computing center provided by an embodiment of the present invention. Figure 1 As shown, the following steps are included:

[0059] Step S1: asynchronously filling first training data obtained by preprocessing the original data into a first buffer;

[0060] In this step, the raw data is preprocessed to obtain preprocessed first training data; then the first training data is filled into the first buffer. The first buffer serves as a temporary storage area and can be used to store data that has been preprocessed but has not yet entered the training stage. In the process of preprocessing the raw data, it can be executed asynchronously, that is, in a background thread, without blocking the operation of other threads. This allows data preprocessing and subsequent training steps to be performed in parallel, and the first training data obtained after preprocessing is cached in the first buffer. In the subsequent training process, the first training data can be read directly from the first buffer, reducing the situation where the training data loading time is too long due to waiting for preprocessing, thereby shortening the idle time of computing resources and improving the utilization rate of computing resources.

[0061] Step S2: When the second training data in the second buffer is filled into the training thread, at least part of the first training data in the first buffer is filled into the second buffer. The training thread is a thread in the GPU used to perform model training on the training data. The capacity of the first buffer is greater than the capacity of the second buffer. The second training data is the training data of the batch before the first training data.

[0062] In this step, when the second training data in the second buffer (i.e., the training data of the previous batch) is filled into the training thread, the model is used to train according to the second training data; at this time, at least part of the first training data in the first buffer (i.e., the data of the current batch) can be filled into the second buffer, so that the model can be trained according to the first training data later. By setting the second buffer before the second buffer and enabling the first buffer to store more preprocessed training data, a continuous data supply is provided for the second buffer. In this way, filling the preprocessed training data into the first buffer reduces the situation where the preprocessing process is slowed down or stopped due to the capacity limitation of the second buffer, thereby ensuring that the preprocessing process and the subsequent training process are continuously carried out in parallel, thereby shortening the idle time of computing resources and improving the utilization rate and training efficiency of the GPU.

[0063] In one embodiment, the first buffer is a memory buffer, and the second buffer is a video memory buffer.

[0064] For example, Figure 2As shown, the first buffer can be a memory buffer, which is used as a temporary storage area for the raw data after DALIPipeline preprocesses it. The first training data is asynchronously cached by taking advantage of the large memory capacity and efficient access, so that the preprocessing process and the subsequent training process can be carried out in parallel; the second buffer can be a video memory buffer, which takes advantage of the high bandwidth of the video memory and the direct access of the GPU. During training, the training data in the second buffer is continuously supplied to the GPU training thread, while new training data is obtained from the first buffer, avoiding preprocessing interruptions caused by insufficient video memory. By setting up a memory buffer and a video memory buffer, the memory buffer is used to cache the preprocessed training data, and the video memory buffer directly serves the training thread for subsequent model calculations, thereby maximizing the utilization of computing resources. In addition, it avoids the situation where the cost of model training increases sharply due to the increase in video memory.

[0065] In an embodiment of the present application, the first training data after preprocessing of the original data is first asynchronously filled into the first buffer, and the preprocessing and training processes are processed in parallel, thereby reducing the situation where the GPU waits for preprocessing and causes the training data loading time to be too long; then, when the second training data in the second buffer is used for training, at least part of the first training data in the first buffer is filled into the second buffer, and the large capacity of the first buffer is utilized to ensure that there is a continuous supply of training data in the second buffer, eliminating the bottleneck of the preprocessing process caused by the capacity limitation of the second buffer, thereby ensuring that the preprocessing process and the subsequent training process are continuously carried out in parallel, significantly shortening the time for training waiting for data loading, and improving the computing resource utilization rate and model training efficiency of the intelligent computing center.

[0066] In addition, it avoids the increase in model training costs caused by increasing video memory, and has better economic efficiency.

[0067] In one embodiment, before step S1, the method further includes:

[0068] Step S3: preprocess the original data through DALIPipeline to obtain multiple batches of training data, where the multiple batches of training data include the first training data and the second training data. The preprocessing includes decoding processing, image size processing, and label processing, and the preprocessing is performed in the GPU.

[0069] In this embodiment, the NVIDIA Data Loading Library Pipeline (DALIPipeline) can be used to perform GPU accelerated preprocessing of the raw data. DALIPipeline is a high-performance data processing pipeline developed by NVIDIA, which is designed for deep learning training and reasoning optimization. DALIPipeline improves the processing speed by migrating data preprocessing tasks (such as image decoding, scaling, and normalization) performed by the CPU to the GPU, utilizing the parallel computing capabilities of the GPU. From reading the raw data from the disk to outputting the preprocessed training data, the entire process is completed in the GPU, avoiding the data transmission overhead between the CPU and the GPU. Among them, preprocessing can include decoding processing (i.e., decoding and data loading), image size processing, label processing, and normalization and half-precision conversion.

[0070] For example, the decoding process can use NVIDIA DALI's mixed-mode decoding (device="mixed") to convert compressed images (such as JPEG / PNG) in the raw data into RGB format. The GPU performs the computationally intensive decoding operations in parallel, fully utilizing the GPU's thousands of cores to accelerate processing. The output RGB image can maintain the original bit depth (typically 8 bits per channel), providing a standard input format for subsequent processing.

[0071] Image resizing can be performed by resizing all images in the original data to a uniform 224×224 pixel size using bilinear interpolation (types.INTERP_LINEAR). Linear interpolation strikes a balance between preserving image detail and computational efficiency, making it particularly suitable for batch processing of large datasets. This ensures that all input images meet the model's expected size. GPU parallel computing reduces the time-consuming resizing operation to milliseconds, providing standardized input for subsequent batch training.

[0072] Label processing involves processing the label data corresponding to an image. First, labels.gpu() can be used to migrate the labels from CPU memory to GPU memory, ensuring they reside on the same device as the image data and eliminating cross-device transmission latency. The label data type is then converted to INT64, conforming to the classification label format requirements of mainstream deep learning frameworks such as PyTorch. Label processing can be performed efficiently on the GPU, concurrently with image preprocessing, ultimately outputting image-label pairs that can be directly used for model training, seamlessly integrating data loading and model computation.

[0073] During normalization and half-precision conversion, the crop_mirror_normalize function first normalizes image pixel values from the range [0, 255] to a standard normal distribution (using the mean and standard deviation pre-calculated by ImageNet), enabling deep learning model convergence. Secondly, the data type is converted to 16-bit floating-point numbers (FLOAT16), saving 50% of video memory space compared to traditional 32-bit floating-point numbers while leveraging the Tensor Cores of modern GPUs to accelerate computation. Finally, all operations are performed within GPU memory, preventing data from flowing back to the CPU and achieving end-to-end GPU optimization.

[0074] This allows preprocessing and training to run in parallel. While the GPU is training, the DALIPipeline is processing the next batch of data simultaneously. DALIPipeline preprocesses the raw data, providing an efficient and continuous data flow for the subsequent double buffering mechanism.

[0075] In one embodiment, step S1 includes:

[0076] Step S11: creating a double-buffer queue in the first buffer, the double-buffer queue including a producer thread and a consumer thread, the producer thread being used to obtain the multiple batches of training data from the DALIPipeline based on a time sequence, and the consumer thread being used to fill at least part of the multiple batches of training data into the second buffer, the producer thread and the consumer thread being parallel processing threads;

[0077] Step S12: When the producer thread obtains the first training data in the multiple batches of training data, the first training data is buffered in a queue corresponding to the consumer thread.

[0078] In this embodiment, a double-buffered queue consisting of a producer thread and a consumer thread is first created in the first buffer. The two threads execute in parallel to eliminate bottlenecks in the preprocessing process. The producer thread continuously obtains multiple batches of preprocessed training data from the DALIPipeline in chronological order, while the consumer thread fills at least some of these batches into the second buffer for GPU training. When the producer thread obtains the first training data, it buffers this first training data into the queue corresponding to the consumer thread, allowing the consumer thread to directly obtain data from the queue and fill it into the video memory for subsequent training processes, independent of subsequent operations of the producer thread. This achieves pipeline parallel processing of the three stages of preprocessing, memory buffering, and video memory filling, reducing the idle time of computing resources, thereby maximizing GPU utilization and reducing data loading latency.

[0079] In one embodiment, before step S2, the method further includes:

[0080] Step S4: monitoring the status information of the training thread in real time, where the status information is used to indicate the training progress of the GPU;

[0081] Step S5: When the status information indicates that the GPU has completed training of the third training data, fill the second training data in the second buffer into the training thread, where the third training data is training data of a batch before the second training data.

[0082] In this embodiment, the status information of the GPU training thread is monitored in real time, and the status information reflects the real-time training progress of the GPU. When the GPU completes training on the third training data (i.e., the previous batch of training data earlier than the second training data), the second training data prepared in the second buffer is immediately filled into the training thread, ensuring that the GPU seamlessly switches to training the next batch of data, avoiding idle computing due to untimely data supply. Through the state-driven scheduling strategy, the training rhythm can be accurately grasped, achieving efficient coordination between data preprocessing and model calculation, and further improving GPU utilization.

[0083] In one embodiment, step S2 includes:

[0084] When the second training data in the second buffer is filled into the training thread, and when the capacity corresponding to the first training data is greater than the remaining capacity of the second buffer, first sub-training data in the first buffer is filled into the second buffer, where the capacity corresponding to the first sub-training data is less than or equal to the remaining capacity of the second buffer, and the first sub-training data is part of the first training data;

[0085] When the second training data in the second buffer is filled into the training thread, and when the capacity corresponding to the first training data is less than or equal to the remaining capacity in the second buffer, the first training data in the first buffer is filled into the second buffer.

[0086] In this embodiment, when the second training data in the second buffer is being processed by the GPU training thread, the capacity of the first training data and the remaining space of the second buffer are compared in real time. If the capacity of the first training data exceeds the remaining space of the second buffer, only part of its data (i.e., the first sub-training data, the capacity of the first sub-training data ≤ the remaining capacity of the second buffer) is filled into the second buffer, and the remaining data is temporarily stored in the first buffer; if the capacity of the first training data does not exceed the remaining space, it is completely filled into the second buffer. The hierarchical filling mechanism adapts to the dynamic capacity of the video memory to ensure the continuity of data transmission, while preventing video memory overflow caused by loading too large a batch at one time. It is particularly suitable for memory optimization when processing high-resolution images or large-scale data sets.

[0087] See Figure 3 , Figure 3 This is a structural diagram of a model training device implemented by the computing power of an intelligent computing center, provided by an embodiment of the present invention. Figure 3 As shown, the model training device 300 implemented by the computing power of the intelligent computing center includes:

[0088] A first filling module 301 is configured to asynchronously fill a first buffer with first training data obtained by preprocessing the original data;

[0089] The second filling module 302 is used to fill at least part of the first training data in the first buffer into the second buffer when the second training data in the second buffer is filled into the training thread. The training thread is a thread in the GPU used to perform model training on the training data. The capacity of the first buffer is greater than the capacity of the second buffer, and the second training data is the training data of the batch before the first training data.

[0090] In one embodiment, the model training device 300 implemented by the computing power of the intelligent computing center further includes:

[0091] A preprocessing module is used to preprocess the raw data through DALIPipeline to obtain multiple batches of training data, where the multiple batches of training data include the first training data and the second training data. The preprocessing includes decoding processing, image size processing, and label processing. The preprocessing is performed in the GPU.

[0092] In one embodiment, the first filling module 301 is specifically configured to:

[0093] Creating a double-buffer queue in the first buffer, the double-buffer queue including a producer thread and a consumer thread, the producer thread being used to obtain the multiple batches of training data from the DALIPipeline based on a timing sequence, and the consumer thread being used to fill at least a portion of the multiple batches of training data into the second buffer, the producer thread and the consumer thread being parallel processing threads;

[0094] When the producer thread obtains the first training data in the multiple batches of training data, the first training data is buffered in a queue corresponding to the consumer thread.

[0095] In one embodiment, the model training device 300 implemented by the computing power of the intelligent computing center further includes:

[0096] A detection module, configured to monitor status information of the training thread in real time, wherein the status information is used to indicate the training progress of the GPU;

[0097] A third filling module is configured to fill the second training data in the second buffer into the training thread when the status information indicates that the GPU has completed training of the third training data, where the third training data is training data of a batch before the second training data.

[0098] In one embodiment, the second filling module 302 is specifically configured to:

[0099] When the second training data in the second buffer is filled into the training thread, and when the capacity corresponding to the first training data is greater than the remaining capacity of the second buffer, first sub-training data in the first buffer is filled into the second buffer, where the capacity corresponding to the first sub-training data is less than or equal to the remaining capacity of the second buffer, and the first sub-training data is part of the first training data;

[0100] When the second training data in the second buffer is filled into the training thread, and when the capacity corresponding to the first training data is less than or equal to the remaining capacity in the second buffer, the first training data in the first buffer is filled into the second buffer.

[0101] In one embodiment, the first buffer is a memory buffer, and the second buffer is a video memory buffer.

[0102] The model training device provided by the embodiment of the present invention through the computing power of the intelligent computing center is capable of realizing the various processes of the various embodiments of the above-mentioned model training method through the computing power of the intelligent computing center. The technical features correspond one to one and can achieve the same technical effects. To avoid repetition, they will not be described here.

[0103] It should be noted that the device for implementing model training through the computing power of an intelligent computing center in the embodiment of the present invention can be a device, or a component, integrated circuit, or chip in an electronic device.

[0104] The embodiment of the present invention further provides an electronic device, see Figure 4 , Figure 4 The electronic device includes a memory 401, a processor 402, and a program or instruction stored in the memory 401 and executed by the processor 402. Figure 1 Correspondingly, any step in the model training method embodiment is implemented through the computing power of the intelligent computing center and the same beneficial effects are achieved, which will not be repeated here.

[0105] The processor 402 may be a CPU, an ASIC, an FPGA, or a GPU.

[0106] Those skilled in the art will understand that all or part of the steps of the above-mentioned embodiment of the model training method using the computing power of an intelligent computing center can be completed through hardware related to program instructions, and the program can be stored in a readable medium.

[0107] The embodiment of the present invention further provides a readable storage medium, on which a computer program is stored, which can realize the above-mentioned Figure 1 Correspondingly, any step in the embodiment of the model training method is implemented by the computing power of the intelligent computing center, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The storage medium is such as read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0108] The present invention also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the above Figure 1 Correspondingly, each process of the embodiment of the model training method is realized through the computing power of the intelligent computing center, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0109] The terms "first", "second" etc. in the embodiments of the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or that are inherent to these processes, methods, products or equipment. In addition, "and / or" is used in this application to represent at least one of the connected objects, for example A and / or B and / or C, which means comprising 7 situations including single A, single B, single C, and both A and B exist, both B and C exist, both A and C exist, and both A, B and C exist.

[0110] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of each embodiment of the present application.

[0112] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for implementing model training by utilizing the computing power of an intelligent computing center, characterized in that: The method comprises: Step S1: asynchronously filling first training data obtained by preprocessing the original data into a first buffer; Step S2: When the second training data in the second buffer is filled into the training thread, at least part of the first training data in the first buffer is filled into the second buffer. The training thread is a thread in the GPU used to perform model training on the training data. The capacity of the first buffer is greater than the capacity of the second buffer. The second training data is the training data of the batch before the first training data.

2. The method according to claim 1, wherein Before step S1, the method further includes: Step S3: preprocess the original data through DALIPipeline to obtain multiple batches of training data, where the multiple batches of training data include the first training data and the second training data. The preprocessing includes decoding processing, image size processing, and label processing, and the preprocessing is performed in the GPU.

3. The method according to claim 2, wherein The step S1 includes: Step S11: creating a double-buffer queue in the first buffer, the double-buffer queue including a producer thread and a consumer thread, the producer thread being used to obtain the multiple batches of training data from the DALIPipeline based on a time sequence, and the consumer thread being used to fill at least part of the multiple batches of training data into the second buffer, the producer thread and the consumer thread being parallel processing threads; Step S12: When the producer thread obtains the first training data in the multiple batches of training data, the first training data is buffered in a queue corresponding to the consumer thread.

4. The method according to claim 1, wherein Before step S2, the method further includes: Step S4: monitoring the status information of the training thread in real time, where the status information is used to indicate the training progress of the GPU; Step S5: When the status information indicates that the GPU has completed training of the third training data, fill the second training data in the second buffer into the training thread, where the third training data is training data of a batch before the second training data.

5. The method according to claim 1, wherein The step S2 includes: When the second training data in the second buffer is filled into the training thread, and when the capacity corresponding to the first training data is greater than the remaining capacity of the second buffer, first sub-training data in the first buffer is filled into the second buffer, where the capacity corresponding to the first sub-training data is less than or equal to the remaining capacity of the second buffer, and the first sub-training data is part of the first training data; When the second training data in the second buffer is filled into the training thread, and when the capacity corresponding to the first training data is less than or equal to the remaining capacity in the second buffer, the first training data in the first buffer is filled into the second buffer.

6. The method according to any one of claims 1 to 5, characterized in that The first buffer is a memory buffer, and the second buffer is a video memory buffer.

7. A model training device using the computing power of an intelligent computing center, characterized in that: include: A first filling module is used to asynchronously fill the first training data obtained after preprocessing the original data into the first buffer; The second filling module is used to fill at least part of the first training data in the first buffer into the second buffer when the second training data in the second buffer is filled into the training thread. The training thread is a thread in the GPU used to perform model training on the training data. The capacity of the first buffer is greater than the capacity of the second buffer. The second training data is the training data of the batch before the first training data.

8. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of a method for implementing model training through the computing power of an intelligent computing center as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the model training method using the computing power of an intelligent computing center as described in any one of claims 1 to 6.

10. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the model training method using the computing power of an intelligent computing center as described in any one of claims 1 to 6.

Images