Data processing method, data processing device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method, a data processing device, electronic equipment and a storage medium. The data processing method comprises the steps of obtaining a current operation record according to user interface information and running state information; generating a data scheduling strategy according to the historical operation record and the current operation record; and executing a corresponding data transfer operation according to the data scheduling strategy. According to the data processing method, the data scheduling strategy is automatically generated by analyzing the training state and the operation record in real time, the data scheduling strategy can be dynamically adjusted and optimized in the training process, and plug-and-play convenience is achieved. Moreover, the data processing method is wide in application range and can be applied to any type of tensor. Therefore, powerful support is provided for training of a complex model, and the efficiency and the stability of the system are improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a data processing method, a data processing device, an electronic device, and a storage medium. Background Art

[0002] With the development of technology, artificial intelligence (AI) technology has been widely used in many fields. Deep learning is one of the important technologies of artificial intelligence technology. Deep learning technology based on artificial neural networks has made great progress in fields such as object classification, text processing, image search and human-computer dialogue.

[0003] As the complexity of problems increases, the depth and scale of neural networks continue to increase. However, the storage capacity of AI accelerators such as graphics processing units (GPUs), tensor processing units (TPUs), and field programmable gate arrays (FPGAs) is growing much slower than the scale of neural networks, making it difficult for them to meet the training needs of large-scale deep neural network models. Summary of the invention

[0004] At least one embodiment of the present disclosure provides a data processing method, which includes: obtaining a current operation record based on user interface information and operating status information; generating a data scheduling strategy based on historical operation records and the current operation record; and executing a corresponding data transfer operation according to the data scheduling strategy.

[0005] For example, in the data processing method provided in at least one embodiment of the present disclosure, the user interface information includes an interface name, the running status information includes a current running position, and obtaining the current operation record based on the user interface information and the running status information includes: combining the interface name and the current running position to obtain the current operation record.

[0006] For example, the data processing method provided by at least one embodiment of the present disclosure further includes: storing the current operation record in an operation record list; and reading the historical operation record from the operation record list.

[0007] For example, in the data processing method provided in at least one embodiment of the present disclosure, the data scheduling strategy includes task information, and the task information is used to generate tasks. The data scheduling strategy is generated according to the historical operation records and the current operation records, including: generating the task information corresponding to the current operation record according to the relative order of the current operation record in the historical operation records.

[0008] For example, in the data processing method provided by at least one embodiment of the present disclosure, executing the corresponding data transfer operation according to the data scheduling strategy includes: in response to the task information corresponding to an unloading task or a prefetching task, performing a validity check on the task information; in response to the task information passing the validity check, reading the task corresponding to the task information from the corresponding task queue and executing it, so as to implement the corresponding data transfer operation.

[0009] For example, the data processing method provided by at least one embodiment of the present disclosure also includes: generating a new task in response to executing the task, and setting the state of the new task based on the state of the task; and inserting the new task into the corresponding task queue.

[0010] For example, in the data processing method provided by at least one embodiment of the present disclosure, the state of the task includes a ready state, an unloaded state, a prefetched state or a completed state, and in response to executing the task, a new task is generated, and the state of the new task is set based on the state of the task, including: in response to executing an unloaded task and the state of the task is a ready state, a new task is generated, and the state of the new task is set to an unloaded state; in response to executing a prefetched task and the state of the task is an unloaded state, a new task is generated, and the state of the new task is set to a prefetched state; in response to the completion of the prefetched task and the state of the task is a prefetched state, a new task is generated, and the state of the new task is set to a completed state.

[0011] For example, the data processing method provided by at least one embodiment of the present disclosure further includes: in response to the user interface information corresponding to the task information indicating preparation for uninstallation, initializing the state of the task to a ready state.

[0012] For example, the data processing method provided by at least one embodiment of the present disclosure further includes: in response to determining whether the data object of the task is returned and the state of the task is a preparation state, updating the state of the task to a completion state.

[0013] For example, in the data processing method provided by at least one embodiment of the present disclosure, after generating the task information corresponding to the current operation record, the method further includes: in response to user interface information corresponding to the task information indicating preparation for uninstallation, inserting the task corresponding to the task information into a corresponding task queue.

[0014] For example, in the data processing method provided by at least one embodiment of the present disclosure, after generating the task information corresponding to the current operation record, the method further includes: in response to the user interface information corresponding to the task information indicating loading, performing a validity check on the task information; in response to the task information passing the validity check, reading the task corresponding to the task information from the corresponding task queue and determining whether to return the data object of the task.

[0015] For example, in the data processing method provided by at least one embodiment of the present disclosure, tasks in different states correspond to different task queues.

[0016] For example, in the data processing method provided by at least one embodiment of the present disclosure, the data transfer operation includes an unloading operation or a pre-fetching operation for a tensor performed between a central processing unit and a graphics processing unit.

[0017] At least one embodiment of the present disclosure provides a data processing device, which includes: an acquisition module, configured to acquire a current operation record based on user interface information and operating status information; a generation module, configured to generate a data scheduling strategy based on historical operation records and the current operation record; and an execution module, configured to execute a corresponding data transfer operation according to the data scheduling strategy.

[0018] At least one embodiment of the present disclosure provides an electronic device, comprising: at least one processor; at least one memory storing one or more computer program modules; wherein the one or more computer program modules are configured to be executed by the at least one processor to execute instructions for implementing the data processing method described in any embodiment of the present disclosure.

[0019] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by at least one processor, perform the data processing method described in any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0021] Figure 1A is a schematic diagram of a data parallel mode;

[0022] Figure 1B A schematic diagram of a tensor parallel mode;

[0023] Figure 1C A schematic diagram of a pipeline parallel mode;

[0024] Figure 1D A schematic diagram of a micro-batch pipeline parallel mode;

[0025] Figure 1E A schematic diagram of cross-executing forward calculation and backward calculation of micro-batch data;

[0026] Figure 1F A schematic diagram of a recalculation;

[0027] Figure 2 A schematic block diagram of a distributed training system provided for at least one embodiment of the present disclosure;

[0028] Figure 3 A flowchart of a data processing method provided for at least one embodiment of the present disclosure;

[0029] Figure 4A An exemplary schematic diagram of a data processing method provided for at least one embodiment of the present disclosure;

[0030] Figure 4B An exemplary schematic diagram of a data processing method provided for at least one embodiment of the present disclosure;

[0031] Figure 5A An exemplary schematic diagram of a data processing method provided for at least one embodiment of the present disclosure;

[0032] Figure 5B A state machine conversion schematic diagram provided for at least one embodiment of the present disclosure;

[0033] Fig. 6A A schematic block diagram of a data processing device provided for at least one embodiment of the present disclosure;

[0034] Figure 6B A schematic block diagram of a data processing device provided for at least one embodiment of the present disclosure;

[0035] Figure 7 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure;

[0036] Figure 8 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure; and

[0037] Fig. 9 A schematic block diagram of a non-transitory computer-readable storage medium is provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0039] Flowcharts are used in this disclosure to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0040] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0041] The present disclosure is described below through several specific embodiments. In order to keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components (elements) may be omitted. When any component (component) of the embodiments of the present disclosure appears in more than one drawing, the component (component) is represented by the same or similar reference numeral in each drawing.

[0042] There are many types of processors used for model training, such as graphics processing units (GPUs), general-purpose graphics processing units (GPGPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), artificial intelligence (AI) accelerators or coprocessors, etc. Currently, GPUs are usually used as the main training device, combined with central processing units (CPUs) for auxiliary tasks and overall process management. For specific scenarios or needs, ASICs, FPGAs, or AI accelerators may be selected for customized acceleration.

[0043] Memory is the main memory in a computing system, which usually has high bandwidth, low latency, and fast access speed. The capacity of memory is relatively large. For example, the memory capacity of a server can reach hundreds of GB (gigabytes) or even TB (terabytes). As the dedicated memory of GPU, video memory has higher bandwidth, but the storage capacity is usually small. For example, the video memory capacity of a GPU is usually between a dozen GB (gigabytes) and dozens of GB, such as 16GB, 32GB, etc. The storage capacity of other accelerators besides GPU is also relatively small. For example, the high bandwidth memory (HBM) of a TPU can be 16GB, 32GB, 64GB or 128GB, etc. For the convenience of description, the main memory with large storage capacity is called "host memory" here, and the video memory of GPU or the small capacity memory of other accelerators is called "device memory".

[0044] Since a large-scale model may have hundreds of billions of parameters, a single machine and a single GPU are no longer capable of training such large-scale models, so distributed training is widely used for large-scale model training in artificial intelligence. Distributed training refers to the use of multiple machines to work together to speed up the training process of large deep learning models. In distributed training, the original model, data set, and training process are decomposed and distributed to multiple machines for simultaneous processing, thereby effectively utilizing more computing resources and shortening training time.

[0045] In the distributed training of large models, common parallel technologies include data parallelism and model parallelism. Model parallelism includes tensor parallelism and pipeline parallelism.

[0046] The data parallel mode refers to dividing the model's training data set into multiple sub-data sets. The divided sub-data sets are distributed to multiple devices. Each device holds a complete copy of the model, so that the sub-data sets assigned to it can be trained independently. For example, each device can independently complete the forward propagation and back propagation calculations of the sub-data sets.

[0047] Figure 1A is a schematic diagram of a data parallel mode. Figure 1AAs shown in the figure, a training data set is divided into three parts and distributed to device 0, device 1 and device 2 respectively. Each of device 0, device 1 and device 2 has a complete model. After each device completes back propagation, the gradients calculated by each device need to be aggregated (All Reduce) and the global model parameters are updated.

[0048] Figure 1B This is a diagram of a tensor parallel mode. Tensor parallel mode refers to splitting a layer (or operator) of the model and placing part of the operator's weight on different devices, thereby reducing the memory usage on each device (for example, GPU). A neural network model consists of multiple layers ( Figure 1B ), each layer can be understood as a function that can perform specific mathematical operations on the input tensor. For example, the fully connected layer can perform linear transformation operations on the tensor, the convolutional layer can perform convolution operations on the tensor, and the pooling layer can perform pooling operations on the tensor to extract or transform features. These operations can change the content and shape of the tensor, realizing the gradual transformation from raw data to high-level abstract representations. Figure 1B As shown in the figure, in the tensor parallel mode, the tensor of a layer is divided into multiple parts, and the divided parts of the tensor are respectively distributed to multiple devices, such as device 0, device 1 and device 2. Each device is only responsible for a part of the model calculation and exchanges necessary intermediate results through communication.

[0049] The pipeline parallel mode is to divide the different layers of the model into multiple stages, and assign multiple stages to multiple devices to form a pipeline. The activation values ​​and gradients are passed sequentially between different devices. For example, during the forward propagation process, each device passes the activation value to the device where the next pipeline stage is located. During the reverse propagation process, each device passes the gradient back to the device where the previous pipeline stage is located. In this way, multiple devices can be used for calculation at the same time to improve the calculation efficiency.

[0050] Figure 1C is a schematic diagram of a pipeline parallel mode. Figure 1C As shown, a model includes 4 layers, which are divided into 3 stages, and each stage is assigned to a device. For example, the first layer is assigned to device 0, the second layer is assigned to device 1, and the third and fourth layers are assigned to device 2.

[0051] Figure 1D The figure is a schematic diagram of a micro-batch pipeline parallel mode. By dividing the global batch data into micro-batch data and artificially creating pipelines to solve the device idle problem, different devices are allowed to participate in the calculation process at the same time, which can significantly improve the utilization rate of pipeline parallel devices and reduce the idle time of devices.

[0052] like Figure 1D As shown, the multiple layers of the target model are split into 4 stages, each stage includes, for example, 4 mini-batches of data, F i,j represents the forward propagation of the jth micro-batch data in the i-th stage, B x,y represents the back propagation of the yth micro-batch data in the xth stage. In the forward calculation process, the calculation of the later stage depends on the calculation of the previous stage, for example, F 1,0 Must be in F 0,0 The calculation can only be started after the calculation is completed. Similarly, in the reverse calculation process, the calculation of the previous stage needs to depend on the calculation of the next stage, for example, B 2,3 Must be in B 3,3 Calculation can only start after the calculation is completed.

[0053] In pipeline parallel mode, data is transmitted between adjacent devices through communication links. For example, in the forward calculation process, the input data is first calculated on device 0 through the first layer to obtain the intermediate result, and the intermediate result is transmitted to device 1. Then the output of the second layer is calculated on device 1, and the output result of the second layer of the model is transmitted to device 2. The final result of the forward calculation is obtained through the calculation of the last layer on device 2. The reverse propagation process is similar. Finally, the network layer on each device will use the gradient calculated by the reverse propagation process to update the parameters.

[0054] exist Figure 1D In the pipeline parallel mode shown, the reverse calculation can only be started after all devices have completed the forward calculation, which means that the intermediate results of the forward calculation in the first stage need to be retained until all the forward calculations in all stages are completed before they can be released. This leads to a large number of pipeline bubbles, that is, idle cycles or gaps in the pipeline. Too many bubbles will lead to a decrease in processor performance, so reducing pipeline bubbles becomes a key goal for optimizing processor performance.

[0055] Figure 1E Schematic diagram of cross-execution of forward and reverse calculations of micro-batch data. Figure 1E As shown, the forward calculation and backward calculation of different micro-batch data are performed alternately in a cross-wise manner, which reduces pipeline bubbles to a certain extent.

[0056] like Figure 1E As shown in the figure, the earlier stages will save more activation values ​​of micro-batch data than the later stages. For example, stage 3 only needs to save the activation value of 1 micro-batch data, while stage 0 needs to save the activation values ​​of 4 micro-batch data, which leads to an imbalance in device memory between different stages.

[0057] Figure 1F is a schematic diagram of recalculation. Figure 1F As shown in the figure, since the back propagation needs to rely on the intermediate results generated in the forward propagation, the activation values ​​generated by the forward propagation need to be stored in the device memory all the time when recalculation is not used. By using recalculation, only the activation values ​​of several recovery points can be saved, and the activation values ​​of non-recovery points can be released, thereby reducing the demand for video memory. In the process of reverse calculation, the activation values ​​of the saved recovery points are used as input to perform the forward calculation again, and the activation values ​​required for the reverse calculation can be obtained. It can be seen that although recalculation reduces the amount of video memory occupied, it will increase the amount of additional calculations, resulting in an increase in calculation time.

[0058] As the model size increases, using only one parallel mode often cannot meet the requirements of both device memory limitations and high computing efficiency. Therefore, for large-scale models, it is usually necessary to combine multiple parallel technologies such as data parallelism, model parallelism, and pipeline parallelism for distributed training.

[0059] Since device memory resources are relatively precious, some data in the device memory can be transferred to the host memory first, and then transferred back to the device memory when the data is needed, thereby reducing the device memory usage. Specifically, the following operations are involved:

[0060] 1. Offload: The operation of transferring certain data from a device (such as an accelerator such as a GPU, GPGPU, or TPU) to a host (such as a CPU) during training. By offloading temporarily unneeded data, the accelerator memory space can be freed up to make room for other computing tasks. For example, when the model scale exceeds the GPU memory, by offloading part of the data to the CPU, larger-scale model training can be supported.

[0061] 2. Load: The operation of loading certain data from the host to the device. Through the load operation, it can be ensured that the device has the data required to perform the current computing task. For example, during the training process, some data may only be needed at a specific stage, and the load operation can be used to load it on demand, thereby effectively utilizing resources.

[0062] 3. Prefetch: Also known as preloading, it is an advance loading operation that aims to move data from the host to the device before it is actually used. For example, in the back propagation phase, prefetching can be performed asynchronously so that when a certain data is needed, it has already been loaded to the accelerator, reducing the waiting time for data loading.

[0063] It should be noted that loading is an operation that directly serves the current request, while prefetching is an operation taken in advance to improve the response speed of future requests.

[0064] In related technologies, some asynchronous offloading-based methods determine when to perform offloading and prefetching operations by analyzing factors such as memory, video memory, and operator performance. Specifically, it is necessary to analyze the distributed parallel strategy and model structure, use a video memory simulator to infer the amount of data to be offloaded, and use manual analysis and scheduling to optimize the data scheduling strategy in order to minimize performance overhead.

[0065] However, the inventors of the present disclosure have noticed that the above method does not provide an effective mechanism to automatically generate data scheduling strategies, nor can it achieve automatic adjustment and optimization of these strategies. The above method adopts a one-way command execution mode and does not fully consider the actual operating status of the uninstallation operation. Due to the lack of an effective monitoring and feedback mechanism for the execution status, once a command fails to execute, it may cause the entire system to crash. In order to avoid system crashes, it is necessary to ensure that the uninstallation system can adapt to all training scenarios, which usually requires a lot of manpower and material resources. As business needs grow, the workload of manual management will increase exponentially, greatly limiting the flexibility and maintainability of the system.

[0066] Other approaches based on asynchronous offloading introduce independent offloading flows and loading flows, which means that the offloading tasks and loading tasks are scheduled to be executed in separate flows, separated from the computation flow.

[0067] However, the inventors of the present disclosure note that the above method places the unloading and loading operations in separate streams, which may cause resource competition problems when kernels overlap. This competition will lead to additional performance overhead. In addition, the method requires at least three execution streams (computation stream, unloading stream, and loading stream), which is highly dependent on the support of hardware resources. However, not all hardware configurations can meet these requirements. For hardware that cannot support multiple concurrent execution streams, such as those with limited bandwidth or computing power, this method is not only difficult to achieve the expected performance gain, but may also lead to reduced system efficiency.

[0068] At least one embodiment of the present disclosure provides a data processing method, which includes: obtaining current operation records based on user interface information and operating status information; generating a data scheduling strategy based on historical operation records and current operation records; and executing corresponding data transfer operations according to the data scheduling strategy.

[0069] The data processing method provided by at least one embodiment of the disclosed embodiments can be applied to the training system to automatically generate a data scheduling strategy by analyzing the training status and operation records in real time. This method can dynamically adjust and optimize the data scheduling strategy during the training process without relying on a customized data scheduling analyzer or a handwritten scheduling program, and has the convenience of plug-and-play. At the same time, since the process of generating the data scheduling strategy does not depend on the distributed parallel strategy and model structure, it has a wider range of applications and can be applied to any type of tensor. This provides strong support for the training of complex models and improves the efficiency and stability of the system.

[0070] Furthermore, by introducing the task queue and state management mechanism, this method realizes the execution mode based on task push and subscription. On the one hand, it ensures the stability of the system and avoids the system crash caused by task execution failure. On the other hand, it decouples scheduling and execution, making the system more flexible in expansion capabilities.

[0071] Figure 2 A schematic block diagram of a distributed training system provided by at least one embodiment of the present disclosure. For example, Figure 2 The distributed training system shown includes multiple computing nodes 0~m, each computing node includes a host and multiple devices 0~n, where m and n are positive integers and can be set according to actual needs. In each computing node, data can be transmitted between the host and the device through a communication link. For example, the device can also be called a "slave device" and can be directly connected to the host through a hardware interface, or work with the host in a software-defined manner (such as virtualization technology) through the network.

[0072] For example, in at least one example of the disclosed embodiment, the host may include a central processing unit (CPU), and the device may include a graphics processing unit (GPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an AI accelerator, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and other accelerators. For example, the distributed training system 500 may include 8, 16, or 32 computing nodes, and each computing node may include 4 or 8 GPUs.

[0073] The data processing method provided by at least one embodiment of the present disclosure can be applied to various types of training systems, including but not limited to: Figure 2 A distributed training system and a training system containing only a single compute node are shown.

[0074] Figure 3 A flowchart of a data processing method provided by at least one embodiment of the present disclosure. Figure 3 As shown, the data processing method provided by at least one embodiment of the present disclosure includes the following steps S101 to S103.

[0075] Step S101: Acquire current operation records according to user interface information and operation status information.

[0076] For example, in step S101, the user interface may be embodied as an application programming interface (Application Programming Interface, API), which may include a module interface or a tensor interface, etc. The user may call the user interface.

[0077] For example, the module interface is used to integrate with existing deep learning frameworks (such as PyTorch), and data transfer (i.e., unloading, loading, etc.) functions can be enabled with almost no modification to the original code. In PyTorch, Module is the base class for building neural network models, which provides the basic functions required to define, manage, and train neural networks. The above module interface can be implemented by extending the Module class, for example, so that users can specify that the data of the entire model or part of the model (such as certain modules or layers) needs to be transferred without changing the model definition.

[0078] For example, the tensor interface allows users to perform data transfer operations on tensors, which can be applied to any type of tensor, including but not limited to activation values, weights, optimizer states, etc. Users can flexibly embed the tensor interface into the training system to load or unload specific tensors as needed.

[0079] For example, user interface information refers to descriptions related to the user interface, and may include interface name, interface type, function description, etc. Interface names may be, for example, Offload, Prefetch, Prepareoffload, Load, etc.

[0080] For example, in step S101, the running state information may refer to the running state information of the training system, which is used to provide the state of the target model during execution, that is, the runtime state, which may include information such as the current running position, the name of the current operation (OP), a timestamp, or the current resource usage. For example, the running state information may be obtained using tools such as a runtime profiler, which is not limited in the embodiments of the present disclosure.

[0081] For example, the current running position is used to indicate the specific position to which the target model is currently executing during the training process. For example, when using micro-batch technology, the current running position may include a micro-batch index to indicate the micro-batch currently being processed. For example, a micro-batch index of 0 indicates that the first micro-batch is currently being processed. For example, the target model is usually composed of multiple layers, and the current running position may include a layer index to indicate the layer currently being processed in the target model. For example, a layer index of 2 indicates that the third layer is currently being processed. According to actual needs, the current running position may also include other information, such as a step index, a module index, etc., wherein the step index is used to indicate the current training step (Step), and the module index is used to indicate the module currently being processed. It should be noted that the above are only some examples, and the various indexes may be in digital form, or in other forms such as names or a combination of the two, and the embodiments of the present disclosure are not limited to this.

[0082] The target model described above may be a machine learning model to be trained, such as a deep learning model. A deep learning model may be a neural network structure including multiple layers (3 layers, 4 layers, 8 layers or more layers), such as a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), etc. The structure of these neural networks usually includes an input layer, a hidden layer, and an output layer. The hidden layer refers to the layers between the input layer and the output layer in the neural network. The hidden layer is also called a processing layer. The input layer is used to receive data to be processed, such as an image to be processed, etc. The output layer is used to output processing results, such as a processed image, etc. The processing layer may include a convolutional layer, a pooling layer, a batch normalization layer, a fully connected layer, etc. Depending on the structure of the neural network, the processing layer may include different contents and combinations. In some examples, the target model may be, for example, a large language model (LLM), which is a deep learning model trained based on a large amount of text data and capable of understanding and generating natural language text. Large language models are usually based on the Transformer architecture and learn the statistical laws and semantic information of language from massive text data through self-supervised learning. They are usually used for tasks such as text generation, translation, question answering, and summarization.

[0083] For example, in step S101, the current operation record can be obtained according to the user interface information and the running status information. The current operation record (Trace) can be regarded as a log for recording the operation at the current moment. The current operation record not only records the operation performed by the user through the interface, but also records the running status information of the training system.

[0084] In some examples, the user interface information includes an interface name, and the operation status information includes a current operation location. An example of step S101 may be: combining the interface name and the current operation location to obtain a current operation record.

[0085] For example, the interface name and the current running position may be combined into a string or a structured data format as a current operation record.

[0086] Step S102: Generate a data scheduling strategy according to historical operation records and current operation records.

[0087] For example, in step S102, the historical operation record may be the operation record generated by step S101 in the previous training step. The data scheduling strategy refers to the timing of pre-fetching or unloading operations. In some examples, the historical operation record may also be the operation record generated by a trained model during the training process, which is suitable for scenarios where single-machine training is expanded to multi-machine training. Based on the similarity of the execution of each step in the target model training process, the tasks that need to be performed subsequently can be inferred through the historical operation records and the current operation records, which will be described in detail later.

[0088] Step S103: Execute corresponding data transfer operations according to the data scheduling strategy.

[0089] For example, in step S103, the data transfer operation may include an offload operation or a pre-fetch operation performed between the host and the device. For example, in some examples, the data transfer operation may include an offload operation or a pre-fetch operation for tensors performed between the CPU and the GPU.

[0090] The data processing method provided by at least one embodiment of the present disclosure automatically generates a data scheduling strategy by analyzing the training status and operation records in real time. This method can dynamically adjust and optimize the data scheduling strategy during the training process without relying on a customized data scheduling analyzer or a handwritten scheduling program, and has the convenience of plug-and-play. At the same time, since the process of generating the data scheduling strategy does not depend on the distributed parallel strategy and model structure, it has a wider range of applications and can be applied to any type of tensor. This provides strong support for the training of complex models and improves the efficiency and stability of the system.

[0091] Each operation record generated according to the above method is unique within a training step, so an operation record list can be maintained to store all operation records within the training step.

[0092] The data processing method provided by at least one embodiment of the present disclosure may further include the following steps S104-S105.

[0093] Step S104: storing the current operation record in the operation record list.

[0094] Step S105: Read historical operation records from the operation record list.

[0095] For example, the above steps S104 to S105 can be performed after step S101 and before step S102.

[0096] For example, at the beginning of training (e.g., the first training step Step 1), the operation record list can be initialized. The operation record list can be initialized based on the following principle: all tensors in the target model are unloaded. Furthermore, the operation record list can be initialized based on the principle of unloading as early as possible and loading (or prefetching) as late as possible. Through the above initialization principle, all tensors are first unloaded to release accelerator storage resources (e.g., video memory), and are only reloaded back to the accelerator when they are about to be used, thereby optimizing computing efficiency and resource utilization.

[0097] For example, assuming that the target model contains three tensors t1, t2, and t3, according to the above initialization principles, the operation record list can be initialized as follows:

[0098] {{Offload_t1},{Offload_t2},{Offload_t3},{Load_t1},{Load_t2},{Load_t3}}

[0099] Among them, Offload_t1 is the first record in the operation record list, indicating unloading tensor t1; Load_t3 is the last record in the operation record list, indicating loading tensor t3.

[0100] In the subsequent training step (eg, the second training step Step 2), each operation record in the above initialized operation record list may be regarded as a historical operation record.

[0101] According to the data processing method provided by at least one embodiment of the present application, operation records are continuously collected during the training process and stored in the operation record list. According to the similarity of the execution of each step in the target model training process, the tasks to be performed subsequently can be inferred through historical operation records and current operation records. This inference algorithm does not need to rely on distributed strategies or model information, and can be implemented only through simple and general scheduling rules. This is because although there are differences in schedulers with different distributed configurations, they are highly similar in scheduling timing. For example, the similarity of scheduling timing in the above text can be specifically reflected in at least the following two aspects:

[0102] 1. Prefetch timing: Prefetch operations are always performed one unit distance (such as one operation or one layer) in advance. For example, data of layer N+1 is prefetched when layer N is currently being processed. This ensures that the data is ready when the calculation reaches the time point when the data is needed, thereby reducing the waiting time for data loading.

[0103] 2. Redundant operation optimization: Reduce redundant unloading or loading operations at the beginning and end of the training step, and overlap them with the computing tasks as much as possible to reduce unnecessary memory management overhead.

[0104] Therefore, in at least one embodiment of the present disclosure, the scheduling rules are independent of the specific distributed strategy, but are determined by the relative order of the current operation in the overall operation process, which means that simple general scheduling rules based on operation records can be applied to achieve efficient scheduling.

[0105] In the data processing method provided by at least one embodiment of the present disclosure, the data scheduling strategy may include task information, and the task information is used to generate tasks. For example, the task information records the detailed information necessary for generating tasks. Based on different task information, various different types of tasks can be generated, such as offloading tasks, pre-fetching tasks, etc.

[0106] An example of step S102 may be: generating task information corresponding to the current operation record according to the relative order of the current operation record in the historical operation records.

[0107] For example, the relative position of the current operation record in the entire process can be analyzed and matched with the corresponding position in the historical operation record, so as to obtain the relative order of the current operation record in the historical operation record. By analyzing the entire historical operation record, the operation that needs to be performed currently can be predicted. For example, the pre-fetching time can be determined according to the relative order of the current operation record in the historical operation record, or redundant operations can be optimized.

[0108] In the data processing method provided by the above embodiments of the present disclosure, it is not necessary to understand the specific distributed strategy or model details, but only rely on the general rules between operations (such as pre-fetching timing and redundant operation optimization). Therefore, even if the configuration of the system changes, as long as the relative order between operations remains consistent, the general scheduling rules can be applied to achieve efficient scheduling.

[0109] For example, Figure 4A The following historical operation records are shown: Offload_t1, Offload_t2, Offload_t3, Load_t1, Load_t2, Load_t3, that is, in the previous training steps, the operations of unloading tensor t1, unloading tensor t2, unloading tensor t3, loading tensor t1, loading tensor t2 and loading tensor t3 were performed in sequence.

[0110] Taking the example that the relative order of the current operation record corresponds to Offload_t3 in the historical operation record, this means that the current operation is in the same position as Offload_t3 in the historical operation record. By analyzing the historical operation records, the operation that needs to be performed currently can be predicted. Specifically, in the above historical operation records, the operation after Offload_t3 is Load_t1. In order to optimize performance, it is necessary to perform a prefetch operation one unit distance in advance before the load operation so that the data is ready when needed. Therefore, the task information corresponding to the current operation record is the prefetch tensor t1, which can be recorded as Prefetch_t1. Based on the task information, a prefetch task for tensor t1 can be generated.

[0111] By analogy, assuming that the relative order of the current operation record corresponds to Load_t1 in the historical operation record, the task information corresponding to the current operation record is the prefetch tensor t2, which can be recorded as Prefetch_t2; assuming that the relative order of the current operation record corresponds to Load_t2 in the historical operation record, the task information corresponding to the current operation record is the prefetch tensor t3, which can be recorded as Prefetch_t3.

[0112] For example, based on historical operation records and current operation records, more in-depth automatic scheduling optimization can be achieved. For example, historical operation records can be regarded as nodes arranged in a linear order in the computational graph, and redundant operations can be eliminated through graph optimization technology. Specifically, through graph optimization technology, operations that do not save video memory but introduce additional overhead can be identified and eliminated, such as adjacent unloading and loading operations. Graph optimization technology can also identify unmatched unloading operations (i.e., there are only unloading operations without corresponding loading operations) and treat them as redundant operations for elimination.

[0113] For example, Figure 4B The following historical operation records are shown: Offload_t1, Offload_t2, Offload_t3, Load_t3, Load_t2, that is, in the previous training steps, the operations of unloading tensor t1, unloading tensor t2, unloading tensor t3, loading tensor t3 and loading tensor t2 were performed in sequence.

[0114] By analyzing the above historical operation records based on graph optimization technology, it can be found that the adjacent operations of unloading tensor t3 and loading tensor t3 are redundant. This operation not only fails to save video memory, but also introduces additional overhead, so it can be skipped in the current operation. Moreover, for tensor t1, the above historical operation record only contains the operation of unloading tensor t1, and there is no corresponding operation of loading tensor t1, which indicates that the unloading operation is unnecessary and can also be skipped. Therefore, assuming that the relative order of the current operation record corresponds to Offload_t1 in the historical operation record, the task information corresponding to the current operation record can be empty.

[0115] In addition, Figure 4A Similar to the example, Figure 4B In the example, assuming that the relative order of the current operation record corresponds to Load_t3 in the historical operation record, the task information corresponding to the current operation record is the prefetch tensor t2, which can be recorded as Prefetch_t2.

[0116] In the traditional imperative offloading execution scheme, the offloading system sends offloading tasks to the training system unidirectionally. Due to the lack of an effective feedback mechanism for the execution status of offloading tasks, the offloading system cannot know the execution stage of the task, nor can it know whether the task is successfully executed. Once there is any deviation between the offloading system and the training system (such as an instruction error, etc.), the entire training program may crash.

[0117] In the data processing method provided in at least one embodiment of the present disclosure, a task queue and state management mechanism are introduced to implement a task-based push and subscription execution mode, which ensures the stability of the system on the one hand and decouples scheduling and execution on the other hand, so that the system has more flexible expansion capabilities. Compared with the traditional imperative solution, it is more adaptable to complex and changeable actual application scenarios, which will be specifically introduced below.

[0118] In the data processing method provided by at least one embodiment of the present disclosure, tasks in different states correspond to different task queues.

[0119] For example, each task follows a preset state machine, ensuring that the task can proceed smoothly according to the predetermined process and preventing the system from crashing due to instruction errors. The state machine will be described in detail later. Tasks in different states are managed using separate task queues, which not only helps to organize and manage tasks, but also allows real-time tracking of the status of each task, so that potential problems can be discovered and handled in a timely manner.

[0120] In the data processing method provided in at least one embodiment of the present disclosure, an example of step S103 may include the following steps S201~S202.

[0121] Step S201: In response to the task information corresponding to an offload task or a pre-fetch task, a validity check is performed on the task information.

[0122] For example, in step S201, it is necessary to verify whether the offloading task or prefetching task to be executed is valid to avoid executing the wrong task. Validity checks may include, but are not limited to, resource availability, state consistency, dependencies, etc. Resource availability is used to confirm whether there are sufficient resources to perform the required operations, such as whether there is sufficient storage space available for offloading tasks, or whether there is sufficient bandwidth to support prefetching tasks. State consistency is used to ensure that the state of the current task is consistent with expectations and avoid errors caused by state mismatches. Dependencies are used to check whether there are unfinished predecessor tasks. If a task depends on the results of other tasks, it is necessary to confirm that these predecessor tasks have been completed.

[0123] Step S202: In response to the task information passing the validity check, the task corresponding to the task information is read from the corresponding task queue and executed to implement the corresponding data transfer operation.

[0124] For example, if the task information passes the validity check, it means that the corresponding task meets the execution conditions. At this time, the task is read from the corresponding task queue and executed, and the corresponding data transfer operation can be realized. For example, if the task information corresponds to an unloading task, the unloading task corresponding to the task information is read from the task queue storing the unloading task and executed, so as to realize the data unloading operation; if the task information corresponds to a prefetch task, the prefetch task corresponding to the task information is read from the task queue storing the prefetch task and executed, so as to realize the data unloading operation. For example, when executing a task, interaction with the underlying hardware can be triggered, thereby realizing data transfer operations related to specific hardware.

[0125] Through the above mechanism, different types of tasks can be effectively managed and scheduled, ensuring that each task will be executed only after it meets all prerequisites and passes the validity check. This method not only improves the stability and reliability of the system, but also optimizes the efficiency of resource utilization, making data unloading and prefetching operations more efficient and orderly.

[0126] In the data processing method provided in at least one embodiment of the present disclosure, the following steps S203A~S203B may also be included.

[0127] Step S203A: In response to executing the task, a new task is generated, and the status of the new task is set based on the status of the task.

[0128] Step S203B: insert the new task into the corresponding task queue.

[0129] It should be noted that the task executed in step S203A refers to the task corresponding to the task information read from the corresponding task queue in the above step S202. For example, after executing an unload task or a pre-fetch task, a new task needs to be generated, and based on the state of the unload task or the pre-fetch task executed, the state of the new task is set according to the preset state machine, and the new task after the state is set is inserted into the task queue corresponding to its state. When a new task is generated, the old task will be destroyed, which can be understood as an update of the task. This step enables the task to be correctly advanced according to the preset state machine process, which helps to maintain the stability of the system.

[0130] After generating the task information corresponding to the current operation record, the data processing method provided by at least one embodiment of the present disclosure may further include the following step S204.

[0131] Step S204: In response to the user interface information corresponding to the task information indicating that the task is ready to be uninstalled, insert the task corresponding to the task information into the corresponding task queue.

[0132] For example, in step S204, when it is detected that the user interface information corresponding to the task information indicates that the task is ready to be uninstalled, it means that the user has called the user interface named "prepare to uninstall", and the generated task can be directly inserted into the task queue without performing validity check. This step can be regarded as an initialization operation.

[0133] After generating the task information corresponding to the current operation record, the data processing method provided by at least one embodiment of the present disclosure may further include the following steps S205 to S206.

[0134] Step S205: In response to the user interface information corresponding to the task information indicating loading, the task information is checked for validity.

[0135] Step S206: In response to the task information passing the validity check, the task corresponding to the task information is read from the corresponding task queue and it is determined whether to return the data object of the task.

[0136] For example, in step S205, when it is detected that the user interface information corresponding to the task information indicates loading, it means that the user has called the user interface with the interface name "loading", and the task information needs to be checked for validity before subsequent operations. The relevant content of the validity check is similar to that in step S202 above, and will not be repeated here.

[0137] For example, in step S206, each task is associated with a data object that needs to be processed, such as a tensor that needs to be unloaded or prefetched. When it is detected that the user interface information corresponding to the task information indicates loading, it means that the user has called the user interface with the interface name "load", which means that a request to return the data object corresponding to the current task has been received. Before returning the data object of the task, it is necessary to determine whether it is the right time to return the data object of the task. For example, if the unloading task has not yet been executed, the data object of the task can be returned. The data object can be returned to the user or directly used in subsequent training processes; if the unloading task or prefetch task is currently being executed, it is necessary to wait for the prefetch task to be completed before returning the data object of the task to avoid process errors.

[0138] Figure 5A An exemplary schematic diagram of a data processing method provided for at least one embodiment of the present disclosure. Figure 5A An example of the above steps S201 to S206 is shown.

[0139] For example, Figure 5A As shown, when it is detected that the user interface information corresponding to the task information indicates that the task is ready to be uninstalled, the task generated according to the task information is directly inserted into the task queue without performing a validity check.

[0140] For example, Figure 5A As shown, the task to be executed is queried according to the data scheduling strategy, as follows: If the task information corresponds to an unloaded task, the task information needs to be checked for validity first. If the task information passes the validity check, the unloaded task corresponding to the task information is read from the task queue storing the unloaded task and executed to implement the unload operation performed between the host and the device. When the unloaded task is executed, a new task is generated, and the state of the new task is set according to the preset state machine based on the state of the unloaded task, and the new task after the state is set is inserted into the task queue corresponding to its state.

[0141] If the task information corresponds to a pre-fetch task, the task information needs to be checked for validity first. If the task information passes the validity check, the pre-fetch task corresponding to the task information is read from the task queue storing the pre-fetch task and executed to implement the pre-fetch operation performed between the host and the device. When the pre-fetch task is executed, a new task is generated, and the state of the new task is set according to the preset state machine based on the state of the pre-fetch task, and the new task after the state is set is inserted into the task queue corresponding to its state.

[0142] For example, Figure 5AAs shown, when the user interface information corresponding to the task information indicates loading, the task information needs to be checked for validity first. If the task information passes the validity check, the task corresponding to the task information is read from the corresponding task queue and it is determined whether the data object of the task is returned. The specific determination method has been introduced above and will not be repeated here.

[0143] This method not only improves the robustness and reliability of the system by introducing task queues and validity check mechanisms, but also makes resource management more flexible and efficient, and can adapt to complex and changing actual application scenarios.

[0144] In the data processing method provided in at least one embodiment of the present disclosure, the status of a task includes a preparation status, an unloading status, a pre-fetch status, and a completion status. The preparation status indicates that the data object corresponding to the task is about to be unloaded; the unloading status indicates that the data object corresponding to the task is being unloaded; the pre-fetch status indicates that the data object corresponding to the task is being pre-fetched; the completion status indicates that the data object corresponding to the task has completed pre-fetching and is ready to be returned. Tasks in different states can be managed by different task queues. For example, a preparation queue, an unloading queue, a pre-fetch queue, and a completion queue can be set to manage the tasks in the above different states respectively. In some examples, there is no need to set a completion queue, and a selection can be made according to actual needs.

[0145] Accordingly, the data processing method provided by at least one embodiment of the present disclosure may further include the following step S301.

[0146] Step S301: in response to the user interface information corresponding to the task information indicating that the task is ready to be uninstalled, the task state is initialized to a ready state.

[0147] For example, in step S301, when it is detected that the user interface information corresponding to the task information indicates ready to uninstall, it means that the user has called a user interface named "ready to uninstall". At this time, a state initialization operation is performed on the current task to set its state to the ready state.

[0148] In the data processing method provided in at least one embodiment of the present disclosure, the above-mentioned step S203A may include the following steps S302 to S304.

[0149] Step S302: In response to executing the uninstall task and the task status being in the preparation status, a new task is generated, and the status of the new task is set to the uninstall status.

[0150] Step S303: In response to executing the pre-fetch task and the task status being the unload status, a new task is generated, and the status of the new task is set to the pre-fetch status.

[0151] Step S304: in response to the pre-fetch task being completed and the task being in the pre-fetch state, generating a new task, and setting the state of the new task to the completed state.

[0152] For example, in steps S302 to S304, the timing of executing the unloading task and the prefetching task is determined according to the data scheduling strategy. When the unloading task is executed and its state is the preparing state, a new task is generated, and the state of the new task is set to the unloading state, which is transformed from the state of the unloading task (preparing state); when the prefetching task is executed and its state is the unloading state, a new task is generated, and the state of the new task is set to the prefetching state, which is transformed from the state of the prefetching task (unloading state); when the prefetching task is completed and its state is the prefetching state, a new task is generated, and the state of the new task is set to the completing state, which is transformed from the prefetching state.

[0153] Accordingly, the data processing method provided by at least one embodiment of the present disclosure may further include the following step S305.

[0154] Step S305: in response to determining whether the data object of the task is returned and the task status is in the preparation status, the task status is updated to the completion status.

[0155] For example, in step S305, the description of "determining whether to return the data object of the task" can be referred to in the above step S206, which will not be repeated here. When determining whether to return the data object of the task, if the current task state is the preparation state, the unloading state and the pre-fetching state can be skipped and directly updated to the completion state, at which time the data object of the task can be returned.

[0156] Figure 5B A state machine conversion diagram is provided for at least one embodiment of the present disclosure. Figure 5B An example of the above steps S301 to S305 is shown. Figure 5B As shown, the task status includes preparation status, unloading status, pre-fetching status and completion status. Four task queues (not shown) including preparation queue, unloading queue, pre-fetching queue and completion queue can be set accordingly to manage the tasks in the above different statuses respectively.

[0157] refer to Figure 5A and Figure 5BFirst, when it is detected that the user interface information corresponding to the task information indicates that it is ready to be unloaded, the state of the task is initialized to the ready state and directly inserted into the ready queue. Next, the task to be executed is queried according to the data scheduling strategy generated in the previous step, that is, when the task information corresponds to the unload task and passes the validity check, the task corresponding to the task information (unload task) is read from the ready queue and executed, a new task is generated, and the state of the new task is set to the unload state, and the new task is inserted into the unload queue; further, the task to be executed continues to be queried according to the data scheduling strategy, that is, when the task information corresponds to the prefetch task and passes the validity check, the task corresponding to the task information (prefetch task) is read from the unload queue and executed, a new task is generated, and the state of the new task is set to the prefetch state, and the new task is inserted into the prefetch queue. Finally, when the prefetch task is executed, the task is taken out of the prefetch queue, a new task is generated, and the state of the new task is set to the completed state, and the new task is inserted into the completed queue.

[0158] For example, when it is detected that the user interface information corresponding to the task information indicates loading, the task information needs to be checked for validity first. If the task information passes the validity check, the task corresponding to the task information is read from the corresponding task queue and it is determined whether to return the task data object. When determining whether to return the task data object, there are two situations depending on the task status:

[0159] If the current task is in the ready state, the unloading state and pre-fetching state can be skipped and the task can be directly updated to the completed state to avoid redundant calculations. When the task is in the completed state, the task data object can be returned.

[0160] If the current task is in the offloading state or prefetching state, you need to follow Figure 5B The process defined by the state machine shown is carried out until the pre-fetch task is completed and the task status is updated to the completed status, and then the data object of the task can be returned to ensure data integrity and consistency.

[0161] Fig. 6A The data processing device provided by at least one embodiment of the present disclosure can be embedded in various types of training systems, including but not limited to: Figure 2 The distributed training system shown and the training system including only a single computing node. The data processing device provided in at least one embodiment of the present disclosure can also be deployed independently and interact with the training system through a communication connection, which can be selected according to actual needs.

[0162] For example, Fig. 6AAs shown, the data processing device 600 provided by the embodiment of the present disclosure includes an acquisition module 601 , a generation module 602 and an execution module 603 .

[0163] For example, the acquisition module 601 is configured to acquire the current operation record according to the user interface information and the operation status information. For the relevant contents of the acquisition module 601, reference may be made to the relevant description of step S101 in the embodiment of the above data processing method, which will not be repeated here.

[0164] For example, the generation module 602 is configured to generate a data scheduling strategy according to the historical operation records and the current operation records. For the relevant contents of the generation module 602, reference may be made to the relevant description of step S102 in the embodiment of the above data processing method, which will not be repeated here.

[0165] For example, the execution module 603 is configured to execute the corresponding data transfer operation according to the data scheduling strategy. For the relevant content of the execution module 603, reference can be made to the relevant description of step S103 in the embodiment of the above data processing method, which will not be repeated here.

[0166] For example, in at least one embodiment of the present disclosure, the user interface information includes the interface name, the operation status information includes the current operation location, and the acquisition module 601 is further configured to: combine the interface name and the current operation location to obtain the current operation record.

[0167] For example, in at least one embodiment of the present disclosure, the acquisition module 601 is further configured to: store the current operation record into the operation record list; the generation module 602 is further configured to: read the historical operation record from the operation record list.

[0168] For example, in at least one embodiment of the present disclosure, the data scheduling strategy includes task information, and the task information is used to generate tasks. The generation module 602 is further configured to: generate task information corresponding to the current operation record according to the relative order of the current operation record in the historical operation record.

[0169] For example, in at least one embodiment of the present disclosure, the execution module 603 includes a task management unit, which is configured to: in response to the task information corresponding to an unloading task or a prefetching task, perform a validity check on the task information; and in response to the task information passing the validity check, read the task corresponding to the task information from the corresponding task queue and execute it to implement the corresponding data transfer operation.

[0170] For example, in at least one embodiment of the present disclosure, the data processing device 600 may further include an update module. The update module is configured to generate a new task in response to executing a task, set the status of the new task based on the status of the task, and insert the new task into the corresponding task queue.

[0171] For example, in at least one embodiment of the present disclosure, the state of a task includes a ready state, an unload state, a prefetch state or a completed state, and the update module is further configured to: in response to executing an unload task and the state of the task is a ready state, generate a new task and set the state of the new task to an unload state; in response to executing a prefetch task and the state of the task is an unload state, generate a new task and set the state of the new task to a prefetch state; and in response to the completion of the prefetch task and the state of the task is a prefetch state, generate a new task and set the state of the new task to a completed state.

[0172] For example, in at least one embodiment of the present disclosure, the update module is further configured to: in response to the user interface information corresponding to the task information indicating preparation for uninstallation, initialize the state of the task to a ready state.

[0173] For example, in at least one embodiment of the present disclosure, the update module is further configured to: in response to determining whether the data object of the task is returned and the task state is in a ready state, update the task state to a completed state.

[0174] For example, in at least one embodiment of the present disclosure, the task management unit may be configured to: in response to user interface information corresponding to the task information indicating that the task is ready to be uninstalled, insert the task corresponding to the task information into a corresponding task queue.

[0175] For example, in at least one embodiment of the present disclosure, the task management unit can be configured to: in response to user interface information corresponding to the task information indicating loading, perform a validity check on the task information; in response to the task information passing the validity check, read the task corresponding to the task information from the corresponding task queue and determine whether to return the data object of the task.

[0176] For example, in at least one embodiment of the present disclosure, tasks in different states correspond to different task queues.

[0177] For example, in at least one embodiment of the present disclosure, the data transfer operation includes an unloading operation or a pre-fetching operation on tensors performed between the CPU and the GPU.

[0178] For example, in at least one embodiment of the present disclosure, the data processing device 600 may further include an underlying execution module, which is configured to directly interact with the hardware according to the tasks executed by the task management unit to implement data transfer operations related to specific hardware. The underlying execution unit may also implement a stream synchronization function. There may be multiple streams (such as calculation streams, data transfer streams, etc.) in the system. The underlying execution unit can manage and coordinate the dependencies between the multiple streams to ensure data consistency and correctness.

[0179] It should be noted that the above-mentioned various modules and units can be implemented by software, hardware, firmware or any combination thereof. For example, the acquisition module, generation module and execution module can be implemented as an acquisition circuit, a generation circuit and an execution circuit respectively, and the embodiments of the present disclosure do not limit their specific implementation methods.

[0180] It should be understood that the data processing device 600 provided in at least one embodiment of the present disclosure can be used to implement the aforementioned data processing method, and can also achieve technical effects similar to the aforementioned data processing method, which will not be elaborated herein.

[0181] It should be noted that in the embodiments of the present disclosure, the data processing device 600 may include more or fewer modules or units, and the connection relationship between the modules or units is not limited and can be determined according to actual needs. The specific configuration of each module or unit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or in other applicable ways.

[0182] Figure 6B A schematic block diagram of a data processing device provided by at least one embodiment of the present disclosure. For example, Figure 6B This is a specific example of the data device provided in the above embodiment of the present disclosure.

[0183] For example, Figure 6B As shown, the data processing device provided by at least one embodiment of the present disclosure is embedded in a training system, including an acquisition module, an execution module, a generation module and a bottom-level execution module. The acquisition module obtains user interface information and running status information from the training system, and obtains the current operation record based on the user interface information and the running status information, and stores the obtained current operation record in the operation record list. The generation module reads the historical operation record from the operation record list, and generates a data scheduling strategy based on the historical operation record and the current operation record. The data scheduling strategy is embodied as task information. The execution module maintains the operation record list and the task queue to realize the function of the coordination center. The execution module includes a task management unit. The task management unit receives the task information generated by the generation module, and can realize the functions of validity check, task queue update, task execution, etc. For details, please refer to the description of the above embodiment, which will not be repeated here. The bottom-level execution module directly interacts with the hardware according to the tasks performed by the task management unit to realize data transfer operations (prefetching, unloading operations) related to specific hardware. It should be noted that Figure 6B The data processing device shown is only an example, and the modules therein may be increased or decreased according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0184] Figure 7 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0185] For example, Figure 7 As shown, the electronic device 700 includes at least one processor 701 and at least one memory 702. The at least one memory 702 includes one or more computer program modules. The one or more computer program modules are stored in the memory 702 and are configured to be executed by the at least one processor 701. The one or more computer program modules include instructions for executing the above-mentioned data processing method. When the instructions are executed by the at least one processor 701, one or more steps in the data processing method provided in at least one embodiment of the present disclosure can be executed. The memory 702 and the processor 701 can be interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0186] For example, the processor 701 may be a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a general purpose graphics processing unit (GPGPU), an artificial intelligence (AI) accelerator, or other forms of processing units with data processing capabilities and / or program execution capabilities, such as a field programmable gate array (FPGA), etc.; for example, the central processing unit (CPU) may be an X86, ARM, RISC-V architecture, etc. The processor 701 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 700 to perform desired functions.

[0187] For example, the memory 702 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (Cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc.

[0188] Figure 8 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0189] The electronic devices in at least one embodiment of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 8The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0190] The electronic device includes at least one processor and a memory. The processor here may be referred to as the processing device 801 described below, and the memory may include at least one of the read-only memory (ROM), random access memory (RAM) and storage device 808 described below. The memory is used to store programs for executing the methods described in the above-mentioned various method embodiments; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0191] like Figure 8 As shown, the electronic device 800 may include a processing device 801 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device 808 into a random access memory (RAM). In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface is also connected to the bus 804.

[0192] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a display, a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0193] In particular, according to at least one embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, at least one embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of at least one embodiment of the present disclosure are executed.

[0194] It should be noted that the computer-readable medium of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In at least one embodiment of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In at least one embodiment of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination of the above.

[0195] The computer-readable medium may be included in the electronic device 800 ; or may exist independently without being assembled into the electronic device 800 .

[0196] Fig. 9 A schematic block diagram of a non-transitory computer-readable storage medium is provided for at least one embodiment of the present disclosure.

[0197] For example, Fig. 9 As shown, a non-transitory computer-readable storage medium 900 stores computer-readable instructions 901 , and when the computer-readable instructions 901 are executed by at least one processor, one or more steps of the above-mentioned data processing method are performed.

[0198] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other applicable storage media. For example, the readable storage medium may also be Figure 7 For the memory 702 in, the related description can be referred to the aforementioned content and will not be repeated here.

[0199] Although the disclosure has been described in detail above with general descriptions and specific implementation methods, it is obvious to those skilled in the art that some modifications or improvements may be made to the embodiments of the disclosure. Therefore, these modifications or improvements made without departing from the spirit of the disclosure are within the scope of protection claimed by the disclosure.

[0200] There are a few points to note about this disclosure:

[0201] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to the general design.

[0202] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present disclosure, the thickness of layers or regions is enlarged or reduced, that is, these drawings are not drawn according to the actual scale.

[0203] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0204] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A data processing method, comprising: Obtain current operation records based on user interface information and operation status information; Generate a data scheduling strategy according to the historical operation records and the current operation records; The corresponding data transfer operation is performed according to the data scheduling strategy.

2. The data processing method according to claim 1, wherein: The user interface information includes an interface name, the running status information includes a current running position, and obtaining the current operation record according to the user interface information and the running status information includes: The interface name and the current running position are combined to obtain the current operation record.

3. The data processing method according to claim 1, further comprising: Storing the current operation record in an operation record list; The historical operation record is read from the operation record list.

4. The data processing method according to claim 1, wherein: The data scheduling strategy includes task information, and the task information is used to generate tasks. The generating of the data scheduling strategy according to the historical operation records and the current operation records includes: The task information corresponding to the current operation record is generated according to the relative order of the current operation record in the historical operation record.

5. The data processing method according to claim 4, wherein: The performing a corresponding data transfer operation according to the data scheduling strategy includes: In response to the task information corresponding to an offload task or a prefetch task, performing a validity check on the task information; In response to the task information passing the validity check, the task corresponding to the task information is read from the corresponding task queue and executed to implement the corresponding data transfer operation.

6. The data processing method according to claim 5, further comprising: In response to executing the task, generating a new task, and setting a status of the new task based on a status of the task; Insert the new task into the corresponding task queue.

7. The data processing method according to claim 6, wherein: The state of the task includes a preparation state, an unloading state, a pre-fetching state or a completion state, and in response to executing the task, generating a new task and setting the state of the new task based on the state of the task includes: In response to executing an uninstall task and the state of the task being in a preparation state, generating a new task, and setting the state of the new task to an uninstall state; In response to executing a pre-fetch task and the state of the task being an unloaded state, generating a new task, and setting the state of the new task to a pre-fetch state; In response to the pre-fetch task being executed and the task being in the pre-fetch state, a new task is generated, and the state of the new task is set to the completed state.

8. The data processing method according to claim 7, further comprising: In response to the user interface information corresponding to the task information indicating that the task is ready to be uninstalled, the state of the task is initialized to a ready state.

9. The data processing method according to claim 7, further comprising: In response to determining whether the data object of the task is returned and the state of the task is a preparation state, the state of the task is updated to a completion state.

10. The data processing method according to claim 4, wherein: After generating the task information corresponding to the current operation record, the method further includes: In response to the user interface information corresponding to the task information indicating that the task is ready to be uninstalled, the task corresponding to the task information is inserted into a corresponding task queue.

11. The data processing method according to claim 4, wherein: After generating the task information corresponding to the current operation record, the method further includes: In response to the user interface information corresponding to the task information indicating loading, performing validity check on the task information; In response to the task information passing the validity check, the task corresponding to the task information is read from the corresponding task queue and it is determined whether to return the data object of the task.

12. The data processing method according to any one of claims 5 to 11, wherein: Tasks in different states correspond to different task queues.

13. The data processing method according to claim 1, wherein: The data transfer operation includes an unloading operation or a pre-fetching operation for a tensor executed between a central processing unit and a graphics processing unit.

14. A data processing device, comprising: An acquisition module is configured to acquire current operation records according to user interface information and operation status information; A generating module, configured to generate a data scheduling strategy according to the historical operation records and the current operation records; The execution module is configured to execute the corresponding data transfer operation according to the data scheduling strategy.

15. An electronic device, comprising: at least one processor; at least one memory storing one or more computer program modules; The one or more computer program modules are configured to be executed by the at least one processor to execute instructions of the data processing method according to any one of claims 1 to 13.

16. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon, wherein: When the computer-readable instructions are executed by at least one processor, the data processing method according to any one of claims 1 to 13 is performed.

Images