Data processing method, apparatus and system

Abstract

The application provides a data processing method and device system, and relates to the technical field of machine learning. The method comprises the following steps: a first group of processing units receive first group model data from a second group of processing units, the first group model data being at least one group of model data in multiple groups of model data corresponding to a first machine learning model. The method further comprises the following step: the first group of processing units load the first machine learning model according to the first group model data. In this way, the multiple groups of processing units can quickly transmit model data, greatly improving the loading efficiency of the model.

Description

Technical Field

[0001] The embodiments of this application primarily relate to the field of machine learning. More specifically, the embodiments of this application particularly relate to data processing methods, apparatus, and systems. Background Technology

[0002] Currently, users may need to deploy different types of machine learning models in various scenarios. Massive amounts of model files are typically stored centrally in a data warehouse. When creating a model instance, the model files need to be retrieved from the data warehouse. In actual business operations, a large number of model instances often need to be launched simultaneously. In this case, storage nodes and compute nodes may face enormous pressure.

[0003] In existing technologies, model files are typically pulled from a remote storage cluster and deployed using an independent resource pool. However, due to the periodic and sudden nature of inference services, the network traffic pressure for data transmission is relatively high, and resource utilization is insufficient. Summary of the Invention

[0004] The embodiments of this application provide a data processing method and apparatus to solve the loading problem of machine learning models in the prior art and improve the loading efficiency of machine learning models.

[0005] According to a first aspect of this application, a data processing method is provided. The method includes a first set of processing units receiving a first set of model data from a second set of processing units, wherein the first set of model data is at least one set of model data from a plurality of sets of model data corresponding to a first machine learning model. The method further includes the first set of processing units loading a first machine learning model based on the first set of model data.

[0006] This method involves fine-grained division of model data and the use of high-speed channels between processing units to obtain the divided model data from other processing units. This allows model data to be quickly transferred between processing unit groups, reducing the time required to acquire model data and thus improving the loading efficiency of machine learning models.

[0007] In some embodiments, the data processing system further includes a storage node, wherein the method further includes: in response to the absence of model data for a first machine learning model in multiple processing units, the first processing unit receives a second set of model data from the storage node, wherein the second set of model data is at least one set of model data that differs from the first set of model data among multiple sets of model data corresponding to the first machine learning model, and the first set of model data is the model data received by the second processing unit from the storage node. In this manner, when model data is absent in multiple processing nodes, model data for the machine learning model can be obtained from the storage node, improving the flexibility of data processing.

[0008] In some embodiments, the first set of processing units loads a first machine learning model based on a first set of model data, including: the first set of processing units loads the first machine learning model based on a first set of model data and a second set of model data. In this manner, the first set of processing units loads the first machine learning model by receiving multiple sets of model data, thus improving the efficiency of model loading.

[0009] In some embodiments, the method further includes: the first group of processing units sending a second set of model data to the second group of processing units. In this way, the first group of processing nodes can send the model data obtained from the storage node to the second group of processing units, avoiding the second group of processing units obtaining model data from the storage node, thus fully utilizing the data transmission links between multiple groups of processing units and improving model loading efficiency.

[0010] In some embodiments, the method further includes: in response to the presence of model data for a first machine learning model in a plurality of processing units, determining a second set of processing units that have loaded the first machine learning model. This approach allows all model data to be received from other processing units in the plurality of processing units that have already loaded the first machine learning model, and uses all the received model data to load a complete model instance of the first machine learning model, enhancing the scalability of the model instance during the loading process and reducing the loading time of the machine learning model.

[0011] In some embodiments, the plurality of processing units further includes a third group of processing units, and the method further includes: in response to the presence of model data for a first machine learning model in the plurality of processing units, determining a third group of processing units that has loaded the first machine learning model, wherein the third group of processing units is at least one group of processing units different from the second group of processing units; and the first group of processing units receiving a third group of model data from the third group of processing units, wherein the third group of model data is at least one group of model data different from the first group of model data among the plurality of model data corresponding to the first machine learning model. In this manner, processing units other than the first and second groups of processing units that contain model data can be identified in the plurality of processing units, and the data transmission links between the plurality of processing units can be utilized to improve the efficiency of model loading.

[0012] In some embodiments, the first set of processing units loads a first machine learning model based on a first set of model data, comprising: the first set of processing units loading the first machine learning model based on the first set of model data and a third set of model data. This method allows model data to be received from processing units other than the first and second sets of processing units, improving the flexibility and efficiency of loading the first machine learning model.

[0013] In some embodiments, the multiple sets of model data are obtained by processing multiple tensors of a first machine learning model. The method further includes: determining multiple tensors of the first machine learning model; dividing the multiple tensors sequentially to determine multiple fragment files; and receiving multiple sets of model data based on the multiple fragment files. This method divides the model files of the first machine learning model at the tensor level before multiple processing units receive the multiple sets of model data, improving the transmission efficiency of the multiple sets of model data and the overall loading efficiency of the first machine learning model.

[0014] In some embodiments, the method further includes: multiple processing units receiving multiple sets of model data from a storage node through multiple processes. This approach utilizes multiple processes to simultaneously transmit data during the transmission of multiple sets of model data, improving the transmission efficiency of the multiple sets of model data and increasing the loading efficiency of the first machine learning model.

[0015] In some embodiments, the method further includes: receiving a loading request for loading the first machine learning model; determining, based on the loading request, whether the first machine learning model has been loaded by at least one group of processing units among a plurality of processing units; in response to the first machine learning model not being loaded by at least one group of processing units, obtaining a first loading strategy for loading the first machine learning model; in response to the first machine learning model being loaded by at least one group of processing units, obtaining a second loading strategy for loading the first machine learning model; and loading the first machine learning model based on the first loading strategy or the second loading strategy. This approach allows for the generation of different loading strategies based on the loading status of the first machine learning model in the plurality of processing units, and the flexible adjustment of the loading process for the first machine learning model based on different loading strategies, thereby improving the scheduling capability of model loading and increasing the efficiency of model loading.

[0016] According to a second aspect of this application, a model loading apparatus is provided. The apparatus includes: a data transceiver module and a data processing module, wherein the data transceiver module is used for a first group of processing units to receive a first group of model data from a second group of processing units, the first group of model data being at least one set of model data from a plurality of sets of model data corresponding to a first machine learning model; and the data processing module is used for the first group of processing units to load a first machine learning model based on the first group of model data.

[0017] In some embodiments, the data processing system further includes a storage node, and the first set of model data is received from the storage node by the second set of processing units. The apparatus further includes: a data transceiver module is further configured to respond to the absence of model data of the first machine learning model in the plurality of processing units, wherein the first set of processing units receives the second set of model data from the storage node, wherein the second set of model data is at least one set of model data that is different from the first set of model data in the plurality of model data corresponding to the first machine learning model, and the first set of model data is the model data received from the storage node by the second set of processing units.

[0018] In some embodiments, the first set of processing units loads a first machine learning model based on a first set of model data, including: the data processing module is further configured to load the first machine learning model based on the first set of model data and a second set of model data.

[0019] In some embodiments, the apparatus further includes a data transceiver module for sending a second set of model data from the first set of processing units to the second set of processing units.

[0020] In some embodiments, the apparatus further includes: a data processing module further configured to determine a second set of processing units that have loaded the first machine learning model in response to model data of the first machine learning model existing in the plurality of processing units.

[0021] In some embodiments, the plurality of processing units further includes a third set of processing units, and the apparatus further includes: a data processing module further configured to, in response to data in which a first machine learning model exists in the plurality of processing units, determine a third set of processing units that have loaded the first machine learning model, wherein the third set of processing units is at least one set of processing units different from the second set of processing units; and a data transceiver module further configured to, wherein the first set of processing units receives third set of model data from the third set of processing units, wherein the third set of model data is at least one set of model data different from the first set of model data among the plurality of model data corresponding to the first machine learning model.

[0022] In some embodiments, the apparatus further includes: an original file acquisition unit configured to acquire an original file for a machine learning model, the original file including a plurality of tensors; a sorting unit configured to sort the plurality of tensors based on tensor information in the plurality of tensors; and a fragmented file generation unit configured to generate a plurality of fragmented files by dividing the sorted plurality of tensors.

[0023] In some embodiments, the first set of processing units loads a first machine learning model based on a first set of model data, including: the data processing module is further configured to load the first machine learning model based on the first set of model data and a third set of model data.

[0024] In some embodiments, where multiple sets of model data are obtained by processing multiple tensors of a first machine learning model, the apparatus further includes: a data processing module further configured to determine multiple tensors of the first machine learning model; a data processing module further configured to divide the multiple tensors sequentially to determine multiple fragment files; and a data transceiver module further configured to receive multiple sets of model data based on the multiple fragment files. It should be noted that the multiple fragment files correspond one-to-one with the multiple sets of model data. For example, when fragment file A and fragment file B are determined and the first set of model data and the second set of model data are received based on fragment file A and fragment file B respectively, then fragment file A corresponds to the first set of model data, and fragment file B corresponds to the second set of model data.

[0025] In some embodiments, the apparatus further includes a data transceiver module for receiving multiple sets of model data from a storage node through multiple processes by multiple processing units.

[0026] In some embodiments, the apparatus further includes: a data transceiver module further configured to receive a loading request for loading the first machine learning model; a data processing module further configured to determine, based on the loading request, whether the first machine learning model has been loaded by at least one group of processing units among a plurality of processing units; the data processing module further configured to, in response to the first machine learning model not being loaded by at least one group of processing units among a plurality of processing units, obtain a first loading strategy for loading the first machine learning model; the data processing module further configured to, in response to the first machine learning model being loaded by at least one group of processing units among a plurality of processing units, obtain a second loading strategy for loading the first machine learning model; and the data processing module further configured to load the first machine learning model based on the first loading strategy or the second loading strategy.

[0027] According to a third aspect of this application, an electronic device is also provided, comprising: at least one computing unit; and at least one memory coupled to the at least one computing unit and storing instructions for execution by the at least one computing unit, the instructions, when executed by the at least one computing unit, causing the device to perform the method according to the first aspect of this application.

[0028] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the method described according to a first aspect of this application.

[0029] According to a fifth aspect of this application, a computer program product is also provided, including computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method described according to a first aspect of this application.

[0030] According to a sixth aspect of this application, a data processing system is also provided, comprising a plurality of processing units, wherein at least one of the plurality of processing units is configured to implement the method according to a first aspect of this application.

[0031] Understandably, the apparatus of the second aspect, the electronic device of the third aspect, the computer storage medium of the fourth aspect, the computer program product of the fifth aspect, or the data processing system of the sixth aspect provided above are used to perform the method provided in the first aspect. Therefore, the explanations or descriptions regarding the first aspect also apply to the second, third, fourth, fifth, and sixth aspects. Furthermore, the beneficial effects achievable by the second, third, fourth, fifth, and sixth aspects can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0032] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0033] Figure 1 A schematic diagram of a system architecture in which several embodiments of this application can be implemented is shown;

[0034] Figure 2 A schematic diagram of an example architecture of a computing cluster for data processing according to some embodiments of this application is shown;

[0035] Figure 3A A schematic flowchart of a method for data processing according to some embodiments of this application is shown;

[0036] Figure 3B Another schematic flowchart of a method for data processing according to some embodiments of this application is shown;

[0037] Figure 3C Another schematic flowchart of a method for data processing according to some embodiments of this application is shown;

[0038] Figure 3D Another schematic flowchart of a method for data processing according to some embodiments of this application is shown;

[0039] Figure 3E Another schematic flowchart of a method for data processing according to some embodiments of this application is shown;

[0040] Figure 4 A schematic diagram illustrates an example environment in which several embodiments of this application can be implemented;

[0041] Figure 5 A schematic diagram illustrating an example of model file segmentation according to some embodiments of this application is shown;

[0042] Figure 6 A schematic diagram illustrating an example of model piecewise loading according to some embodiments of this application is shown;

[0043] Figure 7 A schematic diagram illustrating another example of model piecewise loading according to some embodiments of this application is shown;

[0044] Figure 8 A schematic diagram illustrating an example of elastic scaling of model instances according to some embodiments of this application is shown;

[0045] Figure 9 A schematic diagram illustrating an example of a registration loading request according to some embodiments of this application is shown;

[0046] Figure 10 A schematic diagram illustrating examples of loading strategy generation and execution according to some embodiments of this application is shown;

[0047] Figure 11 Block diagrams of apparatuses according to some embodiments of this application are shown; and

[0048] Figure 12 A block diagram of a computing device capable of implementing several embodiments of the present application is shown. Detailed Implementation

[0049] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0050] In the description of embodiments of this application, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same single or multiple objects. Other explicit and implicit definitions may also be included below.

[0051] As mentioned above, for machine learning model deployment tasks, model files are typically pulled from a remote storage cluster, and a separate resource pool is used for model deployment. Therefore, in the traditional approach to deploying machine learning models, the weight files of the machine learning model are first pulled one by one from a remote model repository to a local disk (such as a solid-state drive), then the files are loaded into Dynamic Random-Access Memory (DRAM), and finally transferred to the high-bandwidth memory (HBM) of the processing unit to load the machine learning model. However, in this traditional approach, because file transfer is performed serially, the file retrieval speed is limited by network bandwidth and remote storage performance, resulting in a slow actual model loading process.

[0052] To address the aforementioned traditional methods for deploying machine learning models, existing technologies have proposed further improvements. One approach involves loading model weight files one by one from the local disk into DRAM, and then transferring the data from DRAM to the processing unit's HBM to load the machine learning model. However, this approach suffers from limited model file reading speed, resulting in a slow actual model loading process. Furthermore, the system may schedule the file reading process, increasing deployment latency, slowing down model loading, and impacting overall system performance. Another approach proposed in existing technologies utilizes a memory pool to store model data. A memory pool is a dynamic memory management technique that pre-allocates and reuses memory blocks. It caches frequently used or commonly accessed model data, reducing latency from repeated loading. In this approach, some models are pre-loaded into the memory pool, and then the model data is directly loaded from the memory pool into the processing unit's HBM. However, this approach is limited by memory capacity and scheduling strategy optimization. An unreasonable scheduling strategy may lead to memory pool resource contention, increasing the risk of system failure propagation and impacting overall system performance.

[0053] To address at least some of the aforementioned problems and other potential issues, in the embodiments of this application, a first group of processing units first receives a first set of model data from a second group of processing units. This first set of model data comprises at least one set of model data from multiple sets of model data corresponding to a first machine learning model. Finally, the first group of processing units loads the first machine learning model based on the first set of model data. This method, by finely dividing the model data and utilizing high-speed channels between processing units to obtain the divided model data from other processing units, enables rapid transmission of model data between processing unit groups, reducing model data acquisition time and thereby improving the loading efficiency of the machine learning model.

[0054] Figure 1A schematic diagram of a system architecture 100 in which several embodiments of this application can be implemented is shown. For example... Figure 1 As shown, the system architecture 100 includes a computing cluster 102. The computing cluster 102 includes computing nodes 104, user terminals 106, network nodes 108, and storage nodes 110. Computing nodes 104 may include dedicated computing devices for processing machine learning model-related tasks, such as neural network processing units (NPUs), graphics processing units (GPUs), etc. User terminals 106 include, but are not limited to, in-vehicle computing devices, personal computers, servers, handheld or laptop devices, mobile devices, multiprocessor systems, consumer electronics, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. Network nodes 108 may include devices for network communication between computing nodes 104, user terminals 106, and storage nodes 110; for example, network node 108 may include a network interface card. Storage nodes 110 may be devices capable of storing model data; for example, storage nodes 110 may be disk drives. It should be noted that the devices in computing cluster 102 can also be any other devices capable of implementing this application, and this application does not impose any restrictions. Additionally, computing node 104, storage node 110, and user terminal 110 can communicate directly locally.

[0055] The above combination Figure 1 A schematic diagram illustrating a system architecture 100 in which embodiments of this application can be implemented is described. The following is in conjunction with... Figure 2 The diagram illustrates the architecture of a computing cluster for data processing according to some embodiments of this application. This example architecture is... Figure 1 An example architecture of computing cluster 102 in the example is provided. In some embodiments, the architecture of example 200 may be an architecture for processing artificial intelligence-related tasks, in which case the processing unit is a device for processing artificial intelligence-related tasks, such as a graphics processing unit or a tensor processing unit (TPU).

[0056] As shown in Example 200, an example architecture of the computing cluster 102 used to load the first machine learning model includes an inter-node processing unit interconnection network 202, computing nodes 204 and 214. In one example, computing node 104 includes multiple computing nodes, and computing nodes 204 and 214 are two of the multiple computing nodes in computing node 104. Both computing nodes 204 and 214 can be used to load the first machine learning model. The following detailed explanation uses node 204 as an example.

[0057] The computing node 204 includes a first group of processing units, and the computing node 214 includes a second group of processing units. In one example, processing units 1, 2, to n in computing node 204 constitute the first group of processing units, and processing units 1, 2, to n in computing node 214 constitute the second group of processing units. Furthermore, data transmission between the first group of processing units in computing node 204 and the second group of processing units in computing node 214 is achieved through an inter-node processing unit interconnection network 202.

[0058] In some embodiments, the first group of processing units of computing node 204 includes processing unit 1, processing unit 2 to processing unit n. The number of processing units can be set as needed; for example, two processing units can be set in the first group. In one example, the computing power of the processing units is the same; for example, processing unit 1 and processing unit 2 both have a computing power of 10 Flops (Flops per second). In another example, the computing power of the processing units is different; for example, processing unit 1 has a computing power of 10 Flops, while processing unit 2 has a computing power of 20 Flops. Additionally, during the loading of the machine learning model, each processing unit in the first group needs to load model data. In this way, the computing resources of each processing unit in each group are fully utilized, improving the efficiency of model loading and increasing the utilization rate of computing resources.

[0059] In some embodiments, the computing node 204 may further include a dynamic random access memory 206, which can cache model data of machine learning models to avoid repeatedly pulling model data from a remote model repository 212 or other remote storage devices each time a model is deployed, thus avoiding resource waste.

[0060] In some embodiments, the computing node 204 further includes a disk 208, such as a solid-state drive or other disk with high bandwidth and high performance, to store the model data pulled from the model repository 212, so as to improve the overall efficiency of model deployment.

[0061] In some embodiments, compute node 204 further includes a network interface card 210, which can establish a network connection with model repository 212 and then pull a set of model data from model repository 212. Model repository 212 is used to store model data for machine learning models. Since machine learning model data typically occupies a large amount of storage space, it needs to be stored in a remote model repository 212. The compute node then pulls the required model data from the remote model repository 212 via network transmission. In one example, model repository 212 can be used to store model data for different types of machine learning models, such as model data for large language models, visual models, and audio models. Additionally, the architecture of node 214 is the same as that of node 204.

[0062] In some embodiments, the architecture of Example 200 may not include Model Repository 212. In this case, the processing unit in Node 204 loads the machine learning model by retrieving model data stored in Disk 208 or Dynamic Random Access Memory 206.

[0063] In some embodiments, the architecture of Example 200 may include multiple nodes, and multiple sets of processing units within the multiple nodes transmit model data of machine learning models via an interconnection network 202. Additionally, processing units within a node may also transmit data via an intra-node interconnection network; for example, model data of machine learning models may be transmitted via intra-node interconnection networks 216 and 218.

[0064] In some embodiments, the inter-node processing unit interconnection network 202 features high bandwidth and high transmission rate. In one example, the inter-node processing unit interconnection network 202 may consist of an InfiniBand (IB) or Ethernet network based on the Remote Direct Memory Access (RDMA) protocol, thereby improving data transmission efficiency during the deployment of model instances.

[0065] This approach establishes a high-speed interconnection network between multiple processing units. By simultaneously pulling multiple sets of model data from a remote model repository and loading them onto multiple processing units, the loading efficiency of model files is greatly improved, fully utilizing computing resources and thus increasing the efficiency of deploying model instances.

[0066] The above combination Figure 2A schematic diagram of an example architecture of a computing cluster for data processing according to some embodiments of this application is described below. A schematic flowchart of a data processing method according to some embodiments of this application is described below with reference to FIG3. Example method 300A can be... Figure 1 It is executed at the control device of the computing node 104 in the computing cluster 102 and any other suitable computing device.

[0067] At box 302, the first set of processing units in the data processing system first receives the first set of model data from the second set of processing units. Additionally, the first set of model data is at least one set of model data from a plurality of sets of model data.

[0068] In some embodiments, the data processing system further includes a storage node for storing multiple sets of model data required to load the first machine learning model. Additionally, the storage node also stores other model data for loading other types of machine learning models; for example, the storage node may also store model data for a second machine learning model, which is a different type of machine learning model than the first machine learning model.

[0069] In some embodiments, when model data for the first machine learning model is absent in the multiple processing units, the first processing unit receives a second set of model data from the storage node. The second set of model data received by the first processing unit is at least one set of model data from the multiple sets of model data that is different from the first set of model data. Alternatively, the second processing unit may also receive the first set of model data from the storage node. The first set of model data is at least one set of model data from the multiple sets of model data that is different from the second set of model data.

[0070] Additionally, the first and second processing units can simultaneously receive the second set of model data and the first set of model data from the storage node through different processes. For example, the first processing unit receives the second set of model data from the storage node through a first process, and the second processing unit receives the first set of model data from the storage node through a second process. The first process is different from the second process, the first process corresponds to the first processing unit, and the second process corresponds to the second processing unit.

[0071] In some embodiments, the second group of processing units is at least one group of processing units that is different from the first group of processing units, and the second group of processing units corresponds one-to-one with at least one group of model data in the multiple groups of model data. The first group of processing units is also at least one group of processing units that is different from the second group of processing units, and the first group of processing units corresponds one-to-one with at least one group of model data in the multiple groups of model data. Additionally, each group of processing units in the multiple groups of processing units includes at least one processing unit.

[0072] Additionally, when the first processing unit receives the first set of model data from the second processing unit, the first processing unit can also send the second set of model data received from the storage node to the second processing unit, so that the second processing unit can also load the first machine learning model based on the first and second set of model data. In this way, both the first and second processing units can load the first machine learning model, making full use of processing unit resources and improving model loading efficiency.

[0073] At box 304, after the first processing unit receives the first set of model data from the second processing unit, the first processing unit loads the first machine learning model based on the first set of model data. In one example, after the first processing unit receives the first set of model data from the second processing unit, it can further load the first machine learning model based on the second set of model data received from the storage node and in combination with the first set of model data.

[0074] In some embodiments, when model data for a first machine learning model exists in multiple processing units, a second processing unit that has already loaded the first machine learning model is first identified. This second processing unit then includes multiple sets of model data. In one example, the second processing unit sends the multiple sets of model data to the first processing unit. After receiving the multiple sets of model data, the first processing unit can load the first machine learning model. This method allows for the elastic scaling of machine learning models across multiple processing units, significantly improving the efficiency of scaling machine learning model instances and substantially saving resource overhead.

[0075] Additionally, the multi-group processing units also include a third group of processing units, which is at least one group of processing units different from the second group of processing units. When model data for the first machine learning model exists in the multi-group processing units, a third group of processing units including the third group of model data can be determined first. The third group of model data is at least one group of model data that is different from the first group of model data. Subsequently, the first group of processing units can receive the third group of model data from the third group of processing units and load the first machine learning model based on the first group of model data and the third group of model data. In this way, the first group of processing units can receive the first group of model data from the second group of processing units and the third group of model data from the third group of processing units, avoiding receiving model data from remote storage nodes. By utilizing the transmission links between the multi-group processing units, the transmission efficiency of model data is improved, and the overall efficiency of loading the first machine learning model is improved.

[0076] In some embodiments, before performing the task of loading the first machine learning model, it is further required to preprocess the first machine learning model to obtain multiple sets of model data for the first machine learning model. The preprocessing process may be completed in a data processing system. The multiple sets of model data for the first machine learning model are obtained by processing a plurality of tensors of the first machine learning model. In an example, after the plurality of tensors of the first machine learning model are determined, the plurality of tensors are further sorted in order, for example, the plurality of tensors are sorted in descending order. The sorted plurality of tensors are then partitioned to obtain a plurality of shard files.

[0077] Additionally, when the first set of processing units needs to use the first machine learning model to perform an inference task, the first set of processing units needs to load a complete model instance for the first machine learning model, that is, the first set of processing units needs to load the complete multiple sets of model data. In an example, after the first machine learning model is partitioned at the tensor level into n shard files, the multiple sets of processing units may further receive the complete n sets of model data according to the n shard files, for example, the first set of processing units receives m sets of model data (m<n), and the second set of processing units receives (n-m) sets of model data. At this time, only m sets of model data are loaded in the first set of processing units, and the first set of processing units cannot form a complete model instance using the received m sets of model data. Therefore, it is also required to receive the remaining (n-m) sets of model data from the second set of processing units, and finally form a complete model instance at the first set of processing units according to the complete n sets of model data, so as to perform the inference task using the first machine learning model at the first set of processing units.

[0078] In some embodiments, the first set of processing units may further receive a loading request for loading the first machine learning model, and after receiving the loading request, determine whether the first machine learning model has been loaded by the multiple sets of processing units. Additionally, the multiple sets of processing units include a scheduling unit, and the scheduling unit may receive the loading request for the first machine learning model, and control the multiple sets of processing units to load the first machine learning model according to the received loading request.

[0079] In some embodiments, when the first machine learning model has not been loaded by the multiple sets of processing units, a first loading policy for loading the first machine learning model is obtained. After the first loading policy is obtained, the scheduling unit may control the multiple sets of processing units to load the first machine learning model according to the first loading policy.

[0080] In some embodiments, when a first machine learning model has been loaded by multiple processing units, a second loading strategy for loading the first machine learning model is obtained. After obtaining the second loading strategy, the scheduling unit can control the multiple processing units to load the first machine learning model according to the second loading strategy.

[0081] This method involves fine-grained division of model data and the use of high-speed channels between processing units to obtain the divided model data from other processing units. This allows model data to be quickly transferred between processing unit groups, reducing the time required to acquire model data and thus improving the loading efficiency of machine learning models.

[0082] The above combination Figure 3A A schematic flowchart illustrating a data processing method according to some embodiments of this application is described below. Figure 3B A schematic flowchart illustrating another data processing method according to some embodiments of this application is provided. Example method 300B can be... Figure 1 It is executed at the control device of the computing node 104 in the computing cluster 102 and any other suitable computing device.

[0083] In box 306, in response to the absence of model data for the first machine learning model in multiple processing units, the first processing unit receives a second set of model data from the storage node. The second set of model data is at least one set of model data that is different from the first set of model data among multiple sets of model data corresponding to the first machine learning model. The first set of model data is the model data received by the second processing unit from the storage node.

[0084] In some embodiments, when model data for a first machine learning model is absent in the multiple processing units, the first processing unit can receive a second set of model data from the storage node. This second set of model data is at least one set of model data from the multiple sets of model data that differs from the first set. Furthermore, the second processing unit can also obtain the first set of model data from the storage node. This first set of model data is at least one set of model data from the multiple sets of model data that differs from the second set. Additionally, the first and second sets of model data can be all the model data from the multiple sets of model data. In this way, each processing unit in the multiple processing units can receive different model data from the storage node, improving the efficiency of model data transmission.

[0085] In box 308, the first set of processing units loads the first machine learning model based on the first set of model data and the second set of model data.

[0086] In some embodiments, after receiving a second set of model data from a storage node and a first set of model data from a second set of processing units, the first set of processing units loads a first machine learning model based on the received first and second sets of model data. In this way, the first set of processing units can load a complete machine learning model for performing inference tasks.

[0087] In box 310, the first group of processing units sends the second group of model data to the second group of processing units.

[0088] In some embodiments, after receiving the second set of model data from the storage node, the first set of processing units can further send the received second set of model data to the second set of processing units. The second set of processing units can then load a first machine learning model based on the second set of model data received from the first set of processing units and the first set of model data received from the storage node. In this way, the second set of processing units can also load a complete first machine learning model based on the first set of model data and the second set of model data for performing inference tasks.

[0089] The above combination Figure 3B Another schematic flowchart illustrating a data processing method according to some embodiments of this application is described below. Figure 3C This is yet another schematic flowchart illustrating a data processing method according to some embodiments of the present application. Example method 300C can be... Figure 1 It is executed at the control device of the computing node 104 in the computing cluster 102 and any other suitable computing device.

[0090] In box 312, in response to the presence of model data for the first machine learning model in multiple processing units, a second set of processing units that has loaded the first machine learning model is determined.

[0091] In some embodiments, when loading a model task, after determining that model data of a first machine learning model already exists in multiple groups of processing units, a second group of processing units that has already loaded the first machine learning model is identified. Additionally, the second group of processing units may include partial model data of the first machine learning model, or it may include complete model data of the first machine learning model. In one example, when the second group of processing units includes complete model data of the first machine learning model, the first group of processing units can receive the complete model data from the second group of processing units and load the complete first machine learning model. In this way, the first group of processing units can quickly transfer model data and quickly load the first machine learning model through data links between processing unit groups.

[0092] In block 314, in response to the presence of model data for the first machine learning model in multiple processing units, a third group of processing units that has loaded the first machine learning model is determined. The third group of processing units is at least one group of processing units that is different from the second group of processing units.

[0093] In some embodiments, the third set of processing units is at least one set of processing units different from the second set of processing units. Additionally, the first machine learning model in the third set of processing units is pre-loaded for use in performing relevant inference tasks at any time.

[0094] In box 316, the first group of processing units receives the third group of model data from the third group of processing units. The third group of model data is at least one group of model data that is different from the first group of model data among the multiple groups of model data corresponding to the first machine learning model.

[0095] In some embodiments, the first group of processing units can receive a third group of model data from the third group of processing units via a data transmission link. The third group of model data is at least one set of model data that differs from the first group of model data. Additionally, the third group of model data may be the same as the second group of model data.

[0096] In box 318, the first set of processing units loads the first machine learning model based on the first set of model data and the third set of model data.

[0097] In some embodiments, after receiving the third set of model data, the first group of processing units loads a first machine learning model based on the first set of model data received from the second group of processing units and the third set of model data. In this way, the first group of processing units can receive model data from other processing units in a plurality of processing units and use it to load model data, utilizing the data transmission links between the plurality of processing units to accelerate data transmission and improve the efficiency of loading the model.

[0098] The above combination Figure 3C Another schematic flowchart illustrating a data processing method according to some embodiments of this application is described below. Figure 3D Further schematic flowcharts depicting data processing methods according to some embodiments of this application are provided. Example method 300D can be... Figure 1 It is executed at the control device of the computing node 104 in the computing cluster 102 and any other suitable computing device.

[0099] In box 320, multiple tensors of the first machine learning model are defined.

[0100] In box 322, multiple tensors are divided in sequence to determine multiple fragment files.

[0101] In some embodiments, the first machine learning model is partitioned at the tensor level to determine multiple fragment files for the first machine learning model. The determined multiple fragment files correspond to multiple sets of model data. In one example, after fragment file A and fragment file B are partitioned at the tensor level, fragment file A corresponds to one set of model data, and fragment file B corresponds to a different set of model data. Additionally, after fragment files A, B, C, and D are partitioned at the tensor level, fragment files A and B correspond to one set of model data, and fragment files C and D correspond to a different set of model data. It should be noted that the above examples are only for describing this disclosure and are not intended to limit this disclosure. By partitioning the model at the tensor level in this way, the multiple fragment files can be transmitted simultaneously during transmission, improving the data transmission efficiency of multiple processing units receiving multiple sets of model data.

[0102] In box 324, multiple sets of model data are received based on multiple fragment files.

[0103] In box 326, multiple processing units receive multiple sets of model data from the storage node through multiple processes.

[0104] In some embodiments, multiple processing units simultaneously receive multiple sets of model data from the storage node through multiple processes. In one example, the first and second processing units receive the second and first sets of model data respectively through different processes. In another example, the first processing unit receives the second set of model data through two processes, and the second processing unit receives the first set of model data through two other different processes. Additionally, the first and second processing units receive model data through at least one process, and the first and second processing units receive model data through different processes. This approach significantly improves the efficiency of multiple processing units receiving multiple sets of models, thereby improving the efficiency of model loading.

[0105] The above combination Figure 3D Another schematic flowchart of a data processing method according to some embodiments of this application is described below. Figure 3E Another schematic flowchart describes a data processing method according to some embodiments of this application. Example method 300E can be... Figure 1 It is executed at the control device of the computing node 104 in the computing cluster 102 and any other suitable computing device.

[0106] In box 328, a loading request for loading the first machine learning model is received.

[0107] In box 330, based on the loading request, it is determined whether the first machine learning model has been loaded by at least one of the multiple sets of processing units.

[0108] In some embodiments, before loading the model, it is necessary to determine whether at least one set of processing units among the multiple sets of processing units has already loaded the first machine learning model. This way, based on the determination result, different strategies can be determined for loading the model, improving the flexibility of model loading. Additionally, the loading strategy is generated by a scheduling unit associated with the multiple sets of processing units, and the scheduling unit can also monitor the loading status of the first machine learning model.

[0109] In box 332, in response to the fact that the first machine learning model has not been loaded by at least one of the multiple sets of processing units, a first loading strategy for loading the first machine learning model is obtained.

[0110] In some embodiments, when it is determined that the first machine learning model has not been loaded by multiple processing units, a first loading strategy is obtained. The first loading strategy enables multiple processing units to receive multiple sets of model data from the storage node and load the first machine learning model according to the received multiple sets of model data. Furthermore, by determining multiple shard files and receiving model data in parallel by multiple processes, the efficiency of loading the model for the first time is improved.

[0111] In box 334, in response to the fact that the first machine learning model has been loaded by at least one of the multiple sets of processing units, a second loading strategy for loading the first machine learning model is obtained.

[0112] In some embodiments, when it is determined that the first machine learning model has been loaded by multiple processing units, a second loading strategy is obtained. The second strategy enables the first processing unit in the multiple processing units to obtain the loaded first and third model data from the second or third processing unit that has already loaded the machine learning model, and improves the transmission efficiency of model data by utilizing the data transmission link between the data processing unit groups, thereby improving the efficiency of model loading.

[0113] In box 336, the first machine learning model is loaded based on either the first or the second loading strategy.

[0114] The above combination Figure 3E A further schematic flowchart illustrating examples of data processing methods according to some embodiments of this application is provided below. Figure 4 A schematic diagram illustrating an example environment in which various embodiments of this application can be implemented.

[0115] Figure 4 A schematic diagram of an example environment 400 in which several embodiments of this application can be implemented is shown. For example... Figure 1 As shown, the example environment 400 includes a data processing system 402. In some embodiments, the data processing system 402 includes dedicated computing devices, such as NPUs, GPUs, etc., that can be used to process tasks related to machine learning models. In some embodiments, the data processing system 402 is a host device that includes an NPU device or a GPU device. In some embodiments, the data processing system 402 includes nodes specifically designed for processing artificial intelligence model inference tasks, such as inference tasks for large language models. In some embodiments, the data processing system 402 includes a computing cluster specifically designed for processing artificial intelligence model inference tasks, such as an artificial intelligence computer cluster including multiple computing nodes. In some embodiments, the data processing system 402 includes, but is not limited to, in-vehicle computing devices, personal computers, servers, handheld or laptop devices, mobile devices, multiprocessor systems, consumer electronics, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0116] like Figure 4 As shown, the multiple processing units 410 in the data processing system 402 are used to deploy model instances of the first machine learning model 404, such as model instances of a large language model. The multiple processing units 410 include a first processing unit 412 and a second processing unit 414. The first processing unit 412 and the second processing unit 414 can be used to load a portion of the model data of the first machine learning model 404, or they can be used to load all the model data of the first machine learning model 404. The multiple sets of model data 406 are determined by partitioning multiple tensors in the first machine learning model 404. After the multiple tensors are partitioned, the data processing system 402 receives the multiple sets of model data 406 through multiple processes. In one example, each processing unit in the multiple processing units 410 of the data processing system 402 can receive one set of model data from the multiple sets of model data 406 through one or more processes.

[0117] For the data processing system 402, it may include an artificial intelligence computer cluster for deploying machine learning model instances, and the artificial intelligence computer cluster may include computing nodes for processing machine learning model deployment tasks. For example, the multiple sets of processing units 410 in the nodes of the data processing system 402 include a first set of processing units 412 and a second set of processing units 414. Additionally, the artificial intelligence computer cluster includes multiple nodes, and each node includes at least one set of processing units for performing tasks to deploy model instances of machine learning models.

[0118] For the multiple processing units 410, each processing unit transmits model data to the others via a designated data transmission link. Additionally, when the data processing system 402 includes other nodes, the multiple processing units 410 also transmit model data to the processing units in other nodes via designated data transmission links.

[0119] like Figure 4 As shown, during the deployment of a model instance for the first machine learning model 404, the first processing unit 412 in the multiple processing units 410 of the data processing system 402 can load the data based on the received first set of model data 408 after receiving the first set of model data 408 from the second processing unit 414. The first set of model data 408 is at least one set of model data from multiple sets of model data 406. In one example, the multiple sets of model data 406 include two sets of model data, namely the first set of model data 408 and the second set of model data.

[0120] In some embodiments, the data processing system 402 may perform a loading task for a first machine learning model 404, for example, loading multiple model instances for the first machine learning model 404 at multiple processing units 410. In one example, when the multiple processing units 410 include two processing units, two model instances for the first machine learning model 404 may be loaded at the multiple processing units 410. In another example, when the multiple processing units 410 include five processing units, five model instances for the first machine learning model 404 may be loaded at the multiple processing units 410.

[0121] In some embodiments, when executing a loading task for a first machine learning model 404, the data processing system 402 also obtains a loading strategy for loading the first machine learning model 404 and executes the loading task according to the specific strategy in the loading strategy. The specific loading strategy may include a sharding strategy, a concurrent read strategy, storage location information, a data source selection strategy, a data fetching strategy, a sharded loading strategy, and an elastic scaling strategy, etc. In one example, the data processing system 402 processes multiple tensors in the first machine learning model 404 and determines multiple sets of model data 406 according to the sharding strategy. It should be noted that the process of the data processing system 402 determining multiple sets of model data 406 is a preprocessing process and has been completed before the loading task for the first machine learning model 404. Additionally, the data processing system 402 also includes a storage node, which can be used to store multiple sets of model data 406 to facilitate the execution of the loading task for the first machine learning model 404.

[0122] This method involves fine-grained division of model data and the use of high-speed channels between processing units to obtain the divided model data from other processing units. This allows model data to be quickly transferred between processing unit groups, reducing the time required to acquire model data and thus improving the loading efficiency of machine learning models.

[0123] The above combination Figure 4 A schematic flowchart illustrating examples of data processing methods according to some embodiments of this application is provided below. Figure 5 A schematic diagram illustrating examples of model file segmentation according to some embodiments of this application.

[0124] As shown in Example 500, the original file 502 for the machine learning model includes multiple model layers, and each model layer includes a tensor. For example, the original file 502 in Example 500 includes six model layers, each including one tensor, and these six tensors are not all of the same size. Therefore, the partitioning of the original file 502 can have multiple results, such as partitioning result 504.

[0125] In partitioning result 504, it can be seen that the original file 502 was divided into two fragment files, corresponding to two sets of model data. Furthermore, both fragment files contain three tensors, with each fragment containing one larger tensor and two smaller tensors. In this case, the sizes of the two fragment files are relatively balanced, thus fully utilizing data transmission resources during fragment file retrieval. Additionally, during the partitioning of the original file for the machine learning model, the size error of the partitioned files can be controlled within a certain threshold. For example, when the original file is divided into 10 fragment files according to the tensor level, the size error of the 10 partitioned files is controlled within 5%. By maintaining the fragment files as similar in size as possible, the efficiency of receiving model data can be improved, and the overall loading efficiency of the machine learning model can be improved. It is understood that the above is only used to describe this disclosure and is not intended to limit this disclosure.

[0126] In some embodiments, during the partitioning of the original file of the machine learning model, the number of partitioned file segments is greater than or equal to the number of processing units in the multiple processing units. In one example, when the multiple processing units include a first processing unit and a second processing unit, and both the first and second processing units include one processing unit, the original file can be divided into two partitioned files, for example, partitioned file A and partitioned file B. The first and second processing units can then load partitioned files A and B respectively to obtain the first and second sets of model data. In another example, the original file can be divided into four partitioned files, for example, partitioned files A, B, C, and D. In this case, the first processing unit loads partitioned files A and B, and the second processing unit loads partitioned files C and D. Additionally, each processing unit in the multiple processing units needs to load a partitioned file of the original file; this maximizes the utilization of the computational resources of the multiple processing units.

[0127] The above, in conjunction with section 5, describes schematic diagrams illustrating examples of model file segmentation according to some embodiments of this application. The following, in conjunction with... Figure 6 A schematic diagram illustrating an example of model piecewise loading according to some embodiments of this application.

[0128] As illustrated in Example 600, this example uses an accelerator card (e.g., a GPU) as one of the processing units in a set of processing units. The original files of the machine learning model are divided into slice file 1 and slice file 2, corresponding to two sets of model data. The multiple processing units include processing unit 1 and processing unit 2. Processing unit 1 includes accelerator card 1, and processing unit 2 includes accelerator card 2. It should be noted that, in Example 500, accelerator card 1 in processing unit 1 performs the inference task, while accelerator card 2 in processing unit 2 does not perform the inference task and is only responsible for loading slice file 2.

[0129] In Example 600, actions related to accelerator card 1 are shown at 602, and actions related to accelerator card 2 are shown at 604. When processing unit 1 needs to perform an inference task, it first initializes the model in the HBM of accelerator card 1 at 606, then loads fragment file 1 from local or remote location to DRAM or ultimately to HBM at 608, and simultaneously loads fragment file 1 from local or remote location to HBM at 610.

[0130] Subsequently, at point 612, accelerator card 1 updates the fragment weight 1 corresponding to fragment file 1. Then, at point 614, processing unit 1 and processing unit 2 establish a communication domain. By establishing an interconnection network among multiple sets of processing units, at point 616, accelerator card 2 sends the model fragment weight 2 to accelerator card 1. Accelerator card 1 receives and updates the model fragment weight 2, thereby updating the machine learning model. Finally, using the updated machine learning model, at point 618, accelerator card 1 performs the model inference task.

[0131] This method allows for the loading of fragmented files by utilizing other processing units that are idle during inference tasks, and the loaded model data can be quickly transmitted to the processing units that need to perform inference tasks via the inter-card interconnection network, thereby saving deployment time for machine learning models and improving deployment efficiency.

[0132] The above combination Figure 6 A schematic diagram illustrating an example of model piecewise loading according to some embodiments of this application is described below, in conjunction with... Figure 7 A schematic diagram illustrating another example of model piecewise loading according to some embodiments of this application.

[0133] As shown in Example 700, combined with the previous Example 600, the original files of the machine learning model are divided into slice file 1 and slice file 2, which correspond to two sets of model data. Multiple processing units include processing unit 1 and processing unit 2. Processing unit 1 includes accelerator card 1, and processing unit 2 includes accelerator card 2. It should be noted that, in the case of Example 700, both accelerator card 1 and accelerator card 2 in processing unit 1 need to perform inference tasks.

[0134] In Example 600, actions related to accelerator card 1 are shown at 702, and actions related to accelerator card 2 are shown at 704. When both processing unit 1 and processing unit 2 need to perform inference tasks, models are initialized in the HBM of accelerator card 1 and accelerator card 2 at 706 and 708, respectively. Subsequently, accelerator card 1 loads fragment file 1 from local or remote location into DRAM or ultimately into HBM at 710, and accelerator card 2 loads fragment file 2 from local or remote location into DRAM or ultimately into HBM at 712.

[0135] Next, accelerator card 1 updates the model fragment weight 1 for fragment file 1 at 714, and accelerator card 2 updates the model fragment weight 2 for fragment file 2 at 716. After accelerator cards 1 and 2 have finished loading the model fragment weights, a communication domain is established between accelerator cards 1 and 2 at 718. By establishing an interconnection network between multiple processing units, accelerator card 2 sends the model fragment weight 2 to accelerator card 1 at 720. Accelerator card 1 receives and updates the model fragment weight 2. Simultaneously, at 722, accelerator card 1 sends the model fragment weight 1 to accelerator card 2, and accelerator card 2 receives and updates the model fragment weight 1, thereby updating the machine learning model at both accelerator cards 1 and 2. Finally, accelerator card 1 performs the model inference task at 724, and accelerator card 2 performs the model inference task at 726.

[0136] In some embodiments, the fragment file 1 and fragment file 2 also include multiple indexes formed by multiple tensors. The multiple indexes are used to quickly match fragment file 1 and fragment file 2 with multiple sets of processing units, so that multiple sets of processing units can quickly lock the fragment file corresponding to the index and quickly transmit model data through the data transmission channel between multiple sets of processing units.

[0137] This method allows loaded fragment files to be transferred between processing units that need to perform inference tasks via an inter-card interconnection network, and model data can be quickly transferred based on the index of the fragment files. This saves time in deploying multiple model instances on multiple processing units and improves the deployment efficiency of multiple model instances.

[0138] The above combination Figure 7 A schematic diagram illustrating another example of model piecewise loading according to some embodiments of this application is described below, in conjunction with... Figure 8 A schematic diagram illustrating an example of elastic scaling of a model instance according to some embodiments of this application.

[0139] In Example 800, taking the redeployment of the same model instance as an example, a set of processing units that have already loaded the model instance is called processing unit 1, and a set of processing units that need to redeploy the same model instance is called processing unit 2. The processing units in the two sets of processing units are accelerator cards.

[0140] First, at point 802, the inference process is started at processing unit 2, and at point 804, it is determined that the current inference process of processing unit 1 is in an idle state. Then, at point 806, the model is initialized, and at point 808, the inference communication domain of processing unit 1 is disconnected. After the initialization of processing unit 2 is completed and the inference communication domain of processing unit 1 is disconnected, at points 810 and 812, a communication domain is simultaneously established between processing unit 1 and processing unit 2, for example, by establishing an inter-card interconnection network using an InfiniBand (IB) network based on the Remote Direct Memory Access (RDMA) protocol or an Ethernet network. Next, at point 820, processing unit 1 traverses the model tensors in the loaded multiple sets of model data, and at point 818, it sends the loaded multiple sets of model data. Simultaneously, processing unit 2 receives the loaded multiple sets of model data at point 816 through the interconnection network between processing unit 1 and processing unit 2, and at point 814, iterates over the model tensors in the received multiple sets of model data. Then, after completing the parameter transmission, processing unit 1 re-establishes the inference communication domain at point 824 to facilitate the execution of subsequent inference tasks. Processing unit 2, after receiving complete sets of model data, establishes its own inference communication domain at point 822. Finally, both processing units 1 and 2 deploy the same model instance, thus enabling both to execute inference tasks. At point 826, they await the arrival of inference requests to execute the relevant inference tasks.

[0141] Additionally, when processing unit 1 sends multiple sets of model data to processing unit 2, the transmission order of the multiple sets of model data can be determined first, for example, by determining the transmission order based on the size of the tensors in each set of model data. In one example, the priority of the transmission order is determined according to the tensors in each set of model data from largest to smallest; the larger the tensor in each set of model data, the higher its transmission order. In another example, the smaller the tensor in each set of model data, the higher its transmission order. It should be noted that the above examples are only used to describe this disclosure and are not intended to limit this disclosure.

[0142] This method enables elastic scaling of identical model instances across cards by transmitting already loaded model instances to processing units that need to load the same model instances via the inter-card interconnection network. This leverages the high transmission rate of the inter-card interconnection network, significantly reducing redeployment time and improving redeployment efficiency.

[0143] The above combination Figure 8 The following is a schematic diagram illustrating examples of elastic scaling of model instances according to some embodiments of this application, in conjunction with... Figure 9A schematic diagram illustrating an example of a registration loading request according to some embodiments of this application.

[0144] In Example 900, the deployment environment includes a user 902, a client 904, and a server (controller / scheduler) 910. Client 904 includes a collector 906 and an interaction layer 908, while server 912 includes an interaction layer 912 and a database 914. Client 904 and server 910 communicate through interaction layers 908 and 912. The collector 906 is responsible for accepting user model instance deployment requests, and the database is used to output the deployment requests and execution status of each client.

[0145] It should be noted that client 904 may include multiple clients, for example, M clients, and each client needs to deploy at least one model instance. In some embodiments, when user 902 requests the generation of N model instances at 916, the number N can be defined by the user, or it can be a pre-set fixed value, such as N being fixed at 5. The steps in box 938 are executed in parallel.

[0146] In the process of deploying machine learning model instances across multiple clients, client 904 first initializes the deployment request at 918, then calls interaction layer 908 at 920 to communicate with interaction layer 912 via Transmission Control Protocol (TCP) at 922. Subsequently, at 924, the server retrieves the deployment request task queue and registers the deployment requests for N model instances, then calls database 914 at 926 to store the deployment requests in database 914 at 928. Finally, at 930, the registration result for the deployment requests is returned, the registration request task queue 932 is returned to the client via TCP at 934, and the registration result is returned to collector 906 in the client at 936.

[0147] This method significantly reduces the time required to deploy model instances and improves the overall efficiency of model instance deployment by registering deployment requests for multiple model instances in parallel during the deployment process.

[0148] The above combination Figure 9 The following is a schematic diagram illustrating an example of a registration loading request according to some embodiments of this application, in conjunction with... Figure 10 A schematic diagram illustrating examples of loading strategies generated and executed according to some embodiments of this application.

[0149] In Example 1000, combined with the previous Example 900, the deployment environment includes, in addition to user 1002, client 1004, server (controller / scheduler) 1010, interaction layer 1008, interaction layer 1012, and database 1016, an executor 1006, a task tracker 1014, and a task generator 1018. The executor 1006 executes specific loading steps during model deployment, the task tracker 1014 tracks the deployment progress of each model instance, and the task generator 1018 analyzes and generates specific model loading steps. The server 1010 receives loading requests from client 1004, generates corresponding loading strategies based on the loading status of machine learning models in client 1004, and can monitor the loading status of model instances on the client.

[0150] In some embodiments, upon receiving a deployment request from client 1004, the deployment environment will repeatedly execute the steps in blocks 1020 and 1032. Additionally, the time interval for this loop is defined by the user, for example, a loop interval of 1 second. The determination of the end of the loop can be achieved through resource consumption in the deployment environment or the completion status of deployment tasks. For example, the loop stops if no new deployment request is received within a predetermined time and all current deployment requests have been completed; for instance, the loop stops if no new deployment request is received within 1 minute and all current deployment requests have been completed.

[0151] During the generation of the loading strategy, task generator 1018 first queries the database 1016 for deployment requests at point 1022. Then, database 1016 returns the query results to task generator 1018 at point 1024. Upon receiving the results, task generator 1018 generates a loading strategy for the deployment request at point 1026 and decomposes the loading strategy so that the client can execute different loading strategies in parallel. After generating the loading strategy, task generator 1018 queries the deployment request again at point 1028. Database 1016 returns the task execution status to task generator 1030. This iterative query allows for continuous monitoring of whether any new deployment requests from clients have been registered.

[0152] Meanwhile, task tracker 1014 queries the task at 1034. By querying the generated loading strategy, the database returns the generated loading strategy task to task tracker 1014 at 1036. Subsequently, at 1038, task tracker 1014 calls interaction layer 1012 to conduct TCP communication with interaction layer 1008 of client 1004 at 1040, and sends the loading strategy task generated by server 1010 to interaction layer 1008.

[0153] Then, after receiving the loading strategy task generated by the server 1010, the interaction layer 1008 sends the loading strategy task to the executor 1006 at 1042, and then the executor 1006 executes the loading strategy task at 1044.

[0154] After the loading strategy task is completed, at point 1046, executor 1006 sends the execution result to interaction layer 1008. Then, at point 1048, interaction layer sends the execution result to server 1010 via TCP communication with interaction layer 1012. Subsequently, at point 1050, server 1010 writes the execution result to database 1016. Finally, at point 1052, client 1004 sends a task completion notification to user 1002.

[0155] In this way, by formulating a loading strategy for model loading during the model deployment process, the transmission of model data during the deployment process can be dynamically scheduled. It can also monitor the resource consumption status of the entire system, the progress of model loading, and the data processing tasks executed in the system, thereby improving the resource utilization of the entire system and giving full play to the performance of the equipment and the bandwidth in the link.

[0156] Figure 11 A block diagram of a model loading apparatus 1100 according to an embodiment of this application is further shown. The apparatus 1100 is applied to a computing device and may include multiple modules for performing tasks such as... Figure 2 The corresponding steps in process 200 discussed herein. In some embodiments, apparatus 1100 may be adapted to Figure 1 The system architecture 100 shown is... Figure 4 In the example environment 400 shown, execution Figures 3A to 3C The method shown. Figure 11 As shown, the device 1100 includes: a data transceiver module 1102 and a data processing module 1104, wherein the data transceiver module is used for the first group of processing units to receive a first group of model data from the second group of processing units, the first group of model data being at least one group of model data among multiple groups of model data corresponding to the first machine learning model; and the data processing module is used for the first group of processing units to load the first machine learning model according to the first group of model data.

[0157] In some embodiments, the data processing system further includes a storage node, and the first set of model data is received from the storage node by the second set of processing units. The device 1100 further includes: a data transceiver module 1102 is further configured to, in response to the absence of model data of the first machine learning model in the multiple sets of processing units, receive the second set of model data from the storage node, wherein the second set of model data is at least one set of model data that is different from the first set of model data among the multiple sets of model data corresponding to the first machine learning model, and the first set of model data is the model data received from the storage node by the second set of processing units.

[0158] In some embodiments, the first set of processing units loads a first machine learning model based on a first set of model data, including: the data processing module 1104 is further configured to load the first machine learning model based on the first set of model data and the second set of model data.

[0159] In some embodiments, the device 1100 further includes a data transceiver module 1102, which is also used to send a second set of model data from the first set of processing units to the second set of processing units.

[0160] In some embodiments, the apparatus 1100 further includes: a data processing module 1104 is further configured to determine a second set of processing units that have loaded the first machine learning model in response to model data of the first machine learning model existing in the plurality of processing units.

[0161] In some embodiments, the plurality of processing units further includes a third set of processing units, and the apparatus further includes: a data processing module 1104 further configured to, in response to the presence of model data of a first machine learning model in the plurality of processing units, determine a third set of processing units that have loaded the first machine learning model, wherein the third set of processing units is at least one set of processing units different from the second set of processing units; and a data transceiver module 1102 further configured to, wherein the first set of processing units receives a third set of model data from the third set of processing units, wherein the third set of model data is at least one set of model data different from the first set of model data among the plurality of model data corresponding to the first machine learning model.

[0162] In some embodiments, the first set of processing units loads a first machine learning model based on a first set of model data, including: the data processing module 1104 is further configured to load the first machine learning model based on the first set of model data and a third set of model data.

[0163] In some embodiments, wherein multiple sets of model data are obtained by processing multiple tensors of a first machine learning model, the apparatus 1100 further includes: a data processing module 1104 further configured to determine multiple tensors of the first machine learning model; the data processing module 1104 further configured to divide the multiple tensors in sequence to determine multiple fragment files; and a data transceiver module 1102 further configured to receive multiple sets of model data based on the multiple fragment files.

[0164] In some embodiments, the device 1100 further includes a data transceiver module 1102, which is also used for multiple processing units to receive multiple sets of model data from the storage node through multiple processes.

[0165] In some embodiments, the apparatus 1100 further includes: a data transceiver module 1102 further configured to receive a loading request for loading the first machine learning model; a data processing module 1104 further configured to determine, based on the loading request, whether the first machine learning model has been loaded by at least one group of processing units among a plurality of processing units; the data processing module 1104 further configured to, in response to the first machine learning model not being loaded by at least one group of processing units among a plurality of processing units, obtain a first loading strategy for loading the first machine learning model; the data processing module 1104 further configured to, in response to the first machine learning model being loaded by at least one group of processing units among a plurality of processing units, obtain a second loading strategy for loading the first machine learning model; and the data processing module 1104 further configured to, based on the first loading strategy or the second loading strategy, load the first machine learning model.

[0166] Figure 12 A schematic block diagram of an example device 1200 that can be used to implement embodiments of the present application is shown. For example, according to embodiments of the present application... Figure 4 The data processing system 102 can be implemented by example device 1200. As shown, device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 1202 or loaded from storage unit 1208 into random access memory (RAM) 1203. Various programs and data required for the operation of device 1200 can also be stored in RAM 1203. CPU 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0167] Multiple components in device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of monitors, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0168] The various processes and procedures described above, such as methods 300 to 300E, can be executed by processing unit 1201. For example, in some embodiments, method 300 can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by CPU 1201, one or more actions of methods 300A to 300E described above can be performed.

[0169] This application may be a method, apparatus, system, chip, and / or computer program product. A chip may include a processing unit and a communication interface, the processing unit being capable of processing program instructions received from the communication interface. A computer program product may include a computer-readable storage medium on which computer-readable program instructions for performing various aspects of this application are stored.

[0170] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0171] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0172] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0173] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0174] These computer-readable program instructions can be provided to the processing unit of a general-purpose computer, a special-purpose computer, or other programmable model loading apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable model loading apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable model loading apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0175] Computer-readable program instructions may also be loaded onto a computer, other programmable model loading device, or other device to cause a series of operational steps to be performed on the computer, other programmable model loading device, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable model loading device, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0176] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0177] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A data processing method, characterized in that, The data processing system includes multiple sets of processing units, including a first set of processing units and a second set of processing units. The method includes: The first group of processing units receives a first group of model data from the second group of processing units, wherein the first group of model data is at least one group of model data from multiple groups of model data corresponding to the first machine learning model; and The first set of processing units loads the first machine learning model based on the first set of model data.

2. The method according to claim 1, characterized in that, The data processing system further includes a storage node, and the method further includes: In response to the absence of model data for the first machine learning model in the plurality of processing units, the first processing unit receives a second set of model data from the storage node. The second set of model data is at least one set of model data that is different from the first set of model data among the plurality of model data corresponding to the first machine learning model. The first set of model data is the model data received by the second processing unit from the storage node.

3. The method according to claim 2, characterized in that, The first set of processing units loads the first machine learning model based on the first set of model data, including: The first set of processing units loads the first machine learning model based on the first set of model data and the second set of model data.

4. The method according to claim 2 or 3, characterized in that, Also includes: The first group of processing units sends the second group of model data to the second group of processing units.

5. The method according to claim 1, characterized in that, The method further includes: In response to the presence of model data for the first machine learning model in the plurality of processing units, the second group of processing units that has loaded the first machine learning model is determined.

6. The method according to claim 5, characterized in that, The plurality of processing units further includes a third processing unit, and the method further includes: In response to the presence of data for the first machine learning model in the plurality of processing units, a third group of processing units that has been loaded with the first machine learning model is determined, wherein the third group of processing units is at least one group of processing units different from the second group of processing units; and The first group of processing units receives the third group of model data from the third group of processing units. The third group of model data is at least one group of model data that is different from the first group of model data among multiple groups of model data corresponding to the first machine learning model.

7. The method according to claim 6, characterized in that, The first set of processing units loads the first machine learning model based on the first set of model data, including: The first set of processing units loads the first machine learning model based on the first set of model data and the third set of model data.

8. The method according to any one of claims 1-7, characterized in that, The multiple sets of model data are obtained by processing multiple tensors of the first machine learning model, and the method further includes: Determine multiple tensors of the first machine learning model; The multiple tensors are divided sequentially to determine multiple fragmented files; and Based on the multiple fragmented files, the multiple sets of model data are received.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: The multiple processing units receive multiple sets of model data from the storage node through multiple processes.

10. The method according to claim 1, characterized in that, The method further includes: Receive a loading request for loading the first machine learning model; Based on the loading request, it is determined whether the first machine learning model has been loaded by at least one of the multiple sets of processing units; In response to the fact that the first machine learning model has not been loaded by at least one group of processing units among the plurality of processing units, a first loading strategy for loading the first machine learning model is obtained; In response to the first machine learning model having been loaded by at least one group of processing units among the plurality of processing units, a second loading strategy for loading the first machine learning model is obtained; and The first machine learning model is loaded based on either the first or the second loading strategy.

11. A first device, characterized in that, The first device includes a data transceiver module and a data processing module, wherein: The data transceiver module is configured to allow the first group of processing units to receive a first group of model data from the second group of processing units, wherein the first group of model data is at least one group of model data among multiple groups of model data corresponding to a first machine learning model; and The data processing module is used by the first group of processing units to load the first machine learning model based on the first group of model data.

12. The apparatus according to claim 11, characterized in that, The data processing system also includes a storage node, wherein the first set of model data is received by the second set of processing units from the storage node, and further includes: The data transceiver module is further configured to respond to the absence of model data for the first machine learning model in the plurality of processing units, wherein the first processing unit receives a second set of model data from the storage node, wherein the second set of model data is at least one set of model data that is different from the first set of model data among the plurality of model data corresponding to the first machine learning model; and the first set of model data is the model data received by the second processing unit from the storage node.

13. The apparatus according to claim 12, characterized in that, The first set of processing units loads the first machine learning model based on the first set of model data, including: The data processing module is also used by the first group of processing units to load the first machine learning model based on the first group of model data and the second group of model data.

14. The apparatus according to claim 12 or 13, characterized in that, Also includes: The data transceiver module is also used to send the second set of model data from the first set of processing units to the second set of processing units.

15. The apparatus according to claim 11, characterized in that, Also includes: The data processing module is further configured to, in response to the presence of data of the first machine learning model in the plurality of processing units, determine the second group of processing units that have loaded the first machine learning model.

16. The apparatus according to claim 15, characterized in that, The multiple processing units also include a third processing unit, which further includes: The data processing module is further configured to, in response to data containing the first machine learning model in the plurality of processing units, determine the third group of processing units that has loaded the first machine learning model, wherein the third group of processing units is at least one group of processing units different from the second group of processing units; and The data transceiver module is further configured to receive the third set of model data from the third set of processing units, wherein the third set of model data is at least one set of model data that is different from the first set of model data among multiple sets of model data corresponding to the first machine learning model.

17. The apparatus according to claim 16, characterized in that, The first set of processing units loads the first machine learning model based on the first set of model data, including: The data processing module is also used by the first group of processing units to load the first machine learning model based on the first group of model data and the third group of model data.

18. The apparatus according to claims 11-17, characterized in that, Also includes: The data processing module is also used to determine multiple tensors of the first machine learning model; The data processing module is further configured to divide the multiple tensors sequentially to determine multiple fragment files; and The data transceiver module is also used to receive the multiple sets of model data based on the multiple fragmented files.

19. The apparatus according to claims 11-18, characterized in that, Also includes: The data transceiver module is also used by the multiple processing units to receive multiple sets of model data from the storage node through multiple processes.

20. The apparatus according to claim 11, characterized in that, Also includes: The data transceiver module is also used to receive a loading request for loading the first machine learning model; The data processing module is further configured to determine, based on the loading request, whether the first machine learning model has been loaded by at least one of the multiple sets of processing units; The data processing module is further configured to, in response to the fact that the first machine learning model has not been loaded by at least one group of processing units among the plurality of processing units, obtain a first loading strategy for loading the first machine learning model; The data processing module is further configured to, in response to the first machine learning model having been loaded by at least one group of processing units among the plurality of processing units, obtain a second loading strategy for loading the first machine learning model; and The data processing module is also used to load the first machine learning model based on the first loading strategy or the second loading strategy.

21. An electronic device, comprising: At least one processor; as well as A storage device for storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the method according to any one of claims 1-10.

22. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method according to any one of claims 1-10.

23. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1-10.

24. A data processing system comprising a plurality of processing units, wherein at least one of the plurality of processing units is configured to implement the method according to any one of claims 1-10.

Images