Model file loading method and device and related equipment

Abstract

The invention provides a model file loading method and device and related equipment, and the method comprises the steps that a unified control layer divides a model file into a plurality of sub-files according to the composition structure and / or the size of the model file; loading the plurality of sub-files through a plurality of reasoning node fragments; and finally, sending information to the reasoning node running the model to indicate the reasoning node running the model to gather a plurality of sub-files from a plurality of reasoning nodes so as to obtain a model file. In the method, when the reasoning node corresponding to the model needs to obtain the file corresponding to the model, the multiple segmented files only need to be gathered from the multiple reasoning nodes, and compared with the mode that each reasoning node independently obtains the file corresponding to the model from persistent storage, the method can obtain the file corresponding to the model under the condition that the bandwidth of the persistent storage is not changed. And the total time for loading the model file by the plurality of inference nodes is reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model file loading method, device and related equipment. Background Art

[0002] In the current field of artificial intelligence technology, due to the continuous expansion and increasing complexity of business, a model often needs to correspond to a large number of inference nodes. For example, a deep learning model often needs to correspond to hundreds of inference nodes.

[0003] Since each inference node needs to load the file corresponding to the model from the persistent storage and initialize the model structure before providing inference services, all inference nodes running a certain model need to obtain the file corresponding to the model from the persistent storage. When the bandwidth of the persistent storage is low, the total time for all inference nodes to obtain the file corresponding to the model is long. For example, there is a file corresponding to model A in the persistent storage, and the size of the file corresponding to model A is 100 Gigabytes (GB). The number of inference nodes corresponding to model A is 100. When these 100 inference nodes start the inference process to process the business, the total size of the file that needs to be loaded from the persistent storage is 100×100=10000GB. When the bandwidth of the persistent storage is 50GB per second, the total time required for all inference nodes corresponding to model A to obtain the file corresponding to model A is 10000÷50=200 seconds. Summary of the invention

[0004] The present application provides a model file loading method, device and related equipment, which divides the files corresponding to the model and loads the divided files through multiple inference nodes, so that the inference nodes corresponding to the model can aggregate multiple segmented files from multiple inference nodes, and then obtain the files corresponding to the model, thereby reducing the total time for multiple inference nodes to load the model files while keeping the bandwidth of persistent storage unchanged.

[0005] In a first aspect, the present application provides a model file loading method, the method comprising: a unified control layer divides a first file into multiple second files, wherein the first file is a file corresponding to a first model; the unified control layer sends first information to multiple first nodes, so that the multiple first nodes load multiple second files, wherein one first node among the multiple first nodes loads at least one second file among the multiple second files, and the multiple first nodes are used to run one or more models including the first model; the unified control layer sends second information to a second node, so that the second node aggregates multiple second files from multiple first nodes, and obtains the first file based on the multiple second files, wherein the second node is used to run the first model, and the multiple first nodes include the second node.

[0006] In the existing model file loading method, since each inference node corresponding to the model needs to obtain the file corresponding to the model from the persistent storage, the time calculation formula corresponding to the existing model file loading method can be expressed as:

[0007] In the above formula 1, T1 is used to indicate the total time for multiple inference nodes corresponding to the model to obtain the files corresponding to the model from the persistent storage; V is used to indicate the size of the model; N is used to indicate the number of inference nodes corresponding to the model; hp_bw is used to indicate the bandwidth of the persistent storage. For example, when the size of the model is 100GB, the number of inference nodes corresponding to the model is 100, and the bandwidth of the persistent storage is 50GB per second, according to formula 1, it can be known that the total time for multiple inference nodes corresponding to the model to obtain the files corresponding to the model from the persistent storage is 100×100÷50=200 seconds.

[0008] In the model file loading method shown in the first aspect above, by segmenting the file corresponding to a certain model, multiple sub-files corresponding to the model are obtained, and then the multiple sub-files are loaded in pieces through multiple inference nodes, so that each of the multiple inference nodes retains part of the file of the model. When some inference nodes need to obtain the complete file corresponding to the model, they only need to aggregate the sub-files corresponding to the model from other multiple inference nodes to obtain the complete file corresponding to the model. Therefore, the time calculation formula corresponding to the model file loading method provided in the present application can be expressed as:

[0009] In the above formula 2, T2 is used to indicate the total time for multiple inference nodes corresponding to the model to obtain the files corresponding to the model; V is used to indicate the size of the model; N is used to indicate the number of inference nodes corresponding to the model; hp_bw is used to indicate the bandwidth of persistent storage; hh_bw is used to indicate the bandwidth between each inference node; a is used to indicate the start time required to establish a communication connection between each inference node. V÷hp_bw is used to indicate the time taken by multiple inference nodes to load multiple sub-files corresponding to the model from persistent storage; 2×V÷hh_bw is used to indicate the time taken by each inference node among the multiple inference nodes to transmit the sub-files corresponding to the model to other inference nodes and receive the sub-files corresponding to the model transmitted by other inference nodes. For example, when the size of the model is 100GB, the number of inference nodes corresponding to the model is 100, the bandwidth of persistent storage is 50GB per second, the bandwidth between each inference node is 10GB per second, and the start time required to establish a communication connection between each inference node is 60 seconds, according to Formula 2, the total time for multiple inference nodes corresponding to the model to obtain the file corresponding to the model is 100÷50+2×100×99÷100÷10+60≈82 seconds.

[0010] By implementing the above solution, the file corresponding to the model is divided into multiple files, and then the multiple split files are loaded through multiple inference nodes in slices, so that the inference node corresponding to the model can gather multiple split files from multiple inference nodes, and then obtain the file corresponding to the model. Under the condition that the bandwidth of persistent storage remains unchanged, the total time for multiple inference nodes to load model files is reduced.

[0011] In a possible implementation of the first aspect, the method further includes: the unified control layer sends third information to multiple first nodes, wherein the third information is used to instruct the multiple first nodes to delete the loaded files when the time for which the loaded files have not been used exceeds a first threshold.

[0012] In the above solution, if a file has not been used for a long time, it means that multiple inference nodes will no longer apply the model corresponding to the file in a short period of time, that is, the file is likely to be idle. By deleting idle files, the memory space occupied by the idle files can be released in time, so that the storage resources of each inference node can be used more efficiently.

[0013] Implementing the above solution can ensure the timeliness of files loaded by multiple inference nodes, make the files loaded by multiple inference nodes match the current business, and avoid the problem that the files corresponding to the model cannot be loaded due to insufficient memory of some inference nodes, thereby ensuring the implementation of the model file loading method.

[0014] In a possible implementation of the first aspect, the unified control layer divides the first file into multiple second files, including: the unified control layer divides the first file into multiple second files according to the number of first fragments and the size of the first file and / or the composition structure of the first file, wherein the size of the multiple second files is less than or equal to the number of first fragments.

[0015] In the above scheme, since the system (unified control layer) divides the file corresponding to the model into multiple sub-files in various ways, it is very likely that there will be a huge size difference between the multiple sub-files after the division. When the multiple reasoning nodes corresponding to the model are loading these multiple sub-files, some reasoning nodes may be overloaded and some reasoning nodes may be too idle. For example, taking the file size corresponding to the model as 60GB and the number of reasoning nodes corresponding to the model as 3, the system can divide the file corresponding to the model into 3 sub-files, where the size of sub-file 1 is 10GB, the size of sub-file 2 is 20GB, and the size of sub-file 3 is 30GB. When the 3 reasoning nodes load these 3 sub-files in pieces, the reasoning node that loads sub-file 1 will first complete the loading task, enter the idle state, and wait for the loading of sub-file 2 and sub-file 3; the reasoning node that loads sub-file 2 will then complete the loading task, enter the idle state, and wait for the loading of sub-file 3; the reasoning node that loads sub-file 3 will finally complete the loading task.

[0016] By implementing the above solution, we can unify the size of the model file divisions and ensure that the sizes of multiple sub-files after division are the same or close, thereby ensuring that the file sizes loaded and processed by each inference node are roughly the same. This not only makes full use of the memory resources of each inference node and improves the efficiency of model file loading, but also reduces the situation where some inference nodes are overloaded and some nodes are too idle.

[0017] In a possible implementation of the first aspect, before the unified control layer sends the second information to the second node, the method also includes: the unified control layer receives a first quantity, the first quantity is used to indicate the number of nodes running the first model; the unified control layer sends the second information to the second node, so that the second node aggregates multiple second files from multiple first nodes, and obtains the first file based on the multiple second files, including: the unified control layer sends the second information to multiple second nodes with a first number, so that the multiple second nodes aggregate multiple second files from multiple first nodes, and obtain the first file based on the multiple second files.

[0018] In the above scheme, since the scale of the business to be processed by the entire system (unified control layer) is unknown, in most cases, when the system processes a business related to a certain model, it will start all the inference nodes corresponding to the model to ensure that the business is completed in a shorter time. Since the scale of the business varies, each time all the inference nodes corresponding to the model are started, more or less idle inference nodes may be generated, resulting in a waste of computing resources. For example, business A and business B both need to be processed using model C. The scale of business A is small and requires 2 inference nodes. The scale of business B is large and requires 9 inference nodes. The number of inference nodes corresponding to model C in the system is 10. Then when the system processes business B, the number of idle inference nodes is 1; when processing business A, the number of idle inference nodes is 8.

[0019] By implementing the above solution, by obtaining the number of inference nodes that need to be started, and then starting the inference nodes according to the number, computing resources can be accurately allocated according to actual business needs to avoid idleness or waste of computing resources, thereby improving the utilization rate of computing resources.

[0020] In a possible implementation of the first aspect, before the unified control layer divides the first file into multiple second files, the method also includes: the unified control layer receives the file name of the first file, and obtains one or more of the size of the first file and the composition structure of the first file according to the file name of the first file; after the unified control layer sends the second information to the second node, the method also includes: the unified control layer sends the Internet Protocol IP address of the second node to the sender of the file name of the first file.

[0021] In the above scheme, when a user needs to process a certain business, he can send the name of the file corresponding to the model involved in the business to the system (unified control layer). After the system loads the model file to the corresponding reasoning node through the method provided by this application, it returns the Internet Protocol (IP) address of the reasoning node, so that the user can call the corresponding reasoning node according to the IP address to realize the processing of the business.

[0022] By implementing the above solution and managing the input and output of model file loading, when an exception occurs (for example, a connection problem occurs between an inference node and persistent storage or a connection cannot be established with other inference nodes), the entire system operation process can be quickly traced back with the information recorded in the system, the problem link can be identified and repaired in time. For example, when a user uses an inference node to process business, if a certain inference node is found to be unable to work properly, the problem at the inference node can be quickly located and repaired based on the IP address of the inference node.

[0023] In a second aspect, the present application provides a model file loading device, the device comprising: a segmentation module, used to divide a first file into multiple second files, wherein the first file is a file corresponding to the first model; a sending module, used to send first information to multiple first nodes, so that the multiple first nodes load multiple second files, wherein one first node among the multiple first nodes loads at least one second file among the multiple second files, and the multiple first nodes are used to run one or more models including the first model; the sending module is also used to send second information to the second node, so that the second node aggregates multiple second files from the multiple first nodes, and obtains the first file based on the multiple second files, wherein the second node is used to run the first model, and the multiple first nodes include the second node.

[0024] In a possible implementation of the second aspect, the sending module is further used to: send third information to the multiple first nodes, wherein the third information is used to instruct the multiple first nodes to delete the loaded file when the loaded file has not been used for a period of time exceeding a first threshold.

[0025] In a possible implementation of the second aspect, the segmentation module is specifically used to: divide the first file into multiple second files according to the number of first fragments and the size of the first file and / or the composition structure of the first file, wherein the size of the multiple second files is less than or equal to the number of first fragments.

[0026] In a possible implementation of the second aspect, the device also includes an acquisition module, which is used to: receive a first number, where the first number is used to indicate the number of nodes running the first model; and a sending module, which is specifically used to: send second information to a plurality of second nodes whose number is the first number, so that the plurality of second nodes aggregate a plurality of second files from a plurality of first nodes, and obtain the first file based on the plurality of second files.

[0027] In a possible implementation of the second aspect, the acquisition module is specifically used to: receive the file name of the first file, and obtain one or more of the size of the first file and the composition structure of the first file according to the file name of the first file; the sending module is also used to send the IP address of the second node to the sender of the file name of the first file.

[0028] In a third aspect, the present application provides a computing device, comprising a processor and a memory, wherein the processor is used to execute instructions stored in the memory so that the computing device executes a model file loading method as in the first aspect and any possible implementation of the first aspect.

[0029] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device, enables the computing device to execute a model file loading method as in the first aspect and any possible implementation of the first aspect.

[0030] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the model file loading method in the first aspect and any possible implementation of the first aspect.

[0031] The above-mentioned second aspect, third aspect, fourth aspect and fifth aspect all have various possible designs similar to the first aspect and any possible implementation method of the first aspect, and can produce corresponding technical effects, which will not be elaborated here.

[0032] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of a model file loading process involved in this application; Figure 2 This is a schematic diagram of a model file loading process provided by this application; Figure 3 This is a schematic diagram of a model file loading method provided by this application; Figure 4 It is a schematic diagram of the process of a reasoning node aggregation model file provided by this application; Figure 5 It is a schematic diagram of the architecture of a telemetry device for a network service provided by the present application; Figure 6 It is a structural schematic diagram of a computing device provided by this application. DETAILED DESCRIPTION

[0034] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] With the continuous development of artificial intelligence technology, the business covered by this field is becoming increasingly broad and the business scale is also gradually expanding. For example, disease diagnosis can be achieved through medical images, intelligent customer service can be achieved with the help of language processing, and road condition prediction and vehicle dispatch can be achieved based on traffic information.

[0036] The model is the core part of artificial intelligence, which is used to process the input data to output the expected results. For example, the input of the disease prediction model can be multiple pictures, and the output is the corresponding disease name. The inference node is a node that performs specific operations, logical judgments, and other operations to realize the inference process from input to output of the model. That is, the inference node can load the file corresponding to the model to realize the calculation operation corresponding to the model. Since the workload of a single inference node is limited and can only process a certain amount of data, when the business scale is large, it is often necessary to invest multiple inference nodes that load the same model file to process it. That is, when the business scale is large, a model often needs to correspond to a large number of inference nodes.

[0037] The files corresponding to the model (model files) are usually stored in persistent storage. By connecting to persistent storage, the inference node can remotely load the model files stored in persistent storage. Persistent storage is a storage method for long-term data preservation. Common persistent storage includes hard disks and disks. By storing the files corresponding to the model in persistent storage, it is not only convenient for the inference node to load at any time, but also ensures that the model files will not be lost due to system restart, system network failure, etc.

[0038] The current model file loading method is that multiple inference nodes load the corresponding model files from the persistent storage when they need to process business. Figure 1 , Figure 1 This is a schematic diagram of a model file loading process involved in this application, such as Figure 1 As shown, in Figure 1 It includes an inference cluster (including inference node 1, inference node 2, ..., inference node n) and persistent storage, wherein the inference nodes in the inference cluster are all inference nodes corresponding to model A; and the files corresponding to model A are stored in the persistent storage.

[0039] according to Figure 1It can be seen that when the system needs to process business related to model A, n inference nodes need to load the files corresponding to model A from the persistent storage in slices, and initialize the loaded files before processing the business. Since n inference nodes load the files corresponding to model A from the persistent storage at the same time, when the number of inference nodes is large and the bandwidth of the persistent storage is small, the total time for these n inference nodes to load the files corresponding to model A will be longer. For example, when the file size corresponding to model A is 200GB, the size of n is 200, and the bandwidth of the persistent storage is 50GB per second, the total size of the files loaded from the persistent storage by all inference nodes corresponding to model A is 200×200=40000GB, so the total time for all inference nodes corresponding to model A to load the files corresponding to model A from the persistent storage is 40000÷50=800 seconds. It should be understood here that due to the large number of inference nodes, when multiple inference nodes load files in the persistent storage at the same time, the amount of data that these multiple inference nodes can load from the persistent storage per second is related to the bandwidth of the persistent storage.

[0040] In order to solve the problem of long model file loading time in the current model file loading method, the present application provides a model file loading method, device, chip and related equipment.

[0041] Among them, the model file loading method includes: the unified control layer divides the model file into multiple sub-files according to the composition structure and / or size of the model file; then loads the multiple sub-files in pieces through multiple inference nodes; finally, sends information to the inference node running the model to instruct the inference node running the model to gather multiple sub-files from multiple inference nodes, and then obtain the model file. In the above method, when the inference node corresponding to the model needs to obtain the file corresponding to the model, it only needs to gather multiple segmented files from multiple inference nodes. Compared with each inference node separately obtaining the file corresponding to the model from the persistent storage, the above method can reduce the total time for multiple inference nodes to load the model file while keeping the bandwidth of the persistent storage unchanged.

[0042] See also Figure 2 , Figure 2 This is a schematic diagram of the model file loading process provided by this application, such as Figure 2 As shown, Figure 2 It includes a unified control layer 201, a persistent storage 202 and an inference cluster 203 (including inference node 1, inference node 2, ..., inference node n), wherein the inference nodes in the inference cluster 203 are all inference nodes corresponding to model A; the persistent storage 202 stores files corresponding to model A; the unified control layer 201 is connected and communicated with the inference cluster 203 and the persistent storage 202.

[0043] exist Figure 2 In the structure shown, the inference cluster 203 is used to load the file corresponding to model A stored in the persistent storage 202, and after loading the file corresponding to model A, execute the calculation operation corresponding to model A; the persistent storage 202 is used to store files corresponding to multiple models including the file corresponding to model A; the unified control layer 201 is used to control the inference node to load the file corresponding to model A.

[0044] The above-mentioned reasoning cluster 203, persistent storage 202 and unified control layer 201 can be deployed in a computing device, which includes a virtual machine, a container or a server. Among them, a virtual machine is a virtualization technology implemented at the computer software level, which can enable a single physical computer to create multiple virtual operating systems and program application environments, and each virtual machine can independently run its operating system and application; a container is a lightweight software packaging method used to package an application and the application environment of the application so that the application can run in the same way in different environments. Unlike a virtual machine, a container does not contain a complete operating system, but shares the operating system of the physical computer where it is located, which makes the container more lightweight; a server, that is, a general physical server, includes an ARM server or an X86 server.

[0045] The above-mentioned inference cluster 203, persistent storage 202 and unified control layer 201 can also be deployed in terminal devices, including computer terminal devices, mobile terminal devices and network terminal devices, etc., among which computer terminal devices include personal computers, laptops and tablet computers; mobile terminal devices include smart phones; network terminal devices include routers, switches and modems.

[0046] It is explained here that the above-mentioned reasoning cluster 203, persistent storage 202 and unified control layer 201 can also be deployed on different computing devices or mobile terminals. For example, the reasoning cluster 203, persistent storage 202 and unified control layer 201 can be deployed on different computing devices in the computing device cluster. Whether the reasoning cluster 203, persistent storage 202 and unified control layer 201 are deployed on the same computing device depends on the specific application environment, and this application does not make specific limitations here.

[0047] The following is based on Figure 2 The architecture shown and the combination Figure 3 To introduce the model file loading method provided by this application, see Figure 3 , Figure 3 This is a schematic diagram of a model file loading method provided by this application, such as Figure 3 As shown, the steps of the model file loading method include: S301: The unified control layer 201 obtains metadata of a first file.

[0048] The above-mentioned first file is the file corresponding to model A in persistent storage 202. Here, the file corresponding to model A is used to carry relevant information of model A. The inference node loads the file corresponding to model A and completes the initialization of the model A structure according to the file, and can execute the corresponding operation of model A.

[0049] It should be noted here that the above-mentioned model A does not specifically refer to a certain type or a certain model. Model A can be a model in a variety of academic fields, including computer science, engineering and physics, mathematics and statistics, etc. For example, model A can be a neural network model in the field of computer science; model A can be a thermodynamic model in the field of engineering and physics; model A can be a linear regression model in the field of data and statistics.

[0050] The file corresponding to model A can be built based on one or more frameworks, including TensorFlow framework, PyTorch framework, Scikit learn framework, etc. The model file format under the Tensor Flow framework is the SavedModel format. For example, in a scenario where image recognition is required, the Tensor Flow framework can be used to build a convolutional neural network model for image classification, and then the model is trained through a large number of images. The trained model is finally saved in a SavedModel file, which can represent the information in the above convolutional neural network model, including the number and weight parameters of the convolutional layers, pooling layers, and fully connected layers in the convolutional neural network. When inference node A needs to classify a bunch of pictures, inference node A can load the above SavedModel file and complete the initialization of the convolutional neural network model structure for image classification. At this time, inference node A can classify the pile of pictures.

[0051] The model file format under the PyTorch framework is pth or pkl format. For example, in the scenario where you need to analyze whether a language is positive, you can use the PyTorch framework to build a natural language recognition model, and then train the model with a large amount of positive and negative texts. The trained model is finally saved in a pth file, which can represent the information in the above natural language recognition model. When reasoning node B needs to judge the positivity of a manuscript, you can let reasoning node B load the above pth file and complete the initialization of the natural language recognition model. At this time, reasoning node B can judge the positivity of the manuscript.

[0052] The model file format under the Scikit learn framework is in pickle format. For example, in the scenario where you need to understand the relationship between the sales volume of a product and its price and advertising fee, you can use the Scikit learn framework to build a multiple linear regression model, and train the model with data sets from previous years (including the sales volume of the product, the price of the product, and the advertising fee of the product each month). The trained model is finally saved in a pickle file, which can represent the information in the multiple linear regression model. When the inference node C needs to predict the sales volume of a product based on its price and advertising fee, the inference node C can load the pickle file and complete the initialization of the multiple linear regression model. At this time, the inference node C can predict the sales volume of the product based on its price and advertising fee.

[0053] The metadata of the first file refers to information describing the characteristics and attributes of the file itself. The metadata of the first file includes the composition structure of the first file and / or the file size of the first file. In some specific usage scenarios, the metadata of the first file may also include the file name, file type, creation time, modification time and storage location of the first file. Exemplarily, since the first file is the file corresponding to model A, the metadata of the first file may include: ① file name: model A; ② file size: 100GB; ③ file type: SavedModel; ④ creation time: x month y day; ⑤ modification time: x month y day; ⑥ storage location: E:\model file; ⑦ composition structure: composed of sub-model 1, sub-model 2 and sub-model 3.

[0054] The specific steps of the unified control layer 201 obtaining the metadata of the first file may be: the unified control layer 201 receives a message containing the name of model A sent by the user terminal; the unified control layer 201 obtains the name of model A by parsing the message; the unified control layer 201 obtains the metadata of the file corresponding to model A from the persistent storage 202 according to the name of model A. Exemplarily, when a user needs to predict the sales of a certain product (corresponding to the multivariate linear regression model of the product sales), the user can send the name of the multivariate linear regression model for predicting the product sales to the unified control layer 201 through the terminal, and the unified control layer 201 obtains the metadata of the file corresponding to the model from the persistent storage 202 according to the name of the multivariate linear regression model for predicting the product sales.

[0055] The user terminal may send the message containing the name of model A to the unified control layer 201 by wired or wireless means, wherein the wired means include optical fiber communication, Ethernet, serial and parallel interfaces, etc.; the wireless means include wireless network (Wireless Fidelity, Wi-Fi), Bluetooth, ZigBee, etc. It should be understood that the present application does not limit the way in which the user terminal sends the message information to the unified control layer 201.

[0056] Optionally, the unified control layer 201 stores a metadata table of the model, and the unified control layer 201 periodically obtains the metadata of the model file from the persistent storage 202 to update the metadata table of the model. By setting a table in the unified control layer 201 to record all model metadata in the persistent storage 202, after the user terminal sends a message containing the name of model A to the unified control layer 201, the unified control layer 201 can quickly obtain the metadata of the file corresponding to model A from the metadata table of the model according to the name of model A.

[0057] Table 1 below is an example of a metadata table of a model provided in this application.

[0058] Table 1: Metadata table of the model

[0059] According to Table 1 above, the persistent storage 202 stores a total of three models, including model A, model B, and model C. The file size of model A is 100 GB, the storage location is E:\model file, and the composition structure includes sub-model 1, sub-model 2, and sub-model 3. The file size of model B is 150GB, the storage location is E:\model file, and the composition structure includes sub-model 4 and sub-model 5; the file size of model C is 200GB, the storage location is E:\model file, and the composition structure includes sub-model 6, sub-model 7 and sub-model 8. It should be understood here that the above Table 1 is only an example provided for the introduction of this application and should not be regarded as a specific limitation. The metadata table in the above Table 1 can also include more or fewer elements. For example, the above Table 1 can also include model D or the above Table 1 can also include the file type column, and in a specific scenario, the metadata table of the above model can be stored in the persistent storage 202, and the unified control layer 201 can obtain the metadata of the file corresponding to model A from the metadata table in the persistent storage 202 according to the name of model A.

[0060] S302: The unified control layer 201 divides the first file into a plurality of second files according to the metadata of the first file.

[0061] Since the second files are obtained by dividing the first file, each of the multiple second files inherits part of the content in the first file, and the multiple second files retain the metadata associated with the first file, and generate their own unique metadata. For example, after the first file is divided into multiple second files, each second file will obtain metadata related to itself, such as the file name, file size, and storage location. The unified control layer 201 divides the first file into multiple second files according to the metadata of the first file, specifically including: after obtaining the metadata of the first file, the unified control layer 201 determines the location of the first file in the persistent storage according to the storage location of the first file, and divides the first file into multiple second files according to the composition structure of the first file.

[0062] It is explained here that the number of the above-mentioned multiple second files should be selected according to the actual application scenario. For example, since the rate at which multiple inference nodes load files in pieces is higher than the rate at which a single inference node loads the entire file, theoretically, when the number of divided second files does not exceed the number of all inference nodes in the inference cluster 203, the more divided second files there are, the shorter the time it takes to load the entire first file from the persistent storage 202 using multiple inference nodes. For example, there are 10 inference nodes in the inference cluster, and the bandwidth of each inference node is the same. When the first file is divided into 5 second files, the 5 second files will be loaded by 5 inference nodes; when the first file is divided into 10 second files, the 10 second files can be loaded by 10 inference nodes.

[0063] Optionally, the unified control layer 201 may also divide the first file into multiple second files according to the number of first shards and the size of the first file, and the file size of the multiple second files is less than or equal to the number of first shards. Here, the first number of shards refers to a certain file size, that is, the first file is divided with a unified specification, thereby reducing the file size gap between the multiple second files after the division, and further reducing the time difference between the multiple inference node shards loading the multiple second files.

[0064] It is explained here that in the above-mentioned unified control layer 201, the first file is divided into multiple second files according to the composition structure of the first file. Due to the diversity of the model composition structure, the composition structure of the file (first file) corresponding to model A is also diverse. There are situations where some composition structures account for a large proportion and some structures account for a small proportion. Therefore, the unified control layer 201 divides the first file into multiple second files in various ways according to the composition structure of the first file. It is very likely that there will be a huge size difference between the multiple second files after division. When multiple inference nodes are loading these multiple second files, some inference nodes may be overloaded and some inference nodes may be too idle.

[0065] For example, taking the case where the size of the first file is 100 GB, the number of inference nodes is 4, and the composition structure of the first file includes a file corresponding to sub-model 1 (sub-1 file), a file corresponding to sub-model 2 (sub-2 file), a file corresponding to sub-model 3 (sub-3 file), and a file corresponding to sub-model 4 (sub-4 file), where the size of the sub-1 file is 10 GB, the size of the sub-2 file is 20 GB, the size of the sub-3 file is 30 GB, and the size of the sub-4 file is 40 GB, the first file is divided into sub-1 files according to the composition structure of the first file at the unified control layer 201. After the sub-file, sub-2 file, sub-3 file and sub-4 file are loaded, when the four inference nodes load these four files in pieces, the inference node that loads the sub-1 file will first complete the loading task, enter the idle state, and wait for the loading of the sub-2 file, the sub-3 file and the sub-4 file; the inference node that loads the sub-2 file will then complete the loading task, enter the idle state, and wait for the loading of the sub-3 file and the sub-4 file; the inference node that loads the sub-3 file will then complete the loading task, enter the idle state, and wait for the loading of the sub-4 file; the inference node that loads the sub-4 file will finally complete the loading task.

[0066] By unifying the size of the model file divisions, we can ensure that the sizes of multiple sub-files after division are the same or close, and thus ensure that the file sizes loaded and processed by each inference node are roughly the same. This not only makes full use of the memory resources of each inference node and improves the efficiency of model file loading, but also reduces the situation where some inference nodes are overloaded and some nodes are too idle.

[0067] Exemplarily, taking the first file size of 100GB, the number of inference nodes of 4 and the size of the first shard number of 25GB as an example, after the unified control layer 201 divides the first file into 4 second files according to the first shard number, the size of each second file is 25GB. When the 4 inference nodes load the 4 files in shards, since the bandwidth of each inference node is similar and the size of the loaded second files is also close, the 4 inference nodes can complete the loading task at the same time with minimal time difference. It is explained here that the above examples are only examples provided by this application for illustration. The specific size of the first shard number should be selected according to the actual application scenario. For example, the size of the first shard number can also be 30GB. At this time, the unified control layer 201 divides the first file into 4 second files according to the first shard number. The sizes of the 4 second files are 30GB, 30GB, 30GB and 10GB. At this time, the inference node loading 10GB will complete the loading task first and wait for the loading of other second files. Selecting an appropriate first shard number can reduce the time difference in loading the second file between each inference node.

[0068] Optionally, the unified control layer 201 divides the first file into multiple second files according to the metadata of the first file. The specific steps may also be: the unified control layer 201 obtains the first file from the persistent storage 202 according to the location of the first file; and then divides the first file into multiple second files according to the composition structure of the first file or the number of first shards and the file size of the first file.

[0069] It is explained here that the unified control layer 201 in S302 divides the first file into multiple second files according to the metadata of the first file, which can be a logical division or a physical division, wherein the unified control layer 201 divides the first file in the persistent storage 202 into multiple second files according to the metadata of the first file, which is a logical division; the unified control layer 201 divides the obtained first file into multiple second files according to the metadata of the first file, which is a physical division.

[0070] S303: The unified control layer 201 sends first information to the multiple first nodes, so that the multiple first nodes load the multiple second files.

[0071] The first information is used to instruct the multiple inference nodes (first nodes) to load the multiple second files divided by the unified control layer 201 in segments.

[0072] The unified control layer 201 sends first information to multiple first nodes so that the multiple first nodes load multiple second files, specifically including: after the unified control layer 201 divides the first file in the persistent storage 202 into multiple second files, it sends information to notify multiple inference nodes to load multiple second files in fragments from the persistent storage 202, wherein each inference node loads at least one second file.

[0073] In S303, the multiple inference nodes load the multiple second files divided by the unified control layer 201. Therefore, in the entire model loading system, the time spent in the above S301 to S303 can be expressed as the following formula 3:

[0074] In the above formula three, t1 is used to indicate the time for multiple inference node shards to load multiple second files divided by the unified control layer 201; V is used to indicate the size of model A; hp_bw is used to indicate the bandwidth of persistent storage 202; a1 is used to indicate the time required for the unified control layer 201 to obtain the metadata of the first file and divide the first file into multiple second files according to the metadata of the first file. This time depends on the performance of the unified control layer 201 and the persistent storage 202. In this application, 10 seconds is taken as an example.

[0075] It is explained here that although multiple inference nodes load multiple second files, these multiple second files are divided from the first file, and when multiple inference nodes load files from the persistent storage 202, the file loading rate depends on the bandwidth of the persistent storage. Therefore, V÷hp_bw can represent the time required for the inference cluster 203 (multiple inference nodes) to load multiple second files from the persistent storage 202.

[0076] Optionally, when in S302, the specific steps of the unified control layer 201 dividing the first file into multiple second files according to the metadata of the first file are that the unified control layer 201 obtains the first file from the persistent storage 202 according to the location of the first file; and then divides the first file into multiple second files according to the composition structure of the first file or the number of first shards and the file size of the first file. The unified control layer 201 sends the first information to the multiple first nodes so that the multiple first nodes load the multiple second files, specifically including: the unified control layer 201 sends the multiple divided second files to the multiple inference nodes respectively, so that these inference nodes load the multiple second files in slices.

[0077] S304: The unified control layer 201 sends second information to the second node, so that the second node aggregates multiple second files from multiple first nodes.

[0078] The second information is used to instruct the inference node (second node) corresponding to the model A to aggregate multiple second files from multiple inference nodes, and then form a first file through the multiple second files.

[0079] The unified control layer 201 sends the second information to the second node so that the second node gathers multiple second files from multiple first nodes, specifically including: after the multiple inference nodes have loaded the multiple second files, the unified control layer 201 sends information to notify the inference node corresponding to the model A (the inference node running the model A) to obtain the multiple second files from the multiple inference nodes, and generate the first file based on the multiple second files.

[0080] See also Figure 4 , Figure 4 This is a schematic diagram of the process of inference node aggregation model file provided by this application, such as Figure 4 As shown, after the unified control layer 201 sends information to instruct multiple inference nodes to load multiple second files in slices, each inference node stores a partial file corresponding to model A. For example, Figure 4 The inference node 1 shown in FIG. 1 stores S1 in the file corresponding to the model A; Figure 4 The inference node 2 shown in FIG. 1 stores S2 in the file corresponding to the model A; Figure 4 The inference node 3 shown stores S3 in the file corresponding to model A. After the unified control layer 201 sends information to instruct the inference node corresponding to model A to aggregate multiple second files from multiple inference nodes, the inference node corresponding to model A stores all the files corresponding to model A. For example, Figure 4 The inference node 1 shown in the figure stores S1, S2 and S3 in the file corresponding to the model A; Figure 4 The inference node 2 shown in the figure stores S1, S2 and S3 in the file corresponding to the model A; Figure 4 The inference node 3 shown in FIG. 1 stores S1, S2 and S3 in the file corresponding to model A. Figure 4 Inference node 1, inference node 2, and inference node 3 are all inference nodes of model A.

[0081] In S304, the time taken for the inference node corresponding to the model A to obtain multiple second files from multiple inference nodes and generate the first file according to the multiple second files can be expressed as the following formula 4:

[0082] In the above formula 4, t2 is used to indicate the time taken by the inference node corresponding to model A to obtain multiple second files from multiple inference nodes, and to generate the first file based on the multiple second files; V is used to indicate the size of model A; N is used to indicate the number of inference nodes corresponding to model A; hh_bw is used to indicate the bandwidth of communication between multiple inference nodes; a2 is used to indicate the time taken to establish a connection between multiple inference nodes and the sum of the time taken by the inference node corresponding to model A to generate the first file based on the multiple second files, and 50 seconds is taken as an example in this application. It is explained here that the above formula 4 is obtained when the unified control layer 201 divides the first file into multiple second files according to the number of first shards and the size of the first file, that is, when the file size difference of the second file loaded by each inference node in the multiple inference nodes is small.

[0083] In the above formula 4, since the model A is run by some inference nodes in the partial inference cluster 203, the inference node corresponding to the model A needs to send the second file loaded by itself to all other inference nodes and receive the second file sent by other nodes, and generate the first file according to multiple second files; for the inference node not corresponding to the model A, it needs to send the second file loaded by itself to all other inference nodes, so according to the barrel effect, the time taken for the whole process to be completed depends on the time taken by the inference node corresponding to the model A to send the second file loaded by itself to all other inference nodes and receive the second file sent by other nodes, and generate the first file according to multiple second files. Since the file size of the second file loaded by a single inference node can be expressed as V×(1÷N), the time taken by a single inference node to send the second file loaded by itself to all other inference nodes is V×(1÷N)×(N-1)÷hh_bw; therefore, the time taken by a single inference node to receive the second file sent by all other inference nodes is also V×(1÷N)×(N-1)÷hh_bw.

[0084] Optionally, when the number of inference nodes in the inference cluster 203 is large, for example, the number of inference nodes in the inference cluster 203 exceeds 100, then (N-1) divided by N in the above formula 4 is approximately 1. Therefore, when the number of inference nodes in the inference cluster 203 is large, the time taken for the inference node corresponding to the model A to obtain multiple second files from multiple inference nodes and generate the first file according to the multiple second files can also be expressed as the following formula 5:

[0085] In the above formula 5, the description of t2, V, hh_bw and a2 can refer to the description in the above formula 4, and this application will not elaborate on them here.

[0086] According to Formula 3 and Formula 5, in the model loading method provided in the present application, the time required for the inference node corresponding to model A to load the file corresponding to model A from the persistent storage can be expressed as the following Formula 6:

[0087] In the above formula 6, t1 can refer to the relevant description in the above formula 3, and t2 can refer to the relevant description in the above formula 4 or formula 5, and the present application will not elaborate on them here.

[0088] The following is a specific example to describe the effect of the model loading method provided by the present application on the time taken by multiple inference nodes to load model files.

[0089] In this embodiment, the size of the file corresponding to model A is 200GB; the inference cluster 203 includes 100 inference nodes, and these 100 inferences are all inference nodes corresponding to model A; the bandwidth of persistent storage is 50GB per second; the bandwidth between each inference node is 10GB per second; the sum of the time required to establish a connection between multiple inference nodes and the time required for the inference node corresponding to model A to generate a first file based on multiple second files is 50 seconds (that is, the value of a2 in the above formula four or formula five is 50); the time required for the unified control layer 201 to obtain the metadata of the first file and divide the first file into multiple second files based on the metadata of the first file is 10 seconds (that is, the value of a1 in the above formula three is 10).

[0090] In the existing model file loading method, since each inference node corresponding to the model needs to obtain the file corresponding to the model from the persistent storage, the time calculation formula corresponding to the existing model file loading method can be expressed as:

[0091] In the above formula 1, T1 is used to indicate the total time taken by multiple inference nodes corresponding to model A to obtain the file corresponding to model A from persistent storage; V is used to indicate the size of model A; N is used to indicate the number of inference nodes corresponding to model A; and hp_bw is used to indicate the bandwidth of persistent storage.

[0092] According to Formula 1, when the existing model file loading method is used in this embodiment, the time required for multiple inference nodes corresponding to Model A to obtain the file corresponding to Model A from persistent storage is 200×100÷50=400 seconds.

[0093] According to the above formulas 3, 5 and 6, when the model file loading method provided by the present application is used in this embodiment, the time required for multiple inference nodes corresponding to model A to obtain the file corresponding to model A from persistent storage is 200÷50+2×200÷10+10+50=104 seconds. It is noted here that in actual application scenarios, the values ​​of a1 and a2 above will fluctuate from the values ​​set in this embodiment.

[0094] Optionally, the unified control layer 201 will also send third information to multiple inference nodes, where the third information is used to indicate that the loaded file will be deleted when the time for which the loaded file has not been used exceeds a first threshold. By deleting files that have not been used for a long time in multiple inference nodes, the timeliness of the files loaded by the multiple inference nodes can be guaranteed, so that the files loaded by the multiple inference nodes match the current business, and at the same time, the problem of the inability to load the files corresponding to the model due to insufficient memory of some inference nodes is avoided, thereby ensuring the implementation of the model file loading method.

[0095] It should be understood here that if a file in an inference node has not been aggregated for a long time, it means that multiple inference nodes will no longer apply the model corresponding to the file in a short period of time, that is, these files are likely to be idle. By deleting these idle files, the memory space occupied by these idle files can be released in time, so that the storage resources of each inference node can be used more efficiently.

[0096] Optionally, in S301, the unified control layer 201 also receives a message containing a first number sent by a user terminal, where the first number is used to indicate the number of nodes that need to be started, and the number of inference nodes that model A finally corresponds to is the same. By obtaining the number of inference nodes that need to be started, and then starting the inference nodes according to the number, computing resources are accurately allocated according to actual business needs, avoiding idleness or waste of computing resources, thereby improving the utilization of computing resources.

[0097] It should be understood here that since the scale of the business to be processed by the entire system is unknown, in most cases, when the system processes the business corresponding to a certain model, the unified control layer 201 will start all the inference nodes corresponding to the model to ensure that the business is completed in a shorter time. Since the scale of the business changes unpredictably, each time all the inference nodes corresponding to the model are started, more or less idle inference nodes may be generated, resulting in a waste of computing resources.

[0098] Optionally, after the unified control layer 201 sends information to instruct the inference node corresponding to model A to aggregate multiple second files from multiple inference nodes and generate a first file based on multiple second files, it will also send the IP address of the inference node corresponding to the started model A to the user terminal. By managing the input and output of the model file loading, when an abnormality occurs, the entire system operation process can be quickly traced back with the information recorded in the system, the problem link can be identified and repaired in time. For example, when a user uses an inference node to process business, if a certain inference node is found to be unable to work properly, the problem at the inference node can be quickly located and repaired based on the IP of the inference node.

[0099] It should be understood here that when a user needs to process a certain business, the user can send the name of the file corresponding to the model involved in the business and the number of inference nodes that need to be started as output to the system (unified control layer 201). After the system loads the model file to the corresponding inference node through the method provided in this application, it returns the IP of the inference node, so that the user can call the corresponding inference node through the IP of the inference node to realize the processing of the business.

[0100] In summary, the present application provides a model file loading method, which includes: the unified control layer 201 divides the model file into multiple sub-files according to the metadata of the model file; then instructs the inference cluster 203 (multiple inference nodes) to load the multiple sub-files in pieces from the persistent storage 202; finally, information is sent to the inference node running the model to instruct the inference node running the model to aggregate multiple sub-files from multiple inference nodes, and then obtain the model file. In the above method, when the inference node corresponding to the model needs to obtain the file corresponding to the model, it only needs to aggregate multiple segmented files from multiple inference nodes. Compared with each inference node separately obtaining the file corresponding to the model from the persistent storage, the above method can reduce the total time for multiple inference nodes to load model files while keeping the bandwidth of the persistent storage unchanged.

[0101] For the above method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present application. Other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of the present application. Secondly, those skilled in the art should also be familiar with that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present application.

[0102] See also Figure 5 , Figure 5 This is a schematic diagram of the architecture of a telemetry device for a network service provided by the present application. Figure 5 As shown, Figure 5 The system includes a model file loading device 500, a persistent storage 510 and an inference cluster 520, wherein the model file loading device 500 is used to segment the model file in the persistent storage 510 and control the inference nodes in the inference cluster 520 to load the model file; the persistent storage 510 is used to store files corresponding to multiple models including model B; the inference cluster 520 is used to load the model file and complete the initialization of the corresponding model structure according to the model file. The model file loading device 500 can be further divided into a plurality of unit modules, such as Figure 5 As shown, the model file loading device 500 further includes an acquisition module 501, a segmentation module 502 and a sending module 503. It should be understood that Figure 5 The number and names of the unit modules included in the model file loading device 500 are only examples provided in the present application. The model file loading device 500 may also include more or fewer unit modules, and the names of the unit modules are not limited to Figure 5 The name of the unit module in, for example, the above-mentioned model file loading device 500 may also include a network environment detection module, the network environment detection module is used to detect the transmission status of the network transmission path between the model file loading device 500 and the persistent storage 510, and send the detected result to the sending module 503 and the obtaining module 501; Figure 5 The acquisition module 501 and the sending module 503 in the embodiment can be combined into a communication module; Figure 5 The name of the segmentation module 502 in the embodiment can be changed to the segmentation module. It should be understood that the above examples are only for illustration and should not be regarded as specific limitations.

[0103] Optionally, each inference node in the above-mentioned inference cluster 520 may also include a memory management module and a loading control module, wherein the memory management module is used to manage the memory of the inference node and provide a software development kit (SDK) for the inference node to run the model. When the inference node needs to load a model file, it can use SCK to send a loading request to the memory management module to load the model file in the memory; the loading control module is used to send a loading request to other inference nodes to obtain the model files loaded by other inference nodes, and receive loading requests sent by other inference nodes, and send the model files loaded by the inference node to other inference nodes.

[0104] Among them, the acquisition module 501, the segmentation module 502 and the sending module 503 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the acquisition module 501 is described below by taking the acquisition module 501 as an example. Similarly, the implementation of the segmentation module 502 and the sending module 503 can refer to the implementation of the acquisition module 501.

[0105] As an example of a software functional unit, the acquisition module 501 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance may be one or more. For example, the acquisition module 501 may include code running on multiple hosts / virtual machines / containers.

[0106] As an example of a hardware functional unit, the acquisition module 501 may include at least one computing device, such as a server, etc. Alternatively, the acquisition module 501 may also be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a device implemented by a programmable logic device (PLD). The above-mentioned PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system on chip (SoC), an offload card, an acceleration card, or any combination thereof.

[0107] The acquisition module 501, the segmentation module 502 and the sending module 503 in the model file loading device 500 are introduced below.

[0108] The acquisition module 501 is used to acquire metadata of the file corresponding to the model B in the persistent storage 510 .

[0109] This module can be used to perform the above Figure 3 S301 of the model file loading method.

[0110] The above-mentioned model B does not specifically refer to a certain type or a certain model. Model B can be a model in various academic fields, including computer science, engineering and physics, mathematics and statistics, etc. For example, model B can be a neural network model in the field of computer science; model B can be a thermodynamic model in the field of engineering and physics; model B can be a linear regression model in the field of data and statistics.

[0111] The file corresponding to model B can be built based on one or more frameworks, including TensorFlow framework, PyTorch framework, Scikit learn framework, etc.

[0112] The acquisition module 501 acquires the metadata of the file corresponding to the model B in the persistent storage 510, specifically including: the model file loading device 500 receives a message containing the name of the model B sent by the user terminal; the acquisition module 501 acquires the name of the model B by parsing the message; the acquisition module 501 acquires the metadata of the file corresponding to the model B from the persistent storage 510 according to the name of the model B.

[0113] The user terminal may send the message containing the name of model B to the model file loading device 500 by wired or wireless means, wherein the wired means include optical fiber communication, Ethernet, serial and parallel interfaces, etc.; the wireless means include wireless network, Bluetooth, ZigBee, etc. It should be understood that the present application does not limit the manner in which the user terminal sends the message information to the model file loading device 500.

[0114] Optionally, the model file loading device 500 stores a metadata table of the model, and the metadata table of the model stores metadata of files corresponding to multiple models in multiple persistent storages 510. The acquisition module 501 updates the metadata table of the model by periodically acquiring the metadata of the model from the persistent storage 510. By setting a table for recording metadata of all models in the persistent storage 510 in the model file loading device 500, after the user terminal sends a message containing the name of model B to the model file loading device 500, the acquisition module 501 can quickly acquire the file metadata corresponding to model B from the metadata table of the model according to the name of model B.

[0115] Optionally, the acquisition module 501 also acquires a message containing a first number sent by the user terminal, where the first number is used to indicate the number of nodes that need to be started, that is, the number of inference nodes that model B finally corresponds to is the same as the first number. The model file loading device 500 acquires the number of inference nodes that need to be started, and then starts the inference nodes according to the number, and accurately allocates computing resources according to actual business needs, thereby avoiding idleness or waste of inference node resources, thereby improving system resource utilization.

[0116] It should be understood here that since the scale of the business to be processed by the entire system is unknown, in most cases, when the system is processing the business corresponding to a certain model, the model file loading device 500 will start all the inference nodes corresponding to the model in the inference cluster 520 to ensure that the business is completed in a shorter time. Since the scale of the business changes unpredictably, each time all the inference nodes corresponding to the model are started, more or less idle inference nodes may be generated, resulting in a waste of computing resources.

[0117] The segmentation module 502 is used to divide the files corresponding to the model B in the persistent storage 510 .

[0118] This module can be used to perform the above Figure 3 S302 of the model file loading method.

[0119] The segmentation module 502 divides the file corresponding to the model B in the persistent storage 510 , specifically including: the segmentation module 502 divides the file corresponding to the model B into multiple sub-files corresponding to the model B according to the metadata of the file corresponding to the model B.

[0120] Since the multiple sub-files corresponding to model B are obtained by dividing the file corresponding to model B, the multiple sub-files each inherit part of the content in the file corresponding to model B, and the multiple sub-files retain the related metadata in the file corresponding to model B, and generate their own unique metadata.

[0121] It is explained here that the number of the above-mentioned multiple sub-files should be selected according to the actual application scenario. Since the rate at which multiple inference nodes load files in slices will be higher than the rate at which a single inference node loads the entire file, theoretically, when the number of divided sub-files does not exceed the number of all inference nodes in the inference cluster 520, the more the number of divided second files is, the shorter the time it takes to load the files corresponding to the entire model B from the persistent storage 510 using multiple inference nodes.

[0122] Optionally, the above-mentioned segmentation module 502 can also divide the file corresponding to model B into multiple sub-files according to the first number of shards and the size of the file corresponding to model B, and the file size of the multiple sub-files is less than or equal to the first number of shards. Here, the first number of shards refers to a certain file size, that is, the first file is divided with a uniform specification, thereby reducing the file size gap between the multiple sub-files after the division, and further reducing the time difference between the multiple inference node shards loading the multiple sub-files.

[0123] It is explained here that due to the diversity of the model composition structure, the composition structure of the file corresponding to model B is also diverse. There are situations where some composition structures account for a large proportion and some structures account for a small proportion. Therefore, when the segmentation module 502 divides the file corresponding to model B into multiple sub-files according to the composition structure of the file corresponding to model B, it is very likely that there will be a huge size difference between the multiple sub-files after the division. When multiple inference nodes are loading these multiple sub-files, some inference nodes may be overloaded and some inference nodes may be too idle.

[0124] By unifying the size of the model file divisions, we can ensure that the sizes of multiple sub-files after division are the same or close, and thus ensure that the file sizes loaded and processed by each inference node are roughly the same. This not only makes full use of the memory resources of each inference node and improves the efficiency of model file loading, but also reduces the situation where some inference nodes are overloaded and some nodes are too idle.

[0125] Optionally, the above-mentioned splitting module 502 divides the file corresponding to model B into multiple sub-files according to the metadata of the file corresponding to model B. The specific steps may also be: the splitting module 502 obtains the file corresponding to model B from the persistent storage 510 according to the location of the file corresponding to model B; and then splits the file corresponding to model B into multiple sub-files according to the composition structure of the file corresponding to model B or the first number of fragments and the size of the file corresponding to model B.

[0126] It is explained here that the splitting module 502 divides the file corresponding to model B into multiple sub-files according to the metadata of the file corresponding to model B. The division can be a logical division or a physical division. Among them, the splitting module 502 divides the file corresponding to model B in the persistent storage 510 into multiple sub-files according to the metadata of the file corresponding to model B, which is a logical division; the splitting module 502 splits the acquired file corresponding to model B into multiple sub-files according to the metadata of the file corresponding to model B, which is a physical division.

[0127] The sending module 503 is used to send first information to the inference cluster 520 so that multiple first nodes load multiple sub-files; send second information to the inference cluster 520 so that the second node aggregates multiple sub-files from multiple first nodes; send third information to the inference cluster 520 so that the multiple first nodes delete the loaded sub-files when the time for which the loaded sub-files are not aggregated exceeds a first threshold.

[0128] This module can be used to perform the above Figure 3 S303 and S304 of the model file loading method.

[0129] The first nodes are nodes in the inference cluster 520 , and the number of the first nodes is the same as the number of inference nodes included in the inference cluster 520 .

[0130] The second node mentioned above is the inference node corresponding to model B, that is, the second node is used to load the file corresponding to model B to perform corresponding operations.

[0131] The sending module 503 sends the first information to the inference cluster 520 so that multiple first nodes load multiple sub-files, specifically including: after the splitting module 502 divides the file corresponding to the model B in the persistent storage 510 into multiple sub-files, the sending module 503 notifies all the inference nodes in the inference cluster 520 to load multiple sub-files in slices from the persistent storage 510 by sending information, wherein each inference node loads at least one sub-file.

[0132] Optionally, when the acquisition module 501 acquires the file corresponding to the model B from the persistent storage 510 according to the location of the file corresponding to the model B; and the segmentation module 502 segments the file corresponding to the model B into multiple sub-files according to the composition structure of the file corresponding to the model B or the number of first shards and the size of the file corresponding to the model B. The sending module 503 sends the first information to the inference cluster 520 so that the multiple first nodes load the multiple sub-files, specifically including: the sending module 503 sends the multiple segmented sub-files to the multiple inference nodes in the inference cluster 520 respectively, so that the multiple inference nodes load the multiple sub-files in segments.

[0133] The sending module 503 sends the second information to the inference cluster 520 so that the second node aggregates multiple sub-files from multiple first nodes, specifically including: after the multiple inference nodes in the inference cluster 520 load the multiple sub-files, the sending module 503 sends information to the inference cluster 520, notifying the inference node corresponding to model B in the inference cluster 520 to obtain the multiple sub-files from the multiple inference nodes, and generate a file corresponding to model B based on the multiple sub-files.

[0134] The method for calculating the time required for the model file loading device 500 to control the inference nodes in the inference cluster 520 to load the file corresponding to model B in the persistent storage 510 can be derived based on the above formulas 3 to 6, and this application will not go into details here.

[0135] The sending module sends the third information to the inference cluster 520, so that the multiple first nodes delete the loaded sub-files when the time for which the loaded sub-files are not aggregated exceeds the first threshold. By deleting the files that have not been used for a long time in the multiple inference nodes, it can be ensured that the files loaded by the multiple inference nodes in the inference cluster 520 have good timeliness, and prevent the inference nodes in the inference cluster 520 from being unable to load the model files due to insufficient memory of some inference nodes when loading the model files.

[0136] It should be understood here that if a file in an inference node has not been aggregated for a long time, it means that multiple inference nodes will no longer apply the model corresponding to the file in a short period of time, and the file is likely to be idle. By deleting idle files, the memory space occupied by these idle files can be released in time, so that the storage resources of each inference node can be used more efficiently.

[0137] Optionally, after sending information to instruct the inference node corresponding to model B to aggregate multiple sub-files from multiple inference nodes and generate a file corresponding to model B based on multiple sub-files, the sending module 503 will also send the IP address of the started inference node corresponding to model B to the user terminal. By managing the input and output of model file loading, when an exception occurs, the entire system operation process can be quickly traced back with the information recorded in the system, the problem link can be determined and repaired in time.

[0138] It should be noted that, in addition to the functions described above, in different embodiments, the functions of each module can be flexibly adjusted. For example, the acquisition module 501 can be used to execute Figure 3 In any step of the model file loading method, the segmentation module 502 can be used to perform Figure 3 The sending module 503 can be used to execute any step in the model file loading method. The steps implemented by the acquisition module 501, the segmentation module 502 and the sending module 503 can be specified as needed. Figure 3 The different steps in the model file loading method are used to realize all the functions of the receiving end.

[0139] In summary, the present application provides a model file loading device, wherein the model file loading device 500 obtains the metadata of the model file through the acquisition module 501; the segmentation module 502 divides the model file into multiple sub-files according to the metadata of the model file; the sending module 503 instructs the inference cluster 520 (multiple inference nodes) to load the multiple sub-files in pieces from the persistent storage 510; finally, the sending module 503 sends information to the inference node running the model to instruct the inference node running the model to gather multiple sub-files from multiple inference nodes, thereby obtaining the model file. By using the above-mentioned model file loading device 500, when the inference node corresponding to the model needs to obtain the file corresponding to the model, it only needs to gather multiple segmented files from multiple inference nodes. Compared with each inference node separately obtaining the file corresponding to the model from the persistent storage, the total time for multiple inference nodes to load the model file can be reduced while the bandwidth of the persistent storage remains unchanged.

[0140] The present application also provides a computing device. Figure 6 , Figure 6 is a schematic diagram of the structure of a computing device provided by the present application, such as Figure 6 As shown, computing device 600 includes: bus 602, processor 604, memory 606 and communication interface 608. Processor 604, memory 606 and communication interface 608 communicate through bus 602. Computing device 600 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 600.

[0141] The bus 602 may be a peripheral component interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. Among them, the unified bus is also called a Lingqu bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 The fact that only one line is used in the diagram does not mean that there is only one bus or only one type of bus.

[0142] The processor 604 may include any one or more computing devices such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP), an ASIC, an FPGA, a CPLD, an NPU, a SoC, an offload card, an acceleration card, etc.

[0143] The memory 606 may include a volatile memory, such as a random access memory (RAM). The processor 604 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). In addition, the memory 606 may also be implemented by a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0144] It is worth noting that the same type of storage medium can be configured in the same computing device to implement the function of memory 606, or two or more types of storage media can be configured to implement the function of memory 606, and this application does not limit this.

[0145] The memory 606 stores executable program codes, and the processor 604 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module 501, the segmentation module 502, and the sending module 503, thereby implementing the aforementioned Figure 3 That is, the memory 606 stores the model file loading method for executing the above Figure 3 Instructions for the model file loading method in .

[0146] The communication interface 608 uses, but is not limited to, a transceiver module such as a network interface card or a transceiver to implement communication between the computing device 600 and the persistent storage and reasoning cluster.

[0147] As a possible implementation method, the computing device 600 may also include a chip system, the chip system includes a processor and a power supply circuit, the power supply circuit is used to power the processor, and the processor is used to execute the operation steps corresponding to the model file loading method. For the sake of brevity, it is not repeated here. Among them, the processor can be implemented by a GPU, and can also be implemented by a computing device or AI chip such as a DPU, NPU, XPU, SoC, offload card, acceleration card, etc.

[0148] As a possible implementation, the computing device 600 may include multiple types of processors 604, that is, the computing device 600 is a heterogeneous device, for example, the computing device 600 includes a CPU and a GPU, and at least one of the processors 604 may perform the operation steps corresponding to the model file loading method. For the sake of brevity, it will not be described here.

[0149] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the model file loading method.

[0150] The present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the model file loading method.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A model file loading method, characterized in that: The method comprises: The unified control layer divides the first file into a plurality of second files, wherein the first file is a file corresponding to the first model; The unified control layer sends first information to a plurality of first nodes, so that the plurality of first nodes load the plurality of second files, wherein one first node among the plurality of first nodes loads at least one second file among the plurality of second files, and the plurality of first nodes are used to run one or more models including the first model; The unified control layer sends second information to the second node so that the second node aggregates the multiple second files from the multiple first nodes and obtains the first file based on the multiple second files, wherein the second node is used to run the first model and the multiple first nodes include the second node.

2. The method according to claim 1, characterized in that The method further comprises: The unified control layer sends third information to the multiple first nodes, wherein the third information is used to instruct the multiple first nodes to delete the loaded files when a time period during which the loaded files have not been used exceeds a first threshold.

3. The method according to claim 1 or 2, characterized in that: The unified control layer divides the first file into a plurality of second files, including: The unified control layer divides the first file into multiple second files according to the number of first fragments, the size of the first file and / or the composition structure of the first file, wherein the size of the multiple second files is less than or equal to the number of the first fragments.

4. The method according to claim 3, characterized in that Before the unified control layer sends the second information to the second node, the method further includes: the unified control layer receiving a first number, the first number being used to indicate the number of nodes running the first model; The unified control layer sends second information to the second node so that the second node aggregates the multiple second files from the multiple first nodes and obtains the first file based on the multiple second files, including: the unified control layer sends second information to a plurality of second nodes whose number is the first number so that the multiple second nodes aggregate the multiple second files from the multiple first nodes and obtain the first file based on the multiple second files.

5. The method according to claim 4, characterized in that Before the unified control layer divides the first file into a plurality of second files, the method further includes: the unified control layer receives a file name of the first file, and acquires one or more of a size of the first file and a component structure of the first file according to the file name of the first file; After the unified control layer sends the second information to the second node, the method further includes: the unified control layer sending the Internet Protocol IP address of the second node to the sender of the file name of the first file.

6. A model file loading device, characterized in that: The device comprises: A segmentation module, used to divide the first file into a plurality of second files, wherein the first file is a file corresponding to the first model; a sending module, configured to send first information to a plurality of first nodes, so that the plurality of first nodes load the plurality of second files, wherein one first node among the plurality of first nodes loads at least one second file among the plurality of second files, and the plurality of first nodes are configured to run one or more models including the first model; The sending module is also used to send second information to the second node so that the second node gathers the multiple second files from the multiple first nodes and obtains the first file based on the multiple second files, wherein the second node is used to run the first model and the multiple first nodes include the second node.

7. The device according to claim 6, characterized in that The sending module is further used for: Sending third information to the multiple first nodes, wherein the third information is used to instruct the multiple first nodes to delete the loaded file when a time period during which the loaded file has not been used exceeds a first threshold.

8. The device according to claim 6 or 7, characterized in that The segmentation module is specifically used for: The first file is divided into a plurality of second files according to the number of first fragments, the size of the first file and / or the composition structure of the first file, wherein the size of the plurality of second files is less than or equal to the number of the first fragments.

9. The device according to claim 8, characterized in that The device also includes an acquisition module, which is used to: receiving a first number indicating a number of nodes running the first model; The sending module is specifically used to send second information to a plurality of second nodes whose number is the first number, so that the plurality of second nodes gather the plurality of second files from the plurality of first nodes and obtain the first file according to the plurality of second files.

10. The device according to claim 9, characterized in that The acquisition module is specifically used for: receiving a file name of the first file, and acquiring one or more items of a size of the first file and a component structure of the first file according to the file name of the first file; The sending module is further configured to send the IP address of the second node to a sender of the file name of the first file.

11. A computing device, characterized in that: The computing device includes a processor and a memory; The processor is configured to execute instructions stored in the memory so that the computing device performs the operation steps of the method according to any one of claims 1 to 5.

12. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the operation steps of the method according to any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the operation steps of the method according to any one of claims 1 to 5.

Images