A model file loading method, system, computer device and storage medium

Abstract

The present invention relates to the field of machine learning technology, and discloses a method, system, computer device and storage medium for loading a model file, the method comprising: obtaining an update request when an inference service is started; if it is detected that the update request carries a local storage path of a model file, using the local storage path as a target storage volume of a target scheduling unit; mounting the target storage volume in an initialization container of the target scheduling unit, generating a target mount item of the initialization container; generating response information based on the target storage volume and the target mount item, the response information is used to update the target scheduling unit to establish a communication link between the updated target scheduling unit and a local directory; sending the response information to a first slave node, so that the first slave node updates the target scheduling unit and loads the model file based on the communication link. The present invention can solve the problems of high transmission delay, redundant occupation of hard disk resources, and namespace limitations during model file loading.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a model file loading method, system, computer equipment and storage medium. Background Art

[0002] With the development of container technology and large language models, the number of model internal parameters and the file size required to store the model continue to increase. The storage and loading requirements of such large-scale models bring about two major technical problems: one is how to efficiently store such a large model file, and the other is how to reduce the latency when loading and reading the model file. The Inference Service is designed to simplify the deployment and service of machine learning models. It provides a simple, scalable, and framework-independent way to turn machine learning models into production-level services. The Inference Service takes advantage of its powerful container orchestration capabilities, making it easy to deploy and manage models in a cloud-native environment.

[0003] Currently, the inference service supports many ways to load models. For example, model files can be loaded from mainstream cloud storage services. However, due to the large size of the model files, the transmission delay of the model files loaded into each container unit pod will be high, and a large amount of hard disk resources will be redundantly occupied. Another example is to use a persistent volume (PVC) to load the model file, but the loaded model file is only allowed to be used in the namespace corresponding to the PVC. Another example is to load the Docker image including the model file into each pod. However, since there is a copy in each pod, and the Docker image often requires users to move and change it frequently, it will consume pod resources and delay the pod startup. Summary of the invention

[0004] In view of this, the present invention provides a model file loading method, system, computer device and storage medium to solve the problems of high transmission delay, redundant occupation of hard disk resources, namespace limitation, consumption of pod resources and pod startup delay in the model file loading process supported by inference service.

[0005] In a first aspect, the present invention provides a method for loading a model file, which is applied to a master node in a cluster, and a shared directory on the master node stores a model file; the method comprises: when an inference service is started, obtaining an update request, and the update request is used to update a target scheduling unit of a first slave node in the cluster; detecting request information carried in the update request, and if the request information carries a local storage path of the model file, using the local storage path as a target storage volume of the target scheduling unit, and the local storage path is a path of the model file obtained and stored by the first slave node from the shared directory in the local directory; mounting the target storage volume in an initialization container of the target scheduling unit, and generating a target mount item of the initialization container, and the target mount item is used to specify the target storage volume; based on the target storage volume and the target mount item, generating response information, and the response information is used to update the target scheduling unit, and to establish a communication link between the updated target scheduling unit and the local directory; sending the response information to the first slave node, so that the first slave node updates the target scheduling unit, and loads the model file based on the communication link through the updated target scheduling unit.

[0006] Based on the method of the first aspect above, when the inference service is started, the master node can obtain an update request, and when the update request carries the local storage path of the model file, the local storage path is used as the target storage volume of the target scheduling unit, the target storage volume is mounted in the initialization container of the target scheduling unit, and a target mount item of the initialization container is generated. Based on the target storage volume and the target mount item, a response message is generated and sent to the first slave node.

[0007] When the inference service is started, the master node can use the local storage path as the target storage volume of the target scheduling unit, mount the target storage volume in the initialization container of the target scheduling unit, generate the target mount item of the initialization container, and then update the target scheduling unit based on the response information generated by the target storage volume and the target mount item to establish a communication link between the updated target scheduling unit and the local directory, and the local directory of the first slave node and the shared directory of the master node have established a communication link in advance, so that the updated target scheduling unit can mount the shared directory, so that the target scheduling unit in the inference service can directly read the model file as it reads the local directory, thereby reducing the transmission delay of loading the model file, removing the restriction of the namespace, avoiding the consumption of pod resources and avoiding pod startup delays. In addition, the embodiment of the present invention only needs to store the model file in the master node, so the problem of redundant copies on other slave nodes can be avoided.

[0008] In an optional embodiment, the request information carried in the update request is detected, including: parsing the update request, obtaining the request information of the target scheduling unit, the request information including annotation information of the first preset field, the annotation information including the header field and the storage path field; detecting whether the header field conforms to the preset format; if so, determining that the update request carries the local storage path of the model file, and the local storage path is the path indicated by the storage path field.

[0009] Based on the above method, the master node can obtain the request information of the target scheduling unit, and when it detects that the header field of the request information conforms to the preset format, it determines the local storage path of the model file, so as to facilitate the subsequent mounting of the local storage path as a storage volume in the target scheduling unit, so that the updated target scheduling unit can establish a communication link with the local directory.

[0010] In an optional embodiment, the request information also includes a volume list of the target scheduling unit; if the request information carries the local storage path of the model file, the local storage path is used as the target storage volume of the target scheduling unit, including: if the update request carries the local storage path, the local storage path is obtained; the local storage path is used as the target storage volume of the target scheduling unit, the target storage volume is added to the volume list, and the volume type and access type of the target storage volume are set.

[0011] Based on the above method, the volume type and access type can be set to indicate that the local storage path must be an existing directory, so that the first slave node can load the model file from the local storage path. The local storage path is used as the target storage volume and added to the volume list so that the master node can determine that the local storage path can be mounted to the target scheduling unit.

[0012] In an optional embodiment, the request information also includes a mount item list for each container in the target scheduling unit; mounting the target storage volume in the initialization container of the target scheduling unit to generate a target mount item for the initialization container, including: obtaining a container name of the initialization container; filtering a mount item list for the initialization container corresponding to the container name from the request information; mounting the target storage volume at an agreed path of the initialization container to generate a target mount item, and adding the target mount item to the mount item list, wherein the agreed path is used to indicate that all containers of the target scheduling unit can access the local directory of the first slave node through the agreed path.

[0013] Based on the above method, the target mount item can be added to the mount item list of the initialization container, so that the target storage volume is mounted at the agreed path of the initialization container, so that all containers in the target scheduling unit can access the local directory through the agreed path, and then obtain the model file stored in the shared directory through the local directory.

[0014] In an optional embodiment, the method further includes: obtaining creation information of a shared directory, the creation information including multiple slave nodes allowed to access the shared directory and read and write permissions of each slave node; creating a shared directory based on the creation information, and copying the model file stored locally on the master node to the shared directory through a command line tool.

[0015] Based on the above method, you can create a shared directory on the master node and configure appropriate read and write permissions for the shared directory to ensure that the shared directory can be accessed by multiple slave nodes that access the shared directory.

[0016] In an optional implementation, when the update request does not carry a local storage path, a first response message is returned to the first slave node, where the first response message is used to instruct the first slave node to update the target scheduling unit based on the update request.

[0017] Based on the above method, when the update request does not carry a local storage path, it indicates that the update request specifies a non-local path to obtain the model file. Therefore, the master node directly returns the first response information to the first slave node, so that the first slave node directly updates the target scheduling unit according to the update request.

[0018] In a second aspect, the present invention provides a method for loading a model file, which is applied to a first slave node in a cluster; the cluster also includes a master node, and a shared directory on the master node stores a model file; the method includes: receiving a response message sent from the master node, the response message is generated by the master node based on a target storage volume of a target scheduling unit of the first slave node and a target mount item of an initialization container of the target scheduling unit when the inference service is started, the target storage volume and the target mount item are determined based on a local storage path of the model file, and the local storage path is a path of the model file in the local directory obtained and stored by the first slave node from the shared directory; updating the target scheduling unit according to the response information to establish a communication link between the updated target scheduling unit and the local directory; and loading the model file based on the communication link through the updated target scheduling unit.

[0019] Based on the method of the second aspect mentioned above, the first slave node receives the response information sent from the master node, updates the target scheduling unit according to the response information to establish a communication link between the updated target scheduling unit and the local directory; and loads the model file based on the communication link through the updated target scheduling unit.

[0020] Since the response information is generated by the master node based on the target storage volume of the target scheduling unit of the first slave node and the target mount item of the initialization container of the target scheduling unit when the inference service is started, and the target storage volume and the target mount item are determined based on the local storage path of the model file, the target scheduling unit can be updated based on the response information to establish a communication link between the updated target scheduling unit and the local directory, and the local directory of the first slave node and the shared directory of the master node have established a communication link in advance, so that the updated target scheduling unit can mount the shared directory, so that the target scheduling unit in the inference service can directly read the model file as it reads the local directory, thereby reducing the transmission delay of loading the model file, removing the restriction of the namespace, avoiding the consumption of pod resources and avoiding pod startup delays. In addition, the embodiment of the present invention only needs to store the model file in the master node, so the problem of redundant copies on other slave nodes can be avoided.

[0021] In an optional embodiment, a mount command is sent to the master node, the mount command includes the network address of the master node, the path of the shared directory and the path of the local directory, the mount command is used to instruct the master node to mount the shared directory to the local file system of the first slave node, and return a mount response message, the mount response message is used to indicate whether a first communication link between the local directory on the local file system and the shared directory is successfully established; and the mount response message returned by the master node is received.

[0022] Based on the above method, the first slave node sends a mount command to the master node and receives the mount response information returned by the master node, thereby establishing a first communication link between the local directory on the local file system and the shared directory, so that the target scheduling unit can load the model file on the shared directory based on the first communication link.

[0023] In an optional embodiment, the updated target scheduling unit is deployed with a model loader; the model file is loaded based on the communication link through the updated target scheduling unit, including: obtaining the file name of the model file through the model loader; querying through the model loader whether there is a model file corresponding to the file name in the local directory; if so, obtaining the total memory occupied by the model file through the model loader; calculating through the model loader based on the total memory and the preset data block size that the model file is divided into a total number of data blocks N of the preset data block size; dividing the storage address segment corresponding to the model file into N sub-address segments, and loading the model file in parallel through the model loader according to the N sub-address segments.

[0024] Based on the above method, the total number of data blocks of preset data block size into which the model file is divided can be calculated, the storage address segment corresponding to the model file can be divided into N sub-address segments, and the model files can be loaded in parallel to improve the speed of reading or loading the model files.

[0025] In an optional embodiment, the model file is loaded in parallel through the model loader according to N sub-address segments, including: creating N read tasks in sequence according to the N sub-address segments through the model loader, each read task is used to read the data of each sub-address segment; initializing a preset thread pool through the model loader, the preset thread pool is used to manage multiple threads, each thread is used to execute at least one read task; and executing N read tasks in parallel through multiple threads to load the model file.

[0026] Based on the above method, the preset thread pool can be initialized through the model loader to manage multiple threads. The use of the thread pool can limit the number of threads running simultaneously, avoid excessive occupation of system resources, and ensure the scheduling and execution efficiency of concurrent tasks.

[0027] In an optional implementation, the method further includes: after the multiple threads have finished loading the model file, closing the thread pool through the model loader and releasing the resources occupied by the multiple threads.

[0028] Based on the above method, after multiple threads have finished loading the model file, the thread pool is closed and all thread resources are released to ensure that system resources will not be occupied for a long time, thus avoiding potential resource leakage problems.

[0029] In a third aspect, the present invention provides a model file loading device, which is applied to a master node in a cluster, and a shared directory on the master node stores the model file;

[0030] A first transceiver module is used to obtain an update request when the inference service is started, and the update request is used to update the target scheduling unit of the first slave node in the cluster;

[0031] A first processing module is used to detect request information carried in the update request. If the request information carries a local storage path of the model file, the local storage path is used as the target storage volume of the target scheduling unit, where the local storage path is the path of the model file in the local directory obtained and stored by the first slave node from the shared directory;

[0032] The first processing module is further used to mount the target storage volume in the initialization container of the target scheduling unit, and generate a target mount item of the initialization container, where the target mount item is used to specify the target storage volume;

[0033] The first processing module is further used to generate response information based on the target storage volume and the target mount item, the response information is used to update the target scheduling unit, and to establish a communication link between the updated target scheduling unit and the local directory;

[0034] The first transceiver module is also used to send response information to the first slave node so that the first slave node updates the target scheduling unit, and loads the model file based on the communication link through the updated target scheduling unit.

[0035] In a fourth aspect, the present invention provides another model file loading device, which is applied to the first slave node in the cluster; the cluster also includes a master node, and the shared directory on the master node stores the model file. A second transceiver module is used to receive a response message sent from the master node, and the response message is generated by the master node based on the target storage volume of the target scheduling unit of the first slave node and the target mount item of the initialization container of the target scheduling unit when the inference service is started. The target storage volume and the target mount item are determined based on the local storage path of the model file, and the local storage path is the path of the model file in the local directory obtained and stored by the first slave node from the shared directory. A second processing module is used to update the target scheduling unit according to the response information to establish a communication link between the updated target scheduling unit and the local directory; and load the model file based on the communication link through the updated target scheduling unit.

[0036] In a fifth aspect, the present invention provides a model file loading system, the model file loading system comprises a master node and at least one slave node, the at least one slave node comprises a first slave node; the master node is used to execute the model file loading method of the above-mentioned first aspect or any corresponding embodiment thereof; the first slave node is used to execute the model file loading method of the above-mentioned second aspect or any corresponding embodiment thereof.

[0037] In a sixth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the method for loading a model file of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions, or executes the method for loading a model file of the above-mentioned second aspect or any corresponding embodiment thereof.

[0038] In the seventh aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for loading a model file of the first aspect or any corresponding embodiment thereof, or to execute the method for loading a model file of the second aspect or any corresponding embodiment thereof.

[0039] In an eighth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the method for loading a model file of the above-mentioned first aspect or any corresponding embodiment thereof, or to execute the method for loading a model file of the above-mentioned second aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 is a topological diagram of a system for loading model files according to an embodiment of the present invention;

[0042] Figure 2 is a flowchart of a method for loading a model file according to an embodiment of the present invention;

[0043] Figure 3 is a flowchart of another method for loading a model file according to an embodiment of the present invention;

[0044] Figure 4 is a flowchart of another method for loading a model file according to an embodiment of the present invention;

[0045] Figure 5 is a structural block diagram of a device for loading a model file according to an embodiment of the present invention;

[0046] Figure 6 is a structural block diagram of another model file loading device according to an embodiment of the present invention;

[0047] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0049] Before introducing the specific embodiments of the present invention, relevant technical terms of the present invention are first introduced.

[0050] Container orchestration platform: The container orchestration platform aims to automate the deployment, expansion and management of containerized applications. Kubernetes provides a complete set of management and scheduling mechanisms that can efficiently run, manage and scale containerized applications. Optionally, the container orchestration platform can be a Kubernetes container orchestration platform. The container orchestration platform provides resources such as scheduling units Pod, deployment modules Deployment, and service modules Service, which can be used to run and access workloads. Persistent Volume Claim (PVC) is a native resource in the container orchestration platform that is used to request persistent storage. PVC represents the user or application's demand for storage resources, and the container orchestration platform will dynamically or statically provide the corresponding storage resources through PVC. In addition, PVC is an encapsulation of Persistent Volume (PV) resources, allowing users to focus on storage needs without directly managing PV. The container orchestration platform will automatically schedule and allocate.

[0051] Namespace: Namespace is a mechanism for logically isolating resources in a cluster. It provides a multi-tenant management method for different users, teams, or projects in a cluster. Namespace helps users manage and organize resources in the same cluster, avoid resource naming conflicts, and make management clearer and more flexible.

[0052] The Mutating Admission Webhook is a mechanism for intercepting and modifying incoming requests. Mutating Admission Webhook can intercept requests and modify them when resources (e.g. Pods) are created or modified, such as adding labels, environment variables, mounting volumes, etc.

[0053] Webhook Configuration (Mutating Webhook Configuration): Mutating Webhook Configuration is a resource type in the container orchestration platform, which is used to configure Mutating Admission Webhook. In Mutating Webhook Configuration, you can define multiple Webhook configuration rules and instruct Kubernetes in what order and under what circumstances to call Mutating Admission Webhook.

[0054] Volumes: A volume is a resource configuration in the container orchestration platform that describes how containers in a Pod access storage resources.

[0055] Management component (Knative): Knative provides a serverless computing model that uses custom resource definitions (CRDs) to extend the functionality of the container orchestration platform. Through CRDs, Knative defines knative's Service, Revision, Router and other resources, and allows users to manage these resources on the container orchestration platform. Knative enables applications to automatically scale based on traffic, or even stop completely when there are no requests, greatly improving resource utilization and reducing costs. Knative has certain defects. For example, Knative itself does not support mounting local directories (e.g., local directories cannot be mounted through volumeMounts in the container orchestration platform).

[0056] Inference Service (kserve): kserve is an inference service framework based on knative, designed for large-scale machine learning inference, and can provide efficient, flexible and scalable inference services. kserve customizes the resource InferenceService, which is usually deployed in the form of instantiating InferenceService. When instantiating, a pod is created and a storage-initializer container is injected into the pod, in which the model can be downloaded from the website, etc. The implementation of InferenceService is based on knative's Service, so its features inherit the features and defects of knative.

[0057] The Network File System (NFS) protocol is a network protocol. The NFS storage system recommended based on this protocol is a client-server architecture. The client accesses the NFS file system in the cluster through the NFS protocol. The server provides storage resources and allows the client to operate. It enables different nodes in the cluster to read and write files to the storage of another node as if accessing a local directory.

[0058] The embodiment of the present invention is applied to a scenario in which each node in a cluster loads a model file during the process of creating a Kserve service.

[0059] As described in the background technology, the kserve service currently supports many model loading methods, but there are defects such as high transmission latency for model file loading, redundant use of a large amount of hard disk resources, namespace restrictions for loading model files, consumption of pod resources, and pod startup delays.

[0060] In order to solve the above problems, an embodiment of the present invention provides a method for loading a model file. When the inference service is started, an update request is obtained; when the update request carries the local storage path of the model file, the local storage path is used as the target storage volume of the target scheduling unit; the target storage volume is mounted in the initialization container of the target scheduling unit to generate a target mount item of the initialization container; response information is generated based on the target storage volume and the target mount item, and the response information is used to update the target scheduling unit to establish a communication link between the updated target scheduling unit and the local directory; the response information is sent to the first slave node so that the first slave node updates the target scheduling unit, and the model file is loaded based on the communication link through the updated target scheduling unit. In order to achieve the effect of solving the problems of high transmission delay, redundant occupation of hard disk resources, namespace limitations, consumption of pod resources and pod startup delay in the model file loading process supported by the inference service.

[0061] Below Figure 1 Taking the model file loading system 100 shown as an example, the method provided in the embodiment of the present application is described. Figure 1 It is only a schematic diagram and does not constitute a limitation on the applicable scenarios of the technical solution provided in this application.

[0062] like Figure 1 As shown, Figure 1 It is a topological diagram of a system for loading model files according to an embodiment of the present invention. Figure 1 In the embodiment, the model file loading system 100 may include a master node 101, a first slave node 102, and a second slave node 103. In addition, the system 100 may also include more slave nodes, such as a third slave node and a fourth slave node, which is not limited in this embodiment.

[0063] The master node 101 in this embodiment may be any device with computing and communication capabilities. For example, the master node 101 may be a server or a cloud server.

[0064] It is understandable that the master node 101 can be deployed with an NFS service. The shared directory on the master node stores the model file. The master node 101 is the master node in the cluster. The master node 101 includes a management module and a Mutating AdmissionWebhook service module (abbreviated as MW service module). Optionally, the management module can be a Kubernetes module.

[0065] Specifically, the master node 101 obtains the Docker image of the MW service module from the container image repository, and deploys the Mutating Admission Webhook service module using the native Service and Deployment methods of the container orchestration platform.

[0066] Specifically, when deploying the Mutating Admission Webhook service module in Kubernetes, the cluster administrator can use the Deployment of the management module to define how to run the container deployed with the Mutating Admission Webhook service module. By specifying the image, number of copies, and port information of the container in the Deployment. The MW service module also requires a security certificate, so the management module needs to mount the security certificate to the container.

[0067] In order for the management module to access the Webhook server, the cluster administrator needs to create a KubernetesService. Service is an abstract layer used in the management module to define how to access a group of Pods. Service is used to expose the port of the MW service module (i.e. port 443) so that the management module can access the MW service module through Service. At the same time, the cluster administrator needs to specify a name (for example: hostpath-webhook-server-service) and namespace for the Service.

[0068] The cluster administrator pre-configures the Mutating Webhook Configuration required by the Mutating Admission Webhook service module in the cluster, and specifies that the MW service module will be triggered when the target scheduling unit is created or updated. The service provider saves the model file to the shared directory, creates an inference service based on InferenceService, and sets the model file location of the inference service to the location saved in the shared directory.

[0069] The cluster administrator specifies the name of the MW service module in MutatingWebhookConfiguration (for example, hostpath-webhook-server-service), as well as the namespace and access path of the MW service module; configures a failure strategy for the MW service module. If the MW service module fails to respond or an error occurs, the management module will reject the current request (such as a request to create a pod or update a pod). Configure applicable rules for the MW service module. The rule specifies that the MW service module will be triggered when a pod request is created or updated.

[0070] The first slave node 102 or the second slave node 103 in the embodiment of the present application can be any device with computing power and communication capability. For example, the first slave node 102 or the second slave node 103 can be a server or a cloud server. It can be understood that the first slave node 102 or the second slave node 103 can be deployed with an NFS client. The first slave node 102 or the second slave node 103 is any slave node in the cluster.

[0071] Figure 1 The model file loading system 100 shown is only used as an example and is not used to limit the technical solution of the present application. Those skilled in the art should understand that in the specific implementation process, the model file loading system 100 may also include other slave nodes, and the number of slave nodes may also be determined according to specific needs without limitation.

[0072] According to an embodiment of the present invention, an embodiment of a method for loading a model file is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0073] In this embodiment, a method for loading a model file is provided, which can be used for the above-mentioned master node. Figure 2 is a flow chart of a method for loading a model file according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0074] S201: When the inference service is started, an update request is obtained.

[0075] The update request is used to update the target scheduling unit of the first slave node in the cluster. The update request includes the pod configuration information of the target scheduling unit and the request operation details.

[0076] The pod configuration information includes all definition information of the target scheduling unit, such as the container definition, volume definition, and environment variables of the target scheduling unit. The pod configuration information also includes the request information of the target scheduling unit. The request information includes the annotation information of the first preset field, the volume list (volumes list) of the target scheduling unit, and the mount item list (volumeMounts list) of each container in the target scheduling unit.

[0077] The first preset field is the "model-path" field. The annotation information is used to indicate the storage path of the model file. The annotation information includes a header field and a storage path field.

[0078] The request operation details are used to indicate whether the requested operation is a create pod request operation or an update pod request operation.

[0079] Exemplarily, when the inference service is started, the Kubernete module of the master node sends an update request in JSON format to the MW service module of the master node through the access path of the MW service module; the MW service module receives and obtains the update request.

[0080] In one example, when the inference service is started, the master node obtains the creation information of the shared directory before obtaining the update request; creates the shared directory based on the creation information, and copies the model file stored locally on the master node to the shared directory through the command line tool.

[0081] The creation information includes multiple slave nodes that are allowed to access the shared directory and the read and write permissions of each slave node.

[0082] It can be understood that the master node can create a shared directory on the master node, configure appropriate read and write permissions for the shared directory, and ensure that the shared directory can be accessed by multiple slave nodes that access the shared directory.

[0083] S202: Detect request information carried in the update request, and if the request information carries a local storage path of the model file, use the local storage path as a target storage volume of the target scheduling unit.

[0084] The local storage path is the path of the model file in the local directory that is obtained and stored by the first slave node from the shared directory.

[0085] In some optional embodiments, before using the local storage path as the target storage volume of the target scheduling unit, the MW service module can parse the update request and obtain the request information of the target scheduling unit; detect whether the header field conforms to a preset format; if so, determine that the update request carries the local storage path of the model file.

[0086] The local storage path is the path indicated by the storage path field.

[0087] Optional, the default format is hostpath: / / format.

[0088] It is understandable that if the MW service module detects that the header field does not conform to the preset format, it determines that the update request does not carry the local storage path of the model file.

[0089] In some optional implementations, when the update request does not carry a local storage path, the MW service module returns a first response message to the first slave node.

[0090] The first response information is used to instruct the first slave node to update the target scheduling unit based on the update request.

[0091] In some optional embodiments, when the update request carries a local storage path, the MW service module obtains the local storage path; uses the local storage path as the target storage volume of the target scheduling unit, adds the target storage volume to the volume list, and sets the volume type and access type of the target storage volume.

[0092] Exemplarily, when an update request carries a local storage path, the MW service module removes the "hostpath: / / " prefix in the annotation information and obtains the local storage path (for example, if the original annotation information is hostpath: / / / mnt / test / models, after removing "hostpath: / / ", the local storage path obtained is / mnt / test / models); the local storage path is used as the target storage volume of the target scheduling unit, and the target storage volume is added to the volume list, and the volume type of the target storage volume is set to the hostPath type and the access type is set to the Directory type.

[0093] It can be understood that when the volume type is hostPath type and the access type is Directory type, it means that the local storage path must be an existing directory. If the local storage path is not an existing directory, the management module will refuse to mount the local storage path.

[0094] S203: Mount the target storage volume in the initialization container of the target scheduling unit to generate a target mount item of the initialization container.

[0095] The target mount item is used to specify the target storage volume.

[0096] InitContainers are special containers that Kubernetes modules execute before running the main container. The container name of the init container is storage-initializer.

[0097] A mount is a path in a container configuration that defines how to mount a storage volume into a container.

[0098] In some optional implementations, the MW service module obtains the container name of the initialization container; filters the mount item list of the initialization container corresponding to the container name from the request information; mounts the target storage volume at the agreed path of the initialization container, generates a target mount item, and adds the target mount item to the mount item list.

[0099] The agreed path is used to indicate that all containers of the target scheduling unit can access the local directory of the first slave node through the agreed path.

[0100] S204: Generate response information based on the target storage volume and the target mount item.

[0101] The response information is used to update the target scheduling unit and to establish a communication link between the updated target scheduling unit and the local directory. The response information includes an updated volumeMounts list and a volumes list including the target storage volume.

[0102] Exemplarily, the MW service module generates response information based on the target storage volume and the target mount item, and sends the response information to the Kubernete module. The Kubernete module receives the response information.

[0103] S205: Send response information to the first slave node, so that the first slave node updates the target scheduling unit, and loads the model file based on the communication link through the updated target scheduling unit.

[0104] Exemplarily, the Kubernetes module sends a response message to the first slave node.

[0105] It can be understood that when the inference service is started, that is, when an inference service based on InferenceService is created in the cluster, the local directory of the model file will be mounted in the target scheduling unit of the first slave node to the agreed path of the target scheduling unit, so that the inference service can directly read the model file like reading local storage.

[0106] Based on the above Figure 2 According to the method, when the inference service is started, the master node can obtain an update request, and when the update request carries a local storage path of a model file, the local storage path is used as a target storage volume of a target scheduling unit, the target storage volume is mounted in an initialization container of the target scheduling unit, a target mount item of the initialization container is generated, and a response message is generated and sent to the first slave node based on the target storage volume and the target mount item.

[0107] When the inference service is started, the master node can use the local storage path as the target storage volume of the target scheduling unit, mount the target storage volume in the initialization container of the target scheduling unit, generate the target mount item of the initialization container, and then update the target scheduling unit based on the response information generated by the target storage volume and the target mount item to establish a communication link between the updated target scheduling unit and the local directory, and the local directory of the first slave node and the shared directory of the master node have established a communication link in advance, so that the updated target scheduling unit can mount the shared directory, so that the target scheduling unit in the inference service can directly read the model file as it reads the local directory, thereby reducing the transmission delay of loading the model file, removing the restriction of the namespace, avoiding the consumption of pod resources and avoiding pod startup delays. In addition, the embodiment of the present invention only needs to store the model file in the master node, so the problem of redundant copies on other slave nodes can be avoided.

[0108] In this embodiment, another method for loading a model file is provided, which can be used for the first slave node mentioned above. Figure 3 is a flowchart of another method for loading a model file according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0109] S301: Receive response information sent from the master node.

[0110] The response information is generated by the master node based on the target storage volume of the target scheduling unit of the first slave node and the target mount item of the initialization container of the target scheduling unit when the inference service is started.

[0111] S302: Update the target scheduling unit according to the response information to establish a communication link between the updated target scheduling unit and the local directory.

[0112] The first slave node updates the target scheduling unit, that is, the first slave node removes the original target scheduling unit and creates a new scheduling unit based on the response information. The new scheduling unit in the first slave node is mounted with a local directory, that is, the new scheduling unit is mounted with a local directory and a communication link is established.

[0113] S303: Load the model file based on the communication link through the updated target scheduling unit.

[0114] The updated target scheduling unit is deployed with a model loader, which may be a PyTorch loader.

[0115] In some optional embodiments, the first slave node obtains the file name of the model file through the model loader; queries through the model loader whether there is a model file corresponding to the file name in the local directory; if so, obtains the total memory occupied by the model file through the model loader; calculates the total number N of data blocks of the preset data block size into which the model file is divided based on the total memory and the preset data block size through the model loader; divides the storage address segment corresponding to the model file into N sub-address segments, and loads the model file in parallel through the model loader according to the N sub-address segments.

[0116] The preset data block size can be set according to actual needs. For example, the preset data block size can be 1MB.

[0117] Each sub-address segment includes a start byte and an end byte for indicating a read range.

[0118] In some optional embodiments, the first slave node may obtain the maximum transmission rate, concurrent task volume and expected response time of the communication link; and determine the preset data block size based on a preset algorithm according to the maximum transmission rate, concurrent task volume and expected response time.

[0119] The concurrent task volume is the number of sub-address segments that concurrently read the data stored in the sub-address segments.

[0120] The expected response time is the sum of the response delay and the time expected to read the entire preset data block.

[0121] The preset algorithm is: S=A×B / T, where A is the maximum transmission rate, B is the number of concurrent tasks, and T is the expected response time.

[0122] It is understandable that the first slave node can comprehensively consider the maximum transmission rate of the communication link, the amount of concurrent tasks, and the expected response time to accurately determine the appropriate preset data block size. In addition, the preset data block size can also balance the network input and output I / O performance and memory usage. If the preset data block size is too large, it may cause memory pressure, while if the preset data block size is too small, it will increase the number of I / O operations and affect the reading efficiency.

[0123] In some optional implementations, the first slave node queries the local directory through the model loader and finds that the model file corresponding to the file name does not exist, and displays an error message to the client of the first slave node, wherein the error message is used to indicate that the model file failed to load.

[0124] In one example, the first slave node creates N read tasks in sequence according to N sub-address segments through a model loader; initializes a preset thread pool through the model loader, the preset thread pool is used to manage multiple threads, each thread is used to execute at least one read task; executes N read tasks in parallel through multiple threads to load the model file.

[0125] Each read task is used to read data of each sub-address segment. Each read task starts to read data from the start byte of each sub-address segment.

[0126] In one example, the first slave node opens the model file through each thread, moves the file pointer to a specified starting position (i.e., the start byte), reads data of a preset data block size, and splices the read data into a complete byte stream and stores it in memory.

[0127] Understandably, the thread pool can limit the number of threads running at the same time, avoid excessive occupation of system resources, and ensure the scheduling and execution efficiency of concurrent tasks. Multiple threads utilize the parallel I / O capability of the NFS protocol. Multiple threads can read data corresponding to different sub-address segments from the master node at the same time, thereby improving the reading speed.

[0128] In one example, after multiple threads have finished loading the model file, the first slave node closes the thread pool through the model loader and releases the resources occupied by the multiple threads.

[0129] Exemplarily, after multiple threads have finished loading the model file, the first slave node obtains the loaded model file from the memory through the model loader, closes the thread pool, and releases the resources occupied by the multiple threads.

[0130] It is understandable that after obtaining the model file, the first slave node will close the thread pool to ensure that system resources will not be occupied for a long time, thus avoiding potential resource leakage problems.

[0131] In some optional implementations, the first slave node sends a mount command to the master node, and receives mount response information returned by the master node.

[0132] The mount command includes the network address of the master node, the path of the shared directory and the path of the local directory. The mount command is used to instruct the master node to mount the shared directory to the local file system of the first slave node and return mount response information.

[0133] The mount response information is used to indicate whether the first communication link between the local directory and the shared directory on the local file system is established successfully.

[0134] Based on the above Figure 3The method comprises the following steps: a first slave node receives a response message sent from a master node, updates a target scheduling unit according to the response message, so as to establish a communication link between the updated target scheduling unit and a local directory; and loads a model file based on the communication link through the updated target scheduling unit.

[0135] Since the response information is generated by the master node based on the target storage volume of the target scheduling unit of the first slave node and the target mount item of the initialization container of the target scheduling unit when the inference service is started, and the target storage volume and the target mount item are determined based on the local storage path of the model file, the target scheduling unit can be updated based on the response information to establish a communication link between the updated target scheduling unit and the local directory, and the local directory of the first slave node and the shared directory of the master node have established a communication link in advance, so that the updated target scheduling unit can mount the shared directory, so that the target scheduling unit in the inference service can directly read the model file as it reads the local directory, thereby reducing the transmission delay of loading the model file, removing the restriction of the namespace, avoiding the consumption of pod resources and avoiding pod startup delays. In addition, the embodiment of the present invention only needs to store the model file in the master node, so the problem of redundant copies on other slave nodes can be avoided.

[0136] In this embodiment, another method for loading a model file is provided, which can be used for the above-mentioned master node. Figure 4 is a flowchart of another method for loading a model file according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0137] S401: The master node initializes the MW service module.

[0138] S402: When the inference service is started, whether the MW service module obtains an update request.

[0139] S403: If not, the MW service module continues to monitor the update request.

[0140] S404: If yes, obtain annotation information of the first preset field of the update request.

[0141] S405: Check whether the header field in the annotation information conforms to a preset format.

[0142] S406: If not, directly return the first response information.

[0143] S407: If yes, extract the local storage path and use the local storage path as the target storage volume of the target scheduling unit;

[0144] S408: Mount the target storage volume in the initialization container of the target scheduling unit to generate a target mount item of the initialization container.

[0145] S409: Generate response information based on the target storage volume and the target mount item, and send the response information to the first slave node.

[0146] In this embodiment, a model file loading system is provided. The model file loading system includes a master node and at least one slave node. The at least one slave node includes a first slave node.

[0147] The master node is used to obtain an update request when the inference service is started; when the update request carries the local storage path of the model file, the local storage path is used as the target storage volume of the target scheduling unit; the target storage volume is mounted in the initialization container of the target scheduling unit, and a target mount item of the initialization container is generated; based on the target storage volume and the target mount item, a response message is generated; and the response message is sent to the first slave node.

[0148] The first slave node is used to receive the response information sent from the master node; update the target scheduling unit according to the response information, and load the model file based on the communication link through the updated target scheduling unit.

[0149] The main node in the model file loading system is used to execute the above Figure 2 The method flow shown; at least one slave node in the model file loading system, such as the first slave node for executing the aforementioned Figure 3 The method flow shown in the figure. The updated target scheduling unit can mount the shared directory, so that the target scheduling unit in the inference service can directly read the model file as if it reads the local directory, thereby reducing the transmission delay of loading the model file, removing the namespace restriction, avoiding the consumption of pod resources and avoiding the delay of pod startup.

[0150] In the present embodiment, a loading device for a model file is also provided, and the device is used to implement the above-mentioned embodiment and the preferred implementation mode, and the description has been made no further. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0151] This embodiment provides a model file loading device, such as Figure 5 As shown, Figure 5 1 is a structural block diagram of a device for loading a model file according to an embodiment of the present invention; the device is applied to a master node in a cluster, and a model file is stored in a shared directory on the master node. The device includes:

[0152] The first transceiver module 501 is used to obtain an update request when the inference service is started, and the update request is used to update the target scheduling unit of the first slave node in the cluster.

[0153] The first processing module 502 is used to detect the request information carried in the update request. If the request information carries the local storage path of the model file, the local storage path is used as the target storage volume of the target scheduling unit. The local storage path is the path of the model file in the local directory obtained and stored by the first slave node from the shared directory.

[0154] The first processing module 502 is further used to mount the target storage volume in the initialization container of the target scheduling unit, and generate a target mount item of the initialization container, where the target mount item is used to specify the target storage volume.

[0155] The first processing module 502 is further used to generate response information based on the target storage volume and the target mount item, and the response information is used to update the target scheduling unit and to establish a communication link between the updated target scheduling unit and the local directory.

[0156] The first transceiver module 501 is further used to send response information to the first slave node so that the first slave node updates the target scheduling unit, and loads the model file based on the communication link through the updated target scheduling unit.

[0157] In some optional embodiments, before the local storage path is used as the target storage volume of the target scheduling unit, the first processing module 502 is specifically used to parse the update request and obtain the request information of the target scheduling unit, the request information includes annotation information of the first preset field, and the annotation information includes a header field and a storage path field; the first processing module 502 is also specifically used to detect whether the header field conforms to a preset format; the first processing module 502 is also specifically used to, if so, determine that the update request carries the local storage path of the model file, and the local storage path is the path indicated by the storage path field.

[0158] In some optional embodiments, the request information also includes a volume list of the target scheduling unit; the first processing module 502 is also specifically used to obtain the local storage path if the update request carries a local storage path; the first processing module 502 is also specifically used to use the local storage path as the target storage volume of the target scheduling unit, add the target storage volume to the volume list, and set the volume type and access type of the target storage volume.

[0159] In some optional embodiments, the request information also includes a mount item list for each container in the target scheduling unit; the first processing module 502 is also specifically used to obtain the container name of the initialization container; the first processing module 502 is also specifically used to filter the mount item list of the initialization container corresponding to the container name from the request information; the first processing module 502 is also specifically used to mount the target storage volume on the agreed path of the initialization container, generate a target mount item, and add the target mount item to the mount item list, the agreed path is used to indicate that all containers of the target scheduling unit can access the local directory of the first slave node through the agreed path.

[0160] In some optional embodiments, the first transceiver module 501 is also used to obtain creation information of a shared directory, the creation information including multiple slave nodes allowed to access the shared directory and the read and write permissions of each slave node; the first processing module 502 is also used to create a shared directory based on the creation information, and copy the model file stored locally on the master node to the shared directory through a command line tool.

[0161] In some optional embodiments, the first transceiver module 501 is further used to return a first response message to the first slave node when the update request does not carry a local storage path, and the first response message is used to instruct the first slave node to update the target scheduling unit based on the update request.

[0162] This embodiment provides another model file loading device, such as Figure 6 As shown, Figure 6 It is a structural block diagram of another model file loading device according to an embodiment of the present invention; it is applied to the first slave node in the cluster; the cluster also includes a master node, and the shared directory on the master node stores the model file. The device includes: a second transceiver module 601, which is used to receive response information sent from the master node, and the response information is generated by the master node based on the target storage volume of the target scheduling unit of the first slave node and the target mount item of the initialization container of the target scheduling unit when the inference service is started. The target storage volume and the target mount item are determined based on the local storage path of the model file, and the local storage path is the path of the model file in the local directory obtained and stored by the first slave node from the shared directory.

[0163] The second processing module 602 is used to update the target scheduling unit according to the response information to establish a communication link between the updated target scheduling unit and the local directory; and load the model file through the updated target scheduling unit based on the communication link.

[0164] In some optional embodiments, the second transceiver module 601 is also used to send a mount command to the master node, the mount command includes the network address of the master node, the path of the shared directory and the path of the local directory, the mount command is used to instruct the master node to mount the shared directory to the local file system of the first slave node, and return the mount response information, the mount response information is used to indicate whether the first communication link between the local directory and the shared directory on the local file system is successfully established; the second transceiver module 601 is also used to receive the mount response information returned by the master node.

[0165] In some optional embodiments, the updated target scheduling unit is deployed with a model loader; the second processing module 602 is specifically used to obtain the file name of the model file through the model loader; query through the model loader whether there is a model file corresponding to the file name in the local directory; if so, obtain the total memory occupied by the model file through the model loader; calculate the total number N of data blocks of the preset data block size that the model file is divided into based on the total memory and the preset data block size through the model loader; divide the storage address segment corresponding to the model file into N sub-address segments, and load the model file in parallel through the model loader according to the N sub-address segments.

[0166] In some optional embodiments, the second processing module 602 is further specifically used to create N read tasks in sequence according to N sub-address segments through a model loader, and each read task is used to read data in each sub-address segment; the second processing module 602 is further specifically used to initialize a preset thread pool through a model loader, and the preset thread pool is used to manage multiple threads, and each thread is used to execute at least one read task; the second processing module 602 is further specifically used to execute N read tasks in parallel through multiple threads to load a model file.

[0167] In some optional implementations, the second processing module 602 is further configured to close the thread pool through the model loader and release resources occupied by the multiple threads after the multiple threads have finished loading the model file.

[0168] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0169] The model file loading device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0170] The embodiment of the present invention also provides a computer device having the above Figure 5 or Figure 6 The model file is shown in the loading device.

[0171] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0172] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0173] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0174] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0175] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0176] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0177] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0178] Part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes but is not limited to source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0179] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for loading a model file, characterized in that: Applied to a master node in a cluster, a shared directory on the master node stores a model file; the method comprises: When the inference service is started, an update request is obtained, where the update request is used to update a target scheduling unit of a first slave node in the cluster; Detecting the request information carried in the update request, and if the request information carries the local storage path of the model file, using the local storage path as the target storage volume of the target scheduling unit, where the local storage path is the path of the model file in the local directory obtained and stored by the first slave node from the shared directory; Mounting the target storage volume in the initialization container of the target scheduling unit, generating a target mount item of the initialization container, wherein the target mount item is used to specify the target storage volume; Generate response information based on the target storage volume and the target mount item, wherein the response information is used to update the target scheduling unit and to establish a communication link between the updated target scheduling unit and the local directory; The response information is sent to the first slave node so that the first slave node updates the target scheduling unit, and loads the model file based on the communication link through the updated target scheduling unit.

2. The method according to claim 1, characterized in that The detecting the request information carried in the update request includes: Parsing the update request to obtain the request information of the target scheduling unit, wherein the request information includes annotation information of a first preset field, and the annotation information includes a header field and a storage path field; Detecting whether the header field conforms to a preset format; If so, it is determined that the update request carries the local storage path of the model file, and the local storage path is the path indicated by the storage path field.

3. The method according to claim 2, characterized in that The request information also includes a volume list of the target scheduling unit; if the request information carries a local storage path of the model file, the local storage path is used as the target storage volume of the target scheduling unit, including: If the request information carries the local storage path, obtaining the local storage path; The local storage path is used as the target storage volume of the target scheduling unit, the target storage volume is added to the volume list, and the volume type and access type of the target storage volume are set.

4. The method according to claim 2, characterized in that: The request information also includes a list of mount items for each container in the target scheduling unit; The step of mounting the target storage volume in the initialization container of the target scheduling unit and generating a target mount item of the initialization container includes: Obtaining the container name of the initialization container; Filtering a mount item list of the initialization container corresponding to the container name from the request information; The target storage volume is mounted on the agreed path of the initialization container, the target mount item is generated, and the target mount item is added to the mount item list, wherein the agreed path is used to indicate that all containers of the target scheduling unit can access the local directory of the first slave node through the agreed path.

5. The method according to claim 4, characterized in that The method further comprises: Acquire creation information of the shared directory, the creation information including a plurality of slave nodes allowed to access the shared directory and the read and write permissions of each of the slave nodes; The shared directory is created based on the creation information, and the model file stored locally on the master node is copied to the shared directory through a command line tool.

6. The method according to claim 5, characterized in that The method further comprises: When the update request does not carry the local storage path, a first response message is returned to the first slave node, where the first response message is used to instruct the first slave node to update the target scheduling unit based on the update request.

7. A method for loading a model file, characterized in that: The method is applied to a first slave node in a cluster; the cluster further includes a master node, and a shared directory on the master node stores a model file; the method includes: Receive response information sent from the master node, the response information is generated by the master node based on the target storage volume of the target scheduling unit of the first slave node and the target mount item of the initialization container of the target scheduling unit when the inference service is started, the target storage volume and the target mount item are determined based on the local storage path of the model file, and the local storage path is the path of the model file in the local directory obtained and stored by the first slave node from the shared directory; updating the target scheduling unit according to the response information to establish a communication link between the updated target scheduling unit and the local directory; The model file is loaded based on the communication link through the updated target scheduling unit.

8. The method according to claim 7, characterized in that The method further comprises: Sending a mount command to the master node, the mount command including: the network address of the master node, the path of the shared directory and the path of the local directory, the mount command being used to instruct the master node to mount the shared directory to the local file system of the first slave node, and returning mount response information, the mount response information being used to indicate whether a first communication link between the local directory on the local file system and the shared directory is successfully established; Receive the mount response information returned by the master node.

9. The method according to claim 7, characterized in that: The updated target scheduling unit is deployed with a model loader; the loading of the model file based on the communication link by the updated target scheduling unit includes: Obtaining the file name of the model file through the model loader; Querying, by the model loader, whether the model file corresponding to the file name exists in the local directory; If it exists, the total memory occupied by the model file is obtained through the model loader; Calculating, by the model loader, a total number N of data blocks of the preset data block size into which the model file is divided based on the total memory and the preset data block size; The storage address segment corresponding to the model file is divided into N sub-address segments, and the model file is loaded in parallel through the model loader according to the N sub-address segments.

10. The method according to claim 9, characterized in that The loading the model files in parallel by the model loader according to the N sub-address segments includes: Creating N read tasks in sequence according to the N sub-address segments by the model loader, each read task being used to read data of each sub-address segment; Initialize a preset thread pool through the model loader, the preset thread pool is used to manage multiple threads, each of the threads is used to execute at least one of the reading tasks; The N reading tasks are executed in parallel by the multiple threads to load the model file.

11. The method according to claim 10, characterized in that: The method further comprises: After the multiple threads have finished loading the model file, the thread pool is closed by the model loader, and the resources occupied by the multiple threads are released.

12. A model file loading system, characterized in that: The model file loading system includes a master node and at least one slave node, wherein the at least one slave node includes a first slave node; The master node is used to execute the model file loading method described in any one of claims 1 to 6; The first slave node is used to execute the model file loading method described in any one of claims 7 to 11.

13. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model file loading method described in any one of claims 1 to 6 or any one of claims 7 to 11 by executing the computer instructions.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute any one of claims 1 to 6, or the model file loading method of any one of claims 7 to 11.

15. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the model file loading method according to any one of claims 1 to 6 or any one of claims 7 to 11.

Images