Model service method and device and storage medium

Abstract

The embodiment of the invention provides a model service method and device and a storage medium. It is proposed to build a plurality of service instances deployed for the model service as a data transmission network. After the service instance receives the model calling request, whether model data needed by model loading exists in the local storage space or not can be inquired firstly, and if not, the needed model data can be remotely downloaded from other service instances on the basis of a data transmission network, and the downloaded model data is stored in the local storage space. Therefore, model data sharing can be realized based on the data transmission network established among the plurality of service instances, and the network transmission rate among the service instances in the data transmission network is far higher than the back-to-source downloading rate, so that even if the service instance does not have the model data required for loading locally, the service instance can not be loaded locally. Model data can also be downloaded remotely from other service instances. Therefore, the model loading efficiency can be effectively improved, so that the response efficiency to the model calling request is improved.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a model service method, device, and storage medium. Background Art

[0002] Model servitization refers to providing services to other services or clients through APIs after the model is deployed. Model servitization technology enables users to deploy models as online inference services or A-WEB applications with a single click.

[0003] With the development of model service technology, the technical concept of dynamic loading of multiple models has been proposed. Under the technical concept of dynamic loading of multiple models, the loaded models can be dynamically switched according to usage requirements, so that the appropriate model can be used to respond to usage needs.

[0004] Currently, model data for various models are usually stored in the model repository. When dynamically loading a model, it is necessary to first download the corresponding model data through back-to-source before the loading can be completed, which results in poor response efficiency of the model service. Summary of the Invention

[0005] Various aspects of the present application provide a model service method, device, and storage medium to improve the response efficiency of model services.

[0006] An embodiment of the present application provides a model service method, wherein multiple service instances are deployed for a model service, and the method is applicable to any service instance among the multiple service instances, and the method includes:

[0007] In response to the received model call request, determining the target model to be loaded;

[0008] If the model data required for loading the target model does not exist in the local storage space, obtaining the model data from the target service instance storing the model data and storing the model data in the local storage space, wherein the multiple service instances are pre-organized into a data transmission network;

[0009] The target model is loaded based on the model data stored in the local storage space.

[0010] Furthermore, obtaining the model data from the target service instance storing the model data includes:

[0011] querying storage status information maintained for each service instance in the data transmission network to select the target service instance storing the model data;

[0012] Downloading the model data from the target service instance;

[0013] The storage status information is used to record the description information of the model data stored in the corresponding service instance.

[0014] Furthermore, the method further comprises:

[0015] After storing the model data in the local storage space, a broadcast carrying the identification information of the model data is sent to other service instances in the data transmission network, so that each other service instance updates the storage status information maintained by itself for the service instance based on the identification information of the model data.

[0016] Furthermore, the method further comprises:

[0017] If there are multiple service instances storing the model data in the data transmission network, then

[0018] Selecting the service instance with the lowest load as the target service instance;

[0019] Each service instance in the data transmission network broadcasts its own load status respectively.

[0020] Furthermore, the method further comprises:

[0021] If the model data does not exist in any other service instance in the data transmission network, performing a back-to-source download operation to download the model data from the data source;

[0022] The downloaded model data is stored in the local storage space.

[0023] Furthermore, the data source is mounted on a user-mode file system constructed in the service instance, and the back-to-source download operation includes:

[0024] Querying first file description information corresponding to the model data in the data source from metadata information maintained for the user-mode file system;

[0025] The model data is downloaded from the data source according to the first file description information.

[0026] Furthermore, the local storage space is mounted on a user-mode file system constructed in the service instance, and the method further includes:

[0027] Querying the second file description information corresponding to the model data in the local storage space from the metadata information maintained for the user-mode file system;

[0028] If no data matching the second file description information is found in the local storage space, it is determined that the model data does not exist in the local storage space.

[0029] Furthermore, the method further comprises:

[0030] After the service instance is created, request a list of service instances corresponding to the model service from the management instance deployed for the model service;

[0031] Join the data transmission network that has been established by each service instance in the service instance list.

[0032] Furthermore, the method further comprises:

[0033] If a remote download instruction is received from any other service instance in the data transmission network, determining the target model data requested by the remote download instruction;

[0034] Reading the target model data from the local storage space;

[0035] The target model data is sent to the service instance that issues the remote download instruction through the data transmission network.

[0036] The embodiment of the present application further provides a computing device, including a memory, a processor, and a communication component. The computing device carries a service instance deployed for a model service, the service instance is used to dynamically load a model, and the service instance is pre-connected to other service instance components deployed for the model service as a data transmission network;

[0037] The memory is used to store one or more computer instructions;

[0038] The processor is coupled to the memory and the communication component, and is configured to execute the one or more computer instructions to perform the aforementioned model service method.

[0039] The embodiment of the present application further provides a model service device, which is respectively arranged in multiple service instances deployed for the model service, and the model service device includes a service module, a processing model and a network building module;

[0040] The network building module is used to control the service instance to build a data transmission network with other service instances deployed for the model service;

[0041] The service module is used to determine the target model to be loaded in response to the received model call request;

[0042] The processing module is used to obtain the model data from the target service instance storing the model data and store it in the local storage space if the model data required for loading the target model does not exist in the local storage space of the service instance; and load the target model based on the model data stored in the local storage space.

[0043] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, which, when executed by one or more processors, causes the one or more processors to execute the aforementioned model service method.

[0044] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by one or more processors, causes the one or more processors to execute the aforementioned model service method.

[0045] In an embodiment of the present application, it is proposed to form a plurality of service instances deployed for the model service into a data transmission network, and the processing logic of the model call request in each service instance is also modified. On this basis, from the perspective of a single service instance, after receiving a model call request, it can first query whether the model data required for model loading exists in the local storage space. If not, it can try to remotely download the required model data from other service instances based on the data transmission network, and store the downloaded model data in the local storage space. In this way, in a multi-model dynamic loading scenario, model data sharing can be achieved based on the data transmission network formed between multiple service instances. Since the network transmission rate between each service instance in the data transmission network is much higher than the back-to-source download rate, even if the model data required for loading does not exist locally in the service instance, the model data can be efficiently downloaded remotely from other service instances based on the data transmission network, thereby effectively improving the model loading efficiency and thus improving the response efficiency to the model call request. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 A flowchart of a model service method provided by an exemplary embodiment of the present application;

[0048] Figure 2 A logical diagram of a model service method provided by an exemplary embodiment of the present application;

[0049] Figure 3A schematic diagram of an exemplary internal logical structure of a model service device provided by an exemplary embodiment of the present application;

[0050] Figure 4 A schematic structural diagram of a computing device provided as another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] Before describing the technical solutions provided in the embodiments of the present application, several technical concepts that may be involved in the present application are briefly explained as follows.

[0053] Model servitization means that after the model is deployed, it provides services to the outside world through API interfaces and other means for other services or clients to call.

[0054] Model Serving:

[0055] Model service refers to a software service based on machine learning models. It provides prediction or inference capabilities for machine learning models by encapsulating them into callable APIs or services.

[0056] Model loading:

[0057] Model loading involves loading a machine learning model from a storage medium (such as a disk, database, or cloud storage) into a service instance for use in inference or other machine learning tasks. This process requires the model data corresponding to the machine learning model, also known as the model file. This model data includes various data required for model loading, such as model parameters.

[0058] Service Instance:

[0059] A service instance is a resource instance deployed to implement a model service. This embodiment does not limit the type of service instance. For example, a service instance can be implemented as a resource group POD, a container, a virtual machine, or a bare metal server.

[0060] As introduced in the background technology, a mechanism for dynamic model loading has been proposed in model servitization technology. From the perspective of a single service instance, the models required for different model call requests it receives may be different, which requires the service instance to be able to dynamically switch the loaded model on demand. At present, the number of models involved in the dynamic model loading mechanism is increasing, and the local storage space of the service instance itself is limited. Therefore, after receiving each model call request, the service instance almost always needs to perform a back-to-source download to complete the model loading. The back-to-source download takes a relatively long time, which causes the service instance to respond too slowly to the model call request.

[0061] To this end, this embodiment proposes a model service method to improve the response efficiency to model call requests in a multi-model dynamic loading scenario.

[0062] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0063] Figure 1 A flow chart of a model service method provided for an exemplary embodiment of the present application is provided. The method can be executed by a model service device. The model service device can be implemented as software, hardware, or a combination of software and hardware. The model service device can run separately in each service instance deployed for the model service. Figure 2 A logical diagram of a model service method provided by an exemplary embodiment of the present application. Figure 2 In this embodiment, a distributed service system can be established for the model service, which includes the aforementioned multiple service instances. In practical applications, architectures such as Kubernetes (K8S) can be used to orchestrate and manage multiple service instances in the distributed service system, but this embodiment does not limit this.

[0064] Among them, the model call request submitted to the model service reaches the service instance in the distributed service system through the distributed scheduling mechanism. That is, the service instance will respond to the model call request. As introduced in the background technology, the model call request may come from other services or clients, etc., and this embodiment does not limit the initiator of the model call request. In addition, as mentioned above, in the multi-model dynamic switching scenario, any service instance deployed for the model service can be used to dynamically load the model. Here, dynamic model loading can be understood as dynamically switching the loaded model according to the received model call request to adapt to the model call request.

[0065] For ease of description, the following will use any service instance deployed by the model service as an example to describe the model service method provided by this embodiment in detail. However, it should be understood that each service instance deployed for the model service can run the model service device provided by this embodiment to execute the model service method provided by this embodiment.

[0066] refer to Figure 2 In this embodiment, under the premise of keeping the distributed service system unchanged, it is proposed to pre-organize multiple service instances deployed for the model service into a data transmission network. The data transmission network in this embodiment can adopt a peer-to-peer network or other network that can support data transmission between service instances. This embodiment does not limit the type of data transmission network. Among them, the peer-to-peer network (Peer-to-Peer, abbreviated as P2P) is a network architecture that allows participants (Peers) in the network to directly share and use their hardware resources, such as processor power, storage space, network connection capabilities and printers. In this network architecture, participants are both providers and consumers of resources. It should be understood that the data transmission network established in this embodiment does not destroy the original network relationship between the service instances in the distributed service system, but can be understood as an extended network (overlay network) built on the original network relationship. In addition, it should be understood that the multiple service instances deployed for the model service are usually located in the same virtual private network VPC. Therefore, based on the data transmission network constructed in this embodiment, multiple service instances can use the original VPC network resources for data transmission, which enables a relatively good network transmission rate to be obtained between the service instances.

[0067] On this basis, reference Figure 1 The model service method provided in this embodiment may include:

[0068] Step 100: In response to the received model call request, determine the target model to be loaded;

[0069] Step 102: If the model data required for loading the target model does not exist in the local storage space, the model data is obtained from the target service instance storing the model data and stored in the local storage space, wherein multiple service instances are pre-organized into a data transmission network;

[0070] Step 103: Load the target model based on the model data stored in the local storage space.

[0071] As mentioned above, the model service method provided in this embodiment can be executed by a model service device. Figure 3 This is a schematic diagram of an exemplary internal logical structure of a model service device provided by an exemplary embodiment of the present application. Figure 3 In this embodiment, the model service device may include at least a service module, a processing module, and a network assembly module. It should be understood that the module division within the model service device is illustrative only and this embodiment is not limited thereto. The model service method provided in this embodiment will be described in detail below, combining the various modules included in the model service device.

[0072] refer to Figure 3 The network building module can be set to control the service instance in which it is located and other service instances deployed for the model service to build a data transmission network. Figure 3 After the service instance is created, the network assembly module deployed therein can request the service instance list corresponding to the model service from the aforementioned control instance to obtain relevant information of other service instances deployed for the model service. Based on this, the current service instance can be controlled to join the data transmission network that has been established by each service instance in the service instance list. In this way, based on the network assembly module in each service instance, it can be ensured that the service instances created for the model service can be smoothly added to the corresponding data transmission network. Moreover, based on the network assembly module in each service instance, it can also be ensured that the data transmission networks between different model services are isolated from each other to ensure the data security in the service instance. Optionally, in this embodiment, decentralized tools such as serf can be used in the network assembly module to realize the establishment of the data transmission network. It should be understood that this embodiment is not limited to this.

[0073] refer to Figure 1 and Figure 3 , the service module can execute the aforementioned step 100. In addition, refer to Figure 3 In actual applications, the distributed service system may also include a management instance for service discovery. The management instance can be used to discover information about each service instance deployed for the model service and maintain a service instance list for the model service. The management instance can control each service instance in the service instance list to provide services. In some exemplary design schemes, a network proxy can be deployed in the management instance, and all model call requests sent to the model service can be distributed by the network proxy. Figure 3 , the model call request will be distributed to the service instance.

[0074] In step 100, the service module can respond to the received model call request to determine the target model to load. As mentioned above, different model call requests may require different models to be loaded. If the target model required by the model call request received in step 100 is different from the model required to be loaded by the previous model call request, a model switch will occur in the service instance.

[0075] In this embodiment, local storage space can be applied for the service instance. It should be understood that the service instance is hosted on a physical server, and based on virtualization technology, the hardware resources (such as memory, hard disk, etc.) on the physical server can be provided to the service instance for use. Based on this, in this embodiment, based on the local storage space applied for the service instance, the service instance can be given local caching capabilities. In this way, the service instance can use its local storage space to temporarily store the model data corresponding to the models that have been loaded in the past.

[0076] Continue to refer Figure 1 and Figure 3 , steps 101 and 102 can be implemented by the processing module. In step 101, it can first be determined whether the model data required for loading the target model exists in the local storage space of the service instance. However, during the research process, the inventors found that the local storage space applied for the service instance is usually relatively limited. Therefore, the local storage space can usually only temporarily store model data for a limited number of models, which is far less than the number of models involved in the multi-model dynamic loading scenario. As a result, the probability of hitting the required model data in the local storage space of the service instance is usually relatively low.

[0077] For this purpose, refer to Figure 2 In step 101, it is further proposed that when the model data required for loading the target model does not exist in the local storage space, an attempt can be made to obtain the model data from other service instances deployed for the model service.

[0078] Based on the pre-established data transmission network, the multiple service instances deployed for the model service are able to communicate with each other. Step 101 fully utilizes this communication capability to share model data among the multiple service instances. Thus, if the required model data is stored on any other service instance deployed for the model service, the required model data can be retrieved from the target service instance storing the model data in step 101 based on the data transmission network established between the service instances.

[0079] refer to Figure 3 The model service device provided in this embodiment may also include a remote download module, based on which a remote download model can be used to execute the operation of obtaining the required model data from the target service instance. Figure 3After determining that the required model data does not exist in the local storage space, the processing module can trigger the remote download module to query whether the required model data is stored on other service instances. If so, it can initiate a remote download instruction to the target service instance to obtain the required model data from the target service instance and store it in the local storage space. If the remote download module finds that the required model data does not exist on other service instances, it can provide feedback to the processing module, allowing it to trigger other remedial logic.

[0080] Correspondingly, a service instance may also receive remote download instructions from other service instances in the data transmission network. If a service instance receives a remote download instruction from any other service instance in the data transmission network, it determines the target model data requested by the remote download instruction, reads the target model data from the local storage space, and sends the read target model data to the service instance that issued the remote download instruction via the data transmission network. Figure 3 ,There is a communication channel between the remote download modules in each ,service instance in the data transmission network. Based on this, each remote download module can send remote download ,instructions to each other to achieve the mutual transmission of model ,data, which is model data sharing.

[0081] During the research process, the inventors found that although from the perspective of a single service instance, the model data that can be temporarily stored in its local storage space is limited, from the global perspective of multiple service instances deployed for the model service, the temporarily stored model data is still relatively comprehensive. This is because the model data temporarily stored in different service instances may be different. Therefore, the model data corresponding to the models provided by the model service is usually not yet covered, especially the model data corresponding to the models recently provided by the model service are basically stored on a certain service instance. In this way, in step 101, even if the required model data does not exist in the local storage space of the service instance, the required model data can usually be obtained from other service instances. In addition, during the research process, the inventors also found that multiple service instances are usually located in the same virtual private network (VPN), which ensures that excellent network transmission rates can be obtained between service instances. Therefore, the service instance can obtain model data from other service instances very efficiently, and the process is very short.

[0082] After obtaining the required model data from other service instances, the service instance can store the obtained model data in the local storage space.

[0083] Continue to refer Figure 1In step 102, the target model can be loaded based on the model data stored in the local storage space. That is, after the model data required to load the target model is stored in the local storage space of the service instance, the service instance can complete the loading of the target model. After loading the target model, the target model can be used to respond to the aforementioned model call request. In actual applications, the model call request typically carries an inference task, and the loaded target model can complete the inference task. The service instance can then use the processing results generated by the target model as the response to the model call request.

[0084] It is understandable that after the service instance receives other model call requests subsequently, if the other model call requests no longer require loading the target model but a new model, the service instance needs to load the new model according to the model service method provided in this embodiment to achieve efficient model switching.

[0085] Furthermore, in this embodiment, if the model data does not exist in other service instances in the data transmission network, the model data can be downloaded from the data source; the downloaded model data is stored in the local storage space. As mentioned above, reference Figure 3 If the remote download module finds that the required model data does not exist on other service instances, it can provide feedback to the processing module. After receiving this feedback, the processing module can trigger the remedial logic of downloading the model data from the data source. Figure 3 The model service device provided in this embodiment may also include a back-to-source download module. After receiving this feedback from the remote download module, the processing module may trigger the back-to-source download module to download the required model data from the data source and store it in the local storage space.

[0086] Among them, various model data are stored in the data source, which can be understood as a relatively comprehensive model library. In this way, based on the data source, a bottom-line guarantee can be provided for model call requests. It should be understood that the network transmission rate between the data source and the service instance is usually relatively low, so the time required for the service instance to download back to the source will be relatively high. However, as mentioned above, based on the model service method provided by this embodiment, after receiving the model call request, the local storage space is first queried. If the required model data does not exist in the local storage space, it will try to obtain the required model data from other service instances. In actual applications, it is highly likely that the required model data can be obtained through the direct model data sharing mechanism of multiple service instances in the data transmission network. Therefore, the probability of returning to the source can be effectively reduced.

[0087] In summary, in this embodiment, it is proposed to organize multiple service instances deployed for model services into a data transmission network, and install the model service device provided by this embodiment in the service instance, so as to realize the transformation of the processing logic of the model call request in the service instance. On this basis, from the perspective of a single service instance, after receiving a model call request, it can first query whether the model data required for model loading exists in the local storage space. If not, it can try to remotely download the required model data from other service instances based on the data transmission network, and store the downloaded model data in the local storage space. In this way, in the scenario of dynamic loading of multiple models, model data sharing can be achieved based on the data transmission network established between multiple service instances. Since the network transmission rate between each service instance in the data transmission network is much higher than the back-to-source download rate, even if the model data required for loading does not exist locally in the service instance, the model data can be efficiently downloaded remotely from other service instances based on the data transmission network, thereby effectively improving the efficiency of model loading and thus improving the efficiency of responding to model call requests.

[0088] In the above or following embodiments, it is also proposed that storage status information can be created for each service instance in the data transmission network. The storage status information can be used to record descriptions of the model data stored in the corresponding service instance. In this way, the storage status information can reflect the model data stored in the service instance.

[0089] Based on this, this embodiment proposes that within a service instance, storage status information can be maintained for each of the other service instances in the data transmission network. Furthermore, if the model data required to load the target model does not exist in the aforementioned local storage space, the storage status information maintained by each service instance in the data transmission network can be queried to select the target service instance that stores the model data.

[0090] Furthermore, this embodiment introduces a broadcast mechanism into the data transmission network: after a service instance stores model data in its local storage space, it can broadcast to other service instances in the data transmission network, allowing the other service instances in the data transmission network to update the storage status information corresponding to the service instance. That is, after any service instance in the data transmission network stores model data in its local storage space, it can broadcast the model data storage event that occurred to it to other service instances based on the broadcast mechanism introduced in the data transmission network. This can trigger the other service instances to promptly update the storage status information they maintain for the service instance. Thus, by introducing a broadcast mechanism into the data transmission network and utilizing it to promptly synchronize the model data storage status of each service instance in the data transmission network, the storage status information maintained by each service instance in the data transmission network can be maintained consistent and updated promptly, thereby ensuring that model data can be accurately shared between the service instances.

[0091] Optionally, in this embodiment, a broadcast mechanism may be implemented between service instances in the data transmission network based on a gossip protocol or the like. This embodiment does not limit the broadcast protocol used in the broadcast mechanism.

[0092] Undertake Figure 3 The internal logical structure of the model service device allows the processing module to maintain the aforementioned storage status information, and the network component module to perform broadcast operations. Based on this, after completing the model data storage operation, the processing module can trigger the network component module to perform the broadcast operation; and after receiving broadcasts from other service instances, the network component module can trigger the processing module to promptly update the storage status information maintained for the corresponding service instance. In this way, the processing module and the network component module cooperate to ensure that the storage status information maintained by each service instance in the data transmission network remains consistent and is updated in a timely manner. This also provides the processing module with an accurate basis to ensure that the selected target service instance actually stores the required model data.

[0093] Furthermore, during their research, the inventors discovered that there may be multiple service instances storing the same model data in a data transmission network. That is, there may be more than one service instance in the data transmission network capable of providing the model data required to load the target model. To this end, this embodiment further proposes a remote download instruction balancing mechanism: if more than one service instance stores the same model data, the service instance with the lowest load responds to the remote download instruction for that model data. In an exemplary balancing scheme, each service instance in the data transmission network can broadcast its load status to each other based on the broadcast mechanism introduced in the aforementioned network assembly module. In this way, a single service instance can maintain load status information for each of the other service instances in the data transmission network. Accordingly, the load status information can record information such as the number of remote download instructions received by the corresponding service instance, thereby reflecting the load status of the corresponding service instance. Based on this, if a service instance needs to remotely download model data and there are multiple service instances in the data transmission network storing the model data, the current service instance can identify the service instance that stores the required model data and has the lowest load as the target service instance and initiate a remote download instruction to the target service instance.

[0094] In this way, remote download instructions for the same model data can be distributed to related service instances for response, rather than being concentrated on one service instance and thus burdening the service instance.

[0095] In summary, this embodiment introduces a broadcast mechanism into the data transmission network and innovatively proposes broadcasting the storage status of service instances based on the broadcast mechanism to ensure that each service instance in the data transmission network can accurately perceive the model data present in the data transmission network and on which service instance(s) the model data is stored, thereby enabling service instances to seamlessly perceive the service instances that can store the required model data. Furthermore, an innovative method is also proposed to broadcast the load status of service instances based on the broadcast mechanism to ensure that each service instance in the data transmission network can accurately perceive the load status of other service instances, thereby achieving load balancing of remote download instructions and avoiding burdening some service instances.

[0096] In the above or following embodiments, it is also proposed that a user-mode file system can be introduced to manage the model data stored in the service instance.

[0097] For this purpose, refer to Figure 3, a Filesystem in Userspace (FUSE) component can be configured in the processing module of the service instance, and a user-space file system can be established in the service instance based on this component. It should be understood that the user-space file system pre-creates file paths for various model data that may be used by the service instance, and these file paths can be maintained in the metadata information corresponding to the user-space file system. The implementation principles of the user-space file system will not be elaborated here.

[0098] On this basis, reference Figure 3 This embodiment also proposes that the aforementioned data source can be mounted on a user-state file system in a service instance. Moreover, the metadata information corresponding to the user-state file system can record the file description information corresponding to the model data in the data source, including but not limited to file identification, file specifications, file type, and storage location.

[0099] In this way, when the processing module needs to download the model data from the data source, it can query the first file description information corresponding to the model data in the data source from the metadata information maintained for the user-mode file system; and download the model data from the data source based on the first file description information. Figure 3 The processing module can provide the first file description information to the back-to-source download module, which can then download data matching the first file description information from the data source, i.e., the model data required for loading the target model. This ensures that the required model data is correctly downloaded from the data source.

[0100] refer to Figure 3 In this embodiment, it is also proposed that the local storage space in the service instance can be mounted to the user-state file system in the service instance. Moreover, the metadata information corresponding to the user-state file system can record the file description information corresponding to the model data in the local storage space, including but not limited to file identification, file specifications, file type, and storage location.

[0101] In this way, the processing module can provide an access interface to the service module based on the user-mode file system created for the service instance. After receiving the model call request, the service module can query the file path of the required model data from the user-mode file system and initiate a file access request to the access interface of the processing module based on the queried file path. After receiving the file access request, the processing module can query the metadata information maintained for the user-mode file system for the second file description information corresponding to the required model data in the local storage space, and then query whether there is data in the local storage space that matches the second file description information. If so, it can be determined that the required model data already exists in the local storage space; if not, it can be determined that the required model data does not exist in the local storage space.

[0102] In summary, this embodiment proposes introducing a user-mode file system and mounting a local storage space on this user-mode file system. For the target model required for loading a model usage request, the user-mode file system can provide the file path corresponding to the model data required for loading the target model. However, the model data may not necessarily already exist in the local storage space. Therefore, this embodiment proposes, based on the mounting relationship between the user-mode file system and the local storage space, to conveniently and quickly query whether the required model data exists in the local storage space. If not, the required model data can be promptly downloaded from the target service instance storing the model data. If other service instances in the data transmission network also do not store the required model data, a back-to-source download operation can be triggered. This embodiment also proposes mounting the data source on the user-mode file system, which ensures that the required model data can be accurately downloaded during the back-to-source download process. As can be seen, by introducing the user-mode file system, data access operations to the current storage space and data source can be converted into unified user-mode file system access operations. This not only ensures that the required model data can be accurately used, but also effectively reduces the logical complexity within the service instance and effectively improves the service instance's response efficiency to model call requests.

[0103] It should be noted that in some of the processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different file description information, etc., and do not represent the order of precedence, nor do they limit "first" and "second" to be different types.

[0104] Figure 4 This is a schematic diagram of a computing device provided by another exemplary embodiment of the present application. Figure 4 As shown, the computing device includes: a memory 40, a processor 41 and a communication component 42. The computing device carries a service instance deployed for the model service, the service instance is used to dynamically load the model, and the service instance is pre-deployed with other service instance components deployed for the model service as a data transmission network;

[0105] Memory 40, for storing one or more computer instructions;

[0106] The processor 41 is coupled to the memory 40 and the communication component 42 for executing the one or more computer instructions for:

[0107] In response to the received model call request, determining the target model to be loaded;

[0108] If the model data required for loading the target model does not exist in the local storage space, obtaining the model data from the target service instance storing the model data and storing the model data in the local storage space, wherein the multiple service instances are pre-organized into a data transmission network;

[0109] The target model is loaded based on the model data stored in the local storage space.

[0110] In an optional embodiment, when the processor 41 obtains the model data from the target service instance storing the model data, it may be specifically configured to:

[0111] querying storage status information maintained for each service instance in the data transmission network to select the target service instance storing the model data;

[0112] Downloading the model data from the target service instance;

[0113] The storage status information is used to record description information of the model data stored in the corresponding service instance.

[0114] In an optional embodiment, the processor 41 may also be configured to:

[0115] After storing the model data in the local storage space, a broadcast carrying the identification information of the model data is sent to other service instances in the data transmission network, so that each other service instance updates the storage status information maintained by itself for the service instance based on the identification information of the model data.

[0116] In an optional embodiment, the processor 41 may also be configured to:

[0117] If there are multiple service instances storing the model data in the data transmission network, then

[0118] Selecting the service instance with the lowest load as the target service instance;

[0119] Each service instance in the data transmission network broadcasts its own load status respectively.

[0120] In an optional embodiment, the processor 41 may also be configured to:

[0121] If the model data does not exist in any other service instance in the data transmission network, performing a back-to-source download operation to download the model data from the data source;

[0122] The downloaded model data is stored in the local storage space.

[0123] In an optional embodiment, the data source is mounted on a user-mode file system constructed in the service instance. When executing the back-to-source download operation, the processor 41 may be specifically configured to:

[0124] Querying first file description information corresponding to the model data in the data source from metadata information maintained for the user-mode file system;

[0125] The model data is downloaded from the data source according to the first file description information.

[0126] In an optional embodiment, the local storage space is mounted on a user-mode file system constructed in the service instance, and the processor 41 may further be configured to:

[0127] Querying the second file description information corresponding to the model data in the local storage space from the metadata information maintained for the user-mode file system;

[0128] If no data matching the second file description information is found in the local storage space, it is determined that the model data does not exist in the local storage space.

[0129] In an optional embodiment, the processor 41 may also be configured to:

[0130] After the service instance is created, request a list of service instances corresponding to the model service from the management instance deployed for the model service;

[0131] Join the data transmission network that has been established by each service instance in the service instance list.

[0132] In an optional embodiment, the processor 41 may also be configured to:

[0133] If a remote download instruction is received from any other service instance in the data transmission network, determining the target model data requested by the remote download instruction;

[0134] Reading the target model data from the local storage space;

[0135] The target model data is sent to the service instance that issues the remote download instruction through the data transmission network.

[0136] Further, if Figure 4 As shown, the computing device also includes: a power supply component 43 and other components. Figure 4 Only some components are shown schematically, and it does not mean that the computing device only includes Figure 4 Components shown.

[0137] It is worth noting that the technical details in the above-mentioned embodiments of the computing device can be referred to the relevant description in the aforementioned method embodiment. In order to save space, they will not be repeated here, but this should not cause any loss of the scope of protection of this application.

[0138] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which can implement the steps in the above method embodiment when the computer program is executed.

[0139] Accordingly, an embodiment of the present application further provides a computer program product, which can implement the steps in the above method embodiment when the computer program contained therein is executed.

[0140] above Figure 4 The memory in the computing platform is used to store computer programs and can be configured to store various other data to support operations on the computing platform. Examples of such data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc. The memory can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0141] above Figure 4 The communication component in is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G / LTE, 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0142] above Figure 4 The power supply component in a device provides power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0143] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0144] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0145] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0147] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0148] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0149] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included in the protection scope of the present application.

Claims

1. A model service method, characterized in that: Multiple service instances are deployed for the model service, and the method is applicable to any service instance of the multiple service instances, and the method includes: In response to the received model call request, determining the target model to be loaded; If the model data required for loading the target model does not exist in the local storage space, obtaining the model data from the target service instance storing the model data and storing the model data in the local storage space, wherein the multiple service instances are pre-organized into a data transmission network; The target model is loaded based on the model data stored in the local storage space.

2. The method according to claim 1, characterized in that Acquiring the model data from a target service instance storing the model data includes: querying storage status information maintained for each service instance in the data transmission network to select the target service instance storing the model data; Downloading the model data from the target service instance; The storage status information is used to record description information of the model data stored in the corresponding service instance.

3. The method according to claim 2, characterized in that Also includes: After storing the model data in the local storage space, a broadcast carrying the identification information of the model data is sent to other service instances in the data transmission network, so that each other service instance updates the storage status information maintained by itself for the service instance based on the identification information of the model data.

4. The method according to claim 2, characterized in that Also includes: If there are multiple service instances storing the model data in the data transmission network, then Selecting the service instance with the lowest load as the target service instance; Each service instance in the data transmission network broadcasts its own load status respectively.

5. The method according to claim 1, wherein Also includes: If the model data does not exist in any other service instance in the data transmission network, performing a back-to-source download operation to download the model data from the data source; The downloaded model data is stored in the local storage space.

6. The method according to claim 5, characterized in that The data source is mounted on a user-mode file system constructed in the service instance. The back-to-source download operation includes: Querying first file description information corresponding to the model data in the data source from metadata information maintained for the user-mode file system; The model data is downloaded from the data source according to the first file description information.

7. The method according to claim 1, characterized in that The local storage space is mounted on a user-mode file system constructed in the service instance, and the method further includes: Querying the second file description information corresponding to the model data in the local storage space from the metadata information maintained for the user-mode file system; If no data matching the second file description information is found in the local storage space, it is determined that the model data does not exist in the local storage space.

8. The method according to claim 1, characterized in that Also includes: After the service instance is created, request a list of service instances corresponding to the model service from the management instance deployed for the model service; Join the data transmission network that has been established by each service instance in the service instance list.

9. The method according to claim 1, characterized in that Also includes: If a remote download instruction is received from any other service instance in the data transmission network, determining the target model data requested by the remote download instruction; Reading the target model data from the local storage space; The target model data is sent to the service instance that issues the remote download instruction through the data transmission network.

10. A computing device, characterized in that The computing device includes a memory, a processor, and a communication component. The computing device carries a service instance deployed for the model service. The service instance is used to dynamically load the model. The service instance is pre-connected with other service instance components deployed for the model service as a data transmission network. The memory is used to store one or more computer instructions; The processor is coupled to the memory and the communication component, and is configured to execute the one or more computer instructions for executing the model service method according to any one of claims 1 to 9.

11. A model service device, characterized in that: are respectively arranged in a plurality of service instances deployed for the model service, wherein the model service device includes a service module, a processing model and a network building module; The network building module is used to control the service instance to build a data transmission network with other service instances deployed for the model service; The service module is used to determine the target model to be loaded in response to the received model call request; The processing module is configured to, if the model data required for loading the target model does not exist in the local storage space of the service instance, obtain the model data from the target service instance storing the model data and store it in the local storage space; The target model is loaded based on the model data stored in the local storage space.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by one or more processors, the one or more processors are caused to execute the model service method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The invention comprises a computer program, which, when executed by one or more processors, causes the one or more processors to execute the model service method according to any one of claims 1 to 9.

Images