Model inference service method and apparatus, and storage medium

Abstract

The present disclosure provides a model inference service method and device and a storage medium, relates to the technical field of communication, and solves the technical problem that a client waits for a model inference service result for too long in related technologies. The method comprises the following steps: analyzing model parameters, determining a plurality of model inference service identifiers and a plurality of model inference methods in the model parameters; constructing a model inference service class according to the plurality of model inference service identifiers and the plurality of model inference methods; obtaining a model inference service request from a client; calling a first function in the model inference service class, and searching for a target model inference service identifier according to the first function; and calling a model inference method corresponding to the target model inference service identifier in the model inference service class, and processing the model inference service request. The present disclosure is used in the scene of an artificial intelligence (AI) model service application.

Description

Technical Field

[0001] This disclosure relates to the field of communication technology, and in particular to a model reasoning service method, apparatus and storage medium. Background Technology

[0002] Currently, in the application of artificial intelligence (AI) models as services, a common method is to place the AI ​​model inference service program in the background of a web server. When the web server receives an inference service request from the client, it calls the model inference program through the web server's application program interface (API) to load the model parameters and perform inference calculations according to the inference method of the target model inference object, thereby realizing the model inference service.

[0003] However, since this method needs to repeatedly load the same model parameters after receiving a model inference request from the client, and the model parameter file is often very large, it takes a long time to load the model file into memory space, resulting in a long waiting time for the client to receive the result. Summary of the Invention

[0004] This disclosure provides a model inference service method, apparatus, and storage medium. It solves the technical problems in related technologies where AI model service applications require repeatedly loading the same model parameters, resulting in significant time consumption when loading model files into memory and long client wait times for results.

[0005] To achieve the above objectives, the present disclosure adopts the following technical solution:

[0006] Firstly, a model inference service method is provided, comprising: parsing model parameters to determine multiple model inference service identifiers and multiple model inference methods in the model parameters, wherein each model inference service identifier corresponds one-to-one with a model inference method; constructing a model inference service class based on the multiple model inference service identifiers and multiple model inference methods; the model inference service class includes a first function and multiple model inference methods, wherein multiple variables of the first function are the multiple model inference service identifiers; obtaining a model inference service request from a client; the model inference service request includes a target model inference service identifier; calling the first function in the model inference service class to find the target model inference service identifier based on the first function; and calling the model inference method in the model inference service class corresponding to the target model inference service identifier to process the model inference service request.

[0007] In conjunction with the first aspect above, in one possible implementation, the method further includes: constructing a second function, the second function comprising multiple variables; assigning multiple model inference service identifiers to the multiple variables of the second function to determine a first function; one variable of the first function pointing to the model inference method corresponding to the model inference service identifier assigned to the variable; and constructing a model inference service class based on the first function and the multiple model inference methods.

[0008] In conjunction with the first aspect above, in one possible implementation, the method further includes: a second function being a function written in the system language of the Web server; the Web server being used to process model inference service requests; and the model inference service class being a service class written in the system language of the Web server.

[0009] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: invoking an HTTP service listener to periodically monitor the client's service requests; and after listening to the client's model inference service request, parsing the target model inference service identifier from the model inference service request.

[0010] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: using an object-oriented reflection mechanism to search for the target model inference service identifier in the first function; determining the target model inference method pointed to by the variable in the first function that is assigned the target model inference service identifier; and calling the target model inference method to process the model inference service request.

[0011] In conjunction with the first aspect mentioned above, in one possible implementation, the method further includes: determining the system language of the web server; and, based on the system language, writing multiple model inference methods and storing them in system files.

[0012] Secondly, a model inference service apparatus is provided, comprising: a communication unit and a processing unit; the processing unit is configured to parse model parameters, determine multiple model inference service identifiers and multiple model inference methods in the model parameters, wherein each model inference service identifier corresponds one-to-one with a model inference method; the processing unit is configured to construct a model inference service class based on the multiple model inference service identifiers and multiple model inference methods; the model inference service class includes a first function and multiple model inference methods, wherein multiple variables of the first function are multiple model inference service identifiers; the communication unit is configured to obtain a model inference service request from a client; the model inference service request includes a target model inference service identifier; the processing unit is configured to call the first function in the model inference service class and search for the target model inference service identifier based on the first function; the processing unit is configured to call the model inference method in the model inference service class corresponding to the target model inference service identifier and process the model inference service request.

[0013] In conjunction with the second aspect above, in one possible implementation, the processing unit is specifically used for: constructing a second function, the second function including multiple variables; assigning multiple model inference service identifiers to the multiple variables of the second function to determine a first function; one variable of the first function pointing to the model inference method corresponding to the model inference service identifier assigned to the variable; and constructing a model inference service class based on the first function and the multiple model inference methods.

[0014] In conjunction with the second aspect above, in one possible implementation, the second function is a function written in the system language of the Web server; the Web server is used to process model inference service requests; and the model inference service class is a service class written in the system language of the Web server.

[0015] In conjunction with the second aspect above, in one possible implementation, the communication unit is specifically used to call the HTTP service listener to periodically monitor the client's service requests; the processing unit is specifically used to parse the target model inference service identifier from the model inference service request after listening to the client's model inference service request.

[0016] In conjunction with the second aspect above, in one possible implementation, the processing unit is specifically used to: use an object-oriented reflection mechanism to find the target model inference service identifier from the first function; determine the target model inference method pointed to by the variable in the first function that is assigned the target model inference service identifier; and call the target model inference method to process the model inference service request.

[0017] In conjunction with the second aspect above, in one possible implementation, after parsing the model parameters and determining multiple model inference service identifiers and multiple model inference methods in the model parameters, the processing unit is further configured to: determine the system language of the Web server; and, based on the system language, write multiple model inference methods and store them in system files.

[0018] Thirdly, a model inference service apparatus is provided, comprising: a processor and a memory; wherein the memory is used to store computer execution instructions, and when the model inference service apparatus is running, the processor executes the computer execution instructions stored in the memory to cause the model inference service apparatus to perform the model inference service method described in the first aspect and any possible implementation thereof.

[0019] Fourthly, a computer-readable storage medium is provided, which stores instructions that, when executed by a processor of a model inference service device, cause the model inference service device to perform the model inference service method described in the first aspect and any possible implementation thereof.

[0020] In this disclosure, the names of the aforementioned model inference service devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this disclosure, they fall within the scope of this disclosure and its equivalents.

[0021] These or other aspects of this disclosure will become more readily apparent in the following description.

[0022] The technical solution provided in this disclosure offers at least the following advantages: The model inference service device loads model parameters, identifies multiple model inference service identifiers and multiple model inference methods within the model parameters, and constructs a model inference service class in the storage system of the model inference service device based on these identifiers and methods. When the model inference service device receives a model inference service request from a client, it only needs to construct a model inference service object with the same name as the client's model inference service request in the model inference service class determined by the object reflection mechanism, and call the corresponding model inference method to perform model inference calculations. This avoids the need to repeatedly load model parameters every time a model inference service request from a client is received, thereby shortening the client's waiting time for the model inference service result. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0024] Figure 1 A schematic diagram of the hardware structure of a model inference service device provided in an embodiment of this disclosure;

[0025] Figure 2 This is a schematic diagram of a model reasoning service framework provided in an embodiment of the present disclosure;

[0026] Figure 3 A flowchart illustrating a model inference service method provided in an embodiment of this disclosure;

[0027] Figure 4 A flowchart illustrating yet another model reasoning service method provided in this disclosure embodiment;

[0028] Figure 5 A flowchart illustrating yet another model reasoning service method provided in this disclosure embodiment;

[0029] Figure 6 A flowchart illustrating yet another model reasoning service method provided in this disclosure embodiment;

[0030] Figure 7A flowchart illustrating yet another model reasoning service method provided in this disclosure embodiment;

[0031] Figure 8 This is a schematic diagram of the structure of a model reasoning service device provided in an embodiment of this disclosure. Detailed Implementation

[0032] The following description, in conjunction with the accompanying drawings, details a model reasoning service method, apparatus, and storage medium provided in the embodiments of this disclosure.

[0033] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0034] The terms “first” and “second” in this disclosure and its accompanying drawings are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a particular order of objects.

[0035] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this disclosure are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus. It should be noted that in the embodiments of this disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this disclosure should not be construed as preferred or advantageous over other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0036] In the description of this disclosure, unless otherwise stated, "multiple" means two or more.

[0037] Figure 1 This is a schematic diagram of the structure of a model inference service device provided in an embodiment of this disclosure, as shown below. Figure 1 As shown, the model inference service device 100 includes at least one processor 101, a communication line 102, and at least one communication interface 104, and may also include a memory 103. The processor 101, memory 103, and communication interface 104 can be connected via the communication line 102.

[0038] The processor 101 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this disclosure, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0039] Communication line 102 may include a path for transmitting information between the aforementioned components.

[0040] The communication interface 104 is used to communicate with other devices or communication networks. It can use any transceiver-like device, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0041] The memory 103 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of including or storing desired program code having the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0042] In one possible design, the memory 103 can exist independently of the processor 101, meaning the memory 103 can be an external memory of the processor 101. In this case, the memory 103 can be connected to the processor 101 via a communication line 102 to store execution instructions or application code, and its execution is controlled by the processor 101 to implement the model inference service method provided in the following embodiments of this disclosure. In another possible design, the memory 103 can also be integrated with the processor 101, meaning the memory 103 can be an internal memory of the processor 101. For example, the memory 103 can be a cache, which can be used to temporarily store some data and instruction information.

[0043] As one possible implementation, processor 101 may include one or more CPUs, for example Figure 1 CPU0 and CPU1 in the example. Alternatively, the model inference service device 100 may include multiple processors, such as CPU0 and CPU1. Figure 1 The processors 101 and 107 are included. Alternatively, the model inference service device 100 may also include an output device 105 and an input device 106.

[0044] Through the above description of the implementation methods, those skilled in the art will clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the network node can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, modules, and network nodes described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0045] The following explanations of the terms used in the embodiments of this disclosure are provided to facilitate the reader's understanding.

[0046] 1. AI Model Inference

[0047] AI model inference service is a service method that uses AI model inference methods to perform inference calculations, thereby enabling rapid processing of data resources.

[0048] Currently, with the rapid development of artificial intelligence, deep learning-based AI model inference has become one of the important methods for rapidly processing data resources. Deep learning mainly includes two processes: 1.1 AI model training and 1.2 model inference.

[0049] 1.1. AI Model Training

[0050] Model training is the process of using a large training dataset and a specific deep learning algorithm to train and obtain a set of optimal neural network model parameters.

[0051] 1.2. Model Reasoning

[0052] Model inference is the process of using pre-trained model parameters to solve real-world problems.

[0053] The inference process of deep learning-based AI models is generally completed through the following two steps, 1 and 2:

[0054] Step 1, Loading Model Parameters: Load the model parameters from the pre-trained model parameter file into memory and create a model inference object.

[0055] Step 2, Model Inference: Based on the parameters from the client's request data, the model inference method of the corresponding model inference object is called to perform inference calculations, thereby obtaining the model prediction results.

[0056] Currently, most common AI model inference service platforms involve technical personnel encapsulating AI model inference programs into services and placing the encapsulated AI model inference programs in the background of a web server. After the web server receives a model inference service request from the client, it loads the model parameters, calls the corresponding model inference methods in the model inference program through the application program interface (API), performs inference calculations according to the model inference methods, and returns the inference calculation results to the client, thereby realizing the service-oriented nature of AI model inference.

[0057] However, the above method combines the model parameter loading and model inference steps into a single API service program. Each time the web server receives a model inference request from a client, it needs to execute steps 1 and 2 sequentially in the server background to complete the AI ​​model inference service and return the prediction result to the client. In other words, every time a user's inference service request is received, all model parameters need to be loaded repeatedly. Since model parameter files are often very large, loading them takes a considerable amount of time, resulting in a long waiting time for the client to receive the result.

[0058] To solve the above-mentioned technical problems, this disclosure provides a method such as Figure 2 The model inference service framework 20 shown is as follows: Figure 2 As shown, the model inference service framework 20 mainly includes: a client 201, a model inference service program 202, and a web server main program 203. These are described in detail below.

[0059] (1) Client 201 sends a model inference service request to Web server main program 203. The model inference service request instructs Web server main program 203 to call the corresponding model inference method and execute the model inference service. The model inference service request includes the name of the model inference service object. For example, client 201 can be a terminal used by the user, such as a mobile phone, computer, smart wearable device, etc., which is not limited in this application.

[0060] (2) The model inference service program 202 is used to determine the model inference object and its corresponding model inference method after loading the model parameter file. Optionally, the model inference service program 202 defines a model inference service class, which includes a constructor (__init__) and several model service methods. The constructor (__init__) is used to define several member variables, and the initial value of each member variable is the model inference object obtained by loading the model parameter file into memory.

[0061] For example, the programming language that defines the constructor (__init__) can be:

[0062] def(__init__)(self):

[0063] self.model_a=load_model_from_file(' / models / model_a.model')

[0064] self.model_b=load_model_from_file(' / models / model_b.model')

[0065] self.model_c=load_model_from_file(' / models / model_c.model')

[0066] For example, the programming language for defining the service methods of the model can be:

[0067] def model_a_service(self,*args):

[0068] result=self.model_a.predict(args)

[0069] return result

[0070] def model_b_service(self,*args):

[0071] result=self.model_b.predict(args)

[0072] return result

[0073] def model_c_service(self,*args):

[0074] result=self.model_c.predict(args)

[0075] return result

[0076] The programming language for creating instances of the Model Inference Service object can be: model_services = ModelServices()

[0077] (3) During initialization, the Web server main program 203 loads the model parameter file and assigns values ​​to the member variables of the constructor (__init__) in the model inference service class based on the model parameter file, thereby determining the model inference object and the corresponding model inference method in the model inference service class. After loading the model parameter file during initialization, when the Web server main program 203 receives a model inference service request from the client, it only needs to use the object-oriented reflection mechanism to find the model inference service object with the same name in the model inference service object and call the corresponding model inference method to perform inference calculation, quickly obtain the result of this model inference calculation, and send the model inference service result to the client.

[0078] For example, a model inference service class, `ModelServices`, is defined in the model inference service program. This `ModelServices` class defines a constructor (`__init__`) and several model service methods. In the constructor of the `ModelServices` class, several member variables are defined. Each member variable is initialized with a model inference object loaded from the model parameter file into memory. When a model inference service request is received from a client, the `ModelServices` class can call the inference method of the corresponding member variable, i.e., the model inference object, to perform the actual model inference calculation. During the initialization and startup phase of the web server main program, the statement `from services import model_services` is executed, importing the variable `model_services` from the `services.py` program module. A `ModelServices` object is created based on the imported variable `model_services` and assigned to the variable `model_services`. During the creation of the `ModelServices` object, the constructor (`__init__`) is executed, loading model inference objects from each model parameter file and assigning them to the respective member variables.

[0079] In one possible implementation, the web server listens for and receives HTTP service requests from clients, and determines from the service requests that the model inference service to be accessed is named animal_classify_service.

[0080] The web server uses the model inference service name and, using object-oriented reflection, searches for a method object with the same name in the `model_services` object. Based on the statement `service_obj = getattr(model_services, 'animal_classify_service')`, it calls the corresponding model inference method object, performs the actual model inference calculation, and returns the result to the client.

[0081] The above process is executed continuously in a loop. During this process, each time the web server receives a request for the inference service from the client, it only needs to find the corresponding pre-loaded model inference object and call its inference method when calling the background model inference service according to the client's request. Instead of loading the model inference object from the model parameter file every time, it can shorten the model service response time, improve the model service performance, and shorten the client's waiting time.

[0082] The above provides a detailed description of the model reasoning service framework 20 involved in the embodiments of this application.

[0083] Currently, in AI model service applications, a common approach is to place the AI ​​model inference service program in the backend of a web server. When the web server receives an inference service request from a client, it calls the model inference program through the web server's application programming interface (API) to load model parameters and perform inference calculations based on the inference method of the target model inference object, thus realizing the model inference service. However, because this method requires repeatedly loading the same model parameters after receiving a model inference request from the client, and model parameter files are often very large, loading the model file into memory takes a long time, resulting in a long waiting time for the client to receive the results.

[0084] To address the technical problems existing in related technologies, this disclosure provides a model inference service method. The model inference service device loads model parameters, determines multiple model inference service identifiers and multiple model inference methods within the model parameters, and constructs a model inference service class in the storage system of the model inference service device based on the multiple model inference service identifiers and multiple model inference methods. When the model inference service device receives a model inference service request from a client, it only needs to construct a model inference service object with the same name as the client's model inference service request in the model inference service class determined by the object reflection mechanism, and call the corresponding model inference method to perform model inference calculations. This avoids the need to repeatedly load model parameters every time a model inference service request is received from a client, thereby shortening the client's waiting time for the model inference service result.

[0085] like Figure 3 As shown, Figure 3 The model inference service method provided in this disclosure is used to shorten the time for clients to wait for model inference service results. The method includes the following S301-S305, which are described in detail below.

[0086] S301, The model inference service device parses the model parameters and determines multiple model inference service identifiers and multiple model inference methods in the model parameters.

[0087] The model reasoning service identifier corresponds one-to-one with the model reasoning method.

[0088] In one possible implementation, the model inference service device loads model parameters stored on the hard disk during initial startup, determines multiple model inference service identifiers in the model parameters, and the model inference method corresponding to each model inference service identifier.

[0089] For example, the model inference service identifier can be the name of the model inference service object.

[0090] S302. The model reasoning service device constructs a model reasoning service class based on multiple model reasoning service identifiers and multiple model reasoning methods.

[0091] The model inference service class includes a first function and multiple model inference methods. The multiple variables of the first function are multiple model inference service identifiers.

[0092] In one possible implementation, after determining multiple model reasoning service identifiers in the model parameters and multiple model reasoning methods corresponding to the model reasoning service identifiers, the model reasoning service device determines a model reasoning service class based on the multiple model reasoning service identifiers, and determines the correspondence between the first function in the model reasoning service class and the model reasoning method.

[0093] For example, if the model inference object is animal recognition (animal_classify), and the model parameters for animal recognition are stored in the file " / models / animal_classify.model", then the first function is:

[0094] def(__init__)(self):

[0095] The model inference object is `self.animal_classify = load_model_from_file(' / models / animal_classify.model')`. The model parameters for handwriting recognition are stored in the file ` / models / handwriting.model`. Therefore, the first function is:

[0096] def(__init__)(self):

[0097] self.handwriting=load_model_from_file(' / models / handwriting.model')

[0098] S303, The model inference service device receives a model inference service request from the client.

[0099] The model inference service request includes the target model inference service identifier.

[0100] For example, when a client needs to perform model inference services for animal identification, the animal identification identifier is written into the model inference service request. After receiving the model inference service request from the client, the model inference service device parses the model inference service and determines that the object of this model inference service is animal identification.

[0101] S304. The model reasoning service device calls the first function in the model reasoning service class to find the target model reasoning service identifier from the first function.

[0102] In one possible implementation, the model inference service device determines the model inference service object corresponding to the first function and its corresponding model inference method based on the model inference service identifier in the loaded model parameters. When a model inference service request is received from the client, the device determines the model inference service object with the same name as the target model inference service identifier.

[0103] For example, after determining that the model reasoning service object is animal recognition, the model reasoning service device determines that the model reasoning service class contains a first function for animal recognition.

[0104] S305. The model reasoning service device calls the model reasoning method corresponding to the target model reasoning service identifier in the model reasoning service class to process the model reasoning service request.

[0105] For example, after determining that the model inference service class contains the target first function for animal recognition and handwritten digit recognition, the model inference service device calls the animal recognition model inference method and the handwritten digit recognition model inference method corresponding to the first function to process the client's model inference service request for animal recognition and handwritten digit recognition.

[0106] For example, in the above example, the following two model service methods need to be defined:

[0107] def animal_classify_service(self,*args):

[0108] result=self.animal_classify.predict(args)

[0109] return result

[0110] def handwriting_service(self,*args):

[0111] result=self.handwriting.predict(args)

[0112] return result

[0113] The technical solution provided by the above embodiments can bring at least the following beneficial effects: The model inference service device loads model parameters, determines multiple model inference service identifiers and multiple model inference methods in the model parameters, and constructs a model inference service class in the storage system of the model inference service device based on the multiple model inference service identifiers and multiple model inference methods. When the model inference service device receives a model inference service request from the client, it only needs to construct a model inference service object with the same name as the client's model inference service request in the model inference service class determined by the object reflection mechanism, and call the corresponding model inference method to perform model inference calculation. This avoids the need to repeatedly load model parameters every time a model inference service request from the client is received, thereby shortening the client's waiting time for the model inference service result.

[0114] In one possible implementation, combining Figure 3 ,like Figure 4 As shown, S302 above constructs a model reasoning service class based on multiple model reasoning service identifiers and multiple model reasoning methods. This method specifically includes the following S401-S403, which will be described in detail below.

[0115] S401, The model reasoning service device constructs the second function.

[0116] The second function includes multiple variables.

[0117] In one possible implementation, the second function is a constructor (__init__) used to construct the first function mentioned above based on the model parameters, thereby determining the specific model inference service object and its corresponding model inference method.

[0118] For example, the number of variables in the second function can be determined based on the number of identifiers in the model parameters. If the model parameters contain three model inference service identifiers, then three first functions can be constructed based on the model inference service identifiers.

[0119] S402, The model reasoning service device assigns multiple model reasoning service identifiers to multiple variables of the second function to determine the first function.

[0120] In this context, one variable of the first function points to the model inference method corresponding to the model inference service identifier assigned to a variable.

[0121] For example, if the model inference service identifiers include animal_classify, handwriting, and face recognition, then substituting animal_classify, handwriting, and face recognition into the second function can determine the first function for animal_classify, handwriting, and face recognition.

[0122] For example, a model inference method for animal recognition is derived from a first function for animal recognition determined by the model parameters and the second function.

[0123] S403. The model reasoning service device constructs a model reasoning service class based on the first function and multiple model reasoning methods.

[0124] In one possible implementation, after loading the model parameters, the model inference service device includes multiple model inference service objects and their corresponding model inference methods in the model inference service class. When a model inference service request is received from the client, the target model inference service object and its corresponding model inference method can be directly invoked.

[0125] The technical solution provided by the above embodiments can bring at least the following beneficial effects: The model inference service device loads model parameters, determines multiple model inference service identifiers and multiple model inference methods in the model parameters, and constructs a model inference service class in the storage system of the model inference service device based on the multiple model inference service identifiers and multiple model inference methods. By loading model parameters when the model inference service device is initialized and started, when a model inference service request is received from a client, the target model inference service object and its corresponding model inference method can be directly invoked, shortening the time the client waits for the model inference service result.

[0126] In one possible implementation, the second function is a function written in the system language of the Web server, wherein the Web server is used to process model inference service requests; and the model inference service class is a service class written in the system language of the Web server.

[0127] For example, if the system language of the web server is Python, then the second function is also in Python.

[0128] In one possible implementation, combining Figure 4 ,like Figure 5 As shown, in step S303 above, the model inference service device obtains a model inference service request from the client. This method specifically includes steps S501-S502, which will be described in detail below.

[0129] S501, the model inference service device calls the HTTP service listener to periodically monitor the client's service requests.

[0130] In one possible implementation, after the model inference service device has loaded the model parameters once, an HTTP service listener can be invoked to monitor model inference service requests from clients. Whenever a model inference service request from a client is received, the relevant model inference method is invoked, and model inference calculations can be performed directly without reloading the model parameters.

[0131] S502. After listening to the client's model inference service request, the model inference service device parses the target model inference service identifier from the model inference service request.

[0132] For example, the model inference service device can determine that the model inference object of this model inference service request is animal recognition based on the fields in the client's model inference service request.

[0133] The technical solution provided by the above embodiments can bring at least the following beneficial effects: when the model inference service device receives a model inference service request from the client, it can directly call the target model inference service object and its corresponding model inference method, which shortens the time for the client to wait for the model inference service result.

[0134] In one possible implementation, combining Figure 5 ,like Figure 6 As shown in S305 above, the model reasoning service device calls the model reasoning method corresponding to the target model reasoning service identifier in the model reasoning service class to process the model reasoning service request. This method specifically includes the following S601-S603, which will be described in detail below.

[0135] S601, The model reasoning service device uses an object-oriented reflection mechanism to search for the target model reasoning service identifier from the first function.

[0136] In one possible implementation, the model inference service identifier can be the name of the model inference object.

[0137] For example, the statements that need to be executed by the method object that obtains the animal identification service using the object-oriented reflection mechanism are:

[0138] service_obj=getattr(model_services,'animal_classify_service')

[0139] S602, The model reasoning service device determines the target model reasoning method pointed to by the variable in the first function that is assigned the value of the target model reasoning service identifier.

[0140] For example, if the model inference service device determines that the assignment value in the first function is animal recognition and the model inference service identifier is animal recognition, it can determine that the object of this model inference is animal recognition and the target model inference method is the model inference method of animal recognition.

[0141] S603. Call the target model inference method to handle the model inference service request.

[0142] In one possible implementation, combining Figure 4 ,like Figure 7 As shown, after the above-mentioned S301, the model inference service device parses the model parameters, determines the multiple model inference service identifiers and multiple model inference methods in the model parameters, the method also includes S701-S702, which will be described in detail below.

[0143] S701, The model reasoning service device determines the system language of the web server.

[0144] For example, the model inference service device determines that the system language of the web server is Python.

[0145] S702. The model reasoning service device writes multiple model reasoning methods according to the system language and stores them in system files.

[0146] Understandably, the model inference service device writes multiple models and stores them in system files based on the system language. Since the model inference is written based on the system language, when the model inference service device receives a model inference service request from the client, it can directly call the target model inference method for processing.

[0147] The technical solution provided by the above embodiments can bring at least the following beneficial effects: The model inference service device loads model parameters, determines multiple model inference service identifiers and multiple model inference methods in the model parameters, and constructs a model inference service class in the storage system of the model inference service device based on the multiple model inference service identifiers and multiple model inference methods. By loading model parameters when the model inference service device is initialized and started, when a model inference service request is received from a client, the target model inference service object and its corresponding model inference method can be directly invoked, shortening the time the client waits for the model inference service result.

[0148] The above provides a detailed description of the model inference service method involved in the embodiments of this disclosure.

[0149] As can be seen, the above mainly describes the technical solutions provided by the embodiments of this disclosure from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0150] This disclosure embodiment can divide the model inference service device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this disclosure embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0151] like Figure 8 The diagram shown is a structural schematic of a model reasoning service device 800 provided in an embodiment of this disclosure.

[0152] The model inference service device 800 includes: a communication unit 801 and a processing unit 802; the processing unit 802 is used to parse model parameters, determine multiple model inference service identifiers and multiple model inference methods in the model parameters, wherein each model inference service identifier corresponds one-to-one with a model inference method; the processing unit 802 is used to construct a model inference service class based on the multiple model inference service identifiers and multiple model inference methods; the model inference service class includes a first function and multiple model inference methods, wherein multiple variables of the first function are multiple model inference service identifiers; the communication unit 801 is used to obtain a model inference service request from a client; the model inference service request includes a target model inference service identifier; the processing unit 802 is used to call the first function in the model inference service class and find the target model inference service identifier based on the first function; the processing unit 802 is used to call the model inference method in the model inference service class corresponding to the target model inference service identifier and process the model inference service request.

[0153] In one possible implementation, the processing unit 802 is specifically used for: constructing a second function, the second function including multiple variables; assigning multiple model inference service identifiers to the multiple variables of the second function to determine a first function; a variable of the first function pointing to the model inference method corresponding to the model inference service identifier assigned to the variable; and constructing a model inference service class based on the first function and the multiple model inference methods.

[0154] In one possible implementation, the second function is a function written in the system language of the web server; the web server is used to process model inference service requests; and the model inference service class is a service class written in the system language of the web server.

[0155] In one possible implementation, the communication unit 801 is specifically used to call the HTTP service listener to periodically monitor the client's service requests; the processing unit 802 is specifically used to parse the target model inference service identifier from the model inference service request after listening to the client's model inference service request.

[0156] In one possible implementation, the processing unit 802 is specifically used to: use an object-oriented reflection mechanism to find the target model inference service identifier in the first function; determine the target model inference method pointed to by the variable in the first function that is assigned the target model inference service identifier; and call the target model inference method to process the model inference service request.

[0157] In one possible implementation, after parsing the model parameters and determining multiple model inference service identifiers and multiple model inference methods in the model parameters, the processing unit 802 is further configured to: determine the system language of the Web server; and, based on the system language, write multiple model inference methods and store them in a system file.

[0158] This disclosure also provides a model inference service apparatus, which includes a processor and a memory; wherein the memory is used to store computer execution instructions, and when the model inference service is running, the processor executes the computer execution instructions stored in the memory so that the model inference service apparatus performs the model inference service method described in this disclosure.

[0159] Embodiments of this disclosure provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the model inference service method described in the above method embodiments.

[0160] Embodiments of this disclosure provide a chip including a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run computer programs or instructions to implement the model inference service method as described in the above method embodiments.

[0161] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), registers, hard disks, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing, or any other form of computer-readable storage medium in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0162] Since the apparatus, devices, computer-readable storage media, and computer program products in the embodiments of this disclosure can be applied to the above methods, the technical effects they can achieve can also be referred to the above method embodiments. The embodiments of this disclosure will not be repeated here.

[0163] The above descriptions are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions within the technical scope disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A model reasoning service method, characterized in that, include: When the Web server main program is initialized and started, the model parameter file is loaded, the model parameters are parsed, and multiple model inference service identifiers and multiple model inference methods in the model parameters are determined. The model inference service identifiers correspond one-to-one with the model inference methods. Construct a second function, which includes multiple variables; The multiple model inference service identifiers are assigned to multiple variables of the second function to determine the first function; one variable of the first function points to the model inference method corresponding to the model inference service identifier assigned to the first variable. Based on the first function and the plurality of model inference methods, the model inference service class is constructed; the model inference service class includes the first function and the plurality of model inference methods, wherein the plurality of variables of the first function are the identifiers of the plurality of model inference services; Obtain a model inference service request from the client; the model inference service request includes a target model inference service identifier; The first function in the model inference service class is called, and the object-oriented reflection mechanism is used to find the target model inference service identifier from the first function; Determine the target model inference method that the variable in the first function, which is assigned the value of the target model inference service identifier, points to. The target model inference method is invoked to process the model inference service request.

2. The method according to claim 1, characterized in that, The second function is a function written in the system language of the web server; the web server is used to process the model inference service request; The model reasoning service class is a service class written in the system language of the web server.

3. The method according to claim 2, characterized in that, The step of obtaining the model inference service request from the client includes: Invoke the Hypertext Transfer Protocol (HTTP) service listener to periodically monitor the client's service requests; After listening to the client's model inference service request, the target model inference service identifier is parsed from the model inference service request.

4. The method according to any one of claims 1-2, characterized in that, After parsing the model parameters and determining the multiple model inference service identifiers and multiple model inference methods within the model parameters, the process further includes: Determine the system language of the web server; Based on the system language, the multiple model inference methods are written and stored in system files.

5. A model reasoning service device, characterized in that, include: Communication unit and processing unit; The processing unit is used to load the model parameter file, parse the model parameters, and determine multiple model inference service identifiers and multiple model inference methods in the model parameters when the Web server main program is initialized and started. The model inference service identifiers correspond one-to-one with the model inference methods. The processing unit is configured to construct a second function, the second function including multiple variables; assign the multiple model inference service identifiers to the multiple variables of the second function to determine a first function; one variable of the first function points to the model inference method corresponding to the model inference service identifier assigned to the first variable; construct the model inference service class based on the first function and the multiple model inference methods; the model inference service class includes the first function and the multiple model inference methods, and the multiple variables of the first function are the multiple model inference service identifiers respectively; The communication unit is used to obtain a model inference service request from the client; the model inference service request includes a target model inference service identifier; The processing unit is configured to call the first function in the model inference service class, use object-oriented reflection mechanism to find the target model inference service identifier in the first function; determine the target model inference method pointed to by the variable in the first function that is assigned the target model inference service identifier; and call the target model inference method to process the model inference service request.

6. The apparatus according to claim 5, characterized in that, The second function is a function written in the system language of the web server; the web server is used to process the model inference service request; The model reasoning service class is a service class written in the system language of the web server.

7. The apparatus according to claim 5, characterized in that, The communication unit is specifically used to call the Hypertext Transfer Protocol (HTTP) service listening program to periodically monitor the service requests of the client. The processing unit is specifically used to parse the target model inference service identifier from the model inference service request after listening to the model inference service request from the client.

8. The apparatus according to any one of claims 5-6, characterized in that, After parsing the model parameters and determining multiple model inference service identifiers and multiple model inference methods within the model parameters, the processing unit is further configured to: Determine the system language of the web server; Based on the system language, the multiple model inference methods are written and stored in system files.

9. A model reasoning service device, characterized in that, include: A processor and a memory; wherein the memory is used to store computer execution instructions, and when the model inference service device is running, the processor executes the computer execution instructions stored in the memory to cause the model inference service device to perform the model inference service method according to any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by the processor of the model inference service device, cause the model inference service device to perform the model inference service method according to any one of claims 1-4.

Images