Service delivery methods and apparatus, electronic devices, storage media

Abstract

This application relates to a service provision method and apparatus, electronic device, and storage medium. The method includes: after initiating the creation of an executor using a non-blocking method, starting a target server, wherein the executor is used to obtain a prototype class from a target SDK and instantiate the prototype class to obtain a reasoning instance with a completed reasoning model initialization; the target SDK is an SDK used to build a target service; if at least one target executor that has become a consumer exists, the target server begins receiving service requests, wherein the service request is used to request the invocation of a target service, and the executor containing the reasoning instance becomes a consumer when it is capable of handling service requests; when the target server receives a target service request, it processes the target service request through the reasoning instance in the target executor. This application solves the technical problem in related technologies where the time spent on model initialization leads to excessively high service processing latency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a service delivery method and apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, more and more applications require AI algorithms to empower them. The final implementation of algorithms is usually encapsulated in services and then invoked through the service's external interface. On the server side, invoking the AI ​​model can be broken down into two stages: the initialization of the inference model instance and the invocation of the model instance's inference method. The time spent on model initialization can lead to excessively high service processing latency.

[0003] There is currently no effective solution to the problem of excessive service processing latency caused by the time spent on model initialization in the aforementioned technologies. Summary of the Invention

[0004] This application provides a service provision method and apparatus, electronic device, and storage medium to at least solve the technical problem in the related art where the service processing latency is too high due to the time spent on model initialization.

[0005] According to one aspect of the embodiments of this application, a service provision method is provided, comprising: after starting the creation of an executor through a non-blocking method, starting a target server, wherein the executor is used to obtain a prototype class in a target SDK and instantiate the prototype class to obtain a reasoning instance that has completed the initialization of the reasoning model, and the target SDK is an SDK used to build a target service; in the presence of at least one target executor that has been converted into a consumer, the target server starts to receive service requests, wherein the service requests are used to request the invocation of a target service, and the executor containing the reasoning instance is converted into a consumer in a state where it can handle service requests; and when the target server receives a target service request, processing the target service request through the reasoning instance in the target executor.

[0006] According to another aspect of the embodiments of this application, a service providing apparatus is also provided, comprising: a startup module, configured to start a target server after an executor is created using a non-blocking method, wherein the executor is configured to obtain a prototype class in a target SDK and instantiate the prototype class to obtain a reasoning instance that has completed the initialization of the reasoning model, and the target SDK is an SDK for building a target service; a receiving module, configured to receive service requests when at least one target executor is converted into a consumer, wherein the service request is configured to request the invocation of a target service, and the executor containing the reasoning instance is converted into a consumer when it is in a state capable of handling service requests; and a processing module, configured to process the target service request through the reasoning instance in the target executor when the target server receives the target service request.

[0007] According to another aspect of the embodiments of this application, a storage medium is also provided, the storage medium including a stored program that executes the above-described method when the program is run.

[0008] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store a computer program; and the processor is used to execute the computer program to implement the method described in any of the foregoing embodiments.

[0009] According to one aspect of this application, a computer-readable storage medium is provided, the storage medium including a stored program, wherein the program, when executed, performs the method as described in any of the preceding claims.

[0010] In this embodiment, the method adopted is as follows: "After starting the creation of the executor using a non-blocking method, the target server is started, wherein the executor is used to obtain the prototype class in the target SDK and instantiate the prototype class to obtain a reasoning instance that has completed the initialization of the reasoning model; the target SDK is an SDK used to build the target service; if there is at least one target executor that has been converted into a consumer, the target server starts to receive service requests, wherein the service request is used to request to call the target service, and the executor containing the reasoning instance is converted into a consumer in a state where it can handle service requests; when the target server receives the target service request, it processes the target service request through the reasoning instance in the target executor." This method starts the creation process using a non-blocking method. The executor initializes the inference model while continuing the server startup process, thus hiding the delay in the inference model initialization within the executor. Once at least one executor has completed the inference model initialization and become a consumer, the server begins receiving service requests. Since an executor has already completed the inference model initialization, the service processing time does not include the inference model initialization time. Furthermore, any executor that has completed the inference model initialization does not need to perform inference model initialization again in subsequent service processing, saving inference model initialization time for each service processing, improving service processing efficiency, and thus solving the technical problem of excessive service processing latency caused by the time consumption of model initialization in related technologies. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0012] Figure 1 This is a schematic diagram of the hardware environment of the service provision method according to an embodiment of this application;

[0013] Figure 2 This is a flowchart of an optional service provision method according to an embodiment of this application;

[0014] Figure 3 This is an optional task diagram according to an embodiment of this application;

[0015] Figure 4 This is a schematic diagram of the monitoring process of an optional actuator monitor according to an embodiment of this application;

[0016] Figure 5 This is a schematic diagram illustrating the overall concept of an optional service provision scheme according to an embodiment of this application;

[0017] Figure 6This is a schematic diagram of an optional service providing apparatus according to an embodiment of this application; and,

[0018] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0022] SDK: Software Development Kit, is a collection of development tools used by software engineers when building application software for specific software packages, software frameworks, hardware platforms, operating systems, etc.

[0023] A class is used to describe a collection of objects that share the same attributes and methods. It defines the attributes and methods common to each object in the collection. When writing a class, we give it some common behaviors. By creating objects based on the class, we can ensure that each object possesses these common behaviors, and then assign unique characteristics to each object as needed. An object is an instance of a class; creating an object based on a class is called instantiation.

[0024] Class inheritance: Inheritance is a way to create a new class. The new class can inherit from one or more base classes. In this case, the new class is called a subclass or derived class, and the base class is called a parent class or superclass.

[0025] Abstract classes are often used to represent abstract concepts derived from the analysis and design of a problem domain. They are abstractions of a series of seemingly different but essentially the same concrete concepts. Classes modified with "abstract" in programming statements are typically abstract classes. In C++, a class containing pure virtual functions is called an abstract class, and it cannot be instantiated. In Java, a class containing abstract methods is called an abstract class, and similarly, it cannot be instantiated. Python also has the concept of abstract classes, defining a series of methods that must be implemented by derived classes. Abstract classes are incomplete; they can only be used as base classes. In object-oriented programming, abstract classes are mainly used for type hiding and as global variables.

[0026] An interface is an abstract type, a collection of abstract methods. Interfaces are typically declared with the keyword "interface". In some object-oriented languages, a class can implement multiple interfaces, but can only inherit from one abstract (or any other type) class. A class implementing an interface must implement all the methods described in the interface; otherwise, it must be declared as an abstract class. In Python, interfaces are implemented using abstract classes and abstract methods.

[0027] DSL: Domain Specific Language, is a language designed for a specific domain context. Here, "domain" refers to a business context (e.g., banking, insurance) or an application context (e.g., web applications, databases, AI algorithms). In contrast, there's the concept of GPL (general-purpose language), which can be widely used for various business or application problems. Common GPLs include Java and Python.

[0028] Concurrency: Executing multiple tasks alternately over a period of time. For example, when multiple threads are operating, the CPU time is divided into several time slots, and these time slots are allocated to each thread for execution. While the code of one thread is running during a time slot, the other threads are suspended.

[0029] Parallelism: The true simultaneous execution of multiple tasks over a period of time. For multi-core CPUs handling multiple tasks, the operating system assigns a task to each core of the CPU, and multiple cores truly execute multiple tasks simultaneously.

[0030] Process: It is the smallest unit of resource allocation. It is the basic unit for the operating system to allocate and schedule resources. Once a process is created, a certain amount of resources will be allocated to it.

[0031] Thread: The smallest unit of program execution. In fact, the process is only responsible for allocating resources, while the threads are the ones that use these resources to execute the program. In other words, the process is a container for threads. There must be at least one thread in a process to be responsible for executing the program. At the same time, the thread itself does not own system resources, it only needs a little bit of resources that are essential for running, but it can share all the resources owned by the process with other threads belonging to the same process.

[0032] Blocking and non-blocking refer to the states of the caller (program) while waiting for a result (or input). In blocking mode, the current thread is suspended until the result is returned, and returns after receiving the result. In non-blocking mode, if the result cannot be obtained immediately, the caller will not block the current thread. Therefore, in the non-blocking case, the caller needs to periodically poll to check the processing status.

[0033] According to one aspect of the embodiments of this application, a method embodiment for providing a service is provided.

[0034] Optionally, in this embodiment, the above-described service provision method can be applied to, for example, Figure 1 The hardware environment shown consists of terminal 101 and server 103. Figure 1 As shown, server 103 is connected to terminal 101 via a network and can be used to provide services to the terminal or clients installed on the terminal. Database 105 can be set up on the server or independently of the server to provide data storage services to server 103. The aforementioned network includes, but is not limited to, wide area networks (WANs), metropolitan area networks (MANs), or local area networks (LANs). Terminal 101 is not limited to PCs, mobile phones, tablets, etc. The service provision method of this application embodiment can be executed by server 103, by terminal 101, or by both server 103 and terminal 101. The service provision method of this application embodiment can also be executed by a client installed on terminal 101.

[0035] Figure 2 This is a flowchart of an optional service provision method according to an embodiment of this application, such as... Figure 2 As shown, the method may include the following steps:

[0036] Step S202: After the executor is created using a non-blocking method, the target server is started. The executor is used to obtain the prototype class in the target SDK and instantiate the prototype class to obtain a reasoning instance that has completed the initialization of the reasoning model. The target SDK is the SDK used to build the target service.

[0037] In step S204, if there is at least one target executor that has been converted into a consumer, the target server begins to receive service requests, wherein the service request is used to request the invocation of the target service, and the executor containing the inference instance is converted into a consumer in a state where it can handle the service request.

[0038] Step S206: When the target server receives a target service request, the target service request is processed through the inference instance in the target executor.

[0039] Through steps S202 to S206 above, an executor is created using a non-blocking method. The executor completes the initialization of the inference model, while the server startup process continues. This hides the delay in the initialization of the inference model in the executor. When at least one executor completes the initialization of the inference model and becomes a consumer, the server begins to receive service requests. Since an executor has already completed the initialization of the inference model, the service processing time does not include the inference model initialization time. This solves the technical problem of excessive service processing delay caused by the time consumption of model initialization in related technologies.

[0040] This application can be applied to the actual production process of intelligent content production, review, and distribution, including but not limited to intelligent advertising, and can be used as a way to improve the external performance of services and reduce the time spent on model initialization in the monetization of artificial intelligence algorithms.

[0041] In the technical solution provided in step S202, after the executor is created using a non-blocking method, the target server is started. The executor is used to obtain the prototype class in the target SDK and instantiate the prototype class to obtain a reasoning instance that has completed the initialization of the reasoning model. The target SDK is an SDK used to build the target service.

[0042] Each executor is a thread or process, typically appearing concurrently or in parallel. Executors can be created by an executor manager, which is an object of an executor management class. The executor management class can define various methods (such as executor creation methods, executor refresh methods, and executor status acquisition and return methods) and properties (such as executor container properties).

[0043] Because the executor creation process uses a non-blocking method, the process of starting the target server and the process of creating the executor begin almost simultaneously. After the executor creation begins, the processes on the target server side and the executor side occur in parallel / concurrently. On the target server side, the target server is started and subsequent operations are performed (e.g., the target server's listening port starts working and begins receiving health check requests, etc.); on the executor side, the executor obtains the prototype class from the target SDK, instantiates the prototype class, and after obtaining the inference instance with the inference model initialized, it is transformed into a consumer capable of handling service requests.

[0044] The class in the target SDK that implements the functionality required by the target service is the prototype class. There can be one or more prototype classes in the target SDK. The prototype class can be obtained by inheriting from the target service class (that is, the prototype class is a subclass of the target service class as the base class).

[0045] The target service class can define interfaces and methods. It can define abstract interfaces that must be implemented by subclasses, or it can define generic methods or interfaces that can be inherited or overridden.

[0046] The target service class can define one or more target interfaces that must be implemented by subclasses. For example, target interfaces can be `load_model` (an interface for importing the inference model), `init_model` (an interface for initializing the inference model), `infer` (an inference interface), and `test` (a unit test interface). The target service class can also define interfaces that are not required to be implemented, such as `unload_model` (an interface for releasing resources occupied by the inference model), `warm_up` (an inference model warm-up interface), and `accelerate` (an interface for accelerating the inference model).

[0047] Instantiating the prototype class yields a reasoning instance with the reasoning model already initialized; that is, the reasoning model initialization method is implemented during prototype class instantiation. The reasoning model initialization method can be constructed using interfaces such as load_model, init_model, and warm_up.

[0048] For example, on the target SDK implementation side, the inference model initialization method can be constructed by overriding or inheriting the `load_model` and `init_model` interfaces defined in the target service class. Then, the `test` interface in the target service class can be used to verify whether the implementation of the aforementioned two interfaces meets the requirements. Subsequent performance monitoring can also determine whether to call or override the `warm_up` interface in the target service class. If the target service takes a long time to process on its first attempt, it may be because the inference model of the target service needs warm-up before it can be fully initialized. In this case, the inference model initialization method should include implementing or calling the `warm_up` interface.

[0049] In each executor, the inference model is initialized only once. An executor with an inference instance can be used to process service requests, reducing the time spent on inference model initialization. The number of inference model initializations is at the executor level, achieving decoupling from service request processing (inference computation). The initialization of the inference model is not coupled with the processing status of service requests. Even if the processing of a service request fails, it will not cause the inference model to be reinitialized.

[0050] In the technical solution provided in step S204, when there is at least one target executor that has been transformed into a consumer, the target server begins to receive service requests, wherein the service request is used to request the invocation of the target service, and the executor containing the inference instance is transformed into a consumer in a state where it can handle the service request.

[0051] Once created, the executor retrieves the prototype class from the target SDK and instantiates it. The executor with the inference instance can then begin to listen in a blocking manner for new tasks (i.e., service requests to be processed) to arrive in the task container, at which point it has successfully transformed into a consumer.

[0052] If there is at least one target executor that transforms into a consumer, it indicates that an executor that has completed the inference model initialization and can handle service requests has appeared and can start providing services. The target server then begins to receive service requests.

[0053] If at least one target executor that has been transformed into a consumer exists when the process on the target server side ends, the target server can start receiving service requests, completely concealing the time of inference model initialization.

[0054] If the process on the target server side ends before any executor has become a consumer, the target server will return a message indicating that the consumer is not yet ready, until at least one target executor becomes a consumer before it starts accepting service requests. In this case, although the initialization time of the inference model is not completely hidden, it does effectively hide some latency. Moreover, the time consumed is the waiting time for initiating the service request, not the processing time of the service request, and will not affect the performance of the service providing the target service to the outside world.

[0055] In the technical solution provided in step S206, when the target server receives a target service request, the target service request is processed through the inference instance in the target executor.

[0056] In the presence of multiple target executors, the target service request can be processed through the inference instance in any idle target executor.

[0057] After the target server receives the target service request, the time spent by the target executor in processing the target service request includes inference time, but does not include the time for inference model initialization, thus improving resource utilization.

[0058] As an optional embodiment, the production-consumption pattern can be implemented by designing task objects, a task manager, and a task submitter. Step S206, when the target server receives a target service request, processes the target service request through the inference instance in the target executor, and further includes the following steps:

[0059] Step S61: When the target server receives the target service request, it transforms the target service request into a target task object through the task submitter and submits the target task object to the task container of the task manager.

[0060] In step S62, when the target executor detects the presence of a target task object in the task container, the target executor obtains the target task object and processes it through the inference instance in the target executor.

[0061] When a target server receives a target service request, it can use the task submitter to transform the request into a target task object according to the target task class and place the target task object in a task container. When any target executor detects the presence of this target task object in the task container, it can retrieve the target task object and process it through inference instances to obtain the return result corresponding to the target service request.

[0062] The target task class can define result container properties and task methods. By transforming the target service request into a target task object according to the target task class, we can obtain a target task object containing a target result container. Since the target task class defines task methods, the target task object also contains task methods.

[0063] A task object is an instance of a data type class. The target task class can be a base class, and the concrete implementation class is used to instantiate the task object.

[0064] A task container is a container used in the Task Manager to store task objects. Before the target server begins receiving service requests, a task container needs to be created through the Task Manager. The Task Manager is a type of task management class object that defines a task container property for storing task objects and a task submission method for submitting task objects to the task container.

[0065] The task submitter is a thread used to call the task manager's task submission method to submit task objects to the task container.

[0066] Optionally, in this embodiment, before starting the target server, the method further includes the following steps:

[0067] Step S11: After the service framework starts executing, a task container is created through the task manager. The service framework is used to access the target SDK to obtain the target service. The task manager is a task management class object. The task management class defines a task container property for storing task objects and a task submission method for submitting task objects to the task container.

[0068] The task management class can define a task container property for storing task objects and task management methods. Task management methods include, but are not limited to: a task submission method for submitting task objects to the task container, and a method for getting and returning the task execution status.

[0069] The execution order specified in step S11 is only an optional implementation method. The step of creating the task container by the task manager can be completed before the target server starts receiving service requests.

[0070] Optionally, in this embodiment, after starting the creation of the executor using a non-blocking method, the method further includes the following steps:

[0071] Step S211: Create an executor through the executor manager and place it in the executor container to obtain a candidate executor;

[0072] In step S212, if the candidate executor obtains the prototype class in the target SDK, instantiates the prototype class into a reasoning instance that has completed the initialization of the reasoning model, and converts it into a consumer that listens to the task objects in the task container, then the candidate executor is determined as the target executor. Here, the prototype class in the target SDK is a subclass of the target service class as the base class. The target service class defines at least one target interface that must be implemented by the subclass. The at least one target interface includes an interface for initializing the reasoning model. Each target executor in the executor container is a thread or process for processing task objects.

[0073] After the executor is created, it is necessary to complete steps such as obtaining the prototype class in the target SDK, instantiating the prototype class into a reasoning instance that has completed the initialization of the reasoning model, and then converting it into a consumer that listens to the task objects in the task container to start the executor.

[0074] The above candidate executors are executors to be started in the executor container, and the above target executors are executors that have already been started in the execution container.

[0075] Optionally, in this embodiment, after starting the creation of the executor using a non-blocking method, the method further includes the following steps:

[0076] Step S221: If the executor monitor detects that the health status of the first executor in the executor container is abnormal, the refresh method in the executor manager is called through the executor monitor to close the first executor, create a second executor and place it in the executor container to replace the first executor. The executor monitor is used to monitor the health status of each executor in the executor container.

[0077] In step S221, after obtaining the prototype class in the target SDK and instantiating the prototype class into a reasoning instance that has completed the initialization of the reasoning model, the second executor is converted into a consumer that listens to the task objects in the task container.

[0078] An executor monitor is a thread or process used to monitor the running status of an executor. The running status of an executor includes, but is not limited to, the following: (1) Timeout state, where the executor's runtime exceeds the target duration (the predetermined time for task execution); (2) Abnormal state, where the executor is in a dead, zombie, or other abnormal state (e.g., due to unreasonable request parameters or the request being a boundary case not covered by the algorithm, the executor cannot continue the inference process and enters a dead, zombie, or stopped abnormal state); (3) Exhaustion state, where the executor consumes excessive resources (e.g., reaching the boundary value of memory or video memory capacity, continuing execution carries the risk of overflow). Abnormal health states can include the timeout state, abnormal state, and exhaustion state mentioned above.

[0079] The executor monitor can monitor each executor in the executor container, including executors that are currently processing tasks and those that are not.

[0080] If the first executor is one that is currently processing a task, the executor monitor can call the result write-back method in the first task object to write the processing exception result back to the result container of the first task object, so that the first task object can retrieve the result and return it. The first task object is the task object that the first executor is processing when the executor monitor detects an abnormal health status of the first executor.

[0081] Optionally, in this embodiment, after starting the target server, the method further includes the following steps:

[0082] Step S231: Listen for health check requests through the target server's listening port;

[0083] In step S232, if no executor that has been converted into a consumer appears, when the target server receives a health check request, it responds to the health check request by returning target shutdown information, wherein the target shutdown information is used to indicate that the target server has not started receiving service requests.

[0084] Once the target server is started, its listening port can function normally, receiving health check requests from the deployment platform or task scheduler.

[0085] When the target server receives a health check request, it can return the check result based on the current executor status. If no executor has yet become a consumer, it returns a target shutdown message.

[0086] Optionally, in this embodiment, step S204, where at least one target executor is converted into a consumer, involves the target server starting to receive service requests, including the following steps:

[0087] In step S41, if there is at least one target executor that has been converted into a consumer, when the target server receives a health check request, it returns target startup information in response to the health check request, wherein the target startup information is used to indicate that the target server can start receiving service requests.

[0088] If an executor has already been converted into a consumer when the target server receives the health check request, then the target startup information is returned.

[0089] As an optional embodiment, the technical solution of this application is illustrated below with reference to specific embodiments:

[0090] As more and more applications require AI to power them, research investment in AI algorithms is increasing daily. The final form of algorithm deployment and monetization is usually encapsulated in services and then invoked through the service's external interface. On the server side, invoking an AI model can be broken down into two stages: the initialization of the inference model instance and the invocation of the model instance's inference method. For service providers, AI-powered applications typically require significant resources (CPU, GPU, memory, video memory, etc.), and resources like GPUs are quite expensive. Therefore, to increase the proportion of computing resources used for inference computation, the time consumed by non-computational model initialization should be minimized. For the caller, the model instance initialization stage is imperceptible, and its time is factored into the service's latency performance. Therefore, to improve the service's external performance, initialization time should also be minimized.

[0091] The following issues affecting service performance exist in related technologies: ① Non-standard model initialization interfaces. The initialization interfaces delivered by algorithm SDKs vary in form and quality, failing to guarantee an independent and complete interface for model initialization; the "thoroughness" of model initialization cannot be guaranteed. ② The time consumption of model initialization and inference often differs significantly. Initialization often takes seconds or even minutes, while the inference time in many applications is an order of magnitude lower than the initialization time. If model initialization accompanies every inference iteration, the resource consumption mainly comes from initialization rather than inference, which also affects the improvement of external service performance. ③ There is a conflict between improving concurrent computing capabilities and reducing initialization time. Many learning frameworks, such as Keras, require that the initialization of inference model instances and the inference method calls be implemented in the same thread. However, to improve the utilization of computing resources, a service usually needs to have concurrent / parallel computing capabilities. This means that for the above learning frameworks, model initialization must be performed in different threads, and without special design, the more threads there are, the more time is spent on initialization.

[0092] To address the aforementioned issues, this embodiment provides the following solutions: ① For the model initialization interface itself, inheritance and implementation of the parent class method in the DSL module are used, ensuring that the model initialization interface of the successfully delivered algorithm SDK conforms to the standard. ② By designing task objects, task managers, task submitters, executors, executor managers, and executor monitors, and implementing a producer-consumer model, task instantiation and inference are decoupled. ③ The execution method of the executor (thread / process) subclass is overridden, instantiating the model first and then transforming itself into a consumer. ④ The executor is created and started using a non-blocking method, while the main process continues to start the server and complete the health check of the deployment platform, thereby hiding the delay in model initialization in the executor. ⑤ The server only receives inference tasks when at least one consumer is ready. Thus, the unhidden model initialization time is part of the service deployment startup time and does not affect task processing performance. ⑥ The executor transformed into a consumer only performs inference operations when consuming tasks, without any further initialization. ⑦ In concurrent / parallel scenarios, the executor that is first converted into a consumer executes the task first, thus masking the initialization time of model instances in executors that have not yet completed the conversion.

[0093] Overview of the overall approach of this plan:

[0094] (I) Introduction to Basic Concepts

[0095] 1. Task: Use Figure 3The diagram illustrates this. It represents an instance of a data type class. The actual class representing the task (i.e., the target task class in this application) is a base class with two concrete implementation classes used to instantiate synchronous and asynchronous task objects, respectively. Its key attribute for decoupling from execution is the result container attribute. This attribute acts as a container object when the executor operates in a multi-threaded manner; and as a container proxy object when the executor operates in a multi-process manner.

[0096] 2. Task Manager: A constructed task management class object. It has a container property to store task objects and methods for submitting these task objects to the container; methods for getting and returning task execution status, etc.

[0097] 3. Task Submitter: Used to call the Task Manager's task submission method to submit tasks to threads in the task container.

[0098] 4. Executor: A thread / process used for result calculation (mainly model inference). It usually appears in a concurrent or parallel form. When the executor is successfully created and is in a state where it can handle tasks, it enters the role of consumer.

[0099] 5. Executor Manager: A class object that manages the executors. It has methods for generating, refreshing, getting, and returning executors, as well as properties such as an executor container.

[0100] 6. Executor Monitor: A thread / process that monitors the running status of the executor. Based on different state definitions, such as timeout (executor runtime exceeds the predetermined task execution time), unhealthy state (executor is in a dead-busy or zombie state), and resource-consumption state (excessive resource usage), it takes corresponding measures, such as writing back task results and restarting the executor. An overall diagram of the executor manager with the executor monitor is shown below. Figure 4 As shown.

[0101] (II) Overall Approach and Process

[0102] The overall approach of this solution is to reduce initialization time in two ways: firstly, by minimizing the number of model instance initializations; and secondly, by using concurrent / parallel processes to mask the model initialization time as much as possible.

[0103] I. Prerequisites:

[0104] 1. The model instance initialization interface itself conforms to certain standards.

[0105] 2. The Task Submitter and Executor Manager, using the Task Manager as a medium, implement a producer-consumer model, decoupling task submission and execution.

[0106] II. Approach: The overall approach diagram of this plan is shown below. Figure 5 As shown.

[0107] 1. Reduce model initialization times. ① Single executor: Perform model instance initialization once before transforming into a consumer role. After transforming into a consumer, the consumption task only performs inference and no further initialization is needed. ② Multiple executors in concurrency / parallelism: Since many model implementation frameworks require initialization and inference to be implemented in the same thread, the above single executor mechanism can be implemented in each executor using dynamic module loading, which can also minimize the number of initialization times.

[0108] 2. Model Initialization Delay Masking: ① The creation of executors is implemented using a non-blocking method, utilizing the continued execution of the main process to mask the delay before the executor transforms into a consumer. ② Executors that first transform into consumer roles process tasks first: From the perspective of the service outside, this masks the delay from service startup to normal service provision; it also masks the consumer role transformation time of concurrent / parallel executors, including normal startup and abnormal restarts.

[0109] 3. Model initialization time category conversion: Whether the model initialization is completed determines whether there are ready consumers, which in turn determines whether the service can receive inference tasks. Therefore, the initialization time affects the service deployment time, rather than the task processing performance.

[0110] The specific implementation steps of this solution are as follows:

[0111] (1) On the algorithm SDK implementation side, inherit the SdkBase class (equivalent to the target service class in this application) from the DSL module (the DSL module may include one or more classes), override or inherit its load_model and init_model interfaces to complete the construction of the model instance initialization method, and then verify whether the implementation of the aforementioned two interfaces meets the requirements through the test interface of the SdkBase class. Subsequent performance monitoring determines whether to call or override the warm_up interface in the parent class. That is, if the time taken for the first task is significantly greater than the time taken for subsequent tasks, it may be that the model needs warm_up to be fully initialized. In this case, the model initialization method should include implementing / calling the warm_up interface.

[0112] (2) After the service framework is executed, before the server starts, the task manager creates a task container.

[0113] (3) The executor manager then calls a non-blocking method to create a predetermined number of executors and place them in the executor container.

[0114] (4) Since the above is a non-blocking method, the main process can enter the server startup stage almost simultaneously.

[0115] (5) On the executor side, after the corresponding thread / process starts, it first dynamically imports the modules containing all inference prototype classes, then obtains all inference prototype classes and instantiates them, thus ensuring that each module is imported only once in each process and each inference prototype class is instantiated only once; the executor with inference instances will listen in a blocking manner to see if any new tasks arrive in the task manager, and at this time it will successfully become a consumer.

[0116] (6) The executor startup process after (3) occurs in parallel / concurrently with (4) above. After the server starts, the server's listening port can work normally and can receive health check requests from the deployment platform or task scheduler. Such requests do not include actual inference tasks in the design, so they can be executed normally without waiting for the process in (5).

[0117] (7) If the initialization time of the inference instance has been completely hidden by the end of the process in (6), that is, at least one executor has become a consumer, the server can receive normal task requests and perform inference processing. If no executor has become a consumer by the end of the process in (6), the server will return the information that the consumer is not ready until at least one consumer is ready, and then start receiving real inference tasks. In this case, although the initialization time of the inference instance is not completely hidden, it still has the effect of hiding some delays, and the impact of the time consumption is the waiting for the request to be initiated rather than the performance impact of the request processing.

[0118] (8) As the first executor becomes a consumer and receives a task, parallel / concurrent executors will successively complete the inference instance initialization, become consumers and receive tasks, thus realizing the delay masking between executors.

[0119] (9) If subsequent tasks are executed normally, the consumer's effective computing time is all inference time, thus improving resource utilization.

[0120] (10) When the actuator monitor detects that one or more actuators have timed out, are in an abnormal state, or are over-consuming resources, it will terminate them one by one and start a new consumer in a process similar to (3) and (5) above, thus still ensuring the effect of delay concealment of the normal process.

[0121] This solution can achieve the following beneficial effects:

[0122] (I) The model initialization method is standardized through DSL inheritance and mandatory implementation: 1. This ensures that the model initialization method is an independent and complete interface. 2. Thorough initialization is essential to ensure that it does not affect the performance of the inference task, which is a prerequisite for this solution to achieve its intended purpose.

[0123] (ii) Model initialization times are at the executor granularity, decoupled from task inference: 1. By dynamically loading the module containing the prototype class of the model, it is ensured that even if the model of each executor comes from a thread-critical framework, the model instance can be initialized only once, reducing the time consumption caused by initialization. 2. Model initialization is not coupled with the execution status of the task, that is, if the execution of a certain task fails, it will not cause the model to be re-initialized.

[0124] (iii) By designing concurrent / parallel processes, the model initialization is delayed and hidden: other processes and the model initialization process are concurrent / parallel, achieving the effect of mutual delay and hiding, thereby improving the performance of service startup and service execution.

[0125] (iv) The role of the executor is determined by the model initialization progress. Only when the model initialization is complete can the executor become a consumer, and only when a consumer is ready will the service receive external requests. Therefore, this scheme converts the unhidden model initialization time into service deployment and startup time, which will not affect the performance after receiving actual inference tasks.

[0126] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0128] According to another aspect of the embodiments of this application, a service provisioning apparatus for implementing the above-described service provisioning method is also provided. Figure 6 This is a schematic diagram of an optional service providing apparatus according to an embodiment of this application, such as... Figure 6 As shown, the device may include:

[0129] Startup module 22 is used to start the target server after the executor is created in a non-blocking method. The executor is used to obtain the prototype class in the target SDK and instantiate the prototype class to obtain the inference instance that has completed the inference model initialization. The target SDK is the SDK used to build the target service.

[0130] In the case of at least one target executor that has been converted into a consumer, the target server begins to receive service requests, wherein the service request is used to request the invocation of the target service, and the executor containing the inference instance is converted into a consumer in the state of being able to handle the service request.

[0131] Processing module 26 processes the target service request through the inference instance in the target executor when the target server receives the target service request.

[0132] It should be noted that the startup module 22 in this embodiment can be used to execute step S202 in this application embodiment, the receiving module 24 in this embodiment can be used to execute step S204 in this application embodiment, and the processing module 26 in this embodiment can be used to execute step S206 in this application embodiment.

[0133] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented either through software or through hardware.

[0134] Through the above modules, executors are created using a non-blocking method. The executors complete the initialization of the inference model while the server startup process continues. This hides the delay in the initialization of the inference model within the executors. Once at least one executor has completed the initialization of the inference model and become a consumer, the server begins to receive service requests. Since an executor has already completed the initialization of the inference model, the service processing time does not include the inference model initialization time. Furthermore, any executor that has completed the initialization of the inference model does not need to perform inference model initialization again in subsequent service processing, saving inference model initialization time for each service processing, improving service processing efficiency, and thus solving the technical problem of excessive service processing latency caused by the time consumption of model initialization in related technologies.

[0135] As an optional embodiment, the processing module 26 includes: a submission unit, configured to, when the target server receives a target service request, convert the target service request into a target task object through a task submitter and submit the target task object to the task container of the task manager; and a processing unit, configured to, when the target executor detects the appearance of a target task object in the task container, the target executor obtains the target task object and processes the target task object through the inference instance in the target executor.

[0136] Optionally, the device further includes a task container creation unit, used to create a task container through the task manager before starting the target server and after the service framework starts executing. The service framework is used to access the target SDK to obtain the target service. The task manager is a task management class object. The task management class defines a task container property for storing task objects and a task submission method for submitting task objects to the task container.

[0137] Optionally, the device further includes an executor creation unit, used to create executors through an executor manager and place them in an executor container after starting executor creation via a non-blocking method to obtain candidate executors; the device further includes an executor determination unit, used to determine the candidate executor as the target executor when the candidate executor obtains the prototype class in the target SDK, instantiates the prototype class into a reasoning instance that has completed the reasoning model initialization, and converts it into a consumer that listens to the task objects in the task container, wherein the prototype class in the target SDK is a subclass of the target service class as the base class, the target service class defines at least one target interface that must be implemented by the subclass, the at least one target interface includes an interface for reasoning model initialization, and each target executor in the executor container is a thread or process for processing task objects.

[0138] Optionally, the device further includes an executor refresh unit, configured to: after starting executor creation via a non-blocking method, if the executor monitor detects an abnormal health status of the first executor in the executor container, call the refresh method in the executor manager through the executor monitor to close the first executor, create a second executor and place it in the executor container to replace the first executor, wherein the executor monitor is used to monitor the health status of each executor in the executor container; after the second executor obtains the prototype class in the target SDK and instantiates the prototype class into a reasoning instance that has completed the initialization of the reasoning model, it is converted into a consumer that listens to the task objects in the task container.

[0139] As an optional embodiment, the receiving module 24 is further configured to: after starting the target server, listen for health check requests through the target server's listening port; if no executor that has become a consumer appears, when the target server receives a health check request, respond to the health check request by returning target shutdown information, wherein the target shutdown information is used to indicate that the target server has not started receiving service requests.

[0140] Optionally, the receiving module 24 is further configured to: when there is at least one target actuator that has been converted into a consumer, the target server, upon receiving a health check request, responds to the health check request by returning target startup information, wherein the target startup information is used to indicate that the target server can begin receiving service requests.

[0141] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented through software or hardware, and the hardware environment includes the network environment.

[0142] According to another aspect of the embodiments of this application, an electronic device is also provided for implementing the above-described service provision method, including: Figure 7 As shown, the electronic device may include: a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504, wherein the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.

[0143] Memory 1503 is used to store computer programs;

[0144] When the processor 1501 executes the program stored in the memory 1503, it implements the steps of the above method embodiment.

[0145] The bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0146] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0147] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0148] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0149] Those skilled in the art will understand that Figure 7 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 7 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 7 The different configurations shown.

[0150] This application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the method steps of the above method embodiments when it runs.

[0151] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0152] This application provides a service provisioning scheme. By using a non-blocking method to start creating an executor, which completes the initialization of the inference model while the server startup process continues, the delay in the initialization of the inference model in the executor is hidden. When at least one executor has completed the initialization of the inference model and become a consumer, the server begins to receive service requests. Since an executor has already completed the initialization of the inference model, the service processing time does not include the inference model initialization time, thus solving the technical problem in related technologies where the service processing latency is too high due to the time consumed by model initialization.

[0153] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0154] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0155] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0156] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0157] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between units or modules, and may be electrical or other forms.

[0159] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0161] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A service provision method, characterized in that, include: After the executor is created using a non-blocking method, the target server is started. The executor is an independent thread or process created by the executor manager to obtain the prototype class from the target SDK and instantiate the prototype class to obtain an inference instance with the inference model initialized. The target SDK is an SDK used to build the target service. The prototype class inherits from the target service class. The target service class defines interfaces that must be implemented by subclasses, including the load_model interface for importing the inference model, the init_model interface for initializing the inference model, and the infer interface for performing inference. When there is at least one target executor that has been converted into a consumer, the target server begins to receive service requests, wherein the service request is used to request the invocation of the target service, the executor containing the inference instance is converted into a consumer in a state where it can handle the service request, and the inference model in each executor is initialized only once; When the target server receives a target service request, it processes the target service request through the inference instance in the target executor.

2. The method according to claim 1, characterized in that, When the target server receives a target service request, it processes the target service request through a reasoning instance in the target executor, including: When the target server receives a target service request, it transforms the target service request into a target task object through the task submitter and submits the target task object to the task container of the task manager. When the target executor detects the presence of the target task object in the task container, the target executor obtains the target task object and processes it through the inference instance in the target executor.

3. The method according to claim 2, characterized in that, Before starting the target server, the method further includes: After the service framework starts executing, the task container is created through the task manager. The service framework is used to access the target SDK to obtain the target service. The task manager is a task management class object. The task management class defines a task container attribute for storing task objects and a task submission method for submitting task objects to the task container.

4. The method according to claim 2, characterized in that, After starting the creation of the executor using a non-blocking method, the method further includes: The executor manager creates executors and places them in the executor container to obtain candidate executors; When a candidate executor obtains the prototype class from the target SDK, instantiates the prototype class into a reasoning instance that has completed the initialization of the reasoning model, and converts it into a consumer that listens to the task objects in the task container, the candidate executor is determined as the target executor. Here, the prototype class in the target SDK is a subclass of the target service class as the base class. The target service class defines at least one target interface that must be implemented by the subclass. The at least one target interface includes an interface for initializing the reasoning model. Each target executor in the executor container is a thread or process for processing task objects.

5. The method according to claim 4, characterized in that, After starting the creation of the executor using a non-blocking method, the method further includes: If the executor monitor detects that the health status of the first executor in the executor container is abnormal, the executor monitor calls the refresh method in the executor manager to shut down the first executor, create a second executor and place it in the executor container to replace the first executor. The executor monitor is used to monitor the health status of each executor in the executor container. After obtaining the prototype class from the target SDK and instantiating the prototype class into a reasoning instance that has completed the initialization of the reasoning model, the second executor is converted into a consumer that listens to the task objects in the task container.

6. The method according to claim 1, characterized in that, After starting the target server, the method further includes: Listen for health check requests through the target server's listening port; In the absence of an actuator that transforms into a consumer, when the target server receives a health check request, it responds to the health check request by returning target shutdown information, wherein the target shutdown information is used to indicate that the target server has not started receiving service requests.

7. The method according to claim 6, characterized in that, In the presence of at least one target executor that transforms into a consumer, the target server begins receiving service requests, including: In the presence of at least one target actuator that transforms into a consumer, when the target server receives a health check request, it returns target startup information in response to the health check request, wherein the target startup information is used to indicate that the target server can begin receiving service requests.

8. A service providing device, characterized in that, include: The startup module is used to start the target server after the executor is created using a non-blocking method. The executor is an independent thread or process created by the executor manager. It is used to obtain the prototype class in the target SDK and instantiate the prototype class to obtain an inference instance with the inference model initialized. The target SDK is an SDK used to build the target service. The prototype class inherits from the target service class. The target service class defines interfaces that must be implemented by subclasses, including the load_model interface for importing the inference model, the init_model interface for initializing the inference model, and the infer interface for performing inference. The receiving module, in the presence of at least one target executor that has been converted into a consumer, the target server begins to receive service requests, wherein the service request is used to request the invocation of the target service, the executor containing the inference instance is converted into a consumer in the state of being able to handle the service request, and the inference model in each executor is initialized only once; The processing module processes the target service request through the inference instance in the target executor when the target server receives the target service request.

9. An electronic device, characterized in that, include: The processor, communication interface, memory, and communication bus are provided, wherein the processor, communication interface, and memory communicate with each other via the communication bus. The memory is used to store computer programs; The processor, when executing the computer program, implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 7 when it is run.

Images