End-side multi-runtime artificial intelligence service method and apparatus, electronic device, and storage medium

Abstract

The application provides an end-side multi-Runtime artificial intelligence service method, device and equipment. The method is applied to an end-side deployed unified AI Runtime service, and includes the following steps: receiving a service request submitted by a calling party and carrying a target model identifier; querying a pre-stored model directory according to the target model identifier, determining a target model, and reading corresponding attributes and current state data of the target model; judging a request type corresponding to the service request; when the service request is a model ensure request, performing a full life cycle management process of the target model based on the attributes and the current state data; and when the service request is an inference routing request, performing an inference routing process corresponding to the service request through a Runtime corresponding to the target model. Through a unified interface and model life cycle management, the method solves the problems of non-uniform interfaces and unobservable resource states of end-side multi-Runtime.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence service technology, specifically to a method, apparatus, electronic device, and storage medium for edge-side multi-Runtime artificial intelligence services. Background Technology

[0002] With the development of artificial intelligence technology, mobile terminals (such as AI smart devices) may simultaneously deploy multiple different types of AI runtimes, such as Large Language Models (LLM), speech recognition, image generation, and multimodal inference. These runtimes differ in model format, download method, loading interface, and warm-up method. In existing technologies, if an application or AIAgent needs to call the capabilities of the edge model, it typically needs to directly adapt to the specific interfaces of each runtime. This not only increases the complexity of integration, but more importantly, it makes it difficult for the caller to uniformly and in real-time obtain the preparation status of the required model resources (such as download, verification, loading, and warm-up progress) before initiating an inference request. Traditional OpenAI-compatible interfaces for cloud-based models are primarily designed to handle the inference request itself and do not cover the complete lifecycle management of edge model files.

[0003] Existing technologies for handling model resource preparation suffer from issues such as monolithic implementation or reliance on manual intervention. For example, the caller is tightly coupled with specific backend or system capabilities; backend upgrades, process restarts, or changes in resource status directly impact the stability of the main interaction chain, potentially leading to inference failures. More importantly, existing solutions lack a structured, streaming mechanism that allows the caller to observe the entire lifecycle of the model from download to readiness, and to automate status assessments and anomaly recovery in scenarios with limited mobile resources or unstable networks. Consequently, this results in a poor user experience and difficulty in guaranteeing system stability. Summary of the Invention

[0004] In view of the aforementioned problems, this application is made to provide an edge-side multi-Runtime artificial intelligence service method, apparatus, electronic device, and storage medium that overcomes or at least partially solves the aforementioned problems, including: A method for providing multi-Runtime AI services on the edge, applied to a unified AI Runtime service deployed on the edge, includes: Receive service requests submitted by the caller that carry the target model identifier; The target model is determined by querying the pre-stored model directory based on the target model identifier, and the corresponding attributes and current status data of the target model are read. Determine the request type corresponding to the service request; When the service request is a model ensure request, the full lifecycle management process of the target model is executed based on the attribute and the current state data; When the service request is an inference routing request, the inference routing process corresponding to the service request is executed through the Runtime corresponding to the target model.

[0005] Furthermore, the full lifecycle management process includes a download phase, a verification phase, a loading phase, and a warm-up phase. The execution of the full lifecycle management process for the target model based on the attributes and the current state data includes: Based on the attributes and the current state data, determine whether the local file corresponding to the target model is complete; If the local file is incomplete, the download process begins, the download task corresponding to the local file is started, and the download progress is recorded. After the download task is completed, the verification step is performed to check the integrity of the local file. If the verification passes, the loading process begins, where the loader corresponding to the target model is invoked based on the local file to load the target model. After the target model is loaded, it is determined whether the target model needs to be preheated based on the preheating flag in the service request. If the target model needs to be preheated, the preheating process is initiated, and a preheating input is sent to the corresponding Runtime of the target model to complete the initialization and obtain ready status data.

[0006] Furthermore, after determining the request type corresponding to the service request, the method further includes: When the service request is a model ensure request, the status, progress and exception information of each stage of the full lifecycle management process are returned to the caller in real time through a preset streaming protocol; After the entire lifecycle management process is completed, a ready event is returned to the caller; The preset streaming protocol includes any one of the following native communication protocols: SSE, WebSocket, AIDL, and UDS.

[0007] Furthermore, the real-time return of status, progress, and exception information for each stage of the full lifecycle management process to the caller via a preset streaming protocol includes: During the download process, the number of bytes downloaded, the total number of bytes, the download speed, and the current download source information are returned to the caller in real time. In the verification stage, the loading stage, and the warm-up stage, the execution status information of the corresponding stage is returned to the caller in real time. If any of the download, verification, loading, or warm-up stages encounters an error, the error type and retry prompt information will be returned to the caller in real time.

[0008] Further, the step of sending a warm-up input to the corresponding Runtime of the target model to complete initialization and obtain ready state data includes: Send a preset zero-load inference input to the corresponding Runtime to trigger the corresponding Runtime to complete the core engine initialization and memory allocation operations; Once the ready state is recorded, the ready state data is generated.

[0009] Further, the step of executing the inference routing process corresponding to the service request through the Runtime corresponding to the target model includes: The task type of the service request is parsed, and the task type includes at least one of natural language interaction, image generation, and audio transcription. The service request is routed to the Runtime for inference based on the task type and the target model identifier, and the inference output returned by the Runtime is received. The inference output is formatted according to the OpenAI interface specification to obtain a unified format inference response.

[0010] Furthermore, the unified AI Runtime service is deployed on the Android application layer, the system pre-installed application layer, or the Framework customization layer of the AI ​​smart device, and the AI ​​smart device includes at least one of AI smart cash register, AI smart tablet, AI smartphone, and AI POS machine.

[0011] An edge-side multi-Runtime AI service device, deployed on the edge, includes: The request receiving module is used to receive service requests submitted by the caller that carry the target model identifier; The directory query module is used to query the pre-stored model directory based on the target model identifier, determine the target model, and read the attribute and current status data corresponding to the target model. The request determination module is used to determine the request type corresponding to the service request; The full lifecycle management module is used to execute the full lifecycle management process of the target model based on the attributes and the current state data when the service request is a model ensure request; The inference routing module is used to execute the inference routing process corresponding to the service request through the Runtime corresponding to the target model when the service request is an inference routing request.

[0012] A computer electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the edge-side multi-runtime artificial intelligence service method as described above.

[0013] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the edge-side multi-runtime artificial intelligence service method as described above.

[0014] This application has the following advantages: In the embodiments of this application, addressing the issues of monolithic implementation or reliance on manual intervention in the prior art when processing model resource preparation, this application provides a "structured and streaming mechanism" solution. Specifically, it involves: "receiving a service request submitted by the caller carrying a target model identifier; querying the pre-stored model directory based on the target model identifier to determine the target model, and reading the corresponding attribute and current state data of the target model; determining the request type corresponding to the service request; when the service request is a model ensure request, executing the full lifecycle management process of the target model based on the attribute and current state data; when the service request is an inference routing request, executing the inference routing process corresponding to the service request through the Runtime corresponding to the target model." This application establishes a unified AI Runtime service, which provides a unified entry point for model ensure requests and inference routing requests. Specifically, the service queries the model directory based on the target model identifier to obtain attributes and current state, and determines the request type accordingly. For model ensure requests, the service automatically performs full lifecycle management covering download, validation, loading, and warm-up based on model attributes and status, and provides real-time feedback on the status of each stage to the caller via a streaming protocol. For inference routing requests, the service routes the request to the corresponding runtime and returns an inference response in a unified format. By encapsulating the model resource preparation process into an observable and manageable lifecycle flow and scheduling it through a unified interface, the caller does not need to be concerned with the differences in the backend runtime and the details of resource preparation, thus solving the problems of inconsistent interfaces, unknown model preparation status, and inability to automatically recover in existing technologies. This brings the following benefits: First, by unifying interface encapsulation and automating lifecycle management, the decoupling between the caller and the specific Runtime is achieved, enhancing the stability and maintainability of the system and preventing the entire chain from failing due to changes or anomalies in a single Runtime. Second, by providing real-time feedback on the status, progress, and anomalies at each stage of model preparation through streaming protocols, the observability of the system is significantly improved, enabling the caller to understand the resource preparation status in a timely manner and optimizing the user experience. Finally, by automating the download, verification, loading, and warm-up processes, it is ensured that model resources are correctly ready before inference, improving the success rate and response speed of inference requests, and achieving more efficient resource utilization, especially in resource-constrained environments on mobile devices. Attached Figure Description

[0015] To more clearly illustrate the technical solution of this application, the drawings used in the description of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the steps of a multi-Runtime AI service method on the edge, according to an embodiment of this application. Figure 2 This is a structural block diagram of an edge-side multi-Runtime artificial intelligence service device provided in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer electronic device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of an SSE status event format provided in an embodiment of the present invention; 1. A computer electronic device; 2. An external device; 3. A processing unit; 4. A bus; 5. A network adapter; 6. An I / O interface; 7. A display; 8. Memory; 9. Random access memory; 10. A cache memory; 11. A storage system; 12. A program / utility; 13. A program module. Detailed Implementation

[0017] To make the objectives, features, and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0018] This application's embodiments can be applied to AI smart devices, particularly mobile terminals or embedded devices that integrate multiple AI models, such as AI smart cash registers, AI smart tablets, AI smartphones, or AI POS machines. These devices typically simultaneously host Large Language Model (LLM) Runtimes, speech recognition (such as Whisper) Runtimes, image generation (such as StableDiffusion) Runtimes, or Visual Language Model (VLM) Runtimes. This application refers to the service running on these devices and used to manage and schedule these heterogeneous Runtimes as a "Unified AI Runtime Service."

[0019] Analysis of existing technologies reveals a lack of structured, streaming mechanisms when deploying multiple AI runtimes simultaneously. This lack prevents callers from observing the entire model lifecycle from download to readiness, and from automating status assessments and anomaly recovery in scenarios with limited mobile resources and unstable networks. Consequently, user experience suffers, and system stability is compromised. Therefore, this solution decouples the caller from the complex state management of the underlying model download, verification, loading, and warm-up processes. Through streaming protocols like SSE, the entire lifecycle state is transparently exposed to the caller in real-time, resolving the issues of unknowable, uncontrollable, and inconsistent interfaces in the edge AI model resource preparation process. The resulting benefits include: significantly improved system stability, as calls are not directly bound to volatile backend capabilities, avoiding single-point changes affecting the entire chain; greatly enhanced observability, allowing callers to accurately identify the current service stage (download, loading, execution, failure, or readiness); and optimized resource efficiency, reducing cold start latency for initial inference through preloading, warm-up, and automated management, and achieving more efficient CPU and memory utilization in resource-constrained mobile environments.

[0020] To facilitate understanding, some terms used in the embodiments of this application will be explained first: Model ensure request: A type of service request used to ensure that model file resources are in an available state. It encapsulates the complete lifecycle stages such as model checking, downloading, validating, loading, and warming up, which is different from requests that directly perform inference.

[0021] Model catalog: A pre-stored data structure used to record the core metadata of locally registered models, including the Runtime type, capability set, local storage path, download source address, and current status (such as downloading, ready, etc.) that the model depends on.

[0022] Full lifecycle management process: refers to a series of automated operations performed by the unified AI Runtime service on model resources from an unavailable state to a fully ready state, which may include download, verification, loading and warm-up.

[0023] SSE: Server-Sent Events, an HTTP protocol that allows servers to push streaming events unidirectionally to clients. In this application, it is used to notify the caller of the progress and status of each stage of model preparation in real time.

[0024] Warm-up: This refers to the process of sending simulated inputs to the Runtime after the model file is loaded and before the actual inference request is processed, in order to trigger the core engine initialization and memory pre-allocation.

[0025] Reference Figure 1 This illustration shows an embodiment of a method for providing multi-Runtime AI services on the edge, applied to a unified AI Runtime service deployed on the edge, specifically including: S101. Receive a service request submitted by the caller, which carries the target model identifier; S102. Query the pre-stored model directory according to the target model identifier, determine the target model, and read the attribute and current status data corresponding to the target model; S103. Determine the request type corresponding to the service request; S104. When the service request is a model ensure request, the full lifecycle management process of the target model is executed based on the attribute and the current state data. S105. When the service request is an inference routing request, the inference routing process corresponding to the service request is executed through the Runtime corresponding to the target model. In the embodiments of this application, a unified AI Runtime service is set up, which provides a unified entry point for model ensure requests and inference routing requests. Specifically, the service queries the model directory based on the target model identifier to obtain attributes and current status, and determines the request type accordingly. For model ensure requests, the service automatically performs full lifecycle management covering download, verification, loading, and warm-up based on model attributes and status, and provides real-time feedback on the status of each stage to the caller via a streaming protocol. For inference routing requests, the service routes the request to the corresponding Runtime and returns an inference response in a unified format. By encapsulating the model resource preparation process into an observable and manageable lifecycle flow and scheduling it through a unified interface, the caller does not need to be concerned with the differences in the backend Runtime and the details of resource preparation, thus solving the problems of inconsistent interfaces, unknown model preparation status, and inability to automatically recover in the prior art. This brings the following benefits: First, by unifying interface encapsulation and automating lifecycle management, the decoupling between the caller and the specific Runtime is achieved, enhancing the stability and maintainability of the system and preventing the entire chain from failing due to changes or anomalies in a single Runtime. Second, by providing real-time feedback on the status, progress, and anomalies at each stage of model preparation through streaming protocols, the observability of the system is significantly improved, enabling the caller to understand the resource preparation status in a timely manner and optimizing the user experience. Finally, by automating the download, verification, loading, and warm-up processes, it is ensured that model resources are correctly ready before inference, improving the success rate and response speed of inference requests, and achieving more efficient resource utilization, especially in resource-constrained environments on mobile devices.

[0026] In a specific implementation, the above steps work together. The unified AI Runtime service provides a single entry point through step S101, receiving service requests from different callers (such as applications or AI Agents), which carry the target model identifier. Subsequently, in step S102, the service queries its internal model directory. This directory, as a specific implementation, can be designed as a local database or a structured configuration file, recording the Runtime type (such as LiteRT, ONNX, etc.), capability set (such as dialogue, image generation, speech recognition), local storage path, list of available download sources, and current status data (such as downloading, downloaded, loaded, ready, or error) for each model. By querying the model directory, the service can map the logical identifier passed in by the caller to specific physical resources and current status. Step S103, as the core branch logic, determines the intent of this call based on the request content. If it's a model ensure request (step S104), the service doesn't directly execute inference. Instead, it drives an automated lifecycle management process that includes downloading, verification, loading, and warm-up, ensuring a smooth transition of model resources from their current state to the ready state. If it's an inference routing request (step S105), the service acts as a proxy and coordinator, transparently forwarding the request to the corresponding backend runtime and processing the response. This mechanism decouples the caller from the underlying heterogeneous runtime and complex resource preparation details. The caller only needs to interact with a unified interface, resolving the instability of the main link caused by backend changes or resource state fluctuations.

[0027] Taking a specific application scenario as an example, in iBoxAI OS, the AI ​​Agent or desktop card (Hiboard) needs to use the speech-to-text model for the first time. The caller constructs a service request, setting the model as the target model identifier in the service request. After receiving the service request, the Unified AI Runtime service queries the model directory and learns that the model has not yet been downloaded. Then, the Unified AI Runtime service determines that this is a model ensure request. The Unified AI Runtime service automatically triggers the download task based on the download source address recorded in the model directory, and continuously pushes the download progress (such as "30% downloaded") through a streaming protocol, allowing the desktop card to display a progress bar in real time. After the download is complete, the Unified AI Runtime service automatically performs verification and loading, and starts warm-up according to the warm-up indicator carried in the request. After all steps are completed, it pushes a "ready" event. After receiving the "ready" event, the desktop card can trigger the subsequent inference routing request, which is routed by the Unified AI Runtime service to the Whisper Runtime to perform audio transcription. For the desktop card, the entire process only involves initiating one preparation request and one inference request through the unified interface, without needing to worry about the underlying file operations and runtime differences.

[0028] The following will further explain the edge-side multi-Runtime artificial intelligence service method in this exemplary embodiment.

[0029] In one embodiment of the present invention, the full lifecycle management process includes a download stage, a verification stage, a loading stage, and a warm-up stage. The specific process of "executing the full lifecycle management process of the target model based on the attribute and the current state data" in step S104 can be further described in conjunction with the following description.

[0030] As described in the following steps, based on the attributes and the current state data, it is determined whether the local file corresponding to the target model is complete; if the local file is incomplete, the download phase is initiated, the download task corresponding to the local file is started, and the download progress is recorded; after the download task is completed, the verification phase is initiated to verify the integrity of the local file; if the verification passes, the loading phase is initiated, and the loader corresponding to the target model is called based on the local file to load the target model; after the target model is loaded, it is determined whether the target model needs to be preheated based on the preheating flag in the service request; if the target model needs to be preheated, the preheating phase is initiated, a preheating input is sent to the corresponding Runtime of the target model to complete the initialization, and ready state data is obtained.

[0031] In one specific implementation, the above steps, through well-defined sequential stages, transform the model preparation process from scratch into a controllable and observable automated pipeline. This process first compares the current state data in the model directory (such as whether the local file exists and whether the checksum matches) with the target attributes to intelligently determine the entry point, avoiding redundant work. For example, if the model file already exists but has not been loaded, the process can start directly from the loading stage. The entire process is broken down into sub-stages, each with its own responsibilities. Successful execution of each stage is a prerequisite for proceeding to the next stage; this serialized operation ensures the atomicity and consistency of resource state transitions. As a specific implementation, the model file uploaded to the server can have its hash value (e.g., SHA-256) pre-calculated and recorded in the attributes of the model directory. In the verification stage, the service recalculates the hash value of the downloaded local file and rigorously compares it with the hash value recorded in the model directory; if the comparison matches, the verification is considered successful. This ensures that even if network instability leads to file corruption, an incorrect model will not be loaded into memory, thus improving system robustness.

[0032] In one embodiment of the present invention, the specific features of the end-side multi-Runtime artificial intelligence service method after the above step of "determining the request type corresponding to the service request" can be further described in conjunction with the following description.

[0033] As described in the following steps, when the service request is a model ensure request, the status, progress, and exception information of each stage of the full lifecycle management process are returned to the caller in real time through a preset streaming protocol; after the full lifecycle management process is completed, a ready event is returned to the caller; the preset streaming protocol includes any one of the following native communication protocols: SSE, WebSocket, AIDL, and UDS.

[0034] In one specific implementation, the above steps solve the "black box" waiting problem for the caller during model preparation. By introducing a preset streaming protocol, the service transforms from passive to active, continuously pushing the real-time running status of the internal pipeline to the caller. This design allows the caller not only to know the final "success" or "failure," but also to observe whether it is currently downloading, loading, or warming up, and to obtain the specific progress percentage. This not only improves the user interface's interactivity but also provides a decision-making basis for the caller's own exception handling logic (such as timeout judgment and user-initiated cancellation). As a specific implementation method, the unified AI Runtime service deployed on Android devices can be implemented as a foreground service based on EdgeBox. This service starts a lightweight HTTP server on the local loopback address and selects SSE as the preset streaming protocol. The advantage of choosing SSE is that it is based on the standard HTTP protocol, is simple to implement, and naturally supports reconnection after disconnection, making it very suitable for efficient, one-way event push scenarios on the local machine. Preferably, the service uses WorkManager to schedule background tasks to ensure the reliable execution of the download phase.

[0035] In one embodiment of the present invention, the specific process of the above step "returning the status, progress and exception information of each stage of the full lifecycle management process to the caller in real time through a preset streaming protocol" can be further described in conjunction with the following description.

[0036] As described in the following steps, during the download phase, the number of bytes downloaded, the total number of bytes, the download speed, and the current download source information are returned to the caller in real time; during the verification phase, the loading phase, and the warm-up phase, the execution status information of the corresponding phase is returned to the caller in real time; when any of the download phase, verification phase, loading phase, and warm-up phase encounters an abnormality, the error type and retry prompt information are returned to the caller in real time.

[0037] In one specific implementation, the above steps define the message body of SSE events with fine granularity, distinguishing the push content for different stages. During the download stage, not only is the progress reported, but the current download source and download speed are also displayed. This allows the caller to estimate the remaining time based on the speed information, or to understand the currently used source in a multi-source scenario. In the relatively fixed processing logic of the verification, loading, and warm-up stages, concise status information (such as "Verifying" or "Loading") is pushed. More importantly, a unified exception reporting mechanism is established. Once a stage fails, a structured error event containing the error type (such as "Download Failed - Network Error," "Verification Failed - Hash Mismatch") and retry prompts immediately appears in the event stream. This allows the caller or end user to react accordingly, such as manually retrying or checking storage space, greatly improving system maintainability and user experience. Figure 4 The diagram illustrates an SSE status event format, which may include an event type (such as edgebox.model.status) and a JSON data payload carrying fields such as the number of bytes downloaded, the total number of bytes, and the stage identifier.

[0038] In one embodiment of the present invention, the specific process of the above step "sending a preheating input to the corresponding Runtime of the target model to complete the initialization and obtain ready state data" can be further explained in conjunction with the following description.

[0039] As described in the following steps, a preset zero-load inference input is sent to the corresponding Runtime to trigger the corresponding Runtime to complete the core engine initialization and memory allocation operations; the ready state after completion is recorded, and the ready state data is generated.

[0040] In a specific implementation, the above steps clarify the implementation method of the warm-up phase. The so-called "preset zero-load inference input" refers to a simulated input designed to activate the construction of the runtime's internal computation graph and pre-allocation of memory, but without producing meaningful inference results. For example, for an LLM runtime, the warm-up input could be a very short text containing only a single token; for an image generation runtime, it could be a minimal solid-color noise image. By actively triggering this "empty run" before the official service, the inherent cold start latency of the runtime during the first inference is eliminated, allowing subsequent first inference requests from real users to achieve near-warm-start response speeds, significantly optimizing the end-to-end interactive experience. As a specific implementation, when the runtime is LiteRT, the warm-up operation can be concretized as calling the LiteRT interpreter's `allocateTensors()` method or performing an inference of a preset input tensor.

[0041] In one embodiment of the present invention, the specific process of "executing the inference routing process corresponding to the service request through the Runtime corresponding to the target model" in step S105 can be further described in conjunction with the following description.

[0042] As described in the following steps, the task type of the service request is parsed, the task type including at least one of natural language interaction, image generation, and audio transcription; the service request is routed to the Runtime for inference according to the task type and the target model identifier, and the inference output returned by the Runtime is received; the inference output is formatted according to the OpenAI interface specification to obtain a unified format inference response.

[0043] In one specific implementation, the Unified AI Runtime service acts as a protocol gateway. The service identifies the caller's true intent (chat, drawing, or speech-to-text) by parsing the request content, and then converts this intent and request payload into a format understandable by the specific backend runtime, achieving intelligent routing from task type to runtime. For example, when the task type is determined to be "audio transcription," the service routes the request to the Whisper Runtime. Finally, regardless of the backend runtime, its raw output is uniformly packaged by the service into a standard OpenAI-compatible interface format. This means that an application written for the cloud-based OpenAI interface can seamlessly switch to using multiple local AI models with little or no code modification, significantly reducing the barrier to entry and migration costs for edge AI capabilities.

[0044] In one embodiment of the present invention, the "device-side multi-Runtime artificial intelligence service method" can be further described in conjunction with the following description.

[0045] In one specific implementation, the unified AI Runtime service is deployed at the Android application layer, the system pre-installed application layer, or the Framework customization layer of the AI ​​smart device. The AI ​​smart device includes at least one of AI smart cash registers, AI smart tablets, AI smartphones, and AI POS machines. This clarifies the physical carrier and deployment flexibility of this technical solution. It can run as a regular Android application at the application layer, facilitating independent updates and distribution; it can also run as a system pre-installed application, gaining higher process priority and a more reliable keep-alive mechanism; furthermore, for deeply customized devices, it can be integrated into the Android Framework layer, running as a system service, providing the lowest-level AI capability support for all upper-layer applications. This flexible deployment method allows this solution to adapt to different levels and types of AI smart devices, finding suitable integration paths from general-purpose AI smartphones to specialized AI smart cash registers.

[0046] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0047] Reference Figure 2 This illustration shows an edge-side multi-Runtime AI service device 200 according to an embodiment of this application, used to implement the edge-side multi-Runtime AI service method provided in the embodiments of this application. The device is deployed on the edge and specifically includes: The request receiving module 210 is used to receive service requests submitted by the caller that carry the target model identifier; The directory query module 220 is used to query the pre-stored model directory according to the target model identifier, determine the target model, and read the attribute and current status data corresponding to the target model; Request determination module 230 is used to determine the request type corresponding to the service request; The full lifecycle management module 240 is used to execute the full lifecycle management process of the target model based on the attributes and the current state data when the service request is a model ensure request; The inference routing module 250 is used to execute the inference routing process corresponding to the service request through the Runtime corresponding to the target model when the service request is an inference routing request.

[0048] In one embodiment of the present invention, the full lifecycle management process includes a download stage, a verification stage, a loading stage, and a preheating stage, and the full lifecycle management module 240 is further used for: Based on the attributes and the current state data, determine whether the local file corresponding to the target model is complete; If the local file is incomplete, the download process begins, the download task corresponding to the local file is started, and the download progress is recorded. After the download task is completed, the verification step is performed to check the integrity of the local file. If the verification passes, the loading process begins, where the loader corresponding to the target model is invoked based on the local file to load the target model. After the target model is loaded, it is determined whether the target model needs to be preheated based on the preheating flag in the service request. If the target model needs to be preheated, the preheating process is initiated, and a preheating input is sent to the corresponding Runtime of the target model to complete the initialization and obtain ready status data.

[0049] In one embodiment of the present invention, the full lifecycle management module 240 is further configured to: When the service request is a model ensure request, the status, progress and exception information of each stage of the full lifecycle management process are returned to the caller in real time through a preset streaming protocol; After the entire lifecycle management process is completed, a ready event is returned to the caller; The preset streaming protocol includes any one of the following native communication protocols: SSE, WebSocket, AIDL, and UDS.

[0050] In one embodiment of the present invention, the full lifecycle management module 240 is further configured to: During the download process, the number of bytes downloaded, the total number of bytes, the download speed, and the current download source information are returned to the caller in real time. In the verification stage, the loading stage, and the warm-up stage, the execution status information of the corresponding stage is returned to the caller in real time. If any of the download, verification, loading, or warm-up stages encounters an error, the error type and retry prompt information will be returned to the caller in real time.

[0051] In one embodiment of the present invention, the full lifecycle management module 240 is further configured to: Send a preset zero-load inference input to the corresponding Runtime to trigger the corresponding Runtime to complete the core engine initialization and memory allocation operations; Once the ready state is recorded, the ready state data is generated.

[0052] In one embodiment of the present invention, the inference routing module 250 is further configured to: The task type of the service request is parsed, and the task type includes at least one of natural language interaction, image generation, and audio transcription. The service request is routed to the Runtime for inference based on the task type and the target model identifier, and the inference output returned by the Runtime is received. The inference output is formatted according to the OpenAI interface specification to obtain a unified format inference response.

[0053] In one embodiment of the present invention, the unified AI Runtime service is deployed on the Android application layer, the system pre-installed application layer, or the Framework customization layer of the AI ​​smart device, and the AI ​​smart device includes at least one of AI smart cash register, AI smart tablet, AI smartphone, and AI POS machine.

[0054] Reference Figure 3 The illustration shows a computer electronic device 1 for implementing the edge-side multi-Runtime artificial intelligence service method of the present invention, which may specifically include: The computer electronic device 1 is manifested in the form of a general computing device, including: one or more processors or processing units 3, memory 8, and a bus 4 connecting different system components (including memory 8 and processing unit 3).

[0055] Bus 4 represents one or more of several bus architectures, including memory buses or memory controllers, peripheral buses, graphics acceleration ports, processors, or local buses using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Audio / Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0056] Computer electronic device 1 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computer electronic device 1, including volatile and non-volatile media, removable and non-removable media.

[0057] Memory 8 may include computer system readable media in the form of volatile memory, such as random access memory 9 and / or cache memory 10. A computer electronic device 1 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 11 may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as a "hard disk drive"). Although Figure 3As not shown, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 4 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 13 configured to perform the functions of the embodiments of this application.

[0058] A program / utility 12 having a set (at least one) of program modules 13 may be stored, for example, in memory. Such program modules 13 include—but are not limited to—an operating system, one or more application programs, other program modules 13, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 13 typically perform the functions and / or methods described in the embodiments of this application.

[0059] The computer electronic device 1 can also communicate with one or more external devices 2 (e.g., keyboard, pointing device, display 7, camera, etc.), and with one or more devices that enable an operator to interact with the computer electronic device 1, and / or with any device that enables the computer electronic device 1 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through the I / O interface 6. Furthermore, the computer electronic device 1 can also communicate with one or more networks (e.g., local area network (LAN)), wide area network (WAN), and / or public networks (e.g., the Internet) through the network adapter 5. Figure 3 As shown, network adapter 5 communicates with other modules of computer electronic device 1 via bus 4. It should be understood that, although... Figure 3 Not shown, it may be combined with other hardware and / or software modules, including but not limited to: microcode, device drivers, redundant processing unit 3, external disk drive array, RAID system, tape drive and data backup storage system 11, etc.

[0060] The processing unit 3 executes various functional applications and data processing by running programs stored in memory 8, such as implementing the end-side multi-Runtime artificial intelligence service method provided in the embodiments of this application.

[0061] That is, when the above-mentioned processing unit 3 executes the above program, it implements the following: receiving a service request submitted by the caller carrying a target model identifier; querying the pre-stored model directory according to the target model identifier, determining the target model, and reading the attribute and current status data corresponding to the target model; determining the request type corresponding to the service request; when the service request is a model ensure request, executing the full lifecycle management process of the target model based on the attribute and the current status data; when the service request is an inference routing request, executing the inference routing process corresponding to the service request through the Runtime corresponding to the target model.

[0062] In this application embodiment, the application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the edge-side multi-Runtime artificial intelligence service method as provided in all embodiments of the application.

[0063] That is, when the program is executed by the processor, it implements the following: receiving a service request submitted by the caller carrying a target model identifier; querying the pre-stored model directory based on the target model identifier to determine the target model, and reading the corresponding attributes and current state data of the target model; determining the request type corresponding to the service request; when the service request is a model ensure request, executing the full lifecycle management process of the target model based on the attributes and the current state data; when the service request is an inference routing request, executing the inference routing process corresponding to the service request through the Runtime corresponding to the target model.

[0064] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0065] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0066] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the operator's computer, partially on the operator's computer, as a standalone software package, partially on the operator's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the operator's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider). The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably.

[0067] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0068] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0069] The foregoing has provided a detailed description of the edge-side multi-Runtime artificial intelligence service method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for providing multiple runtime artificial intelligence services on the edge, characterized in that, The unified AIRuntime service deployed on the client side includes: Receive service requests submitted by the caller that carry the target model identifier; The target model is determined by querying the pre-stored model directory based on the target model identifier, and the corresponding attributes and current status data of the target model are read. Determine the request type corresponding to the service request; When the service request is a model ensure request, the full lifecycle management process of the target model is executed based on the attribute and the current state data; When the service request is an inference routing request, the inference routing process corresponding to the service request is executed through the Runtime corresponding to the target model.

2. The method according to claim 1, characterized in that, The full lifecycle management process includes a download phase, a verification phase, a loading phase, and a warm-up phase. The execution of the full lifecycle management process for the target model based on the attributes and the current state data includes: Based on the attributes and the current state data, determine whether the local file corresponding to the target model is complete; If the local file is incomplete, the download process begins, the download task corresponding to the local file is started, and the download progress is recorded. After the download task is completed, the verification step is performed to check the integrity of the local file. If the verification passes, the loading process begins, where the loader corresponding to the target model is invoked based on the local file to load the target model. After the target model is loaded, it is determined whether the target model needs to be preheated based on the preheating flag in the service request. If the target model needs to be preheated, the preheating process is initiated, and a preheating input is sent to the corresponding Runtime of the target model to complete the initialization and obtain ready status data.

3. The method according to claim 2, characterized in that, After determining the request type corresponding to the service request, the method further includes: When the service request is a model ensure request, the status, progress and exception information of each stage of the full lifecycle management process are returned to the caller in real time through a preset streaming protocol; After the entire lifecycle management process is completed, a ready event is returned to the caller; The preset streaming protocol includes any one of the following native communication protocols: SSE, WebSocket, AIDL, and UDS.

4. The method according to claim 3, characterized in that, The method of returning the status, progress, and exception information of each stage of the full lifecycle management process to the caller in real time through a preset streaming protocol includes: During the download process, the number of bytes downloaded, the total number of bytes, the download speed, and the current download source information are returned to the caller in real time. In the verification stage, the loading stage, and the warm-up stage, the execution status information of the corresponding stage is returned to the caller in real time. If any of the download, verification, loading, or warm-up stages encounters an error, the error type and retry prompt information will be returned to the caller in real time.

5. The method according to claim 2, characterized in that, The step of sending a warm-up input to the corresponding Runtime of the target model to complete initialization and obtain ready state data includes: Send a preset zero-load inference input to the corresponding Runtime to trigger the corresponding Runtime to complete the core engine initialization and memory allocation operations; Once the ready state is recorded, the ready state data is generated.

6. The method according to claim 1, characterized in that, The process of executing the inference routing corresponding to the service request through the Runtime corresponding to the target model includes: The task type of the service request is parsed, and the task type includes at least one of natural language interaction, image generation, and audio transcription. The service request is routed to the Runtime for inference based on the task type and the target model identifier, and the inference output returned by the Runtime is received. The inference output is formatted according to the OpenAI interface specification to obtain a unified format inference response.

7. The method according to any one of claims 1 to 6, characterized in that, The unified AI Runtime service is deployed in the Android application layer, the system pre-installed application layer, or the Framework customization layer of the AI ​​smart device. The AI ​​smart device includes at least one of the following: AI smart cash register, AI smart tablet, AI smartphone, and AI POS machine.

8. An edge-side multi-Runtime artificial intelligence service device, characterized in that, Deployed on the edge, including: The request receiving module is used to receive service requests submitted by the caller that carry the target model identifier; The directory query module is used to query the pre-stored model directory based on the target model identifier, determine the target model, and read the attribute and current status data corresponding to the target model. The request determination module is used to determine the request type corresponding to the service request; The full lifecycle management module is used to execute the full lifecycle management process of the target model based on the attributes and the current state data when the service request is a model ensure request; The inference routing module is used to execute the inference routing process corresponding to the service request through the Runtime corresponding to the target model when the service request is an inference routing request.

9. A computer electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when executed by the processor, the computer program implements the steps of the edge-side multi-runtime artificial intelligence service method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when executed by a processor, the computer program implements the steps of the edge-side multi-runtime artificial intelligence service method as described in any one of claims 1 to 7.

Images