Method and apparatus for running a model inference algorithm

Abstract

The embodiment of the application provides a model inference algorithm running method, the method comprises the following steps: a CLI running manager receives a calling request of a CLI package; the CLI running manager reads a declaration file of the CLI package, wherein the CLI package comprises one or more executable files, the declaration file comprises a life cycle management strategy and an interactive data format specification of the CLI package, and the executable file is used for executing model inference when running; the CLI running manager calls the CLI package according to the declaration file, and performs life cycle management on the executable file according to the life cycle management strategy and executes model inference according to the interactive data format specification without parsing the executable file. The technical scheme solves the problem that the adaptability of various model inference algorithms in the prior art is not high, and achieves the technical effect of providing universality.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and edge computing technology, and in particular to a method and apparatus for running a model inference algorithm. Background Technology

[0002] With the widespread application of artificial intelligence technology in edge computing scenarios, an increasing number of model inference algorithms need to be deployed on resource-constrained edge devices. These algorithms are typically implemented using different programming languages ​​and deep learning frameworks, exhibiting high heterogeneity.

[0003] Currently, the industry primarily adopts Docker containerization deployment. This solution encapsulates each model inference algorithm into an independent Docker container, managed through container orchestration tools. However, different model inference algorithms have different runtime environments and dependencies. While containerization achieves environment isolation, it still requires writing specific container image build scripts and runtime configurations for each algorithm, resulting in a large workload for adaptation and insufficient versatility.

[0004] Another approach is to use HTTP or gRPC microservices. This approach encapsulates the model inference algorithm as an HTTP or gRPC service, which is then invoked via network protocols. However, the service interface definition, data format conversion, and communication protocol adaptation for each algorithm all need to be developed separately. The lack of a unified algorithm invocation standard leads to low algorithm integration efficiency and makes it difficult to achieve rapid switching and collaborative inference among multiple algorithms.

[0005] In summary, existing technologies are not highly adaptable to various model inference algorithms, requiring the writing of dedicated adaptation code for each model inference algorithm. This results in poor versatility and fails to meet the needs of rapid integration and flexible deployment of multiple algorithms in edge computing scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide a method and apparatus for running a model inference algorithm, so as to solve the technical problems of poor adaptability to various model inference algorithms in the prior art, the need to write special adaptation code for each model inference algorithm, and poor universality.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, embodiments of the present invention provide a method for running a model inference algorithm, the method comprising: a CLI runtime manager receiving a request to invoke a CLI package; the CLI runtime manager reading a declaration file of the CLI package, wherein the CLI package includes one or more executable files, the declaration file includes a lifecycle management strategy and an interaction data format specification for the CLI package, and the executable file is used to perform model inference when it runs; the CLI runtime manager invoking the CLI package according to the declaration file, and without parsing the executable file, performing lifecycle management on the executable file according to the lifecycle management strategy and performing model inference according to the interaction data format specification.

[0009] Optionally, the declaration file includes the resource requirements required to run the executable file. Lifecycle management of the executable file according to the lifecycle management strategy includes: the CLI runtime manager reading the declaration file to obtain the resource requirements and allocating available resources to the executable file according to the resource requirements.

[0010] Optionally, the declaration file includes a resident startup mode declaration. Lifecycle management of the executable file according to the lifecycle management strategy includes: the CLI run manager starts one or more preheating CLI algorithm processes for the executable file in resident startup mode and marks the preheating CLI algorithm processes as ready; when the CLI run manager receives a request to execute model inference, it selects one of the one or more preheating CLI algorithm processes to execute model inference; the CLI run manager keeps the preheating CLI algorithm process that has completed model inference marked as ready so that it can be reused the next time model inference is executed.

[0011] Optionally, when the CLI Run Manager receives a request to execute model inference, it selects one of one or more warm-up CLI algorithm processes to execute model inference, including: the CLI Run Manager obtains a warm-up CLI algorithm process marked as ready from the process pool; if there is no available warm-up CLI algorithm process in the process pool, the CLI Run Manager creates a new process as needed to execute model inference.

[0012] Optionally, after the CLI Run Manager keeps the preheated CLI algorithm process that has completed model inference marked as ready, the method includes: the CLI Run Manager periodically probing the liveness status of the preheated CLI algorithm process and the last usage time of each preheated CLI algorithm process; if the idle time of the preheated CLI algorithm process exceeds a threshold, the CLI Run Manager shuts down the idle process, wherein the threshold is configured in a declaration file.

[0013] Optionally, the method includes: after receiving a call request, the CLI runtime manager sends input data to the CLI algorithm process through a standard data input channel, wherein the CLI algorithm process loads an executable file to perform model inference based on the input data; the CLI runtime manager receives the model inference results returned by the CLI algorithm process through a standard data output channel.

[0014] Optionally, after the CLI runtime manager receives the call request and sends input data to the CLI algorithm process through the standard data input channel, the method further includes: after receiving the process shutdown instruction, the CLI runtime manager detects whether the CLI algorithm process has completed shutdown within a preset shutdown time; if it has not completed shutdown within the preset shutdown time, the CLI runtime manager forces the CLI algorithm process to terminate.

[0015] Optionally, the declaration file also includes degradation policy information, which includes at least one level of resource degradation options, each degradation option corresponding to a command-line parameter; the method further includes: before initiating an inference call, the CLI runtime manager obtains the available resources of the current device; the CLI runtime manager compares the available resources with the resource requirements declared in the declaration file; when the available resources are less than the declared resource requirements, the CLI runtime manager determines a degradation level that meets the resource constraints from at least one degradation option according to the degradation policy information, and carries the command-line parameter corresponding to the degradation level when starting the executable file, so as to perform model inference according to the resource configuration matched by the degradation level.

[0016] Optionally, before the CLI runtime manager receives a request to invoke a CLI package, the method further includes: parsing different types of executable files and extracting the interface information of the executable files; generating input parameter specifications and output parameter specifications that conform to a preset data format specification based on the interface information; and automatically generating CLI wrapper code based on a predefined code template, input parameter specifications, and output parameter specifications. The CLI wrapper code is used to package the executable files into a CLI package, wherein the CLI package includes a declaration file and an executable file.

[0017] Secondly, embodiments of the present invention provide a running apparatus for a model inference algorithm, comprising: a request unit for receiving a request to invoke a CLI package; a reading unit for reading a declaration file of the CLI package, wherein the CLI package includes one or more executable files, the declaration file includes a lifecycle management strategy and an interaction data format specification for the CLI package, and the executable file is used to perform model inference when run; and an invocation unit for invoking the CLI package according to the declaration file, performing lifecycle management on the executable file according to the lifecycle management strategy and performing model inference according to the interaction data format specification without parsing the executable file.

[0018] This application provides a method for running a model inference algorithm. By introducing a CLI runtime manager and a CLI package containing declaration files and executable files, and setting lifecycle management strategies and interactive data format specifications, algorithms of different programming languages ​​and frameworks can be uniformly scheduled without the need to write dedicated adaptation code for each algorithm. This solves the technical problems of poor adaptability and versatility of existing technologies for various inference algorithms, and achieves the technical effect of providing versatility.

[0019] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0021] Figure 1 This is the overall system architecture diagram of this embodiment;

[0022] Figure 2 This is a flowchart of the operation method of the model inference algorithm according to an embodiment of the present invention;

[0023] Figure 3 This is a flowchart of a degradation decision based on an embodiment of the present invention;

[0024] Figure 4 This is a flowchart of the CLI package according to an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of the running device of the model inference algorithm according to an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0027] It should be understood that in the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. In this embodiment, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0028] This application provides a method for running a model inference algorithm, applicable to model inference algorithms implemented in different programming languages, and deployable on edge devices requiring lightweight computing. This method eliminates the need to write dedicated adaptation code for each algorithm, enabling unified management and scheduling of multiple inference algorithms. It solves the technical problems of poor adaptability and versatility of existing technologies for various inference algorithms, achieving the technical effect of reducing scheduling costs.

[0029] See Figure 1As shown, the overall system architecture provided in this embodiment includes two parts: a cloud-based CLI encapsulation toolchain and an edge device CLI runtime environment. The cloud-based CLI encapsulation toolchain comprises three modules: algorithm source code, CLI encapsulation generator, and CLI package artifacts. The CLI encapsulation generator includes a template engine, a dependency packager, an interface format specification generator, and a test verifier. The algorithm source code can be a Python model, an ONNX model, TensorRT, or a custom algorithm; this application does not restrict the algorithm source code. The CLI package artifacts include executable files, declaration files, interface format specifications, and runtime dependency packages. The edge device CLI runtime environment includes a CLI runtime manager and multiple CLI algorithm processes. The CLI runtime manager includes three sub-modules: process pool management, lifecycle control, and input / output adapters. The multiple CLI algorithm processes can be inference models with different functions, such as object detection models, character recognition models, text analysis models, and speech recognition models. Process pool management includes preset processes, process reuse, and timeout recycling; lifecycle control includes: feasibility file path, startup mode, health check, and graceful shutdown. Input / output adapters include: standard data input (model input), standard data output (structured output), and standard error output (log or error output).

[0030] Figure 2 This is a flowchart illustrating the operation method of the model inference algorithm provided according to an embodiment of the present invention. Figure 2 As shown, the method includes:

[0031] Step S201: The CLI runtime manager receives a request to invoke a CLI package.

[0032] The orchestration engine receives the inference request and sends it to the CLI runtime manager to request the inference request to be invoked by the CLI package corresponding to the inference request.

[0033] In step S203, the CLI runtime manager reads the declaration file of the CLI package. The CLI package includes one or more executable files. The declaration file includes the CLI package's lifecycle management strategy and interactive data format specification. The executable files are used to perform model inference when they are run.

[0034] The declaration document may include the following core content:

[0035] The lifecycle management strategy field declarations include: executable: the relative path to the executable file; startup_mode: the startup mode, supporting both daemon (resident startup mode) and on_demand (on-demand startup mode); warmup_command: the warmup command used for model loading and initialization; health_command: the health check command; inference_timeout_ms: the timeout for a single inference (milliseconds); shutdown_timeout_ms: the graceful shutdown waiting time (milliseconds).

[0036] Resource requirement declaration: This declares the minimum resource configuration required for the algorithm to run, allowing the orchestration engine to make resource pre-allocation and scheduling decisions.

[0037] The interactive data format specification (Schema) includes input interface format specification and output interface format specification fields: defining the type constraints and structural constraints of the input and output.

[0038] The error code mapping (error_codes) field defines the exit code mapping: 0 indicates success, 2 indicates input format error, 3 indicates inference timeout, and 5 indicates insufficient GPU memory.

[0039] The degradation_options field declares three levels of degradation options: level 1 reduces the input resolution to 640x640 to save 256MB of GPU memory, level 2 switches to the YOLOv8-nano lightweight model to save 512MB of GPU memory, and level 3 switches to CPU inference mode to save 1024MB of GPU memory.

[0040] In step S205, the CLI runtime manager calls the CLI package according to the declaration file, performs lifecycle management on the executable file according to the lifecycle management strategy, and performs model inference according to the interactive data format specification without parsing the executable file.

[0041] The CLI runtime manager doesn't need to understand the internal code logic of the executable file (inference algorithm). Instead, it understands how to manage the entire lifecycle of the algorithm by reading a declaration file. The CLI runtime manager only reads the lifecycle management strategy fields defined in the declaration file (such as startup_mode, warmup_command, health_command, shutdown_timeout_ms, etc.). Based on this, the CLI runtime manager simply calls the standardized four-phase CLI interface without parsing the specific implementation details inside the executable file (such as model loading logic, inference engine calls, etc.). Furthermore, the CLI runtime manager doesn't concern itself with the specific details of model inference; it only performs format validation and result pass-through according to the type and structure constraints of the input and output as specified in the interaction data format specification. This protocol-based rather than code-based interaction method makes the CLI runtime manager completely transparent to the algorithm's internals, achieving "zero parsing and zero adaptation of the executable file."

[0042] In summary, this technical solution achieves complete decoupling between "management logic" and "algorithm implementation" through a "declaration + protocol" approach. This allows algorithms from different programming languages ​​and frameworks to be uniformly scheduled without the need to write dedicated adaptation code for each algorithm. It solves the technical problems of poor adaptability and versatility of existing technologies for various inference algorithms, and achieves the technical effect of reducing scheduling costs.

[0043] Optionally, the declaration file includes a resident startup mode declaration. Lifecycle management of the executable file according to the lifecycle management strategy includes: the CLI run manager starts one or more preheating CLI algorithm processes for the executable file in resident startup mode and marks the preheating CLI algorithm processes as ready; when the CLI run manager receives a request to execute model inference, it selects one of the one or more preheating CLI algorithm processes to execute model inference; the CLI run manager keeps the preheating CLI algorithm process that has completed model inference marked as ready so that it can be reused the next time model inference is executed.

[0044] Optionally, when the CLI Run Manager receives a request to execute model inference, it selects one of one or more warm-up CLI algorithm processes to execute model inference, including: the CLI Run Manager obtains a warm-up CLI algorithm process marked as ready from the process pool; if there is no available warm-up CLI algorithm process in the process pool, the CLI Run Manager creates a new process as needed to execute model inference.

[0045] Optionally, after the CLI Run Manager keeps the preheating CLI algorithm process that has completed model inference marked as ready, the method includes: the CLI Run Manager periodically probing the liveness status of the preheating CLI algorithm process and the last usage time of each preheating CLI algorithm process; if the idle time of the preheating CLI algorithm process exceeds a threshold, the CLI Run Manager shuts down the idle process, wherein the threshold is configured in a declaration file.

[0046] Specifically, during system startup, the CLI run manager traverses the preset CLI package deployment paths and reads the declaration file in the root directory of each CLI package. This declaration file declares parameters such as algorithm identification information, startup mode, resource requirements, and timeout configuration. The startup mode includes two types: resident startup mode and on-demand startup mode.

[0047] For algorithms with a persistent startup mode, the CLI run manager calls the algorithm's initialization command to start the corresponding inference process. After the process starts, it continuously outputs status information. When a preset ready flag (such as "READY" or "SERVICE_READY") is output, the CLI run manager determines that the process initialization is complete, marks it as "ready," and adds it to the available process pool. If there are not enough CLI algorithm processes, new processes are created on demand; these new processes are launched in on-demand startup mode.

[0048] Furthermore, during the pre-boot process, the CLI Run Manager also queries the available resources of the current device based on the resource requirements declared in the declaration file (such as the number of CPU cores, memory size, GPU memory, etc.). If the available resources meet the algorithm's requirements, the process starts normally; if resources are insufficient, a resource shortage warning is recorded and the algorithm is skipped to avoid system instability due to resource contention.

[0049] In this embodiment, process pool reuse is implemented using the following mechanism: When an inference request is received, an algorithm process in a ready state is obtained from the process pool.

[0050] Specifically, the CLI Run Manager maintains a process status table. Each record in this table includes at least the following: process PID, algorithm identifier, process status, and last used time. The process status includes at least three states: "Ready" (ready to be used), "Busy" (performing inference), and "Being reclaimed" (being reclaimed).

[0051] Upon receiving an inference request from an external source, the CLI Run Manager first parses the target algorithm identifier carried in the request, and then queries the process status table to find the process list corresponding to that algorithm identifier. If a process with a "ready" status exists, it selects one (e.g., the least recently used process), updates its status to "busy," and assigns the inference task to that process for execution. If no processes are available in the process pool, the CLI Run Manager creates new processes as needed, i.e., it calls the inference command for that algorithm to start a new process instance.

[0052] Optionally, if there are no available processes in the process pool and the current device resources are insufficient to create a new process, the CLI run manager returns an error message indicating insufficient resources, or adds the inference request to the waiting queue, waiting for a process to be released before continuing execution.

[0053] After inference is complete, the process status is updated to "ready," and the process is reused. Specifically, after the process completes the inference task, it outputs the inference result and remains running without actively exiting. Upon receiving the complete inference result, the CLI runtime manager updates the process status from "busy" to "ready" and records the current timestamp as the last used time for subsequent scheduling. In this way, when a new inference request arrives, the pre-warmed process can be reused directly without recreating the process and loading the model, thereby reducing the algorithm call latency from several seconds for cold starts to milliseconds for hot calls.

[0054] Optionally, after the CLI runtime manager receives the call request and sends input data to the CLI algorithm process through the standard data input channel, the method further includes: after receiving the process shutdown instruction, the CLI runtime manager checks whether the CLI algorithm process has completed shutdown within a preset shutdown time; if it has not completed shutdown within the preset shutdown time, the CLI runtime manager forces the CLI algorithm process to terminate.

[0055] In this embodiment, the CLI run manager periodically detects idle processes and reclaims processes that exceed a threshold. The CLI run manager starts a scheduled task to iterate through the last usage time of all processes in the process status table at a preset shutdown time (e.g., every 30 seconds). For processes whose difference between the current time and the last usage time exceeds the idle timeout threshold, the CLI run manager calls the process's shutdown command and performs the following reclamation operations: (1) sends a shutdown signal to the process and waits for the process to exit safely; (2) removes the process from the process status table; and (3) releases the memory and GPU memory resources occupied by the process.

[0056] The default value for the idle timeout threshold is 300 seconds, but it can be configured separately in the declaration file by the timeout field according to the algorithm to adapt to the resource management needs of different scenarios.

[0057] In this embodiment, when the CLI runtime manager passes input data to the CLI algorithm process, it adaptively selects one of the following methods based on the data volume and call frequency:

[0058] (1) Standard data input channel (stdin pipe): When the amount of input data is less than a preset threshold (e.g., 1MB), the data is transmitted through the standard data input channel. This method has the lowest latency and is suitable for transmitting text data, small-sized images, or JSON serialized data.

[0059] (2) File path passing: When the input data volume is large (e.g., high-resolution images, video clips, or batch data files), the data is written to a temporary file, and the file path is passed as a parameter to the algorithm process. This method avoids data truncation or IO blocking problems caused by the pipe buffer size limit.

[0060] (3) Shared memory transfer: When the algorithm is called frequently and the amount of data is large, data references are passed through a shared memory key (shmkey). The CLI runtime manager writes input data into shared memory and passes the shared memory key to the algorithm process; the algorithm process reads the data directly through the shared memory key and returns the shared memory key or output results upon completion. This method avoids the overhead of data copying between processes and is suitable for real-time inference scenarios with high frequency and large data volume.

[0061] The CLI run manager receives the output of the algorithm process, converts the output data into a unified format through an adapter, and then returns it.

[0062] Optionally, the interactive data format specification includes standard data input and standard data output. Performing model inference according to the interactive data format specification includes: after receiving the call request, the CLI run manager sends input data to the CLI algorithm process through the standard data input channel, wherein the CLI algorithm process loads an executable file to perform model inference based on the input data; the CLI run manager receives the model inference results returned by the CLI algorithm process through the standard data output channel.

[0063] To enable model inference without parsing the executable file, the input and output methods are declared in the declaration file. In this embodiment, standard data input, standard data output, and standard error output are declared. Standard data input supports stdin pipe input (--input -), file path input (--input / path / to / file), and shared memory input (--input shm: / / key). Standard data output can output inference results in JSON format. Standard error output can output runtime logs and error messages, which are not mixed with the inference results.

[0064] Optionally, the declaration file also includes degradation policy information, which includes at least one level of resource degradation options, each degradation option corresponding to a command-line parameter; the method further includes: before initiating an inference call, the CLI runtime manager obtains the available resources of the current device; the CLI runtime manager compares the available resources with the resource requirements declared in the declaration file; when the available resources are less than the declared resource requirements, the CLI runtime manager determines a degradation level that meets the resource constraints from at least one degradation option according to the degradation policy information, and carries the command-line parameter corresponding to the degradation level when starting the executable file, so as to perform model inference according to the resource configuration matched by the degradation level.

[0065] When the available resources of the device are insufficient, the orchestration engine uses the command-line parameters carried by the executable file to perform a degradation based on the degradation policy information in the declaration file, so that the resources required by the CLI package after degradation are within the range supported by the available resources of the current device.

[0066] In the following combination Figure 3 This embodiment demonstrates a degradation decision-making process under scenarios where available resources of the device are scarce.

[0067] For example, an edge device has a 4-core CPU, 4GB of memory, and 2GB of GPU memory, and deploys CLI packages for three algorithms: yolo-detect, ocr-engine, and face-detect.

[0068] S301, Receive inference request.

[0069] S302 reads the declaration files of the target CLI packages. During system startup, the CLI Run Manager reads the declaration files of all deployed CLI packages. YOLO-detect requires a 2-core CPU, 512MB of RAM, and 1024MB of GPU memory; OCR-engine requires a 1-core CPU, 256MB of RAM, and 512MB of GPU memory; and Face-detect requires a 1-core CPU, 256MB of RAM, and 512MB of GPU memory. The total resource requirements are a 4-core CPU, 1024MB of RAM, and 2048MB of GPU memory. With sufficient device resources, all components undergo a warm-up boot.

[0070] S303, query the device's currently available resources. During operation, the edge device consumed 500MB of GPU memory due to other system tasks, causing the available GPU memory to drop from 2GB to 1.5GB. The orchestration engine received a new inference request, requiring the execution of the CLI package with the algorithm ID yolo-detect. The currently available GPU memory is 1500MB. The yolo-detect declaration file is read, where the resource requirement is declared as 1024MB.

[0071] S304: Determine if available resources exceed resource requirements. The available resources (1500MB) exceed the resource requirement of 1024MB. However, considering that the ocr-engine and face-detect algorithms are also running, the total requirement is 1024 + 512 + 512 = 2048MB, which is greater than 1500MB, indicating insufficient resources. If yes, proceed to S305 to call the algorithm in default mode. If no, proceed to S306.

[0072] S306, iterate through the YOLO-detect downgrade strategy information. The downgrade strategy is sorted by level from low to high; a larger number indicates a deeper downgrade. Level 1 reduces the resolution to 640x640, saving 256MB of resources. The downgraded resource requirement is 1024-256=768MB, and the total requirement is 768+512+512=1792MB, still greater than 1500MB, so continue iterating. Level 2 switches to YOLOv8-nano, saving 512MB of resources. The downgraded resource requirement is 1024-512=512MB, and the total requirement is 512+512+512=1536MB, which meets the requirement of less than or approximately equal to 1500MB, so level 2 is selected. If downgrading to level 2 still doesn't meet the requirement, continue downgrading to level 3, and so on.

[0073] S307 checks if the resource requirement after downgrading is less than or equal to the available resources. If yes, execute S308 to call the algorithm with command-line arguments for the current level; otherwise, execute S309. The orchestration engine calls the algorithm with command-line arguments for level 2. After receiving the yolov8n parameters, the process unloads the original YOLOv8 standard model, which occupies 1024MB of GPU memory, and loads the YOLOv8-nano lightweight model, which occupies 512MB of GPU memory. The total system GPU memory usage decreases from 2048MB to 1536MB, meeting the available resource constraints, and the system continues to operate normally.

[0074] S309, Determine if there is a higher level? If yes, proceed to S306; otherwise, proceed to S310.

[0075] S310, request denied or queued.

[0076] This embodiment decouples algorithm capabilities from scheduling decisions. By declaring degradation options in the declaration file, there is no need to worry about when degradation will occur. The orchestration engine automatically selects the optimal operating mode based on the real-time resource status, adapting to the dynamic changes in edge device resources.

[0077] Optionally, before the CLI runtime manager receives a request to invoke a CLI package, the method further includes: parsing different types of executable files and extracting the interface information of the executable files; generating input parameter specifications and output parameter specifications that conform to a preset data format specification based on the interface information; and automatically generating CLI wrapper code based on a predefined code template, input parameter specifications, and output parameter specifications. The CLI wrapper code is used to package the executable files into a CLI package, wherein the CLI package includes a declaration file and an executable file.

[0078] The CLI-encapsulated automatic generator executes a six-step process: Step 1 is algorithm analysis. For the Python script yolo_detect.py, the entry function `def detect(image: str, confidence_threshold: float = 0.5)` is parsed using an AST (Abstract Syntax Tree) to extract the parameter names `image` and `confidence_threshold`, the type annotations `str` and `float`, and the default value `0.5`. The import statement is analyzed to determine the dependency package list, including `onnxruntime`, `numpy`, and `opencv-python`. The model loading code segment `session = onnxruntime.InferenceSession("yolov8n.onnx")` is used. For the ONNX model `yolov8n.onnx`, the shape of the input node `images` [1,3,640,640] and the data type `float32` are read, and the shape of the output node `output0` [1,84,8400] and the data type `float32` are read.

[0079] Step two involves automatic schema generation. The Python type annotation `str` is mapped to a JSON schema with the type `string`, and `float` is mapped to a schema with the type `number`. The ONNX input shape [1,3,640,640] is mapped to an array dimension constraint in the schema. The default value of 0.5 is mapped to a schema with the default field set to 0.5. The required parameter `image` is marked as a schema `required` field containing `image`. The automatically generated input schema defines the `image` field as a string with a `base64` format, the `confidence_threshold` field as a number with a minimum of 0, a maximum of 1, a default of 0.5, and the `required` field containing `image`. The output schema defines the `status` field as a string with an enum of either "ok" or "error", the `data` field as a `detections` array, each detection result containing `class`, `confidence`, and `bbox` fields, and the `duration_ms` field as an integer.

[0080] Step 3 involves CLI wrapper code generation, which generates CLI wrapper code based on a predefined template. The command-line argument parsing module uses argparse to parse parameters such as `--mode`, `--input`, and `--resolution`. The input / output adaptation module selects the logic for stdin pipe reading, file path reading, or shared memory reading based on the `--input` parameter. The JSON serialization output module serializes the detection results into the format `{"status": "ok", "data": {"detections": [...]}, "duration_ms": 45}` and outputs it to stdout. The error handling module catches exceptions and maps them to exit codes: exit code 2 for incorrect input format, exit code 3 for inference timeout, and exit code 5 for insufficient GPU memory. The signal handling module registers a SIGTERM handler function; upon receiving a SIGTERM signal, it completes the current inference task, releases resources, and exits. The health check module implements the detection logic of `--modehealth`, checking whether the process is in a normal, serviceable state, and returning exit code 0 to indicate health.

[0081] Step four involves dependency packaging. Using PyInstaller, yolo_detect.py and its dependencies onnxruntime, numpy, and opencv-python are compiled into a standalone executable file . / bin / yolo-detect, containing the Python interpreter and all dependencies. The yolov8n.onnx model file is then packaged to the . / models / directory. A dependency manifest file is generated, listing the required system-level dependencies, including CUDA version 11.8 and glibc version 2.27, for edge device compatibility checks.

[0082] Step five is automatic verification, which automatically executes test cases to verify the completeness of the CLI package functionality. Verification: `--mode init` successfully outputs READY with an exit code of 0; loading the model to the GPU takes approximately 800ms. Verification: `--mode health` returns an exit code of 0, indicating the process is in a normal, serviceable state. Verification: `--mode run` correctly processes the valid input {"image": ""} and returns a result matching the output_schema: {"status": "ok", "data": {"detections": [...]}, "duration_ms": 45}. Verification: `--mode run` correctly rejects the invalid input {"invalid": "data"} and returns error code 2, indicating an incorrect input format. Verification: `--mode shutdown` shuts down normally with an exit code of 0, releasing GPU memory and exiting the process.

[0083] Step six is ​​to output a standard CLI package, generating a CLI package yolo-detect-v1.0 / with a standard directory structure. The bin / directory contains the executable file yolo-detect. The models / directory contains the model file yolov8n.onnx. The lib / directory contains runtime dependency libraries. manifest.json is the declaration file. The schema / directory contains the input.json and output.json input / output schema files. The test / directory contains the sample_input.json and expected_output.json test cases.

[0084] This embodiment enables one-click automatic conversion from algorithm source code or model files to standard CLI packages. Developers do not need to understand the details of the CLI encapsulation protocol; they only need to provide the algorithm source code or model files to generate a standard-compliant CLI package, which greatly reduces the barrier to entry for algorithm submission.

[0085] like Figure 4As shown, this embodiment uses the YOLO object detection algorithm as an example to illustrate the complete CLI encapsulation and calling process. The original algorithm is a YOLOv8 object detection model based on the ONNX format, using a Python inference script.

[0086] The deployment and invocation process for edge devices is as follows:

[0087] System startup preset phase: During system startup, the CLI run manager scans the CLI package deployment directory, reads the declaration file, identifies the startup mode as resident startup mode, and executes the warm-up command. The process loads the YOLOv8 model onto the GPU, which takes approximately 800ms. After loading is complete, it outputs a "ready" flag to the standard data, and the process remains resident, waiting for subsequent inference instructions. The CLI run manager marks the process as ready and adds it to the available process pool.

[0088] Inference invocation phase:

[0089] The orchestration engine receives a call request with the algorithm ID "yolo-detect" and input parameters including an image and a `confidence_threshold` field. The CLI run manager searches the declaration file and verifies if the input conforms to the `input_schema`. It requests an available process from the process pool, obtains the ready "yolo-detect" process, and sends the input data via standard data input (stdin). The CLI algorithm process receives the stdin input, performs model inference, outputs the inference result in JSON format via standard data output (stdout), and returns the result to the CLI run manager. The CLI run manager verifies if the output conforms to the `output_schema`, returns the inference result to the orchestration engine, marks the process as ready for reuse in future calls.

[0090] Exception handling phase:

[0091] The CLI Run Manager periodically executes health check commands to determine if a process is in a healthy, serviceable state, returning an exit code of 0 to indicate a healthy state. The graceful shutdown process is as follows: the CLI Run Manager executes a graceful shutdown command; after completing the current inference task, the process releases GPU memory, closes file handles, cleans up temporary files, and then exits the process. If the process times out after completion without shutting down, an error is returned, and the process is forcibly terminated.

[0092] This embodiment realizes a complete encapsulation process from the original ONNX model and Python inference script to a standard CLI package. The generated CLI package includes an executable file, a declaration file, and an input / output interface format specification file. It supports a four-stage lifecycle interface, supports multi-level degradation options, and achieves complete decoupling of algorithm capabilities and scheduling logic.

[0093] This example demonstrates the coordinated scheduling of multiple algorithms.

[0094] When the system starts, the CLI Run Manager scans the CLI package deployment directory / opt / cli-packages / and finds three CLI packages: yolo-detect, ocr-engine, and face-detect. It reads the manifest.json file of each CLI package and identifies that yolo-detect and face-detect have a startup_mode of daemon, requiring warm-up startup; while ocr-engine has a startup_mode of on_demand, requiring on-demand startup.

[0095] The `. / bin / yolo-detect --mode init` command is executed on yolo-detect to warm up the process. The process loads the YOLOv8 model onto the GPU, which takes approximately 800ms, and outputs a READY flag. The CLI run manager marks the process as ready, adds it to the available process pool, and records the PID as 1234, the algorithm ID as yolo-detect, the status as ready, and the last usage time as the system startup time. The `. / bin / face-detect --mode init` command is executed on face-detect to warm up the process. The process loads the face detection model onto the GPU, which takes approximately 600ms, and outputs a READY flag. The CLI run manager marks the process as ready, adds it to the available process pool, and records the PID as 1235, the algorithm ID as face-detect, the status as ready, and the last usage time as the system startup time.

[0096] According to the resource declaration, the device has 4 CPU cores, 4GB of RAM, and 2GB of GPU memory. YOLO-detect requires 2 CPU cores, 512MB of RAM, and 1024MB of GPU memory, while face-detect requires 1 CPU core, 256MB of RAM, and 512MB of GPU memory. The total requirements are 3 CPU cores, 768MB of RAM, and 1536MB of GPU memory. The device resources are sufficient; warm-up was successful.

[0097] The orchestration engine receives the first inference request with the algorithm ID yolo-detect. The CLI run manager searches the process pool for the process with the algorithm ID yolo-detect and a ready status, finding PID 1234. Input data is sent to PID 1234 via the stdin standard data input channel: `echo '{"image": ""}' | . / bin / yolo-detect --moderun --input -`. The process executes inference, taking approximately 45ms, and returns the result via stdout: `{"status": "ok", "data": {"detections": [...]}, "duration_ms": 45}`. After inference is complete, the CLI run manager marks PID 1234 as ready and updates the last used time to the current time.

[0098] The orchestration engine receives a second inference request with the algorithm ID yolo-detect. The CLI run manager again retrieves PID 1234 from the process pool, reusing the same process to avoid the overhead of repeatedly creating processes and loading models. Input data is sent to PID 1234 via stdin, the process executes inference, which takes approximately 45ms, and the result is returned via stdout. After inference is complete, PID 1234 is marked as ready, and the last used time is updated to the current time.

[0099] The CLI run manager periodically checks the last usage time of each process in the process pool every 60 seconds. It finds that the last usage time of PID 1234 exceeds 300 seconds, classifying it as a long-term idle process. The command `. / bin / yolo-detect --mode shutdown` is sent to PID 1234. After completing its current inference task, the process releases 1024MB of GPU memory and exits. The CLI run manager removes the record for PID 1234 from the process pool, and the freed GPU memory can be used by other algorithms.

[0100] This embodiment implements a process pool preheating and reuse mechanism, reducing algorithm call latency from seconds during cold starts to milliseconds during hot starts. Through a timeout recycling mechanism, processes that have been idle for extended periods automatically release resources, improving resource utilization.

[0101] This application also provides a device for running a model inference algorithm. For example... Figure 5 As shown, the operating device includes:

[0102] Request unit 501 is used to receive a request to invoke a CLI package.

[0103] The reading unit 503 is used to read the declaration file of the CLI package, wherein the CLI package includes one or more executable files, the declaration file includes the lifecycle management strategy and interaction data format specification of the CLI package, and the executable file is used to perform model inference when it runs.

[0104] The calling unit 505 is used to call the CLI package according to the declaration file, and to perform lifecycle management of the executable file according to the lifecycle management strategy and to perform model inference according to the interaction data format specification without parsing the executable file.

[0105] It is understood that the declaration file format in the above embodiments is not limited to JSON; YAML, TOML, and other formats can also be used. For example, the command-line parameters for the four-stage lifecycle interface of the declaration file are not limited to `--mode`; subcommands such as `init` and `run` can also be used. The input / output schema format is not limited to JSON Schema; Protocol Buffers, Apache Avro, and other formats can also be used. The inter-process communication method is not limited to standard data input / standard data output; Unix Domain Socket, Named Pipe, and other methods can also be used. The triggering method for degradation options is not limited to command-line parameter flags; environment variables, configuration files, and other methods can also be used. The CLI-encapsulated automatic generator supports algorithm types not limited to Python / ONNX / TensorRT; it can also be extended to support TorchScript, TFLite, OpenVINO, and other formats.

[0106] In practical applications, the inference algorithm running method provided in this application embodiment can be applied to various electronic devices, such as personal computers, servers, mobile phones and other devices with data processing capabilities. In one implementation, in order to improve processing efficiency, the inference algorithm running method provided in this application embodiment can be applied to high-performance computers.

[0107] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 602, communication interface, and memory 601 communicating with each other via the communication bus.

[0108] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0109] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0110] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0111] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0112] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the running method of any of the above-described inference algorithms.

[0113] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the running method of any of the inference algorithms in the above embodiments.

[0114] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0116] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer storage media, and computer program products are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0117] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for running a model inference algorithm, characterized in that, The method includes: The CLI Run Manager receives requests to invoke CLI packages; The CLI runtime manager reads the declaration file of the CLI package, wherein the CLI package includes one or more executable files, the declaration file includes the lifecycle management strategy and interaction data format specification of the CLI package, and the executable file is used to perform model inference when it runs; The CLI runtime manager calls the CLI package according to the declaration file, performs lifecycle management on the executable file according to the lifecycle management strategy, and performs model inference according to the interaction data format specification without parsing the executable file.

2. The method according to claim 1, characterized in that, The declaration file includes the resource requirements for running the executable file, and the lifecycle management of the executable file according to the lifecycle management strategy includes: The CLI runtime manager reads the declaration file to obtain the resource requirements and allocates available resources to the executable file according to the resource requirements.

3. The method according to claim 1, characterized in that, The declaration file includes a resident startup mode declaration, and the lifecycle management of the executable file according to the lifecycle management strategy includes: The CLI run manager starts one or more preheating CLI algorithm processes for the executable file in resident startup mode, and marks the preheating CLI algorithm processes as ready. When the CLI run manager receives a request to execute model inference, it selects one of the one or more warm-up CLI algorithm processes to execute model inference. The CLI run manager keeps the preheated CLI algorithm process that has completed model inference marked as ready so that it can be reused the next time model inference is executed.

4. The method according to claim 3, characterized in that, When the CLI run manager receives a request to execute model inference, it selects one of the one or more warm-up CLI algorithm processes to execute model inference, including: The CLI run manager obtains warm-up CLI algorithm processes marked as ready from the process pool; If the process pool does not have available preheating CLI algorithm processes, the CLI run manager creates new processes on demand to perform model inference.

5. The method according to claim 3, characterized in that, After the CLI run manager keeps the preheating CLI algorithm process that has completed model inference marked as ready, the method includes: The CLI run manager periodically detects the liveness status of the preheating CLI algorithm processes and the last usage time of each preheating CLI algorithm process; If the idle time of the preheating CLI algorithm process exceeds a threshold, the CLI run manager shuts down the idle process, wherein the threshold is configured in the declaration file.

6. The method according to claim 1, characterized in that, The interactive data format specification includes standard data input and standard data output. Performing model inference according to the interactive data format specification includes: After receiving the call request, the CLI runtime manager sends input data to the CLI algorithm process through the standard data input channel. The CLI algorithm process loads the executable file to perform model inference based on the input data. The CLI run manager receives the model inference results returned by the CLI algorithm process through a standard data output channel.

7. The method according to claim 6, characterized in that, After the CLI runtime manager receives the call request and sends input data to the CLI algorithm process through the standard data input channel, the method further includes: After receiving the process shutdown command, the CLI run manager checks whether the CLI algorithm process has completed shutdown within the preset shutdown time; If the shutdown is not completed within the preset shutdown time, the CLI run manager will force the CLI algorithm process to terminate.

8. The method according to claim 1, characterized in that, The declaration file also includes degradation policy information, which includes at least one level of resource degradation options, each degradation option corresponding to a command-line parameter; the method further includes: Before initiating an inference call, the CLI runtime manager obtains the available resources of the current device; The CLI runtime manager compares the available resources with the resource requirements declared in the declaration file; When the available resources are less than the declared resource requirements, the CLI runtime manager determines a degradation level that meets the resource constraints from the at least one degradation option based on the degradation policy information, and carries the command line parameters corresponding to the degradation level when starting the executable file, so as to perform model inference according to the resource configuration matched by the degradation level.

9. The method according to claim 1, characterized in that, Before the CLI runtime manager receives a request to invoke a CLI package, the method further includes: Parse different types of executable files and extract the interface information of the executable files; Based on the interface information, input parameter specifications and output parameter specifications conforming to the preset data format specifications are generated; CLI wrapper code is automatically generated based on a predefined code template, the input parameter specification, and the output parameter specification. The CLI wrapper code is used to package the executable file into a CLI package, wherein the CLI package includes the declaration file and the executable file.

10. A device for running a model inference algorithm, characterized in that, The device includes: The request unit is used to receive requests to invoke CLI packages; A reading unit is used to read the declaration file of a CLI package, wherein the CLI package includes one or more executable files, the declaration file includes the lifecycle management strategy and interaction data format specification of the CLI package, and the executable file is used to perform model inference when it runs; The invocation unit is used to invoke the CLI package according to the declaration file, and to perform lifecycle management of the executable file according to the lifecycle management strategy and to perform model inference according to the interaction data format specification without parsing the executable file.

Images