A somatic intelligent model cross-platform deployment method, device and medium

Abstract

The application discloses a body intelligent model cross-platform deployment method and device and medium, and relates to the field of body intelligence, and comprises the following steps: in response to a received model deployment instruction, obtaining a deep learning model to be deployed specified by the model deployment instruction; converting the deep learning model into a unified intermediate representation graph; performing hardware-independent optimization on the unified intermediate representation graph to obtain a first optimized intermediate representation graph; obtaining hardware characteristic information corresponding to a target inference backend, performing hardware-related optimization on the first optimized intermediate representation graph based on the hardware characteristic information to obtain a second optimized intermediate representation graph; and generating an executable inference engine supported by the target inference backend based on the second optimized intermediate representation graph. Through the model conversion, hardware-independent optimization, hardware-related optimization and engine generation steps executed in sequence, the application converts an artificial adaptation process into a fully automatic compilation pipeline, and effectively shortens the deployment period of the body intelligent model on multiple heterogeneous computing units.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a method, device and medium for cross-platform deployment of embodied intelligence models. Background Technology

[0002] In the development of embodied intelligent agents, machine vision is heavily relied upon for environmental perception, and it often needs to be combined with physical models to achieve accurate simulation and control. A neural network model with integrated machine vision capabilities, trained using a physical model in a simulation environment, ultimately needs to be deployed to run on various heterogeneous computing units of the physical robot, such as cloud-based graphics processors for high-level decision-making, edge-side embedded neural network processors for real-time visual inference, and microcontrollers for low-level motion control. These computing units each have their own independent inference frameworks and operating environments, and their underlying sets of supported operators, memory models, and computational paradigms differ significantly, making the cross-platform migration of complex graph models involving physical constraints extremely technically complex.

[0003] Currently, in existing technologies, engineers typically perform model adaptation for each target hardware platform manually, one by one. This usually involves converting the machine vision network and physical model computation graph into a proprietary format required by the specific inference framework, and manually performing optimization work such as graph optimization, operator replacement, and memory layout adjustment based on the characteristics of the target chip. Furthermore, in this process, engineering teams have to repeatedly implement a large number of similar but fragmented adaptation processes for each hardware combination. As the types of computing units required to support embodied intelligent products continue to increase, this discrete adaptation mode, which relies heavily on human experience and manual operation, makes the entire deployment cycle lengthy, severely slowing down the iteration speed of products from virtual simulation to physical entity deployment. Summary of the Invention

[0004] This invention provides a method, device, and medium for cross-platform deployment of embodied intelligent models, aiming to solve the technical problem of how to provide a solution that can effectively shorten the deployment cycle of embodied intelligent models on multiple computing units.

[0005] In a first aspect, the present invention provides a method for cross-platform deployment of embodied intelligent models, comprising: In response to a received model deployment instruction, obtain the deep learning model to be deployed specified in the model deployment instruction; The deep learning model is converted into a unified intermediate representation graph; Hardware-independent optimization is performed on the unified intermediate representation graph to obtain a first optimized intermediate representation graph; Obtain the hardware characteristic information corresponding to the target inference backend, and perform hardware-related optimization on the first optimized intermediate representation graph based on the hardware characteristic information to obtain the second optimized intermediate representation graph; Based on the second optimized intermediate representation graph, an executable inference engine supported by the target inference backend is generated.

[0006] Optionally, converting the deep learning model into a unified intermediate representation graph includes: The source training framework format of the deep learning model is parsed, and the original computation graph is extracted. The operators in the original computation graph are mapped to a preset unified operator set to obtain an operator mapping graph; In the operator mapping graph, at least one dimension of each tensor is marked as a dynamic variable. The specific value of the dynamic variable is determined during inference. A unified intermediate representation graph is constructed, which includes the mapped operators in the operator mapping graph and their connection relationships.

[0007] Optionally, performing hardware-independent optimization on the unified intermediate representation graph to obtain a first optimized intermediate representation graph includes: Traverse the unified intermediate representation graph, identify and remove operators that are not dependent on any output, and complete dead code elimination; The operators in the unified intermediate representation graph after dead code elimination that contain only constant inputs are evaluated, and operators that meet the conditions are replaced with constant nodes to complete constant folding. Multiple consecutive operators that satisfy a preset fusion mode in the unified intermediate representation graph after constant folding are merged into a single equivalent operator to complete operator fusion and obtain the first optimized intermediate representation graph.

[0008] Optionally, the deep learning model includes a machine vision sub-network and a physical model computation graph; the step of performing hardware-related optimization on the first optimized intermediate representation graph based on the hardware characteristic information to obtain the second optimized intermediate representation graph includes: Identify the computational boundary between the machine vision sub-network in the first optimized intermediate representation diagram and the physical model computation graph to obtain boundary description information; Based on the boundary description information and the at least two heterogeneous computing units included in the target inference backend, the first optimized intermediate representation graph is divided into multiple subgraphs that correspond one-to-one with the at least two heterogeneous computing units. Perform operator replacement and memory layout transformation on each of the plurality of subgraphs respectively to obtain the subgraphs that are related to the plurality of subgraphs. Figure 1 A plurality of corresponding optimization subgraphs are used as the second optimization intermediate representation graph; The process of generating the executable inference engine supported by the target inference backend includes: compiling the multiple optimized subgraphs into multiple inference engines suitable for the corresponding heterogeneous computing units.

[0009] Optionally, operator replacement is performed on each of the plurality of subgraphs, including: For each subgraph, the hardware characteristic information is parsed to obtain the first set of operators not supported by the heterogeneous computing unit corresponding to the subgraph and the second set of operators supported by it; Traverse the subgraph to identify ill-fitting operators belonging to the first set of operators; By using at least one combination of operators from the second set of operators to replace the ill-fitting operators, an operator-replaced subgraph corresponding to the subgraph is obtained.

[0010] Optionally, a memory layout transformation is performed on each of the plurality of subgraphs, including: For each operator replacement subgraph, the memory alignment constraints and data arrangement preferences of the heterogeneous computing units corresponding to the operator replacement subgraph are extracted from the hardware characteristic information; Analyze the access patterns of the input and output tensors of each operator in the operator replacement subgraph; A tensor rearrangement operation is inserted into the operator replacement subgraph to make the tensor arrangement after the tensor rearrangement operation conform to the memory alignment constraint and data arrangement preference, thereby obtaining an optimized subgraph corresponding to the operator replacement subgraph.

[0011] Optionally, generating the executable inference engine supported by the target inference backend based on the second optimized intermediate representation graph includes: If the second optimized intermediate representation graph contains multiple optimized subgraphs, then each optimized subgraph is used as the graph to be processed; otherwise, the second optimized intermediate representation graph is used as the graph to be processed. For each graph to be processed, determine the target heterogeneous computing unit corresponding to the graph to be processed; Select the corresponding serialization template based on the target heterogeneous computing unit; Traverse the graph to be processed, convert each operator in the graph to a graph operator description defined by the target heterogeneous computing unit, and convert each tensor to a data description defined by the target heterogeneous computing unit; Using the serialization template, the graph operator description, and the data description, an inference engine corresponding to the graph to be processed is generated.

[0012] Optionally, converting each operator in the graph to be processed into a graph operator description defined by the target heterogeneous computing unit includes: Query the preset operator mapping table to obtain the native operator type of the target heterogeneous computing unit corresponding to the operator in the graph to be processed; Based on the hardware characteristic information, set the running attribute parameters for the native operator type; Generate a graph operator description that includes the native operator type and the runtime attribute parameters.

[0013] Secondly, the present invention also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0014] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0015] This invention provides a method, device, and medium for cross-platform deployment of embodied intelligent models. By sequentially executing model conversion, hardware-independent optimization, hardware-dependent optimization, and engine generation steps, the manual adaptation process is transformed into a fully automated compilation pipeline, effectively shortening the deployment cycle of embodied intelligent models on multiple heterogeneous computing units. Specifically, by converting the deep learning model into a unified intermediate representation graph, the format differences between different training frameworks and the target inference backend are masked, providing a unified operation object for cross-platform reuse of subsequent optimization operations. Further, based on this, hardware-independent optimization is performed on the unified intermediate representation graph to obtain a first optimized intermediate representation graph. This optimization does not depend on specific hardware, achieving one-time development and multi-platform sharing of optimization logic. Further, hardware characteristic information corresponding to the target inference backend is obtained, and hardware-dependent optimization is performed on the first optimized intermediate representation graph based on the hardware characteristic information to obtain a second optimized intermediate representation graph, thereby automatically completing adaptation and tuning for specific hardware and avoiding the efficiency bottleneck caused by manual hardware-by-hardware tuning. Further, based on the second optimized intermediate representation graph, an executable inference engine supported by the target inference backend is generated, completing a fully automated conversion from model to deployment package. The above steps form a closed-loop data processing chain, transforming discrete manual adaptation into standardized automated production, significantly shortening the cycle from model development to deployment on multiple hardware platforms. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1A flowchart illustrating a cross-platform deployment method for an embodied intelligent model provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0020] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0023] Please see Figure 1This invention provides a method for cross-platform deployment of embodied intelligent models. This method can be specifically implemented as a model compilation toolchain running on a server or cloud computing instance. This method aims to systematically solve the problem in existing technologies where deploying the same deep learning model to multiple heterogeneous hardware platforms requires extensive repetitive manual adaptation due to the incompatibility of the inference backends on each platform. In an embodied intelligent system, a complete intelligent agent, such as an autonomous navigation robot, typically has its core functions completed collaboratively by two deep learning models: one is a machine vision sub-network used to process sensor data from cameras, LiDAR, etc., to achieve environmental perception; the other is a physical model computation graph used to describe the robot's kinematics and dynamics and to perform motion planning and control based on the perception results. In actual deployment, these models often need to run on different types of computing units on the robot itself, such as high-performance graphics processors for vision processing and low-power embedded neural network processors for real-time control. However, each computing unit has its own dedicated inference backend, such as TensorRT (NVIDIA Inference Engine), MNN (Alibaba Edge Inference Engine), TFLite (TensorFlow Lite Edge Inference Engine), or CANN (Huawei Ascend Computing Language), which differ significantly in operator support, memory layout, and serialization format. In traditional workflows, developers must manually perform model conversion, graph optimization, operator adaptation, and performance tuning for each model-to-backend combination. This process is like manual mode, extremely inefficient, and highly dependent on expert hardware knowledge. The method provided in this embodiment fundamentally changes this situation by building a fully automated compilation pipeline that includes a unified intermediate representation, a hierarchical optimizer, and a multi-backend code generator, achieving "one-time model import, automatic optimization and deployment across multiple platforms." Specifically, the method includes the following steps: S1, in response to the received model deployment instruction, obtain the deep learning model to be deployed specified by the model deployment instruction.

[0024] In practice, in response to a received model deployment instruction, the deep learning model to be deployed, specified by the instruction, is obtained. The model deployment instruction is a task description submitted by the user through a command-line interface or visual configuration interface provided by the compilation toolchain. Its core parameters specify the storage path of the model to be processed, its source training framework format, and one or more target inference backends.

[0025] Furthermore, after receiving the instruction, the toolchain's front-end interface will parse the various parameters and read the deep learning model file to be deployed from the specified file system or object storage according to the storage path. This file can be in a standard format exported by frameworks such as PyTorch or TensorFlow.

[0026] Furthermore, the deep learning model mentioned here specifically refers to composite models that integrate a machine vision sub-network and a physical model computation graph. The machine vision sub-network is a deep neural network used to process visual sensor data to perform perception tasks such as object detection and semantic segmentation; the physical model computation graph is a computational graph representation based on physical laws, used to describe the kinematic and dynamic characteristics of the robot's interaction with its environment, and used to perform physical simulation and motion planning. This invention does not specifically limit the scope of the model.

[0027] S2, convert the deep learning model into a unified intermediate representation graph.

[0028] In practice, the deep learning model is converted into a unified intermediate representation graph. This step is the cornerstone of the entire automated process, aiming to build a standardized model representation independent of any specific training framework or inference backend, providing a unified and reusable foundation for all subsequent optimization and compilation operations. To achieve this, the toolchain incorporates multiple front-end parsers. This step first calls the corresponding front-end parser according to the source training framework format specified in the model deployment instructions to parse the file structure of the deep learning model and extract the original computation graph. Further, the original computation graph is a topological structure composed of nodes representing mathematical operation units and directed edges representing tensor data flow.

[0029] Furthermore, the toolchain maps each operator in the original computation graph to a fully functional, pre-defined unified operator set according to predefined mapping rules, resulting in an operator mapping graph. The unified operator set is the core abstraction of the toolchain; it defines a set of standard, hardware-independent operator interfaces and semantics, covering most of the computational primitives required by AI models, such as convolution, fully connected, activation, pooling, and normalization.

[0030] Furthermore, a unified intermediate representation graph is constructed based on the operator mapping graph. This unified intermediate representation graph contains all mapped operator instances from the operator mapping graph, the connections between operators, and the hyperparameters and weights associated with each operator. This unified intermediate representation graph serves as the sole data source for all subsequent optimization and compilation operations.

[0031] In some preferred embodiments, converting the deep learning model into a unified intermediate representation graph includes: parsing the source training framework format of the deep learning model and extracting the original computation graph; mapping the operators in the original computation graph to a preset unified operator set to obtain an operator mapping graph; in the operator mapping graph, marking at least one dimension of each tensor as a dynamic variable, the specific value of the dynamic variable being determined during inference, and constructing a unified intermediate representation graph, wherein the unified intermediate representation graph contains the mapped operators and their connection relationships in the operator mapping graph.

[0032] In practical implementation, dynamic changes in model input size are common rather than exceptional in embodied intelligence applications. For example, the resolution of the input image in a machine vision sub-network used for object detection may dynamically switch between 640x480 and 1920x1080 depending on the camera settings or task requirements; the number of rigid bodies to be simulated in a physical model computation graph may also change in each inference cycle during multibody dynamics simulation. An inference engine that fixes all tensor shapes at compile time cannot adapt to this dynamism, causing expensive model recompilation to be triggered every time the input size changes, which is unacceptable on resource-constrained edge devices that require real-time response. To address this, this embodiment explicitly introduces a dynamic variable labeling mechanism during the model transformation stage and incorporates this information throughout the construction of a unified intermediate representation graph, enabling the generated IR to inherently express and process dynamic shapes. This lays the foundation for the subsequent generation of a resilient inference engine that supports dynamic input. The specific implementation is as follows: First, the model compilation toolchain parses the source training framework format of the deep learning model and extracts the original computational graph. The source training framework format refers to the original machine learning platform used by the user to train the model, such as PyTorch or TensorFlow. The toolchain recovers the topology of the computational graph from the model file by calling a front-end parser that matches the format.

[0033] Furthermore, the operators in the original computation graph are mapped to a preset unified operator set to obtain an operator mapping graph. This process converts framework-specific operators into unified, toolchain-recognizable standard operators, providing a consistent interface for subsequent processing. The mapping process may involve one-to-one direct conversion or one-to-many operator decomposition. It should be noted that the preset unified operator set is a set of standard, hardware-independent operator interfaces and semantics predefined within the toolchain, which is not specifically limited in this invention.

[0034] Furthermore, in the operator mapping graph, at least one dimension of each tensor is marked as a dynamic variable, constructing a unified intermediate representation graph. This is the core of achieving dynamic shape support. After obtaining the operator mapping graph, the toolchain, combining any additional hints provided by the user in the deployment instructions and information analyzed from the model structure, identifies which dimensions of which tensors in the computation graph should be dynamically determined at inference time. Common scenarios include the 0th dimension representing the data batch size, the height and width dimensions representing the image spatial resolution, and the dimension representing the sequence length. For these identified dimensions, the toolchain does not write a specific numerical value in their size description field, but instead writes a specific enumeration value, which is defined as a dynamic variable internally by the toolchain. The specific value of the dynamic variable is determined at inference time; that is, when the model engine is actually invoked and input data is fed in, its dimension value is obtained from the shape of the input data, and the memory allocation and kernel startup parameters of the entire computation graph are dynamically configured accordingly. Furthermore, after completing the dynamic marking of all necessary dimensions, the toolchain constructs the final unified intermediate representation graph based on this operator mapping graph containing dynamic dimension information. The unified intermediate representation graph not only includes the mapped operators and their connection relationships in the operator mapping graph, but also completely saves metadata information about dynamic dimensions.

[0035] This embodiment utilizes a specific technique of introducing dynamic variable labeling into the tensor dimensions of the unified intermediate representation graph, endowing the lower-level intermediate representation with the ability to model variable-shaped inputs. This capability is not implemented through upper-level application logic but is embedded in the IR structure, thus seamlessly penetrating subsequent optimization and code generation processes. This allows the final executable inference engine to natively support dynamic shapes. Furthermore, during inference, the engine can adaptively and dynamically adjust the computation graph based on the actual input image resolution or the number of simulated objects, completely avoiding the costly overhead of online model recompilation due to changes in input shape. This feature is crucial for embodied intelligent devices deployed in dynamically changing environments with limited battery and computing resources. It enables a single compilation to produce a general-purpose inference engine that flexibly adapts to various input scenarios, greatly improving the robustness and operational efficiency of the deployment scheme.

[0036] S3, perform hardware-independent optimization on the unified intermediate representation graph to obtain the first optimized intermediate representation graph.

[0037] In specific implementation, hardware-independent optimization is performed on the unified intermediate representation graph to obtain a first optimized intermediate representation graph. Hardware-independent optimization refers to general performance improvement methods that are based solely on the structure and logic of the computation graph itself, without considering specific hardware characteristics. Furthermore, the toolchain implements a pluggable optimization pass manager and includes a series of standard optimization passes. First, a dead code elimination pass is executed. This pass analyzes the data dependencies in the unified intermediate representation graph starting from the output nodes, marking all active nodes that are effectively dependent on the output. Any unmarked operator node and its connected tensor edges are considered invalid computation branches and will be safely removed from the graph. Next, a constant folding pass is executed. The constant folding pass scans the unified intermediate representation graph after dead code elimination, searching for operator nodes whose inputs are all constants. Further, this pass immediately executes the operator's operation through a built-in compile-time interpreter, obtaining a definite constant result, and encapsulates this result into a new constant node to replace the original operator node, thereby eliminating unnecessary runtime computation. Finally, an operator fusion pass is executed. The operator fusion pass performs pattern matching on the unified intermediate representation graph after constant folding. Based on a rule base containing various preset fusion patterns (such as convolution + batch normalization + activation function), it searches for consecutive operator combinations in the graph that can be merged. Furthermore, when a specific pattern is matched, these consecutive operators are replaced as a single, functionally equivalent but computationally more efficient fusion operator. This fusion operation significantly reduces bandwidth bottlenecks and kernel startup overhead caused by repeated data transfer between GPU memory and compute cores when running on hardware such as GPUs. After executing this series of optimization passes, the optimized graph becomes the first optimized intermediate representation graph.

[0038] In some preferred embodiments, performing hardware-independent optimization on the unified intermediate representation graph to obtain a first optimized intermediate representation graph includes: traversing the unified intermediate representation graph, identifying and removing operators that are not dependent on any output to complete dead code elimination; evaluating operators in the unified intermediate representation graph after dead code elimination that contain only constant inputs, replacing operators that meet the conditions with constant nodes to complete constant folding; and merging multiple consecutive operators in the unified intermediate representation graph after constant folding that meet a preset fusion mode into a single equivalent operator to complete operator fusion and obtain the first optimized intermediate representation graph.

[0039] In specific implementation, this embodiment defines a strictly progressive and ordered sequence of sub-steps, clarifying the complete data flow and transformation process from the original unified intermediate representation graph to the first optimized intermediate representation graph, thus ensuring the determinism and optimality of the optimization process. The specific implementation method is as follows: First, the model compilation toolchain traverses the unified intermediate representation graph, performing dead code elimination. Starting from the final output node of the graph, the toolchain recursively traces and marks all operator nodes that contribute to the final output. The tracing process crosses all data edges and control edges, ensuring that nodes on all active paths are marked. After marking, a scan of all nodes in the entire graph is performed; any unmarked operator node is considered dead code, and it, along with its exclusive output tensor edge that has no subsequent data flow, is safely removed from the graph. This step completes the first round of cleanup of invalid computations.

[0040] Furthermore, constant folding is performed on the unified intermediate representation graph after dead code elimination. Following the simplification in the previous step, the toolchain traverses all operator nodes in the current graph. For each node, it checks the source of all its input tensors. If it finds that all inputs to an operator come from constant nodes, it calls a lightweight compile-time evaluator to perform the computation of that operator, resulting in a new constant tensor. Subsequently, the operator node is deleted, and all data edges that originally pointed to that operator are redirected to a newly created constant node that directly outputs the computation result. This step solidifies computations that can be determined at compile time, reducing runtime computation instructions.

[0041] Furthermore, operator fusion is performed on the unified intermediate representation graph after constant folding. The constant folding in the previous step may have created new, more direct adjacent operator patterns. The toolchain traverses the graph again, using a rule base containing multiple preset fusion patterns (such as Conv+BN+ReLU) for pattern matching. When a specific pattern is matched, the toolchain replaces these three consecutive operator nodes with a single, equivalent operator that internally implements composite computational logic. After completing all possible pattern matching and replacement, the toolchain marks the computation graph in memory, which has undergone three rounds of ordered transformations, as the final output, i.e., the first optimized intermediate representation graph.

[0042] This embodiment constructs a cascaded optimization pipeline with mutual benefits by specifying a strict order for dead code elimination, constant folding, and operator fusion. First, dead code elimination narrows the search range for subsequent stages; then, constant folding optimizes the computational logic through algebraic simplification and may expose new dead code that can be eliminated due to constant propagation or new operator combinations that can be fused; finally, operator fusion implements the most critical hardware utilization improvement on the simplified graph structure. Furthermore, this ordered design is not a simple listing but leverages the coupling effect between optimization techniques to ensure that the hardware-independent optimization stage can systematically and progressively compress the model's complexity from three dimensions: logical simplification, algebraic simplification, and physical execution efficiency, providing a higher-performance starting point for subsequent hardware-related optimizations.

[0043] S4, obtain the hardware characteristic information corresponding to the target inference backend, and perform hardware-related optimization on the first optimized intermediate representation graph based on the hardware characteristic information to obtain the second optimized intermediate representation graph.

[0044] In practice, hardware characteristic information corresponding to the target inference backend is obtained. Based on this hardware characteristic information, hardware-related optimizations are performed on the first optimized intermediate representation graph to obtain a second optimized intermediate representation graph. Hardware-related optimization is a key step in maximizing performance by leveraging the unique architectural advantages of the target hardware, and its optimization decisions are highly dependent on hardware characteristic information. Hardware characteristic information is a structured dataset that details the technical specifications and programming constraints of the hardware corresponding to the target inference backend, such as a list of operators natively supported and unsupported by the hardware, performance recommendations for specific operators, optimal memory data layout preferences, memory alignment requirements, and the capacity and bandwidth of multi-level caches. Further, the toolchain first loads the corresponding hardware characteristic information from its internally maintained hardware characteristic information library according to the target inference backend specified in the deployment instructions. Further, a series of hardware-related optimization passes are executed. For example, the specific operator replacement pass will traverse the first optimized intermediate representation graph based on the list of unsupported operators in the hardware characteristic information to identify operators that the hardware cannot directly execute. For these incompatible operators, this pass uses one or more basic operators natively supported by the hardware for functionally equivalent replacement, forming a new operator subgraph that can be directly executed on the hardware. For example, the memory layout transformation pass analyzes the memory access patterns of the input and output tensors of each operator in the graph based on memory alignment constraints and data arrangement preferences in the hardware characteristic information. Without changing the computational semantics, it inserts explicit tensor data rearrangement operations at appropriate positions in the graph, converting the tensor organization in memory to the format preferred by the hardware, thereby maximizing memory bandwidth utilization. After a series of deep optimizations specific to the hardware, the result is the second optimized intermediate representation graph.

[0045] In some preferred embodiments, the deep learning model includes a machine vision sub-network and a physical model computation graph; the step of performing hardware-related optimization on the first optimized intermediate representation graph based on the hardware characteristic information to obtain a second optimized intermediate representation graph includes: identifying the computational boundary between the machine vision sub-network and the physical model computation graph in the first optimized intermediate representation graph to obtain boundary description information; dividing the first optimized intermediate representation graph into multiple subgraphs corresponding one-to-one with the at least two heterogeneous computing units included in the target inference backend according to the boundary description information; and performing operator replacement and memory layout transformation on each of the multiple subgraphs to obtain a second optimized intermediate representation graph corresponding to the at least two heterogeneous computing units. Figure 1A plurality of corresponding optimized subgraphs are used as the second optimized intermediate representation graph; the generation of the executable inference engine supported by the target inference backend includes: compiling the plurality of optimized subgraphs into a plurality of inference engines suitable for the corresponding heterogeneous computing units.

[0046] In specific implementation, the deep learning model explicitly includes two logically independent components: a machine vision subnetwork and a physical model computation graph. The machine vision subnetwork typically consists of numerous highly parallel operators such as convolution and pooling, making it well-suited for running on throughput-priority hardware like cloud graphics processors. The physical model computation graph, on the other hand, contains numerous scalar and sparse matrix operations that dynamically change with simulation states and have complex data dependencies, making it more efficient on embedded neural network processors optimized for sequential control and low-latency physical interactions. Furthermore, when the target inference backend itself is a composite hardware platform consisting of at least two heterogeneous computing units (e.g., a graphics processor and a neural network processor), compiling and deploying the model as a single unit would fail to fully utilize the strengths of each hardware unit, resulting in an overall performance bottleneck. Therefore, this embodiment introduces mechanisms for computational boundary recognition, subgraph segmentation, and independent optimized compilation, solving for the first time the technical challenge of adaptively and parallelly deploying heterogeneous models onto multiple heterogeneous computing units. The specific implementation is as follows: First, the computational boundaries between the machine vision subnetwork and the physical model computation graph in the first optimized intermediate representation graph are identified. The toolchain utilizes graph-level labeling information that may be carried during model transformation, or through a built-in computational feature-based analyzer, to perform structural analysis on the first optimized intermediate representation graph. For example, the analyzer can identify regions of the machine vision subnetwork by recognizing large sequences of 2D convolution and pooling operations, and regions of the physical model computation graph by recognizing specific operator sequences related to physical parameters and rigid body transformations. The interface nodes for data exchange between the two are identified and marked as computational boundaries. This process ultimately generates a structured boundary description, precisely recording which operator nodes belong to which subnetwork, and the cross-boundary tensor connecting the two subnetworks.

[0047] Furthermore, based on the boundary description information and the at least two heterogeneous computing units included in the target inference backend, the first optimized intermediate representation graph is divided into multiple subgraphs corresponding one-to-one with the at least two heterogeneous computing units. The toolchain performs graph cutting operations based on the cutting points recorded in the boundary description information. The cutting strategy is to intelligently match the computational characteristics of the subgraphs with the hardware characteristics of the computing units. For example, the system will divide the set of nodes corresponding to the computationally intensive machine vision subnetwork into an independent subgraph, denoted as the first subgraph, and allocate it to the graphics processor; at the same time, the set of nodes corresponding to the computational graph of the physical model with complex control logic and latency sensitivity will be divided into another independent subgraph, denoted as the second subgraph, and allocated to the neural network processor. This matching decision is automatically completed by the scheduler inside the toolchain according to preset rules.

[0048] Furthermore, operator replacement and memory layout transformation are performed on each of the plurality of subgraphs respectively to obtain the subgraphs that are consistent with the plurality of subgraphs. Figure 1 Multiple optimized subgraphs are corresponding to each other, and these multiple optimized subgraphs serve as the second optimized intermediate representation graph. The toolchain no longer uses a single set of hardware characteristic information to optimize the entire model. Instead, it loads the corresponding hardware characteristic information for each target computing unit to be deployed in each subgraph. For example, for the first subgraph, the hardware characteristic information of the graphics processor is loaded, and operator replacement and memory alignment optimizations are performed for that hardware to obtain the first optimized subgraph. Further, for the second subgraph, the hardware characteristic information of the neural network processor is loaded, and independent optimization is performed to obtain the second optimized subgraph. This set of optimized subgraphs together constitutes the final output of the hardware-related optimization steps, namely the second optimized intermediate representation graph.

[0049] Furthermore, correspondingly, the step of generating the executable inference engine supported by the target inference backend is limited in this specific case to: compiling the multiple optimization subgraphs into multiple inference engines suitable for the corresponding heterogeneous computing units. The toolchain's code generator will start a compilation task for each optimization subgraph, calling its corresponding backend compiler to generate independent engine files, for example, generating a first inference engine for the first optimization subgraph and a second inference engine for the second optimization subgraph. The runtime scheduler will distribute data to the corresponding inference engine for execution according to the type of inference request.

[0050] Furthermore, in some preferred embodiments, the generated multiple independent inference engines are bound to a unified data exchange pipeline; the data exchange pipeline is used to transfer tensor data across boundaries between inference engines corresponding to different heterogeneous computing units through shared memory or direct memory access (DMA) to reduce cross-engine data transfer latency.

[0051] This embodiment achieves intelligent mapping from model structure to hardware topology by automatically identifying the computational boundaries within heterogeneous models and decoupling a complete model into multiple semantically independent optimized subgraphs suitable for different hardware characteristics, breaking the traditional deployment paradigm of "one model, one engine." Furthermore, this mechanism enables different types of workloads within the embodied intelligent agent, such as visual processing and physical simulation, to be automatically distributed and deployed to heterogeneous computing units best suited to their computational characteristics for parallel execution, achieving heterogeneous collaboration and maximized utilization of computing resources. Furthermore, the subsequent independent and customized optimization process for each subgraph and its target hardware ensures that each sub-engine can fully exploit the ultimate performance of its corresponding hardware, avoiding conservative optimizations made to accommodate general-purpose hardware. Finally, the final solution of generating multiple independent inference engines also increases the modularity and flexibility of the system architecture; upgrading or repairing a single sub-engine does not affect other parts, greatly improving the deployment and operation efficiency of complex embodied intelligent systems.

[0052] In some preferred embodiments, operator replacement is performed on each of the plurality of subgraphs, including: for each subgraph, parsing the hardware characteristic information to obtain a first set of operators not supported by the heterogeneous computing unit corresponding to the subgraph and a second set of operators supported; traversing the subgraph to identify unsuitable operators belonging to the first set of operators; and replacing the unsuitable operators with at least one combination of operators from the second set of operators to obtain an operator-replaced subgraph corresponding to the subgraph.

[0053] In practice, when a model is divided into multiple subgraphs and allocated to different hardware, a prominent contradiction arises: the operators contained in a subgraph may not be fully compatible with the heterogeneous computing unit on which it will be deployed. For example, a highly efficient fusion operator optimized for a specific visual task on a graphics processing unit (GPU) may not have a direct implementation in the operator library of a neural network processor. Manually searching for such incompatible operators for each subgraph and writing equivalent alternative implementations is an extremely tedious, time-consuming, and error-prone task, requiring developers to be proficient in the operator libraries of multiple hardware platforms. Therefore, this embodiment provides a systematic method based on hardware capability set comparison and automatic replacement, completely solving the problem of automated cross-platform adaptation of operators. The specific implementation is as follows: The operator replacement mechanism of the model compilation toolchain will perform a loop processing flow for each subgraph obtained from the segmentation. Take the processing of a single subgraph as an example.

[0054] First, the hardware characteristic information is parsed to obtain the first set of operators not supported and the second set of operators supported by the heterogeneous computing unit corresponding to the subgraph. The hardware characteristic information describes the capability boundaries of the computing unit in detail. The toolchain parses this information and extracts two key lists. One is a list of native operators directly exposed and optimized by the computing unit through its software stack, which developers are advised to use first, i.e., the supported second set of operators. The other is a list of operator types that are explicitly unsupported or will have functional or performance defects when running in the current version of the software stack, i.e., the unsupported first set of operators. These two sets together define the goals and no-go zones for operator adaptation.

[0055] Further, the subgraph is traversed to identify unsuitable operators belonging to the first set of operators. The toolchain examines each operator node in the subgraph one by one, comparing its operator type with the types in the unsupported first set of operators. Once a match is found, the operator is determined to be an unsuitable operator. For example, if a complex multi-branch activation function operator exists in the subgraph, and this operator happens to be in the unsupported list of the target neural network processor, it is marked as an unsuitable operator.

[0056] Furthermore, by using at least one combination of operators from the second set of operators to replace the mismatched operators, an operator-replaced subgraph corresponding to the subgraph is obtained. To address the mismatch issue, the toolchain maintains an operator equivalence transformation knowledge base. This knowledge base predefines one or more functionally equivalent subgraph implementation schemes based on the basic operators in the supported second set of operators for each operator that may appear in the unsupported list. For the aforementioned mismatched operators, the toolchain searches the knowledge base to find the corresponding equivalent scheme. This scheme may be a small-scale computational subgraph formed by connecting multiple basic operators (such as multiplication, addition, sigmoid, etc.) from the second set of operators according to a specific topological structure. The toolchain instantiates this equivalent subgraph in memory and directly replaces the original mismatched operator nodes with it, correctly connecting their input and output tensor edges. After replacing all the mismatched operators in the subgraph, a new subgraph that can be natively supported on the target computational unit is obtained, namely the operator-replaced subgraph.

[0057] This embodiment formalizes hardware capabilities into two clearly defined sets of operators: those that support and those that do not. It also constructs an automatic conversion knowledge base from complex or proprietary operators to standard operator combinations, completely resolving the core operator compatibility issue in cross-hardware platform deployment. Furthermore, this method transforms the complex process—which previously relied on developers' personal experience and required in-depth consultation of different hardware development manuals and manual code porting—into a fully automated, traceable, and reproducible graph rewriting process. Moreover, it not only significantly improves the efficiency of operator adaptation but, more importantly, by using standardized and rigorously validated operator combinations to replace incompatible operators, it ensures the functional consistency and robustness of the adapted model across different hardware, fundamentally eliminating the hidden defects easily introduced during manual porting.

[0058] In some preferred embodiments, memory layout transformation is performed on each of the plurality of subgraphs, including: for each operator replacement subgraph, extracting the memory alignment constraints and data arrangement preferences of the heterogeneous computing units corresponding to the operator replacement subgraph from the hardware characteristic information; analyzing the access patterns of the input and output tensors of each operator in the operator replacement subgraph; inserting a tensor rearrangement operation in the operator replacement subgraph so that the tensor arrangement after the tensor rearrangement operation conforms to the memory alignment constraints and data arrangement preferences, thereby obtaining an optimized subgraph corresponding to the operator replacement subgraph.

[0059] In practice, after solving the executable problem at the operator level, the performance bottleneck often shifts to data handling. Different hardware architectures have drastically different optimal memory layouts. For example, for the same four-dimensional tensor, NCHW (channel-first layout, i.e., batch, channel, height, width) is the fastest for access on a graphics processor, but on some embedded neural network processors, NHWC (channel-last layout, i.e., batch, height, width, channel) maximizes the utilization of its internal dedicated convolutional computation units. If a subgraph that functions correctly after operator replacement has a tensor data layout that does not match the preferences of the target hardware, the hardware will automatically perform implicit, time-consuming data rearrangement on each computation, resulting in actual inference performance far below the theoretical peak. To address this, this embodiment explicitly analyzes and inserts necessary tensor rearrangement operations in the graph, ensuring that the data always exists in the way the hardware is best suited for processing during runtime, thereby eliminating implicit conversion overhead and unlocking the full performance potential of the hardware. The specific implementation is as follows: The memory layout transformation mechanism of the model compilation toolchain will process each operator substitution subgraph obtained in Example 5 one by one. Taking the processing of one operator substitution subgraph as an example.

[0060] First, the memory alignment constraints and data layout preferences of the heterogeneous computing units corresponding to the operator replacement subgraph are extracted from the hardware characteristic information. The hardware characteristic information details the best practices for the memory subsystem of this computing unit. For example, it may stipulate that the starting address of the input tensor of all convolution operators must be aligned to a 64-byte boundary, otherwise it will trigger inefficient alignment exception handling of the direct memory access engine; and it explicitly recommends that the tensor data layout preference is NCHW format to obtain the best first-level data cache line utilization. The toolchain first extracts these key constraint and preference parameters.

[0061] Furthermore, the access patterns of the input and output tensors of each operator in the subgraph are analyzed. The toolchain performs a complete traversal of the subgraph, maintaining a currently inferred data arrangement format attribute for each tensor edge in the graph, and analyzes the sensitivity of each operator to its input tensor format. Some operators (such as element-wise addition) are insensitive to the input format, while some computationally intensive operators (such as convolution) are extremely sensitive to the input format; mismatched formats will lead to a significant decrease in their internal computational efficiency. The goal of the analysis is to accurately identify all incompatibilities between the output format of the previous operator and the preferred input format of the next sensitive operator.

[0062] Furthermore, tensor rearrangement operations are inserted into the operator replacement subgraph. For each incompatibility point identified in the above analysis, the toolchain accurately inserts a tensor rearrangement operator on the incompatible tensor data flow path. This operator is a single-function but highly efficient operator whose sole purpose is to transparently transform the input tensor from one data arrangement (e.g., NCHW) to another (e.g., NHWC) at runtime without changing any numerical content. The insertion position is optimized to connect the preceding and following operators with minimal data migration cost. After all necessary tensor rearrangement operations are inserted, the toolchain saves the modified graph so that the final tensor arrangement conforms to the memory alignment constraints and data arrangement preferences, thus obtaining an optimized subgraph corresponding to the operator replacement subgraph. This optimized subgraph is not only functionally executable but also achieves optimal memory access efficiency for the hardware.

[0063] This embodiment fully automates critical memory performance optimization in deep learning by using hardware memory layout preferences and alignment constraints as explicit inputs for compilation optimization and automatically performing graph-level memory access pattern analysis and rearrangement operator insertion. Furthermore, this mechanism ensures that the data flow within a model deployed on any hardware has been reorganized by the compilation toolchain in a manner most compatible with that hardware, eliminating computational unit waiting and implicit data migration overhead caused by data layout mismatches. Moreover, this allows the hardware to operate at near full capacity, significantly improving inference throughput and response speed. This is a key technical guarantee that ensures the solution achieves theoretically optimal performance across different platforms, rather than merely achieving functional usability.

[0064] S5. Based on the second optimized intermediate representation graph, generate the executable inference engine supported by the target inference backend.

[0065] In practice, based on the second optimized intermediate representation graph, an executable inference engine supported by the target inference backend is generated. This is the final step in translating the optimized abstract graph representation into a concrete runnable file. The toolchain includes code generators for various mainstream inference backends. The code generator selects the corresponding model serialization template and code generation logic based on the identifier of the target inference backend. It traverses the second optimized intermediate representation graph, translating each operator in the graph into a specific operator description object defined by the target backend, and serializing its topology, connectivity, and weight tensor data into a specific format that the target inference backend can directly load and execute, generating an executable inference engine. For example, if the target backend is TensorRT, an execution plan file in .trt format is generated; if the target backend is MNN, its specific model file is generated. This engine can be deployed on corresponding hardware to run high-performance AI inference tasks.

[0066] This embodiment successfully establishes a robust isolation layer between various training frameworks and inference backends by introducing a unified intermediate representation as a core abstraction layer. This allows differences in upstream frameworks to be absorbed by the front-end parser, while differences in downstream hardware are shielded by the back-end code generator. Furthermore, this architecture enables hardware-independent optimization techniques above the intermediate representation layer, such as operator fusion and constant folding, to have unprecedented reusability. A single development effort can benefit all backends, fundamentally solving the problem of repetitive development of optimization logic in multi-platform adaptation. Furthermore, hardware-specific optimization techniques such as operator replacement and memory layout transformation are distilled into automated, rule-driven optimization passes. This makes knowledge previously confined to expert experience explicit and tool-based, allowing ordinary developers to achieve expert-level inference performance with a single click, significantly lowering the technical barrier to multi-platform deployment without requiring in-depth understanding of the chip's underlying architecture. Finally, the entire process from model to engine is constructed as a fully automated compilation pipeline, transforming the traditional discrete, manual work mode into a standardized, automated production mode. This greatly shortens the cycle from model development to multi-platform deployment while ensuring optimal inference performance on each platform.

[0067] In some preferred embodiments, generating the executable inference engine supported by the target inference backend based on the second optimized intermediate representation graph includes: if the second optimized intermediate representation graph contains multiple optimized subgraphs, then each optimized subgraph is used as a graph to be processed; otherwise, the second optimized intermediate representation graph is used as a graph to be processed; for each graph to be processed, a target heterogeneous computing unit corresponding to the graph to be processed is determined; according to the target heterogeneous computing unit, a corresponding serialization template is selected; the graph to be processed is traversed, and each operator in the graph to be processed is converted into a graph operator description defined by the target heterogeneous computing unit, and each tensor is converted into a data description defined by the target heterogeneous computing unit; using the serialization template, the graph operator description, and the data description, an inference engine corresponding to the graph to be processed is generated.

[0068] In practice, the second optimized intermediate representation graph produced by the hardware-related optimization steps in the entire compilation pipeline may present two distinct forms depending on the type of deployment task: for a regular single-end deployment, it is a single, optimized intermediate representation graph; for a heterogeneous multi-end deployment as described in the above embodiment, it is a logical set containing multiple optimized subgraphs. Traditional code generators typically can only handle single graph inputs and cannot cope with this structural uncertainty. This embodiment introduces a pre-construction structural judgment and unified dispatch logic during the generation stage, enabling the same generation framework to handle any scenario without distinction, achieving truly universal compilation. The specific implementation method is as follows: Before starting its work, the code generator performs a judgment operation. If the second optimized intermediate representation graph contains multiple optimized subgraphs, it indicates that this is the product of a heterogeneous deployment scenario. The code generator will decompose the set and treat each optimized subgraph in the set as an independent graph to be processed. Each graph to be processed carries label information of the specific heterogeneous computing unit to which it should be deployed. Otherwise, if the second optimized intermediate representation graph is a single, unified graph without any subgraph structure, it indicates a conventional deployment scenario. The code generator directly treats this entire second optimized intermediate representation graph as a single graph to be processed.

[0069] Furthermore, the code generator enters a loop, performing the exact same compilation process for each graph to be processed. First, it determines the target heterogeneous computing unit corresponding to the graph. In a typical scenario, this target computing unit is the inference backend and its corresponding hardware architecture specified in the user deployment instructions. In a heterogeneous scenario, this target computing unit is the specific computing unit (such as a specified graphics processor or neural network processor) that has been bound to the current graph during the partitioning and optimization phases. This determination process ensures that subsequent compilation generates code for the correct hardware target.

[0070] Furthermore, after determining the target, the code generator selects the corresponding serialization template based on the target heterogeneous computing unit. Internally, the toolchain pre-defines a set of serialization templates for each supported hardware backend. This template is a programming language data structure representation of the target inference backend format (such as protobuf, flatbuffers), defining how the nodes, tensors, attributes, and metadata of the model graph should be organized and encoded.

[0071] Furthermore, the graph to be processed is traversed, and a two-way parallel translation process is performed. One path converts each operator in the graph to be processed into a graph operator description defined by the target heterogeneous computing unit. That is, based on the target backend's application programming interface, the corresponding operator description object is instantiated and its type, parameters, input / output tensor bindings, and other information are filled in. The other path converts each tensor into a data description defined by the target heterogeneous computing unit. That is, tensor shapes, data types, and constant data buffers conforming to the target backend format are generated.

[0072] Furthermore, using the serialization template, the filled graph operator description set, and the data description set, an inference engine corresponding to the graph to be processed is generated. The serialization engine of the code generator starts, serializing these description objects and the original data buffer into a continuous binary byte stream according to the hierarchy and order defined in the template, and writes this byte stream to a disk file. This file can then be directly loaded and executed by the inference runtime of the target hardware.

[0073] This embodiment successfully unifies single-backend deployment and complex multi-subgraph heterogeneous deployment scenarios at the code generation level by introducing a pre-judgment and dispatch logic for the input graph structure during the code generation stage. This design significantly improves the versatility and robustness of the compilation toolchain, ensuring high consistency in its internal interfaces. Regardless of the output of the upstream optimization process, the downstream generator can respond correctly. Furthermore, the subsequent independent compilation process for each graph to be processed includes complete steps such as template selection, operator translation, data transformation, and serialization. This ensures that each generated inference engine is a highly optimized, self-contained entity for its specific runtime environment, guaranteeing the independence and optimality of each engine in multi-terminal deployment.

[0074] In some preferred embodiments, converting each operator in the graph to be processed into a graph operator description defined by the target heterogeneous computing unit includes: querying a preset operator mapping table to obtain the native operator type of the target heterogeneous computing unit corresponding to the operator in the graph to be processed; setting runtime attribute parameters for the native operator type according to the hardware characteristic information; and generating a graph operator description containing the native operator type and the runtime attribute parameters.

[0075] In practice, the most crucial and complex step in generating the inference engine is accurately mapping and instantiating the abstract operators defined in the unified intermediate representation layer into concrete operator description objects that the target inference backend can understand. Different inference backends define rich, hardware-dependent operator variants and runtime attributes to leverage their unique hardware capabilities. A simple convolution operation might correspond to a dozen different operator enumeration values ​​and dozens of configuration parameters on a given backend, depending on its precision, acceleration library preferences, and memory reuse strategies. Incorrect mapping or inappropriate attribute settings can lead to the generated engine failing to run or exhibiting poor performance. Therefore, this embodiment solves this problem of accurate mapping and optimized configuration by introducing a maintainable mapping table and an automatic attribute configuration mechanism based on hardware characteristics. The specific implementation is as follows: When transforming a specific operator in the graph to be processed, the process first queries a pre-defined operator mapping table to obtain the native operator type of the target heterogeneous computing unit corresponding to the operator in the graph. The operator mapping table is a large mapping database stored in file format. Its core is a key-value pair data structure, where the key is a standard operator type defined in a unified operator set (e.g., 'Conv2D'), and the value is a lookup table dependent on the target inference backend. This lookup table then maps the operator's specific parameters (e.g., kernel size, stride, number of groups) to one or more of the most suitable native operator enumeration values ​​or function construction interfaces specific to that backend. The toolchain uses the standard type and parameters of the current operator to be processed as the query key to search the operator mapping table and obtain the corresponding native operator type of the target inference backend.

[0076] Furthermore, after identifying the native operator type, runtime attribute parameters are set for that type based on the hardware characteristic information. These runtime attribute parameters are configuration items related to hardware behavior tuning and are not at the mathematical semantic level. The toolchain analyzes the current hardware characteristic information to obtain best practice recommendations for this type of operator. For example, based on hardware characteristic information, for a convolution operator on a graphics processing unit, runtime attributes might include specific algorithm enumeration values ​​for the recommended CUDA deep neural network acceleration library, whether to enable tensor kernels for mixed-precision computation, and workspace memory size limits. The toolchain will set an optimal set of runtime attribute parameters for the obtained native operator type instance based on these preset rules.

[0077] Further, a graph operator description containing the native operator type and the runtime attribute parameters is generated. The toolchain instantiates an operator description object defined in the target inference backend application programming interface, assigns the retrieved native operator type enumeration value to the type field of the description object, assigns the set runtime attribute parameters to their corresponding attribute fields, and fills in other basic parameters (such as the weight data pointer of the convolution kernel, padding, and stride). At this point, a general operator is completely and accurately converted into a native graph operator description of the target backend that can be directly used for code generation and serialization.

[0078] This embodiment completely decouples compiler logic from hardware knowledge by extracting the complex, many-to-many operator mapping relationships from the core logic of the compiler and storing and managing them in an independent operator mapping table. Furthermore, this design allows support for new hardware backends or new operator versions to be achieved simply by updating the mapping table data, greatly enhancing the scalability and maintainability of the toolchain and avoiding frequent modifications to the compiler's core code due to hardware iterations. Moreover, by automatically setting runtime attributes for operators based on hardware characteristic information and solidifying hardware expert tuning experience into automatic configuration rules, each generated operator instance is embedded with the optimal execution configuration for specific hardware. This not only ensures the correct translation of model functionality but also completes deep performance tuning during the code generation stage, ensuring that the final generated inference engine can fully unleash the maximum potential of the target hardware.

[0079] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0080] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0081] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0082] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to execute a cross-platform deployment method for an embodied intelligence model.

[0083] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0084] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a cross-platform deployment method for an embodied intelligence model.

[0085] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0086] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of a cross-platform deployment method for an embodied intelligent model provided in any of the above method embodiments.

[0087] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0088] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0089] Therefore, the present invention also provides a storage medium. This storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of a cross-platform deployment method for an embodied intelligence model provided in any of the above method embodiments.

[0090] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0091] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0092] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0093] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0094] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0095] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0096] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, this invention is also intended to include these modifications and variations as long as they fall within the scope of the claims of this invention and their equivalents.

[0097] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for cross-platform deployment of embodied intelligence models, characterized in that, include: In response to a received model deployment instruction, obtain the deep learning model to be deployed specified in the model deployment instruction; The deep learning model is converted into a unified intermediate representation graph; Hardware-independent optimization is performed on the unified intermediate representation graph to obtain a first optimized intermediate representation graph; Obtain the hardware characteristic information corresponding to the target inference backend, and perform hardware-related optimization on the first optimized intermediate representation graph based on the hardware characteristic information to obtain the second optimized intermediate representation graph; Based on the second optimized intermediate representation graph, an executable inference engine supported by the target inference backend is generated.

2. The method for cross-platform deployment of embodied intelligent models according to claim 1, characterized in that, The step of converting the deep learning model into a unified intermediate representation graph includes: The source training framework format of the deep learning model is parsed, and the original computation graph is extracted. The operators in the original computation graph are mapped to a preset unified operator set to obtain an operator mapping graph; In the operator mapping graph, at least one dimension of each tensor is marked as a dynamic variable. The specific value of the dynamic variable is determined during inference. A unified intermediate representation graph is constructed, which includes the mapped operators in the operator mapping graph and their connection relationships.

3. The method for cross-platform deployment of embodied intelligent models according to claim 1, characterized in that, The step of performing hardware-independent optimization on the unified intermediate representation graph to obtain a first optimized intermediate representation graph includes: Traverse the unified intermediate representation graph, identify and remove operators that are not dependent on any output, and complete dead code elimination; The operators in the unified intermediate representation graph after dead code elimination that contain only constant inputs are evaluated, and operators that meet the conditions are replaced with constant nodes to complete constant folding. Multiple consecutive operators that satisfy a preset fusion mode in the unified intermediate representation graph after constant folding are merged into a single equivalent operator to complete operator fusion and obtain the first optimized intermediate representation graph.

4. The method for cross-platform deployment of embodied intelligent models according to claim 1, characterized in that, The deep learning model includes a machine vision sub-network and a physical model computation graph; the hardware-related optimization of the first optimized intermediate representation graph based on the hardware characteristic information to obtain the second optimized intermediate representation graph includes: Identify the computational boundary between the machine vision sub-network in the first optimized intermediate representation diagram and the physical model computation graph to obtain boundary description information; Based on the boundary description information and the at least two heterogeneous computing units included in the target inference backend, the first optimized intermediate representation graph is divided into multiple subgraphs that correspond one-to-one with the at least two heterogeneous computing units. Operator replacement and memory layout transformation are performed on each of the multiple subgraphs respectively to obtain multiple optimized subgraphs that correspond one-to-one with the multiple subgraphs, and the multiple optimized subgraphs are used as the second optimized intermediate representation graph; The process of generating the executable inference engine supported by the target inference backend includes: compiling the multiple optimized subgraphs into multiple inference engines suitable for the corresponding heterogeneous computing units.

5. The method for cross-platform deployment of embodied intelligent models according to claim 4, characterized in that, Perform operator replacement on each of the plurality of subgraphs, including: For each subgraph, the hardware characteristic information is parsed to obtain the first set of operators not supported by the heterogeneous computing unit corresponding to the subgraph and the second set of operators supported by it; Traverse the subgraph to identify ill-fitting operators belonging to the first set of operators; By using at least one combination of operators from the second set of operators to replace the ill-fitting operators, an operator-replaced subgraph corresponding to the subgraph is obtained.

6. The method for cross-platform deployment of embodied intelligent models according to claim 5, characterized in that, Perform memory layout transformations on each of the plurality of subgraphs, including: For each operator replacement subgraph, the memory alignment constraints and data arrangement preferences of the heterogeneous computing units corresponding to the operator replacement subgraph are extracted from the hardware characteristic information; Analyze the access patterns of the input and output tensors of each operator in the operator replacement subgraph; A tensor rearrangement operation is inserted into the operator replacement subgraph to make the tensor arrangement after the tensor rearrangement operation conform to the memory alignment constraint and data arrangement preference, thereby obtaining an optimized subgraph corresponding to the operator replacement subgraph.

7. The method for cross-platform deployment of embodied intelligent models according to claim 1, characterized in that, The step of generating the executable inference engine supported by the target inference backend based on the second optimized intermediate representation graph includes: If the second optimized intermediate representation graph contains multiple optimized subgraphs, then each optimized subgraph is used as the graph to be processed; otherwise, the second optimized intermediate representation graph is used as the graph to be processed. For each graph to be processed, determine the target heterogeneous computing unit corresponding to the graph to be processed; Select the corresponding serialization template based on the target heterogeneous computing unit; Traverse the graph to be processed, convert each operator in the graph to a graph operator description defined by the target heterogeneous computing unit, and convert each tensor to a data description defined by the target heterogeneous computing unit; Using the serialization template, the graph operator description, and the data description, an inference engine corresponding to the graph to be processed is generated.

8. The method for cross-platform deployment of embodied intelligent models according to claim 7, characterized in that, The step of converting each operator in the graph to be processed into a graph operator description defined by the target heterogeneous computing unit includes: Query the preset operator mapping table to obtain the native operator type of the target heterogeneous computing unit corresponding to the operator in the graph to be processed; Based on the hardware characteristic information, set the running attribute parameters for the native operator type; Generate a graph operator description that includes the native operator type and the runtime attribute parameters.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images