A heterogeneous compatible unified runtime system and method

Abstract

The application relates to a heterogeneous compatible unified runtime system and method; in the system, a multi-language adaptation module uniformly converts received running call instructions into an intermediate representation; a task arrangement module performs semantic structure analysis based on the semantic structure and constraint conditions of the intermediate representation, and constructs a standardized task graph description; a chip sensing module evaluates the capability and resource state of available heterogeneous chips in real time, and generates an evaluation vector; a scheduling execution module is used for selecting an optimal heterogeneous chip and a scheduling strategy, and sending task execution instructions to the selected heterogeneous chip; a model management module loads, registers, caches and unloads compatible models of various heterogeneous chips, supports compatible conversion, compatibility preprocessing and runtime dynamic switching among the various heterogeneous chips; and a unified runtime interface module encapsulates a bottom layer execution result into a unified runtime interface corresponding to the intermediate representation format, and maps the unified runtime interface into an original upper layer language call environment.

Description

Technical Field

[0001] This invention relates to the field of computer system architecture and program execution environment technology, and in particular to a heterogeneous compatible unified runtime system and method. Background Technology

[0002] With the widespread application of various intelligent chips (such as those from Cambricon, Rockchip, and others) and the parallel use of multiple programming languages ​​in different application scenarios, heterogeneous computing environments have become a typical feature of information infrastructure. Against this backdrop, how to build a unified runtime platform that efficiently and stably supports the collaborative execution of programs written in multiple source languages ​​on processors with different architectures and instruction sets has become a core challenge in system software design.

[0003] Current mainstream runtime systems are mostly optimized for specific programming languages ​​or hardware architectures, lacking a unified abstraction mechanism for language diversity and hardware heterogeneity. For example, while traditional virtual machines (such as the JVM) have some cross-platform capabilities, their execution models are deeply tied to specific language ecosystems, making it difficult to natively support efficient collaboration between system-level and dynamic languages ​​such as C / C++, Python, and Rust in a unified runtime environment. Furthermore, existing runtime systems generally lack sufficient support for processor architectures (such as LoongArch and RISC-V), typically requiring additional porting and adaptation layers, leading to complex deployment, high maintenance costs, and significant runtime overhead.

[0004] Currently, application software needs to have stronger portability and cross-platform adaptability. However, when dealing with multi-instruction set architectures (such as ARM, x86, and LoongArch), existing technologies mainly rely on compile-time static adaptation and lack flexible runtime dynamic compatibility mechanisms, making it difficult to meet the needs of applications for rapid deployment, seamless migration, and consistent execution in heterogeneous environments.

[0005] For example, CN105975261A discloses a runtime system for unified interface calls, including a "runtime system agent" device, a "runtime system" device, a "function interface server" device, a "channel" device, and a "network" device. The runtime system's operation method includes: registering functions from a "third-party function component library"; the "application" loading the "runtime system agent"; the "runtime system agent" forwarding the function to be called by the "application" to the "runtime system"; the "runtime system" finding the corresponding "third-party function component library" based on the function interface information, executing the function instance, and sending the execution result back to the "runtime system agent"; and the "runtime system agent" returning the result to the "application". This technical solution only solves the unified call problem at the interface level and does not address the dynamic compatibility requirements under cross-instruction set architectures. Its operation relies on native function libraries that must exist and be compatible on the target platform, meaning that each platform still needs to independently compile and deploy adapted versions, essentially belonging to a static binding mechanism. This technical solution lacks awareness of the underlying hardware architecture, cannot dynamically schedule based on chip type, load, or computing power status, does not support unified abstraction and intermediate representation conversion of multi-language semantics, and lacks the ability to dynamically migrate models or code between different instruction sets such as ARM, x86, and LoongArch. Therefore, CN105975261A cannot provide effective technical support for the needs of rapid deployment and seamless migration in heterogeneous environments.

[0006] For example, CN117648211A discloses a unified runtime interface, server, and invocation method for an artificial intelligence framework. This unified runtime interface includes a framework interface and a basic runtime interface. The basic runtime interface connects to the runtime interface of the board / device. The framework interface indicates the interfaces included in the unified runtime interface to the artificial intelligence framework and, upon receiving a call request from the artificial intelligence framework, invokes the runtime interface of the board / device through the basic runtime interface. While this technical solution improves the consistency of AI applications' access to hardware to some extent, it is essentially still an adaptation layer built on top of the native runtimes of various manufacturers (such as CANN, CUDA, etc.). These underlying runtimes themselves have strong platform dependencies and instruction set binding characteristics. This technology does not achieve abstraction and decoupling of the underlying hardware and lacks a real-time chip load, memory status, and model support capability awareness mechanism, thus failing to perform intelligent scheduling and resource optimization. Furthermore, it does not support format conversion, cache management, and dynamic switching of models across different heterogeneous computing chips, nor can it handle the unified execution of non-AI scenarios or multi-language mixed tasks. Because it lacks an intermediate representation or language-independent execution environment, its applicability is limited to specific AI frameworks, making it difficult to meet the requirement of consistent execution across architectures in general computing scenarios. Therefore, although CN117648211A has made some progress in interface standardization, it has failed to overcome the technical bottleneck of runtime dynamic compatibility and cannot solve the core defects targeted by this invention.

[0007] As mentioned above, there is an urgent need to build a general-purpose, efficient unified runtime system that can provide language-independent execution abstractions and effectively shield underlying hardware differences through modular design and virtualization technology, enabling unified scheduling and highly consistent execution of multi-language programs on heterogeneous platforms. To this end, this invention proposes a heterogeneous compatible unified runtime system based on language virtualization, aiming to overcome the limitations of existing technologies in terms of the breadth of language support and hardware adaptation flexibility, and to provide highly available and highly consistent runtime support for applications in complex heterogeneous computing environments.

[0008] Furthermore, on the one hand, there are differences in understanding among those skilled in the art; on the other hand, the applicant studied a large number of documents and patents when making this invention, but due to space limitations, not all details and contents were listed in detail. However, this does not mean that the present invention does not possess the features of these prior art. On the contrary, the present invention already possesses all the features of the prior art, and the applicant reserves the right to add relevant prior art to the background art. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a heterogeneous compatible unified runtime system from a first aspect. The system includes a processor, which comprises a multi-language adaptation module, a task orchestration module, a chip awareness module, a scheduling and execution module, a model management module, and a unified runtime interface module.

[0010] The multi-language adaptation module receives and parses runtime call instructions from different upper-level programming languages, converting them into a unified intermediate representation. The task orchestration module performs semantic structure parsing based on the semantic structure and constraints of the intermediate representation, constructing a standardized task graph description for subsequent scheduling and execution. The chip awareness module detects available heterogeneous chips in the current runtime environment and evaluates their capabilities and resource status in real time, generating evaluation vectors. The scheduling and execution module selects the optimal heterogeneous chip and scheduling strategy based on the task graph and evaluation vectors, sending task execution instructions to the selected heterogeneous chip. The model management module loads, registers, caches, and unloads compatible models for various heterogeneous chips, supporting compatibility conversion, compatibility preprocessing, and dynamic runtime switching between different heterogeneous chips. The unified runtime interface module receives task results returned by heterogeneous chips, encapsulates the underlying execution results into a unified runtime interface corresponding to the intermediate representation format, and maps it to the original upper-level language calling environment, such as... Figure 3 As shown.

[0011] The core function of this invention is to shield against the complexities of multilingual differences and heterogeneous hardware: a multilingual adaptation module unifies the intermediate representation, a task orchestration module standardizes execution logic, a chip awareness module senses hardware status in real time, a scheduling and execution module achieves optimal resource matching, a model management module enables cross-chip model compatibility and adaptation, and a unified runtime interface module completes result mapping. Ultimately, this achieves unified scheduling, efficient execution, and seamless compatibility of multilingual programs on heterogeneous chips, solving the pain points of poor adaptability and inefficient scheduling in traditional systems. According to a preferred embodiment, the chip awareness module is configured to: actively detect available heterogeneous chips in the current operating environment during system initialization; monitor key indicator data of available heterogeneous chips in real time during inference; and generate an evaluation vector based on the current task graph requirements and the capabilities and resource status of available heterogeneous chips.

[0012] The chip awareness module overcomes the limitations of existing technologies that rely on one-time hardware probing. It proactively performs a comprehensive probing of available heterogeneous chips during system initialization, and simultaneously monitors key indicators such as chip load, memory usage, and model support capabilities in real time during inference. This enables dynamic, full-cycle awareness of hardware status, avoiding the information lag issues associated with static probing. More importantly, the chip awareness module does not collect hardware data in isolation. Instead, it integrates chip capabilities and resource status into a quantified evaluation vector based on the specific requirements of the current task graph. This provides precise decision-making support for the scheduling and execution module, effectively addressing the pain point of hardware status being disconnected from task requirements in traditional technologies. This significantly improves the rationality and efficiency of heterogeneous chip selection and scheduling strategies.

[0013] According to a preferred embodiment, the scheduling execution module is configured to: upon receiving the task graph and evaluation vector, determine the optimal scheduling scheme based on a heuristic or learning strategy, and select the best chip architecture, task partitioning method, and parallel execution strategy.

[0014] The scheduling and execution module breaks through the limitations of traditional static scheduling by adopting heuristic or learning strategies to dynamically adapt to task requirements and hardware state changes, thereby improving scheduling flexibility. The scheduling and execution module has a more comprehensive scheduling dimension, not only selecting the best-performing chips but also optimizing task partitioning and parallel execution strategies to maximize resource utilization. Based on task graphs and evaluation vectors, the scheduling and execution module makes precise decisions, solving the hardware-task disconnect problem of traditional scheduling and ensuring execution efficiency and adaptability.

[0015] According to a preferred embodiment, the model management module is configured to: read the parameter weights and configuration description of the target model from the memory, perform necessary format conversion and quantization compression processing according to the chip type, and perform model registration and caching strategy management; after loading is completed, send the model handle and initialization status to the scheduling execution module for subsequent inference calls.

[0016] The model management module breaks through the limitations of existing model-chip format binding, and can automatically complete format conversion and quantization compression according to chip type, effectively solving the compatibility problem across heterogeneous chips. At the same time, by executing model registration and caching strategies, it reduces the overhead of repeated loading and improves execution efficiency. After loading, it actively synchronizes the model handle and initialization state to the scheduling execution module to ensure smooth connection of subsequent inference calls and strengthens the inter-module collaboration.

[0017] According to a preferred embodiment, the step of the multi-language adaptation module to uniformly convert runtime call instructions into intermediate representations includes: selecting the corresponding language plugin according to the source language type in the runtime call instructions; the language plugin parses the source code submitted by the user into an abstract syntax tree or a higher-level semantic structure in a sandbox environment, performs structural alignment on the abstract syntax tree or the higher-level semantic structure, and extracts the core execution information; and encodes the core execution information into an internal intermediate representation with complete execution semantics.

[0018] The multi-language adaptation module accurately adapts to different upper-level programming languages ​​through a language plugin mechanism, breaking through the single-language binding limitations of traditional runtime systems. The sandbox environment for parsing source code ensures both execution security and stable extraction of semantic structures such as abstract syntax trees. The core execution information, after structural alignment, is then encoded into an internal intermediate representation with complete execution semantics, achieving unified convergence of multi-language semantics. This provides standardized input for subsequent task orchestration and cross-chip scheduling, effectively solving the core pain point of efficient collaboration of multi-language programs in a unified environment.

[0019] According to a preferred embodiment, the model management module is configured to: receive model call requests from the task orchestration module; retrieve candidate models that meet the criteria in the local model repository and the connected remote model center in order of priority; perform compatibility checks on the retrieved candidate models in sequence; if the compatibility check fails, automatically trigger the model format conversion process, perform consistency verification on the converted new model to ensure that the generated file can be scheduled normally by subsequent modules; package the model path, loading instructions and format description of the converted new model or the original compatible model, send it to the scheduling execution module, and complete the registration and reference statistics update.

[0020] By prioritizing local and remote model repositories, the model management module quickly locates suitable candidate models, improving model acquisition efficiency. Compatibility verification, automatic format conversion, and consistency verification work in tandem to completely solve the model adaptation problem across heterogeneous chips. Packaging and sending model-related information and updating registration and citation statistics ensure accurate calls from the scheduling and execution module, enhancing inter-module collaboration and effectively avoiding the pain points of cumbersome adaptation and call errors in traditional model management, thus ensuring stable and efficient system operation.

[0021] According to a preferred embodiment, the scheduling execution module is further configured to: monitor the status of the heterogeneous chip's task execution process during the process of the heterogeneous chip running based on the received task execution instructions.

[0022] The improved scheduling and execution module fills the gap in existing technology regarding the lack of real-time awareness of heterogeneous chip task execution. By continuously monitoring the chip's status during operation, it can capture key information such as task completion status and abnormal faults in real time, avoiding resource waste or execution interruption due to status lag. At the same time, real-time monitoring data can help determine the chip's operating status and task suitability, providing a basis for subsequent scheduling optimization, effectively ensuring the stability and efficiency of collaborative execution of multiple heterogeneous chips, and solving the pain points of traditional scheduling that focus on heavy allocation and light monitoring.

[0023] According to a preferred embodiment, the scheduling execution module monitors the status of the heterogeneous chip in the running state. The status monitoring methods include: (1) interrupt callback: when the heterogeneous chip completes its task or an exception occurs, an interrupt signal is sent to the processor through the registered callback function to trigger status processing immediately; (2) event polling: in the main running loop, the status query function of the heterogeneous chip driver is called periodically to check whether the task has ended or is in the current execution stage; (3) DMA status channel reading: when the heterogeneous chip writes the running status information to the shared memory through the DMA channel, the running status information is read directly without triggering an interrupt or waiting for a response.

[0024] This improved monitoring method offers significant advantages. It utilizes three complementary mechanisms to adapt to the characteristics of heterogeneous chips. Specifically, interrupt callbacks ensure immediate response upon task completion or in case of an anomaly, avoiding status lag; event polling enables periodic checks during the execution phase, guaranteeing monitoring continuity; and DMA channel reading eliminates interruptions and waiting, reducing resource consumption. This combination covers different operating scenarios while balancing real-time performance and resource efficiency. It addresses the pain points of slow response and high overhead associated with traditional single-monitoring methods, providing precise data support for scheduling optimization and fault handling, and ensuring the stability of heterogeneous chip collaborative operation.

[0025] This invention provides a heterogeneous compatible unified runtime method from a second aspect. The method includes: receiving and parsing runtime call instructions from different upper-level programming languages, and uniformly converting the runtime call instructions into an intermediate representation; performing semantic structure parsing based on the semantic structure and constraints of the intermediate representation to construct a standardized task graph description for subsequent scheduling and execution; detecting available heterogeneous chips in the current runtime environment and evaluating the capabilities and resource status of available heterogeneous chips in real time, generating an evaluation vector; selecting the optimal heterogeneous chip and scheduling strategy based on the task graph and evaluation vector, and sending task execution instructions to the selected heterogeneous chip; loading, registering, caching, and unloading compatibility models for multiple heterogeneous chips, supporting compatibility conversion, compatibility preprocessing, and dynamic runtime switching between multiple heterogeneous chips; receiving task results returned by the heterogeneous chips, encapsulating the underlying execution results into a unified runtime interface corresponding to the intermediate representation format, and mapping it to the original upper-level language call environment.

[0026] In terms of operational efficiency, by unifying intermediate representations and standardizing task graphs to reduce multi-language / hardware adaptation overhead, and combining real-time chip evaluation and optimal scheduling strategies, the method of this invention maximizes the utilization of heterogeneous resources. Model caching and dynamic switching further reduce the time spent on repeated loading. In terms of user experience, the multi-language compatibility of this invention eliminates the need for developers to adapt to different programming environments. The unified interface shields developers from underlying hardware differences, eliminating the need to worry about chip types and model formats, enabling rapid application deployment and seamless migration, significantly reducing development and maintenance costs, and ensuring operational stability.

[0027] According to a preferred embodiment, the method further includes: actively probing available heterogeneous chips in the current operating environment during the system initialization phase; monitoring key indicator data of available heterogeneous chips in real time during the inference process; and generating an evaluation vector based on the current task graph requirements and the capabilities and resource status of available heterogeneous chips.

[0028] In terms of operational efficiency, the system actively probes heterogeneous chips during initialization and monitors key indicators in real time during inference. It then generates evaluation vectors based on task graph requirements, providing data support for precise scheduling and maximizing the utilization of heterogeneous resources. Regarding user experience, the invention automatically completes hardware adaptation and optimization, eliminating the need for manual chip type and status adaptation. This ensures stable program operation, significantly reduces deployment and maintenance costs, and enhances ease of use. Attached Figure Description

[0029] Figure 1 This is a simplified module connection diagram of the heterogeneous compatible unified runtime system provided by the present invention; Figure 2 This is a logical schematic diagram of the heterogeneous compatible unified runtime system provided by the present invention; Figure 3 This is a flowchart illustrating the heterogeneous compatible unified runtime method provided by the present invention; Figure 4 This is a schematic diagram illustrating the effect of the heterogeneous compatible unified runtime system provided by the present invention.

[0030] List of reference numerals 110: Processor; 111: Multi-language adaptation module; 112: Task orchestration module; 113: Chip awareness module; 114: Scheduling and execution module; 115: Model management module; 116: Unified runtime interface module; 120: Communication module; 130: System bus; 140: NPU; 150: Memory; 160: Terminal device; 170: High-level language application program; 180: Heterogeneous chip; 190: User request. Detailed Implementation

[0031] The following is a detailed explanation with reference to the accompanying drawings.

[0032] Currently, in the computer field, existing mainstream runtime environments are typically deeply optimized around a single programming language or a specific hardware platform, failing to provide a unified abstract framework to effectively address diverse language characteristics and heterogeneous hardware architectures. While traditional virtual machines, represented by the JVM, achieve some platform independence, their execution model is tightly coupled to the Java language system, making it impossible to naturally support the efficient coexistence and interaction of system programming languages ​​such as C / C++, Python, or Rust with dynamic languages ​​within the same runtime. Furthermore, for emerging processor instruction set architectures (such as RISC-V and LoongArch), most runtimes lack native support, often requiring additional porting work and compatibility layers. This not only increases the complexity of system deployment but also raises long-term maintenance costs and leads to significant performance degradation.

[0033] As application scenarios become increasingly diverse, the requirements for cross-platform compatibility and portability of software are constantly rising. However, current technical solutions, when dealing with multi-architecture instruction sets (such as x86, ARM, and LoongArch), still mainly rely on static adaptation strategies during the compilation phase, lacking the ability to dynamically adapt to different architectures at runtime. This limitation makes it difficult for applications to achieve rapid deployment, smooth migration, and consistent behavior guarantees in heterogeneous computing environments, restricting their flexible operation and expansion on complex infrastructures.

[0034] To address the shortcomings of existing technologies, this invention provides a heterogeneous compatible unified runtime system and method, which can also be described as a heterogeneous inference system and method oriented towards multi-language support and multi-chip compatibility. This invention can also provide a processor 110 for executing the heterogeneous compatible unified runtime method. This invention can also provide an electronic device, including the processor 110, memory 150, communication module 120, and NPU 140 of this invention, wherein the processor 110 is connected to the memory 150, communication module 120, and NPU 140 via a system bus 130, as shown below. Figure 1 As shown.

[0035] This invention proposes a unified interface architecture for multi-language fusion, supporting transparent calls from multiple programming languages ​​to the system and enabling seamless integration of upper-layer applications across different language ecosystems. Based on this, a low-level runtime system compatible with multiple heterogeneous AI chips is constructed, possessing unified abstraction and efficient scheduling capabilities for heterogeneous hardware resources, significantly improving resource management flexibility. Furthermore, a model management scheme based on dynamic loading and plug-in mechanisms is designed, which can load deep learning models adapted to different hardware platforms on demand and achieve cross-platform inference acceleration. By introducing an inference architecture that combines lightweight and high-performance models, the complementary advantages of these two types of models in response speed and accuracy are fully utilized, achieving intelligent load allocation and optimized resource utilization efficiency. This scheme can be widely applied in multiple fields such as image recognition, not only meeting diverse programming language access requirements but also ensuring the stability and high-performance operation of the system in various heterogeneous AI chip environments.

[0036] The present invention provides the following explanations for certain terms and concepts.

[0037] Language virtualization: refers to the key technology that uses abstraction and bridging mechanisms to uniformly map the semantics of high-level programming languages ​​(such as Python, C++, Rust) into a platform-independent intermediate representation (IR), thereby shielding the differences in underlying heterogeneous computing architectures and instruction sets, and achieving cross-platform portability and execution compatibility of applications.

[0038] Heterogeneous chips (180): These refer to processor units with different computing architectures, instruction sets, or runtime environments, including but not limited to CPUs, GPUs, NPU140s, FPGAs, and customized AI accelerators. These chips differ significantly in execution models, memory hierarchies, data paths, and scheduling mechanisms, posing challenges to unified programming and resource management.

[0039] Unified runtime: refers to a runtime system that provides a consistent interface across various heterogeneous chips, supporting unified task scheduling, memory management, model loading, and performance tuning. This system can coordinate computing tasks across devices and languages, enabling collaborative execution and efficient scheduling of models and applications in heterogeneous environments.

[0040] Intermediate Representation (IR): Refers to a neutral program expression built during language virtualization, serving as a semantic bridge between high-level language logic and low-level hardware execution. IR is independent of the source language and target architecture, and can be further optimized and converted into native executable code or graph-structured computing tasks according to specific chip characteristics, making it the core carrier for achieving cross-platform compatibility.

[0041] Multi-chip adapter: refers to the bridging component responsible for model deployment and execution adaptation in a unified runtime environment, undertaking responsibilities such as resource allocation, execution path translation, instruction mapping, and device-specific optimization. Its role is to dynamically translate the unified intermediate representation into the native execution format required by various heterogeneous chips, realizing the capability of "compile once, run on multiple devices".

[0042] Dynamic model loading refers to the runtime system's ability to dynamically decide on model loading, migration, or unloading operations based on current task load, chip availability, resource status, and historical scheduling information. This mechanism supports on-demand deployment and elastic switching of models across different chips, effectively improving system resource utilization and task response efficiency.

[0043] Example 1 The heterogeneous compatible unified runtime system of this invention can be deployed in devices such as user terminals, edge nodes, or servers. It is composed of hardware including a processor 110, memory 150, communication module 120, NPU 140, unified runtime components, and system bus 130. Figure 1 As shown. Preferably, the processor 110, as the core control unit of the system, integrates functional modules such as a multi-language adaptation module 111, a task orchestration module 112, a chip awareness module 113, a scheduling and execution module 114, a model management module 115, and a unified runtime interface module 116. It is responsible for completing core tasks such as parsing user programs, dynamically scheduling heterogeneous computing resources, calling and executing models, and returning execution results.

[0044] The memory 150 stores intermediate representations, running task structures, chip resource status, loaded model weights, and dynamically cached results, supporting high-speed data reading and writing and task status management during system operation. The communication module 120 connects to the local or remote heterogeneous chip environment 180, collects real-time chip operating status information (such as computational load, memory usage, model support capabilities, etc.), and completes key communication interactions such as task scheduling instruction issuance and execution result feedback. Figure 1 As shown, the communication module 120 is also used to receive user requests 190.

[0045] The aforementioned functional modules are interconnected via system bus 130 to achieve efficient transmission of data and control commands, constructing a hardware operating platform that is modularly decoupled and collaboratively efficient. In actual operation, the system first receives call commands from programming languages ​​such as Python, C++, and Rust via multi-language adaptation module 111, and parses them into standardized intermediate representations; task orchestration module 112 generates a structured execution task graph based on command semantics; chip awareness module 113 monitors and evaluates the computing power and resource status of available heterogeneous chips 180 in real time; scheduling and execution module 114 selects the optimal target chip accordingly and distributes tasks to the corresponding devices for execution; and model management module 115 is responsible for loading the matching model version to the target chip, ensuring accurate loading and efficient operation of computing tasks.

[0046] like Figure 2 As shown, the high-level language application 170 receives images from the terminal device 160. The high-level language application 170 converts the images captured by the terminal device 160 into a programming language and sends the programming language to the multi-language adaptation module 111. The multi-language adaptation module 111 receives the programming language and sends the generated intermediate representation (IR) to the task orchestration module 112. The task orchestration module 112 receives the intermediate representation (IR) and sends the analyzed model call request and task graph to the model management module 115 and chip awareness module 113, respectively. The chip awareness module 113 receives the task graph and sends the hardware information to the unified runtime interface module 116. The model management module 115 receives the model call request from the task orchestration module 112 and sends the optimal loaded model to the unified runtime interface module 116. After receiving the hardware information and loading the model, the unified runtime interface module 116 sends the encapsulated unified runtime interface to the scheduling execution module 114.

[0047] The interaction process of the multilingual adaptation module 111, task orchestration module 112, chip awareness module 113, scheduling execution module 114, model management module 115, and unified runtime interface module 116 is as follows.

[0048] S100: The terminal or front-end application sends a run request, which includes source code, input parameters and execution configuration.

[0049] Specifically, the multi-language adaptation module 111 receives and parses runtime call instructions from different upper-level programming languages, and converts the runtime call instructions into a unified intermediate representation, such as... Figure 3 As shown.

[0050] Preferably, during system operation, the multi-language adaptation module 111 receives execution requests from different upper-level programming languages ​​through the input interface, performs syntax analysis and language binding processing, and uniformly converts diverse language calls into a standardized intermediate representation (IR). This intermediate representation is transmitted by the multi-language adaptation module 111 to the task orchestration module 112 through the system bus 130 for subsequent processing.

[0051] Here, the source code may be in Python / C++ / JavaScript. The programming language may be Python, C++, Rust, etc.

[0052] Preferably, the entry point for the runtime system is initiated by the terminal device 160 (e.g., a graphical interface program, an interactive notebook, an edge acquisition device, etc.) or the front-end service platform (e.g., a Web inference page, an API server) to submit an execution instruction package containing the following content to the unified runtime interface module 116.

[0053] According to a preferred embodiment, the step of the multilingual adaptation module 111 in uniformly converting the runtime call instruction into an intermediate representation includes: S110: After receiving the run call instruction (run request) from the terminal device 160, select the corresponding language plugin according to the source language type in the run call instruction.

[0054] S120: In the sandbox environment, the language plugin parses the user-submitted source code into an abstract syntax tree or a higher-level semantic structure, performs structural alignment on the abstract syntax tree or higher-level semantic structure, and extracts core execution information. This core execution information includes function call graphs, control dependencies, variable lifecycles, and resource binding intents.

[0055] As a pre-parser of the runtime system, the language plugin is responsible for parsing user-submitted source code into an Abstract Syntax Tree (AST) or a higher-level semantic tree structure in the sandbox environment. It then performs static expansion, semantic correlation analysis, and control flow induction to form an internal intermediate representation with complete execution semantics. Therefore, there are significant structural differences. To achieve unified scheduling and optimization across multiple languages, the system introduces a mapping mechanism from the semantic tree to the intermediate representation (IR). This allows code from different programming languages ​​to be converted into a unified intermediate representation, thereby enabling efficient scheduling and optimization.

[0056] The process by which the language plugin extracts core execution information is as follows.

[0057] S121: Traverse the function definition and call nodes in the semantic tree to construct the function call graph: .

[0058] In the above formula, F is the set of functions. This represents the call relationships between functions. This diagram is used to generate a task dependency graph later.

[0059] S122: Extracting control flow dependency graphs based on control structures (such as if-else, for, while, match-case). The diagram illustrates the decision conditions for each branch and path, as well as their nesting relationships, and is formalized as follows: .

[0060] In the above formula, B is the set of basic blocks, and P is the set of jump / branch edges.

[0061] S123: Parse the locations of variable definitions, references, and releases, and construct variable scope mappings to assist in memory binding and resource allocation strategies.

[0062] Variable scope mapping is expressed as: .

[0063] In the above formula, For the timing of variable allocation, This is the timing for releasing variables.

[0064] S124: Identify resource binding intent.

[0065] Identify external resource calls (such as file I / O, GPU instructions, device memory declarations, etc.) involved in semantic nodes and construct a resource access description table: .

[0066] In the above formula, For resource identification, For the timing of resource requests, This is an opportune time to release resources.

[0067] S130: The multilingual adaptation module 111 encodes core execution information into an internal intermediate representation with complete execution semantics. That is, it encodes function call graphs, control flow dependency graphs, variable lifecycles, and resource binding intents into an internal intermediate representation with complete execution semantics.

[0068] For example, function call graphs can be mapped to IR operation nodes, and call relationships can be associated through data edges; control flow dependency graphs can be mapped to IR control edges to ensure execution order; variable scope can be mapped to node attributes, and resource binding intent can be mapped to resource nodes to clarify resource types and lifecycles.

[0069] The intermediate representation has the following characteristics: it is stable within a language, compatible across languages, can be translated into a backend scheduling graph, and carries multi-granularity control and resource information.

[0070] This step overcomes the limitations of traditional multi-language systems where "compilers are bound to specific languages" and "schedulers directly perceive programming syntax," enabling the runtime system to understand and process tasks from different programming languages ​​in a unified, structured manner. This constitutes the core abstraction layer supporting heterogeneous environment compatibility and unified scheduling. Simultaneously, this mechanism provides standardized input for subsequent execution plan decomposition, dependency graph construction, and hardware mapping, playing a crucial role in "semantic convergence" throughout the overall process.

[0071] S200: Maps the semantic trees generated by each language plugin to a standardized intermediate representation.

[0072] Specifically, after the semantic tree is standardized into a unified intermediate representation (IR), the system enters the execution preparation phase. The task orchestration module 112 performs semantic structure parsing based on the semantic structure and constraints of the intermediate representation, constructing a standardized task graph description from the identified computational objectives, model inference requests, data dependencies, and temporal constraints for subsequent scheduling and execution, such as... Figure 3 As shown.

[0073] Preferably, after receiving the intermediate representation, the task orchestration module 112 performs semantic structure parsing to identify key information such as the computational objective, model inference requests, data dependencies, and timing constraints. Next, the task orchestration module 112 constructs this key information into a standardized task graph to guide subsequent chip scheduling and computation execution. The generated task graph is transmitted by the task orchestration module 112 to the chip awareness module 113 for further processing and execution.

[0074] Here, a task graph is an abstract data structure or model used to represent the execution logic of a program or computational task. It is usually represented in the form of a directed acyclic graph (DAG), where nodes represent specific computational tasks or operations, such as matrix multiplication or model inference; edges represent data dependencies or control dependencies between tasks, ensuring that tasks are executed in the correct order.

[0075] A task graph description is a concrete implementation or representation of this abstract model; that is, how to transform this conceptual task graph into a data format that can be stored, transmitted, and parsed. Common task graph description formats include structured data formats such as JSON, YAML, and Protocol Buffers.

[0076] S300: Call the hardware detection API to identify the available heterogeneous chips 180 on the current platform.

[0077] Specifically, the chip sensing module 113 detects available heterogeneous chips 180 in the current operating environment and evaluates the capabilities and resource status of the available heterogeneous chips 180 in real time, generating an evaluation vector, such as... Figure 3 As shown.

[0078] Preferably, during the system initialization phase, the system actively detects available heterogeneous chips 180 in the current operating environment; during the inference process, it monitors key indicator data of available heterogeneous chips 180 in real time. The chip awareness module 113 generates an evaluation vector based on the current task graph requirements and the capabilities and resource status of available heterogeneous chips 180.

[0079] Heterogeneous chips 180 include, but are not limited to, NPU140, GPU, and customized AI accelerators.

[0080] Key performance indicators include the load status, memory usage, task queue length, and model support capabilities of the heterogeneous chip 180 monitored in real time by the chip sensing module 113 during the inference process.

[0081] The specific processing procedure of the chip sensing module 113 is as follows.

[0082] S310: Chip sensing module 113 calls the hardware detection API embedded at runtime.

[0083] The hardware detection API is a set of system-level interfaces for heterogeneous computing environments, used to automatically identify and collect computing resources of the current platform. The chip sensing module 113 uses this interface to perform a comprehensive scan of the physical platform, identifying various hardware devices including mainstream processors (such as x86, ARM, RISC-V), general-purpose accelerators (such as GPUs, NPU140), and heterogeneous chips 180 (such as Compagnus, Cambricon, Rockchip, etc.).

[0084] During the detection process, the chip sensing module 113 dynamically parses various low-level interfaces, such as / proc / cpuinfo, lspci, OpenCL API, and dedicated SDKs provided by chip manufacturers, to obtain key indicator data of the device.

[0085] For the heterogeneous chip 180, the chip sensing module 113 further extracts fine-grained capability information such as its operation unit type, supported operator matrix, communication bus bandwidth and heterogeneous caching strategy by adapting to the manufacturer's specific interface.

[0086] S320: Integrates and encapsulates all collected information to generate standardized equipment capability description information, and extracts and forms evaluation vectors for subsequent task scheduling.

[0087] After completing hardware detection and status acquisition, the chip sensing module 113 integrates hardware capability data, real-time load information, and model support capabilities from different sources into standardized device capability description information. Based on this, the chip sensing module 113 extracts key features directly related to scheduling decisions from the integrated device capability description information. These key features include the chip's computing power indicators, available memory and bandwidth, current operating load, chip compatibility with target model formats and operator sets, and data on chip temperature, power status, and abnormal conditions during operation.

[0088] After normalization, structured encoding, and field conventions, the aforementioned key features constitute a quantifiable feature set, namely the evaluation vector, according to a preset semantic dimension. This evaluation vector comprehensively reflects the target chip's available computing power, resource consumption, and execution risk at the current moment. Based on this, the scheduling and execution module 114 can perform horizontal comparisons of multiple heterogeneous chips 180, thereby selecting the most suitable target chip for the current task graph, and further determining the task partitioning method and execution strategy to achieve dynamic optimization scheduling for heterogeneous hardware environments.

[0089] The device capability description information provides a reliable basis for subsequent task scheduling, model matching and resource allocation, realizing dynamic perception and intelligent selection of heterogeneous chip resources, and is a key foundation for supporting cross-platform compatibility and efficient scheduling under a unified runtime.

[0090] This step, by deeply integrating hardware detection with chip capability modeling, provides fundamental support for the system to achieve adaptive operation in a wide range of heterogeneous environments, and is one of the key paths to support the schedulable and compatible operation of heterogeneous computing chips.

[0091] S400: Based on the required backend model call type of the intermediate representation, query and perform format conversion in the model management module 115, load the model, and encapsulate it into a unified runtime interface by the unified runtime interface module 116.

[0092] Specifically, after completing the hardware resource detection, the model management module 115 will further analyze the backend model call requirements involved based on the generated intermediate representation (IR).

[0093] The backend model call requirements include the model name, model task type, required framework type, and target chip compatibility format. These requirements will be used as input to the model management module 115 to execute subsequent model preparation processes. Model task types include, for example, classification, generation, and retrieval.

[0094] The model management module 115 loads, registers, caches, and unloads compatible models for various heterogeneous chips 180, supporting compatibility conversion, compatibility preprocessing, and dynamic switching at runtime between various heterogeneous chips 180, such as... Figure 3 As shown.

[0095] Preferably, the model management module 115 reads the parameter weights and configuration description of the target model from the memory 150, performs necessary format conversion and quantization compression processing according to the chip type, and performs model registration and caching strategy management; after loading is completed, the model handle and initialization status are sent to the scheduling execution module 114 for subsequent inference calls.

[0096] The specific process of model management module 115 is as follows.

[0097] S410: Receive a model call request from the task orchestration module 112.

[0098] The model call request includes information such as model name, task type, shape and data type of target input and output tensors, and runtime constraints.

[0099] S420: Retrieve candidate models that meet the criteria in order of priority between the local model repository and the connected remote model center.

[0100] S430: Perform compatibility checks on the retrieved candidate models in sequence.

[0101] The model management module 115 will read the model's metadata and match it with the hardware capabilities (such as supported model formats, maximum tensor size, computational precision, etc.) reported by the chip awareness module 113 to ensure that the model can be correctly loaded and executed efficiently on the target device.

[0102] S440: If the compatibility check fails, the model format conversion process is automatically triggered to perform consistency verification on the converted new model, ensuring that the generated file can be scheduled normally by subsequent modules.

[0103] The model format conversion process is as follows.

[0104] S441: The model management module 115 calls the model conversion scheduler to dynamically load the corresponding format converter plugin according to the target chip type (for example, loading the ONNX2FTBModel plugin on the computing platform, the converter reads the original model file, parses its computation graph structure, and performs key conversion operations such as operator semantic mapping, tensor format standardization, computation order optimization and reconstruction). S442: After the conversion is complete, the generated new model file will be written to the local cache directory of the runtime by the model management module 115, marked as "platform-specific copy", and registered in the model index table to support quick search and reuse in subsequent tasks, avoid repeated conversion, and improve execution efficiency.

[0105] Finally, the model management module 115 packages the converted or original compatible model path, loading instructions, and format description together, and passes them to the unified runtime interface module 116 for interface encapsulation. The unified runtime interface module 116 then forwards the packages to the scheduling and execution module 114, completing registration and reference statistics updates. The entire process ensures functional matching, format consistency, and execution reachability between the model and the target chip, providing stable model support for subsequent task scheduling.

[0106] S450: Package the model path, loading instructions, and format description of the converted new model or the original compatible model, send them to the scheduling execution module 114, and complete the registration and reference statistics update.

[0107] S460: The unified runtime interface module 116 receives the task results returned by the heterogeneous chip 180, encapsulates the underlying execution results into a unified runtime interface corresponding to the intermediate representation format, maps it to the original upper-level language calling environment, and finally returns it to the user program through the language adaptation module.

[0108] The unified runtime interface uses standard output structures, such as tensors, scalars, or dictionary structures.

[0109] After the conversion is complete, the unified runtime interface module 116 further encapsulates the model: regardless of the original format or loading method of the model, it is ultimately encapsulated into a standard form conforming to the unified runtime interface. This interface defines core elements such as standardized call entry points, layout specifications for input and output tensors, asynchronous execution and callback mechanisms, and resource release processes. This ensures that the execution engine can schedule and run various models in a unified and consistent manner, shielding the differences in underlying frameworks, chip manufacturers, and deployment formats, and achieving true heterogeneous compatibility and portability.

[0110] S500: The intermediate representation is distributed to the target heterogeneous chip 180 along with the model. The scheduling and execution module 114 calls the local driver and API to start the running task and monitor the running status.

[0111] The scheduling and execution module 114 selects the optimal heterogeneous chip 180 and scheduling strategy based on the task graph and evaluation vector, and sends task execution instructions to the selected heterogeneous chip 180, such as... Figure 3 As shown.

[0112] Specifically, after receiving the task graph and evaluation vector, the scheduling and execution module 114 determines the optimal scheduling scheme based on a heuristic or learning-based strategy, selecting the best heterogeneous chip 180, the task partitioning method, and the parallel execution strategy. The scheduling and execution module 114 sends task execution instructions to the selected heterogeneous chip 180 (the target heterogeneous chip), such as... Figure 2 As shown. The task execution instruction includes an intermediate representation reference, a model loading handle, an input data address, and execution configuration parameters, used to trigger the heterogeneous chip 180 to start the corresponding inference or computation process. The scheduling execution module 114 receives status feedback signals from the heterogeneous chip 180 for subsequent execution monitoring and result feedback.

[0113] After the unified runtime interface module 116 completes the unified encapsulation of the model, the scheduling and execution module 114 packages the standardized intermediate representation (IR) and the corresponding backend model instance into a task execution unit. Based on the results of previous hardware probing and chip awareness, the scheduling and execution module 114 selects the heterogeneous chip 180 with the best performance and sufficient resources as the target execution platform. Subsequently, this task unit is transmitted to the scheduling and execution module 114 bound to the target chip via the system bus 130, which is responsible for coordinating the allocation of underlying computing resources, driver invocation, and task startup to ensure that the model executes efficiently on the target device.

[0114] Here, the task unit is a runtime standard structure defined in this invention, used to describe a schedulable execution task, which contains key metadata such as input / output tensor information, resource requirement declaration, and execution priority.

[0115] During the process of the heterogeneous chip 180 executing the received task instructions, the scheduling execution module 114 monitors the status of the task execution process of the heterogeneous chip 180.

[0116] The scheduling execution module 114 monitors the status of the heterogeneous chip 180 in the running state. The status monitoring methods include: (1) interrupt callback. When the heterogeneous chip 180 completes its task or an exception occurs, it sends an interrupt signal to the processor 110 through the registered callback function to trigger status processing as soon as possible. (2) Event polling: In the main runtime loop, the status query function of the heterogeneous chip 180 driver is called periodically to check whether the task has ended or is currently in the execution stage; (3) DMA status channel reading: When the heterogeneous chip 180 writes the running status information to the shared memory through the DMA channel, the running status information can be read directly without triggering an interrupt or waiting for a response.

[0117] The selection of the above mechanisms is automatically configured based on the chip's capabilities during system initialization and can be bound in the chip awareness module 113. The execution status is continuously monitored through interrupt callbacks, event polling, or DMA status channels. Real-time task execution metrics, such as the current running stage, amount of data processed, core utilization, and error codes, are acquired and periodically written to shared memory or the monitoring channel.

[0118] Through the above mechanism, the system completes the localized scheduling and execution of intermediate representations and models, and realizes full-cycle monitoring of the running status, providing a unified and transparent underlying execution guarantee for cross-language and cross-platform AI applications.

[0119] S600: Collects execution results from heterogeneous chip 180, performs formatting, result fusion, and status reporting.

[0120] After each heterogeneous chip 180 completes its task execution, its calculation results are uniformly read by the system core processor 110 through the chip's driver program. Specifically, the scheduling and execution module 114 continuously monitors the completion flags or task callback mechanisms provided by the chip driver (such as interrupt signals, polling interfaces, or task queue status). After confirming that the chip has completed its task, it retrieves the output data from the designated memory area. This output data may include tensor results (such as image classification probabilities, object detection box coordinates), state scalars (such as execution status codes, processing time), or structured output (such as nested lists, dictionaries, and other composite data types). Since the raw output data of different chips differ in numerical precision, dimensional layout, alignment, and data encapsulation format, the system has an internal result format conversion mechanism to uniformly process this heterogeneous data.

[0121] This mechanism is scheduled and executed by the unified runtime interface module 116, and the processing includes: Identify the numerical precision of the raw output data (such as INT8, FP16, FP32) and upconvert it to standard FP32 if necessary; For tensor data, it is automatically converted to a platform-unified layout order (e.g., from NCHW to NHWC). For structured data results, parse them into standard JSON or system-defined intermediate structure formats; If an illegal encoding, precision overflow, or alignment anomaly is encountered, the system will mark the error and output a log for subsequent analysis.

[0122] After processing, the converted unified format result is sent back to the task orchestration module 112, which then performs subsequent processing such as encapsulation, return, and status reporting of the final result.

[0123] After the fusion results are generated, the system will also encapsulate the operating status, resource consumption, and anomaly records of each heterogeneous chip 180 into a unified status report package. This report includes key indicators such as total execution time, time distribution of each stage, device temperature, power consumption, and failure reason code. The status report is synchronously transmitted to the monitoring module or sent to the front-end application caller via the system bus 130, providing transparent execution feedback for developers or platform maintenance personnel.

[0124] This collection and fusion mechanism ensures that the system can still provide stable, standardized, and traceable operating results when faced with semantic differences in the output of heterogeneous chips, providing key support for subsequent performance analysis, dynamic scheduling, and adaptive training.

[0125] S700: Has an update been detected for the plugin model? If yes, complete the hot update, replace the plugin or model, and re-register; if no, return to step S100.

[0126] After each round of task execution, the system will automatically query the latest version information of plugins and models through the version monitoring module. The sources include local configuration files, cloud version control services, or private plugin repositories registered by developers.

[0127] Once a new version of a plugin or model is detected, the system will trigger a hot update process. The hot update module first suspends further scheduling of the current plugin or model through the module scheduler, ensuring that any suspended tasks have been completed or aborted. Subsequently, the system downloads or decompresses the update package from the repository and calls the plugin registration interface to complete the version replacement. For model updates involving underlying driver adaptation, runtime encapsulation information must also be refreshed synchronously to ensure that the new model can correctly interface with the chip's backend call stack.

[0128] After a hot update is completed, the system automatically rebuilds the plugin index table and re-registers the new plugin or model into the unified runtime framework, making it effective in the next round of task scheduling. All hot update processes are completed asynchronously during the system's task scheduling idle window, ensuring that the user is unaware of the update and that the platform runs without interruption.

[0129] If no updates are found during version detection, the system will maintain the current plugin and model configuration, continue to respond to the next round of running requests, restart the above data interaction process, and form a complete closed loop.

[0130] The entire data interaction process of the heterogeneous compatible unified runtime system is scheduled by the processor 110. Data and control signals between modules are efficiently transmitted through the system bus 130. The memory 150 provides functions such as high-speed buffering, model parameter caching, and intermediate calculation result storage during the process, ensuring the system's real-time performance and high throughput execution efficiency. In actual operation, the system also supports caching the task graph and chip scheduling history to optimize the execution path of subsequent tasks and improve the utilization of heterogeneous chips 180, achieving efficient, stable, and scalable unified runtime support for multi-language programming scenarios.

[0131] This invention provides a specific application example.

[0132] like Figure 4 As shown, a user uploads a photo containing a street view in the front-end application, intending to perform object detection analysis. The user selects image recognition as the task type and specifies the use of the YOLOv8 model. In the system of this invention, the request is first parsed by the multilingual adaptation module 111, which parses the request information and configuration parameters and abstracts the image recognition task into an intermediate representation (IR) structure. Subsequently, the system calls the chip perception module 113 to obtain the hardware environment information of the terminal device 160, discovering that it has two types of chips: Rockchip and Computing Power.

[0133] After task parsing is complete, the model management module 115 loads the corresponding model file from the local or remote model repository according to the YOLOv8 model structure required by IR, completes the format conversion, and passes it to the unified runtime interface module 116. The unified runtime interface module 116 then constructs a unified runtime interface (such as ONNXRuntime, TensorRT, or Cambricon CNRT scheduler). This interface will be injected into the task description for subsequent use.

[0134] Next, the task orchestration module 112 generates a task graph based on the chip resource status, model type, and request configuration (such as whether reduced accuracy is allowed or whether a fast response is required). Since this image recognition task is sensitive to response latency, the system decides to prioritize running it on the low-latency target chip. Before the task officially starts, the scheduling and execution module 114 sends the model, input image, and scheduling instructions to the target chip and initiates inference via the local driver API.

[0135] During inference, the system continuously monitors the task's running status and ultimately collects the recognition results (such as bounding boxes and confidence information for "car", "person", "traffic light" identified in the image). The scheduling and execution module 114 formats the results and returns them to the application front-end; at the same time, the system records metadata such as resource consumption and execution time for this run, which is used to support subsequent performance optimization and hot scheduling path recommendation.

[0136] Figure 4 This paper demonstrates an example of the inference results of the system of the present invention in a real-world image recognition task. The image was taken in a city street scene, showing a sanitation worker wearing an orange reflective vest sweeping the road. After receiving the image input request, the system, through a unified runtime task orchestration and chip scheduling mechanism, distributes the inference task to a locally available heterogeneous intelligent chip platform. After loading and calling the YOLOv8 model deployed on the chip, it successfully identifies the main target category in the image.

[0137] like Figure 4 As shown, the system identifies a "person" object in the image and draws a blue bounding box around it for visualization. The upper right corner of the box displays the identification label "person" and its corresponding confidence score, approximately 0.88 (0.877441), indicating a high level of confidence in the identification result. This label is generated by the "Result Encapsulation and Fusion Module" of the unified runtime, which formats the raw output of the object detection model and returns it to the upper-layer application in a standardized JSON format for front-end system use and display.

[0138] In addition, sanitation carts, lane lines, prohibition signs, parking signs, shop signs, and other street elements can be observed in the background. Despite some obstruction and background interference, the system still accurately identified the target person, demonstrating that the present invention still possesses good robustness and recognition accuracy when running the target detection model in a heterogeneous chip 180 environment.

[0139] During this process, if the system detects that the YOLO model has been updated (such as model accuracy optimization or pruning version release), the model management module 115 will trigger a hot update mechanism, replace the current model weights and re-register the unified runtime interface in the background to ensure that subsequent requests can automatically adapt to the latest model version without restarting the system.

[0140] This embodiment demonstrates the practical application process of the present invention in image recognition tasks, showcasing the efficient collaboration between various modules (language adaptation, chip awareness, task orchestration, model management, scheduling execution, and unified interface) and the flexible adaptability in multi-chip heterogeneous environments.

[0141] Example 2 This embodiment is a further explanation of Embodiment 1, and repeated content will not be repeated.

[0142] This embodiment provides one example of physical hardware for a heterogeneous compatible unified runtime system.

[0143] The physical hardware of the heterogeneous compatible unified runtime system of the present invention includes a processor 110, a memory 150, a communication module 120, a system bus 130, and other hardware.

[0144] Preferably, the processor 110, as the core control unit of the system, integrates functional modules such as a multi-language adaptation module 111, a task orchestration module 112, a chip awareness module 113, a scheduling and execution module 114, a model management module 115, and a unified runtime interface module 116. It is responsible for completing core tasks such as parsing user programs, dynamically scheduling heterogeneous computing resources, calling and executing models, and returning execution results.

[0145] Preferably, the physical hardware of the multilingual adaptation module 111 is a multilingual adaptation circuit. The physical hardware of the task orchestration module 112 is a task orchestration circuit. The physical hardware of the chip sensing module 113 is a chip sensing circuit. The physical hardware of the scheduling execution module 114 is a scheduling execution circuit. The physical hardware of the model management module 115 is a model management circuit. The physical hardware of the unified runtime interface module 116 is a unified runtime interface circuit.

[0146] The physical hardware of the memory 150 is a memory chip. The physical hardware of the communication module 120 is a communication interface chip. The memory chip is used to store intermediate representations, running task structures, chip resource status, loaded model weights, and dynamically cached results, supporting high-speed data reading and writing and task status management during system operation. The communication interface chip is responsible for connecting to the local or remote heterogeneous chip environment 180, collecting chip running status information in real time (such as computational load, memory usage, model support capabilities, etc.), and completing key communication interactions such as task scheduling instruction issuance and execution result feedback. The communication interface chip is also used to receive user requests 190.

[0147] Multi-language adaptation circuits, task orchestration circuits, chip sensing circuits, scheduling and execution circuits, model management circuits, unified runtime interface circuits, storage chips, and communication interface chips are interconnected through the system bus 130 to achieve efficient transmission of data and control commands, thus constructing a modularly decoupled and collaboratively efficient hardware operating platform.

[0148] In actual operation, the system first receives calling instructions from programming languages ​​such as Python, C++, and Rust through a multi-language adaptation circuit, and parses them into a standardized intermediate representation. The task orchestration circuit generates a structured execution task graph based on the instruction semantics. The chip awareness circuit monitors and evaluates the computing power and resource status of the available heterogeneous chips 180 in real time. The scheduling execution circuit selects the optimal target chip accordingly and distributes tasks to the appropriate devices for execution. The model management circuit is responsible for loading the matching model version to the target chip, ensuring accurate loading and efficient execution of the computing tasks.

[0149] The high-level language application 170 converts the images captured by the terminal device 160 into a programming language and sends the programming language to the multilingual adapter circuit. The multilingual adapter circuit receives the programming language and sends the generated intermediate representation (IR) to the task orchestration circuit.

[0150] After receiving the intermediate representation (IR) from the multilingual adaptation circuit, the task orchestration circuit receives the intermediate representation (IR) and sends the analyzed model call request and task graph to the model management circuit and the chip sensing circuit, respectively.

[0151] The chip sensing circuit receives the task diagram and sends the hardware information to the unified runtime interface circuit. The model management circuit receives the model invocation request and sends the optimal loaded model to the unified runtime interface circuit. After receiving the hardware information and the loaded model, the unified runtime interface circuit sends the encapsulated unified runtime interface to the scheduling and execution circuit. Based on this, the scheduling and execution circuit selects the optimal target chip and distributes the task to the corresponding device for execution.

[0152] It should be noted that the specific embodiments described above are exemplary. Those skilled in the art can devise various solutions inspired by the disclosure of this invention, and these solutions all fall within the scope of this invention and its protection. Those skilled in the art should understand that this specification and its accompanying drawings are illustrative and not intended to limit the scope of the claims. The scope of protection of this invention is defined by the claims and their equivalents. This specification contains multiple inventive concepts; terms such as "preferredly," "according to a preferred embodiment," or "optionally" indicate that the corresponding paragraph discloses an independent concept. The applicant reserves the right to file divisional applications based on each inventive concept.

Claims

1. A heterogeneous compatible unified runtime system, characterized in that, The system includes a processor (110), the processor (110) comprising: The multi-language adaptation module (111) receives and parses the execution call instructions from different upper-level programming languages, and converts the execution call instructions into an intermediate representation. The task orchestration module (112) performs semantic structure parsing based on the semantic structure and constraints of the intermediate representation, and constructs a standardized task graph description for subsequent scheduling and execution. The chip sensing module (113) detects available heterogeneous chips (180) in the current operating environment and evaluates the capabilities and resource status of available heterogeneous chips (180) in real time, generating an evaluation vector; The scheduling execution module (114) selects the best heterogeneous chip (180) and scheduling strategy according to the task graph and the evaluation vector, and sends the task execution instruction to the selected heterogeneous chip (180); The model management module (115) loads, registers, caches, and unloads compatible models for various heterogeneous chips (180), and supports compatibility conversion, compatibility preprocessing, and dynamic switching at runtime between various heterogeneous chips (180). The unified runtime interface module (116) receives the task results returned by the heterogeneous chip (180), encapsulates the underlying execution results into a unified runtime interface corresponding to the intermediate representation format, and maps them to the original upper-level language calling environment.

2. The system according to claim 1, characterized in that, The chip sensing module (113) is configured as follows: During the system initialization phase, the system actively detects available heterogeneous chips in the current operating environment (180). Key performance data of available heterogeneous chips (180) are monitored in real time during the inference process; An evaluation vector is generated based on the current task graph requirements and the capabilities and resource status of available heterogeneous chips (180).

3. The system according to claim 1 or 2, characterized in that, The scheduling execution module (114) is configured to: after receiving the task graph and the evaluation vector, determine the best scheduling scheme based on a heuristic or learning strategy, and select the best structural chip (180), task partitioning method and parallel execution strategy.

4. The system according to any one of claims 1 to 3, characterized in that, The model management module (115) is configured as follows: The parameter weights and configuration description of the target model are read from the memory (150), and the necessary format conversion and quantization compression are performed according to the chip type. The model registration and caching strategy management are then executed. After loading is complete, the model handle and initialization status are sent to the scheduling execution module (114) for use in subsequent inference calls.

5. The system according to any one of claims 1 to 4, characterized in that, The steps of the multilingual adaptation module (111) in uniformly converting the execution call instruction into an intermediate representation include: Select the corresponding language plugin based on the source language type in the execution command; The language plugin parses the user-submitted source code into an abstract syntax tree or a higher-level semantic structure in a sandbox environment, performs structural alignment on the abstract syntax tree or higher-level semantic structure, and extracts core execution information. The core execution information is encoded into an internal intermediate representation with complete execution semantics.

6. The system according to any one of claims 1 to 5, characterized in that, The model management module (115) is configured as follows: Receive model call requests from the task orchestration module (112); In the local model repository and the connected remote model center, candidate models that meet the criteria are retrieved in order of priority. Compatibility checks are performed on each of the retrieved candidate models in turn; If the compatibility check fails, the model format conversion process is automatically triggered to perform consistency verification on the converted new model, ensuring that the generated file can be scheduled normally by subsequent modules. The model path, loading instructions, and format description of the converted new model or the original compatible model are packaged and sent to the scheduling execution module (114), and registration and reference statistics updates are completed.

7. The system according to any one of claims 1 to 6, characterized in that, The scheduling execution module (114) is also configured to: During the process of the heterogeneous chip (180) executing the task based on the received task execution instructions, the status of the task execution process of the heterogeneous chip (180) is monitored.

8. The system according to any one of claims 1 to 7, characterized in that, The scheduling execution module (114) monitors the status of the heterogeneous chip (180) in operation. The status monitoring methods include: Interrupt callback: When the heterogeneous chip (180) completes its task or an exception occurs, it sends an interrupt signal to the processor (110) through the registered callback function to trigger status processing as soon as possible. Event polling periodically calls the status query function of the heterogeneous chip (180) driver in the main runtime loop to check whether the task has ended or is in the current execution stage; DMA (Direct Memory Access) status channel reading: When the heterogeneous chip (180) writes the running status information to the shared memory through the DMA channel, the running status information can be read directly without triggering an interrupt or waiting for a response.

9. A heterogeneous compatible unified runtime method, characterized in that, The method includes: Receive and parse execution call instructions from different upper-level programming languages, and convert the execution call instructions into an intermediate representation; Semantic structure parsing is performed based on the semantic structure and constraints of the intermediate representation. The identified computational objectives, model inference requests, data dependencies, and temporal constraints are constructed into a standardized task graph description for subsequent scheduling and execution. The system detects available heterogeneous chips (180) in the current operating environment and evaluates the capabilities and resource status of the available heterogeneous chips (180) in real time, generating an evaluation vector. Based on the task graph and the evaluation vector, the optimal heterogeneous chip (180) and scheduling strategy are selected, and task execution instructions are sent to the selected heterogeneous chip (180). The system loads, registers, caches, and unloads compatibility models for various heterogeneous chips (180), and supports compatibility conversion, compatibility preprocessing, and dynamic switching at runtime between various heterogeneous chips (180). Receive the task results returned by the heterogeneous chip (180), encapsulate the underlying execution results into a unified runtime interface corresponding to the intermediate representation format, and map them to the original upper-level language calling environment.

10. The method according to claim 9, characterized in that, The method further includes: During the system initialization phase, the system actively detects available heterogeneous chips in the current operating environment (180). Key performance data of available heterogeneous chips (180) are monitored in real time during the inference process; An evaluation vector is generated based on the current task graph requirements and the capabilities and resource status of available heterogeneous chips (180).

Images