Operator intelligent optimizer and optimization method based on multi-path parallel and expert hybrid

Abstract

This application provides an operator intelligent optimizer and optimization method based on multi-path parallelism and expert hybridization. The operator intelligent optimizer includes: a multi-dimensional feature extraction and encoding layer, used to extract and encode features of each AI operator in the input AI operator set to obtain the encoding results of each AI operator, and then obtain an encoding result set; a MoE dynamic routing and expert collaboration layer, used to distribute the encoding result set to multiple diagnostic expert networks, and diagnose the encoding result set based on the multiple diagnostic expert networks to obtain the diagnostic results of each diagnostic expert network; and an adaptive decision-making and code generation layer, used to select the strategy combination that best matches the AI ​​operator set from a preset strategy library based on the diagnostic results of each diagnostic expert network, and compile the strategy combination into code that can be directly executed on the target heterogeneous platform, which can save computing resources.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and more specifically, it relates to an operator intelligent optimizer and optimization method based on multi-path parallelism and expert hybridism. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, AI models are becoming more complex and larger at an unprecedented rate, especially in the fields of large language models (LLM) and multimodal models. To support the training and inference of these advanced models, the underlying hardware is also constantly evolving, forming heterogeneous computing systems that combine general-purpose processors (such as ARM CPUs) and dedicated AI accelerators (such as neural processing units, NPUs). This system architecture aims to combine the flexibility of general-purpose processing with the high energy efficiency of dedicated computing and has been widely used in various computing scenarios from edge devices to data centers.

[0003] When dealing with modern AI models and complex ARM-NPU heterogeneous systems, several deep-seated limitations are still exposed. These limitations hinder the full release of hardware potential, lack a fine understanding of the internal microarchitecture of the hardware, and the compiler is unable to make optimal decisions when performing task partitioning and instruction scheduling. This often leads to some computing units being overloaded while others are idle, resulting in a serious waste of computing resources. Summary of the Invention

[0004] To address the aforementioned problems, this application provides an operator-based intelligent optimizer and optimization method based on multi-path parallelism and expert hybridism, thereby solving or at least alleviating one or more of the aforementioned problems and other issues existing in the prior art.

[0005] A first aspect of this application provides an operator-based intelligent optimizer based on multi-path parallelism and expert hybrid optimization, comprising: The multidimensional feature extraction and encoding layer is used to extract and encode features from each AI operator in the input AI operator set, obtain the encoding results of each AI operator, and then obtain the encoding result set. The MoE dynamic routing and expert collaboration layer is used to distribute the encoded result set to multiple diagnostic expert networks, and to diagnose the encoded result set based on the multiple diagnostic expert networks to obtain the diagnostic results of each diagnostic expert network. An adaptive decision-making and code generation layer is used to select a strategy combination that best matches the AI ​​operator set from a preset strategy library based on the diagnostic results of each diagnostic expert network, and compile the strategy combination into code that can be directly executed on the target heterogeneous platform.

[0006] A second aspect of this application provides an operator-based intelligent optimization method based on multi-path parallelism and expert hybrid approaches, comprising: Feature extraction and encoding are performed on each AI operator in the input AI operator set to obtain the encoding result of each AI operator, and then the encoding result set is obtained. The encoded result set is distributed to multiple diagnostic expert networks, and the encoded result set is diagnosed based on the multiple diagnostic expert networks to obtain the diagnostic results of each diagnostic expert network; Based on the diagnostic results of each diagnostic expert network, a strategy combination that best matches the AI ​​operator set is selected from the preset strategy library, and the strategy combination is compiled into code that can be directly executed on the target heterogeneous platform.

[0007] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0008] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0009] The beneficial effects of this application's embodiments are as follows: This application implements a more advanced end-to-end automated process from operator analysis to code generation. It can not only generate optimal strategies for each individual operator in the AI ​​operator set, but also perform global optimization at the computational level, exploring optimization opportunities across operators. It can establish a high-fidelity "operator-hardware" mapping relationship. This enables extremely accurate prediction of operator performance, power consumption, and memory behavior on heterogeneous platforms, and precisely pinpoints the root cause of performance bottlenecks. Therefore, the optimization strategies generated based on this precise analysis are far more effective than traditional methods based on coarse-grained hardware abstraction, enabling greater exploitation of hardware potential and improved utilization of computing resources. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic diagram of the structure of an operator-based intelligent optimizer based on multi-path parallelism and expert hybrid technology provided in an embodiment of this application; Figure 2A schematic diagram of a scenario for an operator-based intelligent optimization method based on multi-path parallelism and expert hybridism provided in an embodiment of this application; Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in the specific implementation of this application should fall within the protection scope of the embodiments of this application.

[0013] To keep the drawings concise, each drawing only schematically shows the parts relevant to the disclosure; these do not represent the actual structure of the product. Furthermore, for ease of understanding, in some drawings, only one of components with the same structure or function is schematically shown, or only one is labeled. In this document, "one" not only means "only one," but can also mean "more than one," and "several" includes "two" and "more than two."

[0014] Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0015] It should be understood that, unless the context clearly states otherwise, the terms "comprising," "including," or "having" as used herein refer to the presence of an element, but do not exclude the presence or addition of one or more other elements. Furthermore, "comprising" and / or "including" as used herein specify the presence of shapes, numbers, steps, operations, members, elements, and / or combinations thereof, and do not exclude the presence or addition of one or more other shapes, numbers, operations, elements, and / or combinations thereof. Some embodiments of this application are described in detail below with reference to the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. The steps in the following method embodiments are for illustrative purposes only and are not intended to limit this application.

[0016] To fully unlock the potential of heterogeneous hardware, advanced compilation techniques and operator optimization frameworks have emerged. Currently, the mainstream technological trend in the industry is to replace traditional manual optimization with automated and learning-driven methods. For example, AI compilers such as Apache TVM and XLA (Accelerated Linear Algebra) have achieved significant performance gains on various hardware platforms by introducing complex intermediate representations (IRs) and automated code generation and tuning mechanisms. These frameworks can translate high-level computational graph descriptions into highly optimized, hardware-specific low-level code, thus bridging the gap between AI algorithms and hardware implementations.

[0017] In recent years, research has further focused on the exploration of the search space and the generation of optimization strategies. By constructing complex cost models and employing advanced search algorithms (such as evolutionary algorithms or reinforcement learning), it is possible to automatically find the optimal operator implementation in a vast optimization strategy space. Meanwhile, to better adapt to diverse hardware architectures, some projects provide a scalable, multi-layered compiler infrastructure that allows developers to define proprietary dialects and optimization processes for new hardware, thereby achieving a higher degree of customization and flexibility.

[0018] Despite significant advancements in existing automated optimization techniques, several deep-seated limitations remain when dealing with modern AI models and complex ARM-NPU heterogeneous systems. These limitations hinder the full realization of hardware potential and introduce new technical challenges. This application aims to address one or more of the following technical problems: First, the hardware abstraction level is too coarse, making it impossible to perceive the characteristics of the microarchitecture.

[0019] Existing optimization frameworks typically treat hardware targets (such as ARM CPUs or NPUs) as a single black box model. This level of abstraction ignores the complex microarchitectural details within modern processors, such as the "big.LITTLE" architecture of ARM processors, cache coherency protocols between different core clusters, and the various dedicated computing resources coexisting within the NPU, including matrix computation engines, vector processing units, and tensor cores. Due to this lack of fine-grained understanding of these microarchitectural details, compilers cannot make optimal decisions when partitioning tasks and scheduling instructions, often resulting in some computing units being overloaded while others remain idle, leading to severe waste of computing resources and performance bottlenecks.

[0020] Second, the optimization objective is too singular and fails to adapt to multi-dimensional constraint scenarios.

[0021] Most existing operator optimization efforts primarily focus on maximizing computational performance (such as reducing latency or increasing throughput) as a single optimization objective. However, in real-world deployment scenarios, especially in power-sensitive edge devices or cost-sensitive cloud inference services, developers must make complex trade-offs between performance, energy consumption, and memory footprint. Existing technologies lack a unified decision-making framework that supports multi-objective optimization, failing to generate matching optimization strategies based on specific application requirements (e.g., minimum power consumption or minimum memory footprint), thus significantly limiting their applicability in practical applications.

[0022] Third, static analysis models are rigid and lack dynamic adaptability and synergistic capabilities.

[0023] Current optimization processes are mostly one-off static analyses. The compiler optimizes operators based on a fixed cost model during compilation, and once executable code is generated, its optimization strategy is fixed. This approach cannot adapt to dynamic changes in the runtime environment, such as system load, temperature variations, or changes in data distribution. More importantly, existing methods treat the optimization of each operator as an isolated problem, lacking a global perspective to explore opportunities for collaborative optimization between operators. For example, a memory layout adjustment for one operator can have a profound impact on the cache hit rate of adjacent operators, and such complex cross-operator dependencies are difficult to effectively model and utilize with current technology.

[0024] Fourth, the model lacks generalization ability and lags behind in supporting emerging operators and architectures.

[0025] Learning-based optimization methods heavily rely on large amounts of high-quality training data. When faced with emerging and structurally unique AI models (such as graph neural networks or Transformer variants with irregular computational patterns), existing cost models often suffer a sharp drop in prediction accuracy due to a lack of relevant training data. Furthermore, with the rapid iteration of hardware, every time a new NPU architecture or instruction set emerges, significant resources are required to re-collect data and retrain models. This results in the iteration speed of optimization frameworks lagging far behind the pace of hardware and algorithm innovation, leading to high ecosystem adaptation costs.

[0026] Therefore, there is an urgent need for a new type of automated optimization engine that can not only deeply understand the microarchitectural characteristics of heterogeneous hardware, but also make intelligent decisions under multidimensional constraints, and has the ability to dynamically adapt and globally coordinate optimization, thereby providing a truly efficient, flexible and scalable compilation optimization solution for the next generation of AI models and hardware.

[0027] This application proposes an innovative engine, specifically named MOSAIC (Multi-path Operator Synthesis and Adaptive Intelligence Coordinator), designed for deep analysis and automated optimization of AI operators on heterogeneous computing platforms, particularly systems containing ARM CPUs and dedicated NPUs. MOSAIC, through a hierarchical, multi-path intelligent system, overcomes the bottlenecks encountered by traditional optimization methods when facing complex hardware architectures and diverse operator characteristics, achieving an end-to-end automated optimization process from feature analysis to code generation.

[0028] Specifically, to address the multi-dimensional challenges of operator optimization in heterogeneous environments, this application abandons traditional single-model paths or static rule-based methods. These traditional methods struggle to capture the highly nonlinear mapping between the intrinsic characteristics of operators and the unique performance characteristics of heterogeneous hardware (such as ARM multi-core CPUs and NPU multi-engines). Therefore, MOSAIC is designed as a deep learning-based, end-to-end intelligent analysis and optimization system. Its core idea is to achieve refined perception of operator characteristics and optimal utilization of hardware resources through multi-path parallel processing and expert dynamic routing mechanisms.

[0029] The following is a detailed description of the scheme proposed in this application.

[0030] This application provides an operator-based intelligent optimizer based on multi-path parallelism and expert hybrid optimization. Figure 1 This is a schematic diagram of the intelligent optimizer for this operator. Specifically, the optimizer includes: a multidimensional feature extraction and encoding layer 11, a MoE dynamic routing and expert collaboration layer 12, and an adaptive decision-making and code generation layer 13; wherein: The multidimensional feature extraction and encoding layer 11 is used to extract and encode features of each AI operator in the input AI operator set, obtain the encoding results of each AI operator, and then obtain the encoding result set; MoE Dynamic Routing and Expert Collaboration Layer 12 is used to distribute the encoded result set to multiple diagnostic expert networks, and to diagnose the encoded result set based on the multiple diagnostic expert networks to obtain the diagnostic results of each diagnostic expert network. The adaptive decision-making and code generation layer 13 is used to select a strategy combination that best matches the AI ​​operator set from a preset strategy library based on the diagnostic results of each diagnostic expert network, and compile the strategy combination into code that can be directly executed on the target heterogeneous platform.

[0031] Optionally, for each coding result in the coding result set, each diagnostic expert network makes a corresponding diagnostic result.

[0032] Optionally, the strategy combination that best matches the set of AI operators can refer to the strategy combination that best satisfies the requirements of the AI ​​operators in the set of AI operators.

[0033] Optionally, the strategy combination in this application refers to the hardware combination of data for processing the set of AI operators.

[0034] The overall architecture of the optimizer can be summarized as a three-layer model including a multi-dimensional feature extraction and encoding layer 11, a MoE dynamic routing and expert collaboration layer 12, and an adaptive decision-making and code generation layer 13.

[0035] Optionally, the multidimensional feature extraction and encoding layer 11 includes: a multidimensional feature extraction layer 111 and a multi-path parallel encoder 112; The multidimensional feature extraction layer 111 is used to extract features from each AI operator in the AI ​​operator set to obtain a 192-dimensional feature vector corresponding to each AI operator. Optionally, the feature vector includes: computational intensity and memory access pattern, hardware topology, data flow dependency, and estimated energy consumption pattern; Specifically, in the multidimensional feature extraction and encoding layer 11, a feature vector extended to 192 dimensions is extracted from each AI operator. This feature vector not only contains computational intensity and memory access patterns, but also adds key information such as hardware topology, data flow dependence, and estimated energy consumption patterns.

[0036] Optionally, the model file is scanned, which is represented as a graph. Each node in the graph corresponds to an operator. The corresponding CUDA code for the operator is queried, and information such as the memory, registers, computation methods, data stations, hardware topology, data flow, and energy consumption estimates used in the code are scanned. Then, the information is concatenated and converted into a 192-dimensional vector feature.

[0037] The multi-path parallel encoder 112 is used to perform multi-path parallel encoding on the 192-dimensional feature vectors corresponding to each AI operator to obtain the 384-dimensional encoding result corresponding to each AI operator, and then obtain the encoding result set.

[0038] Optionally, the multi-path parallel encoder includes three paths: an 8-layer computationally intensive path, a 6-layer memory-intensive path, and a 4-layer communication-intensive path.

[0039] In other words, the feature vectors corresponding to each AI operator are fed into a multi-path parallel encoder 112, which contains three independent, specialized Transformer paths: an 8-layer computationally intensive path, a 6-layer memory-intensive path, and a 4-layer communication-intensive path. This parallel structure can simultaneously encode operator characteristics from different dimensions and extend the hidden layer dimension to 384 dimensions to enhance the model's expressive power.

[0040] Optionally, the MoE dynamic routing and expert collaboration layer 12 is also used to select the plurality of diagnostic expert networks with the highest matching degree with the AI ​​operator subset from a plurality of candidate expert networks.

[0041] Optionally, the multiple candidate expert networks are divided into three categories: hardware expert networks, optimization strategy expert networks, and bottleneck diagnosis expert networks.

[0042] The outputs of the hardware expert network include: power consumption, efficiency, throughput, memory usage, process and thread usage, and concurrency effects of the operators running on the chip.

[0043] The output of the optimization strategy expert network includes: operators or hardware that can be memory merged, which hardware can share memory, which locations can be parallelized by threads, and the range of parameter settings. The output of the bottleneck diagnosis expert network includes: which parts of the code are time-consuming, and where memory allocation is uneven.

[0044] Optionally, the bottleneck diagnosis expert network can identify four bottleneck types: computational bottleneck, memory access bottleneck, bandwidth bottleneck, and synchronization bottleneck. A softmax classifier outputs the probability distribution for each bottleneck type and provides a corresponding confidence score, offering precise guidance for subsequent optimization decisions. Optionally, a collaborative attention mechanism exists between the multiple diagnosis expert networks, enabling information exchange and knowledge fusion among them. The multiple diagnosis expert networks interact with each other through question-and-answer sessions, providing feedback and correcting opinions until they reach a consensus.

[0045] Specifically, after obtaining the encoded result set, the data enters the MoE dynamic routing and expert collaboration layer 12. This layer dynamically allocates the encoded result set to the Top-K (e.g., K=4) most relevant experts from an "expert pool" containing 12 specialized experts (i.e., the aforementioned network of multiple diagnostic experts with the highest matching degree). These experts are divided into three categories: hardware, optimization strategies, and bottleneck diagnosis, and can collaboratively handle specific subtasks.

[0046] Optionally, the prediction and judgment processes in this application can be implemented using a large language model.

[0047] In this application, a MoE model with 12 experts is constructed based on CHCA (Cross-Hardware Collaborative Attention). This model uses a lightweight gating network to calculate routing scores for each operator based on the fusion features of the previous layer's output, and dynamically selects the four most suitable experts (which can be selected from a trained MoE large language expert model) for activation. This design not only significantly improves the model's capacity and expressive power but also achieves high computational efficiency, as only a subset of experts are activated during each inference. The expert pool (i.e., multiple candidate expert networks) is carefully designed into the following three groups: Hardware expert networks (4): Modeling the characteristics of ARM A-series cores, ARM B-series cores, NPU matrix engine, and NPU vector engine respectively.

[0048] Optimization Strategy Expert Network (4 members): Each expert specializes in key optimization areas such as operator fusion, memory layout rearrangement, parallel scheduling, and numerical precision transformation.

[0049] Bottleneck Diagnosis Expert Network (4 members): These are used to diagnose performance bottlenecks at four finer granular levels: computation, memory access, bandwidth, and synchronization.

[0050] In addition, this application also designs a cross-expert collaborative attention mechanism, which allows activated experts to interact and integrate information, thereby producing a more comprehensive and accurate judgment than independent analysis by a single expert.

[0051] The output of this application is a comprehensive optimization scheme comprising multiple components. The hardware expert network can accurately predict the execution latency, peak memory usage, and expected power consumption of operators on ARM and NPU under different core / engine combinations. This prediction is based on end-to-end inference results across the entire three-tier architecture, and its multi-path design and expert collaboration mechanism ensure high-precision modeling of complex nonlinear performance mapping relationships.

[0052] Finally, in the adaptive decision-making and code generation layer 13, the analysis results from the diagnostic expert network can be integrated. Through a multi-objective optimization decision-maker, performance, energy consumption and memory usage are comprehensively evaluated. An optimal strategy combination is generated from a library containing 64 basic optimization strategies and finally compiled into optimized code that can be directly executed on the target heterogeneous platform.

[0053] The adaptive decision-making and code generation layer 13 is the core decision-making unit of this application. Instead of generating a single optimization suggestion, it generates a Pareto-optimal policy sequence by combining and sorting policies from a policy library containing 64 basic optimization strategies. This module employs a multi-objective optimization framework, simultaneously considering performance improvement, energy reduction, and memory optimization as optimization objectives, and allows users to adjust the priority of different objectives by setting weights (W = {w_perf (computational performance), w_energy (energy consumption), w_memory (memory)}). The final output includes not only recommended policies but also a globally graph-optimized policy set and a directly deployable hardware-aware compiler backend that can convert the optimal policies into efficient ARM-NPU hybrid executable code.

[0054] The core innovation of this application lies in the deep integration of its Cross-Hardware Collaborative Attention (CHCA) mechanism with a Mixture-of-Experts (MoE) dynamic routing model. Traditional attention models, even when differentiating between processor types, fail to adequately model the multi-core / multi-engine architecture and complex collaborative relationships within modern heterogeneous chips. The CHCA mechanism addresses this issue by introducing a finer-grained, dedicated attention computation mode for specific hardware microarchitectures. Specifically, it designs independent attention computation branches for the "big.LITTLE" architecture of ARM processors and the "multi-engine" architecture of NPUs (such as matrix computation engines and vector computation engines), and adjusts these branches through learnable weight factors, which can be set empirically. ARM Multi-core Collaborative Attention: CHCA_ARM(Q,K,V) = β_big · Attention_big(Q,K,V) + β_little ·Attention_little(Q,K,V) NPU multi-engine collaborative attention: CHCA_NPU(Q,K,V) = γ_matrix · Attention_matrix(Q,K,V) + γ_vector ·Attention_vector(Q,K,V) Here, β_big and β_little are core affinity weights used to balance the computing resources of ARM big and small cores, while γ_matrix and γ_vector are balancing factors used to balance the load of different computing units within the NPU. Attention_big(Q,K,V) represents the attention of the ARM processor's big cores, and Attention_little(Q,K,V) represents the attention of the ARM processor's little cores. Furthermore, this mechanism introduces a communication overhead-aware factor δ_comm to model the potential overhead of cross-processor communication when finally fusing the attention outputs of ARM and NPU, thereby making the decision closer to real hardware behavior. Cross-hardware converged attention: Attention_fused=LayerNorm(CHCA_ARM(Q,K,V)+CHCA_NPU(Q,K,V)-δ_comm·CommCost).

[0055] Here, Attention_fused refers to cross-hardware fused attention, and CommCost refers to the operator's overhead. In some optional embodiments, the complete workflow of the optimizer in this application follows a phased, closed-loop optimization pattern, ensuring the depth of analysis and continuous improvement of optimization. Specifically, it can be summarized into the following stages: 1. Phase One: Multi-path Feature Extraction and Encoding: The algorithm receives the operator set O, hardware configuration H, and optimization objective W as input. First, 192-dimensional enhanced features are extracted for each operator. Then, they are encoded through three parallel Transformer paths: computation, memory access, and communication. Finally, the three outputs are fused into a unified feature representation.

[0056] 2. Phase Two: Cross-Hardware Collaborative Attention Calculation: Based on hardware configuration H, attention masks are generated for ARM big.LITTLE cores and NPU multi-engines. Subsequently, the collaborative attention of each hardware unit is calculated through the CHCA mechanism, and finally fused to generate a cross-hardware feature representation that takes into account communication overhead.

[0057] 3. Phase Three: MoE Dynamic Routing and Expert Analysis: The gated routing network selects Top-4 experts for each operator based on the fused features. The activated experts process the input in parallel, and their outputs undergo knowledge fusion through a cross-expert attention mechanism to form the final expert analysis opinion.

[0058] 4. Phase Four: Multi-objective Optimization and Policy Generation: This phase integrates performance prediction, bottleneck diagnosis, and policy generation. The system first predicts the multi-dimensional performance indicators of the operator on different hardware, and then, based on the bottleneck diagnosis results, performs multi-objective scoring and Pareto ranking from 64 candidate policies to select the optimal policy.

[0059] 5. Phase Five: Graph-level Global Optimization: After obtaining the optimal strategy for a single operator, the system will construct the entire computation graph and perform graph-level global optimization, such as exploring more operator fusion opportunities.

[0060] 6. Phase Six: Hardware-Aware Code Generation: Finally, the globally optimized policy set is fed into the code generator and compiled into efficient hybrid executable code for a specific ARM-NPU platform.

[0061] 7. Phase Seven: Adaptive Feedback and Continuous Optimization (Optional): After system deployment, MOSAIC can collect actual runtime performance data and compare it with the model's predictions. Through an online learning mechanism, the system can use this feedback data to continuously fine-tune the performance prediction model, the gated routing network, and expert weights, thereby forming a continuously self-improving closed-loop optimization system.

[0062] This application innovatively proposes a multi-path parallel encoder architecture, replacing the traditional single-path processing model. This architecture comprises three independent, specialized Transformer encoding paths, respectively used for deep analysis of the computationally intensive, memory-intensive, and communication-intensive features of operators. This design enables the model to decouple and finely perceive operator characteristics from different dimensions and with different granularities, and captures more comprehensive and in-depth intrinsic operator properties than existing technologies through a 192-dimensional enhanced feature vector. This differs from the single, generalized feature extraction and encoding methods in existing technologies.

[0063] Furthermore, this study is the first to combine a Hybrid Expert (MoE) model with a novel Cross-Hardware Collaborative Attention (CHCA) mechanism for heterogeneous operator optimization. The model comprises a pool of 12 specialized experts and a dynamically gated routing network. The CHCA mechanism enables fine-grained modeling of the microarchitectural characteristics within heterogeneous processors, such as ARM's big.LITTLE cores and NPU's multiple computational engines (matrix / vector), and explicitly incorporates cross-processor communication overhead. The MoE model then intelligently distributes tasks to the most relevant experts based on the CHCA analysis results. This differs from existing static and homogeneous analysis models that are unaware of hardware microarchitectural details.

[0064] Secondly, this application establishes an adaptive decision-making framework supporting multi-objective optimization. This framework expands the operator optimization objective from a single performance dimension to a comprehensive consideration of three key dimensions: performance, energy consumption, and memory usage. Through a configurable weighted multi-objective scoring function and a Pareto-optimal ranking-based policy selector, the system can intelligently weigh different constraints according to the user's specific needs and generate one or more optimal, non-dominated optimization policy combinations from an extended policy library containing 64 basic policies. This differs from existing decision-making methods that commonly use performance as the sole optimization objective.

[0065] Furthermore, this application implements a more advanced end-to-end automated process from operator analysis to code generation. The system can not only generate optimal policies for individual operators but also perform global optimization at the computation graph level, exploring optimization opportunities across operators. More importantly, this application designs an optional adaptive feedback learning loop, enabling the system to monitor actual runtime performance data, compare it with model predictions, and continuously and automatically update and calibrate its internal prediction model, routing network, and expert weights using an online learning mechanism. This differs from the open, one-off, static compilation optimization processes in existing technologies.

[0066] This application can achieve the following technical effects: 1. Extremely high prediction accuracy and optimization effectiveness: Thanks to the multi-path parallel coding architecture's refined deconstruction and perception of operator characteristics, and the CHCA mechanism's deep insight into hardware microarchitecture, the performance prediction model in this application can establish a high-fidelity "operator-hardware" mapping relationship. This enables the system to predict the performance, power consumption, and memory behavior of operators on heterogeneous platforms with extremely high accuracy, and to precisely pinpoint the root cause of performance bottlenecks. Therefore, the optimization strategy generated based on this precise analysis is far more effective than traditional methods based on coarse-grained hardware abstraction, and can tap into the hardware potential to a greater extent.

[0067] 2. Excellent flexibility, scalability, and applicability: The decision-making framework supporting multi-objective optimization endows this application with unprecedented flexibility. Users can dynamically adjust optimization objectives according to different application scenarios (such as cloud training pursuing ultimate performance or edge inference emphasizing energy efficiency), enabling the same system to serve diverse business needs. Meanwhile, the MoE-based expert system has inherent scalability; when new hardware architectures or novel optimization strategies are needed, new experts can simply be added to the expert pool as "plugins," greatly reducing system expansion and maintenance costs and improving adaptability to future technological developments.

[0068] 3. True hardware awareness and maximum resource utilization: By explicitly modeling the ARM big.LITTLE cores, NPU multi-engines, and the communication overhead between them, this application achieves true microarchitecture-level hardware awareness. This enables the system to intelligently allocate the most suitable part of the computing task to the most suitable computing unit, just like an experienced system expert, during task scheduling and resource allocation. This achieves optimal collaboration of heterogeneous resources, avoids idle or overloaded resources, and ultimately improves the hardware resource utilization and energy efficiency of the entire system to a new level.

[0069] 4. Continuously evolving level of automation and intelligence: The introduction of an adaptive feedback learning loop transforms this application from a static analysis tool into an intelligent system capable of continuous learning and self-improvement. It automatically learns "experience" from real-world operational data, constantly optimizing its "judgment" to ensure that its optimization effectiveness does not diminish over time or with changes in workload. This closed-loop, adaptive characteristic represents a higher level of automation and intelligence, significantly reducing reliance on human intervention and ensuring the system's robustness and advanced nature in long-term operation.

[0070] This application also provides an operator-based intelligent optimization method based on multi-path parallelism and expert hybrid approach, including the following steps S1-S3: S1. Extract and encode features from each AI operator in the input AI operator set to obtain the encoding results of each AI operator, and then obtain the encoding result set. S2. Distribute the encoded result set to multiple diagnostic expert networks, and perform diagnosis on the encoded result set based on the multiple diagnostic expert networks to obtain the diagnostic results of each diagnostic expert network; S3. Based on the diagnostic results of each diagnostic expert network, select the strategy combination that best matches the AI ​​operator set from the preset strategy library, and compile the strategy combination into code that can be directly executed on the target heterogeneous platform.

[0071] Furthermore, the scheme of this application can also be found in [reference needed]. Figure 2 As shown. For detailed implementation of this embodiment, please refer to the foregoing content; it will not be repeated here.

[0072] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 3The electronic device 300 shown in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304.

[0073] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0074] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0075] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory.

[0076] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation methods described above in the embodiments of this application, or they can execute the implementation methods of the electronic devices described in the embodiments of this application, which will not be repeated here.

[0077] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0078] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0079] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0080] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0082] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0083] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0084] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An operator-based intelligent optimizer based on multi-path parallelism and expert hybrid optimization, characterized in that, include: The multidimensional feature extraction and encoding layer is used to extract and encode features from each AI operator in the input AI operator set, obtain the encoding results of each AI operator, and then obtain the encoding result set. The MoE dynamic routing and expert collaboration layer is used to distribute the encoded result set to multiple diagnostic expert networks, and to diagnose the encoded result set based on the multiple diagnostic expert networks to obtain the diagnostic results of each diagnostic expert network. An adaptive decision-making and code generation layer is used to select a strategy combination that best matches the AI ​​operator set from a preset strategy library based on the diagnostic results of each diagnostic expert network, and compile the strategy combination into code that can be directly executed on the target heterogeneous platform.

2. The optimizer as described in claim 1, characterized in that, The multidimensional feature extraction and encoding layer includes a multidimensional feature extraction layer and a multi-path parallel encoder; The multidimensional feature extraction layer is used to extract features from each AI operator in the AI ​​operator set to obtain a 192-dimensional feature vector corresponding to each AI operator. The multi-path parallel encoder is used to perform multi-path parallel encoding on the 192-dimensional feature vectors corresponding to each AI operator to obtain the 384-dimensional encoding result corresponding to each AI operator, and then obtain the encoding result set.

3. The optimizer as described in claim 2, characterized in that, The multi-path parallel encoder comprises three paths: an 8-layer compute-intensive path, a 6-layer memory-intensive path, and a 4-layer communication-intensive path.

4. The optimizer as described in claim 1, characterized in that, The MoE dynamic routing and expert collaboration layer is also used to select the multiple diagnostic expert networks with the highest matching degree with the AI ​​operator subset from multiple candidate expert networks.

5. The optimizer as described in claim 4, characterized in that, The multiple candidate expert networks are divided into three categories: hardware expert networks, optimization strategy expert networks, and bottleneck diagnosis expert networks.

6. The optimizer as described in claim 1, characterized in that, There is a collaborative attention mechanism among the multiple diagnostic expert networks, and information exchange and knowledge fusion exist between different diagnostic expert networks.

7. The optimizer as described in claim 5, characterized in that, The bottleneck diagnosis expert network can identify the following four types of bottlenecks: computation bottleneck, memory access bottleneck, bandwidth bottleneck, and synchronization bottleneck.

8. An operator-based intelligent optimization method based on multi-path parallelism and expert hybrid approach, characterized in that, include: Feature extraction and encoding are performed on each AI operator in the input AI operator set to obtain the encoding result of each AI operator, and then the encoding result set is obtained. The encoded result set is distributed to multiple diagnostic expert networks, and the encoded result set is diagnosed based on the multiple diagnostic expert networks to obtain the diagnostic results of each diagnostic expert network; Based on the diagnostic results of each diagnostic expert network, a strategy combination that best matches the AI ​​operator set is selected from the preset strategy library, and the strategy combination is compiled into code that can be directly executed on the target heterogeneous platform.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in claim 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in claim 8.

Images