Heterogeneous edge neural network deployment method and system, and electronic device

Abstract

This application provides a method, system, and electronic device for deploying heterogeneous edge neural networks. The method addresses the adaptation difficulties caused by toolchain dialect barriers and knowledge dispersion by constructing a knowledge base and integrating fragmented hardware knowledge. It replaces scripts that rely on fixed templates by assembling prompt words based on model profiles and target hardware types to generate code, enabling the deployment logic to flexibly adapt to different hardware and model combinations. Through containerized sandbox execution and adaptive correction iterative closed loop based on retrieval and intelligent reasoning, it avoids interruptions upon encountering errors. Execution failures no longer require manual intervention for debugging but instead trigger a diagnosis and repair process until success. Therefore, this method can improve the automation, robustness, and success rate of the deployment process in edge computing scenarios with highly heterogeneous hardware and rapidly evolving toolchains.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence engineering and edge computing technology, and in particular to a method, system and electronic device for deploying heterogeneous edge neural networks. Background Technology

[0002] Edge intelligence technology can be applied to scenarios such as the Industrial Internet of Things (IIoT). To meet power consumption and computing power requirements, it can be implemented using edge computing hardware. However, edge computing hardware differs and is incompatible with each other in terms of architecture, instruction set, programming interface, and toolchain. Adapting and deploying trained deep learning models to heterogeneous hardware involves complex engineering conversion and compilation processes.

[0003] To adapt to complex engineering conversion and compilation processes, template-based conversion tools or parameter-search-based AutoML tools can be used. These tools generate deployment configurations based on predefined rules or parameter spaces, reducing manual coding. However, toolchains provided by different hardware are incompatible; scripts based on conversion tools struggle to adapt to changes, leading to deployment interruptions. Furthermore, the decision-making knowledge required for deployment resides in readable, unstructured documents, while the knowledge of the aforementioned tools is also contained within unstructured documents. Conversion tools cannot understand the reasons for errors, resulting in low deployment success rates. Summary of the Invention

[0004] This application provides a method, system, and electronic device for deploying heterogeneous edge neural networks to solve the problem of low deployment success rate.

[0005] In a first aspect, this application provides a method for deploying heterogeneous edge neural networks, including: A knowledge base is constructed, which is an independent knowledge base built based on unstructured technical documents of heterogeneous edge hardware; Based on the model profile and target hardware type of the neural network model, target knowledge is retrieved from the knowledge base to generate prompt words; Based on the prompt words, conversion and compilation code is generated and input into a preset containerized sandbox environment to execute the conversion and compilation code of the neural network model and output the execution result; If the execution result is a failure, the error information during the execution process will be used as a query vector to retrieve the knowledge base and obtain the relevant solution. Based on the aforementioned correlation solution, the conversion and compilation code is adaptively corrected and iterated until the execution result is successful, and the target model file is output. The target model file is distributed to the target edge device for deployment.

[0006] Secondly, this application provides a heterogeneous edge neural network deployment system, comprising: A knowledge base construction module is used to build a knowledge base, which is an independent knowledge base built based on unstructured technical documents of heterogeneous edge hardware. The semantic analysis module, based on the model profile of the neural network model and the target hardware type, retrieves target knowledge from the knowledge base to generate prompt words; The code generation module generates conversion and compilation code based on the prompt words, and inputs the conversion and compilation code into a preset containerized sandbox environment; The containerized sandbox execution module is used to execute the conversion and compilation code of the neural network model to output the execution results; The self-healing module is used to retrieve the knowledge base by using the error information during the execution process as a query vector when the execution result is failure, in order to obtain an associated solution; and to adaptively correct and iterate the transformation and compilation code based on the associated solution until the execution result is success, and output the target model file. The deployment module distributes the target model file to the target edge device to perform deployment.

[0007] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the heterogeneous edge neural network deployment method as described in any of the first aspects.

[0008] As can be seen from the above technical solutions, this application provides a heterogeneous edge neural network deployment method, system, and electronic device. The method, by constructing a knowledge base and integrating fragmented hardware knowledge, solves the adaptation difficulties caused by toolchain dialect barriers and knowledge dispersion. By assembling prompt words and generating code based on model profiles and target hardware types, it replaces scripts that rely on fixed templates, enabling the deployment logic to flexibly adapt to different hardware and model combinations. Through containerized sandbox execution and adaptive correction iterative closed loop based on retrieval and intelligent reasoning, it avoids interruption upon encountering errors. Execution failures no longer require manual intervention for debugging but instead trigger a diagnosis and repair process until success. Therefore, in edge computing scenarios with highly heterogeneous hardware and rapidly evolving toolchains, this method can improve the automation, robustness, and success rate of the deployment process. Attached Figure Description

[0009] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1A schematic diagram illustrating the neural network deployment process provided in an embodiment of this application; Figure 2 This is a schematic diagram of the heterogeneous edge neural network deployment method provided in the embodiments of this application; Figure 3 A schematic diagram of a containerized sandbox environment provided in an embodiment of this application; Figure 4 This is a schematic diagram of the code self-healing process provided in an embodiment of this application. Detailed Implementation

[0011] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following examples do not represent all embodiments consistent with this application.

[0012] Heterogeneous edge computing refers to edge computing devices with different computing architectures. These devices include chips using neural network processor architectures, such as the Rockchip RK series; chips using dataflow architectures, such as the Hailo series; and chips using graphics processor architectures, such as the NVIDIA Jetson series. These hardware devices differ in their instruction sets and software toolchains.

[0013] Neural network deployment is the process of adapting a trained model to the aforementioned edge devices. Deployment occurs after the model training is completed, during the inference phase on the edge computing device. Deployment work includes model format conversion, operator adaptation, and compilation optimization.

[0014] Deploying neural networks on edge devices offers several advantages. Firstly, it allows models to perform inference directly where the data is generated, reducing reliance on network bandwidth and lowering latency when uploading data to the cloud. Secondly, it better meets data privacy requirements, as raw data can be processed locally.

[0015] For example, in industrial quality inspection scenarios, deployed models can identify product defects in real time. In smart security scenarios, deployed models can perform instant analysis of video streams.

[0016] like Figure 1 As shown, the specific process is as follows: First, model format conversion is performed. For example, the model of the training framework is converted to the Open Neural Network Exchange format. The purpose of the conversion is to obtain an intermediate representation that is generally accepted by hardware vendor toolchains. After the conversion, model optimization is performed. Optimization may include operator fusion to reduce computational and memory access overhead. Optimization may also include quantization, which converts model parameters from floating-point numbers to integers. The purpose of quantization is to reduce computational requirements and speed up inference.

[0017] Developers need to call the compiler toolchain specific to the target hardware. The compiler compiles the optimized model diagram into device-executable binary code. This process requires handling hardware-specific operator implementations and memory layouts, and often necessitates adjusting compilation parameters to adapt to the model structure. Successful compilation generates a proprietary model file, which needs to be integrated into the edge device's application. Developers write application code responsible for loading the model file and managing input / output data streams. The application also needs to handle interfaces with sensors or other systems.

[0018] Developers deploy their applications and model files to physical devices and run the applications to verify functionality and performance. This stage often encounters problems, such as unsupported operators or substandard performance. Developers need to consult documentation, modify configurations, or adjust code to solve these problems.

[0019] When deploying neural networks on heterogeneous hardware, the compilation toolchains provided by different hardware vendors, such as RKNN-Toolkit2, Hailo Dataflow Compiler, and TensorRT, have drastically different API design philosophies, configuration file formats, and quantization rules—a kind of dialect barrier. Furthermore, toolchain SDK versions are updated extremely rapidly, and API interfaces change frequently. Traditional rule-based or template-based transformation tools immediately become invalid when the SDK is upgraded, leading to an exponential increase in maintenance costs.

[0020] During model conversion, quantization, or compilation, complex low-level error messages may appear, such as "Layer mismatch" or "Unsupported op specific attribute". Some technologies can only detect the exception and terminate the automation process, but cannot understand the logical reasons behind the error. This makes the automation link fragile and still requires manual intervention from experts to consult documents and debug code, thus failing to achieve true automation.

[0021] Operator constraints, performance optimization techniques, and best practices for specific hardware are often scattered across unstructured data such as official PDF documents, developer forums, and GitHub Issues. Traditional parameter search optimization methods cannot effectively utilize this natural language knowledge to guide the deployment process.

[0022] To address the aforementioned issues, some embodiments of this application provide a method for deploying heterogeneous edge neural networks. This method utilizes retrieval enhancement techniques to acquire and understand development documents and practical data from different hardware in real time, resolving the knowledge obsolescence problem caused by toolchain version iterations. It also leverages a large language model to automatically write complete and executable conversion and quantization source code based on model characteristics. Furthermore, it establishes an automated closed loop for generation, execution, diagnosis, and repair, automatically analyzing toolchain error logs and reversing code logic until deployment is successful.

[0023] like Figure 2 As shown, the process includes the following steps S100-S600: S100: Construct a knowledge base, which is an independent knowledge base built on unstructured technical documents of heterogeneous edge hardware.

[0024] Heterogeneous edge computing devices are edge computing devices that are incompatible in terms of computing architecture, instruction set, and supporting software toolchain. For example, they may use chips with different architectures such as neural network processors, data stream processors, or graphics processors. It is understood that this application does not hardcode any hardware adaptation logic, but rather maintains a vectorized knowledge base specific to edge hardware.

[0025] These unstructured technical documents may include official SDK documentation for the target hardware, API reference manuals, official model library sample code, and frequently asked questions. The knowledge base uses information extraction and vectorization techniques to transform the document content into a format easily searchable by machines; for example, it forms a vector database that stores semantic information about operator support, configuration parameters, and error codes.

[0026] The above unstructured text can be transformed into a vector index using an embedding model. Key engineering information such as the operator support list, detailed explanation of quantization configuration parameters, and meaning of error codes can be extracted to construct a high-dimensional vector database.

[0027] For example, the latest version of the "RKNN Toolkit2 User Guide" PDF document released by Rockchip and the issue discussion records in the GitHub repository are obtained through the crawler module. These unstructured texts are processed by chunking and embedding and then stored in the vector database.

[0028] S200: Based on the model profile and target hardware type of the neural network model, retrieve target knowledge from the knowledge base to generate prompt words.

[0029] When deploying a neural network model, the target hardware type is first specified, identifying the specific target device platform. Simultaneously, the input neural network model file is parsed, and by analyzing its computational graph topology, a model profile describing its structural characteristics is generated. This profile details the model's input and output dimensions, included special operators (such as Deformable Conv, SiLU), network depth, and complexity, but does not include specific weight values.

[0030] In some embodiments, before generating the prompt words, the process includes: parsing the input neural network model file to obtain the computation graph topology; identifying the model's input / output dimensions, operator types, and network depth information from the computation graph topology; and generating the model profile based on the input / output dimensions, operator types, and network depth information.

[0031] For example, a user uploads an ONNX model file through the system front-end and selects the target platform, i.e., the target hardware type is "Rockchip RK3588". The model parser is called to analyze the ONNX file, identify that the model contains several convolutional layers, SiLU activation layers and a complex post-processing header, generate descriptive text, indicate the input node names of the model and the characteristic of having dynamic dimensions, and thus obtain a model profile.

[0032] By parsing model files to obtain the computation graph topology and identifying key structural information, a model profile that accurately describes the model's features can be generated, providing reliable input for subsequent targeted retrieval and code generation.

[0033] Specifically, in some embodiments, based on the model profile of the neural network model and the target hardware type, target knowledge is retrieved from the knowledge base to generate prompt words, including: using the target hardware type as a first search condition, retrieving toolchain usage examples of the hardware from the knowledge base; using the operator type in the model profile as a second search condition, retrieving operator constraint and optimization knowledge from the knowledge base; and combining the toolchain usage examples with the operator constraint and optimization knowledge to assemble prompt words for guiding code generation.

[0034] After obtaining the model profile, the corresponding toolchain usage examples and operator constraint knowledge are retrieved from the knowledge base based on the target hardware type and the operator type in the model profile. These two parts of knowledge are combined and assembled into a specific system prompt word. This prompt word includes both the programming specifications of the target hardware and the structural characteristics of the current model.

[0035] By combining toolchain examples of hardware types and constraint knowledge related to model operators to assemble prompt words, the generated prompt words can simultaneously cover hardware interface specifications and model adaptation requirements, guiding the generation of more targeted deployment code.

[0036] S300: Based on the prompt words, generate conversion and compilation code, and input the conversion and compilation code into a preset containerized sandbox environment to execute the conversion and compilation code of the neural network model and output the execution result.

[0037] In this embodiment, the conversion and compilation code is generated through a large language model. The large language model is an artificial intelligence model that has been pre-trained on a large amount of text and code data and has powerful language understanding and generation capabilities.

[0038] The large language model receives and understands prompt words, performs inference based on the task context described within them, and automatically generates a complete, executable conversion and compilation script (Python or C++). This script includes all necessary operations such as loading the model, configuring parameters, and calling the compiler. For example, a script for Hailo hardware could be written that includes the entire process of model parsing, quantization, and compilation.

[0039] Traditional methods require rewriting the adapter code with SDK updates. This application only requires updating the documentation files in the knowledge base, and the LLM can automatically learn the usage of the new API without modifying the core system code, thus decoupling the deployment logic from the toolchain version.

[0040] Specifically, in some embodiments, generating conversion and compilation code based on the prompt word includes: inputting the prompt word into a large language model to invoke the large language model to understand the task context, the task context including the target hardware type, model structure features, and conversion and compilation task; invoking the large language model to infer and generate code logic that conforms to the target hardware toolchain specification based on the task context; the large language model outputting an execution script based on the code logic; and using the execution script as the conversion and compilation code.

[0041] In this step, the large language model is used to receive and parse prompt words, understand the deployment tasks they imply, and generate corresponding program code based on them. The large language model achieves the understanding of natural language and programming language patterns through its internal neural network parameters.

[0042] The large model understanding transformation and compilation task ensures that the generated code logic fully covers the entire process from model loading, preprocessing, optimization to compilation and export, with no missing steps, so that the large language model can generate effective code instead of random text.

[0043] Continuing with the example above, the Large Language Model (LLM) Agent receives a task request and retrieves the standard conversion process of RKNN Toolkit2 from the knowledge base based on keywords such as "RK3588" and "ONNX model". Based on this context, the Agent writes a Python script whose logic includes: instantiating an RKNN object, loading the ONNX model, configuring the target platform as rk3588, setting the quantization parameter to i8, and calling the build interface.

[0044] By leveraging the deep understanding capabilities of large language models, this approach addresses the inability of other methods to flexibly handle complex and variable hardware and model combinations, enabling code generation to dynamically adapt to different deployment scenarios. Secondly, through reasoning capabilities based on massive knowledge pre-training, it constructs correct and complete code logic flows, resolving the inefficiency and error-prone nature of manual code writing or rule-based code assembly. Finally, by outputting complete executable scripts, it provides directly usable driving instructions for subsequent automated execution stages.

[0045] To ensure execution security and environmental consistency, a matching container image is selected from the pre-built image library based on the target hardware type and instantiated into a runtime container environment, i.e., a containerized sandbox environment.

[0046] In some embodiments, before inputting the converted and compiled code into a preset containerized sandbox environment, the method further includes building a containerized sandbox environment, specifically including: pre-setting container images including corresponding version toolchains for different types of heterogeneous edge hardware; matching the container image according to the target hardware type and instantiating it into a runtime container environment to establish a preset containerized sandbox environment.

[0047] A containerized sandbox environment is an isolated virtualized execution environment with a pre-built specific toolchain, such as a Docker container instance, which provides a simulated runtime space for transforming and compiling code that is highly consistent with the actual hardware toolchain.

[0048] In this embodiment, Docker containers containing corresponding toolchain environments can be pre-configured for different vendors such as Rockchip, Hailo, and Jetson. The generated scripts are injected into the corresponding sandboxes and simulated in an isolated virtualization environment. This process can complete the compilation of the model and the intermediate format conversion without connecting to physical devices.

[0049] like Figure 3As shown, a containerized sandbox environment can consist of a cluster of multiple pre-built images, but only one specific container is dynamically instantiated as a sandbox each time it is executed. For example, sandbox A is a Docker image built from Rockchip RK3588 and its specific version RKNN-Toolkit2; sandbox B is an image built from Hailo-8 and its Dataflow Compiler; and sandbox C is an image built from NVIDIA Jetson AGX Orin and its specific version TensorRT. These images form a toolkit, each encapsulating a complete, clean, version-locked hardware toolchain environment.

[0050] By pre-configuring and matching container images containing corresponding toolchains for different types of hardware, a sandbox environment is established, providing a highly consistent and isolated simulated execution environment for the generated conversion compilation code that is identical to the target hardware.

[0051] For example, start a Docker container pre-installed with the corresponding version of RKNN Toolkit2 and run the Python script described above. The generated conversion and compilation code is injected into this sandbox environment and executed. After execution, the sandbox environment returns the execution result.

[0052] The execution result indicates whether the execution process is successful or unsuccessful. If successful, ..., if the execution result is unsuccessful, the following steps in S400 are executed.

[0053] For example, if the execution fails, an exception can be thrown during script execution, and Stderr will display: "ValueError: layer 'Reshape_145' shape mismatch". This is because the Reshape operation in the model conflicts with the dynamic shape setting of the input during static compilation.

[0054] S400: If the execution result is a failure, the error information during the execution process is used as a query vector to retrieve the knowledge base to obtain a related solution.

[0055] If the execution fails, the method will not terminate but will initiate a self-healing process. In some embodiments, using the error information during execution as a query vector to retrieve the knowledge base to obtain an associated solution includes: obtaining the standard error stream and exception stack text output by the containerized sandbox environment as error information; converting the error information into a high-dimensional vector as the query vector; performing a similarity search in the vector database of the knowledge base to obtain text fragments semantically similar to the query vector; and determining the text fragments as the associated solution.

[0056] Containerized sandbox environments generate various runtime logs during code execution, including standard error streams and exception stack text. The standard error stream is the standard data channel used by the program to output error and warning information during runtime. It is used to capture abnormal state information generated during program execution. The exception stack text is a generated list of detailed information describing the location of the error and the call sequence; it is used to pinpoint the specific logical path in the code that caused the error.

[0057] Error messages are obtained by acquiring and integrating the content of the standard error stream and exception stack trace. The error message is a collection of all text content output by the containerized sandbox environment that describes the reason for the execution failure. A pre-trained text embedding model is called, and the full text of the error message is taken as input. The embedding model encodes the text and maps it into a fixed-length high-dimensional vector, i.e., the query vector.

[0058] Semantic similarity refers to the high degree of correlation between two text fragments in terms of meaning, the issues they describe, or the concepts they involve. Semantic similarity is a core criterion for judging the relevance of search results. Semantic similarity is measured by comparing the spatial distance between corresponding text vectors; the closer the distance, the more semantically similar the texts.

[0059] After analyzing and sorting the candidate list of search results, the text fragments with the highest similarity scores or the top few are selected. The original text content of these fragments is read and integrated or summarized, thus integrating the search results into a specific and actionable reference knowledge point. This solution text is no longer a messy error message, but a known solution for similar problems extracted from the historical experience base, providing high-quality input for the next step of the large language model to understand the root cause of the problem and generate corrective code.

[0060] S500: Based on the aforementioned association solution, the conversion compilation code is adaptively corrected and iterated until the execution result is successful, and the target model file is output.

[0061] After obtaining the above-mentioned correlation result scheme, since the error message describes the problem but cannot explain the cause of the error, it is necessary to provide knowledge or experience to solve the problem through the correlation solution. However, the error message and the correlation solution cannot match the specific context of the current code. In this embodiment, the application environment is defined by the current transformation and compilation code. Through the input of the three, the large language model can perform reliable reasoning to understand the reason for the current execution failure and then perform modification.

[0062] like Figure 4As shown, in some embodiments, the adaptive correction and iteration of the conversion compilation code based on the association solution includes: inputting the association solution, the error information, and the current conversion compilation code into a large language model to call the large language model to perform logical reasoning and obtain the failure reason; locating and modifying the erroneous logic segment in the conversion compilation code according to the failure reason to generate the corrected conversion compilation code; and performing the next round of iteration based on the corrected conversion compilation code.

[0063] The large language model performs deep logical reasoning, going beyond the surface of phenomena to find the root technical reasons for failure. After identifying the cause of failure, the large language model compares and analyzes the cause with the current code, locating the corresponding logically flawed paragraphs in the code. Subsequently, based on the cause of failure and related solutions, the large language model rewrites or adjusts the located erroneous paragraphs, generating a corrected converted and compiled code. This new code is then sent to the next iteration for execution, thus completing a full "correction-verification" cycle. This process simulates the thought process of an engineer debugging code, but achieves automation and continuous iteration.

[0064] For example, the LLM Agent uses the error message as a query to search the FAQ and Errata in the knowledge base again. The search results show that for a specific version of RKNN, when processing operations involving dynamic shape reshape operators, a mandatory static shape needs to be explicitly specified during the configuration phase. Based on this knowledge, the LLM Agent rewrites the Python script. In the new code, the LLM Agent adds a specific rknn.config function call, explicitly fixes the dimensions of the input tensor, and adjusts incompatible optimization options. The corrected script is run again in the sandbox and compiles successfully, generating the .rknn model file.

[0065] This process forms an iterative closed loop until the script executes successfully and exports the hardware-specific model file, or until the preset maximum number of retries is reached.

[0066] Once the model is successfully compiled into a hardware-specific format (such as .rknn, .hef, .engine), i.e., the target hardware file, it will be automatically distributed over the network to real edge devices (RK3588 boxes, Hailo accelerator cards, or Jetson development boards).

[0067] Traditional parameter search methods cannot resolve code logic errors (such as incorrect API call order or missing dependencies). This application possesses code understanding capabilities, automatically correcting syntax errors and logical loopholes, significantly improving the deployment success rate of complex models.

[0068] S600: Distribute the target model file to the target edge device to perform deployment.

[0069] The target edge device is the actual physical heterogeneous edge computing hardware device. The target edge device is the physical platform for the final operation and service of the model file. The target edge device is connected to the deployment management system through the network.

[0070] In some embodiments, distributing the target model file to the target edge device for deployment includes: transmitting the target model file to the target edge device; loading the target model file on the target edge device and running a benchmark test program to obtain test results; obtaining the test results and sending the test results back to the central server to perform deployment verification.

[0071] To ensure that the deployment is not just a formality but a real capability, a standardized benchmark program will be run on the device to perform actual inference operations on the model. After the run is completed, the resulting performance data will be collected and organized as the test results.

[0072] These test data are sent back to the initiator of the deployment process, namely the central server. The server uses this data to perform a final verification of the overall effect of the deployment. This process not only completes the physical deployment of the model, but more importantly, obtains credible performance evidence through real machine operation.

[0073] By using natural language as an intermediate layer, the huge differences in the underlying implementations of Rockchip, Hailo, and Jetson are eliminated. Users do not need to learn the completely different toolchain details of different manufacturers; they only need to provide standard model files.

[0074] Based on the above-described heterogeneous edge neural network deployment method, some embodiments of this application provide a heterogeneous edge neural network deployment system, including: A knowledge base construction module is used to build a knowledge base, which is an independent knowledge base built based on unstructured technical documents of heterogeneous edge hardware. The semantic analysis module, based on the model profile of the neural network model and the target hardware type, retrieves target knowledge from the knowledge base to generate prompt words; The code generation module generates conversion and compilation code based on the prompt words, and inputs the conversion and compilation code into a preset containerized sandbox environment; The containerized sandbox execution module is used to execute the conversion and compilation code of the neural network model to output the execution results; The self-healing module is used to retrieve the knowledge base by using the error information during the execution process as a query vector when the execution result is failure, in order to obtain an associated solution; and to adaptively correct and iterate the transformation and compilation code based on the associated solution until the execution result is success, and output the target model file. The deployment module distributes the target model file to the target edge device to perform deployment.

[0075] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the heterogeneous edge neural network deployment method as described in any of the preceding claims.

[0076] Similar parts between the embodiments provided in this application can be referred to mutually. The specific implementation methods provided above are only a few examples under the overall concept of this application and do not constitute a limitation on the scope of protection of this application. For those skilled in the art, any other implementation methods extended from the solution of this application without creative effort shall fall within the scope of protection of this application.

Claims

1. A method for deploying heterogeneous edge neural networks, characterized in that, include: A knowledge base is constructed, which is an independent knowledge base built based on unstructured technical documents of heterogeneous edge hardware; Based on the model profile and target hardware type of the neural network model, target knowledge is retrieved from the knowledge base to generate prompt words; Based on the prompt words, conversion and compilation code is generated and input into a preset containerized sandbox environment to execute the conversion and compilation code of the neural network model and output the execution result; If the execution result is a failure, the error information during the execution process will be used as a query vector to retrieve the knowledge base and obtain the relevant solution. Based on the aforementioned correlation solution, the conversion and compilation code is adaptively corrected and iterated until the execution result is successful, and the target model file is output. The target model file is distributed to the target edge device for deployment.

2. The heterogeneous edge neural network deployment method according to claim 1, characterized in that, Before generating the prompt words, the following are included: Parse the input neural network model file to obtain the computation graph topology; From the computation graph topology, identify the model's input / output dimensions, operator types, and network depth information; The model profile is generated based on the input / output dimensions, operator types, and network depth information.

3. The heterogeneous edge neural network deployment method according to claim 1, characterized in that, The model profile and target hardware type based on the neural network model are retrieved from the knowledge base to generate prompt words, including: Using the target hardware type as the first search condition, retrieve toolchain usage examples for that hardware from the knowledge base; Using the operator type in the model profile as the second search condition, retrieve operator constraint and optimization knowledge from the knowledge base; The toolchain usage examples are combined with the operator constraints and optimization knowledge to assemble prompts for generating guiding code.

4. The heterogeneous edge neural network deployment method according to claim 1, characterized in that, The step of generating conversion and compilation code based on the prompt words includes: The prompt words are input into a large language model to invoke the large language model to understand the task context, which includes the target hardware type, model structure features, and conversion and compilation tasks. Based on the task context, the large language model is invoked to infer and generate code logic that conforms to the target hardware toolchain specification. Based on the aforementioned code logic, the large language model outputs an execution script; The execution script is used as the conversion and compilation code.

5. The heterogeneous edge neural network deployment method according to claim 1, characterized in that, Before inputting the converted and compiled code into the preset containerized sandbox environment, the following steps are included: Container images with corresponding toolchain versions are pre-configured for different types of heterogeneous edge hardware; Match the container image according to the target hardware type and instantiate it as a runtime container environment to establish a preset containerized sandbox environment.

6. The heterogeneous edge neural network deployment method according to claim 5, characterized in that, The step of using error information during execution as a query vector to retrieve relevant solutions from the knowledge base includes: Obtain the standard error stream and exception stack text output by the containerized sandbox environment as error information; The error information is converted into a high-dimensional vector, which is then used as the query vector. Perform a similarity search in the vector database of the knowledge base to obtain text fragments that are semantically similar to the query vector; The text fragment is identified as the associated solution.

7. The heterogeneous edge neural network deployment method according to claim 1, characterized in that, The adaptive correction and iteration of the conversion compilation code based on the aforementioned correlation solution includes: The associated solution, the error message, and the current conversion and compilation code are input into the large language model to call the large language model to perform logical reasoning and obtain the reason for failure. Based on the stated cause of failure, locate and modify the erroneous logic section in the conversion compilation code to generate corrected conversion compilation code; The next iteration will be executed based on the revised transformation compilation code.

8. The heterogeneous edge neural network deployment method according to claim 1, characterized in that, The step of distributing the target model file to the target edge device for deployment includes: The target model file is transferred to the target edge device; On the target edge device, the target model file is loaded, and a benchmark test program is run to obtain test results; The test results are obtained and sent back to the central server to perform deployment verification.

9. A heterogeneous edge neural network deployment system, characterized in that, include: A knowledge base construction module is used to build a knowledge base, which is an independent knowledge base built based on unstructured technical documents of heterogeneous edge hardware. The semantic analysis module, based on the model profile of the neural network model and the target hardware type, retrieves target knowledge from the knowledge base to generate prompt words; The code generation module generates conversion and compilation code based on the prompt words, and inputs the conversion and compilation code into a preset containerized sandbox environment; The containerized sandbox execution module is used to execute the conversion and compilation code of the neural network model to output the execution results; The self-healing module is used to retrieve the knowledge base by using the error information during the execution process as a query vector in the event of an execution failure, so as to obtain a relevant solution. Based on the aforementioned correlation solution, the conversion and compilation code is adaptively corrected and iterated until the execution result is successful, and the target model file is output. The deployment module distributes the target model file to the target edge device to perform deployment.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the heterogeneous edge neural network deployment method as described in any one of claims 1 to 8.

Images