Model deployment method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a model deployment method and device, electronic equipment and a storage medium. A target model is determined; executing an operator type conversion operation and a memory allocation strategy optimization operation on the target model, and converting the target model into a target graph representation; determining an operator sequence for target graph representation and operator parameters for the operator sequence; generating a binary file based on the operator sequence and the operator parameters, and sending the binary file to embedded terminal equipment; the embedded terminal equipment is configured to download a binary file and create self-defined middleware for analyzing the binary file; according to the method, the target model is loaded and operated in the environment without the operating system based on the user-defined middleware, the AI reasoning framework designed for the complex operating system is adapted to the embedded equipment with limited resources and simplified operating systems, and meanwhile, the efficient memory utilization rate and reasoning performance are guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of model deployment methods, and in particular to a model deployment method, a model deployment device, an electronic device and a computer-readable storage medium. Background Art

[0002] The embedded AI inference framework is a software tool that provides the necessary environment and tools for running artificial intelligence models on resource-limited embedded devices. Simply put, it is like a bridge that connects complex artificial intelligence models and embedded devices, allowing these models to run in real time on devices such as mobile phones, smart watches, and drones.

[0003] The embedded AI reasoning framework of related technologies usually creates graphs and allocates memory on the embedded side, and then implements reasoning. This method is friendly to algorithm deployers, but at the same time, it also has stricter requirements on hardware performance and software support.

[0004] For chips with limited resources, it is impossible to carry a complex operating system (OS) to achieve good scheduling of memory resources, so the reasoning framework cannot be implemented on such chips. Summary of the invention

[0005] The embodiments of the present invention provide a model deployment method, device, electronic device and computer-readable storage medium to overcome the above problems or at least partially solve the above problems.

[0006] An embodiment of the present invention discloses a model deployment method, characterized in that the method is applied to a development end device, the development end device is provided with a corresponding embedded end device, and the method includes:

[0007] Determine the target model;

[0008] Performing an operator type conversion operation and a memory allocation strategy optimization operation on the target model to convert the target model into a target graph representation;

[0009] Determining an operator sequence for the target graph representation and operator parameters for the operator sequence;

[0010] Generate a binary file based on the operator sequence and the operator parameters, and send the binary file to the embedded terminal device;

[0011] The embedded terminal device is configured as follows:

[0012] Downloading the binary file and creating a custom middleware for parsing the binary file;

[0013] The target model is loaded and run in a non-operating system environment based on the custom middleware.

[0014] Optionally, the step of performing an operator type conversion operation and a memory allocation strategy optimization operation on the target model to convert the target model into a target graph representation includes:

[0015] Traversing the target model and identifying model operators;

[0016] Performing an operator type conversion operation on the model operator based on a preset conversion rule;

[0017] Calculating the memory usage data required by the model operator after the operator type conversion operation;

[0018] Selecting a minimum memory block that satisfies the memory occupancy data from a free memory block queue based on the memory occupancy data;

[0019] Allocating the minimum memory block to the model operator that has undergone the operator type conversion operation, and creating a node for expressing the model operator that has undergone the operator type conversion operation;

[0020] A target graph representation for the target model is generated based on the nodes.

[0021] Optionally, the step of determining a sequence of operators for the target graph representation comprises:

[0022] Acquire first parameter information related to memory scheduling from the target graph representation, used to construct association information of operator execution order and parameter initialization, and second parameter information unrelated to the reasoning process of the target model;

[0023] The first parameter information, the second parameter information and the associated information are deleted, and an operator sequence represented by the target graph is constructed based on an execution order.

[0024] Optionally, the step of generating a binary file based on the operator sequence and the operator parameters includes:

[0025] Traversing the operator sequence, acquiring a target parameter associated with the pointer from the model parameters of each model operator, and determining the memory size information of the target memory pointed to by the pointer based on the target parameter;

[0026] Allocating continuous memory space for the model operator based on the memory size information to generate an offset address for the target parameter;

[0027] Assign the offset address to the target parameter to generate a binary file.

[0028] Optionally, the embedded terminal device is configured as:

[0029] Loading the binary file and determining the starting position of the binary file;

[0030] Traversing the overall information of the target model, reading the operator type of each model operator, the parameter value of the operator parameter of the operator type, and the offset address of the parameter value;

[0031] Add the offset address to the starting position to obtain the actual address of the parameter value in the memory;

[0032] The target model is loaded and executed in a non-operating system environment based on the actual address.

[0033] Optionally, the step of constructing an operator sequence for the target graph representation based on the execution order includes:

[0034] Determine a node with an in-degree of 0 as a starting node from the target graph representation;

[0035] According to the execution order, all nodes are traversed in sequence, and the traversed nodes are added to the initial operator sequence to construct an operator sequence.

[0036] Optionally, before the step of obtaining first parameter information related to memory scheduling from the target graph representation for constructing association information of operator execution order and parameter initialization and second parameter information unrelated to the reasoning process of the target model, the step further includes:

[0037] The topological relationship in the target graph representation is obtained, and the execution order of each operator is determined based on the topological relationship.

[0038] The embodiment of the present invention further discloses a model deployment method, which is applied to an embedded terminal device, wherein the embedded terminal device is provided with a corresponding development terminal device, and the development terminal device is configured as follows:

[0039] Determine the target model;

[0040] Performing an operator type conversion operation and a memory allocation strategy optimization operation on the target model to convert the target model into a target graph representation;

[0041] Determining an operator sequence for the target graph representation and operator parameters for the operator sequence;

[0042] Generate a binary file based on the operator sequence and the operator parameters, and send the binary file to the embedded terminal device;

[0043] The method comprises:

[0044] Downloading the binary file and creating a custom middleware for parsing the binary file;

[0045] The target model is loaded and run in a non-operating system environment based on the custom middleware.

[0046] The embodiment of the present invention further discloses a model deployment device, which is applied to a development end device, and the development end device is provided with a corresponding embedded end device, and the device includes:

[0047] A target model determination module, used for determining a target model;

[0048] A target graph representation conversion module, used to perform operator type conversion operations and memory allocation strategy optimization operations on the target model, and convert the target model into a target graph representation;

[0049] An operator parameter determination module, used to determine an operator sequence represented by the target graph and operator parameters for the operator sequence;

[0050] A binary file sending module, used for generating a binary file based on the operator sequence and the operator parameters, and sending the binary file to the embedded terminal device;

[0051] The embedded terminal device is configured as follows:

[0052] Downloading the binary file and creating a custom middleware for parsing the binary file;

[0053] The target model is loaded and run in a non-operating system environment based on the custom middleware.

[0054] The embodiment of the present invention also discloses a model deployment device, which is applied to an embedded terminal device, wherein the embedded terminal device is provided with a corresponding development terminal device, and the development terminal device is configured to: determine a target model; perform an operator type conversion operation and a memory allocation strategy optimization operation on the target model to convert the target model into a target graph representation; determine an operator sequence for the target graph representation, and operator parameters for the operator sequence; generate a binary file based on the operator sequence and the operator parameters, and send the binary file to the embedded terminal device;

[0055] The device comprises:

[0056] A binary file download module, used for downloading the binary file and creating a custom middleware for parsing the binary file;

[0057] The target model loading and running module is used to load and run the target model in a non-operating system environment based on the custom middleware.

[0058] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0059] The memory is used to store computer programs;

[0060] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0061] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0062] The embodiments of the present invention include the following advantages:

[0063] In an embodiment of the present invention, a target model is determined; an operator type conversion operation and a memory allocation strategy optimization operation are performed on the target model to convert the target model into a target graph representation; an operator sequence for the target graph representation and operator parameters for the operator sequence are determined; a binary file is generated based on the operator sequence and the operator parameters, and the binary file is sent to the embedded terminal device; the embedded terminal device is configured to: download the binary file and create a custom middleware for parsing the binary file; based on the custom middleware, the target model is loaded and run in an operating system-free environment, thereby adapting an AI reasoning framework designed for a complex operating system to an embedded device with limited resources and a simplified operating system, while ensuring efficient memory utilization and reasoning performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a flowchart of the steps of a model deployment method provided in an embodiment of the present invention;

[0065] Figure 2 It is a flowchart of a model deployment method provided in an embodiment of the present invention;

[0066] Figure 3 It is a schematic diagram of a target graph representation conversion process provided in an embodiment of the present invention;

[0067] Figure 4 is a flow chart of a method for constructing an operator sequence provided in an embodiment of the present invention;

[0068] Figure 5 It is a flowchart of a binary file generation method provided in an embodiment of the present invention;

[0069] Figure 6 It is a schematic diagram of a binary file operator parsing process provided in an embodiment of the present invention;

[0070] Figure 7 is a flowchart of another model deployment method provided in an embodiment of the present invention;

[0071] Figure 8 is a structural block diagram of a model deployment device provided in an embodiment of the present invention;

[0072] Fig. 9 is a structural block diagram of another model deployment device provided in an embodiment of the present invention;

[0073] Fig.10 is a hardware structure block diagram of an electronic device provided in an embodiment of the present invention;

[0074] Fig.11 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] In practical applications, there are two core problems encountered by related technologies when deploying open source embedded AI inference frameworks on chips with limited resources:

[0077] Problem 1: Operating System and Resource Scheduling

[0078] The core of the problem: Chips with limited resources cannot carry complex operating systems, and existing open source frameworks rely on the memory management and scheduling functions provided by the operating system.

[0079] Impact: Real-time memory allocation cannot be achieved on resource-constrained devices, affecting the operating efficiency and stability of the inference framework.

[0080] Question 2: Memory allocation algorithm

[0081] The core of the problem: Existing open source frameworks use an algorithm based on the sequential selection of free memory blocks when allocating memory. This algorithm is random, resulting in low memory utilization.

[0082] Impact: The overall memory usage distribution is not optimal, which may lead to memory fragmentation and affect the inference performance of the model.

[0083] The underlying reasons behind the problem:

[0084] Original intention of framework design: The original intention of designing these frameworks is to facilitate algorithm deployment personnel, so many optimizations have been made in terms of ease of use, but insufficient consideration has been given to performance optimization in resource-constrained environments.

[0085] Hardware limitations: Chips with limited resources, especially memory resources, make traditional memory management methods difficult to apply.

[0086] Algorithm complexity: The AI ​​reasoning framework involves a large number of memory allocation and release operations. When faced with such complex scenarios, the efficiency and performance of traditional memory allocation algorithms are often not ideal.

[0087] How to adapt the AI ​​reasoning framework designed for complex operating systems to embedded devices with limited resources and simplified operating systems while ensuring efficient memory utilization and reasoning performance has become a technical problem that technical personnel in this field need to overcome.

[0088] In order to enable those skilled in the art to better understand the embodiments of the present invention, some technical names involved in the embodiments of the present invention are explained below.

[0089] Tengine-Lite is a lightweight, high-performance, cross-platform deep learning inference engine. It is designed specifically for embedded devices and mobile terminals to efficiently run deep learning models. The core functions of Tengine-Lite include:

[0090] Model conversion: Convert models trained by various deep learning frameworks (such as TensorFlow, PyTorch) into intermediate representations that can be recognized by Tengine-Lite.

[0091] Operator optimization: Optimize the operators in the model to adapt to the target hardware platform and improve inference performance.

[0092] Memory optimization: optimize memory allocation and usage to reduce memory usage.

[0093] Multi-platform support: supports multiple operating systems and hardware platforms, including ARM, x86, etc.

[0094] Simply put, Tengine-Lite is equivalent to an "engine" that runs deep learning models on embedded devices. It is responsible for loading the model onto the device and performing inference calculations efficiently.

[0095] PC pre_run stage

[0096] Pre_run is an important stage in Tengine-Lite, which usually occurs before the model runs on an embedded device. This stage is mainly performed on the PC side to prepare for the reasoning of the model.

[0097] The main tasks of the pre_run phase include:

[0098] Graph creation: The converted model is represented as a computational graph, which describes how the operators in the model are connected.

[0099] Memory allocation: Allocate the required memory space for model execution.

[0100] Parameter initialization: Initialize the parameters of the model.

[0101] Optimization: Optimize the computational graph, such as fusing operators and constant folding, to improve reasoning efficiency.

[0102] The role of the pre_run stage:

[0103] Improve reasoning efficiency: By performing optimization in advance, the amount of calculation during reasoning can be reduced and the reasoning speed can be improved.

[0104] Reduce memory usage: Through reasonable memory allocation, the memory consumption of the model can be reduced during runtime.

[0105] Ensure the correctness of inference: By checking the structure and parameters of the model, potential problems can be discovered in advance.

[0106] Tengine-Lite is a deep learning inference engine designed for embedded devices.

[0107] The pre_run stage is the preparation stage before the model runs on the embedded device, which mainly involves graph creation, memory allocation, parameter initialization and optimization, etc.

[0108] Reference Figure 1 , shows a flow chart of the steps of a model deployment method provided in an embodiment of the present invention, which may specifically include the following steps:

[0109] Step 101, determining a target model;

[0110] Step 102, performing an operator type conversion operation and a memory allocation strategy optimization operation on the target model to convert the target model into a target graph representation;

[0111] Step 103, determining an operator sequence represented by the target graph and operator parameters for the operator sequence;

[0112] Step 104, generating a binary file based on the operator sequence and the operator parameters, and sending the binary file to the embedded terminal device; the embedded terminal device is configured to: download the binary file, and create a custom middleware for parsing the binary file; based on the custom middleware, load and run the target model in an operating system-free environment.

[0113] In a specific implementation, the embodiment of the present invention may be used in a development end device, wherein the development end device is provided with a corresponding embedded end device, and the development end device may be a computer.

[0114] In practical applications, the embedded AI inference framework is a software tool that provides the necessary environment and tools for running artificial intelligence models on resource-limited embedded devices. Simply put, it is like a bridge that connects complex artificial intelligence models and embedded devices, allowing these models to run in real time on devices such as mobile phones, smart watches, and drones.

[0115] Core features:

[0116] Model loading and conversion: Load trained deep learning models (such as TensorFlow and PyTorch models) onto embedded devices and convert them into a format suitable for device hardware.

[0117] Inference calculation: Execute model reasoning, that is, obtain the corresponding output results based on the input data.

[0118] Hardware acceleration: Use dedicated hardware accelerators on the device (such as GPU, NPU) to accelerate the model reasoning process and improve efficiency.

[0119] Memory management: Optimize memory usage to accommodate the limited memory resources of embedded devices.

[0120] Multi-platform support: Supports multiple embedded platforms and operating systems, making it easy for developers to deploy models on different devices.

[0121] The advantages of embedded AI reasoning framework are:

[0122] Real-time: Embedded devices usually require real-time response, and the inference framework can ensure low latency of model inference.

[0123] Power consumption: The battery capacity of embedded devices is limited, and the inference framework needs to optimize the model and calculation process to reduce power consumption.

[0124] Model deployment: Deploying a trained model to an embedded device requires a series of conversions and optimizations. The inference framework can simplify this process.

[0125] Embedded devices may include:

[0126] Smartphones: image recognition, speech recognition, natural language processing, etc.

[0127] Smart home: voice assistant, smart camera, smart door lock, etc.

[0128] Wearable devices: health monitoring, sports tracking, etc.

[0129] Drones: target detection, path planning, etc.

[0130] Industrial automation: defect detection, predictive maintenance, etc.

[0131] In a specific implementation, the target model may be an embedded AI that needs to be deployed to an embedded terminal device.

[0132] The embedded AI reasoning framework provides strong support for deploying AI models on resource-constrained embedded devices, promoting the application of AI in various fields. With the continuous development of hardware and software technologies, the embedded AI reasoning framework will become more efficient and flexible, bringing more intelligent products and services to users.

[0133] The embedded AI reasoning deployment process of the embodiment of the present invention is mainly divided into the following stages:

[0134] PC-side model conversion and optimization: Use Tengine-Lite on the PC side to convert the trained model into an intermediate representation (graph representation) suitable for embedded devices. In this process, the model is optimized, including operator conversion, memory allocation strategy optimization, etc.

[0135] Generate parameters required for reasoning: Based on the optimized graph representation, generate a bin file containing operator sequences and parameters, that is, a binary file. This bin file is the input for embedded-side reasoning.

[0136] Build an inference environment on the embedded side: Build an environment without an operating system on the embedded side, parse the bin file, generate a graph representation of the model, and implement the inference process.

[0137] In a specific implementation, the operator type conversion operation of the embodiment of the present invention can be implemented in the following manner:

[0138] Customized operators: During the model conversion process, operators in the general deep learning framework (such as convolution, pooling, etc.) are converted into customized operators suitable for the target hardware platform. These customized operators are usually optimized for specific hardware features, such as using SIMD instructions, hardware accelerators, etc.

[0139] Operator fusion: Multiple consecutive operators can be fused to reduce the storage and calculation times of intermediate results and improve efficiency.

[0140] Operator sequence optimization: By adjusting the execution order of operators, the number of memory accesses can be reduced and the cache hit rate can be improved.

[0141] Memory allocation strategy optimization can be achieved in the following ways:

[0142] Memory allocation order: The operator sequence determines the order of memory allocation. By optimizing the operator sequence, you can reduce memory fragmentation and improve memory utilization.

[0143] Memory reuse: Some intermediate results can be reused to reduce the number of memory allocations.

[0144] Graph representation: The model is represented as a graph, where nodes represent operators and edges represent data flows. This representation is clear and intuitive, and is easy to optimize and reason.

[0145] Binary file (bin file):

[0146] Recording of operator sequences: The bin file records the optimized operator sequence to ensure that the embedded end executes operators in the correct order.

[0147] The relationship between operator sequence, operator parameters and model optimization is as follows:

[0148] The operator sequence and operator parameters determine the content of the bin file: the optimized operator sequence and parameters will be directly written into the bin file and become the instruction set for embedded-side reasoning.

[0149] The bin file reflects the model optimization results: the information in the bin file reflects the optimizations made to the model on the PC side, including operator conversion, memory allocation optimization, etc.

[0150] The embedded end performs reasoning based on the bin file: The embedded end executes operators one by one according to the operator sequence and parameters recorded in the bin file to complete model reasoning.

[0151] Storage of operator parameters: The bin file stores the parameters of each operator, including weights, biases, etc. These parameters are optimized and may have undergone operations such as quantization or pruning.

[0152] The embodiment of the present invention can determine the operator sequence represented by the target graph and the operator parameters for the operator sequence. The operator sequence and operator parameters are the direct embodiment of the model optimization results and the basis for embedded end reasoning. By optimizing the model, generating efficient operator sequences and parameters, and storing them in a bin file, the reasoning performance of the model on the embedded device can be significantly improved.

[0153] For example, suppose a convolutional neural network model, after optimization, fuses multiple consecutive convolution operations into a large convolution operation and quantizes the weights. These optimization results are reflected in the bin file. When the embedded end performs inference, it directly follows the instructions in the bin file, executes the fused convolution operation, and uses the quantized weights for calculation, thereby improving the inference speed and reducing memory usage.

[0154] refer to Figure 2 , Figure 2 It is a flowchart of a model deployment method provided in an embodiment of the present invention;

[0155] The embedded AI reasoning deployment solution of the embodiment of the present invention mainly involves two main execution entities:

[0156] The PC side (development device) is responsible for model conversion, optimization and compilation.

[0157] Main tasks of PC (development device):

[0158] Use Tengine-Lite to convert the trained model into a graph representation suitable for embedded devices.

[0159] Optimize graph representation, including operator conversion, memory allocation strategy optimization, etc.

[0160] Generate a bin file containing operator sequences and parameters as input for the embedded end.

[0161] Embedded terminal device, responsible for model reasoning.

[0162] Main tasks of embedded terminal devices:

[0163] Parse the received bin file.

[0164] Based on the information in the bin file, a graph representation of the model is constructed.

[0165] Perform inference calculations to obtain the final result.

[0166] Specifically, embedded terminal devices can be used for:

[0167] Download binary file: The embedded device downloads a binary file containing model information from external storage or network. This file is optimized and designed specifically to run on embedded devices.

[0168] Create custom middleware: A software module dedicated to parsing binary files and performing model inference is created on the embedded device, which is called custom middleware. This middleware is tailored for specific hardware and models.

[0169] Load and run the model: The custom middleware loads the downloaded binary file, parses the model structure and parameters, and then performs model inference on the embedded device to obtain the final result.

[0170] No operating system environment: The entire process is carried out in an environment without a traditional operating system, which helps to reduce system overhead and improve the real-time performance of the system.

[0171] In an embodiment of the present invention, a target model is determined; an operator type conversion operation and a memory allocation strategy optimization operation are performed on the target model to convert the target model into a target graph representation; an operator sequence for the target graph representation and operator parameters for the operator sequence are determined; a binary file is generated based on the operator sequence and the operator parameters, and the binary file is sent to the embedded terminal device; the embedded terminal device is configured to: download the binary file and create a custom middleware for parsing the binary file; based on the custom middleware, the target model is loaded and run in an operating system-free environment, thereby adapting an AI reasoning framework designed for a complex operating system to an embedded device with limited resources and a simplified operating system, while ensuring efficient memory utilization and reasoning performance.

[0172] Furthermore, through memory allocation optimization, model representation optimization and reasoning in an operating system-free environment, the target model can be deployed, achieving the following beneficial effects:

[0173] Memory allocation optimization:

[0174] Local optimal principle: In the pre_run phase on the PC side, the local optimal principle is used to allocate memory to each operator, that is, the smallest free memory block that meets the current operator's needs is selected. This can effectively reduce memory fragmentation and improve memory utilization.

[0175] Static memory allocation: On the embedded side, since there is no operating system, static memory allocation is used. When generating the bin file, the memory size and address offset required by each operator are calculated in advance, and the memory is directly allocated on the embedded side.

[0176] Model representation optimization:

[0177] Graph representation: The model is represented as a graph, where nodes represent operators and edges represent data flows. This representation is clear and intuitive, and is easy to optimize and reason.

[0178] Operator optimization: Optimize the operators in the model, select operators suitable for embedded devices, and improve reasoning efficiency.

[0179] Reasoning without an operating system:

[0180] Custom middleware: Build a custom middleware on the embedded side, which is responsible for parsing bin files, executing operators, managing memory, etc.

[0181] Static linking: Statically link all dependent libraries into the middleware to avoid the complexity brought by dynamic linking.

[0182] Advantages and features:

[0183] Efficient memory utilization: Locally optimal memory allocation strategy and static memory allocation method improve memory utilization.

[0184] Flexible model representation: The graphical representation makes the model structure clear and easy to optimize and modify.

[0185] Strong portability: No operating system dependency, allowing the solution to be deployed on a variety of embedded devices.

[0186] Easy to deploy: By generating a bin file, the model deployment process is simplified.

[0187] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0188] In an optional embodiment of the present invention, the step of performing an operator type conversion operation and a memory allocation strategy optimization operation on the target model to convert the target model into a target graph representation includes:

[0189] Traversing the target model and identifying model operators;

[0190] Performing an operator type conversion operation on the model operator based on a preset conversion rule;

[0191] Calculating the memory usage data required by the model operator after the operator type conversion operation;

[0192] Selecting a minimum memory block that satisfies the memory occupancy data from a free memory block queue based on the memory occupancy data;

[0193] Allocating the minimum memory block to the model operator that has undergone the operator type conversion operation, and creating a node for expressing the model operator that has undergone the operator type conversion operation;

[0194] A target graph representation for the target model is generated based on the nodes.

[0195] Exemplarily, the target graph representation conversion process can be implemented in the following manner:

[0196] 1. Input model: Take the target model to be converted (usually expressed in ONNX, TensorFlow, etc.) as input.

[0197] 2. Operator identification and conversion: traverse the target model, identify each model operator in the target model, and perform type conversion according to the preset conversion rules.

[0198] 3. Memory requirement calculation: For each model operator, the memory usage data required by the model operator after the operator type conversion operation is calculated to determine the required memory size.

[0199] 4. Free memory block selection: According to the "local optimal principle", select the most suitable memory block from the free memory blocks.

[0200] Common conversion rules:

[0201] Data types: integer, signed integer, unsigned integer, floating point number.

[0202] Byte order: big endian, little endian.

[0203] Quantization: linear quantization, logarithmic quantization.

[0204] refer to Figure 3 , Figure 3 It is a schematic diagram of a target graph representation conversion process provided in an embodiment of the present invention;

[0205] Specifically, in order to more effectively utilize the limited memory of embedded devices, an embodiment of the present invention can adopt the best adaptation memory allocation strategy during the model deployment process. This strategy follows the principle of local optimality, that is, to allocate the smallest memory block that can meet its needs to each operator. Specifically, the system can maintain a list of free memory blocks for expressing the free memory block queue, and each memory block has a "number of times to be used" attribute, which records the number of times the memory block needs to be used during the current model execution process. When allocating memory for an operator, the system will traverse the list of free memory blocks and select the memory block with the closest size that can meet the needs of the operator. Once the operator is executed, the memory block it occupies will be released and re-added to the list of free memory blocks. This dynamic memory allocation method can effectively reduce memory fragmentation and improve memory utilization.

[0206] For example:

[0207] Assume that a convolutional neural network model contains multiple convolutional layers and fully connected layers. During model deployment, the system will allocate memory according to the following steps:

[0208] Initialization: The system will allocate a large continuous memory area for the model as the total memory pool.

[0209] First convolution layer: The system calculates the memory size required for the convolution layer, then finds a free block in the free memory pool that is closest in size and greater than or equal to the required memory, allocates the block to the convolution layer, and updates the free memory block list.

[0210] Subsequent layers: For each subsequent layer, repeat step 2 until all layers have been allocated memory.

[0211] Memory release: When the calculation of a layer is completed, the system releases the memory it occupies and adds the memory block back to the list of free memory blocks.

[0212] 5. Memory allocation: Allocate the selected memory block to the current operator and update the list of free memory blocks.

[0213] 6. Create new graph node: Create a node representing the current operator in the new graph.

[0214] 7. Connect nodes: Establish connections between nodes based on the dependencies between operators.

[0215] 8. Add attributes: Add attributes such as operator type, input and output tensors, memory address, etc. to the node.

[0216] Repeat 3-8: until all operators have been processed.

[0217] Output target graph representation: Outputs the target graph representation containing the converted operators and optimized memory allocation information.

[0218] 1. The purpose of operator type conversion is to convert operators in the model (such as convolution, pooling, etc.) into operator types supported by the target platform (such as embedded devices).

[0219] The steps in the operator type conversion phase may include:

[0220] Identify operators: Traverse the model and identify the type and parameters of each operator one by one.

[0221] Type matching: Match the identified operators with the operator types supported by the target platform.

[0222] Conversion rules: According to the conversion rules, unsupported operators are split into multiple supported operators, or replaced with approximate algorithms.

[0223] 2. The purpose of memory allocation strategy optimization is to optimize the memory allocation during model runtime, reduce memory fragmentation, and improve memory utilization.

[0224] The steps in the memory allocation strategy optimization phase may include:

[0225] Memory requirement calculation: For each operator, calculate the required memory size.

[0226] Free memory block selection: According to the memory allocation strategy provided above, the most suitable memory block is selected from the free memory blocks and allocated to the current operator.

[0227] Memory allocation: Allocate the selected memory block to the current operator and update the list of free memory blocks.

[0228] Memory recycling: When an operator is used up, the memory it occupies is recycled and added to the list of free memory blocks.

[0229] 3. The purpose of generating the target graph representation is to generate a new graph representation that contains the converted operators and optimized memory allocation information.

[0230] The steps in the graphical representation generation phase may include:

[0231] Create New Graph: Create a new graph structure to store the converted model.

[0232] Add Node: Add the converted operator as a node to the new graph.

[0233] Add edges: Add edges between nodes based on the dependencies between operators.

[0234] Add attributes: Add corresponding attributes to each node and edge, such as operator type, input and output tensors, memory address, etc.

[0235] Through the above steps, the original model is converted into a target graph representation optimized for the target platform. This process not only realizes the operator type conversion, but also improves the operation efficiency of the model through a reasonable memory allocation strategy.

[0236] In an optional embodiment of the present invention, the step of determining the operator sequence for the target graph representation comprises:

[0237] Acquire first parameter information related to memory scheduling from the target graph representation, used to construct association information of operator execution order and parameter initialization, and second parameter information unrelated to the reasoning process of the target model;

[0238] The first parameter information, the second parameter information and the associated information are deleted, and an operator sequence represented by the target graph is constructed based on an execution order.

[0239] refer to Figure 4 , Figure 4 is a flow chart of a method for constructing an operator sequence provided in an embodiment of the present invention;

[0240] The embodiment of the present invention can perform a series of simplifications and optimizations based on the graph representation created by Tengine-Lite, and finally obtain an operator sequence for executing reasoning.

[0241] In a specific implementation, the first parameter information related to memory scheduling can be deleted from the target graph representation:

[0242] Purpose: Remove parameters related to scheduling such as memory allocation and release in the graph, which are unnecessary when performing inference.

[0243] Operation: Traverse each node in the graph (representing an operator) and delete the memory-related attributes or connections.

[0244] Delete the associated information about the execution order and parameter initialization of the build operator:

[0245] Purpose: Remove information from the graph that is used to determine the order in which operators are executed and to initialize operator parameters. This information is already determined during the inference phase.

[0246] Action: Delete the edge or node attributes in the graph that are used to represent the execution order of operators.

[0247] Delete the second parameter information that is irrelevant to the target model inference:

[0248] Purpose: Remove parameter information in the graph that is not related to the reasoning process, such as parameters and debugging information during training.

[0249] Action: Traverse each node in the graph and remove attributes that are not relevant for reasoning.

[0250] Optionally, the embodiment of the present invention may also obtain the topological relationship in the target graph representation, determine the execution order of each operator based on the topological relationship, and form a linear sequence as the initial operator sequence.

[0251] In a specific implementation, an embodiment of the present invention can find a node with an in-degree of 0 from the target graph representation as the starting node, and then traverse all nodes in sequence according to the rules of topological sorting, and add the traversed nodes to the initial operator sequence to construct an operator sequence.

[0252] Through the above four steps, the pure reasoning execution order can be extracted from the graph representation generated by Tengine-Lite. This operator sequence clarifies the execution order of each operator in the model and provides direct guidance for subsequent reasoning execution.

[0253] Example:

[0254] Assume a convolutional neural network with the following computational graph:

[0255] Input → Convolution 1 → Pooling → Convolution 2 → Full Connection → Output;

[0256] After the above steps, the operator sequence obtained is:

[0257] Convolution 1 → Pooling → Convolution 2 → Full connection;

[0258] This sequence clarifies the execution order of each operator in the model, so that the inference engine can execute each operator in this order and finally obtain the output of the model.

[0259] This process is the process of converting a high-level model representation (graph representation) into a low-level execution instruction (operator sequence). Through this process, the structure of the model can be better understood and provide a basis for subsequent reasoning optimization.

[0260] In an optional embodiment of the present invention, the step of generating a binary file based on the operator sequence and the operator parameters includes:

[0261] Traversing the operator sequence, acquiring a target parameter associated with the pointer from the model parameters of each model operator, and determining the memory size information of the target memory pointed to by the pointer based on the target parameter;

[0262] Allocating continuous memory space for the model operator based on the memory size information to generate an offset address for the target parameter;

[0263] Assign the offset address to the target parameter to generate a binary file.

[0264] refer to Figure 5 , Figure 5 It is a flowchart of a binary file generation method provided in an embodiment of the present invention;

[0265] The embodiment of the present invention can generate a binary file containing all operator parameters based on determining the execution order of operators and the memory size required by each model operator.

[0266] For example:

[0267] Calculate the memory size required for the parameters pointed to by the pointer parameters in each operator:

[0268] Purpose: To determine the memory space occupied by the data pointed to by all pointer type parameters in each operator.

[0269] Operation: Traverse each operator in the operator sequence and analyze its parameter list. Obtain the target parameter associated with the pointer from the model parameters of each model operator. For such target parameters associated with the pointer, that is, the pointer type parameters, calculate the type and size of the data pointed to, so as to obtain the memory size pointed to by the pointer parameter.

[0270] Calculate the offset address of the pointer parameter in the generated bin file:

[0271] Purpose: To assign a unique offset address to each pointer parameter in the generated binary file so that the data pointed to by the parameter can be found correctly at runtime.

[0272] Operation: Starting from the starting address of the binary file, allocate memory space for each operator in turn. For pointer parameters, the offset address is the starting address of the data it points to in the binary file.

[0273] Assign the offset address to the pointer parameter:

[0274] Purpose: Update the pointer parameter in the operator to point to the correct memory address.

[0275] Operation: Assign the calculated offset address to the corresponding pointer parameter.

[0276] Generate binary files:

[0277] Create Binary File: Creates a new binary file in the file system.

[0278] Write data: Write the parameters of each operator into the binary file in sequence according to the calculated offset address.

[0279] Optionally, for other parameters of non-pointer type, directly write their values; for parameters of pointer type, write their offset addresses.

[0280] Close file: After writing is completed, close the binary file.

[0281] For example, consider a simple convolutional neural network with two convolution weight matrices W1 and W2, and a bias b. We can generate the binary file as follows:

[0282] Calculate the size of W1, W2, and b, and allocate contiguous memory space.

[0283] Calculate the starting addresses of W1 and W2, and assign them to the weight parameters of the convolutional layer respectively.

[0284] Calculate the starting address of b and assign it to the bias parameter of the convolutional layer.

[0285] Write the data of W1, W2 and b to a binary file.

[0286] By generating binary files from operator sequences and operator parameters, the model can be deployed on embedded devices or other hardware platforms. This method can improve the running efficiency of the model, reduce memory usage, and facilitate model deployment and management.

[0287] Through the above steps, the embodiment of the present invention successfully converts the operator sequence and operator parameters into a compact binary file. This binary file contains all the information required for model operation and can be directly loaded and executed on the target device.

[0288] In an optional embodiment of the present invention, the embedded terminal device is configured as follows:

[0289] Loading the binary file and determining the starting position of the binary file;

[0290] Traversing the overall information of the target model, reading the operator type of each model operator, the parameter value of the operator parameter of the operator type, and the offset address of the parameter value;

[0291] Add the offset address to the starting position to obtain the actual address of the parameter value in the memory;

[0292] The target model is loaded and executed in a non-operating system environment based on the actual address.

[0293] Custom middleware is a software component that is designed to process specific types of data, such as binary files.

[0294] For example, the main functions of custom middleware are:

[0295] Load Binary File: Reads a binary file stored on disk into memory.

[0296] Parsing binary files: Parse the data in binary files according to the predefined format, including operator types, parameter values, and offset addresses of each parameter.

[0297] Build execution environment: Create a suitable environment in memory for model execution, including allocating memory space, initializing variables, etc.

[0298] Execution model: Execute each operator in the model based on the information obtained from the analysis.

[0299] The relationship between the offset address and the custom middleware:

[0300] The offset address is the key for custom middleware to parse binary files: custom middleware can accurately locate the parameters of each operator and execute the model correctly by parsing the offset address in the binary file.

[0301] The custom middleware provides context for the offset address: the offset address itself is just a relative location and has practical significance only in the execution environment provided by the custom middleware. The custom middleware maps the offset address to a specific address in memory, so that the model can access the parameters correctly.

[0302] refer to Figure 6 , Figure 6 It is a schematic diagram of a binary file operator parsing process provided in an embodiment of the present invention;

[0303] Exemplarily, the embedded terminal device may execute the steps of loading and running the target model in a non-operating system environment based on the custom middleware in the following manner.

[0304] Load binary file: Read the binary file (.bin file) stored in the embedded device into the device's memory.

[0305] Get the starting address of binary file loading: Determine the starting position of the binary file in memory. This address will be used as the basis for subsequent calculation of parameter addresses.

[0306] Parse the binary file header: read the basic information of the model from the header of the binary file, such as the model version, the number of operators, the type of each operator, the number of parameters, etc.

[0307] Parse operator information: According to the information in the binary file header, parse the detailed information of each operator one by one, including the type of operator, the shape of input and output tensors, and the offset address of the parameters.

[0308] Calculate the actual address of the parameter: Add the parameter offset address of each operator to the binary file loading start address to obtain the actual address of the parameter in the memory.

[0309] Create an execution environment: Allocate the necessary space in memory for the execution of the model, including input and output data, intermediate results, and parameters of each operator.

[0310] Generate execution code based on operator type and parameters: Dynamically generate corresponding execution code based on the operator type and parameters obtained through parsing.

[0311] Execution model: The generated execution code is loaded into the execution engine, and each operator is executed in sequence according to the topological structure of the model.

[0312] Output results: Store the output results of the model in the specified memory area or device.

[0313] The core of the whole process is to convert the information in the binary file into executable code. By parsing the binary file, the structure and parameters of the model are obtained, and then the execution code is dynamically generated based on this information and executed in an environment without an operating system.

[0314] The role of the actual address:

[0315] In embedded systems, especially in environments without an operating system, each data (including model parameters, intermediate results, etc.) has a certain address in the memory. This address is called the actual address. The role of the actual address is mainly reflected in the following aspects:

[0316] Data Access:

[0317] Only when you know the actual address of the data can you read or write the data through this address.

[0318] For example, to obtain the weight of a neural network layer, you need to first calculate the actual address of the weight data, and then read it through this address.

[0319] Instruction execution:

[0320] Many instructions (such as loading data and storing data) require specifying the address of the data to be operated.

[0321] The actual address is an important parameter in these instructions.

[0322] Memory Management:

[0323] In the absence of an operating system, memory allocation and management are controlled by the program itself.

[0324] By calculating the actual address, memory space can be allocated reasonably and memory conflicts can be avoided.

[0325] Relationship between actual address and deployment target model

[0326] When deploying the target model, the actual address is closely related to the operation of the model:

[0327] Model loading:

[0328] When loading the model's binary file into memory, you need to determine the model's starting address.

[0329] This starting address is the reference point for all other data. By adding the offset, the actual address of other data can be calculated.

[0330] Parameter access:

[0331] Each parameter of the model (weights, biases, etc.) has an offset in the binary file.

[0332] By adding the offset to the model's starting address, we can get the actual address of the parameter in memory.

[0333] Only by obtaining the correct actual address can the model parameters be accessed to perform inference calculations.

[0334] Memory optimization:

[0335] By properly calculating the actual address, we can optimize memory usage, reduce memory fragmentation, and improve the operating efficiency of the model.

[0336] For example, frequently accessed data can be placed in a continuous memory area to reduce the number of memory accesses.

[0337] The actual address is the bridge that connects the binary representation of the model with the actual data in memory. By calculating the actual address, you can:

[0338] Accurately access model parameters, efficiently execute models, and flexibly manage memory.

[0339] In short, the actual address is a key link in deploying the target model, which directly affects the operating efficiency and accuracy of the model.

[0340] For example:

[0341] Assume that the weight data of a convolutional neural network model has an offset of 1024 bytes in the binary file. If the starting address of the model is 0x10000, then the actual address of the weight is 0x10000+1024=0x11000. Through this actual address, the weight data can be directly accessed.

[0342] Through custom middleware, various types of deep learning models can be flexibly loaded and executed. The offset address is the key link between binary files and model execution. Through reasonable design and implementation, we can build an efficient and reliable deep learning inference system.

[0343] In order to enable those skilled in the art to better understand the embodiment of the present invention, an example is used below to illustrate the embodiment of the present invention.

[0344] The embodiment of the present invention proposes an OS-free embedded AI reasoning solution with low memory usage. It is based on the open source AI reasoning framework tengine-lite. The model parsing tool of the original framework, as well as the construction and pre-run parts of the graph are adopted on the PC side, and its memory allocation scheme is optimized to make its memory allocation method more optimal. The compilation part is added to delete and compile the constructed graph, leaving only the parameters required for reasoning and the memory allocation parameters. In order to effectively use the 100+ software operators of tengine-lite, it is necessary to create a usable graph when deploying on the embedded side, and to perform optimal memory construction and compilation on the PC side through effective resources. When deployed on the embedded side, the graph can be created without an OS, solving the problem that the simple OS on the embedded side cannot effectively schedule memory resources, resulting in the inability to perform AI reasoning.

[0345] The specific implementation steps are as follows:

[0346] 1: Reference Figure 2 , Figure 2 It is a flowchart of a model deployment method provided in an embodiment of the present invention;

[0347] PC: tengine-lite part, on the PC side, the tengine_lite framework is used to convert and parse the model, and finally a graph representation of the model is obtained. The graph representation is obtained by parsing the model parameters and contains not only all the information required for forward reasoning, but also other information required for operator conversion, memory calculation, etc.

[0348] 2: PC: tengine-lite part, type conversion and memory allocation of some operators in graph representation are performed in the pre_run stage:

[0349] The memory allocation scheme of the related art selects memory blocks in the order of creation, which is random;

[0350] The revised new plan is as follows Figure 3 As shown, Figure 3 It is a target graph provided in an embodiment of the present invention, which represents a schematic diagram of the conversion process; the local optimal principle is adopted to select the most suitable memory block at present, that is, the smallest memory block that meets the memory requirements of the current operator is selected from the free memory blocks. This scheme can make better use of memory than the original scheme. The free memory block indicates that the memory block is created but not used, the non-free memory block indicates that the memory block is created but used, and the number of times to be used indicates the number of times the memory block needs to be used when it is created or occupied. It is obtained by analyzing the model parameters. After one use, the parameter is reduced by 1. When the number of uses reaches 0, it indicates that the memory block has been used up, and then the memory block changes from a non-free memory block to a free memory block.

[0351] 3: PC: compile part, compile and generate the necessary parameter bin file required for reasoning. The bin file consists of operator sequence and parameters. The operator only contains parameters related to reasoning calculation, such as operator type, operator input and output size, operator parameters, operator input and output, etc. The specific implementation is as follows Figure 4 , Figure 5 As shown, Figure 4 is a flow chart of a method for constructing an operator sequence provided in an embodiment of the present invention; Figure 5 It is a flowchart of a binary file generation method provided in an embodiment of the present invention.

[0352] 4: embedded end part: build the middleware file without OS dependency, parse the operator sequence of bin file, and generate the graph representation of the embedded end model. Since there is no OS dependency, real-time scheduling and allocation of memory cannot be performed. Therefore, Figure 6 As shown, Figure 6 It is a schematic diagram of a binary file operator parsing process provided in an embodiment of the present invention; in the process of generating bin, the memory occupancy and address offset of the graph representation need to be calculated, and when constructing the graph representation, direct occupancy and calling are performed according to the information provided by the bin file.

[0353] 5: embedded end part: build the model reasoning middleware file, build the model reasoning middleware file containing the tengine-lite software operator, which runs the model graph representation and realizes the reasoning of the graph representation.

[0354] 6: download and embedded end part: implement inference deployment, import the bin file into the embedded terminal, build a complete AI inference process according to the provided middleware file, and implement AI inference deployment.

[0355] Reference Figure 7 , shows a flowchart of another model deployment method provided in an embodiment of the present invention, which may specifically include the following steps:

[0356] Step 701, downloading the binary file and creating a custom middleware for parsing the binary file;

[0357] Step 702: Load and run the target model in a non-operating system environment based on the custom middleware.

[0358] In a specific implementation, an embodiment of the present invention is applied to an embedded terminal device, which is provided with a corresponding development terminal device, and the development terminal device is configured to: determine a target model; perform operator type conversion operations and memory allocation strategy optimization operations on the target model to convert the target model into a target graph representation; determine an operator sequence for the target graph representation, and operator parameters for the operator sequence; generate a binary file based on the operator sequence and the operator parameters, and send the binary file to the embedded terminal device.

[0359] As for the embedded terminal device embodiment, since it is basically similar to the development terminal device embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the development terminal device embodiment.

[0360] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0361] Reference Figure 8 , shows a structural block diagram of a model deployment device provided in an embodiment of the present invention, which may specifically include the following modules:

[0362] The target model determination module 801 is used to determine the target model;

[0363] A target graph representation conversion module 802 is used to perform an operator type conversion operation and a memory allocation strategy optimization operation on the target model to convert the target model into a target graph representation;

[0364] An operator parameter determination module 803 is used to determine an operator sequence represented by the target graph and operator parameters for the operator sequence;

[0365] A binary file sending module 804, configured to generate a binary file based on the operator sequence and the operator parameters, and send the binary file to the embedded terminal device;

[0366] The embedded terminal device is configured as follows:

[0367] Downloading the binary file and creating a custom middleware for parsing the binary file;

[0368] The target model is loaded and run in a non-operating system environment based on the custom middleware.

[0369] Reference Fig. 9 , shows a structural block diagram of another model deployment device provided in an embodiment of the present invention, which may specifically include the following modules:

[0370] A binary file download module 901, used to download the binary file and create a custom middleware for parsing the binary file;

[0371] The target model loading and running module 902 is used to load and run the target model in a non-operating system environment based on the custom middleware.

[0372] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0373] In addition, an embodiment of the present invention further provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned model deployment method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0374] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned model deployment method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0375] Fig.10 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0376] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, a processor 1010, and a power supply 1011. Those skilled in the art will appreciate that Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0377] It should be understood that in the embodiment of the present invention, the radio frequency unit 1001 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving downlink data from the base station, it is sent to the processor 1010 for processing; in addition, uplink data is sent to the base station. Generally, the radio frequency unit 1001 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 1001 can also communicate with the network and other devices through a wireless communication system.

[0378] The electronic device provides users with wireless broadband Internet access through the network module 1002, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0379] The audio output unit 1003 can convert the audio data received by the RF unit 1001 or the network module 1002 or stored in the memory 1009 into an audio signal and output it as sound. Moreover, the audio output unit 1003 can also provide audio output related to a specific function performed by the electronic device 1000 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 1003 includes a speaker, a buzzer, a receiver, etc.

[0380] The input unit 1004 is used to receive audio or video signals. The input unit 1004 may include a graphics processor (GPU) 10041 and a microphone 10042. The graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 1006. The image frame processed by the graphics processor 10041 can be stored in the memory 1009 (or other storage medium) or sent via the radio frequency unit 1001 or the network module 1002. The microphone 10042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 1001 in the case of a telephone call mode.

[0381] The electronic device 1000 also includes at least one sensor 1005, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 10061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 10061 and / or the backlight when the electronic device 1000 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 1005 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0382] The display unit 1006 is used to display information input by the user or information provided to the user. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0383] The user input unit 1007 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 1007 includes a touch panel 10071 and other input devices 10072. The touch panel 10071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 10071 or near the touch panel 10071 using any suitable object or accessory such as a finger, stylus, etc.). The touch panel 10071 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the coordinates of the contact point, and then sends it to the processor 1010, receives the command sent by the processor 1010 and executes it. In addition, the touch panel 10071 can be implemented in various types such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch panel 10071, the user input unit 1007 may also include other input devices 10072. Specifically, other input devices 10072 may include but are not limited to physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be repeated here.

[0384] Furthermore, the touch panel 10071 may be overlaid on the display panel 10061. When the touch panel 10071 detects a touch operation on or near it, it is transmitted to the processor 1010 to determine the type of touch event. Then, the processor 1010 provides corresponding visual output on the display panel 10061 according to the type of touch event. Fig.10 In the figure, the touch panel 10071 and the display panel 10061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 10071 and the display panel 10061 can be integrated to realize the input and output functions of the electronic device, which is not limited here.

[0385] The interface unit 1008 is an interface for connecting an external device to the electronic device 1000. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 1008 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 1000 or may be used to transmit data between the electronic device 1000 and an external device.

[0386] The memory 1009 can be used to store software programs and various data. The memory 1009 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1009 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0387] The processor 1010 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 1009 and calling data stored in the memory 1009, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 1010 may include one or more processing units; preferably, the processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1010.

[0388] The electronic device 1000 may also include a power supply 1011 (such as a battery) for supplying power to each component. Preferably, the power supply 1011 may be logically connected to the processor 1010 via a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0389] In addition, the electronic device 1000 includes some functional modules not shown, which will not be described in detail here.

[0390] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0391] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0392] like Fig.11 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1101 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the model deployment method described in the above embodiment.

[0393] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0394] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0395] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0396] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0397] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0398] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0399] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0400] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A model deployment method, characterized in that: The method is applied to a development terminal device, the development terminal device is provided with a corresponding embedded terminal device, and the method comprises: Determine the target model; Performing an operator type conversion operation and a memory allocation strategy optimization operation on the target model to convert the target model into a target graph representation; Determining an operator sequence for the target graph representation and operator parameters for the operator sequence; Generate a binary file based on the operator sequence and the operator parameters, and send the binary file to the embedded terminal device; The embedded terminal device is configured as follows: Downloading the binary file and creating a custom middleware for parsing the binary file; The target model is loaded and run in a non-operating system environment based on the custom middleware.

2. The method according to claim 1, characterized in that The step of performing an operator type conversion operation and a memory allocation strategy optimization operation on the target model to convert the target model into a target graph representation comprises: Traversing the target model and identifying model operators; Performing an operator type conversion operation on the model operator based on a preset conversion rule; Calculating the memory usage data required by the model operator after the operator type conversion operation; Selecting a minimum memory block that satisfies the memory occupancy data from a free memory block queue based on the memory occupancy data; Allocating the minimum memory block to the model operator that has undergone the operator type conversion operation, and creating a node for expressing the model operator that has undergone the operator type conversion operation; A target graph representation for the target model is generated based on the nodes.

3. The method according to claim 1, characterized in that The step of determining a sequence of operators for the target graph representation comprises: Acquire first parameter information related to memory scheduling from the target graph representation, used to construct association information of operator execution order and parameter initialization, and second parameter information unrelated to the reasoning process of the target model; The first parameter information, the second parameter information and the associated information are deleted, and an operator sequence represented by the target graph is constructed based on an execution order.

4. The method according to claim 2, characterized in that: The step of generating a binary file based on the operator sequence and the operator parameters comprises: Traversing the operator sequence, acquiring a target parameter associated with the pointer from the model parameters of each model operator, and determining the memory size information of the target memory pointed to by the pointer based on the target parameter; Allocating continuous memory space for the model operator based on the memory size information to generate an offset address for the target parameter; Assign the offset address to the target parameter to generate a binary file.

5. The method according to claim 4, characterized in that The embedded terminal device is configured as follows: Loading the binary file and determining the starting position of the binary file; Traversing the overall information of the target model, reading the operator type of each model operator, the parameter value of the operator parameter of the operator type, and the offset address of the parameter value; Add the offset address to the starting position to obtain the actual address of the parameter value in the memory; The target model is loaded and executed in a non-operating system environment based on the actual address.

6. The method according to claim 3, characterized in that: The step of constructing an operator sequence for the target graph representation based on the execution order comprises: Determine a node with an in-degree of 0 as a starting node from the target graph representation; According to the execution order, all nodes are traversed in sequence, and the traversed nodes are added to the initial operator sequence to construct an operator sequence.

7. The method according to claim 4, characterized in that Before the step of obtaining first parameter information related to memory scheduling from the target graph representation for building association information of operator execution order and parameter initialization, and second parameter information unrelated to the reasoning process of the target model, the method further includes: The topological relationship in the target graph representation is obtained, and the execution order of each operator is determined based on the topological relationship.

8. A model deployment method, characterized in that: The method is applied to an embedded terminal device, wherein the embedded terminal device is provided with a corresponding development terminal device, and the development terminal device is configured as follows: Determine the target model; Performing an operator type conversion operation and a memory allocation strategy optimization operation on the target model to convert the target model into a target graph representation; Determining an operator sequence for the target graph representation and operator parameters for the operator sequence; Generate a binary file based on the operator sequence and the operator parameters, and send the binary file to the embedded terminal device; The method comprises: Downloading the binary file and creating a custom middleware for parsing the binary file; The target model is loaded and run in a non-operating system environment based on the custom middleware.

9. A model deployment device, characterized in that: The device is applied to a development terminal device, the development terminal device is provided with a corresponding embedded terminal device, and the device includes: A target model determination module, used for determining a target model; A target graph representation conversion module is used to perform operator type conversion operations and memory allocation strategy optimization operations on the target model to convert the target model into a target graph representation; An operator parameter determination module, used to determine an operator sequence represented by the target graph and operator parameters for the operator sequence; A binary file sending module, used for generating a binary file based on the operator sequence and the operator parameters, and sending the binary file to the embedded terminal device; The embedded terminal device is configured as follows: Downloading the binary file and creating a custom middleware for parsing the binary file; The target model is loaded and run in a non-operating system environment based on the custom middleware.

10. A model deployment device, characterized in that: The device is applied to an embedded terminal device, the embedded terminal device is provided with a corresponding development terminal device, and the development terminal device is configured to: determine a target model; perform an operator type conversion operation and a memory allocation strategy optimization operation on the target model to convert the target model into a target graph representation; determine an operator sequence for the target graph representation, and operator parameters for the operator sequence; Generate a binary file based on the operator sequence and the operator parameters, and send the binary file to the embedded terminal device; The device comprises: A binary file download module, used for downloading the binary file and creating a custom middleware for parsing the binary file; The target model loading and running module is used to load and run the target model in a non-operating system environment based on the custom middleware.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1-7 or 8 when executing the program stored in the memory.

12. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7 or 8.

Images