Intelligent driving end-side model deployment method and device

Abstract

A kind of intelligent driving end side model deployment method and device, method includes: obtaining the network model for intelligent driving of pre-training, network model includes the graph consisting of node;Network model is converted and compiled, wherein in conversion and compilation, uniform external call interface is called, and interface is encapsulated into basic functional plug-in interface in the form of functional plug-in;Input data is obtained by functional plug-in interface, and input data is inferred based on the network model after compilation, and inference result is obtained and output.The method and device in the process of deploying network model, by uniform external call interface in the conversion and compilation of network model, interface is encapsulated into basic functional plug-in interface in the form of functional plug-in, can solve the problem that end side perception inference framework is numerous, interface is not uniform, so as to improve the repeatability of code, reduce the difficulty of algorithm deployment, speed up algorithm landing, save development cost, shorten the development cycle of product.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and more specifically to a method and apparatus for deploying an intelligent driving edge model. Background Technology

[0002] Currently, in the practical application of intelligent driving, edge AI chips (CPU, Neural-network Processing Unit (NPU), GPU) with different computing power are often used depending on the functional scenario. Because the chip structures and functions produced by various hardware manufacturers differ greatly, the performance of the same code implemented on different hardware also varies significantly. Therefore, to maximize chip performance, code optimization for the hardware itself is necessary. This means that the same algorithm will have different optimized code on different hardware, causing significant problems for developers. If a certain function needs to be implemented on a different platform, the original code becomes unusable. This results in the inability to decouple software and hardware, the inability to reuse software code, and the need to rewrite the upper-level application software. Summary of the Invention

[0003] This application is proposed to address the aforementioned problems. According to one aspect of this application, a method for deploying an edge-side model for intelligent driving is provided. The method includes: acquiring a pre-trained network model for intelligent driving, the network model comprising a graph composed of nodes; converting and compiling the network model, wherein the conversion and compilation involves a unified external interface call, and the interface is encapsulated into a basic functional plugin interface in the form of functional plugins; acquiring input data through the functional plugin interface; performing inference on the input data based on the compiled network model; and obtaining and outputting the inference result.

[0004] In one embodiment of this application, the unified external call interface is encapsulated into a basic functional plugin interface in the form of functional plugins, including: extracting a common external startup interface for the startup initialization process, AI chip backend resource initialization, startup, and parameter configuration; extracting a unified external interface for the model inference process; encapsulating an external call interface for subscribing to and pushing messages from the message bus; and encapsulating the common startup interface, the unified external interface, and the external call interface into a basic functional plugin interface in the form of functional plugins.

[0005] In one embodiment of this application, the conversion and compilation of the network model includes: performing online conversion and compilation of the network model.

[0006] In one embodiment of this application, the conversion and compilation of the network model includes: loading end-side configuration information; allocating memory in the computer to create weight and bias information based on the configuration information; loading the graph information corresponding to the network model; setting all nodes of the graph; adding connection information between each node and weight and bias data within each node; and setting the output nodes of the graph.

[0007] In one embodiment of this application, before performing inference on the input data, the method further includes: preprocessing the input data, wherein the preprocessed data is used for the inference, and the preprocessing is implemented through multi-stream parallel processing.

[0008] In one embodiment of this application, the preprocessing is implemented through multi-stream parallel processing, including: dividing the input data into at least two parts and allocating a processing stream to each part of the input data; each processing stream simultaneously processes its respective part of the data.

[0009] In one embodiment of this application, before acquiring input data, the method further includes: segmenting the compiled network model according to the artificial intelligence chip and graph search strategy on the intelligent driving terminal side, associating the segmented subgraphs with the corresponding artificial intelligence chips, wherein the hardware driver and acceleration functions of the artificial intelligence chips are encapsulated into standard interfaces; customizing kernel functions based on the subgraphs and the standard interfaces to achieve heterogeneous execution of the preprocessing of the input data; wherein, the reasoning of the input data based on the compiled network model includes: calling the corresponding artificial intelligence chip to perform reasoning based on the preprocessed data.

[0010] In one embodiment of this application, the network model is a model of an open protocol neural network exchange format.

[0011] In one embodiment of this application, the output of the inference result includes: publishing the inference result to a bus so that an application can receive the inference result if it subscribes to the corresponding topic.

[0012] According to another aspect of this application, an intelligent driving end-side model deployment apparatus is provided. The apparatus includes a memory and a processor. The memory stores a computer program that is executed by the processor. When the computer program is executed by the processor, it causes the processor to perform the above-described intelligent driving end-side model deployment method.

[0013] According to another aspect of this application, a storage medium is provided, on which a computer program is stored, which, when running, executes the above-described intelligent driving edge-side model deployment method.

[0014] The intelligent driving edge-side model deployment method and apparatus according to the embodiments of this application solve the problems of numerous edge-side perception and inference frameworks and inconsistent interfaces by unifying the external calling interface during the conversion and compilation of the network model and encapsulating the interface into a basic functional plug-in interface in the form of functional plug-ins during the deployment process. This can improve the repeatability of the code, reduce the difficulty of algorithm deployment, accelerate the implementation of the algorithm, save development costs, and shorten the product development cycle. Attached Figure Description

[0015] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 A schematic flowchart illustrating a method for deploying an intelligent driving edge-side model according to an embodiment of this application is shown.

[0017] Figure 2 A schematic diagram of the structure of an ADAS vision framework according to an embodiment of this application is shown.

[0018] Figure 3 This diagram illustrates the message bus data flow of an ADAS vision framework according to an embodiment of this application.

[0019] Figure 4 A flowchart illustrating the ADAS vision framework according to an embodiment of this application is shown.

[0020] Figure 5 This document illustrates the execution flowchart of nodes during model inference in an ADAS vision framework according to an embodiment of this application.

[0021] Figure 6 A schematic structural block diagram of an intelligent driving end-side model deployment device according to an embodiment of this application is shown. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of this application, and not all of the embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0023] Deep learning has become a major research direction in the field of artificial intelligence in recent years. The main task of deep learning is to construct deep convolutional neural networks (DNNs) and use a large amount of sample data as input to the network. After a parameter training process, a model with powerful analytical and recognition capabilities is obtained. The trained network model contains the features of the sample data and can be used in autonomous driving to identify obstacles around the vehicle. Depending on the application scenario, the trained network model can run on large servers or small embedded devices (edge-side applications). Applying the trained network model to different scenarios is called "inference" or "deployment."

[0024] With the advent of the Internet of Things and the 5G era and the rise of edge computing, deep learning object detection networks are gradually developing from the server side to the embedded side, realizing the interconnection and perception of everything. As a result, deep learning-based intelligent driving technology has also emerged.

[0025] In the application of intelligent driving, various sensors installed on the car, including cameras, lidar, millimeter-wave radar, and other devices, collect data on the vehicle's surrounding environment and input it into a deep learning model for inference, thereby enabling the vehicle to perceive its surrounding environment. This perception includes information on traffic participants (vehicles, pedestrians), road infrastructure (lane markings, traffic signs, traffic lights), or surrounding obstacles (curb stones, potholes, shrubs), etc. This information is then given to the planning and control module of the intelligent driving system to enable the system to make decisions and control the vehicle's next action.

[0026] In Advanced Driving Assistance Systems (ADAS) applications on the edge of intelligent driving systems, model deployment involves inferring the trained deep learning network at the edge. Inference can be understood as the output process of the previously learned data. The network uses what it has learned during training to perceive the real-world scene and thus understand the vehicle's surrounding environment. Supporting the network model's inference output is a set of environmental perception software and a perception inference framework running on the edge, in addition to the trained model. The inference framework can be understood as an ordered combination of software interfaces; its constituent units are interfaces that perform computational operations on the model. In practical use, the corresponding interface in the framework is called to load the trained model file (usually a file in a specific format, containing an ordered sequence of computational units, which can be simply understood as ordered matrix vector addition, subtraction, multiplication, and division units). This action includes allocating computing resources (memory, CPU, GPU, process space, etc.). The computation process is roughly as follows: based on the arrangement and combination of operators in the model file, and their order, each computational unit within the model is calculated to obtain the final result. This process is inference, also known as model deployment.

[0027] In edge-side intelligent driving, the process of a vehicle perceiving its surroundings is roughly as follows: First, sensors installed on the vehicle (cameras, radar, inertial measurement units (IMUs), etc.) collect road condition information (including information on road infrastructure and surrounding obstacles mentioned earlier). Second, this information is parsed, extracted, and processed by algorithms to obtain real-time information about the vehicle's surroundings and the vehicle itself. This includes specific state information (object attribute information, object position coordinates relative to the vehicle, object motion state) of pedestrians, vehicles, parking spaces, obstacles, etc., as well as the vehicle's own speed, angle, and acceleration information. Finally, this information is transmitted to the vehicle planning and control terminal via Ethernet or Controller Area Network (CAN). The vehicle planning and control terminal then makes decisions about the vehicle's next action based on this information.

[0028] The inventors discovered that in the practical application of intelligent driving, edge AI chips (CPU, NPU, GPU) with different computing power are often used depending on the functional scenario. Due to the scarcity of chip resources, chips from different manufacturers and with different architectures are used. Because the chip structures and functions produced by various hardware manufacturers differ greatly, the performance of the same code implemented on different hardware varies significantly. Therefore, to maximize chip performance, code optimization for the hardware itself is necessary. This means that the same algorithm will have different optimized code on different hardware, causing significant problems for developers. If a certain function needs to be implemented on a different platform, the original code becomes unusable. This results in a lack of decoupling between software and hardware, non-reusable software code, and the need to rewrite the upper-level application software.

[0029] Furthermore, the solutions provided by intelligent driving software and hardware vendors are all based on specific hardware architectures. Examples include the MDC intelligent driving solution based on Huawei's Ascend chip; the intelligent driving solution based on NVIDIA series chips; and the intelligent driving solution based on Horizon Robotics chips. These solutions are all based on their own chips. This presents a problem: each chip manufacturer has its own software and hardware architecture and AI toolchain. For parties that need to use different AI chips due to different application scenarios, such as automotive OEMs, when support for a particular AI chip is required, the OEM needs to understand the chip's software and hardware solutions, implement a completely new software and hardware architecture based on that chip's solution, and then re-implement the application (APP) based on that. This is undoubtedly redundant and inefficient.

[0030] Based on the aforementioned issues of coupling between multiple hardware vendors and multiple software and hardware solutions, this application provides a solution for deploying intelligent driving edge-side models, which is described below with reference to the accompanying drawings.

[0031] Figure 1 A schematic flowchart of an intelligent driving edge-side model deployment method 100 according to an embodiment of this application is shown. Figure 1 As shown, the intelligent driving edge-side model deployment method 100 may include the following steps:

[0032] In step S110, a pre-trained network model for intelligent driving is obtained, the network model including a graph composed of nodes.

[0033] In step S120, the network model is converted and compiled. During the conversion and compilation, a unified external interface is called, and the interface is encapsulated into a basic functional plugin interface in the form of functional plugins.

[0034] In step S130, input data is obtained through the functional plug-in interface, and inference is performed on the input data based on the compiled network model to obtain and output the inference result.

[0035] In the embodiments of this application, the concept of a graph is introduced, viewing the network model as a graph composed of one or more nodes, as defined by the OpenVX international standard. This standard defines a set of general standard interfaces in the field of computer vision and references the concept of "flow." A flow refers to the execution of tasks like a pipeline, where each task execution point performs its corresponding computational task. The entity executing the task is called the target, and the task being executed is called a node. A directed acyclic sequence composed of individual nodes is called a graph in the standard. In the intelligent driving application of this application, the process of acquiring sensor data, parsing and processing the data, feeding it to the perception algorithm, and then outputting the perception algorithm can be regarded as a pipeline, and the entire process is represented by a graph. The graph consists of one or more functional nodes. A node is a small unit of data processing, which can be understood as a small data processing module. It has its corresponding input and output data, kernel function (which can be understood as a function with hardware acceleration), and the processing function can select the target device for its computation (DSP, GPU, or NPU hardware architecture) according to the characteristics of the computation. Nodes are added to the graph through a node creation function. The advantage of using the concept of a graph to implement the above process lies in the layered implementation of the vision processing system. Application-layer nodes are developed on programmable processors, and these nodes are easily connected to a graph. For different chips, only the chip's hardware drivers and optimization acceleration functions need to be encapsulated into standard international interfaces. Based on this, optimized execution across different heterogeneous processors can be achieved, unifying the upper-layer application interface and enabling adaptation to multiple hardware processors. Furthermore, in this application, during the deployment of the network model, by unifying the external calling interface during the conversion and compilation of the network model, and encapsulating the interface into basic functional plugin interfaces in the form of functional plugins, the problem of numerous edge-side perception inference frameworks and inconsistent interfaces can be solved. This improves code repeatability, reduces the difficulty of algorithm deployment, accelerates algorithm implementation, saves development costs, and shortens the product development cycle.

[0036] In the embodiments of this application, the aforementioned unified external call interface may include: extracting a common startup interface for the startup initialization process, AI chip backend resource initialization, startup, and parameter configuration; extracting a unified external interface for the model inference process; encapsulating the external call interface for subscribing to and pushing messages from the message bus; and encapsulating these unified interfaces into basic functional plugin interfaces in the form of functional plugins, and implementing the specific inference process of each network model in specific implementation classes.

[0037] In the embodiments of this application, the conversion compilation in step S120 can be performed online by a separate model compilation module within the inference framework on the driver's side. The inventors have found that the deployment of edge AI chips typically involves converting the trained network model into a lightweight network model that can run on the edge. This process is generally done using offline model conversion tools. However, this presents a problem: once the edge inference framework is developed, the algorithm is constantly iterating, possibly provided by a third-party algorithm vendor. This can lead to inconsistencies between the inference engine and the conversion tool versions, resulting in the converted model being unusable on edges with different engine versions. Therefore, to completely solve this problem, this application integrates model conversion compilation into the inference framework as a separate model compilation module, enabling online conversion of the algorithm model and allowing for multiple reuses after a single conversion.

[0038] In the embodiments of this application, the conversion and compilation of the network model may include: loading the terminal configuration information (including the AI ​​chip processor type); based on the configuration information and the compiled model file, allocating memory in the computer to create weight and bias information (with marking); loading the information of the graph (also called the computation graph) corresponding to the network model; setting all nodes of the computation graph; adding connection information between each node and weight and bias data within each node; and setting the output nodes of the computation graph.

[0039] This involves adding connection information between the parent and child nodes of each node, as well as weight and bias data within each node. (The trained model contains information about several nodes, each containing inputs, outputs, weights, and biases. These weights and biases are parameters in the model and exist from the moment the model is trained. This is based on the test results (inference accuracy) of the test samples after network training.) After setting the output nodes of the computation graph, the entire model structure information is loaded into the structure computation graph. The final output node settings are also determined by the model; the output is different for each algorithm model. Here, the output node is the node that ultimately outputs the computation result. It should be noted that the algorithm model is actually composed of individual nodes. The work done in this application is to allocate resources more efficiently to execute the nodes in the algorithm model to obtain the inference result.

[0040] In embodiments of this application, before inference on the input data, method 100 may further include the step of: preprocessing the input data, with the preprocessed data used for inference, wherein the preprocessing is implemented through multi-stream parallel processing. The inventors have found that currently, model deployment uses a single edge processor and a single stream for processing a single model, without employing multi-core, multi-device, and multi-stream parallel computing. However, some hardware platforms with heterogeneous computing support parallel computing operations to improve inference performance, which is undoubtedly a boon for real-time applications like autonomous driving. Based on the multi-core, multi-device, and multi-stream parallel computing described above, operations involving numerous preprocessing steps for model input data, and computationally intensive operations (such as copying, format conversion, scaling, and filtering of image data), can be processed using operator-based streaming or distributed to suitable target devices. This also greatly helps reduce inference time.

[0041] For example, preprocessing can be implemented through multi-stream parallel processing, including: dividing the input data into at least two parts and assigning a processing stream to each part of the input data, with each processing stream processing the input data simultaneously. Each processing stream can be a single stream. During data processing, a single stream first acquires its portion of the input data, transfers this portion of the input data from CPU memory to GPU memory, then the GPU performs the computation and saves the result in GPU memory. Finally, the processing result of this portion is copied from GPU memory to CPU memory. After obtaining the processing results of all single streams, the preprocessing of the input data is achieved. By performing data preprocessing through the above-described multi-stream parallel processing method, the system's computing power is effectively utilized, improving the efficiency of data processing.

[0042] In the embodiments of this application, before acquiring input data, method 100 may further include the following steps: segmenting the compiled network model according to the artificial intelligence chip and graph search strategy on the intelligent driving terminal side, and associating the segmented subgraphs with the corresponding artificial intelligence chips, wherein the hardware driver and acceleration functions of the artificial intelligence chips are encapsulated into standard interfaces; customizing kernel functions based on subgraphs and standard interfaces to achieve heterogeneous execution of preprocessing of input data; wherein, inference based on the compiled network model includes: calling the corresponding artificial intelligence chip for inference based on the preprocessed data.

[0043] As mentioned earlier, based on the OpenVX international standard, the network model can be viewed as a graph composed of nodes. Nodes are added to the graph through node creation functions, and application-layer nodes are developed on programmable processors, easily connecting these nodes to a graph. For different chips, only the chip's hardware drivers and optimization acceleration functions need to be encapsulated into a standard international interface. Based on this, optimized execution across different heterogeneous processors can be achieved, unifying the upper-layer application interface and realizing adaptation to multiple hardware processors. Since this standard only supports user kernel functions on the host CPU side, custom target hardware devices (target hardware refers to the AI ​​hardware processor or chip used, which will depend on the supported ADAS) can be created based on this standard. This adds kernel function acceleration support to other target hardware in the system, achieving adaptation to different processors.

[0044] Because the model input data undergoes numerous preprocessing operations, some of which are computationally intensive, the network model can be divided into subgraphs. Each subgraph can be assigned a corresponding AI chip, allowing these chips to be used to preprocess the relevant nodes, achieving heterogeneous execution of preprocessing. This heterogeneous execution fully utilizes the computing power of edge AI chips, accelerating algorithm inference and enhancing the real-time performance of model perception. This is crucial for scenarios with high real-time requirements, such as autonomous driving ADAS systems. Furthermore, some operators in the model may not be supported on the edge, with some operators better suited for CPUs and others for GPUs. In such cases, online memory node switching categorizes and merges nodes containing various operators, enabling heterogeneous computation of the edge model without requiring code rewriting due to hardware changes.

[0045] Following this, network model inference can be introduced based on the standard protocol interface, as described in step S130. OpenVX also supports neural networks. A trained network model is actually a graph composed of individual nodes, where each node is a layer in the network. As mentioned earlier, a node is an instance of a kernel function. For each AI chip platform, it is only necessary to implement the corresponding node support according to the standard interface, including the input and output data objects that the node depends on, the kernel function, and setting the target device for the node. In this way, the OpenVX framework can immediately route to the corresponding device during inference.

[0046] The aforementioned nodes are divided into modules according to the process flow, which can be divided into sensor module, input preprocessing module, output postprocessing module, model inference module, and display module. Each module's corresponding data structure type, nodes of the standard data type, node input / output parameter configuration, the next node associated with the output, output file path, and other key member variables are established. Interfaces related to creating this module are provided, including interfaces for creating and destroying the data objects required for nodes, interfaces for creating nodes based on data objects, adding the node to the graph, and interfaces for creating the result output nodes associated with the node. Based on the above-defined modules, graphs for each sub-function are constructed. Image acquisition is used as one graph to provide services to other functions, parking space detection algorithms are used as another graph, and object detection algorithms are used as yet another graph. Function scheduling is performed on these graphs according to the product's functional definition, distributing data acquired by the camera to specific graphs under certain operating conditions.

[0047] The external interface calls are extracted, including the startup initialization process, AI chip backend resource initialization, startup, and parameter configuration, resulting in a common startup interface. The edge-side model inference process is abstracted, and a unified external interface is extracted. External call interfaces for subscribing to and pushing messages from the message bus are encapsulated. These unified interfaces are encapsulated into basic functional plugin interfaces in the form of functional plugins. The specific inference process of each network model is implemented in the specific implementation class, as described above.

[0048] Finally, the inferred results are published to the bus via the communication module. Specific applications only need to subscribe to the topic to receive the corresponding inference results (multiple applications can subscribe at the same time), thus achieving low coupling between the application layer and the inference framework.

[0049] The following is combined Figures 2 to 5 This document describes in more detail an example of a method for deploying an intelligent driving edge model according to embodiments of this application.

[0050] Figure 2 A schematic diagram of the ADAS vision framework according to an embodiment of this application is shown. The diagram illustrates a flowchart of acquiring data from sensors (including cameras, LiDAR, GPS, and IMU) installed on the vehicle. The process is as follows: The specific sensor device is accessed via its sensor interface; corresponding sensor data is acquired, such as YUV or RGB data from the camera, point cloud data from the LiDAR, and acceleration and angular velocity information from the IMU. The data is then formatted as required by the algorithm input and pushed to the BUS bus via a message push module. The inference framework subscription module then subscribes to the specific sensor data.

[0051] This process is manifested in Figure 3The above is the image data acquisition graph. This graph consists of a node whose input is raw YUV data from the sensor, and whose output is an RGB image. The node's kernel function converts NV12 to RGB format. In the implementation, the node can run on the corresponding hardware device according to the hardware environment. For example, if there is a DSP device available in the hardware resources, the kernel function can be written as an accelerated function on the DSP chip. The OpenVX standard has a standard definition interface for kernel function pointers. When implementing a node, simply bind the kernel to this node, and then call the standard interface `vxSetNodeTarget` to set the target device for the node to run on the corresponding DSP device. This achieves the binding of the node, kernel, and corresponding target device. Alternatively, priority can be set for the target running device, for example, setting the first priority to DSP and the second priority to CPU. This way, if there is no corresponding DSP hardware in the hardware environment, the CPU will be selected. After binding the kernel and registering it on the Context, the corresponding node can be generated through relevant function calls.

[0052] The general process for generating custom nodes using a custom Kernel function is as follows: 1) Assign a unique name and enumeration value to the custom Kernel. In the standard framework, the Context manages Kernels based on their names and enumeration values. Kernels registered on the Context have unique names and enumeration values; the name is a string. 2) Implement the four functions mentioned above based on the custom Kernel's functionality: initialization function, destructor function, verification function, and execution function. 3) Register the Kernel on the Context by calling the `vxAddUserKernel()` function. 4) Set a "signature" for the Kernel's parameters. 5) Complete the customization by calling the `vxFinalizeKernel()` function. 6) Encapsulate the API for easy generation of the corresponding nodes.

[0053] After generating nodes, the nodes are assembled into a graph, a context is created, the graph is built based on the context, and then the generated nodes are added to the graph. A callback function is registered when creating the graph to push the inference results.

[0054] Figure 3 A message bus data flow diagram of an ADAS vision framework according to an embodiment of this application is shown. Figure 3 As shown, once the graph has completed pushing the sensor data, the data has arrived. Figure 3 The BUS message bus in the framework abstracts a basic interface. This basic interface implements the initialization, shutdown, push, and subscription interfaces for the message inference mechanism. Each implementation graph is derived from these interfaces. For example... Figure 3 The obstacle detection model inference graph shown can be instantiated and the basic interface mentioned above implemented. There is no interference between the newly added functional graphs. In this way, each functional inference module achieves a software design with high cohesion and low coupling. For example, the initialization interface of all communication parts is abstracted to complete the activation of the message bus push mechanism and the loading of device configuration files.

[0055] Figure 4 A flowchart illustrating the ADAS vision framework according to an embodiment of this application is shown.

[0056] like Figure 4 As shown in section 401, a trained ONNX perceptual model with specific functions is obtained. ONNX is an open format used to represent deep learning models. Most mainstream deep learning frameworks support ONNX, such as TensorFlow, PyTorch, and MXNet. The ONNXRuntime is used to convert the models trained by these deep learning frameworks into ONNX format models.

[0057] Section 402 implements the model compilation part of the network model inference framework. The role of this part is to compile the pre-trained network model into a model file that can be recognized by the target AI chip. This file is also a graph (a graph specifically for the neural network model). The steps involved in this process have already been described in the previous steps of converting and compiling the network model, and will not be repeated here.

[0058] The resulting edge-side conversion model is a directed acyclic graph (DAG) based on a specific inference framework, hereinafter referred to as the computation graph. The node list and tensor list together describe the main structure of the network model. Nodes primarily describe the node's dependent tensors and operators. This structure is also bound to the corresponding edge AI chip device to perform the computations corresponding to the graph. Furthermore, based on the capabilities of the operators supported by the edge AI chip device, nodes corresponding to operators not supported by the device are first assigned to the CPU. Then, a depth-first search is used to obtain the number of associated nodes after the unsupported node. The operator types of the associated nodes and the computational complexity are then considered to determine whether the unsupported node's operator needs to be implemented on the AI ​​chip. Ultimately, the node and its subsequent nodes are assigned to the AI ​​chip, or simply assigned to the CPU.

[0059] As described above, different subgraphs will be obtained. Then, connected subgraphs on the same device will be merged to prepare for backend inference. The structure describing the computation graph information, st_graph, can have the following members: 1) a list of all nodes in the computation graph; 2) a list of all tensor data in the computation graph; 3) the computing hardware (NPU or GUP) allocated to execute the computation graph; 4) a list of subgraphs running on different computing hardware, which defaults to CPU.

[0060] In section 403, which involves the custom kernel function and node mentioned above, the standard interface of the international standard OpenVX is used to implement a node that can be executed heterogeneously (running on different AI chips), which will not be elaborated here.

[0061] In section 404, the inference process first deserializes the edge model obtained in section 402, loading the computation graph into memory as a structure variable. As mentioned earlier, the main member variables of this structure include a node list, input and output nodes, a tensor list, hardware AI chip information associated with the computation graph, and subgraphs (if any). These variables play a crucial role in actual inference. The inference engine schedules operations based on the AI ​​chip information associated with the graph and whether all inputs to the subgraphs are ready. If all inputs are ready, then the subgraph operations are scheduled for execution. After the subgraph is executed, its output tensor is ready, and the input tensors of the subgraphs connected to it are also ready. The operations are then executed in a loop until the entire computation graph is completed, yielding the output result. Depending on the actual AI chip on the edge, there may be one or more subgraphs running on different AI chips, such as one subgraph running on the CPU and multiple other subgraphs running on the CPU. This is heterogeneous computing as described above. By maximizing the utilization of AI chip resources in this way, inference performance is accelerated. Figure 5 This is the execution flowchart of the node during model inference.

[0062] In section 405: After inference, the data is parsed based on the output data structure of each network model to obtain visualized inference results. The results are then packaged in a specified data format and pushed to the BUS message bus. At this point, the application layer APP only needs to know the topic corresponding to the packaged data to subscribe to and obtain the data.

[0063] The above exemplarily describes a method for deploying an intelligent driving edge-side model according to an embodiment of this application. Based on the above description, the method for deploying an intelligent driving edge-side model according to an embodiment of this application solves the problem of numerous edge-side perception and inference frameworks and inconsistent interfaces by unifying the external calling interface during the conversion and compilation of the network model and encapsulating the interface into a basic functional plugin interface in the form of functional plugins during the deployment process. This improves code repeatability, reduces the difficulty of algorithm deployment, accelerates algorithm implementation, saves development costs, and shortens the product development cycle. Furthermore, the method for deploying an intelligent driving edge-side model according to an embodiment of this application incorporates model conversion and compilation into the inference framework, existing as a separate model compilation module, enabling online conversion of the algorithm model and reuse after a single conversion. Further, the method for deploying an intelligent driving edge-side model according to an embodiment of this application performs multi-stream parallel computation on the edge-side operators, accelerating model inference speed. Further, the method for deploying an intelligent driving edge-side model according to an embodiment of this application uses a scheduler to achieve parallel heterogeneous execution of edge-side multi-processor model inference, fully utilizing the computing power of edge-side AI chips to accelerate the inference process. Furthermore, in the intelligent driving edge model deployment method according to the embodiments of this application, the online memory node conversion of the model categorizes and integrates the nodes containing various operators, thereby realizing heterogeneous computing of the edge model without rewriting the code due to hardware changes.

[0064] The following is combined Figure 6 This application describes another aspect of the intelligent driving end-side model deployment apparatus. Figure 6 A schematic block diagram of an intelligent driving edge-side model deployment device 600 according to an embodiment of this application is shown. Figure 6 As shown, the intelligent driving edge-side model deployment device 600 according to an embodiment of this application may include a memory 610 and a processor 620. The memory 610 stores a computer program executed by the processor 620. When the computer program is executed by the processor 620, the processor 620 performs the intelligent driving edge-side model deployment method described above according to an embodiment of this application. Those skilled in the art can understand the specific operation of the intelligent driving edge-side model deployment device according to the embodiments of this application in conjunction with the foregoing description; for the sake of brevity, it will not be described in detail here.

[0065] Furthermore, according to embodiments of this application, a storage medium is also provided, on which program instructions are stored. When executed by a computer or processor, these program instructions are used to perform corresponding steps of the intelligent driving end-side model deployment method of this application. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0066] In addition, a computer program is provided, which is used by a computer or processor to execute the corresponding steps of the intelligent driving end-side model deployment method of the embodiments of this application.

[0067] Based on the above description, the intelligent driving edge-side model deployment method and apparatus according to the embodiments of this application solve the problem of numerous edge-side perception inference frameworks and inconsistent interfaces by unifying the external calling interface during the network model conversion and compilation process and encapsulating the interface into basic functional plugin interfaces in the form of functional plugins during the network model deployment process. This improves code repeatability, reduces the difficulty of algorithm deployment, accelerates algorithm implementation, saves development costs, and shortens the product development cycle. Furthermore, the intelligent driving edge-side model deployment method and apparatus according to the embodiments of this application incorporate model conversion and compilation into the inference framework, existing as a separate model compilation module, enabling online conversion of the algorithm model and reuse multiple times after a single conversion. Further, the edge-side operators in the intelligent driving edge-side model deployment method and apparatus according to the embodiments of this application perform multi-stream parallel computation, which accelerates model inference speed. Further, the intelligent driving edge-side model deployment method and apparatus according to the embodiments of this application implement parallel heterogeneous execution of edge-side multi-processor model inference through a scheduler, fully utilizing the computing power of edge-side AI chips to accelerate the inference process. Furthermore, according to the intelligent driving end-side model deployment method and device of this application embodiment, the online memory node conversion of the model categorizes and integrates the nodes containing various operators, realizing heterogeneous computing of the end-side model without rewriting the code due to hardware changes.

[0068] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0069] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0071] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0072] Similarly, it should be understood that, in order to streamline this application and aid in understanding one or more of the various inventive aspects, features of this application may sometimes be grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with features fewer than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0073] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0074] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0075] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0076] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0077] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A method for deploying an intelligent driving edge-side model, characterized in that, The method includes: Obtain a pre-trained network model for intelligent driving, the network model comprising a graph of nodes; The network model is transformed and compiled, wherein a unified external interface is called during the transformation and compilation, and the interface is encapsulated into a basic functional plugin interface in the form of functional plugins. Input data is obtained through the functional plug-in interface, and inference is performed on the input data based on the compiled network model to obtain and output the inference result; The method further includes, before performing inference on the input data: The input data is preprocessed, and the preprocessed data is used for the inference. The preprocessing is implemented through multi-stream parallel processing. Before acquiring input data, the method further includes: segmenting the compiled network model according to the artificial intelligence chip and graph search strategy on the intelligent driving terminal side, associating the segmented subgraphs with the corresponding artificial intelligence chips, wherein the hardware driver and acceleration functions of the artificial intelligence chips are encapsulated into standard interfaces; customizing kernel functions based on the subgraphs and the standard interfaces to achieve heterogeneous execution of the preprocessing of the input data; wherein, the reasoning of the input data based on the compiled network model includes: calling the corresponding artificial intelligence chip to perform reasoning based on the preprocessed data.

2. The method according to claim 1, characterized in that, The unified external call interface is encapsulated into basic functional plugin interfaces through functional plugins, including: For the startup initialization process, AI chip backend resource initialization, startup, and parameter configuration, a public startup interface is extracted for external use; For the model inference process, a unified external interface is extracted; Encapsulates external calling interfaces for subscribing to and pushing messages from the message bus; The public startup interface, the unified external interface, and the external call interface are encapsulated into basic functional plugin interfaces in the form of functional plugins.

3. The method according to claim 1 or 2, characterized in that, The process of converting and compiling the network model includes: The network model is converted and compiled online.

4. The method according to claim 3, characterized in that, The process of converting and compiling the network model includes: Load the configuration information on the client side, and allocate memory in the computer to create weight and bias information based on the configuration information; Load the information of the graph corresponding to the network model, and set all nodes of the graph; Add connection information between each node and weight and bias data within each node; Configure the output node of the graph.

5. The method according to claim 1, characterized in that, The preprocessing is implemented through the multi-stream parallel processing, including: The input data is divided into at least two parts, and a separate processing stream is assigned to each part of the input data. Each of the processing streams processes its own portion of the data simultaneously.

6. The method according to claim 1 or 2, characterized in that, The network model is a model of an open protocol neural network exchange format.

7. The method according to claim 1 or 2, characterized in that, The output reasoning result includes: The inference results are published to the bus so that applications that subscribe to the corresponding topic can receive the inference results.

8. A device for deploying intelligent driving edge-side models, characterized in that, The device includes a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform the intelligent driving edge model deployment method as described in any one of claims 1-7.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when running, executes the intelligent driving edge model deployment method as described in any one of claims 1-7.

Images