Monocular depth estimation method and device based on neural network and edge computing chip

Abstract

The application discloses a monocular depth estimation method and device based on a neural network and an edge computing chip, and the method comprises the following steps: acquiring a training data set of a camera image, and training a depth estimation network by using the training data set to obtain a trained depth estimation network; performing quantization operation on network parameters of the trained depth estimation network by using multiple pictures in the training data set to obtain a convolution tensor; dividing the convolution tensor by using an optimal tensor division mode to generate network code data; and obtaining a depth map estimated by a real-time camera image by network calculation of the trained depth estimation network by using the real-time camera image and the network code data. The application realizes deployment and running of a monocular depth estimation network on an ultra-low-power computing chip, which is of great significance to a nano unmanned platform which can only use a monocular camera as a sensing unit, and can significantly improve the understanding ability of the nano unmanned platform for a scene.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a monocular depth estimation method and apparatus based on neural networks and edge computing chips. Background Technology

[0002] Depth information is crucial for many robotic tasks, such as mapping, localization, and obstacle avoidance. Existing depth sensors (e.g., LiDAR, structured light sensors) are typically bulky, heavy, and power-intensive. These limitations make them unsuitable for nano-robot platforms (such as nano-unmanned aerial vehicles). However, the low cost, compact size, and high energy efficiency of nano-unmanned platforms have spurred the motivation to use monocular cameras for depth estimation.

[0003] Previous research on monocular depth estimation has mainly focused on improving accuracy. However, these monocular depth estimation algorithms have high computational complexity and memory consumption, making them difficult to deploy in robotic systems, especially on platforms with limited computing resources and power consumption. Therefore, a key challenge is to balance the accuracy of the algorithm with the computational resource requirements. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] To address this, this invention proposes a monocular depth estimation method based on neural networks and edge computing chips, focusing primarily on a more lightweight monocular depth estimation network, its quantization, and its deployment on low-power chips. At the network design level, a more lightweight encoder is employed, and network pruning is used to trim the network. The trained network parameters are quantized to 8 bits, and a reasonable partitioning method is used to deploy the network's computation on the GAP8 chip. Through various operations, real-time operation of the monocular depth estimation network on the low-power GAP8 chip is achieved. This invention has significant implications for the intelligent application of nano-robot platforms. Furthermore, experimental verification demonstrates that this invention yields reliable results and acceptable speed, possessing good engineering application value.

[0006] Another objective of this invention is to provide a monocular depth estimation device based on neural networks and edge computing chips.

[0007] To achieve the above objectives, this invention proposes a monocular depth estimation method based on neural networks and edge computing chips, comprising:

[0008] Obtain a training dataset of camera images, and use the training dataset to train the depth estimation network to obtain a trained depth estimation network.

[0009] The network parameters of the trained depth estimation network are quantized using multiple images from the training dataset to obtain convolutional tensors.

[0010] The convolutional tensor is partitioned using the optimal tensor partitioning method to generate network code data;

[0011] Using real-time camera images and the network code data, a depth map estimated from the real-time camera images is obtained through network computation of the trained depth estimation network.

[0012] In addition, the monocular depth estimation method based on neural networks and edge computing chips according to the above embodiments of the present invention may also have the following additional technical features:

[0013] Furthermore, in one embodiment of the present invention, after obtaining the trained depth estimation network, the method further includes: pruning the trained depth estimation network using the NetAdapt algorithm to filter out a predetermined number of convolutional kernels in the network.

[0014] Furthermore, in one embodiment of the present invention, the step of quantizing the network parameters of the trained depth estimation network using multiple images from the training dataset to obtain convolutional tensors includes:

[0015] Multiple images from the training dataset are used as input to the trained depth estimation network to obtain the reference range [α] of the tensor t for each convolutional layer. t ,β t ), and map it to an N-bit pure integer tensor.

[0016]

[0017] ε t =(β) t -α t ) / (2 N -1)

[0018] Where, ε t This is the scaling factor.

[0019] Furthermore, in one embodiment of the present invention, the step of pruning the trained depth estimation network using the NetAdapt algorithm to filter out a predetermined number of convolutional kernels in the network includes: performing multiple rounds of iterative pruning on the trained depth estimation network using the NetAdapt pruning method; in each round of iteration, deleting a predetermined number of convolutional kernels from each layer of the network to obtain several sub-networks; and selecting the sub-network with the highest accuracy from the several sub-networks for the next round of iteration.

[0020] Furthermore, in one embodiment of the present invention, the step of using the optimal tensor partitioning method to partition the convolutional tensor to generate network code data includes: using the AutoTiler network tool to select the optimal tensor partitioning method to partition the convolutional tensor, so as to encapsulate the computation process of the trained depth estimation network on the ultra-low power edge computing chip into code data represented in C language.

[0021] To achieve the above objectives, another aspect of the present invention proposes a monocular depth estimation device based on neural networks and edge computing chips, comprising:

[0022] The network training module is used to acquire a training dataset of camera images and use the training dataset to train the depth estimation network to obtain a trained depth estimation network.

[0023] The convolution quantization module is used to quantize the network parameters of the trained depth estimation network using multiple images in the training dataset to obtain convolution tensors.

[0024] The tensor partitioning module is used to partition the convolutional tensor using the optimal tensor partitioning method to generate network code data.

[0025] The depth estimation module is used to obtain a depth map estimated from the real-time camera image by using the real-time camera image and the network code data through the network computation of the trained depth estimation network.

[0026] The monocular depth estimation method and apparatus based on neural networks and edge computing chips in this invention utilizes convolutional neural networks to extract image depth features, encode and decode them to estimate the depth values ​​at different pixels in the image, quantize the trained network parameters, and decompose each layer of convolutional operations for deployment on an ultra-low power edge computing chip.

[0027] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0028] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0029] Figure 1 This is a flowchart of a monocular depth estimation method based on neural networks and edge computing chips according to an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the GAP8 chip architecture according to an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of a monocular depth estimation neural network structure according to an embodiment of the present invention;

[0032] Figure 4 This is a demonstration diagram of the operating performance of a monocular depth estimation network based on an ultra-low power chip according to an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of the NetAdapt iterative pruning method according to an embodiment of the present invention;

[0034] Figure 6 This is a schematic diagram of the feature space representation of a convolutional layer according to an embodiment of the present invention;

[0035] Figure 7 This is a schematic diagram of a monocular depth estimation device based on a neural network and an edge computing chip according to an embodiment of the present invention. Detailed Implementation

[0036] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] The following description, with reference to the accompanying drawings, describes a monocular depth estimation method and apparatus based on neural networks and edge computing chips according to embodiments of the present invention.

[0039] Figure 1 This is a flowchart of a monocular depth estimation method based on neural networks and edge computing chips according to an embodiment of the present invention.

[0040] like Figure 1 As shown, the method includes, but is not limited to, the following steps:

[0041] S1. Obtain the training dataset of camera images and use the training dataset to train the depth estimation network to obtain the trained depth estimation network.

[0042] S2, use multiple images from the training dataset to quantize the network parameters of the trained depth estimation network to obtain convolutional tensors;

[0043] S3 uses the optimal tensor partitioning method to partition the convolutional tensor to generate network code data;

[0044] S4 uses real-time camera images and network code data to obtain a depth map estimated from the real-time camera images through network computation of a trained depth estimation network.

[0045] Specifically, in this embodiment of the invention, a suitable depth dataset is selected to train the parameters of the depth estimation network so that the estimation accuracy of the network meets the requirements. In this embodiment, the NYU Depth v2 dataset is used.

[0046] To further reduce the weight of the given network, pruning was performed on the trained network by removing some convolutional kernels. The NetAdapt algorithm was used to prune the network.

[0047] Quantizing network parameters requires first selecting several images as input to obtain the reference range [α] of the tensor t for each convolutional layer. t ,β t Then map it to an N-bit pure integer tensor.

[0048]

[0049] ε t =(β) t -α t ) / (2 N -1)

[0050] Where, ε t Often referred to as the scaling factor, it scales the tensor from a floating-point number to an integer representation. The quantization stream enforces the quantization of the representation of all tensors in the network. Quantization is performed using the NNTOOL tool developed by GWT (GreenWaves Technologies).

[0051] Deploying models on the GAP8 chip enables and utilizes the hardware platform by generating C code that directly controls low-level in-memory computations. The main challenge lies in maximizing the parallel execution of all available cores while minimizing data transfer overhead. On the GAP8 chip, the primary challenge is the limited L1 memory (64kB), which forces the deployment tool to address optimization by dividing tensors in network computations into smaller data blocks called tiles, which are then moved between L2 and L1 memory. This invention uses the AutoTiler tool to select the optimal partitioning method and wraps it in the generated C code.

[0052] The computation process of the network on the ultra-low-power edge computing chip is encapsulated in C language code. Before calling this code, it is necessary to acquire images from the camera. The images are obtained by calling the interface in gap_sdk and stored as an array. Then, the image data is input, and the network's computation yields estimated depth information. By repeating this process iteratively, a real-time depth map estimated from the camera images can be obtained.

[0053] It is understood that this invention proposes a monocular depth estimation method based on convolutional neural networks and ultra-low-power chips. The implementation is deployed on the AIdeck platform, which mainly consists of GreenWaves Technologies' GAP8 chip and a Himax camera, expanding computing power and enabling complex AI-based workloads to run on nano-sized unmanned platforms. The GAP8 chip is a commercial embedded RISC-V multi-core processor derived from the PULP open-source project. The core of the GAP8 consists of an advanced RISC-V MCU and a programmable eight-core processor, with the architecture as follows... Figure 2 As shown. The code implementation in this invention is based on the Python language and uses PyTorch as the deep learning framework.

[0054] This embodiment uses the NYU Depth v2 dataset to train and prune the network and verify the final deployment results. The network structure is similar to FastDepth, but the MobileNetv1 front-end encoder is replaced with the more lightweight MobileNetv2 network. It also incorporates multiple back-end upsampling layers and cross-layer connections. The network structure is as follows: Figure 3 As shown. The NetAdapt pruning method was used to prune the network for a maximum of 23 rounds. The main principle of the pruning algorithm is as follows. Figure 5 In each iteration, NetAdapt selects and removes some convolutional kernels from each layer of the network to obtain several sub-networks, performs FineTune, and selects the sub-network with the highest accuracy for the next iteration.

[0055] This invention employs the NNTOOL tool to perform 8-bit quantization of the network parameters, enabling faster network computation on the chip. NNTOOL is an NN mapping tool developed by GWT (GreenWaves Technologies) as part of the GAP8 software development kit. NNTOOL performs layer fusion, post-training calibration, and quantization (8 / 16 bits), and folds and fuses batch normalization layers (BN layers) into the preceding convolutional layers, avoiding the need for intermediate buffers and reducing memory consumption. This invention uses NNTOOL to perform 8-bit post-training quantization of the network, employing a Conv-BN-ReLU fusion mode. This simplifies quantization and deployment, and reduces quantization errors. The tensor output by the Conv operation requires a higher precision representation than the input and weights—using 32 bits. However, this does not mean that a complete tensor of 32-bit elements needs to be generated and stored in the computation. Instead, each element is generated by Conv in 32 bits, but immediately reduced to 8 bits after passing through the ReLU or BN+ReLU operator. This operation significantly reduces precision loss, speeds up computation, and reduces memory waste.

[0056] The main challenge in deploying the model on the GAP8 chip lies in maximizing the parallel execution of all available kernels while minimizing data transfer overhead. On GAP8, the primary challenge is the limited L1 memory (64kB), which forces the deployment tool to address an optimization problem: dividing tensors in network computation into smaller data blocks, called tiles, and moving them between L2 and L1 memory. This problem is divided into two separate parts: 1) a set of optimized kernels specifically running on L1 data tiles; and 2) a tiling solver that defines the optimal tile size and generates the code for related data transfers between L2 and L1, including double buffering for all tensors. Figure 6 For reference, each layer in a CNN operates on a three-dimensional input tensor representing a feature space (one feature map per channel) and produces a new 3D activation tensor as the output. Specifically, convolutional layers consist of a linear transformation that maps the Kin input feature map to the Kout output feature map using Kin×Kout convolutional filters (or weight matrices). Therefore, in any convolutional layer, we can identify three distinct data spaces that can exist in one or more of the three dimensions (i.e.,...). Figure 6 The space is divided into tiles (W, H, and K). Similar considerations can be made for other layers in a CNN, allowing them to be processed in the same way. This invention uses the AutoTiler tool to help explore subsets of this space, select the optimal tiling configuration, and wrap the generated C code to efficiently move data between L2 and L1 memories using a cluster DMA controller.

[0057] In this example, the parameters are set as follows: chip CL frequency is 175MHz, FC frequency is 250MHz, L1 memory allocation is 46736 bytes, and L2 memory allocation is 250000 bytes. The test results of this embodiment are demonstrated as follows. Figure 4 As shown in the figure, each column from left to right represents the original image (grayscale), the true depth map, and the depth maps output by the original FastDepth network, the newly built monocular depth estimation network, and the networks pruned in iterations 7, 15, and 23, all running on the GAP8 platform. It can be seen that although the network was rebuilt and pruned to improve computational speed, resulting in a slight decrease in depth estimation accuracy compared to the original FastDepth network, it can still clearly distinguish between areas of higher and lower depth in the image, which is sufficient for downstream tasks on unmanned platforms.

[0058] The monocular depth estimation method based on neural networks and edge computing chips according to embodiments of the present invention is applicable to robot platforms with extremely low power consumption requirements. By deploying a lightweight monocular depth estimation neural network on an ultra-low power chip, image computation at a frame rate of 10fps can be achieved with a power consumption of only 393mW, enabling the present invention to be mounted on nano-sized unmanned platforms for intelligent applications on unmanned platforms.

[0059] To achieve the above embodiments, such as Figure 7 As shown, this embodiment also provides a monocular depth estimation device 10 based on neural networks and edge computing chips. The system 10 includes a network training module 100, a convolutional quantization module 200, a tensor partitioning module 300, and a depth estimation module 400.

[0060] The network training module 100 is used to acquire a training dataset of camera images and to train the depth estimation network using the training dataset to obtain a trained depth estimation network.

[0061] The convolutional quantization module 200 is used to quantize the network parameters of the trained depth estimation network using multiple images in the training dataset to obtain convolutional tensors.

[0062] Tensor partitioning module 300 is used to partition convolutional tensors using the optimal tensor partitioning method to generate network code data;

[0063] The depth estimation module 400 is used to obtain a depth map estimated from the real-time camera image by using real-time camera images and network code data through a trained depth estimation network.

[0064] Furthermore, following the aforementioned network training module 100, the following is also included:

[0065] The network pruning module is used to prune the trained depth estimation network using the NetAdapt algorithm to filter out a preset number of convolutional kernels in the network.

[0066] Furthermore, the aforementioned convolutional quantization module 200 is also used for:

[0067] Multiple images from the training dataset are used as input to the trained depth estimation network to obtain the reference range [α] of the tensor t for each convolutional layer. t ,β t ), and map it to an N-bit pure integer tensor.

[0068]

[0069] ε t =(β) t -α t ) / (2 N -1)

[0070] Where, ε t This is the scaling factor.

[0071] Furthermore, the aforementioned network pruning module is also used for:

[0072] The NetAdapt pruning method is used to perform multiple rounds of iterative pruning on the trained depth estimation network. In each round of iteration, a preset number of convolutional kernels are removed from each layer of the network to obtain several sub-networks.

[0073] Select the subnetwork with the highest accuracy from several subnetworks for the next round of iteration.

[0074] Furthermore, the tensor partitioning module 300 is also used to: select the optimal tensor partitioning method using the AutoTiler network tool to partition the convolutional tensors, so as to encapsulate the computation process of the trained depth estimation network on the ultra-low power edge computing chip into code data represented in C language.

[0075] The monocular depth estimation device based on a convolutional neural network and an ultra-low-power edge computing chip according to embodiments of the present invention is applicable to robot platforms with extremely low power consumption requirements. By deploying a lightweight monocular depth estimation neural network on an ultra-low-power chip, image computation at a frame rate of 10fps can be achieved with a power consumption of only 393mW, enabling the present invention to be mounted on nano-sized unmanned platforms for intelligent applications on unmanned platforms.

[0076] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0077] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A monocular depth estimation method based on neural networks and edge computing chips, characterized in that, Includes the following steps: Obtain a training dataset of camera images, and use the training dataset to train the depth estimation network to obtain a trained depth estimation network. The network parameters of the trained depth estimation network are quantized using multiple images from the training dataset to obtain convolutional tensors. The convolutional tensor is partitioned using the optimal tensor partitioning method to generate network code data; Using real-time camera images and the network code data, a depth map estimated from the real-time camera images is obtained through network computation of the trained depth estimation network. The method further includes: The NYU Depth v2 dataset was used to train and prune the network and verify the final deployment results. The network structure was based on the FastDepth network structure, with the MobileNetv1 in the front-end encoder replaced by the more lightweight MobileNetv2 network. Multiple back-end upsampling layers and cross-layer connections were also added. The NetAdapt pruning method was used to prune the network for up to 23 rounds. In each iteration, NetAdapt selects to delete a preset number of convolutional kernels from each layer of the network to obtain several subnetworks, and performs FineTune, selecting the subnetwork with the highest accuracy for the next iteration. The step of quantizing the network parameters of the trained depth estimation network using multiple images from the training dataset to obtain convolutional tensors includes: Multiple images from the training dataset are used as input to the trained depth estimation network to obtain the reference range of the tensor t for each convolutional layer. And map it to an N-bit pure integer tensor : in, This is the scaling factor; The step of partitioning the convolutional tensor using the optimal tensor partitioning method to generate network code data includes: The AutoTiler network tool is used to select the optimal tensor partitioning method to partition the convolutional tensors, so as to encapsulate the computation process of the trained depth estimation network on the ultra-low power edge computing chip into code data represented in C language. The method further includes: Tensors in network computation are divided into smaller data blocks, called tiles, and moved between L2 and L1 memory. This problem is divided into two separate parts: 1) a set of optimized kernels that run specifically on L1 data tiles; 2) a tiling solver that defines the optimal size of the tiles and generates the code for related data transfer between L2 and L1, including double buffering of all tensors. Each layer in the CNN runs on a 3D input tensor representing the feature space and produces a new 3D activation tensor as the output. Convolutional layers consist of linear transformations that map the Kin input feature map to the Kout output feature map using Kin×Kout convolutional filters. In any convolutional layer, three distinct data spaces are identified, which are divided into tiles in one or more of the three dimensions. A subset of these spaces is explored using the AutoTiler tool to select the optimal tiling configuration and packaged into C code that efficiently moves data between L2 and L1 memory using a cluster DMA controller.

2. A monocular depth estimation device based on a convolutional neural network and an ultra-low-power edge computing chip, characterized in that, include: The network training module is used to acquire a training dataset of camera images and use the training dataset to train the depth estimation network to obtain a trained depth estimation network. The convolution quantization module is used to quantize the network parameters of the trained depth estimation network using multiple images in the training dataset to obtain convolution tensors. The tensor partitioning module is used to partition the convolutional tensor using the optimal tensor partitioning method to generate network code data. The depth estimation module is used to obtain a depth map estimated from the real-time camera image by using the real-time camera image and the network code data through the network calculation of the trained depth estimation network. Also used for: The NYU Depth v2 dataset was used to train and prune the network and verify the final deployment results. The network structure was based on the FastDepth network structure, with the MobileNetv1 in the front-end encoder replaced by the more lightweight MobileNetv2 network. Multiple back-end upsampling layers and cross-layer connections were also added. The network pruning module is used to perform up to 23 rounds of iterative pruning on the network using the NetAdapt pruning method. In each iteration, NetAdapt selects and deletes a preset number of convolutional kernels from each layer of the network to obtain several subnetworks, performs FineTune, and selects the subnetwork with the highest accuracy from them for the next iteration. The convolution quantization module is also used for: Multiple images from the training dataset are used as input to the trained depth estimation network to obtain the reference range of the tensor t for each convolutional layer. And map it to an N-bit pure integer tensor : in, This is the scaling factor; The tensor partitioning module is further configured to: use the AutoTiler network tool to select the optimal tensor partitioning method to partition the convolutional tensors, so as to encapsulate the computation process of the trained depth estimation network on the ultra-low power edge computing chip into code data represented in C language. Also includes: Tensors in network computation are divided into smaller data blocks, called tiles, and moved between L2 and L1 memory. This problem is divided into two separate parts: 1) a set of optimized kernels that run specifically on L1 data tiles; 2) a tiling solver that defines the optimal size of the tiles and generates the code for related data transfer between L2 and L1, including double buffering of all tensors. Each layer in the CNN runs on a 3D input tensor representing the feature space and produces a new 3D activation tensor as the output. Convolutional layers consist of linear transformations that map the Kin input feature map to the Kout output feature map using Kin×Kout convolutional filters. In any convolutional layer, three distinct data spaces are identified, which are divided into tiles in one or more of the three dimensions. A subset of these spaces is explored using the AutoTiler tool to select the optimal tiling configuration and packaged into C code that efficiently moves data between L2 and L1 memory using a cluster DMA controller.

Images