Depth image estimation method, apparatus, electrical device, and storage medium

Abstract

This invention discloses a depth image estimation method, apparatus, electrical device, and storage medium. The depth image estimation method is applied to an embedded neural network processor. The method includes: acquiring a first image and a second image captured by a binocular camera; cropping the effective regions of the first and second images respectively to obtain a first effective image and a second effective image; using a pre-trained lightweight image correction network model to align and correct the rows of the first and second effective images to obtain a first corrected image and a second corrected image; and using a pre-trained lightweight depth estimation network model to estimate the depth of the first corrected image and the second corrected image to obtain a depth image. This method reduces computational complexity, computational load, and computation time, while ensuring the real-time performance of the depth estimation method.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a depth image estimation method, apparatus, electrical device, and storage medium. Background Technology

[0002] In related technologies, the traditional binocular depth estimation method follows this process: The left and right images acquired by the binocular camera are corrected; the corrected left and right images are then used to calculate a disparity map through block matching; and a depth image is calculated based on the disparity map. The estimation process of the traditional binocular depth estimation method runs on a CPU (Central Processing Unit).

[0003] Traditional depth estimation methods are computationally intensive due to the large amount of data required for image correction, disparity map calculation, and disparity map to depth map conversion. This is especially true when traditional depth estimation methods are run on the CPUs of embedded development boards at edge devices with limited computing power (such as robotic vacuum cleaners and smart door locks). The slow computation speed makes it difficult to guarantee the real-time performance of traditional depth estimation methods, and the CPU's efficiency is low when performing parallel image-related computations. Summary of the Invention

[0004] This invention aims to at least partially solve one of the technical problems in related technologies. To this end, the first objective of this invention is to propose a depth image estimation method that reduces computational complexity and computational load by cropping the effective regions of a first and second image. Furthermore, the pre-trained lightweight image correction network model and lightweight depth estimation network model offer fast computation speeds, reducing computation time. Additionally, the depth image estimation method runs on an embedded neural network processor, further improving computational speed and ensuring real-time performance.

[0005] A second objective of this invention is to provide a computer-readable storage medium.

[0006] The third objective of this invention is to provide an electrical device.

[0007] The fourth objective of this invention is to provide a depth image estimation device.

[0008] To achieve the above objectives, a depth image estimation method is proposed according to a first aspect of the present invention, applied to an embedded neural network processor. The method includes: acquiring a first image and a second image captured by a binocular camera; cropping the effective regions of the first image and the second image respectively to obtain a first effective image and a second effective image; using a pre-trained lightweight image correction network model to perform alignment correction on the rows of the first effective image and the rows of the second effective image to obtain a first corrected image and a second corrected image; and using a pre-trained lightweight depth estimation network model to estimate the depth of the first corrected image and the second corrected image to obtain a depth image.

[0009] According to the depth image estimation method of this invention, a first image and a second image acquired by a binocular camera are obtained. Effective regions of the first and second images are cropped to obtain a first effective image and a second effective image, respectively. A pre-trained lightweight image correction network model is used to correct the first and second effective images to obtain a first corrected image and a second corrected image. Then, a pre-trained lightweight depth estimation network model is used to estimate the depth of the first corrected image and the second corrected image to obtain a depth image. The depth image estimation method is applied to an embedded neural network processor. Therefore, by cropping the effective regions of the first and second images, the complexity of depth image calculation is reduced, and the computational load is decreased. The lightweight image correction network model and the lightweight depth estimation network model have fast computation speeds, reducing the computation time of the depth image. Furthermore, the depth image estimation method runs in an embedded neural network processor, which has a "data-driven parallel computing" architecture, further improving the computational speed and ensuring the real-time performance of the depth estimation method.

[0010] According to an embodiment of the present invention, before acquiring the first image and the second image captured by the binocular camera, the method further includes: performing network pruning on the pre-trained image correction network model and the depth estimation network model respectively to obtain a lightweight image correction network model and a lightweight depth estimation network model.

[0011] According to an embodiment of the present invention, after obtaining the lightweight image correction network model and the lightweight depth estimation network model, the method further includes: training the lightweight image correction network model using a first target sample set to obtain a pre-trained lightweight image correction network model, wherein the first target sample set includes multiple effective image groups composed of a first effective image and a second effective image, a first target correction image and a second target correction image, the first target correction image and the second target correction image being obtained through the pre-trained image correction network model; training the lightweight depth estimation network model using a second target sample set to obtain a pre-trained lightweight depth estimation network model, wherein the second target sample set includes multiple correction image groups composed of the first correction image and the second correction image, a target depth image, the target depth image being obtained by depth estimation based on the disparity map between the first image and the second image.

[0012] According to one embodiment of the present invention, training a lightweight image correction network model using a first target sample set includes: using multiple effective image groups as input, and a first target correction image and a second target correction image as image correction labels, training the parameters of the lightweight image correction network model until the error between the image correction result output by the lightweight image correction network model and the image correction label satisfies a first preset error condition.

[0013] According to one embodiment of the present invention, training a lightweight depth estimation network model using a second target sample set includes: training the parameters of the lightweight depth estimation network model with multiple sets of calibrated images as input and target depth images as depth image labels, until the error between the depth image estimation result output by the lightweight depth estimation network model and the depth image label satisfies a second preset error condition.

[0014] According to one embodiment of the present invention, after training the lightweight image correction network model and the lightweight depth estimation network model, the method further includes: performing model format conversion on the trained lightweight image correction network model and the lightweight depth estimation network model respectively, so that the converted lightweight image correction network model and the lightweight depth estimation network model are suitable for running on an embedded neural network processor.

[0015] According to one embodiment of the present invention, before performing model format conversion on the trained lightweight image correction network model and the lightweight depth estimation network model respectively, the method further includes: performing int8 quantization on the model parameters of the trained lightweight image correction network model and the lightweight depth estimation network model.

[0016] According to one embodiment of the present invention, the effective regions of a first image and a second image are cropped, including: determining the boundary coordinates of the effective region; cropping the region defined by the line connecting the boundary coordinates from the first image and the second image to obtain a first effective image and a second effective image, wherein the region defined by the line connecting the boundary coordinates is the effective region.

[0017] To achieve the above objectives, a computer-readable storage medium is provided according to a second aspect of the present invention, on which a depth image estimation program is stored, which, when processed by a processor, executes the depth image estimation method of any of the foregoing embodiments.

[0018] According to the computer-readable storage medium of the present invention, by executing the computer program of the above-described depth image estimation method, the computational complexity is reduced and the amount of computation is decreased by cropping the effective regions of the first image and the second image. Furthermore, the pre-trained lightweight image correction network model and lightweight depth estimation network model have fast computation speed, reducing computation time. In addition, the depth image estimation method runs in an embedded neural network processor, further improving the computation speed and ensuring the real-time performance of the depth estimation method.

[0019] To achieve the above objectives, an electrical device is provided according to a third aspect of the present invention, including a memory, a processor, and a depth image estimation program stored in the memory and executable on the processor, wherein the depth image estimation program implements the depth image estimation method of any of the foregoing embodiments.

[0020] According to the electrical device of the present invention, the processor executes the program of the above-mentioned depth image estimation method. By cropping the effective regions of the first image and the second image, the computational complexity is reduced and the amount of computation is decreased. Furthermore, the pre-trained lightweight image correction network model and lightweight depth estimation network model have fast computation speed, reducing computation time. In addition, the depth image estimation method runs in the embedded neural network processor, which further improves the computation speed and ensures the real-time performance of the depth estimation method.

[0021] To achieve the above objectives, a depth image estimation device is proposed according to a fourth aspect embodiment of the present invention, applied to an embedded neural network processor. The device includes: an acquisition module for acquiring a first image and a second image captured by a binocular camera; a cropping module for cropping the effective regions of the first image and the second image respectively to obtain a first effective image and a second effective image; a correction module for aligning and correcting the rows of the first effective image and the second effective image using a pre-trained lightweight image correction network model to obtain a first corrected image and a second corrected image; and an estimation module for estimating the depth of the first corrected image and the second corrected image using a pre-trained lightweight depth estimation network model to obtain a depth image.

[0022] According to an embodiment of the present invention, the depth image estimation apparatus acquires a first image and a second image captured by a binocular camera through an acquisition module. A cropping module then crops the effective regions of the first and second images to obtain a first effective image and a second effective image, respectively. A correction module then corrects the first and second effective images using a pre-trained lightweight image correction network model to obtain a first corrected image and a second corrected image. Finally, an estimation module uses a pre-trained lightweight depth estimation network model to estimate the depth of the first and second corrected images to obtain a depth image. Therefore, by cropping the effective regions of the first and second images, the complexity of depth image calculation is reduced, and the computational load is decreased. The lightweight image correction network model and the lightweight depth estimation network model have fast computation speeds, reducing the computation time for depth image estimation. Furthermore, the depth image estimation method runs in an embedded neural network processor, which has a "data-driven parallel computing" architecture, further improving the computational speed and ensuring the real-time performance of the depth estimation method.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] Figure 1 This is a schematic flowchart of a depth image estimation method according to the first embodiment of the present invention;

[0025] Figure 2 This is a schematic flowchart of a depth image estimation method according to a second embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram illustrating the training process of a lightweight image correction network model and a lightweight depth estimation network model according to an embodiment of the present invention.

[0027] Figure 4This is a schematic diagram of a depth image estimation process according to an embodiment of the present invention;

[0028] Figure 5 This is a system schematic diagram of an electrical device according to an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of a depth image estimation device according to an embodiment of the present invention. Detailed Implementation

[0030] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0031] The depth image estimation method, apparatus, electrical device, and storage medium of the present invention are described below with reference to the accompanying drawings.

[0032] The depth image estimation method of this invention is applied to an embedded neural network processor.

[0033] Because embedded neural network processors employ a "data-driven parallel computing" architecture, they require less time to process massive amounts of multimedia data such as videos and images. Therefore, the depth image estimation method in this embodiment, when applied to an embedded neural network processor, can improve the computational speed of depth image estimation.

[0034] Figure 1 This is a schematic flowchart of a depth image estimation method according to a first embodiment of the present invention. Figure 1 As shown, depth image estimation methods include:

[0035] S101, acquire the first and second images captured by the binocular camera.

[0036] Specifically, the binocular camera is an edge-mounted camera (such as a robot vacuum cleaner or smart door lock), and the first image and the second image are scene data collected by the left camera and the right camera, respectively.

[0037] S102, the effective regions of the first image and the second image are cropped respectively to obtain the first effective image and the second effective image.

[0038] Specifically, by extracting the effective regions of the first and second images, and then performing correction and depth estimation only on the effective regions during subsequent image correction and depth estimation processes, the complexity of depth image calculation is reduced, thereby reducing the amount of computation.

[0039] In some embodiments, the effective regions of the first image and the second image are cropped, including: determining the boundary coordinates of the effective region; cropping the region defined by the line connecting the boundary coordinates from the first image and the second image to obtain the first effective image and the second effective image, wherein the region defined by the line connecting the boundary coordinates is the effective region.

[0040] For example, when the first image and the second image are images captured by a robot vacuum cleaner, the effective area is the area with the best coverage in the image captured by the robot vacuum cleaner, such as the middle area in the image. Therefore, the boundary coordinates of the middle area can be determined, and then the area defined by the line connecting the boundary coordinates can be extracted to obtain the first effective image and the second effective image.

[0041] In one optional embodiment, when the first image and the second image are images captured by a smart door lock, the effective area is the area where the face is located. At this time, an image recognition network can be used to identify the area where the face is located and determine the left side of the boundary of the area. Then, the effective area is cropped from the first image and the second image.

[0042] S103, the rows of the first effective image and the rows of the second effective image are aligned and corrected using a pre-trained lightweight image correction network model to obtain the first corrected image and the second corrected image.

[0043] Because the left and right cameras are spherical, the images captured by the binocular cameras are convex, and objects in the images will be deformed. Therefore, it is necessary to first align and correct the rows of the first and second effective images to make the rows of the first and second effective images aligned, so that the subsequent depth images are more accurate.

[0044] In this embodiment, the image correction network model is a lightweight network model. Lightweight network models have fast computation speed, which reduces the time required for image correction. Furthermore, the first and second effective images are images of the effective regions, which further reduces the time required for image correction.

[0045] S104. Use a pre-trained lightweight depth estimation network model to estimate the depth of the first and second corrected images to obtain a depth image.

[0046] Specifically, the lightweight depth estimation network model has a fast computation speed, reducing the time required for depth image estimation. Furthermore, the first and second corrected images are corrected images of the effective region, requiring less computation and further reducing the time consumption of depth image estimation.

[0047] In the above embodiments, by cropping the effective regions of the first and second images, the computational complexity and computational load are reduced. The computation speed of the lightweight image correction network model and the lightweight depth estimation network model is also reduced, thus reducing computation time. Furthermore, the depth image estimation method runs in an embedded neural network processor, further improving the computational speed and ensuring the real-time performance of the depth estimation method.

[0048] In some embodiments, before acquiring the first and second images captured by the binocular camera, the method further includes: performing network pruning on the pre-trained image correction network model and depth estimation network model respectively to obtain a lightweight image correction network model and a lightweight depth estimation network model.

[0049] Specifically, by cropping the pre-trained image correction network model and depth estimation network model, the number of parameters in the image correction network model and depth estimation network model is reduced, thereby speeding up the computation.

[0050] It should be noted that the pre-trained image correction network model and depth estimation network model include, but are not limited to, MobileNet (a lightweight deep neural network) or ShuffleNet (a lightweight deep neural network), but there are no specific restrictions here.

[0051] In one specific embodiment, network pruning is performed on the pre-trained image correction network model and depth estimation network model, including pruning the network layers and convolutional channels of the pre-trained image correction network model and depth estimation network model, respectively.

[0052] Specifically, redundant network layers in the image correction network model and depth estimation network model are identified by the output values ​​of each network layer module, and these redundant network layers are then pruned. By performing sparse constraint training on the image correction network model and depth estimation network model, the pruning ratio of the convolutional channels of each network layer is determined, and the convolutional channels are pruned according to the pruning ratio.

[0053] In some embodiments, such as Figure 2 As shown, after obtaining the lightweight image correction network model and the lightweight depth estimation network model, the method further includes:

[0054] S201, the lightweight image correction network model is trained using the first target sample set to obtain a pre-trained lightweight image correction network model. The first target sample set includes multiple groups of effective images consisting of a first effective image and a second effective image, a first target correction image, and a second target correction image. The first target correction image and the second target correction image are obtained through the pre-trained image correction network model.

[0055] In some embodiments, training a lightweight image correction network model using a first target sample set includes: training the parameters of the lightweight image correction network model with multiple effective image groups as input, and the first target correction image and the second target correction image as image correction labels, until the error between the image correction result output by the lightweight image correction network model and the image correction label meets a first preset error condition.

[0056] Specifically, because the cropped image correction network model and depth estimation network model may produce large errors, it is necessary to train the lightweight image correction network model and the lightweight depth estimation network model. Using the first and second target correction images obtained from the pre-trained image correction network model as labels, and multiple effective image groups as inputs, the parameters of the lightweight image correction network model are adjusted to reduce the error between the image correction result output by the lightweight image correction network model and the image correction label, until a first preset error condition is met.

[0057] S202, the lightweight depth estimation network model is trained using the second target sample set to obtain a pre-trained lightweight depth estimation network model. The second target sample set includes multiple sets of corrected images consisting of a first corrected image and a second corrected image, and a target depth image. The target depth image is obtained by depth estimation based on the disparity map between the first image and the second image.

[0058] In some embodiments, training a lightweight depth estimation network model using a second target sample set includes: training the parameters of the lightweight depth estimation network model with multiple sets of calibrated images as input and target depth images as depth image labels, until the error between the depth image estimation result output by the lightweight depth estimation network model and the depth image label meets a second preset error condition.

[0059] Specifically, using the target depth image obtained by traditional depth image estimation methods as a label, and multiple sets of calibration images as input, the parameters of the lightweight depth estimation network model are adjusted to reduce the error between the depth image estimation result output by the lightweight depth estimation network model and the depth image label, until a second preset error condition is met. Using the target depth image obtained by traditional depth image estimation methods as a label can further improve the accuracy of the lightweight depth estimation network model, making it more accurate.

[0060] In this embodiment, by training the lightweight image correction network model and the lightweight depth estimation network model, the accuracy of the lightweight image correction network model and the lightweight depth estimation network model is improved, resulting in a more accurate depth image.

[0061] In some embodiments, after training the lightweight image correction network model and the lightweight depth estimation network model, the method further includes: performing model format conversion on the trained lightweight image correction network model and the lightweight depth estimation network model respectively, so that the converted lightweight image correction network model and the lightweight depth estimation network model are suitable for running on an embedded neural network processor.

[0062] Specifically, because the trained model is not compatible with embedded neural network processors, the trained lightweight image correction network model and lightweight depth estimation network model are first converted into ONNX (Open Neural Network Exchange) models, as ONNX models support multiple inference frameworks. Then, the converted lightweight image correction network model and lightweight depth estimation network model are converted into inference frameworks supported by the embedded neural network processor, such as NCNN (an inference framework) and TNN (an inference framework), thus enabling the application of the trained model to the embedded neural network processor.

[0063] In some embodiments, before performing model format conversion on the trained lightweight image correction network model and the lightweight depth estimation network model respectively, the method further includes: performing int8 quantization on the model parameters of the trained lightweight image correction network model and the lightweight depth estimation network model.

[0064] In other words, in order to further accelerate the computation speed of the lightweight image correction network model and the lightweight depth estimation network model, the model parameters of the trained lightweight image correction network model and the lightweight depth estimation network model were adjusted from float32 to int8, thereby further reducing the number of model parameters.

[0065] In one optional implementation, if the output error of the lightweight image correction network model and the lightweight depth estimation network model is large after int8 quantization of the model parameters, the quantized lightweight image correction network model and the lightweight depth estimation network model can be retrained to adjust the model parameters.

[0066] In the above embodiments, by performing int8 quantization on the model parameters of the trained lightweight image correction network model and lightweight depth estimation network model, the number of model parameters is further reduced, thereby further accelerating the model's computation speed.

[0067] In one alternative implementation, the number of parameters of the lightweight image correction network model and the lightweight depth estimation network model can be adjusted according to the computational size of different embedded neural network processors, so as to adjust the computational speed and accuracy of the lightweight image correction network model and the lightweight depth estimation network model.

[0068] The technical solution of this application will be further described in detail below with reference to specific implementation methods:

[0069] like Figure 3 and Figure 4 As shown, the depth image estimation method can include two parts: training a lightweight image correction network model and a lightweight depth estimation network model, and then performing depth image estimation.

[0070] like Figure 3 As shown, training the lightweight image correction network model and the lightweight depth estimation network model can include:

[0071] S301, the network layers and convolutional channels of the pre-trained image correction network model and depth estimation network model are pruned respectively to obtain a lightweight image correction network model and a lightweight depth estimation network model.

[0072] S302, taking multiple effective image groups consisting of the first effective image and the second effective image as input, and the first target correction image and the second target correction image obtained by the pre-trained image correction network model as image correction labels, the parameters of the lightweight image correction network model are trained until the error between the image correction result output by the lightweight image correction network model and the image correction label is less than the first error.

[0073] S303, using multiple sets of corrected images consisting of the first and second corrected images as input, and the target depth image obtained according to the traditional depth estimation method as the depth image label, the parameters of the lightweight depth estimation network model are trained until the error between the depth image estimation result output by the lightweight depth estimation network model and the depth image label is less than the second error.

[0074] S304 performs int8 quantization on the model parameters of the trained lightweight image correction network model and lightweight depth estimation network model.

[0075] S305 converts the trained lightweight image correction network model and lightweight depth estimation network model into an ONNX model, and then converts the converted lightweight image correction network model and lightweight depth estimation network model into an inference framework supported by an embedded neural network processor, thus obtaining a pre-trained lightweight image correction network model and lightweight depth estimation network model.

[0076] The training method for the lightweight image correction network model and lightweight depth estimation network model in this embodiment reduces computational complexity and computational load by cropping the effective regions of the first and second images. Furthermore, by performing network pruning and int8 quantization on the pre-trained image correction network model and depth estimation network model, the number of parameters in the model is reduced, thus accelerating their computational speed. Finally, by converting the lightweight image correction network model and lightweight depth estimation network model into an inference framework supported by an embedded neural network processor, the model can run on the embedded neural network processor, further accelerating their computational speed and ensuring the real-time performance of the depth image estimation algorithm.

[0077] like Figure 4 As shown, depth image estimation may include:

[0078] S401, acquire the first image and the second image captured by the binocular camera, and crop the effective area of ​​the first image and the second image to obtain the first effective image and the second effective image.

[0079] S402, the first valid image and the second valid image are input into a pre-trained lightweight image correction network model for alignment correction, so as to align the rows of the first valid image and the rows of the second valid image to obtain the first corrected image and the second corrected image.

[0080] S403, input the first and second corrected images into the pre-trained lightweight depth estimation network model to perform depth estimation and obtain the depth image.

[0081] The depth image estimation method in this embodiment uses the aforementioned trained lightweight image correction network model and lightweight depth estimation network model, which can quickly obtain depth images and ensure the real-time performance of the depth image estimation algorithm.

[0082] In summary, the depth image estimation method according to embodiments of the present invention acquires a first image and a second image captured by a binocular camera, respectively extracts effective regions from the first and second images to obtain a first effective image and a second effective image, and uses a pre-trained lightweight image correction network model to correct the first and second effective images to obtain a first corrected image and a second corrected image. Then, it uses a pre-trained lightweight depth estimation network model to estimate the depth of the first corrected image and the second corrected image to obtain a depth image. The depth image estimation method is applied to an embedded neural network processor. Therefore, by extracting effective regions from the first and second images, the complexity of depth image computation is reduced, and the computational load is decreased. The lightweight image correction network model and the lightweight depth estimation network model have fast computation speeds, reducing the computation time for depth image estimation. Furthermore, the depth image estimation method runs in an embedded neural network processor, which has a "data-driven parallel computing" architecture, further improving computational speed and ensuring the real-time performance of the depth estimation method.

[0083] Corresponding to the above embodiments, embodiments of the present invention also propose a computer-readable storage medium storing a depth image estimation program thereon, which, when processed by a processor, executes the depth image estimation method of any of the foregoing embodiments.

[0084] According to the computer-readable storage medium of the present invention, by executing the computer program of the above-described depth image estimation method, the computational complexity is reduced and the amount of computation is decreased by cropping the effective regions of the first image and the second image. Furthermore, the pre-trained lightweight image correction network model and lightweight depth estimation network model have fast computation speed, reducing computation time. In addition, the depth image estimation method runs in an embedded neural network processor, further improving the computation speed and ensuring the real-time performance of the depth estimation method.

[0085] Corresponding to the above embodiments, the present invention also proposes an electrical device.

[0086] like Figure 5 As shown, the electrical device 100 includes a memory 110, a processor 120, and a depth image estimation program stored in the memory 110 and run on the processor 120. When the processor 120 performs the depth image estimation program, it implements the depth image estimation method of any of the foregoing embodiments.

[0087] According to the electrical device of the present invention, the processor executes the program of the above-mentioned depth image estimation method. By cropping the effective regions of the first image and the second image, the computational complexity is reduced and the amount of computation is decreased. Furthermore, the pre-trained lightweight image correction network model and lightweight depth estimation network model have fast computation speed, reducing computation time. In addition, the depth image estimation method runs in the embedded neural network processor, which further improves the computation speed and ensures the real-time performance of the depth estimation method.

[0088] Corresponding to the above embodiments, the present invention also proposes a depth image estimation device applied to an embedded neural network processor.

[0089] like Figure 6 As shown, the device includes: an acquisition module 10, an interception module 20, a correction module, and an estimation module 40.

[0090] The acquisition module 10 is used to acquire the first image and the second image captured by the binocular camera; the cropping module 20 is used to crop the effective regions of the first image and the second image respectively to obtain the first effective image and the second effective image; the correction module 30 is used to use a pre-trained lightweight image correction network model to perform alignment correction on the rows of the first effective image and the rows of the second effective image to obtain the first corrected image and the second corrected image; the estimation module 40 is used to use a pre-trained lightweight depth estimation network model to perform depth estimation on the first corrected image and the second corrected image to obtain the depth image.

[0091] In some embodiments, the apparatus further includes a cropping module, which is configured to: before acquiring the first image and the second image captured by the binocular camera, respectively crop the pre-trained image correction network model and the depth estimation network model to obtain a lightweight image correction network model and a lightweight depth estimation network model.

[0092] In some embodiments, the apparatus further includes a training module, which is configured to: after obtaining a lightweight image correction network model and a lightweight depth estimation network model, train the lightweight image correction network model using a first target sample set to obtain a pre-trained lightweight image correction network model, wherein the first target sample set includes multiple groups of valid images consisting of a first valid image and a second valid image, a first target corrected image, and a second target corrected image, the first target corrected image and the second target corrected image being obtained through the pre-trained image correction network model; and train the lightweight depth estimation network model using a second target sample set to obtain a pre-trained lightweight depth estimation network model, wherein the second target sample set includes multiple groups of corrected images consisting of a first corrected image and a second corrected image, a target depth image, the target depth image being obtained by depth estimation based on the disparity map between the first image and the second image.

[0093] In some embodiments, the training module is further configured to: train the parameters of the lightweight image correction network model with multiple effective image groups as input and the first target correction image and the second target correction image as image correction labels, until the error between the image correction result output by the lightweight image correction network model and the image correction label meets the first preset error condition.

[0094] In some embodiments, the training module is further configured to: train the parameters of the lightweight depth estimation network model with multiple sets of calibrated images as input and the target depth image as the depth image label, until the error between the depth image estimation result output by the lightweight depth estimation network model and the depth image label satisfies the second preset error condition.

[0095] In some embodiments, the cropping module is further configured to: crop the network layers and convolutional channels of the pre-trained image correction network model and depth estimation network model, respectively.

[0096] In some embodiments, the apparatus further includes a conversion module, which is configured to perform model format conversion on the trained lightweight image correction network model and lightweight depth estimation network model respectively after training, so that the converted lightweight image correction network model and lightweight depth estimation network model are suitable for running on an embedded neural network processor.

[0097] In some embodiments, the apparatus further includes a quantization module, which performs int8 quantization on the model parameters of the trained lightweight image correction network model and the lightweight depth estimation network model before performing model format conversion on the trained lightweight image correction network model and the lightweight depth estimation network model, respectively.

[0098] In some embodiments, the cropping module 20 is further configured to: determine the boundary coordinates of the effective region; crop the region defined by the line connecting the boundary coordinates from the first image and the second image to obtain the first effective image and the second effective image, wherein the region defined by the line connecting the boundary coordinates is the effective region.

[0099] It should be noted that for details regarding the undisclosed depth image estimation device, please refer to the details disclosed regarding depth image estimation methods; these details will not be repeated here.

[0100] According to an embodiment of the present invention, the depth image estimation apparatus acquires a first image and a second image captured by a binocular camera through an acquisition module. A cropping module then crops the effective regions of the first and second images to obtain a first effective image and a second effective image, respectively. A correction module then corrects the first and second effective images using a pre-trained lightweight image correction network model to obtain a first corrected image and a second corrected image. Finally, an estimation module uses a pre-trained lightweight depth estimation network model to estimate the depth of the first and second corrected images to obtain a depth image. Therefore, by cropping the effective regions of the first and second images, the complexity of depth image calculation is reduced, and the computational load is decreased. The lightweight image correction network model and the lightweight depth estimation network model have fast computation speeds, reducing the computation time for depth image estimation. Furthermore, the depth image estimation method runs in an embedded neural network processor, which has a "data-driven parallel computing" architecture, further improving the computational speed and ensuring the real-time performance of the depth estimation method.

[0101] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0102] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0103] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0104] Furthermore, the terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying relative importance, or implicitly specifying the number of technical features indicated in this embodiment. Therefore, features defined with terms such as "first" and "second" in the embodiments of this invention can explicitly or implicitly indicate that the embodiment includes at least one of those features. In the description of this invention, the word "multiple" means at least two or more, such as two, three, four, etc., unless otherwise explicitly specified in the embodiments.

[0105] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A depth image estimation method, characterized in that, Applied to embedded neural network processors, the method includes: Acquire the first and second images captured by the binocular camera; The effective regions of the first image and the second image are cropped respectively to obtain the first effective image and the second effective image; Alignment correction is performed on rows of the first effective image and rows of the second effective image using a target lightweight image correction network model to obtain a first corrected image and a second corrected image. The target lightweight image correction network model is trained using a group of effective images, the first target corrected image, and the second target corrected image. The first target corrected image and the second target corrected image are obtained through a pre-trained image correction network model. The number of parameters in the image correction network model is greater than the number of parameters in the target lightweight image correction network model. A depth image is obtained by using a target lightweight depth estimation network model to estimate the depth of the first and second corrected images. The method further includes: The pre-trained lightweight image correction network model and the pre-trained lightweight depth estimation network model are respectively converted into ONNX models to obtain the first lightweight image correction network model and the first lightweight depth estimation network model. The first lightweight image correction network model and the first lightweight depth estimation network model are respectively converted into a model format that matches the embedded neural network processor to obtain the target lightweight image correction network model and the target lightweight depth estimation network model.

2. The method according to claim 1, characterized in that, Before acquiring the first and second images captured by the binocular camera, the method further includes: By pruning the pre-trained image correction network model and depth estimation network model, lightweight image correction network model and lightweight depth estimation network model are obtained.

3. The method according to claim 2, characterized in that, After obtaining the lightweight image correction network model and the lightweight depth estimation network model, the method further includes: The lightweight image correction network model is trained using a first target sample set to obtain a pre-trained lightweight image correction network model. The first target sample set includes multiple groups of valid images consisting of the first valid image and the second valid image, the first target correction image, and the second target correction image. The lightweight depth estimation network model is trained using a second target sample set to obtain a pre-trained lightweight depth estimation network model. The second target sample set includes multiple sets of corrected images consisting of the first corrected image and the second corrected image, and a target depth image. The target depth image is obtained by depth estimation based on the disparity map between the first image and the second image.

4. The method according to claim 3, characterized in that, Training the lightweight image correction network model using the first target sample set includes: Using the multiple valid image groups as input and the first target correction image and the second target correction image as image correction labels, the parameters of the lightweight image correction network model are trained until the error between the image correction result output by the lightweight image correction network model and the image correction label meets the first preset error condition.

5. The method according to claim 3, characterized in that, The lightweight depth estimation network model is trained using a second target sample set, including: Using the multiple sets of corrected images as input and the target depth image as the depth image label, the parameters of the lightweight depth estimation network model are trained until the error between the depth image estimation result output by the lightweight depth estimation network model and the depth image label meets the second preset error condition.

6. The method according to claim 1, characterized in that, Before converting the pre-trained lightweight image correction network model and the pre-trained lightweight depth estimation network model into ONNX models respectively, the method further includes: The model parameters of the pre-trained lightweight image correction network model and lightweight depth estimation network model are quantized using int8.

7. The method according to any one of claims 1-6, characterized in that, The effective regions of the first image and the second image are respectively cropped, including: Determine the boundary coordinates of the effective region; The region defined by the line connecting the boundary coordinates is extracted from the first image and the second image to obtain the first valid image and the second valid image, wherein the region defined by the line connecting the boundary coordinates is the valid region.

8. A computer-readable storage medium, characterized in that, It stores a depth image estimation program, which, when executed by a processor, implements the depth image estimation method according to any one of claims 1-7.

9. An electrical appliance, characterized in that, include: The device includes a memory, a processor, and a depth image estimation program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the depth image estimation method according to any one of claims 1-7.

10. A depth image estimation device, characterized in that, The device, used in an embedded neural network processor, includes: The acquisition module is used to acquire the first and second images captured by the binocular camera; The cropping module is used to crop the effective regions of the first image and the second image respectively to obtain the first effective image and the second effective image; The correction module is used to perform alignment correction on rows of the first effective image and rows of the second effective image by the target lightweight image correction network model to obtain a first corrected image and a second corrected image. The target lightweight image correction network model is trained using the effective image group, the first target corrected image, and the second target corrected image. The first target corrected image and the second target corrected image are obtained through a pre-trained image correction network model. The number of parameters of the image correction network model is greater than the number of parameters of the target lightweight image correction network model. The estimation module is used by the target lightweight depth estimation network model to perform depth estimation on the first corrected image and the second corrected image to obtain a depth image; The device also includes: The first conversion module is used to convert the pre-trained lightweight image correction network model and the pre-trained lightweight depth estimation network model into ONNX models respectively, so as to obtain the first lightweight image correction network model and the first lightweight depth estimation network model. The second conversion module is used to convert the first lightweight image correction network model and the first lightweight depth estimation network model into a model format that matches the embedded neural network processor, respectively, to obtain the target lightweight image correction network model and the target lightweight depth estimation network model.

Images