Image Depth Prediction Method, Device, Readable Storage Medium and Electronic Device

Abstract

The present disclosure relates to an image depth prediction method, apparatus, readable storage medium and electronic device. The method includes: acquiring a two-dimensional image and point cloud data of the same scene, projecting the point cloud data onto the two-dimensional image to generate a sparse depth image, estimating the depth of pixel points lacking depth information in the sparse depth image based on the perspective geometry principle to obtain an initial dense depth image, optimizing the parameters of the initial dense depth image through the segmentation results of a neural network and a semantic segmentation model to obtain a dense depth image; fusing the point cloud data and the two-dimensional image to obtain a sparse depth image, estimating the depth of the sparse depth image based on the perspective geometry principle, simply and efficiently eliminating abnormal estimated values, and finally optimizing the depth estimation result through a neural network and a semantic segmentation model to ensure that the depth estimation error of various category information in the image scene is globally minimized, and finally obtaining a relatively accurate global depth prediction of the image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular, to an image depth prediction method, apparatus, readable storage medium, and electronic device. Background Art

[0002] In the related art, the depth of an image is predicted through a neural network, that is, the image to be predicted is predicted based on a depth branch network and a confidence branch network in the neural network, so as to determine the depth of the image to be predicted. However, there are many drawbacks to this prediction method. For example, a multi-layer and complex cascade network model is constructed, and the front-to-back dependence of the multi-stage network in the feature extraction process may cause error propagation, resulting in limited model accuracy. At the same time, the large number of parameter models are difficult to converge, and the prediction efficiency and rate are limited. Summary of the Invention

[0003] The purpose of the present disclosure is to provide an image depth prediction method, apparatus, electronic device, and computer-readable storage medium, so as to solve the problem of limited prediction efficiency and rate caused by predicting the image depth based on the depth branch network and the confidence branch network in the existing neural network.

[0004] According to the first aspect of the embodiments of the present disclosure, an image depth prediction method is provided, including: obtaining a two-dimensional image and point cloud data of the same scene; projecting the point cloud data onto the two-dimensional image to generate a sparse depth image; estimating the depth of pixel points lacking depth information in the sparse depth image based on the perspective geometry principle to obtain an initial dense depth image; optimizing the parameters of the initial dense depth image through the segmentation result of a neural network and a semantic segmentation model to obtain a dense depth image; the segmentation result is the pixel point label classification of the two-dimensional image.

[0005] Optionally, the two-dimensional image is obtained by a camera, and the point cloud data is obtained by a lidar; the step of projecting the point cloud data onto the two-dimensional image to generate a sparse depth image includes:

[0006] Projecting the point cloud data onto the two-dimensional image based on the external parameters jointly calibrated by the camera and the lidar to generate the sparse depth image; the jointly calibrated external parameters represent the data conversion relationship between the pixel points on the same target in the two-dimensional image and the point cloud data.

[0007] Optionally, the neural network is a feedforward neural network based on the backpropagation algorithm; the feedforward neural network includes an input layer, a hidden layer, and an output layer; the input layer, the hidden layer, and the output layer are connected through network weights and biases.

[0008] Optionally, the step of optimizing the parameters of the initial dense depth image based on the segmentation result of the neural network and the semantic segmentation model to obtain a dense depth image includes: using the pixel point coordinates, pixel point class labels, and pixel point depth values of the initial dense depth image as the input of the feed-forward neural network; the pixel point depth value represents the physical distance from the pixel point in the initial dense depth image to the lidar; the feed-forward neural network corrects the pixel point depth value according to the segmentation result to obtain the dense depth image.

[0009] Optionally, it further includes: constructing a loss function of the dense depth image according to the pixel point coordinates, image vanishing point, pixel point class labels, and pixel point depth values of the initial dense depth image; the image vanishing point is the perspective geometric image vanishing point of the initial dense depth image; or the feed-forward neural network constructs a prediction of the loss function by setting a simulated non-linear continuous function with different numbers of network layers.

[0010] Optionally, the semantic segmentation model is pre-trained; the training process of the semantic segmentation model includes: collecting sample images of the measured scene related to the scene; performing class annotation on the sample images; using the annotated sample images as input images and inputting them into an efficient residual decomposition network for training to obtain the semantic segmentation model.

[0011] Optionally, the semantic segmentation model includes a first downsampling layer, a second downsampling layer, a third downsampling layer, a first upsampling layer, a second upsampling layer, and a third upsampling layer; the first downsampling layer includes a convolutional layer, a batch normalization layer, and an activation layer, and is configured to perform feature extraction on the input image to obtain a first feature map; the second downsampling layer includes a convolutional layer, a batch normalization layer, an activation layer, and a five-level cascaded efficient residual decomposition module, and is configured to perform feature extraction and feature fusion on the first feature map to obtain a second feature map; the third downsampling layer includes a convolutional layer, a batch normalization layer, an activation layer, and an eight-level cascaded efficient residual decomposition module, and is configured to perform feature extraction and feature fusion on the second feature map to obtain a third feature map; the first upsampling layer includes a transposed convolutional layer and a batch normalization layer, and is configured to perform an upsampling operation on the third feature map to obtain a fourth feature map for feature recovery; and further includes two-level cascaded efficient residual decomposition modules, and is configured to perform feature fusion on the fourth feature map to obtain a fifth feature map; the second upsampling layer includes a transposed convolutional layer and a batch normalization layer, and is configured to perform an upsampling operation on the fifth feature map to obtain a sixth feature map for feature recovery; and further includes two-level cascaded efficient residual decomposition modules, and is configured to perform feature fusion on the sixth feature map to obtain a seventh feature map; the third upsampling layer includes a transposed convolutional layer and a batch normalization layer, and is configured to perform an upsampling operation on the seventh feature map to obtain an output image; the size of the output image is the same as the size of the input image.

[0012] Optionally, the step of using the labeled sample image as the input image and inputting it into an efficient residual decomposition network for training to obtain the semantic segmentation model includes: obtaining the difference between the input image and the output image; adjusting the parameters of the first downsampling layer, the second downsampling layer, the third downsampling layer, the first upsampling layer, the second upsampling layer, and the third upsampling layer according to the difference to obtain the semantic segmentation model.

[0013] According to a second aspect of the embodiments of the present disclosure, there is provided an image depth prediction device, including: an acquisition module, configured to acquire a two-dimensional image and point cloud data of the same scene; a projection module, configured to project the point cloud data onto the two-dimensional image to generate a sparse depth image; an estimation module, configured to perform depth estimation on the pixel points lacking depth information in the sparse depth image based on the perspective geometry principle to obtain an initial dense depth image; a processing module, configured to optimize the parameters of the initial dense depth image through the segmentation result of a neural network and a semantic segmentation model to obtain a dense depth image; the segmentation result is the pixel point label classification of the two-dimensional image.

[0014] According to a third aspect of the embodiments of the present disclosure, there is provided a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the foregoing image depth prediction method are implemented.

[0015] According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic device, including: a memory on which a computer program is stored; a processor configured to execute the computer program in the memory to implement the steps of the foregoing image depth prediction method.

[0016] The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: By fusing point cloud data and two-dimensional images, a sparse depth image is obtained. Based on the perspective geometry principle, depth estimation is performed on the sparse depth image, and abnormal estimated values are simply and efficiently eliminated. Finally, the results of depth estimation are optimized through a neural network and a semantic segmentation model to ensure that the depth estimation errors of various category information in the image scene are globally minimized, and relatively accurate global depth prediction of the image is finally obtained. It avoids constructing a multi-layer and complex cascade network model, avoids the front-to-back dependence of multi-level networks in the feature extraction process, which may cause error propagation resulting in limited model accuracy, and avoids the convergence difficulty of models with a large number of parameters. The data fusion method can more efficiently and real-time obtain sparse and accurate prior pixel depth values.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings are used to provide a further understanding of the present disclosure, and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the drawings:

[0019] Figure 1 is a schematic structural diagram of a computer system shown in an exemplary embodiment of the present disclosure.

[0020] Figure 2 is a flowchart of an image depth prediction method shown in an exemplary embodiment of the present disclosure.

[0021] Figure 3 is a flowchart of another image depth prediction method shown in an exemplary embodiment of the present disclosure.

[0022] Figure 4 is a block diagram of an image depth prediction device shown in an exemplary embodiment of the present disclosure.

[0023] Figure 5 is a block diagram of an electronic device shown in an exemplary embodiment of the present disclosure.

[0024] Figure 6 It is a block diagram of another electronic device shown in an exemplary embodiment of the present disclosure.

[0025] Description of Reference Numerals

[0026] 120 - Terminal; 140 - Server; 20 - Image Depth Prediction Device; 201 - Acquisition Module; 203 - Projection Module; 205 - Estimation Module; 207 - Processing Module; 400 - Electronic Device; 401 - Processor; 402 - Memory; 403 - Multimedia Component; 404 - I / O Interface; 405 - Communication Component; 500 - Electronic Device; 522 - Processor; 532 - Memory; 526 - Power Supply Component; 550 - Communication Component; 558 - I / O Interface. Detailed Embodiments

[0027] The following describes the detailed embodiments of the present disclosure with reference to the accompanying drawings. It should be understood that the detailed embodiments described herein are only for explaining and illustrating the present disclosure, and are not used to limit the present disclosure.

[0028] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0029] Figure 1 It shows a schematic structural diagram of a computer system provided by an exemplary embodiment of the present disclosure. The computer system includes a terminal 120 and a server 140.

[0030] The terminal 120 and the server 140 are interconnected through a wired or wireless network.

[0031] The terminal 120 may include at least one of a smart phone, a laptop computer, a desktop computer, a tablet computer, a smart speaker, and a smart robot.

[0032] The terminal 120 includes a display; the display is used to display image depth prediction Result .

[0033] The terminal 120 includes a first memory and a first processor. A first program is stored in the first memory; the first program is called and executed by the first processor to implement the image depth prediction method provided by the present disclosure. The first memory may include, but is not limited to, the following: Random Access Memory (RAM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). The first processor may be composed of one or more integrated circuit chips. Optionally, the first processor may be a general-purpose processor, such as a Central Processing Unit (CPU) or a Network Processor (NP).

[0034] The server 140 includes a second memory and a second processor. A second program is stored in the second memory, and the second program is called by the second processor to implement the image depth prediction method provided by the present disclosure. Optionally, the second memory may include, but is not limited to, the following: RAM, ROM, PROM, EPROM, EEPROM. Optionally, the second processor may be a general-purpose processor, such as a CPU or an NP.

[0035] The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected through wired or wireless communication methods, and the present disclosure does not limit this.

[0036] Please refer to Figure 2 , Figure 2 which is a flowchart of an image depth prediction method shown in an exemplary embodiment of the present disclosure. This method is executed by a computer device, for example, by the terminal or the server in the computer system shown in Figure 1 .Figure 2 The image depth prediction method shown includes the following steps:

[0037] In step S101, a two-dimensional image and point cloud data of the same scene are obtained.

[0038] The same scene refers to the two-dimensional image and point cloud data of the same scene image. The two-dimensional image and point cloud data contain the same target objects and the same environment. The two-dimensional image and point cloud data are two presentation forms of an image. In one implementation, the two-dimensional image is obtained by a camera, and the point cloud data is obtained by a lidar.

[0039] In step S102, the point cloud data is projected onto the two-dimensional image to generate a sparse depth image.

[0040] Exemplarily, the point cloud data can be projected onto the two-dimensional image through a data fusion module. According to the data conversion relationship between the point cloud data and the two-dimensional image, the pixel points of the point cloud data and the pixel points of the two-dimensional image at the same position in the scene image are corresponded to each other to fuse the two-dimensional image and the point cloud data and generate a sparse depth image. Compared with the two-dimensional image, the number of pixel points after the point cloud data is projected onto the two-dimensional image is small and relatively sparse. Therefore, it is called a sparse depth image.

[0041] In step S103, based on the perspective geometry principle, depth estimation is performed on the pixel points lacking depth information in the sparse depth image to obtain an initial dense depth image.

[0042] The number of pixel points after the point cloud data is projected onto the two-dimensional image is small. Therefore, only a small number of pixel points in the scene image have depth information. Depth estimation can be performed on the pixel points lacking depth information in the sparse depth image based on the perspective geometry principle to obtain an initial dense depth image. Among them, the depth information includes the pixel point depth value, and the pixel point depth value represents the physical distance from the pixel point in the scene image to the electronic device for acquiring the image, such as an electronic device for acquiring images like a camera or a lidar.

[0043] In step S104, the parameters of the initial dense depth image are optimized through the segmentation result of a neural network and a semantic segmentation model to obtain a dense depth image.

[0044] Exemplarily, the neural network can be a feedforward neural network based on the backpropagation algorithm. The semantic segmentation model is pre-trained based on the sample images of the measured scene. The measured scene is a scene related to the aforementioned scene. The segmentation result is the pixel point label classification of a two-dimensional image, such as the label classification of each target object pixel point in the two-dimensional image. Then, based on the backpropagation algorithm and the segmentation result, the parameters of the initially obtained dense depth image are optimized, that is, the depth value of each pixel point in the initially obtained dense depth image is optimized to obtain a dense depth image. For example, some pixel points with obviously incorrect depth values in the initially obtained dense depth image are corrected. Finally, the optimized and relatively accurate dense depth image is used to calibrate the estimation algorithm coefficients and the scene segmentation weights.

[0045] Please refer to Figure 3 , Figure 3 which is a flowchart of another image depth prediction method shown in an exemplary embodiment of the present disclosure. This method is executed by a computer device, for example, by Figure 1 the terminal or server in the computer system shown. It should be noted that Figure 3 the image depth prediction method shown is consistent with the implementation content of Figure 2 the image depth prediction method shown. For the parts not mentioned in Figure 3 , reference can be made to the description in Figure 2 and will not be elaborated here. Figure 3 The image depth prediction method shown includes the following steps:

[0046] In step S101, a two-dimensional image and point cloud data of the same scene are obtained.

[0047] The same scene refers to the two-dimensional image and point cloud data of the same scene image. The two-dimensional image and point cloud data contain the same target objects and the same environment. The two-dimensional image and the point cloud data are two presentation forms of an image. In one implementation, the two-dimensional image is obtained by a camera, and the point cloud data is obtained by a lidar.

[0048] In step S102, the point cloud data is projected onto the two-dimensional image to generate a sparse depth image.

[0049] Exemplarily, through the data fusion module, based on the extrinsic parameters jointly calibrated by the camera and lidar, the point cloud data synchronized with the two-dimensional image can be projected onto the two-dimensional image in real time. According to the data conversion relationship between the point cloud data and the two-dimensional image, the pixel points of the point cloud data and the pixel points of the two-dimensional image at the same position in the scene image are corresponded to each other, so as to fuse the two-dimensional image and the point cloud data to generate a sparse depth image. Compared with the two-dimensional image, the number of pixel points after the point cloud data is projected onto the two-dimensional image is small and sparse, so it is called a sparse depth image. The data conversion relationship is obtained based on the extrinsic parameters jointly calibrated by the camera and lidar.

[0050] For a two-dimensional image, the pixel set of the target object in the image is dense. After the point cloud data is projected onto the two-dimensional image, some pixel points of the target object will obtain depth values, and some other pixel points have no depth values. The pixel points with depth values can be recorded as pixel points containing prior depth, and the pixel points without depth values can be recorded as pixel points without valid prior depth.

[0051] Compared with the pixel depth estimation accuracy and efficiency based on a multi-level neural network by extracting depth feature maps and confidence feature maps in the related art, the data fusion method can obtain sparse and accurate prior pixel depth values more efficiently and in real time.

[0052] In step S103, based on the perspective geometry principle, depth estimation is performed on the pixel points lacking depth information in the sparse depth image to obtain an initial dense depth image.

[0053] The number of pixel points after the point cloud data is projected onto the two-dimensional image is small. Therefore, only a small number of pixel points in the scene image have depth information. Depth estimation can be performed on the pixel points lacking depth information in the sparse depth image based on the perspective geometry principle to obtain the estimated depth value of the pixel points, which is recorded as the initial dense depth image. Among them, the depth information includes the pixel point depth value, and the pixel point depth value represents the physical distance from the pixel point in the scene image to the electronic device for acquiring the image, such as an electronic device for acquiring the image like a camera or lidar.

[0054] The geometric perspective principle uses the perspective phenomenon of objects being larger when closer and smaller when farther away to represent the three-dimensional sense of various target objects in the image. The most obvious phenomenon is the convergence of parallel lines in the image. Just like a road or railway stretching into the distance, the farther away, the smaller, and the closer, the larger. Using this geometric perspective principle, the spatial distance of any target object can be drawn. Sometimes, the shape of the target object also shows this phenomenon of being larger when closer and smaller when farther away visually. Figures of the same size seem to be at the same distance, and figures of different sizes seem to be closer when larger and farther away when smaller. Similarly, for any target object, due to their positions in space and the position of the viewer being different, there will also be differences in height positions in the image, thus creating a sense of space.

[0055] Based on the perspective geometry principle, depth estimation is performed on the pixel points lacking depth information in the sparse depth image, avoiding the construction of a multi-layer and complex cascaded network model, and avoiding the problem that the front-back dependence of multiple networks in the feature extraction process may cause error propagation, resulting in limited model accuracy and difficulty in converging of models with a large number of parameters.

[0056] In step S104, the initial dense depth image is parameter-optimized through the segmentation results of the neural network and the semantic segmentation model to obtain the dense depth image.

[0057] Exemplarily, the neural network can be a feedforward neural network based on the backpropagation algorithm. The feedforward neural network includes an input layer, a hidden layer, and an output layer; the input layer, the hidden layer, and the output layer are connected through network weights and biases, and the loss function for dense depth image estimation is constructed by setting the simulated non-linear continuous functions of different network layers. Or construct the loss function of the dense depth image according to the pixel coordinates of the initial dense depth image, the image vanishing point, the pixel category label, and the pixel depth value; the image vanishing point is the perspective geometry image vanishing point of the initial dense depth image, and the pixel category label is used to distinguish the target objects in the initial dense depth image, such as target objects like the road surface, tracks, vehicles, etc., and the pixel category label can be obtained according to the segmentation results of the semantic segmentation model, and the pixel depth value is the estimated depth value of the initial dense depth image.

[0058] Exemplarily, the calculation formula of the loss function can include:

[0059]

[0060] Among them, formula (1) is the loss function of the pixel points containing the prior depth, and formula (2) is the loss function of the pixel points without the effective prior depth, and the joint estimation can be performed through the prior depth of the neighborhood categories; (x i , y i ) are the pixel coordinates of the initial dense depth image, α, β, and ω are the preset estimation coefficients, λ is the preset harmonic coefficient, (x c , y c ) is the image vanishing point of the initial dense depth image, is the pixel depth value of the initial dense depth image, M*N is the pixel area of a certain target object in the initial dense depth image, or is called the pixel height and width of the target object, and C is the category of a certain target object in the initial dense depth image.

[0061] Exemplarily, the semantic segmentation model is pre-trained based on the sample images of the scene to be measured, and the scene to be measured is a scene related to the scene in step S101, such as a scene related to the driving process of an autonomous vehicle, a scene including tracks, road surfaces, vehicles, etc. The segmentation result is the pixel point label classification of a two-dimensional image, such as the label classification of each target object pixel point in the two-dimensional image, and the label classification is used to represent the category of the target object.

[0062] The training process of the semantic segmentation model includes: collecting sample images of the scene to be measured related to the scene, performing category annotation on the sample images, using the annotated sample images as input images, and inputting them into an efficient residual decomposition network for training to obtain the semantic segmentation model. Performing category annotation on the sample images includes annotating the categories of various target objects in the sample images.

[0063] The semantic segmentation model includes a first downsampling layer, a second downsampling layer, a third downsampling layer, a first upsampling layer, a second upsampling layer, and a third upsampling layer.

[0064] The first downsampling layer includes a convolutional layer, a batch normalization layer, and an activation layer, and is used to extract features from the input image to obtain a first feature map;

[0065] The second downsampling layer includes a convolutional layer, a batch normalization layer, an activation layer, and a five-level cascaded efficient residual decomposition module, and is used to extract features and perform feature fusion on the first feature map to obtain a second feature map;

[0066] The third downsampling layer includes a convolutional layer, a batch normalization layer, an activation layer, and an eight-level cascaded efficient residual decomposition module, and is used to extract features and perform feature fusion on the second feature map to obtain a third feature map;

[0067] The first upsampling layer includes a transposed convolutional layer and a batch normalization layer, and is used to perform an upsampling operation on the third feature map to obtain a fourth feature map to achieve feature recovery; and also includes a two-level cascaded efficient residual decomposition module, and is used to perform feature fusion on the fourth feature map to obtain a fifth feature map;

[0068] The second upsampling layer includes a transposed convolutional layer and a batch normalization layer, and is used to perform an upsampling operation on the fifth feature map to obtain a sixth feature map to achieve feature recovery; and also includes a two-level cascaded efficient residual decomposition module, and is used to perform feature fusion on the sixth feature map to obtain a seventh feature map;

[0069] The third upsampling layer includes a transposed convolutional layer and a batch normalization layer, and is used to perform an upsampling operation on the seventh feature map to obtain an output image; the size of the output image is the same as the size of the input image.

[0070] Taking the annotated sample image as the input image, input it into an efficient residual decomposition network for training. The obtained semantic segmentation model includes: obtaining the difference between the input image and the output image, and adjusting the parameters of the first downsampling layer, the second downsampling layer, the third downsampling layer, the first upsampling layer, the second upsampling layer, and the third upsampling layer according to the difference. Without changing the input image, adjust the parameters of each layer to obtain the optimal parameter weights that minimize the difference between the output image and the input image. If the difference is less than a predetermined threshold, then use the model with the optimal parameter weights as the semantic segmentation model.

[0071] Finally, optimize the parameters of the initial dense depth image through the segmentation results of the neural network and the semantic segmentation model to obtain the dense depth image. Exemplarily, the parameters of the initial dense depth image can be optimized based on the feedforward neural network and the segmentation results of the semantic segmentation model using the backpropagation algorithm to obtain the dense depth image. Take the pixel point coordinates, pixel point class labels, and pixel point depth values of the initial dense depth image as the input of the feedforward neural network. The pixel point depth value represents the physical distance from the pixel point in the initial dense depth image to the lidar; then the feedforward neural network corrects the pixel point depth value according to the segmentation result to obtain the dense depth image. Incorporating the segmentation result of the semantic segmentation model helps to correct the depth value deviation and pixel point position of the target object in the initial dense depth image.

[0072] It should be noted that the depth value of each pixel point in the initial dense depth image is a parameter, and these parameters can be optimized based on the backpropagation algorithm. Optimize the parameters of the previously obtained initial dense depth image based on the backpropagation algorithm and the segmentation result, that is, optimize the depth value of each pixel point in the initial dense depth image to obtain the dense depth image. For example, correct some pixel points with significantly incorrect depth values in the initial dense depth image. Finally, use the optimized and relatively accurate dense depth image to calibrate the estimation algorithm coefficients and scene segmentation weights.

[0073] The backpropagation algorithm is an algorithm for adjusting weights from back to front. Starting from the last layer of the neural network, adjust the weights of the neural network layer by layer to make the neural network make better predictions during depth prediction. The specific adjustment algorithm can be the chain rule of gradient calculation or stochastic gradient descent, etc. The present disclosure does not limit this.

[0074] Based on the perspective geometry principle, it is possible to effectively perform pixel depth densification estimation on a priori sparse depth images. Then, by combining the scene category information (segmentation results) of the semantic segmentation model, abnormal estimated values can be simply and efficiently eliminated. The estimation coefficients of the initial dense depth image are corrected through a feedforward neural network using the backpropagation algorithm to ensure the global minimization of depth estimation for various category information in the image scene, that is, finally obtaining a relatively accurate global depth prediction of the image.

[0075] In summary, the image depth prediction method provided by the present disclosure includes obtaining a two-dimensional image and point cloud data of the same scene, projecting the point cloud data onto the two-dimensional image to generate a sparse depth image, performing depth estimation on the pixel points lacking depth information in the sparse depth image based on the perspective geometry principle to obtain an initial dense depth image, and optimizing the parameters of the initial dense depth image through the segmentation results of a neural network and a semantic segmentation model to obtain a dense depth image; by fusing the point cloud data and the two-dimensional image to obtain a sparse depth image, performing depth estimation on the sparse depth image based on the perspective geometry principle, simply and efficiently eliminating abnormal estimated values, and finally optimizing the results of depth estimation through a neural network and a semantic segmentation model to ensure the global minimization of depth estimation errors for various category information in the image scene, and finally obtaining a relatively accurate global depth prediction of the image, avoiding the construction of a multi-layer and complex cascaded network model, avoiding the front-to-back dependence of multi-level networks during the feature extraction process, which may cause error propagation resulting in limited model accuracy, and avoiding the convergence difficulty of models with a large number of parameters. The data fusion method can more efficiently and real-time obtain sparse and accurate prior pixel depth values.

[0076] Figure 4 It is a block diagram of an image depth prediction device shown in an exemplary embodiment of the present disclosure. Referring to Figure 4 , the device 20 includes an acquisition module 201, a projection module 203, an estimation module 205, and a processing module 207.

[0077] The acquisition module 201 is configured to acquire a two-dimensional image and point cloud data of the same scene;

[0078] The projection module 203 is configured to project the point cloud data onto the two-dimensional image to generate a sparse depth image;

[0079] The estimation module 205 is configured to perform depth estimation on the pixel points lacking depth information in the sparse depth image based on the perspective geometry principle to obtain an initial dense depth image;

[0080] The processing module 207 is configured to optimize the parameters of the initial dense depth image through the segmentation results of a neural network and a semantic segmentation model to obtain a dense depth image; the segmentation result is the pixel point label classification of the two-dimensional image.

[0081] Optionally, the projection module 203 is further configured to project the point cloud data onto the two-dimensional image based on the extrinsic parameters jointly calibrated by the camera and the lidar to generate the sparse depth image; the jointly calibrated extrinsic parameters represent the data conversion relationship between the pixels on the same target in the two-dimensional image and the point cloud data.

[0082] Optionally, the neural network is a feedforward neural network based on the backpropagation algorithm;

[0083] The feedforward neural network includes an input layer, a hidden layer, and an output layer; the input layer, the hidden layer, and the output layer are connected by network weights and biases.

[0084] Optionally, the processing module 207 is further configured to use the pixel coordinates, pixel class labels, and pixel depth values of the initial dense depth image as the input of the feedforward neural network; the pixel depth value represents the physical distance from the pixel in the initial dense depth image to the lidar.

[0085] The feedforward neural network corrects the pixel depth value according to the segmentation result to obtain the dense depth image.

[0086] Optionally, the processing module 207 is further configured to construct a loss function of the dense depth image according to the pixel coordinates, image vanishing point, pixel class label, and pixel depth value of the initial dense depth image; the image vanishing point is the perspective geometric image vanishing point of the initial dense depth image.

[0087] Or the feedforward neural network constructs and predicts the loss function by setting a continuous function that simulates non-linearity with different network layers.

[0088] Optionally, the processing module 207 is further configured to collect sample images of the measured scene related to the scene;

[0089] Perform class annotation on the sample images;

[0090] Use the annotated sample images as input images and input them into an efficient residual decomposition network for training to obtain the semantic segmentation model.

[0091] Optionally, the semantic segmentation model includes a first downsampling layer, a second downsampling layer, a third downsampling layer, a first upsampling layer, a second upsampling layer, and a third upsampling layer;

[0092] The first downsampling layer includes a convolutional layer, a batch normalization layer, and an activation layer, and is configured to perform feature extraction on the input image to obtain a first feature map;

[0093] The second downsampling layer includes a convolutional layer, a batch normalization layer, an activation layer, and a five-level cascaded efficient residual decomposition module, which are used to perform feature extraction and feature fusion on the first feature map to obtain a second feature map;

[0094] The third downsampling layer includes a convolutional layer, a batch normalization layer, an activation layer, and an eight-level cascaded efficient residual decomposition module, which are used to perform feature extraction and feature fusion on the second feature map to obtain a third feature map;

[0095] The first upsampling layer includes a transposed convolutional layer and a batch normalization layer, which are used to perform an upsampling operation on the third feature map to obtain a fourth feature map for feature recovery; and it also includes a two-level cascaded efficient residual decomposition module, which is used to perform feature fusion on the fourth feature map to obtain a fifth feature map;

[0096] The second upsampling layer includes a transposed convolutional layer and a batch normalization layer, which are used to perform an upsampling operation on the fifth feature map to obtain a sixth feature map for feature recovery; and it also includes a two-level cascaded efficient residual decomposition module, which is used to perform feature fusion on the sixth feature map to obtain a seventh feature map;

[0097] The third upsampling layer includes a transposed convolutional layer and a batch normalization layer, which are used to perform an upsampling operation on the seventh feature map to obtain an output image; the size of the output image is the same as that of the input image.

[0098] Optionally, the processing module 207 is further configured to obtain the difference between the input image and the output image;

[0099] According to the difference, adjust the parameters of the first downsampling layer, the second downsampling layer, the third downsampling layer, the first upsampling layer, the second upsampling layer, and the third upsampling layer to obtain the semantic segmentation model.

[0100] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0101] Figure 5 is a block diagram of an electronic device shown according to an exemplary embodiment. As Figure 5 shown, the electronic device 400 may include: a processor 401, a memory 402. The electronic device 400 may further include one or more of a multimedia component 403, an input / output (I / O) interface 404, and a communication component 405.

[0102] Among them, the processor 401 is used to control the overall operation of the electronic device 400 to complete all or part of the steps in the above image depth prediction method. The memory 402 is used to store various types of data to support the operation of the electronic device 400. These data may include, for example, instructions for any application or method operating on the electronic device 400, as well as application-related data, such as contact data, received and sent messages, pictures, audio, video, and so on. The memory 402 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc. The multimedia component 403 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 402 or sent through the communication component 405. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 404 provides an interface between the processor 401 and other interface modules, and the above other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 405 is used for wired or wireless communication between the electronic device 400 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G, etc., or a combination of one or more of them, is not limited herein. Therefore, the corresponding communication component 405 may include: a Wi-Fi module, a Bluetooth module, an NFC module, and so on.

[0103] In one exemplary embodiment, the electronic device 400 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components, and is used to execute the above-mentioned image depth prediction method.

[0104] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When the program instructions are executed by a processor, the steps of the above-mentioned image depth prediction method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 402 including program instructions, and the above-mentioned program instructions can be executed by the processor 401 of the electronic device 400 to complete the above-mentioned image depth prediction method.

[0105] Figure 6 is a block diagram of another electronic device shown according to an exemplary embodiment. For example, the electronic device 500 can be provided as a server. Referring to Figure 6 , the electronic device 500 includes a processor 522, the number of which can be one or more, and a memory 532 for storing computer programs executable by the processor 522. The computer programs stored in the memory 532 can include one or more modules each corresponding to a set of instructions. In addition, the processor 522 can be configured to execute the computer program to execute the above-mentioned image depth prediction method.

[0106] In addition, the electronic device 500 can further include a power supply component 526 and a communication component 550. The power supply component 526 can be configured to perform power management of the electronic device 500, and the communication component 550 can be configured to implement communication of the electronic device 500, for example, wired or wireless communication. In addition, the electronic device 500 can further include an input / output (I / O) interface 558. The electronic device 500 can operate based on an operating system stored in the memory 532, such as Windows Server TM , Mac OSX TM , Unix TM , Linux TM and so on.

[0107] In another exemplary embodiment, there is also provided a computer-readable storage medium including program instructions, which, when executed by a processor, implement the steps of the above-described image depth prediction method. For example, the non-transitory computer-readable storage medium may be the above-described memory 532 including program instructions, and the above program instructions may be executed by the processor 522 of the electronic device 500 to complete the above-described image depth prediction method.

[0108] In another exemplary embodiment, there is also provided a computer program product, which includes a computer program executable by a programmable device, and the computer program has a code portion for executing the above-described image depth prediction method when executed by the programmable device.

[0109] The preferred embodiments of the present disclosure have been described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0110] In addition, it should be noted that, in the above specific embodiments, the various specific technical features described can be combined in any suitable manner without conflict. To avoid unnecessary repetition, the present disclosure does not separately describe various possible combination manners.

[0111] Furthermore, any combination can be made between various different embodiments of the present disclosure as long as it does not violate the idea of the present disclosure, and it should also be regarded as the content disclosed by the present disclosure.

Claims

1. An image depth prediction method, characterized in that, Including: Obtaining a two-dimensional image and point cloud data of the same scene; Projecting the point cloud data onto the two-dimensional image to generate a sparse depth image; Based on the perspective geometry principle, estimating the depth of pixel points lacking depth information in the sparse depth image to obtain an initial dense depth image; Optimizing the parameters of the initial dense depth image through the segmentation results of a neural network and a semantic segmentation model to obtain a dense depth image; the segmentation result is the pixel point label classification of the two-dimensional image; the neural network is a feedforward neural network based on the backpropagation algorithm; the feedforward neural network includes an input layer, a hidden layer, and an output layer; the input layer, the hidden layer, and the output layer are connected by network weights and biases; Constructing a loss function of the dense depth image according to the pixel point coordinates, the image vanishing point, the pixel point class label, and the pixel point depth value of the initial dense depth image; the image vanishing point is the perspective geometry image vanishing point of the initial dense depth image; or the feedforward neural network constructs and predicts the loss function by setting a continuous function simulating non-linearity with different network layers; The step of optimizing the parameters of the initial dense depth image through the segmentation results of a neural network and a semantic segmentation model to obtain a dense depth image includes: Taking the pixel point coordinates, the pixel point class label, and the pixel point depth value of the initial dense depth image as the input of the feedforward neural network; The pixel point depth value represents the physical distance from the pixel point in the initial dense depth image to the lidar; The feedforward neural network corrects the pixel point depth value according to the segmentation result to obtain the dense depth image.

2. The method according to claim 1, wherein The two-dimensional image is obtained by a camera, and the point cloud data is obtained by a lidar; the step of projecting the point cloud data onto the two-dimensional image to generate a sparse depth image includes: Based on the external parameters jointly calibrated by the camera and the lidar, projecting the point cloud data onto the two-dimensional image to generate the sparse depth image; the jointly calibrated external parameters represent the data conversion relationship between the pixel points on the same target in the two-dimensional image and the point cloud data.

3. The method according to claim 1, characterized in that, The semantic segmentation model is pre-trained; The training process of the semantic segmentation model includes: Collecting sample images of the measured scene related to the scene; Performing class annotation on the sample images; Taking the annotated sample images as input images and inputting them into an efficient residual decomposition network for training to obtain the semantic segmentation model.

4. The method according to claim 3, characterized in that, The semantic segmentation model includes a first downsampling layer, a second downsampling layer, a third downsampling layer, a first upsampling layer, a second upsampling layer, and a third upsampling layer; The first downsampling layer includes a convolutional layer, a batch normalization layer, and an activation layer, and is used for feature extraction of the input image to obtain a first feature map; The second downsampling layer includes a convolutional layer, a batch normalization layer, an activation layer, and a five-level cascaded efficient residual decomposition module, and is used for feature extraction and feature fusion of the first feature map to obtain a second feature map; The third downsampling layer includes a convolutional layer, a batch normalization layer, an activation layer, and an eight-level cascaded efficient residual decomposition module, which are used to perform feature extraction and feature fusion on the second feature map to obtain a third feature map; The first upsampling layer includes a transposed convolutional layer and a batch normalization layer, which are used to perform an upsampling operation on the third feature map to obtain a fourth feature map for feature recovery; and further includes two cascaded efficient residual decomposition modules, which are used to perform feature fusion on the fourth feature map to obtain a fifth feature map; The second upsampling layer includes a transposed convolutional layer and a batch normalization layer, which are used to perform an upsampling operation on the fifth feature map to obtain a sixth feature map for feature recovery; and further includes two cascaded efficient residual decomposition modules, which are used to perform feature fusion on the sixth feature map to obtain a seventh feature map; The third upsampling layer includes a transposed convolutional layer and a batch normalization layer, which are used to perform an upsampling operation on the seventh feature map to obtain an output image; the size of the output image is the same as that of the input image.

5. The method according to claim 4, wherein The step of taking the labeled sample image as the input image and inputting it into the efficient residual decomposition network for training to obtain the semantic segmentation model includes: Obtaining the difference between the input image and the output image; Adjusting the parameters of the first downsampling layer, the second downsampling layer, the third downsampling layer, the first upsampling layer, the second upsampling layer, and the third upsampling layer according to the difference to obtain the semantic segmentation model.

6. An image depth prediction device, characterized in that, Including: An acquisition module, which is used to acquire the two-dimensional image and point cloud data of the same scene; A projection module, which is used to project the point cloud data onto the two-dimensional image to generate a sparse depth image; An estimation module, which is used to estimate the depth of the pixel points lacking depth information in the sparse depth image based on the perspective geometry principle to obtain an initial dense depth image; A processing module, which is used to optimize the parameters of the initial dense depth image through the segmentation result of the neural network and the semantic segmentation model to obtain a dense depth image; the segmentation result is the pixel point label classification of the two-dimensional image; the neural network is a feedforward neural network based on the backpropagation algorithm; the feedforward neural network includes an input layer, a hidden layer, and an output layer; the input layer, the hidden layer, and the output layer are connected through network weights and biases; The step of optimizing the parameters of the initial dense depth image through the segmentation result of the neural network and the semantic segmentation model to obtain a dense depth image includes: taking the pixel point coordinates, pixel point category labels, and pixel point depth values of the initial dense depth image as the input of the feedforward neural network; The pixel point depth value represents the physical distance from the pixel point in the initial dense depth image to the lidar; the feedforward neural network corrects the pixel point depth value according to the segmentation result to obtain the dense depth image; The processing module is further used for: Construct a loss function for the dense depth image based on the pixel point coordinates, the image vanishing point, the pixel point class label, and the pixel point depth value of the initial dense depth image; the image vanishing point is the perspective geometric image vanishing point of the initial dense depth image; Alternatively, the feedforward neural network constructs to predict the loss function by setting a continuous function that simulates non-linearity with different numbers of network layers.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-5.

8. An electronic device, characterized in that, Comprising: A memory storing a computer program thereon; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-5.

Images