Depth estimation method, device, storage medium and vehicle based on surround view camera

Abstract

The present invention relates to the field of autonomous driving technology, and specifically provides a depth estimation method, device, storage medium and vehicle based on a surround-view camera, aiming to solve the problem of how to perform effective depth estimation of a close-range environment while ensuring low cost and high robustness. To this end, the present invention applies a depth estimation model to perform depth estimation on input multiple frames of image data based on continuous frame image data collected by a surround-view camera to obtain a depth estimation result of the image data. Since the depth estimation model is obtained by training based on the pixel relationship between continuous frame image samples, it can make full use of the pixel relationship between the previous and next frames as supervision, and only uses the image data collected by the surround-view camera to obtain the depth estimation result, which can effectively reduce costs. Since the surround-view camera can perceive the close-range environment of the vehicle, the depth estimation method of the present invention can achieve depth perception within a close range near the vehicle.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and specifically provides a depth estimation method, device, storage medium and vehicle based on a surround-view camera. Background Art

[0002] In recent years, with the continuous development of deep learning technology, more and more deep learning-based autonomous driving solutions have been implemented in real-world applications. Depth estimation is a crucial component of autonomous driving technology. While lidar can provide relatively good distance and depth information, it is relatively expensive and has certain limitations in its ability to perceive environmental conditions, being easily affected by environmental factors such as weather. Consequently, purely visual solutions for monocular depth estimation have also gained attention.

[0003] In monocular cameras, pinhole cameras need to capture an image of the environment ahead with equal triangular focal lengths. This means they have a relatively long sensing range and are not suitable for close-range environmental perception near the vehicle.

[0004] Accordingly, a new depth estimation solution is needed in this field to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects, the present invention is proposed to provide a solution or at least partially solve the problem of how to perform effective depth estimation in a close-range environment while ensuring low cost and high robustness.

[0006] In a first aspect, the present invention provides a depth estimation method based on a surround view camera, the method comprising:

[0007] The continuous frame image data collected by the surround view camera is input into a trained depth estimation model to obtain a depth estimation result, wherein the trained depth estimation model is trained based on the pixel relationship between the continuous frame image samples.

[0008] In one technical solution of the above-mentioned depth estimation method based on a surround view camera, the method further includes training the depth estimation model according to the following steps:

[0009] Performing depth prediction on a current frame image sample among the continuous frame image samples to obtain a predicted depth map of the current frame image sample;

[0010] Obtaining a point cloud truth map based on calibration parameters between the surround-view camera and the radar and point cloud data samples corresponding to the current frame image samples;

[0011] Obtaining a first loss according to a difference between the predicted depth map of the current frame image sample and the point cloud true value map;

[0012] Obtaining a photometric error of pixels of a predicted depth map between the current frame image sample and an adjacent frame image sample as a second loss;

[0013] The depth estimation model is trained based on the first loss and the second loss.

[0014] In one technical solution of the above-mentioned depth estimation method based on a surround-view camera, obtaining a point cloud ground truth map based on calibration parameters between the surround-view camera and the radar and point cloud data samples corresponding to the current frame image samples includes:

[0015] Projecting the point cloud data samples to the pixel coordinate system of the surround-view camera according to the calibration parameters to obtain an initial depth truth map;

[0016] The initial depth truth map is processed by a removal of obstacle occlusion algorithm to obtain the point cloud truth map.

[0017] In one technical solution of the above-mentioned depth estimation method based on a surround view camera, the step of performing a removal of obstacle occlusion algorithm on the initial depth truth map to obtain the point cloud truth map includes:

[0018] Dividing the depth values ​​of the initial depth truth image into N depth intervals according to preset intervals;

[0019] For the nth depth interval, assign the corresponding depth value to the point cloud data in the depth interval;

[0020] For the (n+1)th depth interval, compare the point cloud data in the depth interval with the point cloud data in the (n)th depth interval. If the point cloud data in the depth interval has the same depth value as the point cloud data in the (n)th depth interval, filter out the pixels in the (n+1)th depth interval corresponding to the depth value to obtain the point cloud ground truth map.

[0021] Here, n traverses from 1 to N-1, and the depth value of the n+1th depth interval is greater than the depth value of the nth depth interval.

[0022] In one technical solution of the above-mentioned depth estimation method based on a surround view camera, obtaining the photometric error of pixels of the predicted depth map between the current frame image sample and the adjacent frame image sample includes:

[0023] Obtaining a posture matrix from the adjacent frame image samples to the current frame image samples;

[0024] Projecting the predicted depth map corresponding to the adjacent frame image samples onto the predicted depth map corresponding to the current frame image samples according to the posture matrix;

[0025] Calculating an overlapping area on a predicted depth map corresponding to the current frame image sample according to the projection;

[0026] The photometric errors of the pixels in the overlapping region are calculated.

[0027] In a technical solution of the above-mentioned depth estimation method based on a surround-view camera, the adjacent frame image samples are a previous frame image sample and a next frame image sample of the current frame image sample.

[0028] In one technical solution of the above-mentioned depth estimation method based on a surround view camera, the calculating of the overlapping area on the predicted depth map corresponding to the current frame image sample includes:

[0029] If a pixel exists in both the predicted depth map corresponding to the previous frame image sample and the predicted depth map corresponding to the next frame image sample, but only appears once in the predicted depth map corresponding to the current frame image sample, then the minimum pixel value of the pixel in the predicted depth map corresponding to the previous frame image sample and the predicted depth map corresponding to the next frame image sample shall be used as the value of its corresponding pixel in the predicted depth map corresponding to the current frame image sample.

[0030] In one technical solution of the above-mentioned depth estimation method based on a surround view camera, the method further includes:

[0031] Estimate, according to the speed calibration file, the position of the predicted depth map corresponding to the adjacent frame image samples in the predicted depth map corresponding to the current frame image samples;

[0032] The projection is compensated according to the estimation result.

[0033] In a second aspect, a control device is provided, which includes at least one processor and at least one storage device, wherein the storage device is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the depth estimation method based on the surround-view camera described in any one of the technical solutions of the depth estimation method based on the surround-view camera.

[0034] In a third aspect, a computer-readable storage medium is provided, which stores a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the depth estimation method based on a surround-view camera as described in any one of the technical solutions of the depth estimation method based on a surround-view camera.

[0035] In a fourth aspect, a vehicle is provided, comprising a surround-view camera and the control device described in the above-mentioned control device technical solution.

[0036] The above one or more technical solutions of the present invention have at least one or more of the following

[0037] Beneficial effects:

[0038] In the technical solution for implementing the present invention, the present invention applies a depth estimation model to perform depth estimation on the input multi-frame image data based on the continuous frame image data collected by the surround-view camera to obtain a depth estimation result of the image data. Through the above configuration, since the depth estimation model is obtained by training based on the pixel relationship between continuous frame image samples, it can fully utilize the pixel relationship between the previous and next frames as supervision, so that the depth estimation model has better robustness. And only by applying the image data collected by the surround-view camera, the depth estimation result can be obtained, and there is no need to rely on the laser radar, which can effectively reduce the cost and is not easily affected by environmental factors such as weather. At the same time, since the surround-view camera can perceive the close environment of the vehicle, the depth estimation method of the present invention can realize depth perception within a close range near the vehicle, with the characteristics of low cost and high robustness.

[0039] Solution 1. A depth estimation method based on a surround view camera, characterized in that the method comprises:

[0040] The continuous frame image data collected by the surround view camera is input into a trained depth estimation model to obtain a depth estimation result, wherein the trained depth estimation model is trained based on the pixel relationship between the continuous frame image samples.

[0041] Solution 2. The surround view camera-based depth estimation method according to Solution 1, further comprising training the depth estimation model according to the following steps:

[0042] Performing depth prediction on a current frame image sample among the continuous frame image samples to obtain a predicted depth map of the current frame image sample;

[0043] Obtaining a point cloud truth map based on calibration parameters between the surround-view camera and the radar and point cloud data samples corresponding to the current frame image samples;

[0044] Obtaining a first loss according to a difference between the predicted depth map of the current frame image sample and the point cloud true value map;

[0045] Obtaining a photometric error of pixels of a predicted depth map between the current frame image sample and an adjacent frame image sample as a second loss;

[0046] The depth estimation model is trained based on the first loss and the second loss.

[0047] Solution 3. The depth estimation method based on surround view camera according to Solution 2 is characterized in that:

[0048] The step of obtaining a point cloud true value map based on calibration parameters between the surround view camera and the radar and point cloud data samples corresponding to the current frame image samples includes:

[0049] Projecting the point cloud data samples to the pixel coordinate system of the surround-view camera according to the calibration parameters to obtain an initial depth truth map;

[0050] The initial depth truth map is processed by a removal of obstacle occlusion algorithm to obtain the point cloud truth map.

[0051] Solution 4. The depth estimation method based on surround view camera according to Solution 3 is characterized in that:

[0052] The step of performing a removal of obstacle occlusion algorithm on the initial depth true value map to obtain the point cloud true value map includes:

[0053] Dividing the depth values ​​of the initial depth truth image into N depth intervals according to preset intervals;

[0054] For the nth depth interval, assign the corresponding depth value to the point cloud data in the depth interval;

[0055] For the (n+1)th depth interval, compare the point cloud data in the depth interval with the point cloud data in the (n)th depth interval. If the point cloud data in the depth interval has the same depth value as the point cloud data in the (n)th depth interval, filter out the pixels in the (n+1)th depth interval corresponding to the depth value to obtain the point cloud ground truth map.

[0056] Here, n traverses from 1 to N-1, and the depth value of the n+1th depth interval is greater than the depth value of the nth depth interval.

[0057] Solution 5. The depth estimation method based on surround view camera according to Solution 2 is characterized in that:

[0058] The obtaining of a photometric error of a pixel of a predicted depth map between the current frame image sample and an adjacent frame image sample includes:

[0059] Obtaining a posture matrix from the adjacent frame image samples to the current frame image samples;

[0060] Projecting the predicted depth map corresponding to the adjacent frame image samples onto the predicted depth map corresponding to the current frame image samples according to the posture matrix;

[0061] Calculating an overlapping area on a predicted depth map corresponding to the current frame image sample according to the projection;

[0062] The photometric errors of the pixels in the overlapping region are calculated.

[0063] Solution 6. The depth estimation method based on a surround view camera according to Solution 2 or 5 is characterized in that:

[0064] The adjacent frame image samples are the previous frame image sample and the next frame image sample of the current frame image sample.

[0065] Solution 7. The surround-view camera-based depth estimation method according to Solution 6, wherein calculating the overlapping area on the predicted depth map corresponding to the current frame image sample includes:

[0066] If a pixel exists in both the predicted depth map corresponding to the previous frame image sample and the predicted depth map corresponding to the next frame image sample, but only appears once in the predicted depth map corresponding to the current frame image sample, then the minimum pixel value of the pixel in the predicted depth map corresponding to the previous frame image sample and the predicted depth map corresponding to the next frame image sample shall be used as the value of its corresponding pixel in the predicted depth map corresponding to the current frame image sample.

[0067] Solution 8. The depth estimation method based on a surround view camera according to Solution 5, characterized in that the method further comprises:

[0068] Estimate, according to the speed calibration file, the position of the predicted depth map corresponding to the adjacent frame image samples in the predicted depth map corresponding to the current frame image samples;

[0069] The projection is compensated according to the estimation result.

[0070] Solution 9. A control device comprising at least one processor and at least one storage device, wherein the storage device is suitable for storing multiple program codes, and is characterized in that the program codes are suitable for being loaded and run by the processor to execute the depth estimation method based on a surround view camera described in any one of Solutions 1 to 8.

[0071] Solution 10. A computer-readable storage medium storing a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the depth estimation method based on a surround view camera according to any one of Solutions 1 to 8.

[0072] Solution 11. A vehicle, characterized in that the vehicle includes a surround-view camera and the control device described in Solution 9. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Among them:

[0074] Figure 11 is a flow chart showing the main steps of a depth estimation method based on a surround view camera according to an embodiment of the present invention;

[0075] Figure 2 1 is a flow chart of the main steps of training a depth estimation model according to an implementation of an embodiment of the present invention;

[0076] Figure 3 1 is a flow chart of the main steps of obtaining a point cloud truth map according to an embodiment of the present invention. DETAILED DESCRIPTION

[0077] Some embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0078] In the description of the present invention, "module" and "processor" may include hardware, software, or a combination of both. A module may include hardware circuitry, various suitable sensors, communication ports, and memory. It may also include software components, such as program code, or a combination of software and hardware. A processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor may be implemented in software, hardware, or a combination of both. Non-transitory computer-readable storage media include any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, and the like. The term "A and / or B" refers to all possible combinations of A and B, such as only A, only B, or both A and B. The terms "at least one of A or B" or "at least one of A and B" have similar meanings to "A and / or B" and may include only A, only B, or both A and B. The singular forms "one" and "the" may also include the plural forms.

[0079] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart showing the main steps of a depth estimation method based on a surround view camera according to an embodiment of the present invention. Figure 1 As shown, the depth estimation method based on a surround view camera in the embodiment of the present invention mainly includes the following step S101.

[0080] Step S101: inputting continuous frame image data captured by a surround view camera into a trained depth estimation model to obtain a depth estimation result, wherein the trained depth estimation model is trained based on the pixel relationship between continuous frame image samples.

[0081] In this embodiment, the continuous frame image data collected by the surround view camera can be input into a trained depth estimation model to obtain a depth estimation result. The depth estimation model is trained based on the pixel relationship between the continuous frame image samples.

[0082] In one embodiment, the continuous frame image samples may be image samples including the current frame and a frame before the current frame, or may be image samples including the current frame and a frame after the current frame.

[0083] In another embodiment, the continuous frame image samples may include the current frame and the previous and next frames of the current frame. Including the image samples of the previous and next frames of the current frame can obtain a more accurate pixel relationship, thereby better training the depth estimation model.

[0084] Based on the above step S101, the embodiment of the present invention applies a depth estimation model to perform depth estimation on the input multiple frames of image data based on the continuous frame image data collected by the surround-view camera to obtain a depth estimation result of the image data. Through the above configuration, since the depth estimation model is obtained by training based on the pixel relationship between continuous frame image samples, it can fully utilize the pixel relationship between the previous and next frames as supervision, so that the depth estimation model has better robustness. And the depth estimation result can be obtained by only using the image data collected by the surround-view camera, without relying on the laser radar, which can effectively reduce the cost and is not easily affected by environmental factors such as weather. At the same time, since the surround-view camera can perceive the close environment of the vehicle, the depth estimation method of the embodiment of the present invention can realize depth perception within a close range near the vehicle, with the characteristics of low cost and high robustness.

[0085] In one implementation of the embodiment of the present invention, the present invention may further include the following steps S201 to S205, and the depth estimation model is trained according to steps S201 to S205:

[0086] Step S201: performing depth prediction on a current frame image sample among continuous frame image samples to obtain a predicted depth map of the current frame image sample.

[0087] In this embodiment, continuous frame image samples may be input into a depth prediction model to obtain a predicted depth map of each frame image sample.

[0088] Step S202: Obtain a point cloud ground truth map based on the calibration parameters between the surround view camera and the radar and the point cloud data samples corresponding to the current frame image samples.

[0089] In this embodiment, step S202 may further include step S2021 and step S2022:

[0090] Step S2021: Project the point cloud data samples to the pixel coordinate system of the surround view camera according to the calibration parameters to obtain an initial depth truth map.

[0091] In this embodiment, based on the calibration parameters between the surround-view camera and the LiDAR that collects the point cloud data samples, the point cloud data samples corresponding to the current frame image sample can be projected into the pixel coordinate system of the surround-view camera to obtain the initial depth ground truth map. The calibration parameters can include intrinsic and extrinsic parameters between the surround-view camera and the LiDAR.

[0092] Step S2022: The initial depth truth map is processed by an obstacle removal algorithm to obtain a point cloud truth map.

[0093] In this embodiment, step S2022 may further include the following steps S20221 to S20223:

[0094] Step S20221: Divide the depth values ​​of the initial depth truth image into N depth intervals according to preset intervals.

[0095] In this embodiment, the depth values ​​of the initial true depth value map may be divided into N depth intervals according to preset intervals, where N is a positive integer greater than 1.

[0096] Step S20222: for the nth depth interval, assign corresponding depth values ​​to the point cloud data in the depth interval.

[0097] In this embodiment, for the nth depth interval among the N depth intervals, a corresponding depth value may be assigned to the point cloud data in the depth interval.

[0098] Step S20223: For the n+1th depth interval, compare the point cloud data in the depth interval with the point cloud data in the nth depth interval. If the point cloud data in the depth interval has the same depth value as the point cloud data in the nth depth interval, filter out the pixels of the corresponding depth value in the n+1th depth interval to obtain a point cloud true value map; where n traverses from 1 to N-1, and the depth value of the n+1th depth interval is greater than the depth value of the nth depth interval.

[0099] In this embodiment, for the n+1th depth interval among the N depth intervals, the point cloud data in that depth interval can be compared with the point cloud data in the nth depth interval. If the point cloud data in the current depth interval has the same depth value as the point cloud data in the nth depth interval, the pixels with the corresponding depth value in the current depth interval can be filtered out to obtain the point cloud ground truth map. This can effectively remove occlusions in the point cloud data samples and improve the quality of the point cloud ground truth map.

[0100] Step S203: Obtain a first loss based on the difference between the predicted depth map of the current frame image sample and the point cloud true value map.

[0101] In this embodiment, the first loss can be obtained by calculating the difference between the predicted depth map of the current image sample and the ground truth map of the point cloud, wherein the first loss is a supervised loss of the depth estimation model.

[0102] Step S204: Obtain a photometric error between pixels of the predicted depth map of the current frame image sample and the adjacent frame image sample as a second loss.

[0103] In this embodiment, step S204 may further include steps S2041 to S2044:

[0104] Step S2041: Obtain the posture matrix from the adjacent frame image samples to the current frame image samples.

[0105] In this embodiment, a posture prediction network may be applied to obtain a posture matrix from adjacent frame image samples to current frame image samples.

[0106] In one embodiment, the adjacent frame image samples are the previous frame image sample and the next frame image sample of the current frame image sample.

[0107] Step S2042: According to the posture matrix, the predicted depth map corresponding to the adjacent frame image samples is projected onto the predicted depth map corresponding to the current frame image samples.

[0108] In this embodiment, the predicted depth map corresponding to the adjacent frame image samples may be projected onto the predicted depth map corresponding to the current frame image samples according to the posture matrix.

[0109] Step S2043: Calculate the overlapping area on the predicted depth map corresponding to the current frame image sample based on the projection.

[0110] In this embodiment, if a pixel exists in both the predicted depth map corresponding to the previous frame image sample and the predicted depth map corresponding to the next frame image sample, but only appears once in the predicted depth map corresponding to the current frame image sample, the minimum pixel value of the pixel in the predicted depth map corresponding to the previous frame image sample and the predicted depth map corresponding to the next frame image sample is used as the value of the corresponding pixel in the predicted depth map corresponding to the current frame image sample. This method can effectively avoid the problem that the photometric error cannot effectively measure the matching situation when the pixel is blocked.

[0111] Step S2044: Calculate the photometric errors of the pixels in the overlapping area.

[0112] In this embodiment, the photometric error of the pixels in the overlapping area may be calculated as the second loss, which is a self-supervised loss for training the depth estimation model.

[0113] Step S205: training a depth estimation model based on the first loss and the second loss.

[0114] In this embodiment, back propagation can be performed based on the first loss and the second loss to update the network parameters of the depth estimation model, thereby achieving training of the depth estimation model.

[0115] In one embodiment, the first and second losses can be weighted averaged to obtain the loss gradient of the depth estimation model. Backpropagation is then performed based on the loss gradient to update the network parameters. Combining supervised and self-supervised loss to train the depth estimation model can increase its efficiency and robustness, resulting in a smaller error in the depth estimation results obtained when the depth estimation model is used for inference.

[0116] In one embodiment, the present invention may further include the following steps S206 and S207:

[0117] Step S206: Estimate the position of the predicted depth map corresponding to the adjacent frame image samples in the predicted depth map corresponding to the current frame image samples according to the speed calibration file.

[0118] Step S207: Compensate the projection according to the estimation result.

[0119] In this embodiment, the movement of the vehicle causes positional offsets between the preceding and following frames, and the corresponding pixels also experience positional offsets. Therefore, the positional offset can be estimated based on the vehicle's speed calibration file, and the projection of the predicted depth map corresponding to the adjacent frame image samples on the current frame can be compensated based on the estimated result. The speed calibration file contains the vehicle's speed at different times.

[0120] In one embodiment, during the training phase of the depth estimation model, a velocity calibration file can be applied to compensate for the projection, thereby obtaining an accurate projection and further obtaining a more accurate second loss, namely, a self-supervised loss, to improve the training effect of the depth estimation model.

[0121] In another embodiment, after the depth estimation model training is completed, the speed calibration file can be used to fine-tune the depth estimation model. Specifically, the continuous frame image data can be input into the trained depth estimation model, and the projection of the predicted depth map corresponding to the adjacent frame image data on the predicted depth map corresponding to the current frame image data can be compensated based on the speed calibration file, so as to obtain an accurate projection, and further obtain a more accurate second loss, namely the self-supervised loss. In the model fine-tuning stage, the supervised loss can be removed, and only the self-supervised loss can be used to adjust the model parameters of the depth estimation model, so as to achieve fine-tuning of the depth estimation model, thereby obtaining a trained depth estimation model. Using the self-supervised loss for fine-tuning in the model fine-tuning stage can adapt to different scenarios and increase the adaptability and robustness of the model.

[0122] In one embodiment, see the attached Figure 2 , Figure 2 FIG. 1 is a flow chart showing the main steps of training a depth estimation model according to an embodiment of the present invention. Figure 2 As shown in the figure, during the training of the depth estimation model, the continuous frames of the surround view camera (continuous frame image samples), parameter calibration files (calibration parameters), and speed calibration files can be processed and true values ​​generated, and then input into the surround view self-supervised module (depth estimation model) for depth prediction and posture prediction respectively. Among them, the predicted depth map is obtained by ResNet101 and Encoder+Decorder (encoding and decoding); the predicted position and posture (pose matrix) is obtained by PoseNset. Based on the predicted depth map and the predicted position and posture, the predicted depth map of the adjacent frame image samples is projected onto the predicted depth map of the current frame image sample, thereby obtaining the self-supervised photometric loss (second loss) and absolute error loss (first loss). Backpropagation is performed based on the self-supervised photometric loss and absolute error loss to update the model parameters to realize the training of the depth estimation model.

[0123] In one embodiment, see the attached Figure 3 , Figure 3 FIG. 1 is a flow chart showing the main steps of obtaining a point cloud truth map according to an embodiment of the present invention. Figure 3 As shown, the point cloud true value map can be obtained through the following steps S301 to S307.

[0124] Step S301: Input laser point cloud.

[0125] In this embodiment, point cloud data samples collected by a laser radar may be input.

[0126] Step S302: Calibrate parameter projection.

[0127] Step S303: Generate an initial depth ground truth map.

[0128] In this embodiment, the methods described in step S302 and step S303 are similar to the aforementioned step S2021, and for the sake of simplicity, they are not repeated here.

[0129] Step S304: Divide the depth value into depth intervals.

[0130] In this embodiment, the method described in step S304 is similar to the aforementioned step S20221, and for the sake of simplicity, it is not repeated here.

[0131] Step S305: point cloud expansion.

[0132] Step S306: Repeat point filtering.

[0133] In this embodiment, the methods described in step S305 and step S306 are similar to the aforementioned steps S20222 and step S20223, and are not repeated here for simplicity of description.

[0134] Step S307: Generate a point cloud ground truth map.

[0135] In this embodiment, the filtered initial depth truth map can be used as the point cloud truth map.

[0136] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0137] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above-mentioned embodiment of the present invention may also be accomplished by instructing the relevant hardware through a computer program. The computer program may be stored in a computer-readable storage medium. When the computer program is executed by a processor, it may implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal, and software distribution medium capable of carrying the computer program code. It should be noted that the content contained in the computer-readable storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0138] Furthermore, the present invention also provides a control device. In one embodiment of the control device according to the present invention, the control device includes a processor and a storage device. The storage device can be configured to store a program for executing the depth estimation method based on a surround-view camera according to the above-mentioned method embodiment, and the processor can be configured to execute the program in the storage device, which includes but is not limited to a program for executing the depth estimation method based on a surround-view camera according to the above-mentioned method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The control device can be a control device device formed by various electronic devices.

[0139] In an embodiment of the present invention, the control device may be a control device device formed by various electronic devices. In some possible implementations, the control device may include multiple storage devices and multiple processors. The program for executing the depth estimation method based on the surround-view camera of the above-mentioned method embodiment can be divided into multiple subroutines, and each subroutine can be loaded and run by the processor to execute different steps of the depth estimation method based on the surround-view camera of the above-mentioned method embodiment. Specifically, each subroutine can be stored in different storage devices respectively, and each processor can be configured to execute the programs in one or more storage devices to jointly implement the depth estimation method based on the surround-view camera of the above-mentioned method embodiment, that is, each processor executes different steps of the depth estimation method based on the surround-view camera of the above-mentioned method embodiment respectively to jointly implement the depth estimation method based on the surround-view camera of the above-mentioned method embodiment.

[0140] The aforementioned multiple processors may be processors deployed on the same device. For example, the aforementioned control device may be a high-performance device composed of multiple processors, and the aforementioned multiple processors may be processors configured on the high-performance device. Furthermore, the aforementioned multiple processors may also be processors deployed on different devices. For example, the aforementioned control device may be a server cluster, and the aforementioned multiple processors may be processors on different servers in the server cluster.

[0141] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of a computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program for executing the depth estimation method based on a surround-view camera according to the above-mentioned method embodiment. The program can be loaded and run by a processor to implement the above-mentioned depth estimation method based on a surround-view camera. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-transitory computer-readable storage medium.

[0142] Furthermore, the present invention also provides a vehicle. In one embodiment of a vehicle according to the present invention, the vehicle may include a surround view camera and the control device in the control device embodiment.

[0143] Furthermore, it should be understood that since the configuration of each module is merely for the purpose of illustrating the functional units of the apparatus of the present invention, the physical devices corresponding to these modules may be the processor itself, or a portion of the software in the processor, a portion of the hardware, or a combination of software and hardware. Therefore, the number of modules in the figure is merely illustrative.

[0144] Those skilled in the art will appreciate that the various modules in the device can be adaptively split or merged. Such splitting or merging of specific modules does not cause the technical solution to deviate from the principles of the present invention. Therefore, the technical solutions after splitting or merging will fall within the scope of protection of the present invention.

[0145] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A depth estimation method based on a surround view camera, characterized in that: The method comprises: Inputting continuous frame image data collected by the surround view camera into a trained depth estimation model to obtain a depth estimation result, wherein the trained depth estimation model is trained based on the pixel relationship between continuous frame image samples; The method further comprises training the depth estimation model according to the following steps: Performing depth prediction on a current frame image sample among the continuous frame image samples to obtain a predicted depth map of the current frame image sample; Obtaining a point cloud truth map based on calibration parameters between the surround-view camera and the radar and point cloud data samples corresponding to the current frame image samples; Obtaining a first loss according to a difference between the predicted depth map of the current frame image sample and the point cloud true value map; Obtaining a photometric error of pixels of a predicted depth map between the current frame image sample and an adjacent frame image sample as a second loss; The depth estimation model is trained based on the first loss and the second loss.

2. The depth estimation method based on surround view camera according to claim 1, characterized in that The step of obtaining a point cloud true value map based on calibration parameters between the surround view camera and the radar and point cloud data samples corresponding to the current frame image samples includes: Projecting the point cloud data samples to the pixel coordinate system of the surround-view camera according to the calibration parameters to obtain an initial depth truth map; The initial depth truth map is processed by a removal of obstacle occlusion algorithm to obtain the point cloud truth map.

3. The depth estimation method based on surround view camera according to claim 2, characterized in that The step of performing a removal of obstacle occlusion algorithm on the initial depth true value map to obtain the point cloud true value map includes: Dividing the depth values ​​of the initial depth truth image into N depth intervals according to preset intervals; For the nth depth interval, assign the corresponding depth value to the point cloud data in the depth interval; For the (n+1)th depth interval, compare the point cloud data in the depth interval with the point cloud data in the (n)th depth interval. If the point cloud data in the depth interval has the same depth value as the point cloud data in the (n)th depth interval, filter out the pixels in the (n+1)th depth interval corresponding to the depth value to obtain the point cloud ground truth map. Here, n traverses from 1 to N-1, and the depth value of the n+1th depth interval is greater than the depth value of the nth depth interval.

4. The depth estimation method based on surround view camera according to claim 1, characterized in that The obtaining of a photometric error of a pixel of a predicted depth map between the current frame image sample and an adjacent frame image sample includes: Obtaining a posture matrix from the adjacent frame image samples to the current frame image samples; Projecting the predicted depth map corresponding to the adjacent frame image samples onto the predicted depth map corresponding to the current frame image samples according to the posture matrix; Calculating an overlapping area on a predicted depth map corresponding to the current frame image sample according to the projection; The photometric errors of the pixels in the overlapping region are calculated.

5. The depth estimation method based on surround view camera according to claim 4, characterized in that: The adjacent frame image samples are the previous frame image sample and the next frame image sample of the current frame image sample.

6. The depth estimation method based on surround view camera according to claim 5, characterized in that: Calculating the overlapping area on the predicted depth map corresponding to the current frame image sample, including: If a pixel exists in both the predicted depth map corresponding to the previous frame image sample and the predicted depth map corresponding to the next frame image sample, but only appears once in the predicted depth map corresponding to the current frame image sample, then the minimum pixel value of the pixel in the predicted depth map corresponding to the previous frame image sample and the predicted depth map corresponding to the next frame image sample shall be used as the value of its corresponding pixel in the predicted depth map corresponding to the current frame image sample.

7. The depth estimation method based on surround view camera according to claim 4, characterized in that: The method further comprises: According to the speed calibration file, estimating the position of the predicted depth map corresponding to the adjacent frame image samples in the predicted depth map corresponding to the current frame image samples; The projection is compensated according to the estimation result.

8. A control device comprising at least one processor and at least one storage device, wherein the storage device is adapted to store a plurality of program codes, wherein: The program code is suitable for being loaded and run by the processor to execute the depth estimation method based on a surround view camera according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the depth estimation method based on a surround view camera according to any one of claims 1 to 7.

10. A vehicle, characterized in that: The vehicle comprises a surround view camera and the control device of claim 8 .

Images