Method and device for determining depth estimation loss of monocular vision image

Abstract

The embodiment of the invention discloses a method and a device for determining depth estimation loss of a monocular vision image. The method comprises the following steps: acquiring an original image, a true value marking depth image, a prediction depth image, a first mask image of a road, a second mask image of a lane line, and third and fourth mask images of a vertical object; determining a road depth estimation image according to the first mask image and the predicted depth image, determining a road depth calibration image according to the first mask image and the true value marking depth image, and determining a vertical object depth estimation image according to the third mask image and the predicted depth image; determining a total loss value of road depth estimation based on the road depth estimation image, the second mask image and the road depth calibration image; determining the total loss value of the vertical object depth estimation according to the difference between adjacent pixel points in the vertical object depth estimation image and the fourth mask image; and calculating the total loss value of the depth estimation network model according to the two total loss values.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for determining depth estimation loss of a monocular vision image. Background Art

[0002] With the development of deep learning technology, intelligent driving technology has also been greatly improved. There are many solutions for depth estimation of intelligent driving, one of which is to directly predict the depth map based on the monocular camera through deep learning technology.

[0003] In the process of iteratively training the depth estimation network model based on the training set, usually the road environment image in the training set is firstly estimated based on the current depth estimation network model to obtain a predicted depth image, and then the mean difference between the preset depth image and the corresponding pixel points in the true value annotated depth image is calculated as the loss value of the current training, which is used to adjust the model parameters of the current depth estimation network model, and then the next iterative training is performed until the loss value of the current training reaches the convergence condition, and the training is stopped to obtain the final required depth estimation network model. Although the accuracy of depth estimation can be further improved by increasing the number of training samples and the number of scenes to which the training samples belong, inaccuracies still occur in practical applications. Summary of the invention

[0004] The present invention provides a method and device for determining a monocular vision image depth estimation loss, which can improve the depth estimation accuracy of a depth estimation network model by improving the calculation accuracy of the monocular vision image depth estimation loss.

[0005] The specific technical solutions are as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for determining a monocular visual image depth estimation loss, the method comprising:

[0007] Acquire an original road environment image, a true value annotated depth image corresponding to the original road image, a predicted depth image obtained after depth estimation of the original road image based on a depth estimation network model, and a first mask image of the road, a second mask image of each lane line, a third mask image of each vertical object, and a fourth mask image of each vertical object obtained after instance segmentation of the original road environment image based on an image instance segmentation model, wherein the third mask image is a complete mask image of each vertical object, and the fourth mask image is a mask image of a straight line segment in the vertical object;

[0008] Determine a road depth estimation image according to the first mask image and the predicted depth image, determine a road depth calibration image according to the first mask image and the true value annotated depth image, and determine a vertical object depth estimation image corresponding to each third mask image according to each third mask image and the predicted depth image;

[0009] Determine a total loss value of road depth estimation based on a mean value of differences between adjacent pixels in the road depth estimation image in the direction of the transverse coordinate axis, a difference between each pixel on each lane line in the road depth estimation image filtered by the second mask image and each adjacent pixel in all its adjacent pixels, and a difference between the road depth estimation image and the road depth calibration image;

[0010] Determining a total loss value of vertical object depth estimation according to a difference between each pixel point on each straight line segment and each adjacent pixel point of all adjacent pixel points in the vertical object depth estimation image filtered by the fourth mask image;

[0011] The total loss value of the depth estimation of the depth estimation network model is calculated according to the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation.

[0012] In a possible implementation, based on the average of the differences between adjacent pixels in the road depth estimation image in the direction of the transverse coordinate axis, the difference between each pixel on each lane line in the road depth estimation image filtered by the second mask image and each of its adjacent pixels, and the difference between the road depth estimation image and the road depth calibration image, determining the total loss value of the road depth estimation includes:

[0013] Determining a first loss value based on a mean value of differences between adjacent pixels in the road depth estimation image in a direction of a transverse coordinate axis;

[0014] Determine a second loss value by using a difference between each pixel point on each lane line in the road depth estimation image filtered by the second mask image and each adjacent pixel point in all adjacent pixel points;

[0015] determining a third loss value according to a difference between the road depth estimation image and the road depth calibration image;

[0016] A total loss value of road depth estimation is calculated according to the first loss value, the second loss value and the third loss value.

[0017] In a possible implementation, determining the first loss value based on the mean difference between adjacent pixels in the road depth estimation image in the direction of the transverse coordinate axis includes:

[0018] Traversing each pixel in the road depth estimation image;

[0019] Get the first pixel point and the second pixel point adjacent to the currently traversed pixel point in the horizontal coordinate axis direction;

[0020] Calculating the Euclidean distances between the currently traversed pixel point and the first pixel point and the second pixel point respectively, and calculating the sum of the two Euclidean distances as the difference value corresponding to the currently traversed pixel point;

[0021] An average of the difference values ​​corresponding to all pixels in the road depth estimation image is calculated as the first loss value.

[0022] In a possible implementation, determining the second loss value by the difference between each pixel point on each lane line in the road depth estimation image filtered by the second mask image and each adjacent pixel point in all adjacent pixel points thereof includes:

[0023] determining a longitudinal loss on each lane line according to a sum of Euclidean distances between each pixel point on each lane line and each adjacent pixel point in all adjacent pixel points in the road depth estimation image filtered by the second mask image;

[0024] The average of the longitudinal losses of all lane lines is calculated as the second loss value.

[0025] In a possible implementation, calculating a total loss value of road depth estimation according to the first loss value, the second loss value, and the third loss value includes:

[0026] The total loss value of the road depth estimation is calculated by weighted summing the first loss value, the second loss value and the third loss value.

[0027] In a possible implementation, determining a total loss value of vertical object depth estimation according to a difference between each pixel point on each straight line segment in the vertical object depth estimation image filtered by the fourth mask image and each adjacent pixel point of all adjacent pixels thereof includes:

[0028] For each of the vertical object depth estimation images, determine a loss value of each straight line segment according to the sum of the Euclidean distances between each pixel point on each straight line segment and each adjacent pixel point of all adjacent pixel points in the vertical object depth estimation image filtered by the fourth mask image;

[0029] The average loss value of all straight line segments of all vertical objects is calculated as the total loss value of the depth estimation of the vertical object.

[0030] In a possible implementation, calculating the total loss value of depth estimation of the depth estimation network model according to the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation includes:

[0031] The total loss value of the depth estimation of the depth estimation network model is obtained by performing weighted summation on the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation.

[0032] In a second aspect, an embodiment of the present invention provides a device for determining a monocular visual image depth estimation loss, the device comprising:

[0033] an acquisition unit, configured to acquire an original road environment image, a true value annotated depth image corresponding to the original road image, a predicted depth image obtained by performing depth estimation on the original road image based on a depth estimation network model, and a first mask image of the road obtained by performing instance segmentation on the original road environment image based on an image instance segmentation model, a second mask image of each lane line, a third mask image of each vertical object, and a fourth mask image of each vertical object, wherein the third mask image is a complete mask image of each vertical object, and the fourth mask image is a mask image of a straight line segment in the vertical object;

[0034] a first determining unit, configured to determine a road depth estimation image according to the first mask image and the predicted depth image, determine a road depth calibration image according to the first mask image and the true value annotated depth image, and determine a vertical object depth estimation image corresponding to each of the third mask images according to each of the third mask images and the predicted depth image;

[0035] a second determination unit, configured to determine a total loss value of the road depth estimation based on a mean value of differences between adjacent pixels in the road depth estimation image in the direction of the transverse coordinate axis, a difference between each pixel on each lane line in the road depth estimation image screened by the second mask image and each adjacent pixel of all its adjacent pixels, and a difference between the road depth estimation image and the road depth calibration image;

[0036] a third determining unit, configured to determine a total loss value of vertical object depth estimation according to a difference between each pixel point on each straight line segment in the vertical object depth estimation image screened by the fourth mask image and each adjacent pixel point of all adjacent pixels thereto;

[0037] A calculation unit is used to calculate a total loss value of depth estimation of a depth estimation network model according to the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation.

[0038] In a possible implementation manner, the second determining unit includes:

[0039] A first determination module, configured to determine a first loss value based on a mean value of differences between adjacent pixels in the road depth estimation image in a direction of a transverse coordinate axis;

[0040] A second determination module, configured to determine a second loss value by using a difference between each pixel point on each lane line in the road depth estimation image filtered by the second mask image and each adjacent pixel point in all adjacent pixel points;

[0041] A third determination module, configured to determine a third loss value according to a difference between the road depth estimation image and the road depth calibration image;

[0042] The first calculation module is used to calculate a total loss value of road depth estimation according to the first loss value, the second loss value and the third loss value.

[0043] In a possible implementation manner, the first determining module is used to:

[0044] Traversing each pixel in the road depth estimation image;

[0045] Get the first pixel point and the second pixel point adjacent to the currently traversed pixel point in the horizontal coordinate axis direction;

[0046] Calculating the Euclidean distances between the currently traversed pixel point and the first pixel point and the second pixel point respectively, and calculating the sum of the two Euclidean distances as the difference value corresponding to the currently traversed pixel point;

[0047] An average of the difference values ​​corresponding to all pixels in the road depth estimation image is calculated as the first loss value.

[0048] In a possible implementation manner, the second determining module is used to:

[0049] determining a longitudinal loss on each lane line according to a sum of Euclidean distances between each pixel point on each lane line and each adjacent pixel point in all adjacent pixel points in the road depth estimation image filtered by the second mask image;

[0050] The average of the longitudinal losses of all lane lines is calculated as the second loss value.

[0051] In a possible implementation manner, the first calculation module is configured to calculate a total loss value of the road depth estimation by performing a weighted summation on the first loss value, the second loss value, and the third loss value.

[0052] In a possible implementation manner, the third determining unit includes:

[0053] a fourth determination module, configured to determine, for each of the vertical object depth estimation images, a loss value of each straight line segment according to the sum of the Euclidean distances between each pixel point on each straight line segment and each of all its adjacent pixel points in the vertical object depth estimation image filtered by the fourth mask image;

[0054] The second calculation module is used to calculate the average loss value of all straight line segments of all vertical objects as the total loss value of the vertical object depth estimation.

[0055] In a possible implementation, the calculation unit is used to obtain the total loss value of the depth estimation of the depth estimation network model by performing a weighted sum of the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation.

[0056] In a third aspect, an embodiment of the present invention provides a storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method described in the first aspect.

[0057] In a fourth aspect, an embodiment of the present invention provides an electronic device, including:

[0058] one or more processors;

[0059] a storage device for storing one or more programs,

[0060] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect.

[0061] From the above content, it can be seen that the method and device for determining the depth estimation loss of a monocular vision image provided by an embodiment of the present invention, when calculating the depth estimation loss value of the depth estimation network model, firstly segment the instances in the original road environment image into roads, lane lines and vertical objects, obtain a first mask image of the road, a second mask image of each lane line, a third mask image of each vertical object and a fourth mask image of each vertical object, and determine the road depth estimation image based on the first mask image and the predicted depth image, determine the road depth calibration image based on the first mask image and the true value labeled depth image, and determine the vertical object depth estimation image corresponding to each third mask image based on each third mask image and the predicted depth image, and then determine the vertical object depth estimation image corresponding to each third mask image from the loss of the road on the horizontal coordinate axis, the loss on the longitudinal coordinate axis (including the loss of the lane line) , the overall road loss between the road depth estimation image and the road depth calibration image, the three are comprehensively determined to determine the total loss value of the road depth estimation, and the total loss value of the vertical object depth estimation is determined according to the straight line segment loss of all vertical objects, and finally, the total loss value of the depth estimation of the depth estimation network model is comprehensively determined according to the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation. Therefore, compared with the rough calculation method of directly taking the difference between the predicted depth image and the true value annotated depth image as the required depth estimation loss value, the embodiment of the present application first decouples the instance in the image into horizontal and vertical directions to calculate the loss separately, and then merges these two types of losses, and when calculating these two types of losses, it also introduces structured information related to the object, thereby greatly improving the accuracy of the depth estimation loss calculation. In addition, when calculating the difference between each pixel and its adjacent pixel points, the total loss between the pixel point and each adjacent pixel point is represented by the sum of the Euclidean distances between the pixel point and each adjacent pixel point, which can further improve the accuracy of the pixel point loss calculation. After obtaining the first loss value, the second loss value and the third loss value, the total loss value of the road depth estimation is obtained by performing a weighted summation of the three, which can improve the calculation accuracy of the total loss value of the road depth estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work.

[0063] Figure 1 A schematic flow chart of a method for determining depth estimation loss of a monocular visual image provided by an embodiment of the present invention;

[0064] Figure 2A block diagram of a device for determining depth estimation loss of a monocular visual image provided by an embodiment of the present invention;

[0065] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0068] Figure 1 A flow chart of a method for determining a monocular visual image depth estimation loss provided by an embodiment of the present invention. The method may include the following steps:

[0069] S110: Acquire an original road environment image, a true value annotated depth image corresponding to the original road image, a predicted depth image obtained after performing depth estimation on the original road image based on a depth estimation network model, and a first mask image of the road obtained after performing instance segmentation on the original road environment image based on an image instance segmentation model, a second mask image of each lane line, a third mask image of each vertical object, and a fourth mask image of each vertical object.

[0070] The original road environment image is a road environment image captured by a monocular camera on a vehicle during driving, including roads and vertical objects. Vertical objects refer to objects extending in a direction perpendicular to the ground, including vehicles, pedestrians, traffic lights, road signs, buildings, etc. The true value annotated depth image corresponding to the original road image refers to an image annotated with the true depth value of each pixel in the original road image, which can usually be annotated manually.

[0071] The depth estimation network model refers to the currently trained depth estimation network model. After the currently trained depth estimation network model outputs a predicted depth image, the total loss value of the depth estimation of this training can be calculated by the loss calculation method provided in the embodiment of the present application, so as to adjust the model parameters of the depth estimation network model through the total loss value of the depth estimation and perform the next iterative training.

[0072] The image instance segmentation model refers to further refining and classifying the foreground and background of the target based on the detection of the target to achieve pixel-level target separation. Therefore, the electronic device can perform instance segmentation on the original road environment image based on the image instance segmentation model to obtain the mask image of the required instance, including the first mask image (i.e., the mask image about the road), the second mask image (i.e., the mask image of a single lane line), the third mask image (i.e., the complete mask image of a single vertical object), and the fourth mask image (i.e., the mask image of a single straight line segment in a single vertical object). The mask image refers to a mask image, in which all elements are composed of 1 and 0, 1 indicates that the pixel at this position belongs to the pixel of the current instance, and 0 indicates that the pixel at this position does not belong to the pixel of the current instance.

[0073] S120: Determine a road depth estimation image based on the first mask image and the predicted depth image, determine a road depth calibration image based on the first mask image and the true value annotated depth image, and determine a vertical object depth estimation image corresponding to each third mask image based on each third mask image and the predicted depth image.

[0074] After obtaining the first mask image, the predicted depth image, the true value labeled depth image and the third mask image, the first mask image and the predicted depth image can be ANDed to obtain a road depth estimation image, the first mask image and the true value labeled depth image can be ANDed to obtain a road depth calibration image, and each third mask image and the predicted depth image can be ANDed to obtain a vertical object depth estimation image corresponding to each third mask image.

[0075] S130: Determine the total loss value of the road depth estimation based on the average difference of each adjacent pixel point in the road depth estimation image in the direction of the horizontal coordinate axis, the difference between each pixel point on each lane line in the road depth estimation image filtered by the second mask image and each adjacent pixel point of all its adjacent pixel points, and the difference between the road depth estimation image and the road depth calibration image.

[0076] Among them, the average value of the difference between each adjacent pixel point in the road depth estimation image in the horizontal coordinate axis direction can be understood as the loss of the road in the horizontal coordinate axis direction; the difference between each pixel point on each lane line in the road depth estimation image screened by the second mask image and each adjacent pixel point of all its adjacent pixels can be understood as the loss of the road in the longitudinal coordinate axis direction; the difference between the road depth estimation image and the road depth calibration image is similar to the loss calculation method proposed in the background technology, which is the loss predicted from the road image as a whole. Therefore, the embodiment of the present application obtains the total loss value of the road depth estimation by comprehensively calculating these three losses.

[0077] The following describes in detail the calculation process of the total loss value of the road depth estimation through steps A1-A4:

[0078] A1. Determine a first loss value based on the mean difference between adjacent pixels in the road depth estimation image in the direction of the horizontal coordinate axis.

[0079] The specific calculation method of this step includes: traversing each pixel point in the road depth estimation image; obtaining the first pixel point and the second pixel point adjacent to the currently traversed pixel point in the direction of the horizontal coordinate axis; calculating the Euclidean distance between the currently traversed pixel point and the first pixel point and the second pixel point respectively, and calculating the sum of the two Euclidean distances as the difference value corresponding to the currently traversed pixel point; calculating the average of the difference values ​​corresponding to all pixel points in the road depth estimation image as the first loss value.

[0080] The above calculation process is expressed as follows:

[0081]

[0082] Among them, loss1 represents the first loss value, g represents the number of all pixels involved in the calculation, h represents the height of the road depth estimation image, w represents the width of the road depth estimation image, and mpI i,j Represents the pixel value of the pixel with coordinates (i, j) in the road depth estimation image, mpI i,j+1 Represents the pixel value of the pixel with coordinates (i, j+1) in the road depth estimation image, mpI i,j-1 represents the pixel value of the pixel with coordinates (i, j-1) in the road depth estimation image, and | |2 represents the Euclidean distance, such as |mpI i,j -mpI i,j+1 |2 represents the Euclidean distance between the pixel with coordinates (i, j) and the pixel with coordinates (i, j+1).

[0083] A2. Determine a second loss value by estimating the difference between each pixel point on each lane line in the road depth estimation image filtered by the second mask image and each adjacent pixel point in all adjacent pixels.

[0084] Specifically, this step can first determine the longitudinal loss on each lane line based on the sum of the Euclidean distances between each pixel point on each lane line in the road depth estimation image filtered by the second mask image and each of its adjacent pixel points; and then calculate the mean of the longitudinal losses of all lane lines as the second loss value.

[0085] The above calculation process is expressed as follows:

[0086] The calculation formula for the longitudinal loss of a single lane line includes:

[0087]

[0088] Among them, lossl represents the longitudinal loss of a single lane line, z represents the number of all pixels involved in the calculation, (x1, y1) and (x2, y2) represent the starting point and end point coordinates of a single lane line, which can be the starting point and end point of the entire lane line contained in the image, or the starting point and end point of a part of the entire lane line contained in the image. For example, the starting point remains unchanged, and the end point is taken at 1 / 3 of the entire lane line, so as to ensure that the line segment involved in the calculation is a straight line segment as much as possible, thereby improving the accuracy of the longitudinal loss of the lane line. mIla represents the second mask image, Indicates phase and, mpI i-1,j Represents the pixel value of the pixel with coordinates (i-1, j) in the road depth estimation image, mpI i+1,j Represents the pixel value of the pixel with coordinates (i+1,j) in the road depth estimation image.

[0089] The calculation formula of the mean loss of all lane lines, that is, the second loss value, includes:

[0090]

[0091] Among them, loss2 represents the second loss value, k1 represents the number of all lane lines, and lossl n Represents the longitudinal loss of the nth lane line.

[0092] It should be added that because the left and right sides of the lane lines are usually flat areas and their extension properties are similar to those of the lane lines, in order to expand to the constraints of lane line loss, the lane lines are expanded left and right, that is, the left boundary of each lane line pixel value in the mLa image is extended to the left by w1, and the right boundary is extended to the right by w1, that is, the lane line width is increased by 2w1.

[0093] A3. Determine a third loss value according to the difference between the road depth estimation image and the road depth calibration image.

[0094] The calculation formula of the third loss value may include:

[0095]

[0096] Among them, loss3 represents the third loss, h represents the height of the road depth estimation image and the road depth calibration image, w represents the width of the road depth estimation image and the road depth calibration image, and mpI i,j Represents the pixel value of the pixel with coordinates (i, j) in the road depth estimation image, mdI i,j Represents the pixel value of the pixel with coordinates (i, j) in the road depth calibration image.

[0097] A4. Calculate a total loss value of the road depth estimation according to the first loss value, the second loss value and the third loss value.

[0098] The total loss value of the road depth estimation is calculated by weighted summing the first loss value, the second loss value and the third loss value. The calculation formula can be expressed as:

[0099] roadloss=α·loss1+β·loss2+(1-α-β)·loss3

[0100] Among them, roadloss represents the total loss value of road depth estimation, α and β represent weight coefficients. The weight coefficients of the first loss value, the second loss value and the third loss value can be determined based on actual experience.

[0101] It should be added that, in actual applications, after the first loss value, the second loss value and the third loss value, the total loss value of the road depth estimation can also be calculated by other means. For example, the maximum value of the three is taken as the total loss value of the road depth estimation, or the total loss value of the road depth estimation can be obtained by performing other mathematical operations on the three.

[0102] S140: Determine a total loss value of the vertical object depth estimation according to a difference between each pixel point on each straight line segment and each adjacent pixel point in all adjacent pixel points in the vertical object depth estimation image filtered by the fourth mask image.

[0103] The vertical object depth information extension characteristic is related to its framework and physical structure. Its obvious feature is the straight line segments detected on the object image. These straight line segments can represent the physical structure of the object to a certain extent. Therefore, the loss constraint in the embodiment of the present invention also uses the straight line segments as reference features for calculation. It should be noted that in the application of the embodiment of the present invention, the straight line detection can be fragmented, but false alarms need to be avoided. Therefore, the LSD straight line detection algorithm can be used to detect the straight line segments in the third mask image to obtain the fourth mask image.

[0104] Specifically, this step can first determine the loss value of each straight line segment for each vertical object depth estimation image according to the sum of the Euclidean distances between each pixel point on each straight line segment in the vertical object depth estimation image filtered by the fourth mask image and each adjacent pixel point of all its adjacent pixels; and then calculate the average loss value of all straight line segments of all vertical objects as the total loss value of vertical object depth estimation.

[0105] The above calculation process is expressed as follows:

[0106] The loss value calculation formula for a single straight line segment includes:

[0107]

[0108] Among them, lossv represents the loss value of a single straight line segment, z' represents the number of all pixels involved in the calculation, (x3, y3) and (x4, y4) represent the starting point and end point coordinates of a single straight line segment, which can be the starting point and end point of a single straight line segment, mIl represents the fourth mask image, and mlI i,j Represents the pixel value of the pixel with coordinates (i, j) in the vertical object depth estimation image, mlI i,j+1 Represents the pixel value of the pixel with coordinates (i, j+1) in the vertical object depth estimation image, mlI i,j-1 Represents the pixel value of the pixel with coordinate (i, j-1) in the vertical object depth estimation image, mlI i-1,j Represents the pixel value of the pixel with coordinates (i-1, j) in the vertical object depth estimation image, mlI i+1,j Represents the pixel value of the pixel with coordinates (i+1,j) in the vertical object depth estimation image.

[0109] The calculation formula for the average loss value of all straight line segments of all vertical objects, that is, the total loss value of vertical object depth estimation, includes:

[0110]

[0111] Among them, lossver represents the total loss value of vertical object depth estimation, k2 is the number of all straight line segments of all vertical objects, and lossvm Represents the loss value of the mth straight line segment.

[0112] S150: Calculate the total loss value of depth estimation of the depth estimation network model according to the total loss value of road depth estimation and the total loss value of vertical object depth estimation.

[0113] After obtaining the total loss value of road depth estimation and the total loss value of vertical object depth estimation, the largest loss value between the two can be taken as the total loss value of depth estimation of the depth estimation network model, or the result of weighted summation of the two can be taken as the total loss value of depth estimation of the depth estimation network model, or other mathematical operations can be performed on the two to obtain the total loss value of depth estimation of the depth estimation network model.

[0114] When the two are weighted and summed, the calculation formula can be expressed as:

[0115] loss=γ·lossroad+(1-γ)·lossver

[0116] Among them, loss represents the total loss value of depth estimation of the depth estimation network model, and γ represents the weight, which is used to adjust the loss ratio of horizontal and vertical objects.

[0117] The method for determining the depth estimation loss of a monocular vision image provided by an embodiment of the present invention, when calculating the depth estimation loss value of a depth estimation network model, firstly divides the instances in the original road environment image into roads, lane lines and vertical objects, obtains a first mask image of the road, a second mask image of each lane line, a third mask image of each vertical object and a fourth mask image of each of the vertical objects, and determines a road depth estimation image based on the first mask image and the predicted depth image, determines a road depth calibration image based on the first mask image and the true value labeled depth image, and determines a vertical object depth estimation image corresponding to each third mask image based on each third mask image and the predicted depth image, and then calculates the depth estimation loss value of the road on the horizontal coordinate axis, the vertical coordinate axis (including the loss of the lane line), and the road depth. The overall road loss between the estimated image and the road depth calibration image, the three are comprehensively determined to determine the total loss value of the road depth estimation, and the total loss value of the vertical object depth estimation is determined according to the straight line segment loss of all vertical objects, and finally, the total loss value of the depth estimation of the depth estimation network model is comprehensively determined according to the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation. Therefore, compared with the rough calculation method of directly taking the difference between the predicted depth image and the true value annotated depth image as the required depth estimation loss value, the embodiment of the present application first decouples the instance in the image into horizontal and vertical directions to calculate the loss separately, and then merges these two types of losses, and when calculating these two types of losses, it also introduces structured information related to the object, thereby greatly improving the accuracy of the depth estimation loss calculation. In addition, when calculating the difference between each pixel and its adjacent pixel points, the total loss between the pixel point and each adjacent pixel point is represented by the sum of the Euclidean distances between the pixel point and each adjacent pixel point, which can further improve the accuracy of the pixel point loss calculation. After obtaining the first loss value, the second loss value and the third loss value, the total loss value of the road depth estimation is obtained by performing a weighted summation of the three, which can improve the calculation accuracy of the total loss value of the road depth estimation.

[0118] Corresponding to the above method embodiment, the embodiment of the present invention provides a device for determining the depth estimation loss of a monocular visual image, such as Figure 2 As shown, the device comprises:

[0119] An acquisition unit 210 is used to acquire an original road environment image, a true value annotated depth image corresponding to the original road image, a predicted depth image obtained after depth estimation of the original road image based on a depth estimation network model, and a first mask image of the road obtained after instance segmentation of the original road environment image based on an image instance segmentation model, a second mask image of each lane line, a third mask image of each vertical object, and a fourth mask image of each vertical object, wherein the third mask image is a complete mask image of each vertical object, and the fourth mask image is a mask image of a straight line segment in the vertical object;

[0120] A first determining unit 220 is configured to determine a road depth estimation image according to the first mask image and the predicted depth image, determine a road depth calibration image according to the first mask image and the true value annotated depth image, and determine a vertical object depth estimation image corresponding to each of the third mask images according to each of the third mask images and the predicted depth image;

[0121] A second determining unit 230 is used to determine a total loss value of the road depth estimation based on a mean value of differences between adjacent pixels in the road depth estimation image in the direction of the transverse coordinate axis, a difference between each pixel on each lane line in the road depth estimation image filtered by the second mask image and each adjacent pixel of all its adjacent pixels, and a difference between the road depth estimation image and the road depth calibration image;

[0122] a third determining unit 240, configured to determine a total loss value of vertical object depth estimation according to a difference between each pixel point on each straight line segment in the vertical object depth estimation image filtered by the fourth mask image and each adjacent pixel point of all adjacent pixels thereto;

[0123] The calculation unit 250 is used to calculate the total loss value of the depth estimation of the depth estimation network model according to the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation.

[0124] In a possible implementation manner, the second determining unit 230 includes:

[0125] A first determination module, configured to determine a first loss value based on a mean value of differences between adjacent pixels in the road depth estimation image in a direction of a transverse coordinate axis;

[0126] A second determination module, configured to determine a second loss value by using a difference between each pixel point on each lane line in the road depth estimation image filtered by the second mask image and each adjacent pixel point in all adjacent pixel points;

[0127] A third determination module, configured to determine a third loss value according to a difference between the road depth estimation image and the road depth calibration image;

[0128] The first calculation module is used to calculate a total loss value of road depth estimation according to the first loss value, the second loss value and the third loss value.

[0129] In a possible implementation manner, the first determining module is used to:

[0130] Traversing each pixel in the road depth estimation image;

[0131] Get the first pixel point and the second pixel point adjacent to the currently traversed pixel point in the horizontal coordinate axis direction;

[0132] Calculating the Euclidean distances between the currently traversed pixel point and the first pixel point and the second pixel point respectively, and calculating the sum of the two Euclidean distances as the difference value corresponding to the currently traversed pixel point;

[0133] An average of the difference values ​​corresponding to all pixels in the road depth estimation image is calculated as the first loss value.

[0134] In a possible implementation manner, the second determining module is used to:

[0135] determining a longitudinal loss on each lane line according to a sum of Euclidean distances between each pixel point on each lane line and each adjacent pixel point in all adjacent pixel points in the road depth estimation image filtered by the second mask image;

[0136] The average of the longitudinal losses of all lane lines is calculated as the second loss value.

[0137] In a possible implementation manner, the first calculation module is configured to calculate a total loss value of the road depth estimation by performing a weighted summation on the first loss value, the second loss value, and the third loss value.

[0138] In a possible implementation manner, the third determining unit 240 includes:

[0139] a fourth determination module, configured to determine, for each of the vertical object depth estimation images, a loss value of each straight line segment according to the sum of the Euclidean distances between each pixel point on each straight line segment and each of all its adjacent pixel points in the vertical object depth estimation image filtered by the fourth mask image;

[0140] The second calculation module is used to calculate the average loss value of all straight line segments of all vertical objects as the total loss value of the vertical object depth estimation.

[0141] In a possible implementation, the calculation unit 250 is used to obtain the total loss value of the depth estimation of the depth estimation network model by performing a weighted summation on the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation.

[0142] The device for determining the depth estimation loss of a monocular vision image provided by an embodiment of the present invention, when calculating the depth estimation loss value of a depth estimation network model, first segments the instances in the original road environment image into roads, lane lines, and vertical objects, obtains a first mask image of the road, a second mask image of each lane line, a third mask image of each vertical object, and a fourth mask image of each of the vertical objects, and determines a road depth estimation image based on the first mask image and the predicted depth image, determines a road depth calibration image based on the first mask image and the true value labeled depth image, and determines a vertical object depth estimation image corresponding to each third mask image based on each third mask image and the predicted depth image, and then calculates the depth estimation loss value of the road on the horizontal coordinate axis, the vertical coordinate axis (including the loss of the lane line), and the road depth. The overall road loss between the estimated image and the road depth calibration image, the three are comprehensively determined to determine the total loss value of the road depth estimation, and the total loss value of the vertical object depth estimation is determined according to the straight line segment loss of all vertical objects, and finally, the total loss value of the depth estimation of the depth estimation network model is comprehensively determined according to the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation. Therefore, compared with the rough calculation method of directly taking the difference between the predicted depth image and the true value annotated depth image as the required depth estimation loss value, the embodiment of the present application first decouples the instance in the image into horizontal and vertical directions to calculate the loss separately, and then merges these two types of losses, and when calculating these two types of losses, it also introduces structured information related to the object, thereby greatly improving the accuracy of the depth estimation loss calculation. In addition, when calculating the difference between each pixel and its adjacent pixel points, the total loss between the pixel point and each adjacent pixel point is represented by the sum of the Euclidean distances between the pixel point and each adjacent pixel point, which can further improve the accuracy of the pixel point loss calculation. After obtaining the first loss value, the second loss value and the third loss value, the total loss value of the road depth estimation is obtained by performing a weighted summation of the three, which can improve the calculation accuracy of the total loss value of the road depth estimation.

[0143] Based on the above method embodiment, another embodiment of the present invention provides a storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the above method.

[0144] Based on the above method embodiment, another embodiment of the present invention provides an electronic device, such as Figure 3As shown, including:

[0145] one or more processors;

[0146] a storage device for storing one or more programs,

[0147] When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0148] The above-mentioned system and device embodiments correspond to the method embodiments and have the same technical effects as the method embodiments. For specific descriptions, please refer to the method embodiments. The device embodiments are obtained based on the method embodiments. For specific descriptions, please refer to the method embodiment part, which will not be repeated here. It can be understood by those of ordinary skill in the art that the accompanying drawings are only schematic diagrams of one embodiment, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present invention.

[0149] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining the depth estimation loss of a monocular visual image, characterized in that: The method comprises: Acquire an original road environment image, a true value annotated depth image corresponding to the original road image, a predicted depth image obtained after depth estimation of the original road image based on a depth estimation network model, and a first mask image of the road, a second mask image of each lane line, a third mask image of each vertical object, and a fourth mask image of each vertical object obtained after instance segmentation of the original road environment image based on an image instance segmentation model, wherein the third mask image is a complete mask image of each vertical object, and the fourth mask image is a mask image of a straight line segment in the vertical object; Determine a road depth estimation image according to the first mask image and the predicted depth image, determine a road depth calibration image according to the first mask image and the true value annotated depth image, and determine a vertical object depth estimation image corresponding to each third mask image according to each third mask image and the predicted depth image; Determine a total loss value of road depth estimation based on a mean value of differences between adjacent pixels in the road depth estimation image in the direction of the transverse coordinate axis, a difference between each pixel on each lane line in the road depth estimation image filtered by the second mask image and each adjacent pixel in all its adjacent pixels, and a difference between the road depth estimation image and the road depth calibration image; Determining a total loss value of vertical object depth estimation according to a difference between each pixel point on each straight line segment and each adjacent pixel point of all adjacent pixel points in the vertical object depth estimation image filtered by the fourth mask image; The total loss value of the depth estimation of the depth estimation network model is calculated according to the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation.

2. The method according to claim 1, characterized in that Determining a total loss value of road depth estimation based on a mean value of differences between adjacent pixels in the road depth estimation image in the direction of the transverse coordinate axis, a difference between each pixel on each lane line in the road depth estimation image screened by the second mask image and each adjacent pixel of all its adjacent pixels, and a difference between the road depth estimation image and the road depth calibration image, including: Determining a first loss value based on a mean value of differences between adjacent pixels in the road depth estimation image in a direction of a transverse coordinate axis; Determine a second loss value by using a difference between each pixel point on each lane line in the road depth estimation image filtered by the second mask image and each adjacent pixel point in all adjacent pixel points; determining a third loss value according to a difference between the road depth estimation image and the road depth calibration image; A total loss value of road depth estimation is calculated according to the first loss value, the second loss value and the third loss value.

3. The method according to claim 2, characterized in that Determining a first loss value based on a mean value of differences between adjacent pixels in the road depth estimation image in a direction of a transverse coordinate axis includes: Traversing each pixel in the road depth estimation image; Get the first pixel point and the second pixel point adjacent to the currently traversed pixel point in the horizontal coordinate axis direction; Calculating the Euclidean distances between the currently traversed pixel point and the first pixel point and the second pixel point respectively, and calculating the sum of the two Euclidean distances as the difference value corresponding to the currently traversed pixel point; An average of the difference values ​​corresponding to all pixels in the road depth estimation image is calculated as the first loss value.

4. The method according to claim 2, characterized in that: Determining a second loss value by using a difference between each pixel point on each lane line in the road depth estimation image filtered by the second mask image and each adjacent pixel point in all adjacent pixel points thereof includes: determining a longitudinal loss on each lane line according to a sum of Euclidean distances between each pixel point on each lane line and each of all its adjacent pixel points in the road depth estimation image filtered by the second mask image; The average of the longitudinal losses of all lane lines is calculated as the second loss value.

5. The method according to claim 2, characterized in that: Calculating a total loss value of road depth estimation according to the first loss value, the second loss value, and the third loss value includes: The total loss value of the road depth estimation is calculated by weighted summing the first loss value, the second loss value and the third loss value.

6. The method according to claim 1, characterized in that Determining a total loss value of vertical object depth estimation according to a difference between each pixel point on each straight line segment in the vertical object depth estimation image filtered by the fourth mask image and each adjacent pixel point of all adjacent pixels thereof, comprising: For each of the vertical object depth estimation images, determine a loss value of each straight line segment according to the sum of the Euclidean distances between each pixel point on each straight line segment and each adjacent pixel point of all its adjacent pixel points in the vertical object depth estimation image filtered by the fourth mask image; The average loss value of all straight line segments of all vertical objects is calculated as the total loss value of the depth estimation of the vertical object.

7. The method according to any one of claims 1 to 6, characterized in that: Calculating a total loss value of depth estimation of a depth estimation network model according to the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation includes: The total loss value of the depth estimation of the depth estimation network model is obtained by performing weighted summation on the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation.

8. A device for determining the loss of depth estimation of a monocular visual image, characterized in that: The device comprises: an acquisition unit, configured to acquire an original road environment image, a true value annotated depth image corresponding to the original road image, a predicted depth image obtained by performing depth estimation on the original road image based on a depth estimation network model, and a first mask image of the road obtained by performing instance segmentation on the original road environment image based on an image instance segmentation model, a second mask image of each lane line, a third mask image of each vertical object, and a fourth mask image of each vertical object, wherein the third mask image is a complete mask image of each vertical object, and the fourth mask image is a mask image of a straight line segment in the vertical object; a first determining unit, configured to determine a road depth estimation image according to the first mask image and the predicted depth image, determine a road depth calibration image according to the first mask image and the true value annotated depth image, and determine a vertical object depth estimation image corresponding to each of the third mask images according to each of the third mask images and the predicted depth image; a second determination unit, configured to determine a total loss value of the road depth estimation based on a mean value of differences between adjacent pixels in the road depth estimation image in the direction of the transverse coordinate axis, a difference between each pixel on each lane line in the road depth estimation image screened by the second mask image and each adjacent pixel of all its adjacent pixels, and a difference between the road depth estimation image and the road depth calibration image; a third determining unit, configured to determine a total loss value of vertical object depth estimation according to a difference between each pixel point on each straight line segment in the vertical object depth estimation image screened by the fourth mask image and each adjacent pixel point of all adjacent pixels thereto; A calculation unit is used to calculate a total loss value of depth estimation of a depth estimation network model according to the total loss value of the road depth estimation and the total loss value of the vertical object depth estimation.

9. A storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor implements the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 7.

Images