Method and device for self-supervised training of a depth prediction model with moving object tracking

Abstract

The invention relates to a method for managing the operation of a navigation system (2) in a motor vehicle (1) to facilitate the execution of planned routes. The invention also relates to a computer device (100) implementing such a method, as well as a motor vehicle (1) comprising such a device. Figure for the abstract: 1

Description

Title of the invention: Method and device for self-supervised training of a depth prediction model with moving object tracking. Technical field of the invention

[0001] The present invention relates to the field of vision systems for motor vehicles. The invention relates in particular to a method for self-supervised training of a depth prediction model implemented by a neural network associated with a vision system of a motor vehicle, the vision system comprising a monocular camera. The invention also relates to a device implementing such a method, as well as a motor vehicle comprising such a device. The invention is applicable to motor vehicles such as motor vehicles, in particular cars. Prior art

[0002] Many modern vehicles are equipped with Advanced Driver-Assistance Systems (ADAS). Such ADAS systems are passive and active safety systems designed to eliminate human error in driving all types of vehicles. ADAS systems use advanced technologies to assist the driver while driving and thus improve performance. ADAS systems use a combination of sensor technologies to perceive the environment around a vehicle and then provide information to the driver or act on certain vehicle systems.

[0003] There are several levels of ADAS, such as reversing cameras and blind spot sensors, lane departure warning systems, adaptive cruise control or automatic parking systems.

[0004] Vehicle-mounted ADAS systems are powered by data obtained from one or more on-board sensors such as, for example, cameras. These cameras make it possible, in particular, to detect and locate other road users or any obstacles present around a vehicle in order, for example, to adapt the vehicle's lighting according to the presence of other users, to automatically regulate the vehicle's speed, or to activate the braking system in the event of a risk of impact with an object.

[0005] The position of another user or of an obstacle is, for example, determined by a vision system comprising a prediction model of a depth associated with a pixel or of a distance separating the vision system from an object in a scene three-dimensional. Such a model is generally trained using images that must be acquired and processed beforehand to allow the model to be trained. This process of acquiring and processing training images is, however, costly and time-consuming.

[0006] Self-supervised training eliminates this need for training a depth prediction model for a monocular vision system. Such a model typically performs an image reconstruction task in which it reconstructs the current image from a previous and / or subsequent image. To accomplish this, another model predicts the camera's movement.

[0007] However, known methodologies impose a limited number of moving objects (e.g., pedestrians, vehicles) in the images acquired by the monocular vision system's camera. This is because the model predicting camera movements is unable to differentiate between the objects' own motion and the motion caused by the vehicle's movement. While this generally does not hinder the training of a depth prediction model when moving objects occupy a small portion of the acquired images, it significantly reduces the model's predictive capacity if numerous moving objects are present in the images. Summary of the invention

[0008] The invention aims to solve this problem. In particular, it aims to provide a solution for training a depth prediction model from images acquired by a monocular vision system of a motor vehicle in the presence of numerous moving objects. Through this means, the invention aims to maximize the depth prediction efficiency of a monocular vision system of a motor vehicle and thus improve the safety of driver assistance features that rely on the use of such a system.

[0009] In order to achieve these goals, the invention relates, according to a first aspect, to a self-supervised training method for a depth prediction model implemented by a neural network associated with a vision system of a motor vehicle, the vision system comprising a monocular camera, the method comprising the steps of: i. obtaining a first image acquired by the camera at a first instant and a second image acquired by the camera at a second instant prior to the first instant; ii. determination of a first depth map for the first image, a second depth map for the second image and an estimate of the vehicle's displacement between the camera acquisition of the first image and the camera acquisition of the second image; iii. processing of the first image and the second image in order to determine a two-dimensional bounding box for each moving object present in the first image, a two-dimensional bounding box for each moving object present in the second image as well as a two-dimensional segmentation identifying in each image the pixels relating to each moving object; iv. determination, from the first depth map, the second depth map and the moving objects determined by the image segmentation carried out during step iii), of a three-dimensional bounding box for each moving object present in the first image and a three-dimensional bounding box for each moving object present in the second image; v. determination of associations between the two-dimensional bounding boxes of the moving objects present in the first image and the two-dimensional bounding boxes of the moving objects present in the second image determined during step iii) in order to deduce associations between the three-dimensional bounding boxes of the moving objects present in the first image and the three-dimensional bounding boxes of the moving objects present in the second image determined during step iv); vi. determination, based on the estimation of the vehicle's displacement between the acquisition by the camera of the first image and the acquisition by the camera of the second image, of a first coordinate transformation allowing the coordinates in a three-dimensional space of the points relative to the static environment of the vehicle in the first image to be obtained from the coordinates in a three-dimensional space of the points relative to the static environment of the vehicle in the second image; vii. determination, for each moving object present in the second image, based on the estimated vehicle displacement between the camera acquisition of the first image and the camera acquisition of the second image, the associations between the three-dimensional bounding boxes of the moving objects in each image determined during step v) and the movement of the three-dimensional bounding boxes determined for each moving object, of a second coordinate transformation allowing to obtain the coordinates in a three-dimensional space of the points relative to a moving object present in the second image from the coordinates in a three-dimensional space of the points relative to an associated moving object present in the first image; viii. determination from the coordinates in a three-dimensional space of the points relative to the static environment of the vehicle in the second image and from the coordinates in a three-dimensional space of the points relative to each moving object in the second image, of an image reconstructed from the second image; ix. determination of a first loss function establishing a first photometric error as a function of the first image, the reconstructed image and a measure of a structural similarity index and a second photometric error as a function of the second image, the reconstructed image and a measure of a structural similarity index; x. minimization of the first photometric error and the second photometric error; xi. determination of a second loss function establishing sensitivity to object edges as a function of the dimensions of the second image and a first value of the order of a smoothing gradient; and xii. training of the depth prediction model by adding the two loss functions.

[0010] According to one variant, step ii) can be carried out using a monocular depth prediction model.

[0011] According to another variant, step iii) can be carried out using the Mask- model RCNN or the MAE model.

[0012] According to yet another variant, step iv) is carried out using the PointNet model or the PointMAE model.

[0013] According to yet another variant, step v) can be carried out using an object tracking model from the PCL library.

[0014] According to a second aspect, the invention relates to a self-supervised training device for a depth prediction model of a vision system of a motor vehicle, the vision system comprising a monocular camera, the device comprising an information processing unit, with one or more processors, and a data storage medium, which are configured to implement a method as described above.

[0015] According to a third aspect, the invention relates to a computer program comprising program code instructions for executing the steps of a process as described above when said program is executed by at least one processor.

[0016] According to a fourth aspect, the invention relates to a computer-readable medium on which a program as described above is recorded.

[0017] According to a fifth aspect, the invention relates to a motor vehicle equipped with a vision system comprising a monocular camera, the vehicle comprising a device as described above. Brief description of the figures

[0018] Other features and advantages of the invention will become apparent from an examination of the detailed description below, and the accompanying figures, in which:

[0019] [Fig-1] is a schematic illustration of a motor vehicle according to the invention;

[0020] [Fig.2] is a functional diagram of a device according to the invention; and

[0021] [Fig.3] is a flowchart of the steps of a process according to the invention. Detailed description of the invention

[0022] Figure 1 shows a schematic illustration of a motor vehicle 1 according to the invention. This vehicle comprises a vision system 2 equipped with a single camera 3, in other words, a monocular vision system. The camera 3 is configured and positioned in the vehicle 1 to capture images of the driving environment of the vehicle 1, in other words, to generate images of a three-dimensional scene located near the vehicle 1, preferably located in front of the vehicle 1.

[0023] Advantageously, the vehicle 1 according to the invention also includes a self-supervised training device 100 for a depth prediction model of a vision system of a motor vehicle within the meaning of the present invention, as described below, which implements a self-supervised training method for a depth prediction model implemented by a neural network associated with a vision system of a motor vehicle within the meaning of the present invention, as described below.

[0024] Figure 2 illustrates in more detail the self-driving device 100 supervised by a depth prediction model of a vision system of a motor vehicle according to the invention. It is essentially a computer device, which includes at least one information processing unit 101, comprising one or more processors, a data storage medium 102, on which is recorded in particular a program which includes program code instructions for the execution of the steps of the process according to the invention described below, and an input and output interface 103 enabling the reception and transmission of data.

[0025] Preferably, the device 100 according to the invention is integrated into an independent computer and interacts via its input and output interface 103 and by means of a wired vehicle communication network (e.g. CAN, Ethernet, MOST) - represented on [Fig. 1] by a bidirectional arrow - with the vision system 2. Alternatively, the device 100 according to the invention is an integral part of the vision system 2.

[0026] According to the invention, the elements described above contribute to enabling the implementation of a self-supervised training method for a depth prediction model implemented by a neural network associated with a vision system of a motor vehicle, as described below in relation to [Fig.3].

[0027] According to a first step 301 of the method according to the invention, at least one first image acquired by the camera 3 at a first instant and a second image acquired by the camera 3 at a second instant prior to the first instant are obtained. Thus, according to known methodologies, the reconstruction of the second image from the first image is defined by the following equation:

[0028] [Math.Eql] ^,=^(^1^(^-¾.)])

[0029] With: • Pms the coordinates of a pixel in the second image, • 77 a function to go from homogeneous coordinates to coordinates pixels by removing one dimension from a vector, • K is the intrinsic parameter matrix of the camera for projecting points from three-dimensional space to a two-dimensional image, • 0 is the backprojection of a pixel with its corresponding depth in function the intrinsic matrix of the two-dimensional image (specifically the inverse of the matrix), • Pt the coordinates of a pixel in the first image, • Dfnt the depth value in the first image, and • All the movement of the monocular camera between the two images.

[0030] As mentioned above, this equation requires, however, that there be no moving objects in the scene. It also requires that the vehicle be in motion. Indeed, if the vehicle is stationary, we have = [0], and the depth can then take any value to allow for reconstruction. Moreover, it can be observed that if a significant proportion of images acquired when the vehicle is stationary are used, the depth prediction model will tend to predict a single depth value for the entire image. However, we will consider that the invention does not concern this scenario and we will assume that * [0].

[0031] According to a second step 302 of the method according to the invention, the device 100 according to the invention proceeds by determining a first depth map for the first image, a second depth map for the second image, and an estimate of the vehicle's displacement between the acquisition of the first image by camera 3 and the acquisition of the second image by camera 3. For this purpose, it uses a conventional monocular prediction model, such as monodepth2, HR-depth, or packnet.

[0032] According to a third step 303 of the method according to the invention, the device 100 according to the invention processes the first image and the second image to determine a two-dimensional bounding box for each moving object present in the first image, a two-dimensional bounding box for each moving object present in the second image as well as a two-dimensional segmentation identifying in each image the pixels relating to each moving object.

[0033] For this purpose, it uses, for example, a model developed for supervised machine learning, such as Mask-RCNN. Alternatively, it uses a self-supervised model, for example MAE. The use of a self-supervised model has the advantage of matching the monocular depth prediction model, which is a self-supervised model. However, there is no obligation to use a self-supervised model at this stage of the process. Indeed, the training data for MAE are very different from those used for the monocular model; the former requires images containing only a single object, whereas the latter requires sequential images containing a relatively large number of objects.Furthermore, the two-dimensional detection model that enables the determination of bounding boxes remains fixed during the training of the depth prediction model, and its parameters therefore do not vary according to the loss calculation described below.

[0034] According to a fourth step 304 of the method according to the invention, the device 100 according to the invention proceeds by determining, from the first depth map, the second depth map and the moving objects determined by the segmentation of the images carried out during the previous step, a three-dimensional bounding box for each moving object present in the first image and a three-dimensional bounding box for each moving object present in the second image.

[0035] For this purpose, by using the segmentation of an object as a mask, and applying it to the depth maps, the points in three-dimensional space of each object are determined by the term D in equation (1) mentioned above. To implement this step, the device 100 according to the invention can use a supervised model, for example PointNet, or a self-supervised model, such as PointMAE. However, the annotation of the bounding boxes Since three-dimensional processing is relatively computationally intensive, it may be preferable to use a self-supervised model. In any case, both models mentioned allow for the calculation of a three-dimensional bounding box that provides the position of an object indicated by the center of the bounding box and the object's orientation. The device 100 according to the invention thus determines three-dimensional bounding boxes for all objects detected by segmentation in the two images. As in the previous step, the three-dimensional classification model remains static and therefore does not train during the training of the depth prediction model.

[0036] According to a fifth step 305 of the method according to the invention, the device 100 according to the invention proceeds by determining associations between the two-dimensional bounding boxes of the moving objects present in the first image and the two-dimensional bounding boxes of the moving objects present in the second image, which were determined during the third step 303. It then deduces associations between the three-dimensional bounding boxes of the moving objects present in the first image and the three-dimensional bounding boxes of the moving objects present in the second image, determined during the fourth step 304. Such associations of corresponding objects in the two images can be made using a pre-trained object tracking model from the PCL library. It is preferable here again to use a self-supervised model.As in the previous steps, the object tracking model remains fixed and therefore does not train during the training of the depth model.

[0037] According to a sixth step 306 of the method according to the invention, the device 100 according to the invention determines, based on the estimation of the vehicle's displacement between the acquisition by camera 3 of the first image and the acquisition by camera 3 of the second image, a first coordinate transformation enabling the three-dimensional space coordinates of the points relative to the static environment of the vehicle in the first image to be obtained from the three-dimensional space coordinates of the points relative to the static environment of the vehicle in the second image.Similarly, according to a seventh step 307 of the process, it determines, for each moving object present in the second image, based on the estimate of the vehicle's displacement between the acquisition by camera 3 of the first image and the acquisition by camera of the second image, associations between the three-dimensional bounding boxes of the moving objects determined during the fifth step 305 and the movement of the three-dimensional bounding boxes determined for each moving object, a second coordinate transformation allowing the coordinates in a three-dimensional space of the points relative to a moving object present in the second image from the coordinates in a . three-dimensional space of points relative to an associated moving object present in the first image.

[0038] Indeed, image reconstruction requires transforming the coordinates of the points from the coordinate system of the first image to that of the second image. If the environment is completely static, equation (1) can do this by multiplying the coordinates of the points by the vehicle's motion, T^. For moving objects, a separate transformation specific to each object must be performed. First, the 3D points of an object, as well as the object's three-dimensional bounding box, must be transformed into the coordinate system of the second image in the same way as for the static environment. The difference between the two three-dimensional bounding boxes of the same object in the coordinate system of the second image represents the object's proper motion between the images.The points in the three-dimensional space of the object in the first image must therefore be multiplied by this difference to be transformed at the time of the second image. This operation is performed for all objects detected in the second image.

[0039] According to an eighth step 308 of the method according to the invention, the device 100 according to the invention determines, from the coordinates in a three-dimensional space of the points relating to the static environment of the vehicle in the second image and the coordinates in a three-dimensional space of the points relating to each moving object in the second image which have been determined during the two preceding steps, a reconstructed image from the second image.

[0040] Next, according to a ninth step 309 of the method according to the invention, the device 100 according to the invention determines a first loss function which establishes a first photometric error as a function of the first image, the reconstructed image and a measure of a structural similarity index, and a second photometric error as a function of the second image, the reconstructed image and a measure of a structural similarity index.

[0041] Indeed, as in a typical method for training a classical prediction model, image reconstruction is performed using the RGB "source" image (i.e., the second image) and the transformations of the 3D point coordinates with the camera's intrinsic matrix. From a mathematical point of view, the photometric error is defined by the following equations:

[0042] [Math.Eq2] L(Dt)= ( p) Mp, O))

[0043] [Math.Eq3] s(l(p\I(p") ) = (!-«)• \l(p) -I(p) | + a- (1-^SSIM(I(p),I(p) ) )

[0044] With: • the loss function, • I the image considered, • / the reconstructed image, • Dt the depth, and • s the photometric error, which includes two terms, the first being the difference in pixel values ​​and the second being the measure of the structural similarity index SSIM (from the English "Structural Similarity Index Measure").

[0045] Next, according to a tenth step 310 of the process according to the invention, the device 100 according to the invention performs a minimization of the first photometric error and the second photometric error determined during the previous step.

[0046] According to an eleventh step 311 of the method according to the invention, the device 100 according to the invention determines a second loss function establishing a sensitivity to the contours of objects as a function of the dimensions of the second image and a first value on the order of a smoothing gradient. From a mathematical point of view, this second loss function is defined by the following equation:

[0047] [Math.Eq4] L^OWo) WO(P')

[0048] With: • O the input type (here Dts\ • W is a parameter matrix, • ° the order of the smoothing gradient.

[0049] The L1 norm of second-order depth gradients is taken with W = 1, and o = 2, where etx and are the image dimensions. Thus, the reconstruction is bilateral: reconstruction of the current image with the reference image and the predicted depth for the current image, and reconstruction of the reference image with the current image and the predicted depth for that reference image. Both have the same performance and both are capable of avoiding the impact of occlusion: pixels visible only in a single image, through the agglomeration function as minimization, defined by the equation below.

[0050] [Math.Eq5] L* = min(z4(p), L*(p))

[0051] where the index * signifies the digit representing the division, and the indices ; etr represent the two reconstructions.

[0052] Finally, according to a twelfth step 312 of the process according to the invention, the device 100 according to the invention performs the training of the depth prediction model by adding the two loss functions determined during the two previous steps.

[0053] Therefore, thanks to the method and device according to the invention described above, a solution is provided for training a depth prediction model from images acquired by a monocular vision system of a motor vehicle in the presence of numerous moving objects. Through this solution, the device and method according to the invention maximize the depth prediction efficiency of a monocular vision system of a motor vehicle and thus enable an improvement in the safety of driver assistance features that rely on the use of such a system.

Claims

1. Demands A self-supervised training method for a depth prediction model implemented by a neural network associated with a vision system of a motor vehicle, the vision system comprising a monocular camera, said method being implemented by a processor, and being characterized in that it comprises the following steps: i. obtaining a first image acquired by the camera at a first instant and a second image acquired by the camera at a second instant prior to the first instant; ii. determination of a first depth map for the first image, a second depth map for the second image and an estimate of the vehicle's displacement between the camera acquisition of the first image and the camera acquisition of the second image; iii. processing of the first image and the second image in order to determine a two-dimensional bounding box for each moving object present in the first image, a two-dimensional bounding box for each moving object present in the second image as well as a two-dimensional segmentation identifying in each image the pixels relating to each moving object; iv. determination, from the first depth map, the second depth map and the moving objects determined by the image segmentation carried out during step iii), of a three-dimensional bounding box for each moving object present in the first image and a three-dimensional bounding box for each moving object present in the second image; v. determination of associations between the two-dimensional bounding boxes of the moving objects present in the first image and the two-dimensional bounding boxes of the moving objects present in the second image determined during step iii) in order to deduce associations between the three-dimensional bounding boxes of the moving objects present in the first image and the three-dimensional bounding boxes of the moving objects present in the second image determined during step iv); vi. determination, based on the estimation of the vehicle's displacement between the acquisition by the camera of the first image and the acquisition by the camera of the second image, of a first coordinate transformation allowing the coordinates in a three-dimensional space of the points relative to the static environment of the vehicle in the first image to be obtained from the coordinates in a three-dimensional space of the points relative to the static environment of the vehicle in the second image; vii. determination, for each moving object present in the second image, as a function of the estimation of the vehicle's displacement between the acquisition by the camera of the first image and the acquisition by the camera of the second image, of the associations between the three-dimensional bounding boxes of the moving objects in each image determined during step v) and of the movement of the three-dimensional bounding boxes determined for each moving object, of a second coordinate transformation allowing the coordinates in a three-dimensional space of the points relative to a moving object present in the second image from the coordinates in a three-dimensional space of the points relative to an associated moving object present in the first image; viii. determination from the coordinates in a three-dimensional space of the points relative to the static environment of the vehicle in the second image and from the coordinates in a three-dimensional space of the points relative to each moving object in the second image, of an image reconstructed from the second image; ix. determination of a first loss function establishing a first photometric error as a function of the first image, the reconstructed image and a measure of a structural similarity index and a second photometric error as a function of the second image, the reconstructed image and a measure of a structural similarity index; x. minimization of the first photometric error and the second photometric error; xi. determination of a second loss function establishing a sensitivity to the contours of objects as a function of the dimensions of the second image and a first value of the order of a smoothing gradient; and xii. training of the depth prediction model by adding the two loss functions.

2. Method according to claim 1, characterized in that step ii) is carried out using a monocular depth prediction model.

3. A method according to any one of the preceding claims, characterized in that step iii) is carried out using the Mask-RCNN model or the MAE model.

4. A method according to any one of the preceding claims, characterized in that step iv) is carried out using the PointNet model or the PointMAE model.

5. A method according to any one of the preceding claims, characterized in that step v) is carried out using an object tracking model from the PCL library.

6. A self-supervised training device (100) for a depth prediction model of a vision system of a motor vehicle, the vision system comprising a monocular camera, characterized in that the device comprises an information processing unit (101), with one or more processors, and a data storage medium (102), which are configured to implement a method according to any one of the preceding claims.

7. Computer program comprising program code instructions for carrying out the steps of a process according to any one of claims 1 to 5 when said program is executed by at least one processor.

8. A computer-readable medium, characterized in that a program according to claim 7 is stored thereon. 15

9. Motor vehicle (1) equipped with a vision system comprising a monocular camera, characterized in that the vehicle (1) comprises a device (100) according to claim 6.

Images