Method and device for learning a depth prediction model from vertical image strips.

Abstract

Method or device for learning a depth prediction model associated with a vehicle-mounted vision system. The method comprises receiving (31) a batch of images acquired by a camera of the vision system, the images being acquired at distinct time points; dividing (32) these images into vertical bands; comparing vertical bands from two consecutive images when a colorimetric difference (33) between these bands exceeds a threshold; the comparison being based on a photometric error determined (35) for each band compared to a set of pixels determined from the other band to be compared. The depth prediction model is then learned (37) by minimizing the largest photometric errors associated with the vertical bands of each image. Figure for the abstract: Figure 3

Description

Title of the invention: Method and device for learning a depth prediction model from vertical image strips. technical field

[0001] The present invention relates to methods and devices for learning a depth prediction model associated with a stereoscopic vision system embedded in a vehicle, for example, in a motor vehicle. The present invention also relates to a method and device for determining depth and / or measuring the distance separating an object from a vehicle equipped with a vision system. Technological background

[0002] Many modern vehicles are equipped with Advanced Driver-Assistance Systems (ADAS). Such ADAS systems are passive and active safety systems designed to eliminate human error in driving all types of vehicles. ADAS systems use advanced technologies to assist the driver while driving and thus improve performance. ADAS systems use a combination of sensor technologies to perceive the environment around a vehicle and then provide information to the driver or act on certain vehicle systems.

[0003] There are several levels of ADAS, such as reversing cameras and blind spot sensors, lane departure warning systems, adaptive cruise control or automatic parking systems.

[0004] The ADAS systems installed in a vehicle are powered by data obtained from one or more on-board sensors such as, for example, cameras. These cameras make it possible, in particular, to detect and locate other road users or any obstacles present around a vehicle in order, for example: - to adapt the vehicle's lighting according to the presence of other users; - to automatically regulate the vehicle's speed; - to act on the braking system in case of risk of impact with an object.

[0005] The position of another user or an obstacle is, for example, determined by a vision system comprising a prediction model for the depth associated with a pixel or for the distance separating the vision system from an object in a three-dimensional scene. Such a model is, for example, learned using images, these images being obtained from a universal database, for example Kitti® or Sceneflow®. Kitti®, for example, provides images of a city center road environment, but such a database does not encompass all the road environments a vehicle might encounter. The training data is therefore unsuitable for training the predictive model for a vehicle operating in other road environments.

[0006] The quality of the training of the depth or distance prediction model is, however, very important. Indeed, the depths or distances predicted by the model represent, for example, the distances to other road users or obstacles present in the road environment of the vehicle equipped with the vision system and ADAS. The proper functioning of the driver assistance devices using this data therefore depends on the quality of the data emitted by the vision system.

[0007] A vision system comprising a camera, hereinafter referred to as a monocular vision system, uses several consecutive images to predict depths associated with pixels in one of the images. This monocular vision system has the drawback of not being able to accurately predict depths associated with pixels in an image corresponding to moving objects in the three-dimensional scene observed during a training or learning phase, these moving objects being hereinafter referred to as dynamic objects. Indeed, the accuracy of the depths predicted by this monocular vision system for pixels corresponding to a dynamic object is not satisfactory during this learning phase, as the depth prediction model is unable to determine a proportion of disparity or optical flux, for example, arising from the inherent motion of the dynamic object.The greater the number of pixels corresponding to a dynamic object within an image, the less relevant the depth prediction model training will be. Therefore, it is advisable to train the depth prediction model associated with this monocular vision system from images that do not contain pixels corresponding to a dynamic object; however, this type of filtering is not straightforward. Summary of the present invention

[0008] One object of the present invention is to solve at least one of the problems of the technological background described above.

[0009] Another object of the present invention is to improve the learning phase of a depth prediction model from images acquired by a vision system, the depth prediction model being implemented in particular by a neural network and associated with a vision system embedded in a vehicle.

[0010] Another object of the present invention is to improve road safety, in particular by improving the reliability of AD AS systems powered by data obtained from a camera of a vision system.

[0011] According to a first aspect, the present invention relates to a method for learning a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle, the vision system comprising a camera arranged to acquire an image of a three-dimensional scene of an environment external to the vehicle, the process being implemented by at least one processor, and being characterized in that it comprises the following steps: - reception of a batch of images acquired by the camera comprising at least one sequence of images, the images of the batch of images being ordered according to a time instant of acquisition of each image of the batch of images, the batch of images comprising a first set of first images and a second set of second images, a first image of the first set immediately preceding a second image of the second set, the first image being acquired at a first time instant of acquisition prior to a second time instant of acquisition of the second image; - for each image in the set of images, division of the image into a plurality of vertical bands of the same width, to each vertical band in each first image, called the first band, corresponding to a vertical band in the second image immediately preceding the first image, called the second band; - determination, for each first band, of a colorimetric difference associated with the first band by comparing pixels of the first band to pixels of the second band corresponding to the first band; - removal of any first image in the batch of images for which the color difference associated with at least one first band of the first image is less than a threshold value, - determination, for each first band of each first undeleted image in the image batch, of a photometric error associated with the first band, the photometric error being determined from: • a first photometric error determined by comparing the pixels of the second band to pixels of a first set of pixels, the pixels of the first set of pixels being generated from the pixels of the first band and from first depths associated with the pixels of the first band, and • a second photometric error determined by comparing the pixels of the first band to pixels of a second set of pixels, the pixels of the a second set of pixels being generated from the pixels of the second band and second depths associated with the pixels of the second band, the first and second depths being predicted by the depth prediction model from the first and second bands; • selection, for each first image not deleted in the batch of images, of a determined number of vertical bands according to the photometric error associated with each vertical band of the first image, the selected bands being those with the lowest photometric errors; - learning the depth prediction model by minimizing a loss error determined from the photometric errors associated with the selected vertical bands.

[0012] This learning process thus makes it possible to limit: • the consideration of images acquired by a static vision system, the colorimetric difference determined during the step of determining a colorimetric difference allowing the detection of the absence of camera movement between two consecutive image acquisition times in order to not take into account the bands associated with these images during the following steps, and • the influence of the presence of pixels in an image corresponding to a moving object in the observed three-dimensional scene, called a dynamic object, via the selection of bands with the smallest photometric error and therefore the least likely to present this type of pixel.

[0013] Indeed, learning the depth prediction model is not valid from images acquired by a stationary or static camera in the three-dimensional scene, nor from images containing pixels corresponding to a dynamic object. These different steps thus make it possible to filter the bands used to learn the depth prediction model.

[0014] According to a variant of the learning process, the colorimetric difference associated with the first band is defined by the following function: With : • corresponding to the colorimetric difference associated with the first band, denoted Ref, • I{(p) a colorimetric value determined for a pixel p in the first band, the pixel p being defined by its two-dimensional coordinates, , and • I^{p) a colorimetric value determined for pixel p in the second band.

[0015] If two images have been acquired from the same viewpoint, that is, at two distinct times when the camera is not moving, then the pixels of the two images are similar. Their colorimetric values ​​are then almost identical; the colorimetric difference is then practically zero.

[0016] According to another variant of the learning process, the photometric error associated with the first band is determined by the following function: L^Ref] = XP [ min^ ( p ), Lp2( p ) ) ] With: • Lp corresponding to the photometric error associated with the first band, denoted Ref, • Lp} (p) the first photometric error determined for a pixelp in the second band, the pixelp being defined by its coordinates in two dimensions, and • Lp2 ( p ) the second photometric error determined for the pixel p in the first band.

[0017] The fact of retaining only the minimum value among the first and second photometric errors makes it possible in particular not to impact the learning when an object of the three-dimensional scene is occluded in one of the images, that is to say when one image includes pixels corresponding to this object and the other image does not include pixels corresponding to this same object.

[0018] According to yet another variant of the learning process, the loss error is equal to a sum of the photometric errors associated with the selected vertical bands. The set of selected vertical bands is thus taken into account during the training of the depth prediction model.

[0019] According to a further variant of the learning method, the first, respectively the second, photometric error is determined by the following function: Lp* = Zp{\-aY\l{Pyî(P)\+a- With : • Lp^ the first photometric error, denoted Lp\(p), respectively the second photometric error, denoted Lp2(p), • / (p) a colorimetric value of a pixelp in the second band, respectively in the first band / being a pixel defined by its two-dimensional coordinates, * l(p) a value of the pixelp in the first set of pixels, respectively in the second set of pixels, • SSIM is a function that takes into account a local structure, and • has a weighting factor depending in particular on the type of environment in which the vehicle operates.

[0020] According to an additional variant of the learning process, the first set of pixels, respectively the second set of pixels, is generated by the following function: With : • Ps the coordinates of a pixel in the first band, respectively in the second band, •77 a function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector, • K is a camera-associated direction prediction model (11), • Let T be a matrix comprising extrinsic parameters, and • 0 a projection function in the three-dimensional scene of a pixel Pt as a function of its coordinates in the first band, respectively in the second band, and of the depth associated with it.

[0021] According to yet another variant of the learning process, the threshold is determined by the following function: s = kHW' With : •£ the threshold, • k is a threshold difference for a pixel, • H is a number of pixels within a vertical band height, and • W' a number of pixels within a vertical bandwidth.

[0022] The threshold is thus proportional to the number of pixels included in a vertical band, the factor 'k' allowing adjustment of the threshold representing a sensitivity of detection of images acquired from the same point of view.

[0023] According to yet another variant of the learning process, the determined number of vertical bands is between 2 and 3 when the plurality of vertical bands is equal to 16. Indeed, if a dynamic object is present in the observed three-dimensional scene, the pixels corresponding to it are located in a maximum of two or three bands. By not systematically taking into account the two or three bands exhibiting the highest photometric error, the pixels corresponding to this object do not influence the learning of the depth prediction model.

[0024] According to a second aspect, the present invention relates to a device configured to learn a depth prediction model by a vision system embedded in a vehicle, the device comprising a memory associated with at least one processor configured for the implementation of the steps of the process according to the first aspect of the present invention.

[0025] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0026] According to a fourth aspect, the present invention relates to a computer program which includes instructions adapted for carrying out the steps of the process according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0027] Such a computer program may use any programming language and be in the form of source code, object code, or an intermediate form between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0028] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the process according to the first aspect of the present invention.

[0029] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage means, such as a ROM, a CD-ROM or a microelectronic circuit-type ROM, or a magnetic recording means or a hard disk drive.

[0030] On the other hand, this recording medium can also be a transmissible medium such as an electrical or optical signal, such a signal being able to be transmitted via an electrical or optical cable, by conventional or radio frequency, by self-directing laser beam, or by other means. The computer program according to the present invention can, in particular, be downloaded from an Internet-type network.

[0031] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to execute or to be used in the execution of the process in question. Brief description of the figures

[0032] Other features and advantages of the present invention will become apparent from the description of the particular and non-limiting embodiments of the present invention below, with reference to the attached Figures 1 to 4, in which:

[0033] [Fig-1] schematically illustrates a vision system equipping a vehicle, according to a a particular and non-limiting example of a realization of the present invention;

[0034] [Fig.2] illustrates a flowchart of the different steps in a process for determining the depth of a pixel in an image using a prediction model depth associated with a vision system embedded in the vehicle of the [Fig.1], according to a particular and non-limiting embodiment of the present invention;

[0035] [Fig. 3] illustrates a flowchart of the different stages of a method for learning the depth prediction model used in the method of [Fig. 2], according to a particular and non-limiting embodiment of the present invention; and

[0036] [Fig.4] schematically illustrates a device configured to learn a depth prediction model by a vision system embedded in the vehicle of [Fig.1], according to a particular and non-limiting embodiment of the present invention. Description of examples of achievements

[0037] A method and device for learning a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle will now be described in what follows with joint reference to Figures 1 to 4. The same elements are identified with the same reference signs throughout the description that follows.

[0038] The terms "first," "second" (or "firsts," "seconds"), etc., are used in this document by arbitrary convention to allow for the identification and distinction of different elements (such as operations, means, etc.) implemented in the embodiments described below. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0039] For the purposes of this description, receiving an image or a set of images means receiving data representative of an image or a set of images. Generally speaking, any reception, determination, or generation of an object or value amounts to receiving, determining, or generating data representative of that object or value. These abbreviations are intended solely to simplify the description; however, since the processes described below are implemented by one or more processors, it is clear that the input and output data of the various stages of a process are computer data.

[0040] According to a particular and non-limiting embodiment of the present invention, the depth prediction model is learned in a training phase from a set of images acquired by a camera of the vision system associated with the depth prediction model. The image set comprises at least one sequence of images, and the images in the set are ordered according to the acquisition time of each image, with the first image immediately preceding the second image in the set being acquired at a first acquisition time prior to the second acquisition time of the second image.

[0041] Each image in the batch is then divided into a plurality of vertical bands of the same width, with each vertical band in the first image, called the first band, corresponding to a vertical band in the second image, called the second band.

[0042] A colorimetric difference associated with each first band is determined by comparing pixels of the first band to pixels of the second band and a photometric error associated with the first band is determined when the associated colorimetric difference is greater than a threshold value.

[0043] For each first image, a determined number of vertical bands is selected according to the photometric error associated with each vertical band of the first image, the selected bands being those with the lowest photometric errors and the depth prediction model is learned by minimizing a loss error determined from the photometric errors associated with the selected vertical bands.

[0044] Fig. 1 schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting embodiment of the present invention.

[0045] The vehicle 10 is located in an environment 1 corresponding, for example, to a road environment consisting of a network of roads accessible to the vehicle 10.

[0046] In this example, vehicle 10 corresponds to a vehicle with an internal combustion engine, an electric motor(s), or a hybrid vehicle with an internal combustion engine and one or more electric motors. Vehicle 10 thus corresponds, for example, to a land vehicle such as a car, a truck, a bus, or a motorcycle. Finally, vehicle 10 corresponds to an autonomous or non-autonomous vehicle, that is to say, a vehicle operating according to a predetermined level of autonomy or under the total supervision of the driver.

[0047] The vehicle 10 advantageously comprises at least one onboard camera 11, configured to acquire images of a three-dimensional scene unfolding in the environment of the vehicle 10 from a common viewing position. The camera 11 forms a monocular vision system if used alone, as illustrated in [Fig. 1]. However, the present invention is not limited to a monocular vision system comprising a single camera but extends to any vision system comprising at least one camera, for example, 1, 2, 3, or 5 cameras.

[0048] The camera 11 has intrinsic parameters, including: - a focal length, - distortions which are due to imperfections in the optical system of camera 11, - a direction of the optical axis of camera 11, and - a resolution.

[0049] The intrinsic parameters characterize the transformation which associates, for an image point, hereafter called a "point", its three-dimensional coordinates in the camera 11 reference frame at pixel coordinates in an image acquired by camera 11. These parameters do not change if the camera is moved.

[0050] Distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of camera lenses, will deflect the light beams and thus induce a positioning error for the projected point relative to an ideal model. It is then possible to complete the camera model by introducing the three distortions that generate the most significant effects, namely radial, decentering, and prismatic distortions, induced by defects in lens curvature, parallelism, and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, meaning that distortions are not taken into account, and their correction is addressed during image acquisition or calibration.

[0051] The camera 11 is arranged so as to acquire an image of a three-dimensional scene from a defined viewpoint, the viewpoint is for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10 as illustrated in [Fig.1].

[0052] The camera 11, for example, acquires images of a three-dimensional scene located in front of the vehicle 10, the first camera 11 covering an acquisition field 12. An object 13 is placed in the acquisition field 12 of the camera 11, defining an occlusion field 14 for the vision system, an object present in the occlusion field 14 not being observable by the camera 11 from its current observation position.

[0053] It is evident that it is possible to use such a vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with cameras placed and oriented differently, the invention not being limited to the observation of a three-dimensional scene taking place in front of the vehicle 10 carrying the camera 11.

[0054] An image acquired by the camera 11 at a given acquisition time is in the form of data representing pixels characterized by: - ​​coordinates in the image; and - data relating to the colours and brightness of objects in the observed scene in the form of, for example, RGB colourimetric values ​​(from the English "Red Green Blue", in French "Rouge Vert Bleu") or HSL (Tone, Saturation, Luminosity).

[0055] Each pixel of the acquired image represents an object in the three-dimensional scene present in the field of view of the camera 11. Indeed, a pixel of the acquired image is the smallest visible unit and corresponds to a point of light resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When light strikes this object, photons are Light emitted or reflected is captured by a photosensitive sensor in camera 11 after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thus capturing visual details. The combination of millions of pixels creates an image that faithfully represents the physical object observed by camera 11. An image point, as previously described, is therefore a point on the surface of an object in the three-dimensional scene.

[0056] When the vehicle 10 is in motion, then two images acquired by the camera 11 at two distinct time instants represent views of the same three-dimensional scene taken from different viewpoints or observation positions, the observation positions of the camera 11 being distinct. On this three-dimensional scene there are, for example: - buildings; - road infrastructure; - other users or stationary objects, for example a parked vehicle; and / or - other users or moving objects, for example another vehicle, a cyclist or a moving pedestrian.

[0057] According to a particular embodiment, an image acquired by the camera 11 includes a distortion equal to 0.5%, 0.8%, or greater than 1%. The measurement of such distortion corresponds to determining a ratio between: - the maximum spacing of a pixel in the image of a straight line in the first three-dimensional scene whose image is a line touching the longest edge of the first image, either at the center of the image edge or at the corners of the image edge, and - the length of this edge.

[0058] Each image acquired by the camera 11 is for example sent to a processor, for example a computer of a device equipping the vehicle 10, or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10. It is then used during the implementation of a method for determining the depth of one of its pixels and / or during the implementation of a method for learning a depth prediction model associated with this camera of the vision system.

[0059] Figure 2 illustrates a flowchart of the different steps of a method 2 for determining the depth of a pixel of an image by depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle, for example in the vehicle 10 of Figure 1, according to a particular and non-limiting embodiment of the present invention. Method 2 is for example implemented by a device of the vision system on board vehicle 10 or by device 4 of [Fig.4].

[0060] In a step 21, at least one image acquired by the camera 11 is received.

[0061] In a step 22, depths associated with a set of pixels of an image received are determined by the depth prediction model from at least one received image.

[0062] Each determined depth then corresponds to a distance separating the vehicle 10 or a part of the vehicle 10 from an object in the three-dimensional scene to which a pixel is associated, the determination of a depth of a pixel then corresponding to a measurement of a distance separating an object from the vehicle carrying the vision system.

[0063] If the ADAS uses these depths or distances as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another road user, the ADAS is then able to determine this distance precisely. For example, if the ADAS's function is to activate the braking system of the vehicle 10 in the event of a risk of collision with another road user, and the distance between the vehicle 10 and that same road user decreases sharply, then the ADAS is able to detect this sudden closeness and activate the braking system of the vehicle 10 to avoid a possible accident.

[0064] Figure 3 illustrates a flowchart of the different stages of a method for learning the depth prediction model used in a method for determining the depth of a pixel of an image, for example in method 2 of Figure 2, according to a particular and non-limiting embodiment of the present invention.

[0065] The learning process 3 is for example implemented by the device on board the vehicle 10 implementing the depth determination process for the vision system on board a vehicle 10 or by the device 4 of the [Fig.4].

[0066] In a step 31, a batch of images acquired by the camera 11 is received. The image batch comprises at least one image sequence, the images in the batch being ordered according to the acquisition time of each image in the batch. The image batch comprises a first set of first images and a second set of second images, with a first image from the first set immediately preceding a second image from the second set, the first image being acquired at a first acquisition time prior to a second acquisition time of the second image. Note that the first and second sets contain the same number of images; only the extreme images of the batch of images being included in only one of the two sets of images, the other images in the batch of images being included in both the first and second sets of images.

[0067] An image sequence includes, in particular, a plurality of images acquired by the camera 11 of the vehicle 10 at defined time intervals, for example, based on an image acquisition frequency. The camera 11 may, for example, have an image acquisition frequency of thirty Hertz (30 Hz), meaning that the camera 11 acquires 30 images per second when the vehicle 10 is active, for example, when it is moving or when its engine is running or powered if it is an electric or hybrid vehicle. An image sequence then consists of images acquired consecutively representing the same three-dimensional scene at different times and observed from several different viewpoints if the vehicle 10 is moving. An image sequence constitutes, for example, a video.

[0068] It should be noted that if the image set comprises several sequences, then a first image may belong to one sequence while the second image belongs to a second sequence. The acquired images are then very different and do not necessarily represent the same scene. If the training of the depth prediction model uses such a pair of images, it will not improve the training, but it will also be detrimental. Indeed, the images are so different that the training step described below is unable to utilize them.

[0069] It should also be noted that each sequence is not necessarily homogeneous and includes images acquired, for example, in different environments, for example a first sequence includes images acquired on a motorway while the following sequence includes images acquired in a rural environment.

[0070] For the remainder of this description, the acquired images are considered to be of the same resolution. Indeed, since they are acquired by the same camera 11, their resolution should not differ. However, if the resolution of images in the same sequence were different, it is possible to add an additional step consisting of resizing or cropping certain images to obtain a sequence of images comprising only images of the same resolution, that is to say, comprising the same number of pixels in terms of their height and width.

[0071] In a step 32, each image in the set of images is divided into a plurality of vertical strips of the same width, with each vertical strip in each first image, called the first strip, corresponding to a vertical strip in a second image, called the second strip. Each vertical strip has a height equal to the height of the image. The width, on the other hand, is defined according to the size of areas comprising pixels associated with dynamic objects in an image, thus the The width of a vertical band must encompass at least half the width of such an area. For example, if pixels corresponding to a dynamic object occupy half (50%) of the image's width, then the width of a band must be at most 25% of the original image's width. The intended effect is that two or three vertical bands can include pixels corresponding to one or more dynamic objects while guaranteeing the existence of at least one area without pixels corresponding to a dynamic object.

[0072] According to a particular embodiment, the number of vertical bands is 16.

[0073] In a step 33, a colorimetric difference associated with each first band is determined by comparing pixels of the first band to pixels of the second band corresponding to the band.

[0074] According to a particular embodiment, the colorimetric difference associated with the first band is defined by the following function: [Math.l] ZW)=I / 1(P)- / 2(P)I

[0075] With: • corresponding to the colorimetric difference associated with the first band, denoted Ref, • I(p) a colorimetric value determined for a pixel p in the first band, the pixel p being defined by its two-dimensional coordinates, , and • I2(p) a colorimetric value determined for pixel p in the second band.

[0076] Determining the colorimetric difference amounts to determining a level of similarity between two pixels, the pixels being defined by their position or coordinates in two dimensions and which are identical in the first image and in the second image.

[0077] The aim is to determine whether the first image was acquired from the same viewpoint as the second image. An image acquired from the same viewpoint as another image, and in which pixels correspond to fixed objects in the observed three-dimensional scene, then includes pixels with the same colorimetric value as the pixels in the other image. If a large number of pixels in the image are similar to pixels in the other image, then it is highly probable that the image was acquired from the same viewpoint as the other image. However, to ensure the proper training of the depth prediction model, it is necessary to consider only images acquired from different viewpoints.

[0078] According to a particular embodiment, a representative threshold difference for an identical pixel is between 1 and 2 when a colorimetric value of a A pixel is encoded using eight bits. Eight-bit encoding allows for a colorimetric value between 0 and 255; a threshold close to 1.5 is therefore particularly suitable for comparing two pixels of images acquired from the same viewpoint with a very short time interval between the acquisition moments of the two images, for example, with the duration of 1 / 30th of a second corresponding to two consecutive images acquired by a camera acquiring images at a frequency of 30 Hz. Indeed, at such an image acquisition frequency, a variation in the lighting of the three-dimensional scene is not perceptible.

[0079] In order to avoid considering images in a sequence acquired from the same viewpoint, in step 34, the first image is removed from the image set if the color difference associated with a first band of this image is less than a threshold; in other words, if it is highly probable that this image was acquired from the same viewpoint as the reference image. Analyzing the color difference by vertical band in the first image allows for the detection of a first image potentially acquired by a first static camera 11, even when a dynamic object is present in the three-dimensional scene. Indeed, only the vertical bands of the first image containing pixels corresponding to this dynamic object have a significant color difference, and not all the vertical bands of the first image.

[0080] The threshold is, for example, determined by the following function: [Math.2] e=kHW

[0081] With: •e the threshold, • k is a threshold difference for a pixel, • H a number of pixels within a vertical band height, and • VF' a number of pixels within a vertical band width.

[0082] According to the specific embodiment described above, as soon as a vertical band in an image is considered invariant, that is, as soon as the color difference associated with it is less than the threshold defined above, the image containing this vertical band is removed from the set of images. The subsequent learning process 3 therefore only considers the images that were not removed during this additional step.

[0083] In a step 35, a photometric error associated with the first band is determined for each first band for which the associated colorimetric difference is greater than a threshold value.

[0084] The photometric error is determined from: • a first photometric error determined by comparing the pixels of the second band to pixels of a first set of pixels, the pixels of the first set of pixels being generated from the pixels of the first band and from first depths associated with the pixels of the first band, and • a second photometric error determined by comparing the pixels of the first band to pixels of a second set of pixels, the pixels of the second set of pixels being generated from the pixels of the second band and second depths associated with the pixels of the second band, the first and second depths being predicted by the depth prediction model from the first and second bands.

[0085] Determining the depth of a pixel of an image acquired by a moving camera from a depth prediction model implemented by a convolutional neural network is known to those skilled in the art, for example with monodepth2® described in the document "Digging Into Self-Supervised Monocular Depth Estimation" written by Clément Godard, Oisin Mac Aodha, Michael Firman and Gabriel Brostow and published in August 2019. The depth prediction model is then used to predict depths associated with pixels of a vertical band of an image.

[0086] It should be noted that the depth prediction by the depth prediction model associated with the monocular vision system, i.e. including only the camera 11, is accurate for a pixel corresponding to a static object, but is much less accurate for a pixel corresponding to a dynamic object.

[0087] Generating a first set of pixels from the first band of the first image acquired by the camera 11 consists of reprojecting a pixel from the first band of the first image acquired at a first time instant into the three-dimensional scene as a point, and then projecting this point into the image plane of the camera 11 at its position at a second time instant of acquisition corresponding to the time instant of acquisition of the second image, so as to obtain a set of pixels corresponding to a view of the three-dimensional scene from the viewpoint of the camera 11 at the second time instant of acquisition. The image plane of the camera 11 corresponds to a plane defined in the camera 11's frame of reference, normal to the optical axis of the camera 11 and located at the focal length of the camera 11.Thus, the first set of pixels generated from the first band of the first image is comparable to the second band of the second image. Similarly, the second set of pixels generated from the second band of the second image is comparable to the first band of the first image.

[0088] The positions of the camera 11 at the first and second time instants of acquisition are not coincident when the vehicle 10 is in motion and Because objects can obscure other objects in the scene or even move between the two acquisition times, the generated pixel sets are not identical to the image bands acquired by camera 11. Furthermore, the depth prediction, which includes predicting camera 11's movement between the first and second time points, as well as the models used to generate the pixel sets, are not error-free. Therefore, comparing a generated pixel set to a band of an acquired image allows us to evaluate the accuracy of the different models used and the predicted depth values.

[0089] According to a particular embodiment, the first set of pixels, respectively the second set of pixels, is generated by the following function:

[0090] [Math.3]

[0091] With: • Ps the coordinates of a pixel in the first band, respectively in the second band, •77 a function to convert from homogeneous coordinates to pixel coordinates by removing one dimension from a vector, • K is an intrinsic matrix of camera 11, • Let T be a matrix comprising extrinsic parameters, and • 0 a projection function in the three-dimensional scene of a pixel Pt as a function of its coordinates in the first band, respectively in the second band, and of the depth associated with it.

[0092] The use of an intrinsic camera matrix 11 is particularly applicable to pinhole and calibrated cameras.

[0093] According to a second variant, the direction prediction model corresponds to a learned model implementing a second neural network. Such a direction prediction model is known to those skilled in the art; it is notably presented in the document "Digging Into Self-Supervised Monocular Depth Estimation".

[0094] According to this second variant, a direction prediction step associated with the pixels of the first band and the pixels of the second band is necessary.

[0095] The matrix comprising the extrinsic parameters of the moving camera 11 can be obtained in several ways. For example, from data emitted by a driver assistance system of the vehicle 10, called an ADAS system, or from sensors on board the vehicle 10. According to another example, the extrinsic parameters corresponding to the movement of the camera 11 between the first and second time instants are determined in an additional step by a motion prediction model, for example by the one presented in the document "Digging Into Self-Supervised Monocular Depth Estimation".

[0096] The first and / or second photometric error is smaller the more accurate the depth prediction model, i.e., the more effective its training. Conversely, if the depth prediction model lacks accuracy, for example because its training is incomplete, then the first and / or second photometric error is significant.

[0097] According to a first particular embodiment, the first, respectively the second, photometric error is determined by the following function: [Math.4] l-«) • \l(p)-I(p) ] +«■ ( l-iSSIM(l(p), I(p) ) )

[0098] With: • Lp* the first photometric error, denoted Lp\ (p) , respectively the second photometric error, denoted Lp2(p), • j(p) a colorimetric value of a pixel p in the second band, respectively in the first band, where p is a pixel defined by its two-dimensional coordinates, * l[p] a value of the pixel p in the first set of pixels, respectively in the second set of pixels, • SSIM is a function that takes into account a local structure, and • has a weighting factor depending in particular on the type of environment in which the vehicle operates (10).

[0099] According to a second particular embodiment, the first and second photometric errors include a determination of a reconstruction error of a generated set of pixels, the reconstruction error being determined by the following function: [Math.5] ^aD^p)

[0100] With: • ^smooth^P^ the reconstruction error, denoted Lsi( p) for a pixelp of the first set of pixels, respectively the construction error Lg2(p) for a pixelp of the second set of pixels, • D(p) is a depth associated with pixel p in the first band, respectively in the second band; • W is a parameter matrix; • 0 is the order of a smoothing gradient; • an L1 norm of the second order depth gradients is calculated with VF =1 , et0 =2; • x and y are the dimensions of the first and second images; • P is a hyperparameter dependent on the environment in which the vehicle 10 is operating; and • is a colorimetric value of pixel p in the first band, respectively second band.

[0101] This second function is generally used to deal with the discontinuity at the edge of objects (in English "edge aware smoothness").

[0102] The photometric error is thus defined, for example, from the photometric errors and reconstruction errors previously defined.

[0103] According to the first particular embodiment, the photometric error associated with each first band is determined by the following function: [Math.6] L^Ref) = Yp [min^! (p), Lp2(p))]

[0104] With: • Lp corresponding to the photometric error associated with the first band, denoted Ref, • Lp\ ( p) the first photometric error determined for a pixelp in the second band, the pixelp being defined by its coordinates in two dimensions, and • Lp2 ( p ) the second photometric error determined for the pixel p in the first band.

[0105] According to the second particular embodiment, the photometric error associated with each first band is determined by the following function: [Math.7] L^Ref) = Ip[mm(Lpl(p), Lp2(p) ) + Lsï(p) + Ls2(p) ]

[0106] With: • Lp corresponding to the photometric error associated with the first band, denoted Ref, • Lp] (p ) the first photometric error determined for a pixelp in the first band, the pixelp being defined by its two-dimensional coordinates, • Lp2 ( p ) the second photometric error determined for the pixel p in the second band, •Ls](p) 1'reconstruction error associated with a pixel from the first set of pixels, and • L^ip) the reconstruction error associated with a pixel from the second set of pixels.

[0107] In a step 36, a determined number of vertical bands are selected for each first image according to the photometric error associated with each vertical band of the first image, the selected bands being those with the lowest photometric errors.

[0108] Selecting only the vertical bands with the lowest photometric error eliminates vertical bands that would have a high photometric error due to the presence of pixels in that vertical band corresponding to a dynamic object in the observed three-dimensional scene. Thus, by eliminating these bands, which are potentially detrimental to the training of the depth prediction model, the learning process 3 gains in reliability. Pixels corresponding to dynamic objects are not taken into account in the next step.

[0109] The determined number of vertical bands is, for example, between 2 and 3 when the plurality of vertical bands is equal to 16. Thus, whether or not pixels in an image correspond to dynamic objects, the most suspect bands are systematically eliminated from the next step. According to this example, each image acquired by the moving camera 11 is divided into 16 vertical bands of equal width, and the three bands exhibiting the highest photometric error are eliminated; each image then has 13 bands selected for the training step 37.

[0110] In step 37, the depth prediction model is learned by minimizing a loss error determined from the photometric errors associated with the selected vertical bands. The number of vertical bands selected per image is thus always the same, and each image therefore makes the same contribution to this step 37.

[0111] According to a particular embodiment, the loss error is equal to a sum of the photometric errors associated with the selected vertical bands. It is, for example, determined by the following function: [Math.8] L = OW)

[0112] With: • L corresponding to the loss error, * L^Ref} corresponding to the photometric error associated with the vertical band Ref, n representing its position in the list of photometric errors associated with the bands of a first image and ranked in ascending order, • N corresponds to the number of vertical bands per first image, and • N -X corresponding to the determined number of vertical bands from the first image selected, X representing the number of vertical bands dropped.

[0113] According to another particular embodiment, the loss error is equal to an average of the photometric errors associated with the selected vertical bands.

[0114] The training of the depth prediction model consists of adjusting the input parameters of the convolutional neural network implementing the depth prediction model in order to minimize the previously calculated loss error.

[0115] According to a particular embodiment, if the loss error determined for the batch of images is not zero, a gradient descent algorithm is used. Such an algorithm is known to those skilled in the art and is, for example, implemented using the loss.backward() function in PyTorch®.

[0116] An object visible in the field of view of the camera 11 at the first time instant of acquisition and masked in the field of view of the camera 11 at the second time instant of acquisition or vice versa does not impact the loss error thanks to the min function, this process is therefore insensitive to occlusions.

[0117] Learning is performed solely by comparing images acquired from different viewpoints, where the first or second photometric error associated with pixels corresponding to a dynamic object is negligible. Such a learning process thus makes it possible to learn a depth prediction model from image sequences acquired by a camera of the vision system, with the camera in motion.

[0118] Thus, the depth prediction model used for the depth prediction of a pixel of an image acquired by a camera of the vision system is made more reliable through this learning process.

[0119] This learning process is performed using data acquired by the on-board vision system and therefore does not require data annotated by another on-board system or the storage of a training image library. Furthermore, the training data is representative of the data received when the system is in operation or in production; indeed, the training data is representative of real-world environments in which the vehicle carrying the vision system operates or moves. This training data is therefore particularly relevant because it represents real-world conditions for implementing a depth prediction process.

[0120] Figure 4 schematically illustrates a device configured to learn a A depth prediction model using a vision system embedded in a vehicle and / or for predicting the depth associated with a pixel in an image acquired by a camera, according to a particular and non-limiting embodiment of the present invention. Device 4 corresponds, for example, to an embedded device in the first vehicle 10, for example a computer associated with the stereoscopic vision system.

[0121] Device 4 is, for example, configured to carry out the operations described opposite Figures 1 and 4 and / or the steps described opposite Figures 2 and 3. Examples of such a device 4 include, but are not limited to, embedded electronic equipment such as a vehicle's on-board computer, an electronic control unit such as an ECU (Electronic Control Unit), a smartphone, a tablet, or a laptop computer. The elements of device 4, individually or in combination, may be integrated into a single integrated circuit, into several integrated circuits, and / or into discrete components. Device 4 may be implemented in the form of electronic circuits or software (or computer) modules, or a combination of electronic circuits and software modules.

[0122] The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the process and / or for executing instructions from the software embedded in the device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41, for example, volatile and / or non-volatile memory, and / or includes a memory storage device that may include volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk, or optical disk.

[0123] The computer code of the embedded software(s), including the instructions to be loaded and executed by the processor, is for example stored in memory 4L

[0124] According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (Telematic Control Unit), for example via a communication bus or through dedicated input / output ports.

[0125] According to a particular and non-limiting embodiment, the device 4 includes a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 include one or more of the following interfaces: - Radio frequency (RF) interface, for example, Wi-Fi® (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox using UBN (Ultra Narrow Band) radio technology narrow), or LoRa in the 868 MHz frequency band, LTE (from the English "Long-Term Evolution" or in French "Evolution à long terme"), LTE-Advanced (or in French LTE-avancé); - USB interface (from the English "Universal Serial Bus" or "Universal Serial Bus" in French); HD MI interface (from the English "High Definition Multimedia Interface", or "High Definition Multimedia Interface" in French); - LIN interface (from the English "Local Interconnect Network", or in French "Réseau interconnecté local").

[0126] According to another particular and non-limiting embodiment, the device 4 includes a communication interface 43 which enables communication with other devices (such as other computers in the embedded system) via a communication channel 430. The communication interface 43 corresponds, for example, to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds, for example, to a wired network of the CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by ISO 17458) or Ethernet (standardized by ISO / IEC 802-3) type.

[0127] According to a particular and non-limiting embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch or not, one or more speakers 450 and / or other peripherals 460 via the output interfaces 44, 45, 46 respectively. According to a variant, one or more of the external devices is integrated into the device 4.

[0128] Of course, the present invention is not limited to the embodiments described above but extends to a method for determining the depth of a pixel in an image acquired by a vision system, and / or for measuring the distance between an object and a vehicle equipped with a vision system, the depth and / or distance being predicted and / or measured via a depth prediction model learned according to the learning method described above, which would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0129] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-powered vehicle, comprising the device 4 of [Fig.4].

Claims

1. Demands Method for learning a depth prediction model implemented by a convolutional neural network associated with a vision system embedded in a vehicle (10), the vision system comprising a camera (11) arranged to acquire an image of a three-dimensional scene of an environment external to the vehicle (10), said method being implemented by at least one processor, and being characterized in that it comprises the following steps: - reception (31) of a batch of images acquired by the camera (11) comprising at least one sequence of images, the images of the batch of images being ordered according to a time instant of acquisition of each image of the batch of images, said batch of images comprising a first set of first images and a second set of second images, a first image of said first set immediately preceding a second image of said second set, the first image being acquired at a first time instant of acquisition prior to a second time instant of acquisition of the second image; - for each image of said batch of images, division (32) of said each image into a plurality of vertical bands of the same width, to each vertical band in each first image, called first band, corresponding to a vertical band in the second image immediately preceding said each first image, called second band; - determination (33), for each first band, of a colorimetric difference associated with said first band by comparing pixels of said first band to pixels of the second band corresponding to said first band; - deletion (34) of any first image in said batch of images for which the colorimetric difference associated with at least one first band of said first image is less than a threshold value, - determination (35), for each first band of each first undeleted image in said batch of images, of a photometric error associated with said first band, said photometric error being determined from: • of a first photometric error determined by comparing the pixels of said second band to pixels of a first set of pixels, the pixels of the first set of pixels being generated from the pixels of said each first band and first depths associated with the pixels of said each first band, and • of a second photometric error determined by comparing the pixels of said each first band to pixels of a second set of pixels, the pixels of the second set of pixels being generated from the pixels of the second band and second depths associated with the pixels of the second band, said first and second depths being predicted by the depth prediction model from said first and second bands;• selection (36), for each first undeleted image in said batch, of a determined number of vertical bands as a function of the photometric error associated with each vertical band of said first image, the selected bands being those with the lowest photometric errors; - learning (37) the depth prediction model by minimizing a loss error determined from said photometric errors associated with the selected vertical bands.

2. A method according to claim 1, wherein the colorimetric difference associated with said first band is defined by the following function: zW)=IMp)-W)I With: • D corresponding to the colorimetric difference associated with the first band, denoted Ref, • I^p) a colorimetric value determined for a pixel p in the first band, the pixel p being defined by its coordinates in two dimensions, , and • 12( P ) a colorimetric value determined for the pixel p in the second band.

3. A method according to claim 1 or 2, wherein the photometric error associated with said first band is determined by the following function: L^Ref) = Yp[min^i(p), Lpl{p))} With: • Lp corresponding to the photometric error associated with the first band, denoted Ref, • Lp} ( p ) the first photometric error determined for a pixelp in the second band, the pixel p being defined by its coordinates in two dimensions, and • Lp2 ( p ) the second photometric error determined for the pixelp in the first band.

4. A method according to any one of claims 1 to 3, wherein the loss error is equal to a sum of the photometric errors associated with the selected vertical bands.

5. A method according to any one of claims 1 to 4, wherein the first, respectively the second, photometric error is determined by the following function: Lp» = E / l-«) ^Lp) -hp) 1 + «■ ( ) ) With: • Lp* the first photometric error, denoted Lp^p], respectively the second photometric error, denoted Lp2(p), • j(pj a colorimetric value of a pixel p in the second band, respectively in the first band, where p is a pixel defined by its coordinates in two dimensions, * l(p) a value of the pixel p in the first set of pixels, respectively in the second set of pixels, • SSIM a function which takes into account a local structure, and • has a weighting factor depending in particular on the type of environment in which the vehicle operates (10).

6. A method according to any one of claims 1 to 5, wherein the first set of pixels, respectively the second set of pixels, is generated by the following function: Ps = AK[T(Kp^Z>M Where: • Ps the coordinates of a pixel in the first band, respectively in the second band, • 77 a function to go from homogeneous coordinates to pixel coordinates by removing one dimension of a vector, • K a direction prediction model associated with the camera (11), • T a matrix including extrinsic parameters, and • 0 a projection function in the three-dimensional scene of a pixel Pt as a function of its coordinates in the first band, respectively in the second band, and the depth associated with it.

7. A method according to any one of claims 1 to 6, wherein said threshold is determined by the following function: With: •e the threshold, • k a threshold difference for a pixel, • H a number of pixels included in a vertical band height, and • W' a number of pixels included in a vertical band width.

8. A method according to any one of claims 1 to 7, wherein said specified number of vertical bands is between 2 and 3 when said plurality of vertical bands is equal to 16.

9. Device (4) configured to learn a depth prediction model by a vision system embedded in a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured to implement the steps of the method according to any one of claims 1 to 8.

10. Vehicle (10) comprising the device (4) according to claim 9.

Images