Method and device for processing images from a training database of a depth prediction model associated with a vision system.

Abstract

Method or device for processing images from a training database of a depth prediction model associated with a vision system on board a vehicle, the training database comprising a sequence of images acquired by a camera of the vision system. Indeed, the method comprises the selection (21) of two consecutive images in the sequence corresponding to a reference image and to a candidate image and the application (22, 23) of a determined mask to the reference image to select a first set of pixels in the reference image and to the candidate image to select a second set of pixels in the candidate image. A photometric error is determined (24) by comparing the sets of pixels and the candidate image is deleted (25) from the sequence of images when the photometric error is less than a threshold value. Figure for abstract: Figure 2

Description

Title of the invention: Method and device for processing images from a training database of a depth prediction model associated with a vision system. Technical field

[0001] The present invention relates to methods and devices for processing images from a training database of a depth prediction model for a vision system embedded in a vehicle, for example in a motor vehicle.

[0002] The present invention also relates to methods and devices for acquiring training data for a depth prediction model for a vision system embedded in a vehicle, for example in a motor vehicle. Technological background

[0003] Many modern vehicles are equipped with so-called AD AS (Advanced Driver-Assistance System). Such AD AS systems are passive and active safety systems designed to eliminate the element of human error in the driving of vehicles of all types. AD AS use advanced technologies to assist the driver while driving and thus improve their performance. AD AS use a combination of sensor technologies to perceive the environment around a vehicle, then provide information to the driver or act on certain vehicle systems.

[0004] There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control and automatic parking systems.

[0005] The AD AS embedded in a vehicle are supplied with data obtained from one or more embedded sensors such as, for example, cameras. These cameras make it possible in particular to detect and locate other road users or possible obstacles present around a vehicle in order, for example: - to adapt the vehicle's lighting according to the presence of other users; - to automatically regulate the vehicle speed; - to act on the braking system in the event of a risk of impact with an object.

[0006] A position of another user or of an obstacle is for example determined by a vision system comprising a model for predicting a depth associated with a pixel or a distance separating the vision system from an object in a three-dimensional scene. Such a model is for example learned using images, these images being obtained from a universal database, for example Kitti® or Sceneflow®. Kitti®, for example, provides images of a city-center road environment, but such a database does not include all the road environments in which a vehicle can move. The training data is therefore unsuitable for training the prediction model for a vehicle traveling in other road environments.

[0007] The quality of the training of the depth or distance prediction model is however very important, in fact, the depths or distances predicted by the prediction model represent for example distances at which other users or obstacles present in the road environment of the vehicle carrying the vision system and AD AS are located. The proper functioning of the driving assistance peripherals using this data therefore depends on the quality of the data emitted by the vision system.

[0008] A vision system comprising a single camera, hereinafter called a monocular vision system, uses several consecutive images to predict depths associated with pixels of one of the images. This monocular vision system has the disadvantage of not being able to accurately predict depths associated with pixels of an image when the vehicle on which it is mounted is not moving. Indeed, the depth is obtained by comparing positions of pixels corresponding to the same object of the three-dimensional scene in different images. However, if the vehicle is not moving, then the pixels associated with different objects placed at different distances from the monocular vision system are always located at the same position in the images. The depths predicted by the prediction model are then not reliable.

[0009] Furthermore, the monocular vision system has difficulty predicting the distance separating it from moving objects, which can disrupt learning, i.e. degrade the quality of learning of the depth prediction model when pixels corresponding to these moving objects are used for this learning phase. Summary of the present invention

[0010] An object of the present invention is to solve at least one of the problems of the technological background described above.

[0011] Another object of the present invention is to improve the relevance of the training data of a depth prediction model for a monocular vision system on board a vehicle, allowing training not disturbed by series of images acquired when the vehicle carrying the vision system is not in motion or presenting pixels corresponding to objects in the three-dimensional scene in motion.

[0012] Another object of the present invention is to improve road safety, in particular by improving the operational safety of AD AS systems supplied by data obtained from a vision system on board the vehicle.

[0013] According to a first aspect, the present invention relates to a method for processing images from a training database of a depth prediction model associated with a vision system on board a vehicle, the vision system comprising at least one camera arranged to acquire an image of a three-dimensional scene in an environment of the vehicle, the training database comprising a sequence of ordered images acquired at consecutive time instants by the camera, the method being implemented by a processor and being characterized in that it comprises the following steps: A) selecting two consecutive images in the sequence corresponding to a reference image and a candidate image, the reference image preceding the first candidate image in the sequence; B) applying a determined mask to the reference image to select a first set of pixels in the reference image; C) applying the determined mask to the candidate image to select a second set of pixels in the candidate image; D) determining a photometric error by comparing the first set of pixels to the second set of pixels and comparing the photometric error to a threshold value; E) removing the candidate image from the image sequence when the photometric error is less than the threshold value for updating the image sequence, called the updated image sequence.

[0014] According to a variant, the method further comprises a reiteration of steps C, D and E by selecting as candidate image the image corresponding to the image following the reference image in the updated sequence of images, as long as the photometric error is less than the threshold value.

[0015] According to another variant, the method further comprises a reiteration of steps B, C, D and E by selecting: • as reference image the candidate image, and • as candidate image the image corresponding to the image following the reference image in the updated image sequence, when the photometric error is greater than the threshold value.

[0016] According to yet another variant of the method, a photometric error is determined by the following function: L.= E,[(!-«) ■ Idr)-hp)l +«• (t(p) ))] With : • Lp* the photometric error, • I(p) a value of a pixel P of the first set of pixels, P being defined by its coordinates in two dimensions, • a value of the pixel P of the second set of pixels, • SSIM a function that takes into account a local structure, and • has a weighting factor depending in particular on the type of environment in which the vehicle operates.

[0017] According to an additional variant, the method comprises a step of reducing the definition of each image of the sequence of images.

[0018] According to another variant of the method, the reduction in definition comprises a division of a number of vertical pixels and a number of horizontal pixels of each image by a factor of between 3 and 7.

[0019] According to yet another variant of the method, the first and second sets of pixels comprise only pixels representative of static objects in the three-dimensional scene.

[0020] According to a second aspect, the present invention relates to a device for processing images from a learning database of a depth prediction system associated with a vision system on board a vehicle, the device comprising a memory associated with at least one processor configured for implementing the steps of the method according to the first aspect of the present invention.

[0021] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0022] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0023] Such a computer program may use any programming language and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0024] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

[0025] On the one hand, the recording medium can be any entity or device capable of storing the program. For example, the medium may include a storage medium, such as a ROM memory, a CD-ROM or a microelectronic circuit-type ROM memory, or a magnetic recording medium or a hard disk.

[0026] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.

[0027] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures

[0028] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 6, in which:

[0029] [Fig-1] schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting example of embodiment of the present invention;

[0030] [Fig.2] illustrates a flowchart of the different stages of a treatment process of images from a training database of a depth prediction model associated with the vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention;

[0031] [Fig.3] illustrates a diagram representing the evolution of the database during the image processing method illustrated in [Fig.2], according to a particular and non-limiting exemplary embodiment of the present invention;

[0032] [Fig.4] schematically illustrates a device configured to process images of a training database of a depth prediction system associated with the on-board vision system in the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention;

[0033] [Fig.5] schematically illustrates different areas of an image acquired by the camera embedded in the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention; and

[0034] [Fig.6] illustrates a diagram representing a distribution of photo error values metric depending on different situations in which the vehicle of [Fig.l] finds itself, according to a particular and non-limiting exemplary embodiment of the present invention. Description of examples of implementation

[0035] A method and a device for processing images from a training database of a depth prediction model associated with a vision system embedded in a vehicle will now be described in what follows with joint reference to FIGS. 1 to 6. The same elements are identified with the same reference signs throughout the description which follows.

[0036] The terms "first(s)", "second(s)" (or "first(s)", "second(s)"), etc. are used in this document by arbitrary convention to enable different elements (such as operations, means, etc.) implemented in the embodiments described below to be identified and distinguished. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0037] According to a particular and non-limiting example of embodiment of the present invention, a method for processing images from a training database of a depth prediction model associated with a vision system on board a vehicle, the training database comprising a sequence of images acquired by a camera of the vision system.

[0038] Indeed, the method comprises the selection of two consecutive images in the sequence corresponding to a reference image and to a candidate image and the application of a determined mask to the reference image to select a first set of pixels in the reference image and to the candidate image to select a second set of pixels in the candidate image.

[0039] A photometric error is determined by comparing the sets of pixels and the candidate image is removed from the image sequence when the photometric error is less than a threshold value.

[0040] Such a method thus makes it possible to obtain a database adapted to the road environment and to the objects encountered by the vehicle when it is traveling. Indeed, the types of object and the images associated with these objects are acquired by the vision system of a vehicle and are therefore representative of real scenes taking place around a vehicle and as perceived by such an on-board vision system.

[0041] The application of the mask makes it possible to select parts of images acquired by the vision system in which there is little chance that a moving object is located in a three-dimensional scene, the mask then being considered as a mask of static objects.

[0042] Comparing the photometric error makes it possible to compare consecutive images and to eliminate those that would be acquired from the same position, i.e. when the vehicle carrying the vision system is not moving. Indeed, when the vehicle is not moving, the acquired images are similar apart from the pixels corresponding to moving objects. The photometric error is then almost zero.

[0043] [Fig. 1] schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.

[0044] The vehicle 10 is located in an environment 1 corresponding, for example, to a road environment formed of a network of roads accessible to the vehicle 10.

[0045] In this example, the vehicle 10 corresponds to a vehicle with a thermal engine, with an electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle 10 thus corresponds, for example, to a land vehicle such as an automobile, a truck, a bus, a motorcycle. Finally, the vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.

[0046] The vehicle 10 advantageously comprises at least one on-board camera 11, configured to acquire images of a three-dimensional scene taking place in the environment of the vehicle 10 from a current observation position. The camera 11 forms a monocular vision system if it is used alone as illustrated in [Fig.l]. The present invention is however not limited to a monocular vision system comprising a single camera but extends to any vision system comprising at least one camera, for example 1, 2, 3 or 5 cameras.

[0047] The camera 11 has intrinsic parameters, in particular: - a focal length, - distortions which are due to imperfections in the optical system of the camera 11, - a direction of the optical axis of the camera 11, and - a resolution.

[0048] The intrinsic parameters characterize the transformation which associates, for an image point, subsequently called “point”, its three-dimensional coordinates in the frame of reference of the camera 11 with the pixel coordinates in an image acquired by the camera 11. These parameters do not change if the camera is moved.

[0049] The distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of the camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions which generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, that is to say that the distortions are not taken into account, that their correction is processed at the time of image acquisition or at the time of calibration.

[0050] The camera 11 is arranged so as to acquire an image of a three-dimensional scene according to a defined point of view, the point of view is for example located on or in the left rearview mirror of vehicle 10 or at the top of the windshield of vehicle 10 as shown in [Fig.l].

[0051] The camera 11 for example acquires images of a three-dimensional scene located in front of the vehicle 10, the first camera 11 covering an acquisition field 12. An object 13 is placed in the acquisition field 12 of the camera 11, defining an occlusion field 14 for the vision system, an object present in the occlusion field 14 not being observable by the camera 11 from its current observation position.

[0052] It is obvious that it is possible to use such a vision system to take images of scenes located on the sides or behind the vehicle 10 by equipping it with cameras placed and oriented differently, the invention not being limited to the observation of a three-dimensional scene taking place in front of the vehicle 10 carrying the camera 11.

[0053] According to a particular embodiment, the camera 11 is of the “wide-angle” type, a wide-angle camera being for example equipped with a lens designed to acquire an image representative of a three-dimensional scene perceived according to a wider field of vision than that of a standard camera, also sometimes called a panoramic lens. In other words, a wide-angle lens makes it possible to capture a larger portion of the three-dimensional scene taking place in front of or around the camera, which is particularly useful in situations where it is necessary to include more elements in the frame of the image acquired by this camera. The angle a of the field of vision of the camera 11 is for example equal to 120°, 145°, 180° or 360°, whereas a standard camera offers, for example, an open field of vision following an angle of 45° or less.Such a camera 11 corresponds, for example, to a camera equipped with mirrors or even to a "fisheye" camera (in French "fish eye"). Wide-angle lenses have a shorter focal length compared to standard lenses, which makes them suitable for acquiring images of landscapes, architecture, road intersections or any other subject requiring an extended perspective. Wide-angle cameras are, for example, used to capture immersive and dynamic images with an extended depth of field.

[0054] An image acquired by the camera 11 at an acquisition time instant is presented in the form of data representing pixels characterized by: - coordinates in the image; and - data relating to the colors and brightness of objects in the observed scene in the form, for example, of RGB colorimetric values ​​(from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness).

[0055] Each pixel of the acquired image is representative of an object in the three-dimensional scene. sional present in the field of vision of the camera 11. Indeed, a pixel of the acquired image is the smallest visible unit and corresponds to a luminous point resulting from the emission or reflection of light by a physical object present in the three-dimensional scene. When the light hits this object, photons are emitted or reflected, captured by a photosensitive sensor of the camera 11 after passing through its lens. This sensor divides the three-dimensional scene into a grid of pixels. Each pixel records the light intensity at a specific location, thus capturing visual details. The combination of millions of pixels creates an image faithfully representing the physical object observed by the camera 11. An image point previously presented is thus a point on a surface of an object in the three-dimensional scene.

[0056] When the vehicle 10 is in motion, then two images acquired by the camera 11 at two distinct time instants represent views of the same three-dimensional scene taken from different viewpoints or observation positions, the observation positions of the camera 11 being distinct. On this three-dimensional scene are for example: - buildings; - road infrastructure; - other users or stationary objects, for example a parked vehicle; and / or - other users or moving objects, for example another vehicle, a cyclist or a moving pedestrian.

[0057] According to a particular exemplary embodiment, an image acquired by the camera 11 comprises a distortion equal to 0.5%, 0.8% or greater than 1%. The measurement of such a distortion corresponds to the determination of a ratio between: - the maximum spacing of a pixel of the image from a straight line of the first three-dimensional scene whose image is a line touching the longest edge of the first image, either at the center of the edge of the image, or at the corners of the edge of the image, and - the length of this edge.

[0058] Commonly, distortion is considered, in the world of photography, as: • negligible if it is less than 0.3%, • not very sensitive if it is between 0.3% or 0.4%, • sensitive if it is between 0.5% and 0.6%, • very sensitive if it is between 0.7% and 0.9%, and • annoying if it is greater than or equal to 1% or more.

[0059] A barrel distortion is characterized by a positive percentage, while a crescent distortion is characterized by a negative percentage.

[0060] Each image is for example sent to a processor, for example a computer of a device equipping the vehicle 10, or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

[0061] The stored images then constitute a training database for a depth prediction model associated with a vision system on board the vehicle 10. The training database thus comprises a sequence of ordered images acquired at consecutive time instants by the camera 11, such a sequence of images constituting, for example, a video.

[0062] [Fig. 2] illustrates a flowchart of the different steps of a method for processing images from a training database of a depth prediction model associated with the vision system on board a vehicle, for example from the training database comprising the images acquired by the camera 11 on board the vehicle 10 of [Fig. 1], according to a particular and non-limiting exemplary embodiment of the present invention.

[0063] According to a first particular exemplary embodiment, this method is advantageously implemented in the vehicle 10, that is to say by a processor, a computer or a combination of computers of the on-board system of the vehicle 10, for example by the computer(s) in charge of the vision system of the vehicle 10.

[0064] According to a second particular exemplary embodiment, this method is advantageously implemented by a server remote from the vehicle 10, which has received the images acquired by the camera 11 on board the vehicle 10.

[0065] According to a particular exemplary embodiment, the method comprises a step of reducing the definition of each image of the sequence of images. Indeed, reducing the definition of the images allows a saving of time during the processing of these images. A high definition of the images is, within a certain limit, not important because this method makes it possible to filter images, that is to say to delete images that are not necessary or that disturb a method of learning the depth prediction model, and not to determine depths associated with pixels of these images. Thus, reducing the definition of these images comprises a division of a number of vertical pixels and a number of horizontal pixels of each image by a factor of, for example, between 3 and 7.Thus, an image with a definition of two thousand pixels by two thousand pixels (2000x2000) is, for example, reduced to an image of three hundred pixels per side (300x300), five hundred pixels per side (500x500) or seven hundred pixels per side (700x700).

[0066] It should be noted that in the case where the definition of the images is reduced, this is for example reduced for images recorded in a first database for the implementation of the method while the original images are saved in a second database so as to then apply the selection or the filter. applied to the images of the first database to the images of the second database. Thus, the image processing process of the training database is executed quickly on “low” definition images while the “high” definition images are preserved for the subsequent implementation of the training process.

[0067] In a step 21 called step “A”, two consecutive images are selected from the sequence of images, the two images corresponding to a reference image and a candidate image, the reference image preceding the first candidate image in the sequence.

[0068] During a first iteration of step A, the reference image is, for example, the first image of the sequence of images, that is to say the oldest image among the images of the sequence of images acquired by the camera 11. The candidate image is then the image acquired just after, that is to say the second oldest image among the images of the sequence of images acquired by the camera 11.

[0069] In subsequent iterations of step A, the reference image is an image in the sequence of images and the candidate image is the image following the reference image.

[0070] In a step 22, called step “B”, a determined mask is applied to the reference image to select a first set of pixels in the reference image. Such a mask is determined so as to minimize any risk of inclusion, among the pixels of the first set of pixels, of pixels corresponding to a moving object in the three-dimensional scene. Such a mask is in particular determined as a function of the position of the camera 11 in the vehicle 10 and of the environment of the vehicle 10.

[0071] [Fig. 5] illustrates different areas of an image acquired by the camera 11, for example when it is positioned at the top of the windshield of the vehicle 10. The mask is for example determined so as to exclude: • a first zone 51 comprising pixels corresponding to the sky, • a second zone 52 comprising pixels corresponding to a part of the vehicle 10, here the front hood of the vehicle 10, this part of the vehicle 10 always being static in the images acquired by the camera 11, • a third zone 53 comprising pixels corresponding to locations in the three-dimensional scene in which moving objects such as other users are likely to be present. Thus, the mask makes it possible to process only the pixels present, for example, in the fourth zone 54. This corresponds, in this example, to the space in front of the vehicle 10 in which no other user should be present. The pixels therefore correspond, for example, to the road surface or the surface on which the vehicle 10 is moving.

[0072] Thus, the mask is determined as a function of the type of vehicle 10 and the position of the camera 11. It is also determined, for example, as a function of a type of road environment in which the vehicle 10 is moving. A type of road environment belongs, for example, to a set of types of road environments comprising: - a city center in an agglomeration, - an agglomeration (outside the city center), - a departmental or national road in a rural area, - a road with regulated access, and - a motorway. Indeed, another user is likely to be positioned in front of vehicle 10 at a distance of less than five meters (5m) in an urban environment, whereas no user is likely to be positioned at this distance from vehicle 10 when the latter is traveling on the motorway.

[0073] It should be noted that the mask covers one or more areas in an image depending on its determination. Thus, the example illustrated in [Fig.5] comprises a mask covering a single continuous area 54, the invention however not being limited to this example, the mask being able to cover two, three or more distinct areas.

[0074] In a step 23, the mask is applied to the candidate image to select a second set of pixels in the candidate image. The mask applied to the candidate image is the same as that applied to the reference image. Thus, the number of pixels in the first set of pixels is equal to the number of pixels in the second set of pixels. A pixel in the candidate image corresponding to a pixel in the reference image is a pixel having the same coordinates in the candidate image as the pixel in the reference image. If a pixel in the reference image is included in the first set of pixels, then the pixel corresponding to it in the candidate image is included in the second set of pixels and if a pixel in the reference image is not included in the first set of pixels, then the pixel corresponding to it in the candidate image is also not included in the second set of pixels.

[0075] According to the determination of the mask, the first and second sets of pixels comprise only pixels representative of static objects in the three-dimensional scene.

[0076] In a step 24, a photometric error is determined by comparing the first set of pixels to the second set of pixels.

[0077] According to a particular exemplary embodiment, the photometric error is determined by the following function: [Math.l]

[0078] With: • photometric error, • Z(p) a colorimetric value of a pixel P of the first set of pixels, P being defined by its two-dimensional coordinates, • a colorimetric value of the pixel P of the second set of pixels, • SSIM a function that takes into account a local structure, and • H a weighting factor depending in particular on the type of environment in which the vehicle operates (10).

[0079] This function is known to those skilled in the art and is for example presented in the document “Digging Into Self-Supervised Monocular Depth Estimation” written by Clément Godard, Oisin Mac Aodha, Michael Firman and Gabriel Brostow and published in August 2019.

[0080] According to another particular embodiment, the photometric error is determined by the following function:

[0081] [Math.2] EJ | / (p)-î(p) | ]

[0082] With: • z^photometric error, • I(p) a colorimetric value of a pixel P of the first set of pixels, P being defined by its two-dimensional coordinates, and • j^pj a colorimetric value of pixel P of the second set of pixels.

[0083] Thus, each pixel of the first set of pixels is compared to the pixel of the second set of pixels corresponding to it, i.e. having the same coordinates in the candidate image as the pixel in the reference image.

[0084] In a first situation, if the vehicle is static, then the pixels corresponding to the static objects in the three-dimensional scene are identical in the reference image and in the candidate image, that is to say, for example, the colorimetric values ​​associated with them are equal. In this case, the photometric error determined for a pixel corresponding to this static object is zero.

[0085] Even if the mask is determined so as to minimize the chances that a moving object is present in the field of vision of the camera 11 associated with the pixels covered by the mask, it remains possible that a moving object is present there in certain images. In this second situation, pixels of the first set and second set of pixels correspond to this moving object. This object being moving and the reference and candidate images being acquired at two distinct time instants, then the pixels corresponding to the moving object are not at the same position or at the same time. same coordinates in the reference and candidate images. Comparing the corresponding pixels two by two then generates photometric errors when at least one of the two corresponding pixels corresponds to the moving object.

[0086] Conversely, if the vehicle 10 is mobile in a third situation, the pixels of the first and second sets of pixels correspond mainly to static objects of the three-dimensional scene, therefore in relative motion with respect to the mobile vehicle 10. Thus, from one image to another, the values ​​of the pixels corresponding to these static objects in relative motion with respect to the mobile vehicle 10 change constantly. The photometric error for each pixel included in the mask is then significant, its value however depending on the appearance as well as the texture associated with the static object corresponding to the pixel and / or the speed of movement of the vehicle 10.

[0087] A threshold value is then defined to determine whether the photometric error is representative of a situation in which the vehicle 10 is stationary, i.e. immobile relative to the three-dimensional scene, or in motion, i.e. mobile relative to the three-dimensional scene.

[0088] The threshold value is defined so as to identify two scenarios, a first in which the vehicle 10 is stationary, this first scenario corresponding to the first and second situations, and a second scenario where the vehicle 10 is in motion and corresponding to the third situation. Indeed,: • in the first situation, the photometric error is zero or almost zero, • in the second situation, the photometric error is small but not zero, and • in the third situation, the photometric error is significant and greater than that determined in the second situation. It is then possible to define the threshold value so that it lies between a photometric error determined in the second situation and a photometric error determined in the third situation.

[0089] [Fig. 6] illustrates distributions of photometric error values ​​Lp as a function of the different situations presented above. A first set 61 of values ​​comprises the photometric error values ​​Lp determined in the first situation and a second set 62 of values ​​comprises the photometric error values ​​Lp determined in the second situation. The threshold value e must then be defined to be greater than the values ​​included in these two sets 61, 62 and thus encompass these two situations corresponding to a vehicle 10 at a standstill.

[0090] A third set 63 of values ​​comprises the photometric error values ​​Lp determined in the third situation, i.e. when the vehicle 10 is in motion. The threshold value e must then be defined to be lower than the values ​​included in this third set 63.

[0091] The photometric error previously determined by comparing the first and second sets of pixels is then compared to the threshold value.

[0092] In a step 25, the candidate image is deleted from the image sequence when the photometric error is less than the threshold value for updating the image sequence, called the updated image sequence. Indeed, if the photometric error is less than the threshold value, this corresponds to a situation in which the vehicle 10 is not moving. The candidate image must then be deleted because learning the prediction model from the reference and candidate images would be a source of error. Indeed, a depth predicted from two images taken consecutively when the vehicle 10 is stationary has a high chance of being erroneous.Furthermore, if the depth prediction model is coupled to a vehicle movement prediction model 10, i.e. a model predicting the movement of the vehicle between the time instants corresponding to the acquisitions of the reference and candidate images, then the movement of the vehicle predicted between these two time instants is itself erroneous.

[0093] According to a particular exemplary embodiment, steps C, D and E are then repeated as long as the database presents images following the reference and candidate images and as long as the photometric error is less than the threshold value. A candidate image is then selected again, the candidate image being the image corresponding to the image following the reference image in the updated image sequence. This is then different from the candidate image which was deleted during step 25. It corresponds to the image acquired by the camera 11 at the time instant following the time instant of acquisition of the reference image in the updated image sequence.

[0094] Thus, following several iterations of steps C, D and E, all the images corresponding to the images following the reference image and being identified as acquired when the vehicle 10 is stationary, that is to say acquired from the same point of view as that of the reference image, are deleted from the database.

[0095] According to yet another particular exemplary embodiment, steps B, C, D and E are repeated as long as the database presents images following the reference and candidate images and when the photometric error is greater than the threshold value. A new pair of reference and candidate images is selected from the updated database. A reference image is selected and corresponds to the previously selected candidate image, and a candidate image is selected and corresponds to the image following the newly selected reference image in the updated sequence of images.

[0096] Thus, the new reference image corresponds to an image different from the initial reference image, the vehicle having moved between the times of acquisition of these two images.

[0097] The database is then scanned in its entirety so as to keep only the images acquired when the vehicle 10 is moving and to delete any image acquired from the same point of view as an image already acquired, i.e. acquired when the vehicle 10 is stationary.

[0098] [Fig.3] illustrates a diagram representing the evolution of the database during the image processing method, according to a particular and non-limiting exemplary embodiment of the present invention.

[0099] Initially, the database 3a comprises, in this example, nine images acquired by the camera 11 on board the vehicle 10. When initializing the image processing method, the image 31 is selected as the reference image and the image 32 is selected as the candidate image. The image 31 corresponds to the oldest image acquired by the camera 11 and the image 32 corresponds to that acquired at the time instant following the time instant of acquisition of the closest first image 31. After applying the mask to the two images 31, 32, the pixels of the first set of pixels of the image 31 selected as the reference image are compared to the pixels of the second set of pixels of the image 32 selected as the candidate image. The photometric error determined in this example is less than the threshold value. The image 32 is then deleted from the database 3a to obtain the updated database 3b.

[0100] The method is then repeated with the updated database 3b becoming database 3b, the reference image still being image 31 and the candidate image now corresponding to image 33. The pixels of the first set of pixels are then unchanged while the pixels of the second set of pixels now correspond to pixels of image 33 selected as the candidate image. As before, in this example, the photometric error is less than the threshold value. Image 33 is then deleted from database 3b to obtain the updated database 3c.

[0101] The method is repeated with the updated database 3c becoming database 3c, the reference image still being image 31 and the candidate image now corresponding to image 34. The pixels of the first set of pixels are then unchanged while the pixels of the second set of pixels now correspond to pixels of image 34 selected as the candidate image. This time, in this example, the photometric error is greater than the threshold value. Image 34 is then selected as the new reference image, with database 3c remaining unchanged.

[0102] The method is repeated with the database 3c, the reference image now being image 34 and the candidate image now corresponding to image 35. The pixels of the first set of pixels are then those of image 34 included in the mask while the pixels of the second set of pixels now correspond to pixels of image 35 selected as the candidate image. As before, in this example, the photometric error is greater than the threshold value. Image 35 is then selected as the new reference image, the 3c database remaining unchanged.

[0103] The method is repeated with the database 3c, the reference image now being image 35 and the candidate image now corresponding to image 36. The pixels of the first set of pixels are then those of image 35 included in the mask while the pixels of the second set of pixels now correspond to pixels of image 36 included in the mask, image 36 being selected as the candidate image. The photometric error determined in this example is less than the threshold value. Image 36 is then deleted from the database 3c to obtain the updated database 3d.

[0104] The method is repeated with the updated 3d database becoming the 3d database, the reference image still being image 35 and the candidate image now corresponding to image 37. The pixels of the first set of pixels are then unchanged while the pixels of the second set of pixels now correspond to pixels of image 37 selected as the candidate image. As before, in this example, the photometric error is less than the threshold value. Image 37 is then deleted from the 3d database to obtain the updated 3e database.

[0105] The method is repeated with the updated database 3e becoming database 3e, the reference image still being image 35 and the candidate image now corresponding to image 38. The pixels of the first set of pixels are then unchanged while the pixels of the second set of pixels now correspond to pixels of image 38 selected as the candidate image. As before, in this example, the photometric error is less than the threshold value. Image 38 is then deleted from database 3e to obtain the updated database 3f.

[0106] The method is repeated with the updated database 3f becoming database 3f, the reference image still being image 35 and the candidate image now corresponding to image 39. The pixels of the first set of pixels are then unchanged while the pixels of the second set of pixels now correspond to pixels of image 39 selected as the candidate image. This time, in this example, the photometric error is greater than the threshold value. Image 39 is then retained in database 3f and since image 39 is the last image in database 3f, the image processing method ends.

[0107] The initial database 3a was thus filtered to obtain the final database 3f comprising only images acquired at different positions of the vehicle 10, the vehicle 10 being in motion between the time instants of acquisition of these different images 31, 34, 35 and 39 constituting the updated database or learning database.

[0108] According to a particular exemplary embodiment, the depth prediction model is trained from the updated database. The training of such a depth prediction model is known in particular to those skilled in the art. Indeed, many depth prediction models are implemented by convolutional neural networks. Each convolutional neural network is then trained, that is to say its input parameters are adjusted, for example by minimizing loss functions, for example determined from reconstruction errors or from photometric errors.Training such a depth prediction model from the learning database is also called self-supervision and is known to those skilled in the art, for example through the use of algorithms such as monodepth2® presented in the document “Digging Into Self-Supervised Monocular Depth Estimation”, or Depthmotion® presented in the document “Unsupervised Learning of Depth and Ego-Motion from Video” written by Tinghui Zhou, Matthew Brown, Noah Snavely and David G. Lowe and published in August 2017, for a prediction model associated with a monocular vision system.

[0109] This learning database is thus used to adjust the parameters of the depth prediction model associated with the vision system embedded in the vehicle, making the predicted depths more reliable and increasing the robustness of the depth prediction model. Indeed, since the images in the learning database have been filtered, the learning process will not be disturbed by error source data such as images acquired at different time instants but from the same point of view or images comprising disruptive dynamic objects. The learning of the depth prediction model is thus made more reliable.

[0110] If an AD AS uses depths predicted by this prediction model as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another user present on the road, the ADAS is then able to determine this distance precisely. For example, if the ADAS has the function of acting on a braking system of the vehicle 10 in the event of a risk of collision with another road user and the distance separating the vehicle 10 from this same road user decreases significantly, then the ADAS is able to detect this sudden approach and act on the braking system of the vehicle 10 to avoid a possible accident. The safety of users is thus improved.

[0111] This database can further be shared, for example by saving it on a remote server accessible to a plurality of vehicles carrying a similar vision system. Each of these vehicles then benefits from a relevant learning database allowing the depth prediction model associated with the vision system it carries to be trained.

[0112] [Fig. 4] schematically illustrates a device 4 configured to process images from a training database of a depth prediction system associated with a vision system embedded in a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention. The device 4 corresponds for example to a device embedded in the first vehicle 10, for example a computer associated with the vision system.

[0113] The device 4 is for example configured for the implementation of the operations described with regard to figures 1 and 4 and / or steps described with regard to figures 2 and 3. Examples of such a device 4 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 4, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and / or in discrete components. The device 4 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.

[0114] The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0115] The computer code of the embedded software(s) comprising the instructions to be loaded and executed by the processor is for example stored in the 4L memory.

[0116] According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input / output ports.

[0117] According to a particular and non-limiting exemplary embodiment, the device 4 comprises a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 comprise one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); HDMI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French); - LIN interface (from the English “Local Interconnect Network”).

[0118] According to another particular and non-limiting exemplary embodiment, the device 4 comprises a communication interface 43 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 430. The communication interface 43 corresponds for example to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds for example to a wired network of the CAN (Controller Area Network) type, CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO / IEC 802-3 standard).

[0119] According to a particular and non-limiting exemplary embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch-sensitive or not, one or more speakers 450 and / or other peripherals 460 via the output interfaces 44, 45, 46 respectively. According to a variant, one or other of the external devices is integrated into the device 4.

[0120] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for acquiring training data for a depth prediction model for a vision system on board a vehicle, which would include secondary steps without departing from the scope of the present invention. The same would apply to a device configured for the implementation implementation of such a process.

[0121] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 4 of [Fig.4].

Claims

Claims

1. A method for processing images from a training database of a depth prediction model associated with a vision system embedded in a vehicle (10), the vision system comprising at least one camera (11) arranged to acquire an image of a three-dimensional scene in an environment of the vehicle (10), said training database comprising a sequence of ordered images acquired at consecutive time instants by the camera (11), said method being implemented by a processor and being characterized in that it comprises the following steps: A) selecting (21) two consecutive images in said sequence corresponding to a reference image and a candidate image, said reference image preceding said first candidate image in said sequence; B) applying (22) a determined mask to the reference image to select a first set of pixels in the reference image;C) applying (23) said determined mask to the candidate image to select a second set of pixels in the candidate image; D) determining (24) a photometric error by comparing said first set of pixels to said second set of pixels and comparing said photometric error to a threshold value; E) deleting (25) the candidate image from said image sequence when said photometric error is less than said threshold value for updating said image sequence, called the updated image sequence.;

2. The method of claim 1, further comprising repeating steps C, D and E by selecting as the candidate image the image corresponding to the image following the reference image in said updated image sequence, as long as the photometric error is less than the threshold value.

3. Method according to claim 1 or 2, further comprising a reiteration of steps B, C, D and E by selecting: • as reference image the candidate image, and • as candidate image the image corresponding to the image following the reference image in said updated sequence of images, when the photometric error is greater than the threshold value.

4. Method according to one of claims 1 to 3, for which an error photometric error is determined by the following function: L.= ^(la)-\I(p)-î(p)\ + a-[^^ With: • Lp* the photometric error, • I(p) a colorimetric value of a pixel P of the first set of pixels, P being defined by its coordinates in two dimensions, • a colorimetric value of the pixel P of the second set of pixels, • SSIM a function which takes into account a local structure, and • has a weighting factor depending in particular on the type of environment in which the vehicle is moving (10).

5. Method according to one of claims 1 to 4, which comprises a step of reducing the definition of each image of said sequence of images.

6. A method according to claim 5, wherein said reduction in definition comprises dividing a number of vertical pixels and a number of horizontal pixels of said each image by a factor of between 3 and 7.

7. Method according to one of claims 1 to 6, for which the first and second sets of pixels comprise only pixels representative of static objects in the three-dimensional scene.

8. Computer program comprising instructions for implementing the method according to any one of the preceding claims, when these instructions are executed by a processor.

9. Device (4) for processing images from a learning database of a depth prediction system associated with a vision system on board a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured for implementing the steps of the method according to any one of claims 1 to 7.

10. Vehicle (10) comprising the device (4) according to claim 9.

Images