Method and device for generating training images of a depth prediction model for a vision system.

Abstract

Method or device for generating training images of a depth prediction model for a vision system embedded in a vehicle, the vision system comprising two cameras arranged so as to each acquire an image of a three-dimensional scene from a different point of view. Indeed, the method comprises receiving data representative of two images acquired by the two cameras at the same time instant and generating two training images, each training image being obtained from one of the acquired images, by coding data representative of said training images, in: • a first channel obtained from an application of a first filter to an acquired image, • a second channel obtained from an application of a second filter to an acquired image, • a third channel obtained from a conversion into gray levels of an acquired image. Figure for the abstract: Figure 3

Description

Title of the invention: Method and device for generating training images of a depth prediction model for a vision system. Technical field

[0001] The present invention relates to methods and devices for generating training images of a depth prediction model for a vision system embedded in a vehicle, for example in a motor vehicle.

[0002] The present invention also relates to a method for training a depth prediction model for an on-board vision system in a vehicle from training images acquired by the on-board vision system. Technological background

[0003] Many modern vehicles are equipped with so-called AD AS (Advanced Driver-Assistance System). Such AD AS systems are passive and active safety systems designed to eliminate the element of human error in the driving of vehicles of all types. AD AS use advanced technologies to assist the driver while driving and thus improve their performance. AD AS use a combination of sensor technologies to perceive the environment around a vehicle, then provide information to the driver or act on certain vehicle systems.

[0004] There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control and automatic parking systems.

[0005] The AD AS embedded in a vehicle are supplied with data obtained from one or more embedded sensors such as, for example, cameras. These cameras make it possible in particular to detect and locate other road users or possible obstacles present around a vehicle in order, for example: - to adapt the vehicle's lighting according to the presence of other users; - to automatically regulate the vehicle speed; - to act on the braking system in the event of a risk of impact with an object.

[0006] A position of another user or of an obstacle is for example determined by a vision system comprising a model for predicting a depth or a distance. Such a model is for example learned using images, these images being obtained from a universal database, for example Kitti® or Sceneflow®. Kitti® for example provides images of a road environment in a city center, However, such a database does not include all the road environments in which a vehicle can operate. The training data is therefore unsuitable for training the prediction model for a vehicle traveling in other road environments.

[0007] The quality of the training of the prediction model is however very important, in fact, the depths or distances predicted by the prediction model represent for example distances at which the other users or obstacles present in the road environment of the vehicle equipped with the vision system and the AD AS are located. The proper functioning of the driving aid peripherals using this data therefore depends on the quality of the data emitted by the vision system. Summary of the present invention

[0008] An object of the present invention is to solve at least one of the problems of the technological background described above.

[0009] Another object of the present invention is to improve the relevance of the training data of a depth prediction model for an on-board vision system in a vehicle.

[0010] Another object of the present invention is to improve road safety, in particular by improving the operational safety of AD AS systems supplied by data obtained from a vision system on board the vehicle.

[0011] According to a first aspect, the present invention relates to a method for generating training images of a depth prediction model for a vision system embedded in a vehicle, the vision system comprising a set of cameras comprising a first camera and a second camera arranged so as to each acquire an image of a three-dimensional scene from a different point of view, the method comprises the following steps: - reception of first data representative of a first image acquired by the first camera and second data representative of a second image acquired by the second camera, the first and second data being acquired at the same time instant and each being coded in three channels; - generation of data representative of a third image and a fourth image by applying a contour extraction filter to the first and second images respectively, the data representative of the third and fourth images each being coded in a single channel; - generation of data representative of a fifth image and a sixth image by applying a color inversion filter to the third and fourth images respectively, the data representative of the fifth and sixth images each being coded in a single channel; - generation of data representative of a seventh image and an eighth image by converting respectively the first and second images into gray levels, the data representative of the seventh and eighth images each being coded in a single channel; - generation of a ninth image and a tenth image by coding data representative of the ninth and tenth images in: • a first channel obtained from the data representative of the third image and the data representative of the fourth image, respectively, • a second channel obtained from the data representative of the fifth image and the data representative of the sixth image, respectively, • a third channel obtained from the data representative of the seventh image and the data representative of the eighth image respectively.

[0012] The data representative of the ninth and tenth images are thus similar to the data representative of the first and second images and can thus be used for training the depth prediction model for a vision system embedded in a vehicle. The data representative of the ninth and tenth images then allow training of this model, promoting the consideration of geometric data linked in particular to the extracted contours and reducing the influence of the colors present in the first and second images.

[0013] According to a variant of the method, the data representative of the first and second images are coded on eight bits in each channel, each of the three channels being associated with a different color.

[0014] Such coding is generally associated with images whose colors are coded in RGB.

[0015] According to yet another variant of the method, the contour extraction filter assigns a value between 0 and 1 to each pixel of the third and fourth images.

[0016] According to a further variant of the method, the color inversion filter assigns a value between 0 and 1 to each pixel of the fifth and sixth images, the value being defined by the following function: c(ps) = 1 - c(pe) with : • c(ps) the value of a pixel of the fifth, respectively sixth, image, and • c(pe) the value of a pixel of the third, respectively fourth, image, a position of the pixel pe in the fifth, respectively sixth, image corresponding to a position of the pixel ps in the third, respectively fourth, image.

[0017] According to another variant of the method, a value of each pixel of the seventh and respectively eighth image is determined by the following function: c(ps) = mean (cR(pe) + cG(pe) + cB(pe)) / 255 with : • c(ps) the value of a pixel of the seventh, respectively eighth, image, • cR(pe) the value of a pixel pe in a first channel of the first, respectively second, image, a position of the pixel pe in the first, respectively second, image corresponding to a position of the pixel ps in the ninth, respectively tenth, image, • cG(pe) the value of the pixel pe in a second channel of the first, respectively second, image, and • cB(pe) the value of pixel pe in a third channel of the first, respectively second, image.

[0018] According to an additional variant of the method, - the representative data of the ninth image are determined by the following function: (c^pç) ; c2(p9); c3(p9)) = (255 xc(p3); 255 xc(p5); 255 xc(p7)) with : • Ci(p9) a value of a pixel p9 in a first channel of the ninth image, • c2(p9) a value of pixel p9 in a second channel of the ninth image, • c3(p9) a value of pixel p9 in a third channel of the ninth image, • c(p3) a value of a pixel p3 of the third image, a position of the pixel p3 in the third image corresponding to a position of the pixel p9 in the ninth image, • c(p5) a value of a pixel p5 of the fifth image, a position of the pixel p5 in the fifth image corresponding to the position of the pixel p9 in the ninth image, • c(p7) a value of a pixel p7 of the seventh image, a position of the pixel p7 in the seventh image corresponding to the position of the pixel p9 in the ninth image, and - the representative data of the tenth image are determined by the following function: (ci(pio); c2(p10); c3(p10)) = (255 xc(p4); 255 x (p6); 255 x (p8)), with : • Ci(pio) a value of a pixel p10 in a first channel of the tenth image, • c2(pio) a value of pixel p10 in a second channel of the tenth image, • c3(pio) a value of pixel p10 in a third channel of the tenth image, • c(p4) a value of a pixel p4 of the fourth image, a position of the pixel p4 in the fourth image corresponding to a position of the pixel p10 in the tenth image, • c(p6) a value of a pixel p6 of the sixth image, a position of the pixel p6 in the sixth image corresponding to the position of the pixel pi0 in the tenth image, and • c(p8) a value of a pixel p8 of the eighth image, a position of the pixel p8 in the eighth image corresponding to the position of the pixel pi0 in the tenth image.

[0019] According to yet another variant, the method further comprises a step of training the learning model from a first pair of images. comprising the first and second images and from a second pair of images comprising the ninth and tenth images.

[0020] According to a second aspect, the present invention relates to a device for generating learning images of a depth prediction model for a vision system embedded in a vehicle, the device comprising a memory associated with at least one processor configured for implementing the steps of the method according to the first aspect of the present invention.

[0021] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.

[0022] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.

[0023] Such a computer program may use any programming language and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0024] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which is recorded a computer program comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.

[0025] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording means or a hard disk.

[0026] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.

[0027] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures

[0028] Other features and advantages of the present invention will become apparent from the following: Description of the particular and non-limiting examples of embodiment of the present invention below, with reference to the appended figures 1 to 4, in which:

[0029] [Fig.l] schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention;

[0030] [Fig.2] illustrates a flowchart of the different steps of a method for generating training images of a depth prediction model for an on-board vision system in the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention;

[0031] [Fig.3] illustrates a set of images generated by a method for generating training images of a depth prediction model for an on-board vision system in the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention;

[0032] [Fig.4] schematically illustrates a device configured to generate training images of a depth prediction model for an on-board vision system in the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention. Description of examples of implementation

[0033] A method and a device for generating learning images of a depth prediction model for a vision system embedded in a vehicle will now be described in the following with joint reference to FIGS. 1 to 4. The same elements are identified with the same reference signs throughout the description which follows.

[0034] The terms "first(s)", "second(s)" (or "first(s)", "second(s)"), etc. are used in this document by arbitrary convention to enable different elements (such as operations, means, etc.) implemented in the embodiments described below to be identified and distinguished. Such elements may be distinct or correspond to a single element, depending on the embodiment.

[0035] According to a particular and non-limiting example of embodiment of the present invention, a method for generating learning images of a depth prediction model for a vision system on board a vehicle is for example implemented by a computer of the on-board system of the vehicle controlling this vision system.

[0036] The vision system comprises a set of cameras of at least two cameras comprising a first camera and a second camera arranged so as to each acquire an image of a three-dimensional scene from a different point of view.

[0037] Indeed, the method comprises receiving data representative of two images acquired by the first and second cameras at the same time instant and the generation of two training images, each training image being obtained from one of the acquired images, by coding data representative of said training images, in: • a first channel obtained from an application of a first filter to an acquired image, • a second channel obtained from an application of a second filter to an acquired image, • a third channel obtained from a grayscale conversion of an acquired image.

[0038] Such a method thus makes it possible to obtain training images in a format similar to a format of the first and second images and are thus usable for training the depth prediction model for a vision system embedded in a vehicle. The training images generated then allow training of this depth prediction model, promoting the taking into account of geometric data linked in particular to the extracted contours and reducing the influence of the colors present in the images acquired by the vision system.

[0039] [Fig. 1] schematically illustrates a vision system equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.

[0040] An environment 1 corresponds, for example, to a road environment formed of a network of roads accessible to the vehicle 10.

[0041] In this example, the vehicle 10 corresponds to a vehicle with a thermal engine, with an electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle 10 thus corresponds, for example, to a land vehicle such as an automobile, a truck, a bus, a motorcycle. Finally, the vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.

[0042] The vehicle 10 advantageously comprises a set of cameras comprising a first camera 11 and a second camera 12 on board, each configured to acquire images of a three-dimensional scene in the environment 1 of the vehicle 10. This set of cameras forms the vision system. Two cameras are illustrated in [Fig.l]. The present invention is however not limited to a vision system comprising two cameras but extends to any vision system comprising 2 or more cameras, for example 2, 3, 4 or 5 cameras.

[0043] The first 11 and second 12 cameras have known intrinsic parameters. These parameters consist in particular of: - the focal length fl of the first camera 11; - the focal length f2 of the second camera 12; - distortions which are due to imperfections in the optical system of each camera; - the direction Cl of the optical axis of the first camera 11; - the direction C2 of the optical axis of the second camera 12; and - the respective resolutions of cameras 11, 12.

[0044] The intrinsic parameters characterize the transformation which associates, for an image point, the camera coordinates with the pixel coordinates, in each camera. These parameters do not change if the camera is moved.

[0045] The distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of the camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions which generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, that is to say that the distortions are not taken into account or that their correction is processed at the time of image acquisition.

[0046] These first 11 and second 12 cameras are arranged so as to each acquire an image of a three-dimensional scene from a different point of view, the first point of view is for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10, the second point of view is for example located on or in the right rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. In the case where the two cameras are located at the top of the windshield of the vehicle, they are then placed at a certain distance. In this example, the first camera 11 is located at the top of the windshield of the vehicle 10, the second camera 12 is located in the right rearview mirror of the vehicle 10.

[0047] A first marker is associated with the first camera 11: - the direction of the y axis is defined by the position of the second camera 12, so as to place the second camera 12 on the y axis of the first camera 11. The distance B separating the two cameras 11, 12 is called the reference base (in English “baseline”) and the direction separating the two cameras 11, 12 is that of the y axis; - the direction of the x axis is defined orthogonal to that of the y axis and orthogonal to that of the optical axis Cl of the first camera 11; - the direction of the z axis is defined orthogonal to the directions of the x and y axes. The three axes x, y and z thus form an orthonormal reference frame.

[0048] The extrinsic parameters linked to the position of the cameras 11, 12 are the following parameters: - 3 translations in the x, y and z directions: Tx, Ty and Tz constituting the translation vector T; and - 3 rotations around the x, y and z axes: Rx, Ry and Rz, constituting the rotation matrix R.

[0049] The determination of the extrinsic parameters is carried out for example during the calibration of the vision system.

[0050] A stereoscopic vision system is a vision system comprising a plurality of cameras, for example the first camera 11 and the second camera 12. A main constraint of a stereoscopic vision system used in automobiles is, for example, the large distance between the two cameras. Indeed, to be able to cover a measurement range of 200 meters, the reference base must reach 60cm for the cameras commonly used in this field.

[0051] The first 11 and second 12 cameras acquire images of a three-dimensional scene located in front of the vehicle 10, the first camera 11 covering only a first acquisition field 13, the second camera 12 covering only a second acquisition field 14 and the first 11 and second 12 cameras both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic vision of the three-dimensional scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic vision of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic vision of the scene by the stereoscopic vision system composed of the first 11 and second 12 cameras.

[0052] An obstacle 18 is placed in the acquisition field of the cameras, for example in the third acquisition field 15. The presence of the obstacle 18 defines an occlusion field for the stereoscopic vision system composed here of the three fields 16, 17 and 19.

[0053] Among these three fields, field 16 is visible from the second camera 12. The part of the scene present in this field 16 is therefore observable using the monoscopic vision system composed of the second camera 12.

[0054] The field 17 is visible from the first camera 11. The part of the scene present in this field 17 is therefore observable using the monoscopic vision system composed of the first camera 11.

[0055] Finally, field 19 is not visible from any of the cameras. The part of the scene present in this field 19 is therefore not observable.

[0056] When the vehicle 10 is in motion, each of the first 11 and second 12 cameras forms a monoscopic vision system.

[0057] According to an exemplary embodiment, the directions C1, C2 of the optical axes representative of an orientation of the field of vision of each camera are oriented non-parallel so as to obtain the third acquisition field 15 of the environment 1 as wide as possible.

[0058] According to another exemplary embodiment, the directions C1, C2 of the optical axes representative of an orientation of the field of vision of each camera are oriented parallel.

[0059] It is obvious that it is possible to use such a vision system, stereoscopic or monoscopic, to take images of scenes located on the sides or behind the vehicle 10 by equipping it with cameras placed and oriented differently.

[0060] The images acquired by the first 11 and second 12 cameras at a given acquisition time instant are presented in the form of data representing pixels characterized by: - coordinates in each image; and - data relating to the colors and brightness of the objects in the observed scene in the form, for example, of RGB (from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness) colorimetric coordinates.

[0061] The images acquired by the first 11 and second 12 cameras at the same time instant represent views of the same scene taken from different viewpoints, the positions of the cameras being distinct. On this scene there are, for example, objects, each object belonging to a type of object such as: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.

[0062] These images are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.

[0063] A process for generating learning images of a depth prediction model for the vision system on board the vehicle 10 is advantageously implemented by the vehicle 10, i.e. by a computer or a combination of computers of the on-board system of the vehicle 10, for example by the computer(s) in charge of the vision system of the vehicle 10.

[0064] A predicted depth is for example a distance, expressed in meters, separating the vision system from an object present in a field of vision of the first 11 and / or the second 12 camera.

[0065] In a first operation, first data representative of a first image acquired by the first camera 11 and second data representative of a second image acquired by the second camera 12 are received. The first and second images are acquired by the first camera 11 and the second camera 12 respectively at the same time instant, the first and second data are then acquired at the same time instant.

[0066] The first and second data are each coded in three channels, corresponding for example to an RGB or TSL image coding model.

[0067] According to a particular exemplary embodiment, the data representative of the first and second images are coded on eight bits in each channel, each of the three channels being associated with a different color. A first channel is then associated with the color red, a second channel is associated with the color green and a third channel is associated with the color blue. The eight-bit coding then makes it possible to assign to any pixel of the first and second images a value associated with each color, in other words in each channel, between 0 and 255. Thus, any pixel of the first and second images, defined by its position in the first or second image respectively, comprises three values ​​making it possible to assign it a color, each of the values ​​being coded in one of the first, second and third channels.

[0068] In a second operation, data representative of a third image 31 and a fourth image are generated by applying a contour extraction filter to the first and second images respectively. Such an extraction filter is known to those skilled in the art; it corresponds, for example, to a Canny filter.

[0069] According to a first particular embodiment, a value of a pixel of the third 31 and fourth images corresponding to a contour in the first and second images respectively is for example coded in a single channel and equal to 1, corresponding for example to a white pixel. A value of a pixel of the third 31 and fourth images not corresponding to a contour in the first and second images respectively is for example coded in this same channel and equal to 0, corresponding for example to a black pixel. [Fig. 3] thus illustrates this particular embodiment for which any value of a pixel of the third 31 and fourth images then has a value equal to 0 or 1 coded in a single channel.

[0070] According to a second particular embodiment, the values ​​of the pixels of the third and fourth images are between 0 and 255 and are coded in a single channel. It is noted here that the second embodiment can be obtained either by using a contour extraction filter assigning a value between 0 and 255 to each pixel of the third and fourth images, or by applying the first embodiment and then multiplying by 255 the values ​​of the pixels of the third and fourth images previously generated.

[0071] In a third operation, data representative of a fifth image 32 and a sixth image are generated by applying a color inversion filter to the third 31 and fourth images respectively. The data representative of the fifth and sixth images are then each encoded in a single channel. A fifth image 32 thus obtained is illustrated in [Fig.3].

[0072] According to the first particular embodiment previously described, the color inversion filter assigns a value between 0 and 1 to each pixel of the fifth and sixth images, the value being defined by the following function: c(ps) = 1 - c(pe) with : • c(ps) the value of a pixel of the fifth 32, respectively sixth, image, and • c(pe) the value of a pixel of the third 31, respectively fourth, image, a position of the pixel pe in the fifth 32, respectively sixth, image corresponding to a position of the pixel ps in the third 31, respectively fourth, image. Any value of a pixel of the fifth 32 and sixth images then has a value equal to 0 or 1 coded in a single channel. The fifth 32 and sixth images then correspond respectively to a negative of the third 31 and fourth images, the black pixels of the third 31 and fourth images becoming white pixels in the fifth 32 and sixth images respectively and the white pixels of the third 31 and fourth images becoming black pixels in the fifth 32 and sixth images respectively.

[0073] According to the second particular embodiment, the color inversion filter assigns a value between 0 and 255 to each pixel of the fifth and sixth images, the value being defined by the following function: c(ps) = 255 - c(pe) with : • c(ps) the value of a pixel of the fifth 32, respectively sixth, image, and • c(pe) the value of a pixel of the third 31, respectively fourth, image, a position of the pixel pe in the fifth 32, respectively sixth, image corresponding to a position of the pixel ps in the third 31, respectively fourth, image.

[0074] In a fourth operation, data representative of a seventh image 33 and an eighth image are generated by converting the first and second images into grayscale respectively. The data representative of the seventh and eighth images are then each coded in a single channel. Such a seventh image 33 is for example illustrated in [Fig.3].

[0075] According to the first particular embodiment previously described, a value of each pixel of the seventh 33 and respectively eighth image is determined by the following function: c(ps) = mean (cR(pe) + cG(pe) + cB(pe)) / 255 with : • c(ps) the value of a pixel of the seventh, respectively eighth, image, • cR(pe) the value of a pixel pe in a first channel of the first, respectively second, image, a position of the pixel pe in the first, respectively second, image corresponding to a position of the pixel ps in the ninth, respectively tenth, image, • cG(pe) the value of the pixel pe in a second channel of the first, respectively second, image, and • cB(pe) the value of pixel pe in a third channel of the first, respectively second, image.

[0076] According to this first particular embodiment, any value of a pixel of the seventh 33 and eighth images then has a value between 0 or 1 coded in a single channel.

[0077] According to the second embodiment, a value of each pixel of the seventh 33 and respectively eighth image is determined by the following function: c(ps) = mean (cR(pe) + cG(pe) + cB(pe)) with : • c(ps) the value of a pixel of the seventh, respectively eighth, image, • cR(pe) the value of a pixel pe in a first channel of the first, respectively second, image, a position of the pixel pe in the first, respectively second, image corresponding to a position of the pixel ps in the ninth, respectively tenth, image, • cG(pe) the value of the pixel pe in a second channel of the first, respectively second, image, and • cB(pe) the value of pixel pe in a third channel of the first, respectively second, image.

[0078] According to this second embodiment, any value of a pixel of the seventh 33 and eighth images then has a value between 0 or 255 coded in a single channel.

[0079] In a fifth operation, a ninth image and a tenth image are generated by encoding data representative of the ninth and tenth images in: • a first channel obtained from the data representative of the third image and the data representative of the fourth image, respectively, • a second channel obtained from the data representative of the fifth image and the data representative of the sixth image, respectively, and • a third channel obtained from the data representative of the seventh image and the data representative of the eighth image respectively.

[0080] According to the first embodiment, the data representative of the ninth image are determined by the following function: (ci(p9); c2(p9); c3(p9)) = (255 xc(p3); 255 xc(p5); 255 xc(p7)) with : • Ci(p9) a value of a pixel p9 in a first channel of the ninth image, • c2(p9) a value of the pixel p9 in a second channel of the ninth image, • c3(p9) a value of the pixel p9 in a third channel of the ninth image, • c(p3) a value of a pixel p3 of the third image, a position of the pixel p3 in the third image corresponding to a position of the pixel p9 in the ninth image, • c(p5) a value of a pixel p5 of the fifth image, a position of the pixel p5 in the fifth image corresponding to the position of the pixel p9 in the ninth image, • c(p7) a value of a pixel p7 of the seventh image, a position of the pixel p7 in the seventh image corresponding to the position of the pixel p9 in the ninth image, and - the data representative of the tenth image are determined by the following function: (ci(pio); c2(pio); c3(pio)) = (255 xc(p4); 255 x (p6); 255 x (p8)), with: • Ci(pio) a value of a pixel pi0 in a first channel of the tenth image, • c2(pio) a value of the pixel pi0 in a second channel of the tenth image, • c3(pio) a value of the pixel pi0 in a third channel of the tenth image, • c(p4) a value of a pixel p4 of the fourth image, a position of the pixel p4 in the fourth image corresponding to a position of the pixel pi0 in the tenth image, • c(p6) a value of a pixel p6 of the sixth image, a position of the pixel p6 in the sixth image corresponding to the position of the pixel pi0 in the tenth image, and • c(p8) a value of a pixel p8 of the eighth image, a position of the pixel p8 in the eighth image corresponding to the position of the pixel p10 in the tenth image.

[0081] According to the second embodiment, the data representative of the ninth image are determined by the following function: (ci(p9); c2(p9); c3(p9)) = (c(p3); c(p5); c(p7)) with : • Ci(p9) a value of a pixel p9 in a first channel of the ninth image, • c2(p9) a value of the pixel p9 in a second channel of the ninth image, • c3(p9) a value of the pixel p9 in a third channel of the ninth image, • c(p3) a value of a pixel p3 of the third image, a position of the pixel p3 in the third image corresponding to a position of the pixel p9 in the ninth image, • c(p5) a value of a pixel p5 of the fifth image, a position of the pixel p5 in the fifth image corresponding to the position of the pixel p9 in the ninth image, • c(p7) a value of a pixel p7 of the seventh image, a position of the pixel p7 in the seventh image corresponding to the position of the pixel p9 in the ninth image, and - the representative data of the tenth image are determined by the following function: (ci(pio); c2(pio); c3(pio)) = (c(p4); (p6); (p8)), with : • Ci(pio) a value of a pixel p10 in a first channel of the tenth image, • c2(pio) a value of the pixel p10 in a second channel of the tenth image, • c3(pio) a value of the pixel p10 in a third channel of the tenth image, • c(p4) a value of a pixel p4 of the fourth image, a position of the pixel p4 in the fourth image corresponding to a position of the pixel p10 in the tenth image, • c(p6) a value of a pixel p6 of the sixth image, a position of the pixel p6 in the sixth image corresponding to the position of the pixel p10 in the tenth image, and • c(p8) a value of a pixel p8 of the eighth image, a position of the pixel p8 in the eighth image corresponding to the position of the pixel p10 in the tenth image.

[0082] The data representative of the ninth and tenth images then have a structure similar to the data representative of the first and second images, each pixel comprising values ​​coded on three channels, these values ​​being for example between 0 and 255.

[0083] According to a particular exemplary embodiment, in a sixth operation, data representative of the ninth and tenth images are transmitted, for example to a memory or to a computer associated with the vision system embedded in the vehicle 10, or to a remote server accessible to a plurality of vehicles embedding similar vision systems. The latter then benefit from relevant training images making it possible to train the depth prediction model that they embed.

[0084] According to a particular exemplary embodiment, in a seventh operation, the learning model is trained from a first pair of images comprising the first and second images and / or from a second pair of images comprising the ninth and tenth images.

[0085] The depth prediction model for the on-board vision system in the vehicle 10 predicts depths associated with the pixels of a pair of stereoscopic images. The data representative of the ninth and tenth images being similar to the data representative of the first and second images, this prediction model is then able to use as input data a first pair of stereoscopic images acquired by the first 11 and second 12 cameras or a second pair of training images generated from stereoscopic images acquired by the first 11 and second 12 cameras via the operations previously described, in particular if the depth prediction model uses the channels of the pair of input images consecutively, that is to say that the depth prediction model compares values ​​assigned to the same channel of the input images at a time.

[0086] The depth prediction model trained using image pairs generated according to the process previously described is then more sensitive to the geometric characteristics present in the images it receives as input, the geometric characteristics including in particular contours. Indeed, the ninth and tenth images are devoid of characteristics associated with colors, each of the first and second channels of these images containing only pixel values ​​associated with geometric characteristics, the contours, and the third channel containing pixel values ​​associated with a gray level, thus defining a monochrome image.

[0087] Training such a depth prediction model from previously generated training images is also called self-supervision and is known to those skilled in the art, for example through the use of algorithms such as UnOs® for a depth prediction model associated with a stereoscopic vision system.

[0088] These training images are thus used to adjust the parameters of the depth prediction model associated with the vision system on board the vehicle 10, making the predicted depths more reliable and increasing the robustness of the depth prediction model.

[0089] If an ADAS uses depths predicted by this prediction model as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another user present on the road, the ADAS is then able to determine this distance precisely. For example, if the ADAS has the function of acting on a braking system of the vehicle 10 in the event of a risk of collision with another road user and the distance separating the vehicle 10 from this same road user decreases significantly, then the ADAS is able to detect this sudden approach and act on the braking system of the vehicle 10 to avoid a possible accident, including if the other user is only seen by a single camera. The safety of the users is thus improved.

[0090] [Fig. 2] illustrates a flowchart of the different steps of a method 2 for generating training images of a depth prediction model for a vision system embedded in a vehicle comprising a first camera 11 and a second camera 12, for example in the vehicle 10 of [Fig. 1], according to a particular and non-limiting exemplary embodiment of the present invention. The method 2 is for example implemented by a device embedded in the vehicle 10 or by the device 4 of [Fig. 4].

[0091] In a step 21, first data representative of a first image acquired by the first camera 11 and second data representative of a second image acquired by the second 12 cameras are received. The first and second data are in particular acquired at the same time instant and are each coded in three channels.

[0092] In a step 22, data representative of a third image 31 and of a fourth image are generated by applying a contour extraction filter to the first and second images respectively, the data representative of the third and fourth images each being coded in a single channel.

[0093] In a step 23, data representative of a fifth image 32 and a sixth image are generated by applying a color inversion filter to the third 31 and the fourth images respectively, the data representative of the fifth and sixth images each being coded in a single channel.

[0094] In a step 24, data representative of a seventh image 33 and an eighth image are generated by converting respectively the first and second images into gray levels, the data representative of the seventh and eighth images each being coded in a single channel.

[0095] In a step 25, a ninth image and a tenth image are generated by coding data representative of the ninth and tenth images in: • a first channel obtained from the data representative of the third image and the data representative of the fourth image, respectively, • a second channel obtained from the data representative of the fifth image and the data representative of the sixth image, respectively, and • a third channel obtained from the data representative of the seventh image and the data representative of the eighth image respectively.

[0096] According to a variant, the variants and examples of the operations described in relation to figures 1 and 3 apply to the steps of method 2 of [Fig.2].

[0097] [Fig. 4] schematically illustrates a device 4 configured for the generation of training images of a depth prediction model for a vision system embedded in a vehicle, for example in the vehicle 10 of the first figure, according to a particular and non-limiting exemplary embodiment of the present invention. The device 4 corresponds for example to a device embedded in the first vehicle 10, for example a computer.

[0098] The device 4 is for example configured for the implementation of the operations described with regard to figures 1 and 3 and / or steps described with regard to [Fig.2]. Examples of such a device 4 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 4, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and / or in discrete components. The device 4 can be produced in the form of electronic circuits or software modules (or information matics) or a combination of electronic circuits and software modules.

[0099] The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.

[0100] The computer code of the embedded software(s) comprising the instructions to be loaded and executed by the processor is for example stored in the 4L memory.

[0101] According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input / output ports.

[0102] According to a particular and non-limiting exemplary embodiment, the device 4 comprises a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 comprise one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced; - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); HDMI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French); - LIN interface (from the English “Local Interconnect Network”).

[0103] According to another particular and non-limiting exemplary embodiment, the device 4 comprises a communication interface 43 which makes it possible to establish a communication with other devices (such as other computers of the on-board system) via a communication channel 430. The communication interface 43 corresponds for example to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds for example to a wired network of the CAN (Controller Area Network) type, CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO / IEC 802-3 standard).

[0104] According to a particular and non-limiting exemplary embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch-sensitive or not, one or more speakers 450 and / or other peripherals 460 (projection system) via the output interfaces 44, 45, 46 respectively. According to a variant, one or other of the external devices is integrated into the device 4.

[0105] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for generating training images of a depth prediction model for a vision system embedded in a vehicle from images acquired by the on-board vision system, which would include secondary steps without thereby departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.

[0106] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 4 of [Fig.4].

Claims

1. Claims Method for generating training images of a depth prediction model for a vision system embedded in a vehicle, the vision system comprising a set of cameras comprising a first camera (11) and a second camera (12) arranged so as to each acquire an image of a three-dimensional scene from a different point of view, said method comprising the following steps: - reception (21) of first data representative of a first image acquired by the first camera (11) and second data representative of a second image acquired by the second camera (12), said first and second data being acquired at the same time instant and each being coded in three channels; - generation (22) of data representative of a third image (31) and of a fourth image by applying a contour extraction filter to the first and second images respectively, said data representative of the third and fourth images each being coded in a single channel; - generation (23) of data representative of a fifth image (32) and a sixth image by applying a color inversion filter to the third (31) and fourth images respectively, said data representative of the fifth and sixth images each being coded in a single channel; - generation (24) of data representative of a seventh image (33) and an eighth image by converting respectively the first and second images into gray levels, said data representative of the seventh and eighth images each being coded in a single channel; and - generation (25) of a ninth image and a tenth image by coding data representative of said ninth and tenth images in: • a first channel obtained from the data representative of the third image and the data representative of the fourth image, respectively, • a second channel obtained from the data representative of the fifth image and the data representative of the sixth image, respectively, and • a third channel obtained from the data representative of the seventh image and the data representative of the eighth image respectively.

2. A method according to claim 1, wherein said data representative of the first and second images are coded on eight bits in each channel, each of said three channels being associated with a different color.

3. Method according to one of claims 1 to 2, for which the contour extraction filter assigns a value between 0 and 1 to each pixel of the third and fourth images.

4. Method according to claim 3, for which the color inversion filter assigns a value between 0 and 1 to each pixel of the fifth (32) and sixth images, said value being defined by the following function: c(ps) = 1 - c(pe) with: • c(ps) the value of a pixel of the fifth, respectively sixth, image, and • c(pe) the value of a pixel of the third (31), respectively fourth, image, a position of the pixel pe in the fifth, respectively sixth, image corresponding to a position of the pixel ps in the third, respectively fourth, image.

5. Method according to claim 4, for which a value of each pixel of the seventh (33) and respectively eighth image is determined by the following function: c(ps) = mean (cR(pe) + cG(pe) + cB(pe)) / 255 with: • c(ps) the value of a pixel of the seventh, respectively eighth, image, • cR(pe) the value of a pixel pe in a first channel of the first, respectively second, image, a position of the pixel pe in the first, respectively second, image corresponding to a position of the pixel ps in the ninth, respectively tenth, image, • cG(pe) the value of the pixel pe in a second channel of the first, respectively second, image, and • cB(pe) the value of the pixel pe in a third channel of the first, respectively second, image.

6. A method according to claim 5, wherein - the representative data of the ninth image are determined by the following function: (ci(p9); c2(p9); c3(p9)) = (255 xc(p3); 255 xc(p5); 255 xc(p7)) with : • Ci(p9) a value of a pixel p9 in a first channel of the ninth image, • c2(p9) a value of pixel p9 in a second channel of the ninth image, • c3(p9) a value of pixel p9 in a third channel of the ninth image, • c(p3) a value of a pixel p3 of the third image, a position of the pixel p3 in the third image corresponding to a position of the pixel p9 in the ninth image, • c(p5) a value of a pixel p5 of the fifth image, a position of the pixel p5 in the fifth image corresponding to the position of the pixel p9 in the ninth image, • c(p7) a value of a pixel p7 of the seventh image, a position of the pixel p7 in the seventh image corresponding to the position of the pixel p9 in the ninth image, and - the representative data of the tenth image are determined by the following function: (ci(pio); c2(p10); c3(p10)) = (255 xc(p4); 255 x (p6); 255 x (p8)), with : • Ci(pio) a value of a pixel p10 in a first channel of the tenth image, • c2(pio) a value of pixel p10 in a second channel of the tenth image, • c3(pio) a value of pixel p10 in a third channel of the tenth image, • c(p4) a value of a pixel p4 of the fourth image, a position of the pixel p4 in the fourth image corresponding to a position of the pixel p io in the tenth image, • c(p6) a value of a pixel p6 of the sixth image, a position of the pixel p6 in the sixth image corresponding to the position of the pixel p10 in the tenth image, and • c(p8) a value of a pixel p8 of the eighth image, a position of the pixel p8 in the eighth image corresponding to the position of the pixel p10 in the tenth image.

7. Method according to one of claims 1 to 6, further comprising a step of training the learning model from a first pair of images comprising the first and second images and from a second pair of images comprising the ninth and tenth images.

8. Computer program comprising instructions for implementing the method according to any one of the preceding claims, when these instructions are executed by a processor.

9. Device (4) for generating a learning image of a depth prediction model for a vision system on board a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured for implementing the steps of the method according to any one of claims 1 to 7.

10. Vehicle (10) comprising the device (4) according to claim 9.

Images