Method and device for determining depth by a pseudo-stereoscopic vision system.
The method improves depth determination in monoscopic vision systems by leveraging a stereoscopic-supervised prediction model and rectification techniques, enhancing ADAS systems' precision and safety.
Patent Information
- Application Number
- FR2023004623
- Authority / Receiving Office
- FR · FR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing monoscopic vision systems in vehicles lack the precision and accuracy needed for reliable depth perception, which is crucial for advanced driver assistance systems (ADAS) to function effectively.
A method and device for determining depth using a monoscopic vision system on a vehicle, involving a first camera cooperating with a second camera to acquire images from different viewpoints, utilizing a prediction model supervised by a stereoscopic system, and employing rectification and disparity analysis to achieve metric precision.
The method enhances the accuracy of depth calculation, providing coherent and metrically precise depth data for ADAS systems, improving road safety by enabling precise distance measurements between the vehicle and its environment.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Title of the invention: Method and device for determining a depth using a pseudo-stereoscopic vision system. Technical field
[0001] The present invention relates to methods and devices for determining a depth by means of a monoscopic vision system on board a vehicle, for example in a motor vehicle. The present invention also relates to a method and a device for measuring such a depth. The present invention also relates to a method and a device for controlling one or more AD AS systems on board a vehicle from the determined depth. Technological background
[0002] Many modern vehicles are equipped with so-called AD AS (Advanced Driver-Assistance System). Such AD AS systems are passive and active safety systems designed to eliminate the element of human error in the driving of vehicles of all types. AD AS use advanced technologies to assist the driver while driving and thus improve their performance. AD AS use a combination of sensor technologies to perceive the environment around a vehicle, then provide information to the driver or act on certain vehicle systems.
[0003] There are several levels of ADAS, such as rearview cameras and blind spot sensors, lane departure warning systems, adaptive cruise control and automatic parking systems.
[0004] The AD AS embedded in a vehicle are supplied with data obtained from one or more embedded sensors such as, for example, cameras. These cameras make it possible in particular to detect and locate other road users or possible obstacles present around a vehicle in order, for example: - to adapt the vehicle's lighting according to the presence of other users; - to automatically regulate the vehicle speed; - to act on the braking system in the event of a risk of impact with an object.
[0005] The proper functioning of the driving assistance peripherals using this data therefore depends on the quality of the data emitted by a vision system. Summary of the present invention
[0006] An object of the present invention is to solve at least one of the problems of the technological background described above.
[0007] Another object of the present invention is to improve the quality of the data obtained of these vision systems.
[0008] Another object of the present invention is to improve road safety.
[0009] According to a first aspect, the present invention relates to a method for determining a depth by a monoscopic vision system on board a moving vehicle, the monoscopic vision system comprising a first camera capable of cooperating with at least one second camera of a set of cameras of at least two cameras arranged so as to each acquire an image of a three-dimensional scene from a different point of view, the method being characterized in that it comprises the following steps: - reception of first and second data respectively representative of a first and second image acquired by the first camera of the set of cameras respectively at a first acquisition time instant and at a second acquisition time instant; - determination of third data representative of a movement of the first camera between the first acquisition time instant and the second acquisition time instant as a function of the first and second data; - prediction of first depths associated with a first set of pixels of the first image by the monoscopic vision system from a first prediction model learned and supervised by a stereoscopic system composed of the first camera and at least one second camera, the first depths being predicted based on the first data; - rectification of the first and second images according to the third data to obtain a first rectified image and a second rectified image; - determining disparity values associated with a second set of pixels of the first rectified image corresponding to a set of pixels of the second rectified image; and - prediction of second depths associated with the second set of pixels by a pseudo-stereoscopic vision system composed of the first moving camera from a second prediction model learned and supervised by the monoscopic vision system, the second depths being predicted as a function of the disparity values and as a function of a distance separating positions of the first camera at the first acquisition time instant and at the second acquisition time instant, the distance being determined from the third data.
[0010] The method thus makes it possible to calculate the depth of an object in the first image seen by the monoscopic vision system with the metric precision of the stereoscopic vision system.
[0011] According to a method variant, supervision of the second prediction model is obtained by minimizing a loss function defined by the following function: L^^D^p) -Dj^jwith: - DpS(p) the second predicted depth for a pixel P of the first image, and " ^m(p) the First predicted depth for a pixel P of the first image.
[0012] The supervision of the pseudo-stereoscopic vision system by the monoscopic vision system thus makes it possible to obtain coherent first and second depths.
[0013] According to another variant, the method further comprises the steps of: - prediction of third depths associated with a set of pixels of the second image by the monoscopic vision system from the first prediction model, the third depths being predicted based on the second data; - determination of a mask of dynamic objects associated with the first image and representative of a third set of pixels of the first image associated with a moving object in the three-dimensional scene, the learning of the second prediction model being further a function of the dynamic object mask.
[0014] The use of the dynamic object mask for learning the second prediction model thus makes it possible to exclude pixels associated with moving objects during the learning phase.
[0015] According to another variant, the method further comprises a step of determining a first visibility mask associated with the first image and representative of a fourth set of pixels of the first image having at least one corresponding pixel in the second image from the first and second data, the learning of the second prediction model furthermore being a function of the first visibility mask.
[0016] The use of the first visibility mask for training the second prediction model thus makes it possible to exclude pixels that do not have a corresponding pixel in the second image during the training phase.
[0017] According to an additional variant, the method further comprises the steps of: - reception of fourth data representative of a third image acquired by the at least one second camera at the first time instant of acquisition; - prediction of fourth depths associated with a fifth set of pixels of the first image by the stereoscopic vision system from the first and third data, the learning of the first prediction model being further a function of the fourth depths.
[0018] The supervision of the monoscopic vision system by the stereoscopic vision system thus makes it possible to adjust the depths calculated by the monoscopic vision system and to obtain the metric precision of the stereoscopic vision system for the monoscopic vision system.
[0019] According to another variant of the method, supervision of the first prediction model is obtained by minimizing a loss function defined by the following function: L^m = ( p) - Ds(p)[with: - Dm(p) the first predicted depth for a pixel P of the first image, and - Ds(p) the fourth predicted depth for a pixel P of the first image.
[0020] According to a variant, the method further comprises a step of determining a second visibility mask associated with the first image and representative of a sixth set of pixels of the first image having at least one corresponding pixel in the third image from the first and fourth data, the learning of the first prediction model furthermore being a function of the second visibility mask.
[0021] The use of the second visibility mask for training the first prediction model thus makes it possible to exclude aberrant depth values predicted by the stereoscopic vision system.
[0022] According to a second aspect, the present invention relates to a device for determining a depth by a monoscopic vision system on board a vehicle, the device comprising a memory associated with at least one processor configured for implementing the steps of the method according to the first aspect of the present invention.
[0023] According to a third aspect, the present invention relates to a vehicle, for example of the automobile type, comprising a device as described above according to the second aspect of the present invention.
[0024] According to a fourth aspect, the present invention relates to a computer program which comprises instructions adapted for executing the steps of the method according to the first aspect of the present invention, in particular when the computer program is executed by at least one processor.
[0025] Such a computer program may use any programming language and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.
[0026] According to a fifth aspect, the present invention relates to a computer-readable recording medium on which a program is recorded. computer system comprising instructions for carrying out the steps of the method according to the first aspect of the present invention.
[0027] On the one hand, the recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM memory, a CD-ROM or a microelectronic circuit type ROM memory, or a magnetic recording means or a hard disk.
[0028] Furthermore, this recording medium may also be a transmissible medium such as an electrical or optical signal, such a signal being able to be conveyed via an electrical or optical cable, by conventional or hertzian radio or by self-directed laser beam or by other means. The computer program according to the present invention may in particular be downloaded from an Internet-type network.
[0029] Alternatively, the recording medium may be an integrated circuit in which the computer program is incorporated, the integrated circuit being adapted to perform or to be used in performing the method in question. Brief description of the figures
[0030] Other characteristics and advantages of the present invention will emerge from the description of the particular and non-limiting exemplary embodiments of the present invention below, with reference to the appended figures 1 to 4, in which:
[0031] [Fig-1] schematically illustrates a vision system comprising a set of cameras of at least two cameras equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention;
[0032] [Fig.2] schematically illustrates a device configured for determining a depth by monoscopic vision system on board the vehicle of [Fig.l], according to a particular and non-limiting exemplary embodiment of the present invention;
[0033] [Fig.3] illustrates a flowchart of the different operations of a process for determining a depth by a monoscopic vision system on board the vehicle of [Fig.1], according to a particular and non-limiting exemplary embodiment of the present invention;
[0034] [Fig.4] illustrates a flowchart of the different steps of a method for determining a depth by a monoscopic vision system on board the vehicle of [Fig.1], according to a particular and non-limiting exemplary embodiment of the present invention.
[0035] Description of the examples of embodiment
[0036] A method and a device for determining a depth by means of a monoscopic vision system on board a vehicle will now be described in what follows: follow with reference to Figures 1 to 4. The same elements are identified with the same reference signs throughout the description which follows.
[0037] According to a particular and non-limiting example of embodiment of the present invention, a method for determining a depth by a monoscopic vision system on board a vehicle is for example implemented by a computer of the on-board system of the vehicle controlling this vision system.
[0038] The monoscopic vision system comprises a first camera capable of cooperating with at least one second camera of a set of cameras of at least two cameras arranged so as to each acquire an image of a three-dimensional scene from a different point of view.
[0039] For this purpose, the method for determining a depth by a monoscopic vision system on board a vehicle comprises receiving first and second data respectively representative of a first and second image acquired by the first camera of the set of cameras respectively at a first acquisition time instant and at a second acquisition time instant, determining third data representative of a movement of the first camera between the first acquisition time instant and the second acquisition time instant as a function of the first and second data and predicting first depths associated with a first set of pixels of the first image by the monoscopic vision system from a first prediction model learned and supervised by a stereoscopic system composed of the first camera and the at least one second camera.
[0040] The method also comprises rectifying the first and second images as a function of the third data to obtain a first rectified image and a second rectified image, determining disparity values associated with a second set of pixels of the first rectified image corresponding to a set of pixels of the second rectified image and predicting second depths associated with the second set of pixels by a pseudostereoscopic vision system composed of the first moving camera from a second prediction model learned and supervised by the monoscopic vision system, the second depths being predicted as a function of the disparity values and as a function of a distance separating positions of the first camera at the first acquisition time instant and at the second acquisition time instant, the distance being determined from the third data.
[0041] [Fig.l] schematically illustrates a vision system comprising a set of cameras of at least two cameras equipping a vehicle, according to a particular and non-limiting exemplary embodiment of the present invention.
[0042] Such an environment 1 corresponds, for example, to a road environment formed of a network of roads accessible to the vehicle 10.
[0043] In this example, the vehicle 10 corresponds to a vehicle with a thermal engine, with an electric motor(s) or even a hybrid vehicle with a thermal engine and one or more electric motors. The vehicle 10 thus corresponds, for example, to a land vehicle such as an automobile, a truck, a bus, a motorcycle. Finally, the vehicle 10 corresponds to an autonomous vehicle or not, that is to say a vehicle traveling according to a determined level of autonomy or under the total supervision of the driver.
[0044] The vehicle 10 advantageously comprises several on-board cameras 11, 12, each configured to acquire images of a scene in the environment of the vehicle 10. This set of cameras 11, 12 forms a stereoscopic vision system. Two cameras 11 and 12 are illustrated in [Fig.l]. The present invention is however not limited to a vision system comprising two cameras but extends to any vision system comprising 2 or more cameras, for example 2, 3, 4 or 5 cameras. Each camera 11, 12 also forms a monoscopic vision system.
[0045] The two cameras 11, 12 have known intrinsic parameters. These parameters consist in particular of: - the focal length fl of the first camera 11; - the focal length f2 of the second camera 12; - distortions which are due to imperfections in the optical system of each camera; - the direction Cl of the optical axis of the first camera 11; - the direction C2 of the optical axis of the second camera 12; and - the respective resolutions of cameras 11, 12.
[0046] The intrinsic parameters characterize the transformation which associates, for an image point, the camera coordinates with the pixel coordinates, in each camera. These parameters do not change if the camera is moved.
[0047] The distortions, which are due to imperfections in the optical system such as defects in the shape and positioning of the camera lenses, will deflect the light beams and therefore induce a positioning deviation for the projected point compared to an ideal model. It is then possible to complete the camera model by introducing the three distortions which generate the most effects, namely radial, decentering and prismatic distortions, induced by defects in curvature, parallelism of the lenses and coaxiality of the optical axes. In this example, the cameras are assumed to be perfect, that is to say that the distortions are not taken into account or that their correction is processed at the time of image acquisition.
[0048] These two cameras 11, 12 are arranged so as to each acquire an image of a scene from a different point of view, the first point of view being for example located on or in the left rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10, the second point of view is for example located on or in the right rearview mirror of the vehicle 10 or at the top of the windshield of the vehicle 10. In the case where the two cameras are located at the top of the windshield of the vehicle, they are then placed at a certain distance. In this example, the first camera 11 is located at the top of the windshield of the vehicle 10, the second camera 12 is located in the right rearview mirror of the vehicle 10.
[0049] A first marker is associated with the first camera 11: - the direction of the y axis is defined by the position of the second camera 12, so as to place the second camera 12 on the y axis of the first camera 11. The distance B separating the two cameras 11, 12 is called the reference base (in English “baseline”) and the direction separating the two cameras 11, 12 is that of the y axis; - the direction of the x axis is defined orthogonal to that of the y axis and orthogonal to that of the optical axis Cl of the first camera 11; - the direction of the z axis is defined orthogonal to the directions of the x and y axes. The three axes x, y and z thus form an orthonormal reference frame.
[0050] The extrinsic parameters linked to the position of the cameras 11, 12 are the following parameters: - 3 translations in the x, y and z directions: Tx, Ty and Tz constituting the translation vector T; and - 3 rotations around the x, y and z axes: Rx, Ry and Rz, constituting the rotation matrix R.
[0051] Determining the extrinsic parameters constitutes the problem of calibrating a stereoscopic vision system.
[0052] A main constraint of the stereoscopic vision system used in automobiles is, for example, the large distance between the two cameras. Indeed, to be able to cover a measurement range of 200 meters, the "baseline" must reach 60cm for the cameras commonly used in this field.
[0053] The two cameras 11, 12 acquire images of a scene located in front of the vehicle 10, the first camera covering only a first acquisition field 13, the second camera covering only a second acquisition field 14 and the two cameras 11, 12 both covering a third acquisition field 15. The first and third acquisition fields 13, 15 thus allow a monoscopic vision of the scene by the first camera 11, the second and third acquisition fields 14, 15 allow a monoscopic vision of the scene by the second camera 12 and the third acquisition field 15 allows a stereoscopic vision of the scene by the stereoscopic vision system composed of the two cameras 11, 12.
[0054] An obstacle 18 is placed in the acquisition field of the cameras, for example in the third acquisition field 15. The presence of the obstacle 18 defines an occlusion field for the stereoscopic vision system composed here of the three fields 16, 17 and 19.
[0055] Among these three fields, field 16 is visible from the second camera 12. The part of the scene present in this field 16 is therefore observable using the monoscopic vision system composed of the second camera 12.
[0056] The field 17 is visible from the first camera 11. The part of the scene present in this field 17 is therefore observable using the monoscopic vision system composed of the second camera 11.
[0057] Finally, field 19 is not visible from any of the cameras. The part of the scene present in this field 19 is therefore not observable.
[0058] The directions C1, C2 of the optical axes representative of an orientation of the field of vision of each camera are oriented non-parallel so as to obtain the third acquisition field 15 of the environment 1 as wide as possible.
[0059] According to a variant, the vision system is further configured to take images of scenes located on the sides or behind the vehicle 10 using differently placed and oriented cameras.
[0060] The images acquired by the cameras 11, 12 at a given acquisition time instant tl are presented in the form of data representing pixels characterized by: - coordinates in each image; and - data relating to the colors and brightness of the objects in the observed scene in the form, for example, of RGB (from the English “Red Green Blue”) or TSL (Tone, Saturation, Brightness) colorimetric coordinates.
[0061] The images acquired by the cameras 11, 12 represent views of the same scene taken from different viewpoints, the positions of the cameras being distinct. On this scene are found, for example: - buildings; - road infrastructure; - other stationary users, for example a parked vehicle; and / or - other mobile users, for example another vehicle, a cyclist or a moving pedestrian.
[0062] These images are sent to a computer of a device equipping the vehicle 10 or stored in a memory of a device accessible to a computer of a device equipping the vehicle 10.
[0063] [Fig. 2] schematically illustrates a device 4 configured for the determination of a depth by a monoscopic vision system on board a vehicle 10, according to a particular and non-limiting exemplary embodiment of the present invention. The device 4 corresponds for example to a device on board the first vehicle 10, for example a computer.
[0064] The device 4 is for example configured for the implementation of the operations and / or steps described with regard to figures 1, 3 and 4. Examples of such a device 4 include, but are not limited to, on-board electronic equipment such as an on-board computer of a vehicle, an electronic calculator such as an ECU (“Electronic Control Unit”), a smartphone, a tablet, a laptop. The elements of the device 4, individually or in combination, can be integrated in a single integrated circuit, in several integrated circuits, and / or in discrete components. The device 4 can be produced in the form of electronic circuits or software (or computer) modules or even a combination of electronic circuits and software modules.
[0065] The device 4 comprises one (or more) processor(s) 40 configured to execute instructions for carrying out the steps of the method and / or for executing the instructions of the software(s) embedded in the device 4. The processor 40 may include integrated memory, an input / output interface, and various circuits known to those skilled in the art. The device 4 further comprises at least one memory 41 corresponding for example to a volatile and / or non-volatile memory and / or comprises a memory storage device which may comprise volatile and / or non-volatile memory, such as EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic or optical disk.
[0066] The computer code of the embedded software(s) comprising the instructions to be loaded and executed by the processor is for example stored in the 4L memory.
[0067] According to various particular and non-limiting embodiments, the device 4 is coupled in communication with other similar devices or systems (for example other computers) and / or with communication devices, for example a TCU (from the English “Telematic Control Unit” or in French “Telematic Control Unit”), for example via a communication bus or through dedicated input / output ports.
[0068] According to a particular and non-limiting exemplary embodiment, the device 4 comprises a block 42 of interface elements for communicating with external devices. The interface elements of the block 42 comprise one or more of the following interfaces: - RF radio frequency interface, for example Wi-Fi® type (according to IEEE 802.11), for example in the 2.4 or 5 GHz frequency bands, or Bluetooth® type (according to IEEE 802.15.1), in the 2.4 GHz frequency band, or Sigfox type using UBN (Ultra Narrow Band) radio technology, or LoRa in the 868 MHz frequency band, LTE (Long-Term Evolution), LTE-Advanced (or in French LTE-advanced); - USB interface (from the English “Universal Serial Bus” or “Universal Serial Bus” in French); HD MI interface (from the English “High Definition Multimedia Interface” or “High Definition Multimedia Interface” in French); - LIN interface (from the English “Local Interconnect Network”).
[0069] According to another particular and non-limiting exemplary embodiment, the device 4 comprises a communication interface 43 which makes it possible to establish communication with other devices (such as other computers of the on-board system) via a communication channel 430. The communication interface 43 corresponds for example to a transmitter configured to transmit and receive information and / or data via the communication channel 430. The communication interface 43 corresponds for example to a wired network of the CAN (Controller Area Network) type, CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (standardized by the ISO 17458 standard) or Ethernet (standardized by the ISO / IEC 802-3 standard).
[0070] According to a particular and non-limiting exemplary embodiment, the device 4 can provide output signals to one or more external devices, such as a display screen 440, touch-sensitive or not, one or more speakers 450 and / or other peripherals 460 (projection system) via the output interfaces 44, 45, 46 respectively. According to a variant, one or other of the external devices is integrated into the device 4.
[0071] [Fig. 3] illustrates a flowchart of the different operations of a process 2 for determining a depth by a monoscopic vision system on board the vehicle of [Fig. 1], according to a particular and non-limiting exemplary embodiment of the present invention.
[0072] The process is for example implemented by one or more processors of one or more computers on board the vehicle 10, for example by a computer controlling the monoscopic vision system.
[0073] In a first operation 21, the computer receives first data representative of a first image acquired by a first camera 11 of the set of cameras at a first acquisition time instant t1.
[0074] In a second operation 22, the computer receives second data representative of a second image acquired by the first camera 11 of the set of cameras at a second acquisition time instant t2, the acquisition time instant t2 being prior to the acquisition time instant t1.
[0075] The two acquired images correspond to two views of the same scene taking place around the vehicle 10 at two distinct acquisition time instants t1 and t2.
[0076] The viewpoints of the first camera 11 at the two acquisition time instants t1 and t2 are not the same; in fact, the first camera 11 has moved between these two instants. Extrinsic parameters linked to the movement of the first camera 11 between these two instants are the following parameters: - 3 translations in the x, y and z directions: T'x, T'y and T'z constituting the translation vector T'; and - 3 rotations around the x, y and z axes: R'x, R'y and R'z, constituting the rotation matrix R'.
[0077] In a third operation 23, third data representative of the movement of the first camera 11 between the first acquisition time instant t1 and the second acquisition time instant t2 are determined as a function of the first and second data. These data are, for example, determined by a computer associated with the monoscopic vision system. The determination of these data is known to those skilled in the art and presented, for example, in the document Unsu-pervised Learning of Depth and Ego-Motion from Video by Tinghui Zhou, Matthew Brown, Noah Snavely and David G. Lowe published on August 1, 2017.
[0078] In a fourth step 24, first depths associated with a first set of pixels of the first image are predicted by the monoscopic vision system composed of the first camera 11 from a first prediction model learned by a stereoscopic system composed of the first camera 11 and the second camera 12.
[0079] In an operation 28, third depths associated with a set of pixels of the second image are predicted by the monoscopic vision system from the first prediction model, the third depths being predicted based on the second data.
[0080] A method for depth prediction by a monoscopic vision system is described in the document: HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation by Xiaoyang Lyu, Liang Liu, Mengmeng Wang, Xin Kong, Lina Liu, Yong Liu, Xinxin Chen, and Yi Yuan published on December 14, 2020.
[0081] Such a depth prediction method is implemented by a convolutional neural network. In order to improve the quality of the output data, i.e. to improve the prediction of depths by the convolutional neural network, its input parameters must be adjusted. The adjustment of these input parameters is done, for example, by learning carried out via the reconstruction of an image.
[0082] A fourth and a fifth image are reconstructed from the first and second images, from the third data relating to the displacement of the first camera 11 between the acquisition time instants tl and t2 and from the first and third predicted depths.
[0083] The fourth and fifth images respectively are reconstructed using the following formula:
[0084] [Math.l] Pm^(K\r^(p$K'\ ] )
[0085] with: - 77 a function to go from homogeneous coordinates to pixel coordinates by removing a dimension from a vector; - K an intrinsic matrix of the camera 11 associated with the projection of a point in space with 3-dimensional coordinates into the image with 2-dimensional coordinates; - T a displacement matrix between the positions of the camera 11 at the second acquisition time instant t2 and at the first acquisition time instant tl for the reconstruction of the fourth image, respectively a displacement matrix between the positions of the camera 11 at the first acquisition time instant tl and at the second acquisition time instant t2 for the reconstruction of the fifth image; - 0 a backprojection function in the scene of a pixel according to its depth; ■ Drnlp ) is a first depth of the pixel Pt of the first image predicted by the monoscopic vision system for the reconstruction of the fourth image, respectively a third depth of the pixel Pt of the second image predicted by the monoscopic vision system for the reconstruction of the fifth image.
[0086] A first reconstruction error is determined for each pixel of the second image by comparing the second image to the fourth image.
[0087] Similarly, a second reconstruction error is determined for each pixel of the first image by comparing the first image to the fifth image.
[0088] The first and second errors respectively are determined by the following loss function:
[0089] [Math.2] Lm(p) =
[0090] With: - I(p) is a value of the pixel P in the second image, respectively the first image; ' l(p\ is a value of the pixel P in the reconstructed image: the fourth image, res- respectively the fifth image; - SSIM (from the English “structural similarity index measure”) is a function which takes into account a local structure; and - a is a weighting factor depending in particular on the type of environment.
[0091] A third reconstruction error is determined from the first and second reconstruction errors. This third reconstruction error is defined by the following loss function:
[0092] [Math.3] Lm = E^min (Lmt (p), Lms (p))
[0093] with : - Ltn^p) the first reconstruction error for a pixel P for a monoscopic vision system; - Lms(p) the second reconstruction error for a pixel P for a monoscopic vision system.
[0094] The input parameters of the convolutional neural network are then adjusted by minimizing the third reconstruction error. The convolutional neural network is then said to be self-supervised because it is able to autonomously adjust its input parameters in order to improve the output data, here the first and second geometric parameters.
[0095] To improve the automatic learning previously presented, a second visibility mask associated with the first image is, for example, determined in an operation 33. This second visibility mask is representative of a sixth set of pixels of the first image having at least one corresponding pixel in the third image, this is determined by any method known to those skilled in the art from the first and third images.
[0096] For example, a second visibility mask associated with the sixth set of pixels of the first image is determined. This second visibility mask is obtained by respective comparison of the first and second reconstruction errors previously defined with predefined threshold values and is obtained for example by the following function: [Math 4] -1 a function returning 0 or 1; - |J the union of pixels; - Lm / ^p) the first reconstruction error for a pixel P of the second image for a monoscopic vision system; - Lm j,p) the second reconstruction error for a pixel P of the first image for a monoscopic vision system; - an AND operator; and - a, b and c are determined parameters.
[0097] The error of the pixel not visible in the target image, but visible in the source image, must exceed a certain level defined by the value a, when the error of the reconstruction of the other direction must be less than a certain level defined by the value b. The notable difference between a and b is linked to the maturity of the training of the learning model.
[0098] The symbol U signifies the union of pixels, provided that the number of pixels in this union is greater than the criterion c. From experience the value of c is, for example, greater than 4 pixels. The symbol 1 indicates the visibility mask resulting from a logical operator, either 1 or 0.
[0099] In order to improve the accuracy of the prediction of first depths for the monoscopic vision system, the prediction method used is supervised using data from the stereoscopic vision system composed of the first camera 11 and the second camera 12.
[0100] The prediction of depth by the monoscopic vision system is not of metric precision. It is therefore useful, in order to improve the relevance of the first predicted depths, to set up supervised automatic learning (in English "knowledge distillation") of the convolutional neural network making it possible to define the first depth via the monoscopic vision system.
[0101] According to one embodiment, in an operation 31, the computer receives fourth data representative of a third image acquired by the second camera 12 at the first acquisition time instant t1.
[0102] The first and third acquired images correspond to two views of the same scene taking place around the vehicle 10 at the same acquisition time instant t1 from viewpoints located at two distinct positions.
[0103] These two images are two images acquired by the stereoscopic vision system consisting of the two cameras 11, 12.
[0104] In an operation 32, fourth depths associated with a fifth set of pixels of the first image are predicted by the stereoscopic vision system from the first and third images according to any method known to those skilled in the art. The learning of the first prediction model is, for example, a function of the fourth depths.
[0105] This calculation is metrically accurate, meaning that the depth measured here is absolute and not relative. In the case where the extrinsic parameters of the system remain unchanged, this depth calculation is very accurate.
[0106] According to an exemplary embodiment, the convolutional neural network making it possible to calculate the first depth is trained under the supervision of the stereoscopic vision system so as to minimize the calculation error defined by the following loss function:
[0107] [Math.5] = Yp j Dm ( p ) - Ds ( p ) |
[0108] with: - Dm(p) the first predicted depth for a pixel P of the first image, and - Ds ( p ) the fourth predicted depth for a pixel P of the first image.
[0109] Thus, the calculations of first depths are refined, the fourth depths making it possible to obtain first depths predicted by the monoscopic vision system consistent with fourth depths predicted by the stereoscopic vision system.
[0110] Using the third data representative of the movement of the first camera 11 between the first acquisition time instant t1 and the second acquisition time instant t2 determined during operation 23, the first moving camera 11 constitutes a pseudo-stereoscopic vision system whose extrinsic parameters are defined using the third data.
[0111] A stereoscopic system has extrinsic data relating to the posi operation of two cameras: their relative orientation and the distance separating the two cameras, also called "Baseline" (in French "reference base"). For the pseudo-stereoscopic system, these data are obtained from the third data, the "reference base" being the distance traveled by the first camera 11 between the two acquisition time instants t1 and t2.
[0112] The “reference base” is determined from the third data as being the norm of the translation vector T' previously defined.
[0113] In order to facilitate the analysis of the first and second acquired images, the first and second images are rectified in an operation 25, according to a method known to those skilled in the art. Such a method is described, for example, in “Projective Rectification of Non-Calibrated Infrared Stereo Images with Global Consideration of Distortion Minimization” by Benoit Ducarouge, Thierry Sentenac, Florian Bugarin and Michel Devy of July 16, 2009.
[0114] The rectification method consists of reorienting the epipolar lines so that they are parallel with the horizontal axis of the image. This method is described by a transformation which projects the epipoles to infinity and whose corresponding points are necessarily on the same ordinate.
[0115] A rectification algorithm consists, for example, of 4 steps: - Rotate (virtually) the first camera 11 in its position at the first acquisition time instant tl so that the epipole goes to infinity along the horizontal axis of the reference frame associated with it; - Apply the same rotation to the first camera 11 in its position at the acquisition time instant t2 to find itself in the initial geometric configuration; - Rotate the first camera 11 in its position at the acquisition time instant t2 by the rotation associated with the rotation matrix 'R”, corresponding to the extrinsic parameter of the pseudo-stereoscopic vision system; - Adjust the scale in both camera markers.
[0116] Rectification simplifies the matching of image pixels. The corresponding pixel in the second image to a pixel in the first image (and vice versa) is positioned on the same line. Based on knowledge of the epipolar geometry and therefore a fundamental matrix of the pseudostereoscopic vision system, the objective is then to determine a pair of projective transformations, called homographies, which reorient the epipolar projections parallel to the image lines, therefore to the horizontal axis of the rectified cameras.
[0117] The following operation 26 consists of determining disparity values associated with a second set of pixels of the first rectified image corresponding to a set of pixels of the second rectified image.
[0118] The determination of a set of pixels in the second image corresponding to a second set of pixels in the first image, this operation is called “stereo matching” (in English “feature matching”).
[0119] Stereo matching or disparity estimation is the process of finding pixels in stereoscopic views, here pseudo-stereoscopic, that correspond to the same 3D point in the three-dimensional scene. Rectified epipolar geometry simplifies this process of finding correspondences on the same epipolar line. It is not necessary to calculate the coordinates of the 3D point to find the corresponding pixel on the same line of the other image. Disparity is the distance d between a pixel and its correspondence in the other image. This disparity is horizontal when the images are rectified, the horizontality being defined along an x direction of the first image.
[0120] The output data of this operation 26 are first disparities representative of a displacement between each pixel of the second set of pixels of the first image and a corresponding pixel in the second image.
[0121] In the same way, according to an exemplary embodiment, second disparities are associated with pixels of the second image, representative of a displacement between each pixel of the second image and a corresponding pixel in the second set of pixels of the first image.
[0122] In an operation 27, a second depth associated with pixels of the second set of pixels of the first image is calculated as a function of first disparities associated with these pixels. The second depth associated with a pixel is calculated, for example, according to a formula known to those skilled in the art:
[0123] [Math.6] £>(p) = / x -~-7 J d(p)
[0124] With: - D( p) is the depth of pixel P of the first image; - f is the focal length fl of the first camera 11 in pixel units; - B the distance between the two positions of the first camera 11 at the two acquisition time instants tl and t2; - d(p) is the first disparity of pixel P in pixel units, defined as the horizontal displacement of a pixel from the first image to the second image.
[0125] The depth prediction model for the pseudo-stereoscopic vision system is then trained. Machine learning consists of reconstructing a sixth and seventh image from the first and second images, and from the first and second disparities.
[0126] The sixth and seventh images respectively are reconstructed using the following formula:
[0127] [Math.7] p^==p^d(pt)
[0128] with: - p* the abscissa of a pixel of the sixth image, respectively seventh image; - the abscissa of a pixel of the first image, respectively second image; and ■ d( p} a disparity determined for a pixel Pt of the first image, respectively second image.
[0129] A fourth reconstruction error is determined for each pixel of the second image by comparing the second image to the sixth image.
[0130] Similarly, a fifth reconstruction error is determined for each pixel of the first image by comparing the first image to the seventh image.
[0131] The fourth and fifth reconstruction errors respectively are determined by the following loss function: [Math 8] LM = Uj (1 ') ' Uip) ~Kp)\+a- With : - I(p) is a value of the pixel P in the second image, respectively the first image; - is a value of the pixel P in the reconstructed image: the sixth image, respec tively the seventh image; - SSIM (from the English “structural similarity index measure”) is a function which takes into account a local structure; and - a is a weighting factor depending in particular on the type of environment.
[0132] A sixth reconstruction error is determined from the fourth and fifth reconstruction errors. This sixth reconstruction error is defined by the following loss function: [Math 9] - Ls^p) the fourth reconstruction error for a pixel P for a pseudo-stereoscopic vision system using a disparity calculation method; - Ls / p) the fifth reconstruction error for a pixel P for a pseudo-stereoscopic vision system using a disparity calculation method.
[0133] The input parameters of the convolutional neural network are then adjusted by minimizing the sixth error. The convolutional neural network is then said to be self-supervised because it is able to autonomously adjust its input parameters in order to improve the output data, here the first and second disparities.
[0134] It is possible that a pixel of the first image does not find a corresponding pixel in the third image. This phenomenon is explained by the fact that areas of the first image may be occluded in the third image. Indeed, the difference in viewpoint of the first camera 11 at the two acquisition time instants t1 and t2 does not allow the first camera 11 to see all the elements of the scene at two distinct acquisition time instants. An object present in the scene, for example the obstacle 18, may mask a second object of the scene, the second object being visible from the viewpoint of the first camera 11 at the acquisition time instant t1 but being masked by the obstacle 18 from the viewpoint of the first camera 11 at the acquisition time instant t2.
[0135] In an operation 29, a first visibility mask associated with a fourth set of pixels of the first image is determined. This visibility mask is obtained by respective comparison of the fourth and fifth errors to predefined threshold values and is obtained for example by the following function: [Math 10] v„ (p) =1( U,(Up)> a KLM -1 a function returning 0 or 1; - U the union of pixels; - Ls[(p) the fourth reconstruction error for a pixel P of the second image for a pseudo-stereoscopic vision system using a disparity calculation method; - Ls£p) the fifth reconstruction error for a pixel P of the first image for a pseudo-stereoscopic vision system using a disparity calculation method; - an AND operator; and - a, b and 6 of the determined parameters.
[0136] The error of the pixel not visible in the target image, but visible in the source image, must exceed a certain level defined by the value a, when the error of the reconstruction of the other direction must be lower than a certain level defined by the value b, The notable difference between a and b is linked to the maturity of the training of the learning model.
[0137] The symbol U signifies the union of pixels, provided that the number of pixels in this union is greater than the criterion c. From experience the value of c is, for example, greater than 4 pixels. The symbol 1 indicates the visibility mask resulting from a logical operator, either 1 or 0.
[0138] Thus, the definition of a first visibility mask makes it possible to identify the areas where the image reconstruction is not valid. The use of the first visibility mask in the calculations of the reconstruction errors makes it possible not to take into account the pixels not visible in an image and thus makes it possible not to take these pixels into consideration during the calculations of the reconstruction error. The learning of the second depth prediction model is then improved.
[0139] It is possible that a pixel of the first image is associated with a moving object in the three-dimensional scene. Such an object is then located at two different positions in the three-dimensional scene at the acquisition time instants t1 and t2. The pseudo-stereoscopic vision system cannot predict a second depth for this pixel of the first image.
[0140] According to an exemplary embodiment, it is therefore appropriate to determine a mask of dynamic objects representative of a third set of pixels of the first image associated with objects in motion in the three-dimensional scene.
[0141] In an operation 30, a mask of the dynamic objects is defined, for example, using the following formulas:
[0142] [Math. 11] Mh(pt) = )
[0143] [Math. 12] M d (p^ = le*Wll2
[0144] [Math. 13] 0 ( P^Pt™ ( Pt ) ) = Dtm ( p; ) K~}h(jy^
[0145] With: ■ Mh(pj is the mask for the static environment applied to pixel Pt of the first image; - T'^t is the displacement matrix of the first camera 11 between the first acquisition time instant tl and the second acquisition time instant t2; - ÿ is the backprojection of a pixel P with its corresponding depth Dip); ■ p) is the third depth of a source pixel Ps of the third image calculated for the monoscopic vision system; - Dtm^P ) is 'a Prcm'orc depth of a target pixel Pt of the first image calculated for the monoscopic vision system; ■ Md(j) ) is the mask for the dynamic environment applied to the pixel Pt of the first image; - K is the intrinsic matrix of the first camera 11; - || || 2 is the squared error (L2 norm); ■ h{p^ transforms the pixel vector Pt to the homogeneous coordinate by adding a dimension to the pixel coordinate vector Pt to allow matrix multiplication.
[0146] The hypothesis of Mh is that static points in 3D space do not change their positions during the time separating two consecutive images. The mask Md is constructed by an exponential function with a hyperparameter a to find a good criterion for Mh. The static character of the objects is thus considered in a relative manner, the weighting being implemented via the exponential function.
[0147] The learning of the second prediction model is then a function of this mask of dynamic objects.
[0148] Thus, the definition of a dynamic object mask makes it possible to identify the areas where the image reconstruction is not valid. The use of the dynamic object mask in the calculations of reconstruction errors makes it possible not to take into account the pixels of an image associated with moving objects and thus makes it possible not to take these pixels into consideration during the calculations of reconstruction errors. The learning of the second depth prediction model is then further improved.
[0149] According to an exemplary embodiment, the convolutional neural network making it possible to calculate the second depth is trained under the supervision of the monoscopic vision system so as to minimize the calculation error defined by the following loss function:
[0150] [Math. 14] Lnr^ps = ILp ] DpS (p) - (p) |
[0151] with: - DpS(p ) the second predicted depth for a pixel P of the first image, and - Dm(p) the first predicted depth for a pixel P of the first image
[0152] Thus, the first and second depths are consistent.
[0153] It is noted that the prediction model for the pseudo-stereoscopic system learned by the monoscopic vision system, as well as the prediction model for the monoscopic system learned by the stereoscopic vision system, make it possible to obtain first, second and fourth depths that are consistent for the three prediction models. Thus, an AD AS that uses one or more of the previously obtained depths has homogeneous and consistent input data.
[0154] If an ADAS uses the first or second depths as input data to determine the distance between a part of the vehicle 10, for example the front bumper, and another user present on the road, the ADAS is then able to determine this distance precisely. For example, if the ADAS has the function of acting on a braking system of the vehicle 10 in the event of a risk of collision with another road user and the distance separating the vehicle 10 from this same road user decreases significantly, then the ADAS is able to detect this sudden approach and act on the braking system of the vehicle 10 to avoid a possible accident.
[0155] [Fig. 4] illustrates a flowchart of the different steps of a method for determining a depth by a monoscopic vision system on board the vehicle of [Fig. 1], according to a particular and non-limiting example of the present invention. The method is for example implemented by a device on board the first vehicle 10 or by the device 4 of [Fig. 2].
[0156] In a first step 21, first data representative of a first image acquired by a first camera 11 of said set of cameras at a first acquisition time instant t1 are received.
[0157] In a second step 22, second data representative of a second image acquired by the first camera 11 of the set of cameras at a second acquisition time instant t2 are received.
[0158] In a third step 23, third data representative of a movement of the first camera 11 between the first acquisition time instant t1 and the second acquisition time instant t2 are determined as a function of the first and second data.
[0159] In a step 24, first depths associated with a first set of pixels of the first image are predicted by the monoscopic vision system from a first prediction model learned and supervised by a stereoscopic system composed of the first camera 11 and the at least one second camera 12, the first depths being predicted based on the first data, i.e. based on the first image.
[0160] In a step 25, the first and second images are rectified according to the third data to obtain a first rectified image and a second rectified image. The first and second images are obtained from a vision system composed of the first moving camera 11 called a pseudo-stereoscopic vision system.
[0161] In a step 26, disparity values associated with a second set of pixels of the first rectified image corresponding to a set of pixels of the second rectified image are determined.
[0162] In a step 27, second depths associated with the second set of pixels are predicted by the pseudo-stereoscopic vision system composed of the first moving camera 11 from a second prediction model learned and supervised by the monoscopic vision system. The second depths are predicted as a function of the disparity values and as a function of a distance separating positions of the first camera 11 at the first acquisition time instant t1 and at the second acquisition time instant t2, the distance called “reference base” being determined from the third data.
[0163] According to a variant, the variants and examples of the operations described in relation to Figures 1 and 3 apply to the steps of the method of [Fig.4].
[0164] Of course, the present invention is not limited to the exemplary embodiments described above but extends to a method for determining a depth by means of a monoscopic vision system on board a vehicle, which would include secondary steps without thereby departing from the scope of the present invention. The same would apply to a device configured for implementing such a method.
[0165] The present invention also relates to a vehicle, for example an automobile or more generally an autonomous land-based motor vehicle, comprising the device 4 of [Fig.2].
Claims
1. Claims Method for determining a depth by a monoscopic vision system on board a moving vehicle (10), the monoscopic vision system comprising a first camera (11) capable of cooperating with at least one second camera (12) of a set of cameras of at least two cameras (11, 12) arranged so as to each acquire an image of a three-dimensional scene from a different point of view, said method being characterized in that it comprises the following steps: - reception (21, 22) of first and second data respectively representative of a first and second image acquired by said first camera (11) of said set of cameras respectively at a first acquisition time instant and at a second acquisition time instant; - determination (23) of third data representative of a movement of the first camera (11) between said first acquisition time instant and said second acquisition time instant as a function of said first and second data; - prediction (24) of first depths associated with a first set of pixels of said first image by said monoscopic vision system from a first prediction model learned and supervised by a stereoscopic system composed of the first camera (11) and at least one second camera (12), said first depths being predicted as a function of said first data; - rectification (25) of said first and second images as a function of said third data to obtain respectively a first rectified image and a second rectified image; - determining disparity values (26) associated with a second set of pixels of said first rectified image corresponding to a set of pixels of said second rectified image; and - prediction (27) of second depths associated with said second set of pixels by a pseudo-stereoscopic vision system composed of the first camera (11) in motion from a second prediction model learned and supervised by said monoscopic vision system, said second depths being predicted as a function of said disparity values and as a function of a distance separating positions of the first camera (11) at the first acquisition time instant and at the second acquisition time instant, said distance being determined from said third data.
2. Method according to claim 1, for which a supervision of said second prediction model is obtained by minimizing a loss function defined by the following function: L^ïjpM -^w(p)|with: - Dps^p} the second predicted depth for a pixel P of the first image, and - Dm( p} the first predicted depth for a pixel P of the first image.
3. Method according to one of claims 1 to 2, further comprising the steps of: - prediction (28) of third depths associated with a set of pixels of the second image by said monoscopic vision system from said first prediction model, said third depths being predicted as a function of said second data; - determination (30) of a mask of dynamic objects associated with said first image and representative of a third set of pixels of the first image associated with a moving object in said three-dimensional scene, the learning of said second prediction model being furthermore a function of said mask of dynamic objects.
4. Method according to one of claims 1 to 3, further comprising a step of determining (29) a first visibility mask associated with said first image and representative of a fourth set of pixels of the first image having at least one corresponding pixel in said second image from said first and second data, the learning of said second prediction model being furthermore a function of said first visibility mask.
5. Method according to one of claims 1 to 4, further comprising the steps of: - receiving (31) fourth data representative of a third image acquired by said at least one second camera (12) at said first acquisition time instant; - prediction (32) of fourth depths associated with a fifth set of pixels of the first image by said stereoscopic vision system from said first and third data, the learning of said first prediction model being furthermore a function of said fourth depths.
6. Method according to claim 5, for which a supervision of said first prediction model is obtained by minimizing a loss function defined by the following function: L^m = ( P ) - Ds ( p ) | with: - Dm(p) the first predicted depth for a pixel P of the first image, and - D, ( p ) the fourth predicted depth for a pixel P of the first image.
7. Method according to one of claims 5 to 6, further comprising a step of determining (33) a second visibility mask associated with said first image and representative of a sixth set of pixels of the first image having at least one corresponding pixel in said third image from said first and fourth data, the learning of said first prediction model being furthermore a function of said second visibility mask.
8. Computer program comprising instructions for implementing the method according to any one of the preceding claims, when these instructions are executed by a processor.
9. Device (4) for determining a depth by a vision system on board a vehicle (10), said device (4) comprising a memory (41) associated with at least one processor (40) configured for implementing the steps of the method according to any one of claims 1 to 7.
10. Vehicle (10) comprising the device (4) according to claim 9.
Citation Information
Patent Citations
Three-dimensional dense structure from motion with stereo vision
US20170019655A1
