Method for estimating distance, computing element, computer program and vehicle system for implementing the method

Abstract

The invention relates to a method, a computing element, a computer program and a system for estimating distances or depths in the field of view of a vehicle comprising at least one camera (3) under night or low light conditions using an image processing module (1) by the following means: a pre-training process for said module (1) in order to estimate distances based on data corresponding to a plurality of images from said field of view called training images, in which at least one high-contrast drawing called pattern (4) is projected by an illumination device (5) into a projection zone (ZP) during the pre-training phase, which allows said module (1) to generate a plurality of learned parameters corresponding to the characteristics of elements (EP) belonging to said pattern (4) and to the characteristics of elements (EC) belonging to the content of the scenes captured in these training images, according to the distances at which these elements (EP, EC) are located, then to create a map of the distances of these elements within at least one region of interest (ROI) in these images, then, when said vehicle is running under night or low light conditions, to acquire and process the images transmitted by said camera (3) by said module (1) to provide a map of the distances of these elements contained in at least one monitored zone (ZS) within the images acquired when said vehicle is running, using said parameters learned during said training phase.

Description

[0001] This invention relates to the field of motor vehicles, particularly automobiles, and more specifically to the perception and estimation (assessment) of the distance or depth of elements present in the field of vision of a vehicle, even under low light conditions (e.g., at night).

[0002] Known problems in the field of motor vehicles, and especially automobiles, involve visibility at night or in low light conditions, and particularly the perception of distance (or depth). Adverse conditions, such as nighttime (or dawn or dusk) or severe weather conditions (snow, rain, dark skies, thunderstorms, etc.), pose significant challenges to a variety of computer vision applications. Indeed, despite significant progress in autonomous driving, challenges remain in nighttime navigation, particularly due to the complexity of the scenes to be analyzed. Accurate perception of distance or depth is especially important, particularly in conditions of poor visibility, for example, to provide driver assistance or assistance to autonomous vehicles. This perception extends beyond immediate applications and has a profound impact on the overall perception of the scene, especially at night. Information about distance (also known as “depth”) is a key aspect of scene perception.

[0003] Various solutions are known from existing technologies, but they all have various drawbacks. For example, some solutions use LiDAR (Light Detection and Ranging), which provides high depth accuracy both day and night, but its widespread adoption is hindered by high costs. On the other hand, thermal imaging has proven its effectiveness, but it introduces new challenges such as contrast and resolution. Furthermore, camera-based methods perform well in daylight but fail in low-light conditions, leading to the development of dedicated methods.

[0004] Solutions known as Structure of Motion (SfM) also utilize or predict the relative positions of cameras in the video while estimating depth. However, a major challenge associated with SfM is scale inconsistency and its deviation from reality. Attempts to address scale inconsistency to date have introduced losses in scene geometry and / or required the use of other physical properties such as vehicle velocity.

[0005] RGB-D cameras can also be used to obtain depth information. While some schemes rely solely on stereo vision, others also use patterns projected by infrared lasers, a technique known as active stereo vision. A major challenge of active stereo vision is model matching between the projection and the reference, traditionally handled through mathematical models. Recent advances have combined machine learning and deep learning. Unlike monocular depth estimation methods, active stereo vision provides scale information derived from the distance between the camera and the projector. However, research shows that its performance degrades significantly when used outdoors due to ambient lighting conditions and projector performance. In such scenarios, the accuracy of maximum distance estimation and reconstruction decreases. Furthermore, the use of infrared projectors involves additional costs, especially when used in vehicles that are not necessarily equipped with infrared projectors.

[0006] Therefore, the object of the present invention is to provide a solution for estimating the distance of motor vehicles under low light conditions that is accurate and robust, while limiting additional costs, particularly additional costs for additional equipment used in the vehicle.

[0007] This objective is achieved by a method for estimating distance or depth in a vehicle's field of view using an image processing module at night or under low light conditions, the vehicle first including at least one lighting device for projecting at least one luminous design into the field of view, and secondly including at least one camera for acquiring an image from the field of view, the method being characterized in that it includes: - A pre-training process for the module is used to estimate distances based on data corresponding to multiple images, called training images, from the field of view. During the pre-training phase, the lighting device projects at least one high-contrast drawing, called a pattern, onto a projection area. This allows the module to generate multiple learned parameters corresponding to features belonging to elements of the pattern and features belonging to elements of the scene content captured in these training images. Based on the distances of these elements, a distance map of these elements is then created in at least one region of interest within these images. - When the vehicle is operating at night or in low light conditions, the module acquires and processes images transmitted by the camera to provide a distance map of elements contained in at least one monitoring area within these images acquired while the vehicle is operating, using the parameters learned during the training phase.

[0008] This objective is also achieved by a computing element comprising means for performing the steps of the method according to any one of the preceding claims.

[0009] This objective is also achieved by a computer program comprising instructions that, when executed by an image processing module, cause the module to perform the steps of the method according to the invention.

[0010] This objective is also achieved by a vehicle assistance system, which includes: - An image processing module capable of performing the steps of the method according to the present invention; - At least one lighting device, which is capable of projecting at least one pattern into a field of view; - At least one camera, which is used to transmit images acquired while the vehicle is in motion.

[0011] According to another characteristic, the learned parameters correspond to features representing the changes of elements existing in the projection pattern relative to the reference pattern, which are caused by the distance, shape, and orientation of the visible surfaces of elements belonging to the scene in the training image, and correspond to at least one of the parallax, shape, and size of elements belonging to the projection pattern.

[0012] According to another characteristic, the reference pattern is implicitly learned by the module based on the recurrence of the reference pattern in the training images during the pre-training process.

[0013] According to another characteristic, the reference pattern is transmitted to the module at least during the pre-training process.

[0014] According to another characteristic, the region of interest is restricted relative to the projected region within the training image.

[0015] According to another characteristic, the learned parameters used to provide a graph of the distances of elements contained in a monitoring area within an image acquired while the vehicle is in motion correspond at least to the shape and / or size features of elements belonging to the content of the scene captured in the image.

[0016] According to another feature, during the acquisition and processing of images obtained by the camera while the vehicle is in motion by the module, the pattern is also projected by the lighting device onto the projection area in the field of view.

[0017] According to another characteristic, the size of the monitoring area is greater than or equal to the size of the region of interest in the training image and / or the size of the projected region in the image acquired by the camera while the vehicle is in motion.

[0018] According to another feature, during the acquisition and processing of images captured by the camera while the vehicle is in motion, the projection area is limited to a portion of the field of view, which is determined by the detection module based on elements contained in the scene captured by the camera.

[0019] According to another feature, the advantage stems from the projection of at least one second pattern onto the second projection area by at least one lighting device.

[0020] According to another characteristic, the second pattern has the same content as the first pattern, but has the same or different resolution.

[0021] According to another characteristic, the second pattern has content that is different from that of the first pattern.

[0022] According to another characteristic, the first and second patterns are projected into the field of view at different distances from the vehicle.

[0023] According to another feature, the lighting device is able to project a pattern based on the fact that it includes multiple light sources, each of which can illuminate a limited area in the field of view with a variable intensity. The combined control of these intensities according to the location allows the pattern to be obtained.

[0024] Other features and advantages of the invention will become clearer upon reading the following description of various embodiments given with reference to the accompanying drawings. In fact, a set of drawings is provided to supplement the specification and to better understand the invention. These drawings illustrate one embodiment of the invention and form part of the specification; they should not be construed as limiting the scope of the invention, but are merely examples of how the invention can be practiced. These drawings include the following figures: [ Figure 1 ] Figure 1 A schematic plan view of a vehicle equipped with a system for performing methods according to various embodiments of the present invention is shown; [ Figure 2 ] Figure 2 A schematic diagram of a system for performing methods according to some embodiments of the present invention is shown; [ Figure 3 ] Figure 3 The implementation of the invention is shown in addition to Figure 2 A schematic diagram of the system of methods in embodiments other than those shown; [ Figure 4 ] Figure 4 An exemplary reference pattern according to some embodiments of the present invention is shown at the top, and an exemplary pattern projected onto a surface perpendicular to the image acquisition camera is shown at the bottom, the pattern having distortions due to components of its projector; [ Figure 5 ] Figure 5 On the left, an example is shown where a portion of the pattern is projected onto a surface perpendicular to the image-acquiring camera and at a first distance; on the right, an example is shown where the same portion of the pattern is projected onto the same surface but at a second distance greater than the first distance. [ Figure 6 ] Figure 6 The image shown is an image acquired by a camera and in which a pattern is projected, according to some embodiments of the present invention, the image having elements of the pattern and elements of the content of the scene in the image; [ Figure 7 ] Figure 7 An image acquired by a black-and-white camera is shown on the left, and a distance map of that image is shown on the right, obtained by performing a method according to some embodiments of the present invention. [ Figure 8 ] Figure 8 A black-and-white schematic diagram illustrating real-world distances from three different scenes is provided, where elements of the scene content are identified for comparison with... Figure 9 and Figure 10 Compare; [ Figure 9 ] Figure 9 The diagram is schematically shown in black and white. Figure 8 The distance maps of the three scenes are obtained by performing a method according to some embodiments of the invention, which uses patterned projection during training and during the acquisition of images from these scenes while the vehicle is running; [ Figure 10 ] Figure 10 The diagram is schematically shown in black and white. Figure 8 Distance maps of three scenarios, these distance maps are obtained by performing the above-described invention, except... Figure 9 The method obtained is different from those embodiments shown, which uses pattern projection only during training and does not use pattern projection when acquiring images from these scenes while the vehicle is running.

[0025] This invention relates to a method for estimating distances in a vehicle's field of vision, a computing element, a computer program, and a system (vehicle-mounted) for performing the method. The estimation of distance or depth is performed by an image processing module (1) in the vehicle's field of vision, under nighttime or low-light conditions (typically less than 20 lux, or even 10 lux). Typically, the vehicle first includes at least one lighting device (5) for projecting at least one luminous design into the field of vision, and secondly includes at least one camera (3) for acquiring an image from the field of vision. A single "lighting" or "pattern projection" device is sufficient to make the model work. In a vehicle, this projection can be provided by one of the vehicle's lighting devices (i.e., headlights, e.g., particularly low beam headlights), but another lighting device dedicated to patterns can also be used. However, the invention specifically utilizes devices already present in many vehicles, as described in detail below.

[0026] This application relates to an image processing module (1) trained during at least one training process, which estimates distances within an image acquired by a camera (3) by means of the projection of a pattern onto a pattern by at least one lighting device (5), and also to a control unit (2) that can, for example, control the lighting devices of a vehicle and can be combined with driver assistance functions, as known from the prior art. In practice, typically, the control unit (2) controls the lighting devices of a vehicle to adapt the lighting to traffic conditions and brightness with or without manual operation by the driver, and modern control units increasingly incorporate advanced (“intelligent” functions, particularly for driver assistance or autonomous driving. Those skilled in the art will certainly understand that the expression “image processing module (1)” is a functional definition and actually refers to a model, particularly a neural network model, such as a CNN (Convolutional Neural Network) model. Such a model can, for example, be based on a transformer (or “self-attention model”) designed to manage sequential data. The applicant of this application has observed that the present invention can operate using conventional, even basic, models (e.g., models known from the prior art as “U-net”, or models specifically designed for distance estimation (e.g., models known from the prior art as “Adabins” or “depthformer”), and the invention is not limited to the type of model used in module (1). In practice, this model (or module) can be provided by a control unit or other data processing resources (particularly computer resources). Furthermore, it can actually be multiple modules and / or control units dedicated to various tasks or functions and cooperating to perform the invention, or a single unit implementing all the tasks or functions described in this application. Thus, the figures show a control unit (2) for controlling the lighting device (5), which is separate from the image processing module (1), but it will be understood that this configuration is merely a non-limiting example among examples, including other examples. The same applies to the detection module (not shown) described in this application, which is capable of detecting obstacles, objects, or features in the surrounding environment or scene, as such a module (known from the prior art) can be implemented in the same manner. This invention also advantageously falls within the context of vehicles already equipped with driver assistance or even autonomous driving systems by providing a method and system that complements existing systems (which are typically more expensive and use different sensor technologies). This complementarity provides redundancy for the measurements and estimations performed, which is indispensable in the case of autonomous vehicles. Furthermore, those skilled in the art will naturally understand that, due to the nature of the task being performed, such a unit (e.g., comprising multiple modules) will typically have at least one processor executing instructions, and multiple implementations are possible, and therefore it is not necessary to provide details about the types of hardware that can be used or employed.The accompanying drawings shown on the accompanying pages are provided by way of example only and in an illustrative and functional manner, enabling those skilled in the art to conceive of any type of variation based on the content of this application.

[0027] Unless otherwise defined, all terms (including technical and scientific terms) used in this document shall be interpreted in accordance with standard practices in the industry, particularly in the fields of lighting and signaling of motor vehicles, especially automobiles. For example, the term “field of view” (usually expressed in degrees) may be used without implying any particular limitation in this application. It should also be understood that, unless explicitly defined herein, commonly used terms should be interpreted according to conventions in the relevant field, rather than with idealized or overly formal meanings, and should not be construed as restrictive. Here, the term “while the vehicle is in motion” (“image acquired while the vehicle is in motion”) is used to indicate an image from a real-world scenario where a method is being performed “in the field” in a vehicle ready to move; however, unlike some models in the prior art, since the model does not require knowledge of the vehicle's speed, this term is not restrictive, especially regarding whether the vehicle is moving. Therefore, the term “in motion” also encompasses, for example, parking or stationary parking.

[0028] In this application, as is generally accepted in the field of patent applications, the terms “comprises,” “has,” and “includes,” and their derivatives (such as “comprising,” “having,” etc.) should not be understood in an exclusive sense, that is, these terms should not be interpreted as excluding the possibility that the described and defined content may include other elements, steps, etc.

[0029] In this application, the term "lighting device (5)" is understood to more specifically refer to a device for illuminating the environment around a vehicle: - In order to be able to see (e.g., low beam headlights (LB) or high beam headlights (HB), even though the latter is usually prohibited under certain conditions, especially in built-up areas), - Or so that it can be seen (e.g., position lights (PL) or daytime running lights (DRL)). However, given the distances that are typically expected to be estimated, pattern projection will preferably involve low beam headlights (or possibly high beam headlights).

[0030] The present invention uses a projection of a “pattern” (4) or “high-contrast drawing” from an image within a scene, in which distance estimation is performed by an image processing module (1) or a “model”. The pattern can be a drawing, sketch, design, image, or shape, preferably a high-contrast shape, that is, preferably having a contrast (typically brightness contrast) sufficient to be detected by the camera in low light. Contrast can be achieved by alternating dark and light portions, and it is not necessary to use the same high contrast as the black and white shown in the accompanying drawings, which do not limit this. On the other hand, the transition (edges) between dark and light portions will preferably be as sharp as possible to facilitate the detection of deformation. Furthermore, the pattern can include repeating designs or repeating shapes, such as the logo of a vehicle brand. Preferably, the pattern has highly concentrated discontinuities and contrast, particularly with edges and corners. Thus, checkerboard patterns are particularly advantageous because they have a large number of repeating high-contrast designs and edges and corners. However, the invention is not limited to this example, and the module is able to learn deformations of various types of patterns even in the case of such repeating designs, which learns faster. Therefore, the present invention is not limited to the example of the checkerboard pattern shown in the accompanying drawings, as other patterns can certainly be used, preferably those with easily identifiable contrast areas. In fact, the checkerboard pattern was chosen as an example because of its apparent discontinuity and high contrast. In the design, the dense concentration of corners and transitions serves as a feature that makes the model easily identifiable and detectable, but other less regular patterns can also be used. On the other hand, the present invention is not limited to using a single pattern at a time or using the same pattern in all scenarios.

[0031] It should be noted that the term "luminous design" is used in this application to refer to a lighting device projecting a beam of light of various shapes (or luminance) depending on the device and configuration (and specifications). In practice, individual devices emit beams of light specific to that device and having a particular shape defined by luminance. For example, low beam headlights typically have a beam that illuminates a symmetrical area (referred to as the "flat area") closer to the vehicle and an asymmetrical area (referred to as the "knot area") further away, thereby conveying a particular luminous design. However, the term "design" does not imply a limitation on design in the proper sense (e.g., it may be misunderstood as a simplified diagram) and therefore does not necessarily imply a change in luminance. Furthermore, it is known to modify the luminous design, for example, to save energy or to illuminate certain parts of the field of vision to a greater or lesser extent in order to better identify objects or traffic conditions (e.g., by maintaining only the necessary illumination for the flat area or indicating the outline of the vehicle). Therefore, the pattern (4) of the present invention can be projected within the luminous design of the lighting device (5) that projects the pattern, or projected outside of the design, particularly when the design is limited for other reasons. Typically, and preferably, the pattern is projected by the lighting device within its luminous design; however, the pattern may be projected outside the projected design, especially where the design has been limited by the control unit for other reasons (e.g., energy saving or anti-glare). Furthermore, since the pattern is only needed during training, it may not be projected during online image processing (while the vehicle is running), where the images can be used based on illumination provided by the luminous design of at least one lighting device.

[0032] Furthermore, it will be understood that the pattern (4) can be advantageously projected by lighting devices (5) known in the prior art, such as “pixelated headlights”, which include multiple light sources, each capable of illuminating a limited location in the field of vision with variable intensity, the combined control of these intensities according to the location allowing the pattern (4) to be obtained. In fact, there are various known types of headlights with multiple individually controllable light sources (thus forming pixels) that readily allow the projection of the pattern (4) that can be used to perform the invention. For example, luminous lighting devices that emit light by individually controllable pixels (particularly having the advantage of anti-glare functionality in some areas of the field of vision by means of advanced control by a control unit) are particularly known from document EP4251473.

[0033] Therefore, preferably, the present invention does not require an additional dedicated projector in the vehicle, but instead utilizes the fact that the vehicle has lighting devices (5) to project patterns, wherein at least one lighting device is capable of projecting patterns, which represents a favorable saving, especially compared with other methods in the prior art that use other technologies.

[0034] The model is capable of learning at low resolution and provides satisfactory results at a resolution of 320x320 in its region of interest (ROI) or its monitored region (ZS) (described in this application), but it can also work at even lower resolutions. Therefore, the present invention can work on some vehicles without requiring changes to their cameras or lighting devices, which typically have a resolution higher than that required by the model. In fact, cameras typically have resolutions on the order of megapixels, and modern HD headlights have resolutions of hundreds or thousands, or even tens of thousands of pixels. On the other hand, patterns, such as checkerboard patterns (where the squares (or cells) are measured to be 2x2 pixels), have proven sufficient to make the model work, allowing the model to distinguish patterns even at lower resolutions. Therefore, implementing the present invention advantageously does not limit the technical specifications of the devices used. In some embodiments, at least one of the vehicle's lighting devices (5) has a matrix arrangement of rows and columns of luminous pixels (2). For example, a matrix may include at least 1,000 to 2,000 light sources, and devices now known as HD (high-definition) devices have more than 4,000, or even 25,000 or up to 50,000 light sources.

[0035] In general, the present invention is based on a method that uses at least: - For the pre-training process of the module (1), in order to estimate distances based on data corresponding to multiple images called training images from the field of view, in which the lighting device (5) projects at least one high-contrast drawing called pattern (4) into a projection area (ZP) during the pre-training phase, this allows the module (1) to generate multiple learned parameters corresponding to features of elements (EP) belonging to the pattern (4) and features of elements (EC) belonging to the content of the scene captured in these training images, and then, based on the distances of these elements (EP, EC), a distance map of these elements is created in at least one region of interest (ROI) within these images, and then... - When the vehicle is running at night or in low light conditions, the module (1) acquires and processes images transmitted by the camera (3) to provide a map of the distances of elements contained in at least one monitoring area (ZS) within these images acquired while the vehicle is running, using the parameters learned during the training phase.

[0036] Figure 7 The left side shows images acquired at night, where most elements in the scene are very difficult to distinguish and their distances are almost impossible to estimate. In contrast, the right side shows images obtained using this method where elements in the scene are easier to identify and their distances are correctly estimated.

[0037] Therefore, some embodiments relate to methods at least based on this method, and some embodiments relate to computing elements including means for performing the method. Such means may be, for example, a computer program. Therefore, some embodiments relate to a computer program including instructions that, when executed by the image processing module (1), cause the module (1) to perform the method described in this application. Therefore, it will be understood that by integrating such an image processing module (1) or such a model into a data processing device present in a vehicle, a driver assistance system for a motor vehicle is obtained. Therefore, some embodiments relate to a vehicle assistance system comprising: - Image processing module (1), which is capable of executing the steps of the method; - At least one lighting device (5) capable of projecting at least one pattern (4) into the field of view; - At least one camera (3) is used to transmit images acquired while the vehicle is in motion.

[0038] In some such embodiments of the system, the lighting device (5) is capable of projecting a pattern based on the fact that it comprises multiple light sources, each capable of illuminating a limited location in the field of view with variable intensity, and the combined control of these intensities according to these locations allows the pattern (4) to be obtained. On the other hand, the system can actually be integrated into the vehicle itself, and therefore some embodiments involve vehicles that include devices identical to those described above.

[0039] Figure 1 Schematic and non-limiting illustrations of examples of vehicles or systems according to some embodiments are provided, which project a pattern (4) into a projection area (ZP), wherein, as explained below, a region of interest (ROI) is used for training, while a wider monitoring area (ZS) is used for distance estimation. Figure 2 An illustrative and non-limiting illustration of an example system is provided, which includes a camera (3) for transmitting an image from the field of view to an image processing module (1), and in this example, a device (5) controlled by a control unit (2) for projecting a pattern (4) into the field of view of the camera. Figure 3 Another non-limiting example of an embodiment of the system is shown, which includes two devices (5) controlled by a control unit, which includes an image processing module (1) for projecting a pattern. This allows a camera (3) to acquire images from the scene and the pattern so that these images can be transmitted to a module (1) that allows for the estimation of distances in the scene. Based on these examples, it will be understood that many variations are possible.

[0040] On the other hand, it will be understood that training actually allows the model to learn parameters for recognizing features caused by distance. Here, these learned parameters are designated as generated parameters because they are subsequently stored in memory for use when processing images acquired while the vehicle is in motion.

[0041] In various embodiments, the training process performed by the control unit (2) includes using computer-based machine learning algorithms, such as neural networks. This training is referred to as “pre-training” because it is performed before the method steps (acquisition, calculation, detection, comparison, etc.) executed while the vehicle is running. However, it should be noted that the training can be performed before or after installation on the vehicle, allowing the training to be performed under real-world conditions. Thus, the training can be performed using images generated by simulation or real-world acquisitions, either before the vehicle starts running (offline) or, for example, while the vehicle is running (online). Once the corresponding results have been verified, the parameters (and values) obtained during training are used to execute various embodiments of the method “online” while the vehicle is running. On the other hand, for this “online” method, the steps performed by the module (or control unit) while the vehicle is running can be performed continuously (e.g., once the brightness drops below a threshold, such as 20 lux or 10 lux) or only after the driver assistance system detects a specific condition.

[0042] In some embodiments, the learned parameters correspond to features representing variations in elements (EPs) present in the projection pattern (4) relative to a reference pattern (4r), which are caused by the distance, shape, and orientation of the visible surfaces of elements (ECs) belonging to the scene content in the training image, and correspond to at least one of the parameters of parallax, shape, and size of the elements (EPs) belonging to the projection pattern (4). In some of these embodiments, the reference pattern (4r) is implicitly learned by the module (1) based on the repetition of the reference pattern in the training image during the pre-training process. In other embodiments, the reference pattern (4r) is transmitted to the module (1) at least during the pre-training process; however, the reference pattern (4r) is no longer necessary during the processing of the acquired real images and may or may not be transmitted to the module (1) or the model.

[0043] Here, the term "parallax" refers to the positional difference inferred from the known distance between the projector (5) and the camera (3). This term is known in the art but has never been used for patterns projected by vehicle headlights and used by processing modules that process images from cameras on vehicles. Furthermore, the pattern (4) allows the module to identify the shape and / or size of the elements of the projected pattern, which is particularly advantageous for distance estimation. Figure 4The upper portion of the figure shows a reference pattern (4r), and the lower portion shows a pattern projected onto a projection region (ZP), which exhibits commonly observed distortions (due to the optical characteristics of the illumination device) and, in particular, deterioration around the edges. Therefore, it is preferable to use a restricted region of interest (ROI) within the projection region (ZP) for training optimization. Thus, in some embodiments, the ROI is restricted relative to the projection region (ZP) within the training image. Furthermore, as... Figure 5 The projected pattern shown also distorts with distance. The left portion shows a portion of the projected pattern on a surface perpendicular to the camera axis and 10 meters from the camera, while the right portion shows the same portion of the pattern when the surface is 100 meters from the camera. Therefore, it will be understood that the dimensions of the pattern elements (EPs), such as... Figure 5 The dimensions of the squares or cells in this chessboard pattern example provide the model with useful information for estimating the distances to surfaces on which a portion of the pattern is projected. On the other hand, Figure 6 The diagram illustrates pattern projection in a more complex real-world scene, showing the deformation of elements (EPs) based on the orientation of the surface on which the pattern is projected. It can be seen that if the surface is horizontal, the squares of the checkerboard pattern deform into rectangles or trapezoids, and if the surface is curved, the squares deform accordingly. Therefore, these elements (EPs) of the pattern provide information about the elements (ECs) belonging to the scene's content, including the orientation, shape, and distance of these elements from the surface. Figure 6 It also showcases the elements of the scene's content (EC), which the model can recognize outside of the areas where patterns are projected (such as a tree on the left or a wall on the right).

[0044] In some embodiments, the learned parameters for providing a graph of distances to elements contained within a monitoring area (ZS) in an image acquired while the vehicle is in motion correspond at least to the shape and / or size features of elements (ECs) belonging to the content of the scene captured in the image. In fact, even in the absence of a pattern, the module is able to estimate the distances to these elements using their size and shape, as explained above and detailed below.

[0045] In some embodiments, during the acquisition and processing of images of the vehicle by the camera (3) while it is in motion by the module (1), the pattern (4) is also projected by the illumination device (5) into a projection area (ZP) in the field of view. Indeed, it has been observed that modules trained using patterns are able to estimate distances even when the pattern is absent, although the performance of the module sometimes degrades depending on the content of the field of view. Therefore, patterns can be omitted entirely, or their use can be limited, for example by not projecting patterns for a specific period (e.g., to avoid glare to vehicles traveling in the opposite direction) and / or by projecting patterns only in a portion of the field of view (to similarly avoid glare in a portion of the field of view while estimating distances in the rest of the field of view), or by simply improving distance estimation in a portion of the field of view, for example, as defined by an obstacle detected by another module. Figure 8 , Figure 9 and Figure 10 This capability was demonstrated. Figure 8 A true map corresponding to the distance in the scene, while Figure 9 This corresponds to the graph obtained by using pattern projection during both training and scene acquisition. It can be seen that all elements (ECs) of the scene content are identified, and the distances are correctly estimated. Figure 10 In itself, it demonstrates the image obtained by using pattern projection only during training, but not during "online" acquisition of the image from the scene. It can be seen that most elements (ECs) in the scene are identified, but specific elements EC1 and EC6 are particularly blurry, and element EC5 is not identified at all. Therefore, it will be understood that the model works even without a pattern when estimating "while the vehicle is running," but is more efficient if the pattern persists during acquisition "while the vehicle is running."

[0046] On the other hand, in some embodiments, the size of the monitoring area (ZS) is greater than or equal to the size of the region of interest (ROI) within the training image and / or the size of the projected region (ZP) within the image acquired by the camera (3) while the vehicle is in motion. This configuration is, for example, in... Figure 1 The results are shown in [the document], and examples of the results are in [the document]. Figure 6 This is illustrated in the diagram. In fact, it has been observed that even by training the module on a limited region of interest (ROI), it is able to handle a wider field of view after the training phase. Therefore, pattern projection can be limited to the ROI during training, but used with a wider monitoring area (ZS) during vehicle operation. Furthermore, as... Figure 8 , Figure 9 and Figure 10As shown, a model correctly trained with a pattern can then work without a pattern, which eliminates many constraints on the extent of the monitoring region (ZS), even if performance is reduced without a pattern.

[0047] Therefore, embodiments that project images onto patterns acquired during vehicle operation are preferred, particularly because the features identified by the model can also correspond to the parallax as explained above. Furthermore, the model's performance is significantly improved by the information provided by variations in the shape and / or size of the pattern (4), which result from the pattern being projected onto elements (ECs) belonging to the scene's content. Therefore, these embodiments are strongly preferred for effective distance estimation in low ambient light conditions. This is because the model can also distinguish features representing variations in elements (EPs) present in the projected pattern (4) relative to the reference pattern (4r), which result from the distance, shape, and orientation of the visible surfaces of elements (ECs) belonging to the scene's content in the training image, and correspond to at least one of the parameters of parallax, shape, and size of the elements (EPs) belonging to the projected pattern (4). These parameters regarding the elements (EPs) of the pattern make it easier to identify elements (ECs) of the scene's content, and these parameters improve the results (in terms of speed and accuracy).

[0048] In some of these advantageous embodiments, during the acquisition and processing of images captured by the camera (3) while the vehicle is in motion, the projection area (ZP) is restricted to a portion of the field of view, which is determined by the detection module based on elements (ECs) contained in the scene captured by the camera (3). The detection module may, for example, detect vehicles traveling in the opposite direction and thus prevent patterns from being projected onto these vehicles to avoid glare, or it may detect unexpected obstacles and instead specifically project patterns onto those obstacles. This improves distance estimation within the projection area (ZP), and the monitoring area (ZS) can then be restricted to the projection area (or even to a region within the projection area), but the model can still continue to estimate distances outside the restricted projection area, and therefore there is no need to restrict the monitoring area (ZS) unless, for example, computation time needs to be accelerated. This embodiment makes it possible, for example, to target areas in the field of view where estimation becomes critical, or to limit the use of computational resources, for example, to benefit other functions or to save energy.

[0049] Finally, in some embodiments, the method includes projecting at least one second pattern onto, for example, a second projection area (or onto, for example, a second projection area) by at least one lighting device (5). Figure 3(The same area shown). A single area allows for pattern overlay and, for example, enables the presence of a pattern at some locations in the scene by means of one of the devices, even when obstacles prevent another device from projecting the pattern onto those locations. Different patterns combine the advantages of both patterns for identifying the features of the scene's content. In some embodiments of these examples, the second pattern has the same content as the first pattern (4) but with the same or different resolution. Using different resolutions also has advantages in terms of estimation accuracy and processing time. In other embodiments of these examples, the second pattern has content different from that of the first pattern (4). This in particular doubles the amount of information that can be extracted from the pattern. For example, the first pattern may include vertical lines, while the second pattern includes horizontal lines. Combining them provides a good checkerboard pattern for distance estimation, while ensuring a good distance map even if an obstacle hides one of the two patterns in a portion of the field of view. In some embodiments, the first pattern (4) and the second pattern are projected into the field of view at different distances from the vehicle. Thus, optimization of distance estimation by the presence of the pattern "while the vehicle is in motion" can be achieved over a larger portion of the field of view.

[0050] As will be understood from this application, the pattern is defined during training because the model implicitly learns the pattern, but the pattern can also be transmitted for training and / or during vehicle operation (e.g., under real-world conditions and for processing acquired images). Furthermore, some embodiments allow the pattern to be selectively applied to specific regions of interest within a scene. Such targeted application can prove particularly useful in driver assistance or autonomous driving scenarios where accurate information about the distance to specific objects is required (e.g., detecting dropped cargo on a highway).

[0051] Because edge sharpness is generally important regardless of distance, the model is trained to detect edges. Furthermore, the model allows for accurate representation of surfaces in depth maps, thus enabling the identification of flat areas such as roads and buildings, and facilitating the reconstruction of object surfaces, for example, in possible classification tasks.

[0052] As will be understood from this application, the present invention utilizes the pixelated headlights of modern vehicles, which are capable of readily projecting patterns (although other means of projecting patterns are conceivable and therefore within the scope of this application, such as using a mask in front of the headlight). Extensive testing has demonstrated the effectiveness of the method, highlighting significant and robust improvements in depth perception both within and outside the illuminated area. The versatility of the method is also demonstrated by its implementation in conventional models, such as the U-net model, as well as in more complex models with advanced architectures, such as the Adamins model and the DepthFormer model. Therefore, it will be understood that the present invention is not limited to these exemplary models and can be used with other types of past, present, and future models.

[0053] This application describes various technical features and advantages with reference to the accompanying drawings and / or various embodiments. Those skilled in the art will understand that technical features of a given embodiment can actually be combined with features of another embodiment, unless explicitly stated otherwise, or if these features are clearly incompatible, or if the combination does not provide a solution to at least one of the technical problems mentioned in this application. Furthermore, unless explicitly stated otherwise, the technical features described in a given embodiment can be separated from other features of that embodiment.

[0054] A detailed list of reference numerals in the attached figures: 1 Image Processing Module 2 Control Unit 3 cameras 4. Pattern 4r reference pattern 5 lighting fixtures.

Claims

1. A method for estimating distance or depth in a vehicle's field of view using an image processing module (1) under nighttime or low-light conditions, the vehicle firstly including at least one lighting device (5) for projecting at least one luminous design into the field of view, and secondly including at least one camera (3) for acquiring an image from the field of view, the method being characterized in that the method comprises: - A pre-training process for the module (1) is performed to estimate distances based on data corresponding to multiple images, referred to as training images, from the field of view, in which the lighting device (5) projects at least one high-contrast drawing, referred to as a pattern (4), into a projection region (ZP) during the pre-training phase. This allows the module (1) to generate multiple learned parameters corresponding to features of elements (EP) belonging to the pattern (4) and features of elements (EC) belonging to the content of the scene captured in these training images. Based on the distances of these elements (EP, EC), a map of the distances of these elements is then created in at least one region of interest (ROI) within these images. - When the vehicle is running at night or in low light conditions, the module (1) acquires and processes images transmitted by the camera (3) to provide a map of the distances of the elements contained in at least one monitoring area (ZS) within these images acquired while the vehicle is running, using the parameters learned during the training phase.

2. The method as described in claim 1, characterized in that, The learned parameters correspond to features representing the changes of elements (EP) existing in the projection pattern (4) relative to the reference pattern (4r), which are caused by the distance, shape and orientation of the visible surfaces of elements (EC) belonging to the content of the scene in the training image, and correspond to at least one of the parameters of the parallax, shape and size of the elements (EP) belonging to the projection pattern (4).

3. The method as described in claim 2, characterized in that, The reference pattern (4r) is implicitly learned by the module (1) based on the repeated occurrence of the reference pattern in the training image during the pre-training process.

4. The method as described in claim 2 or 3, characterized in that, The reference pattern (4r) is transmitted to the module (1) at least during the pre-training process.

5. The method as described in any one of the preceding claims, characterized in that, The region of interest (ROI) is constrained relative to the projected region (ZP) within the training image.

6. The method as described in any one of the preceding claims, characterized in that, The learned parameters are used to provide a map of the distances of the elements contained in the monitoring area (ZS) within the image acquired while the vehicle is in operation, and the learned parameters correspond at least to the shape and / or size features of elements (ECs) belonging to the content of the scene captured in the image.

7. The method as described in any one of the preceding claims, characterized in that, During the acquisition and processing of the image obtained by the camera (3) of the vehicle during operation by the module (1), the pattern (4) is also projected by the lighting device (5) into the projection area (ZP) in the field of view.

8. The method as described in any one of the preceding claims, characterized in that, The size of the monitoring area (ZS) is greater than or equal to the size of the region of interest (ROI) in the training image and / or the size of the projection area (ZP) in the image acquired by the camera (3) while the vehicle is in motion.

9. The method as described in any one of claims 7 and 8, characterized in that, During the acquisition and processing of images obtained by the camera (3) while the vehicle is in motion, the projection area (ZP) is restricted to a portion of the field of view, which is determined by the detection module based on the elements (EC) contained in the scene captured by the camera (3).

10. The method as described in any one of the preceding claims, characterized in that, This includes projecting at least one second pattern into a second projection area using at least one lighting device (5).

11. The method as described in claim 10, characterized in that, The second pattern has the same content as the first pattern (4), but has the same or different resolution.

12. The method as described in claim 10, characterized in that, The second pattern has content that is different from that of the first pattern (4).

13. The method according to any one of claims 10 to 12, characterized in that, The first pattern (4) and the second pattern are projected into the field of view at different distances from the vehicle.

14. A computing element comprising means for performing steps of the method as described in any of the preceding claims.

15. A computer program comprising instructions which, when executed by an image processing module (1), cause the module (1) to perform the steps of the method as claimed in any one of claims 1 to 13.

16. A vehicle assistance system, comprising: - Image processing module (1), which is capable of performing the steps of the method as described in any one of claims 1 to 13; - At least one lighting device (5) capable of projecting at least one pattern (4) into the field of view; - At least one camera (3), said at least one camera being used to transmit images acquired while the vehicle is in motion.

17. The system (1) as described in the preceding claim, wherein, The lighting device (5) is able to project the pattern based on the fact that it includes multiple light sources, each of which is able to illuminate a restricted area in the field of view with a variable intensity, and the combined control of these intensities according to the location allows the pattern (4) to be obtained.

Images