A process of correcting images acquired using a rolling shutter.

Abstract

Method for correcting an object (OBJ) represented in images acquired (IMG1, IMG2) by a camera (3) by a rolling shutter acquisition process, said method comprising: (700) obtaining a first image (IMG1) of the object (OBJ) at a first acquisition time (T1), associated (710) with initial data including a first distance (X1) of the object (OBJ) relative to the camera (3), (720) acquiring a second image (IMG2) of the object (OBJ), at a second acquisition time (T2), associated (730) with a second distance (X2) of the object (OBJ) relative to the camera (3), if (740) a first criterion is satisfied, (750) determining a corrected distance (X2,corr) of the object (OBJ) relative to the camera (3) at the second acquisition time (T2) from the second distance (X2), said corrected distance (X2,corr) depending on a reading time per line (λ) of the camera sensor (3). Abstract figure: Figure 6

Description

Title of the invention: Method for correcting images acquired via a rolling shutter Technical field

[0001] The present disclosure relates to the field of image acquisition by cameras equipped with electronic shutter sensors, with acquisition via a rolling shutter. In particular, the present disclosure relates to a method for correcting objects in images acquired by such cameras. Prior art

[0002] The acquisition of images by a camera (e.g., a camcorder, a camera, a photo camera, etc.) requires a shutter step during which the photosensitive surface of the device is exposed to light. Such a photosensitive surface typically corresponds to a sensor, for example of the “Charged Coupled Deflected” (or CCD) or “Complementary Metal-Oxide-Semiconductor” (or CMOS) type, or even to a silver film. The precision and quality of an acquired image depend in particular on the parameters linked to the shutter such as the type of shutter (mechanical or electronic), the shutter speed, the exposure conditions or even the shooting conditions (e.g., the brightness).

[0003] In particular, a global image acquisition mechanism, or "global shutter", is distinguished from an image acquisition mechanism via a rolling shutter, or "rolling shutter", also sometimes referred to as line-by-line acquisition. Indeed, in the "global shutter" acquisition mechanism, the photosensitive surface is fully exposed at a single time at each shutter cycle (delimiting an exposure cycle of the photosensitive surface), typically using a mechanical shutter, then fully closed for the transfer of the electrical charges recorded by the photodiodes of the photosensitive surface to the reading zones. In the "rolling shutter" acquisition mechanism, the different lines of pixels forming the photosensitive surface are exposed with a time shift, so that the entire photosensitive surface is not fully read at the same time.

[0004] Most cameras use the rolling shutter acquisition mechanism, particularly with the use of electronic sensors. Indeed, the "rolling shutter" acquisition mechanism makes it possible to shorten the occultation time of the sensor's photodiodes, so that each line of the sensor, once read, can be re-exposed, and can be used in particular for aiming. sensor sensitivity is improved compared to global acquisition. The rolling shutter also reduces image noise compared to voltage fluctuations generated when the image is read all at once in "global shutter". The "rolling shutter" acquisition mechanism then allows for longer exposure times, which makes it possible to capture fast movements, scenes with strong lighting or to achieve more precise zoom effects. In addition, in the absence of a mechanical shutter opening and closing with each exposure cycle, electronic sensors operating in "rolling shutter" are more stable, reliable and silent. Finally, compared to the "global shutter" acquisition mechanism, for which several transistors per pixel are required, the "rolling shutter" acquisition mechanism allows for reduced production costs.

[0005] However, the rolling shutter acquisition mechanism has a main drawback, linked to the line-by-line reading of the camera's sensor. Indeed, the acquisition of data line by line leads to a latency time for the reading of each line of the sensor. In particular, in the case of variations in brightness or rapid movements of the captured object and / or the camera during acquisition, the resulting image may present distortions or deformations of the objects. Typically, the vertical lines of objects captured during a rolling shutter acquisition when the camera moves horizontally present a distortion, called the "jello effect". Another example of deformation is that the elements moving quickly towards the bottom of the image appear thicker while the elements moving quickly towards the top of the image appear thinner than they actually are.The distortions and deformations caused by the line-by-line acquisition mechanism are more generally referred to as the "rolling shutter" phenomenon.

[0006] The “rolling shutter” phenomenon then generates a bias in the estimation of the positions, the occupied space or even the movements of the captured objects. Such a bias can particularly pose a problem in the context of detection and tracking of vehicles implemented by intelligent traffic management systems (e.g., road or air), for which the observed objects and / or observers move quickly (for example, a vehicle carrying an integrated system connected to a camera and observing neighboring vehicles on neighboring highway lanes) and require tracking precision (for example, in the case of autonomous vehicles). Summary

[0007] The present disclosure improves the situation.

[0008] A method is proposed for correcting at least one object represented in images acquired by a camera, said images being acquired by a process of line-by-line reading of a camera sensor, known as the rolling shutter acquisition process, said method comprising: - obtaining a first image of the object, acquired at a first acquisition time, said first image being associated with initial data including at least a first distance of the object relative to the camera, - acquiring a second image of the object, at a second acquisition time subsequent to the first acquisition time, said second image being associated at least with a second distance of the object relative to the camera, - if a first criterion is satisfied, determine a corrected distance of the object relative to the camera at the second acquisition time from at least the second distance, characterized in that said corrected distance depends at least on one reading time per line of the camera sensor.

[0009] Consequently, the method advantageously makes it possible to correct estimation biases during the detection of objects in images acquired by a rolling shutter camera. Such estimation biases in fact lead to errors in estimating the distance between the detected object and the camera in the acquired image, and therefore a fortiori to errors in detecting the position of the object in the acquired image, or even to imprecision in the modeling of a virtual object associated with the detected object. These estimation biases then impact the precision and effectiveness of identification, tracking, alerting or assistance methods which can use the results of object detection and their associated data in the acquired images.

[0010] The proposed method then makes it possible to correct, a posteriori of the image acquisition and the data relating to the object represented, an estimated distance between the object and the camera, which is distorted by the “rolling shutter” effect due to the time lag existing between the start of the acquisition of an image and the acquisition of the object as such in the image. Such a “rolling shutter” effect is moreover intrinsically linked to the reading time per line of the camera sensor, or of the photosensitive surface made up of pixels of the camera, which generates a time lag between the reading of the different pixels allowing the acquisition of an image. The proposed method then corrects the estimated distance of the object by taking into account such a reading time per line.

[0011] By an image acquired at an acquisition time, reference is made to an acquisition time associated with the image. The acquisition time may refer to a specific time at which the acquisition begins. In particular, the acquisition of the image may end at a time different from the time at which the acquisition begins, due to the operation of rolling shutter acquisition.

[0012] By first and second distances of the object relative to the camera, are designated distance values ​​between the object and the camera estimated respectively from the different acquisitions. In particular, the second distance is distinct from the first distance. In other words, there is a relative movement between the object represented and the camera, at least over the period between the first acquisition time and the second acquisition time.

[0013] By a correction of an object represented in images, it is meant a correction of the data relating to the object detected in the images. Such data may for example include a distance from the object to the camera, but also a position, coordinates, segments, optical flows or even a modeling of the detected object. In the following, reference is made indiscriminately to a correction of an object in the image or to a correction of the image.

[0014] According to another aspect, there is proposed a device on board a main vehicle and connected to a camera configured to acquire images of an environment of the main vehicle according to a rolling shutter process, said device including a unit for correcting an object belonging to the environment, said unit being configured to implement the proposed method.

[0015] According to another aspect, there is provided a computer program comprising instructions for implementing the proposed method when this program is executed by a processor.

[0016] According to another aspect, there is provided a non-transitory recording medium readable by a computer on which is recorded a program for implementing the method when this program is executed by a processor.

[0017] The features set out in the following paragraphs may, optionally, be implemented, independently of one another or in combination with one another:

[0018] In one embodiment, the corrected distance is determined as a further function of a line difference between a first position of the object on the first acquired image and a second position of the object on the second acquired image.

[0019] Consequently, the method advantageously makes it possible to take into account a spatial change of the object in the acquired images.

[0020] By first and second positions of the object on the first and second acquired images, reference is made to positions on pixels of the images. In particular, an object detected on an image acquired by rolling shutter is captured during a reading of a given line of pixels. Thus, by a line difference, reference is made to a difference in lines of pixels of the camera sensor read during the acquisition of the first and second images, on which the object is captured.

[0021] In one embodiment, said line difference is related to a relative movement between the object and the camera.

[0022] Consequently, the method advantageously takes into account the relative displacement existing between the object and the camera, which generates undesirable distortion and deformation effects.

[0023] In one embodiment, corrected distance is determined by:

[0024] X(X2-X1)+X1

[0025] where: - X2coir is the corrected distance, - XI is the first distance, - X2 is the second distance, - dt is the difference between the start of the first acquisition time and the start of the second acquisition time, - dl is a line difference between a first position of the object on the first acquired image and a second position of the object on the second acquired image, and - X is the reading time per line.

[0026] In one embodiment, the first criterion is satisfied if one of the camera and the object is static between the first acquisition time and the second acquisition time.

[0027] In one embodiment, the method further comprises: - determine, from at least the corrected distance, a corrected position of the object on the second acquired image.

[0028] Consequently, the method makes it possible to reposition the object as detected on the image, in coherence with the correction of the distance of the object relative to the camera. Thus, the corrected position of the object takes into account the time shift that occurred during the acquisition of the image.

[0029] By a corrected position of the object, it can be understood a new matching between pixels of the second image and the detected object, so that the detected object is associated with a new position, different from the position estimated solely from the acquisition of the second image.

[0030] In one embodiment, the corrected position corresponds to an estimate of the line on which the object is located in the second image at the start of the second acquisition time.

[0031] Consequently, the proposed method advantageously makes it possible to replace the object in relation to the time at which it is actually captured in the image, and not simply compared to the second acquisition time associated with the second image, these two times differing due to the acquisition by rolling shutter.

[0032] In one embodiment, the method further comprises: - from the corrected position and the initial position, determine corrected data relating at least to a relative movement between the object and the camera between the first acquisition time and the second acquisition time.

[0033] Consequently, the proposed method makes it possible to restore the evolution of an object in the images acquired relative to the camera. In particular, such a method then makes it possible to improve the precision of tracking the object in images successively acquired by rolling shutter, despite the presence of a relative movement between the object and the camera. Thus, the known effects of the "jello effect" or "spatial aliasing" type are reduced, allowing better restitution of the object represented in the images, as well as more precise data relating to the object and its evolution, for better purposes of tracking the object represented in the images. Brief description of the drawings

[0034] Other characteristics, details and advantages will appear on reading the detailed description below, and on analyzing the attached drawings, in which: Fig.l

[0035] [Fig.l] [Fig.l] shows an example of an image acquired by a camera via a rolling shutter according to the prior art. Fig. 2

[0036] [Fig.2] [Fig.2] shows a diagram of a step of acquiring an image of a vehicle via a rolling shutter at a first instant according to the prior art. Fig. 3

[0037] [Fig.3] [Fig.3] shows a diagram of a step of acquiring an image of a vehicle via a shutter rolling down at a second instant according to the prior art. Fig. 4

[0038] [Fig.4] [Fig.4] shows a diagram of a step of acquiring an image of a vehicle via a shutter rolling down at a third instant according to the prior art. Fig. 5

[0039] [Fig.5] [Fig.5] shows a schematic of an image acquired from a vehicle via a shutter occurring at the end of the acquisition steps at the first, second and third instants according to the prior art. Fig. 6

[0040] [Fig.6] [Fig.6] shows diagrammatically the steps of the image correction method according to a embodiment. Fig. 7

[0041] [Fig.7] [Fig.7] shows a succession of steps of a method for correcting an image acquired via a rolling shutter according to one embodiment. Fig. 8

[0042] [Fig.8] [Fig.8] shows schematically a step of acquiring an image of a vehicle via a rolling shutter at a first acquisition time according to one embodiment. Fig. 9

[0043] [Fig.9] [Fig.9] a step of correcting an image of a vehicle acquired via a rolling shutter at a second acquisition time according to an embodiment. Description of the embodiments

[0044] Reference is now made to [Fig.l]. [Fig.l] is an example of an image acquired of a road environment by a camera, more simply referred to hereinafter as camera 3, operating with a rolling shutter. For example, such an image may be acquired by a camera 3 mounted on a vehicle, called observer vehicle VO, moving on a fast lane (or motorway). The spots visible in [Fig.l] may in particular correspond to segments or optical flows linked to relative movements existing between the camera 3 (and more generally, the observer vehicle VO) and the objects captured in the image (e.g., other vehicles, road markings, lampposts or traffic signs, etc.).

[0045] The acquisition of images by the rolling shutter camera 3 can generate known effects of distortion and deformation of the objects in the image, designated by the phenomenon of “rolling shutter” (the expression “rolling shutter” designates both the mode of line-by-line acquisition of the images, and the phenomenon of distortion of the objects observable in certain cases during such an acquisition). Such effects can for example be observed in zones A and B of the image of [Fig. 1]. Such effects are linked to the conventionally known operation of the rolling shutter. When acquiring an image of a scene according to a field of view of the camera 3, a photosensitive surface of a sensor of the camera 3 (for example, a CCD or CMOS sensor) is exposed. Such a photosensitive surface is notably constituted by a matrix of pixels, each pixel then capturing a part of the scene, which is restored in the image.In particular, in the case of a rolling shutter, the pixels of the photosensitive surface are not all exposed simultaneously (as in the case of acquisition by "global shutter") but line by line, so that the entire scene is not captured at the same time. Indeed, in an image acquisition by rolling shutter, the scanning of all the pixels of the photosensitive surface is done line by line (i.e., by line of pixels), each . line of pixels being read with a predefined reading time per line X. Such a reading time per line X depends in particular on the sensor of the camera 3. Thus, the reading of each line of pixels of the photosensitive surface is done with a time offset, compared to the lines which precede it.

[0046] The “rolling shutter” effect caused by such a time shift in the reading of the lines of pixels of the sensor of the camera 3 is particularly accentuated when a rapid relative movement exists between an object in the captured scene and the camera 3. Typically, distortions can be observed linked to the “rolling shutter” effect when the camera 3 captures a rapidly moving object, or when the camera 3 moves rapidly relative to a captured object for example. The level of distortion linked to the “rolling shutter” effect during the acquisition of images can in particular depend on the characteristics of the relative movement between the camera and the object but also on the exposure time of the photosensitive surface, the quantity of light captured or even on the characteristics of the camera 3 such as the speed or the reading time per line X.

[0047] The rolling shutter effect caused by the operation of the rolling shutter is schematically illustrated in Figures 2 to 5 as an example. Figures 2 to 4 schematically illustrate the successive lines of pixels during the acquisition of images of a scene by a camera 3 with a rolling shutter at several respective times t1, t2, t3, a vehicle moving in the scene in the direction of the arrow indicated in [Fig. 2]. [Fig. 5] illustrates a final image resulting from the image acquisition of Figures 2 to 4. The camera 3 is considered fixed relative to the scene.

[0048] In [Fig. 2], a phase is considered during an image acquisition of the scene at a first instant t1, during which the pixel line 14 is read. At this first instant t1, the vehicle enters the field of view of camera 3 and is partially captured on a right portion of the image during the reading of line 14.

[0049] In [Fig. 3], another phase is considered during the image acquisition of the scene at a second time t2 subsequent to the first time t1, during which the pixel line 15 is read. Between the first time t1 and the second time t2, the vehicle has moved in the scene so as to be captured on a central portion of the image when reading the line 15.

[0050] In [Fig. 4], another phase is considered during the image acquisition of the scene at a third time t3 subsequent to the second time t2, during which the pixel line 16 is read. Between the second time t2 and the third time t3, the vehicle has moved into the scene and begins to leave the field of view of the camera 3, so that the vehicle is partially captured on a left portion of the image when reading the line 16.

[0051] In [Fig.5] is illustrated a final image constructed at the end of the acquisition step including the acquisition phases at times t1, t2, t3. Typically, the final image is constructed once all the lines of pixels have been read. As schematically represented in [Fig.5], the vehicle rendered in the image is distorted due to the “rolling shutter” acquisition, the vehicle having moved longitudinally in the captured field of view. In other words, between the moment when line 14 is read (in [Fig.2]) and allows the capture of a part of the scene and the moment when line 16 is read (in [Fig.4]) and allows the capture of another, lower, part of the scene, the vehicle has moved and the final image (in [Fig.5]) reproduces the deformed or distorted vehicle. The distortion illustrated in [Fig.5] is purely schematic.

[0052] In particular, the observed distortion depends in particular on the nature of the relative displacement between the object and the camera 3. For example, in the case of horizontal relative displacement between the observed object and the camera 3, the “rolling shutter” effect is similar to an oscillation effect, known as the “jello effect”. Typically, vertical objects in the field of view appear tilted. Thus, with reference to [Fig. 1], zone A illustrates such an oscillation effect, where a vertical lamppost in the scene appears tilted. In another example, in the case of lateral relative displacement between the observed object and the camera 3, the “rolling shutter” effect is similar to a spatial aliasing effect, known as “spatial aliasing”. Typically, objects having a displacement towards the bottom of the image appear elongated. Thus, with reference to [Fig.l], zone B illustrates such a spatial aliasing effect, where the optical flows closest to the ground, typically corresponding to the relative movement between camera 3 and the ground markings, appear elongated.

[0053] Reference is now made to [Fig.6]. [Fig.6] schematizes as an example a context for implementing the image correction method which will be detailed in [Fig.7]. An observer point is considered, corresponding for example to an observer vehicle VO in a scene comprising at least one observed object, for example a wheel OBJ of a neighboring vehicle. The scene considered can correspond to any environment, for example a road, in which, at each instant, the observer point and the observed object have positions, variable or not, which can be expressed by coordinates, for example Cartesian coordinates in three dimensions. [Fig.6] illustrates a simplified example by representing a single dimension along the X axis, but it is assumed that the observer point and the observed object evolve in a three-dimensional environment.

[0054] In particular, a relative movement between the observer point and the observed object is considered. For example, in the context of [Fig.6], the observer vehicle VO moves in the direction and sense of the X axis relative to a neighboring vehicle (and therefore to the wheel OBJ of such a neighboring vehicle). Such a movement is illustrated in the [Fig.6] by several positions occupied by the observer vehicle VO in relation to different acquisition times T, including a first acquisition time T1 and a second acquisition time T2, during which the observer vehicle VO moves along the X axis.

[0055] The observer point coincides with a camera 3 capable of acquiring images according to a known acquisition mode of the “rolling shutter” type, or with a rolling shutter, as described previously. For example, in [Fig. 6], the camera 3 can be on board the observer vehicle VO. The field of view of the camera 3 delimits a scene of the environment, which at least partially and at least temporarily includes the observed object. For example, in [Fig. 6], the camera 3 is on board a front face of the observer vehicle VO, so that the field of view of the camera 3 covers a view upstream of the observer vehicle VO.

[0056] Image acquisition parameters by the camera 3 may be predefined or preconfigured. Such parameters may in particular include parameters linked to the sensor of the camera 3 such as the acquisition time, the acquisition frequency, the acquisition speed, the reading time per line X, the reading speed per line, the number of lines on the photosensitive surface of the camera 3.

[0057] The camera 3 is connected to a processing circuit configured to implement the image correction method which will be detailed in [Fig.7]. Such a processing circuit, illustrated in [Fig.6], may comprise at least one processor 1 and one memory unit 2. The processing circuit may be on board the observer vehicle VO, as shown in [Fig.6]. Alternatively, the processing circuit may also be remote from the observer vehicle VO.

[0058] The context of implementation of the image correction method is illustrated in [Fig.6] as an example. Alternatively, the observer point and / or the observed object may differ from vehicles and the observed environment and scene may differ from a road. Furthermore, the relative displacement between the observer point and the observed object may be multidimensional or in directions other than along the X axis. Furthermore, the relative displacement between the observer point and the observed object may be linked to a movement of the observer point or the observed object.

[0059] Reference is now made to [Fig. 7]. [Fig. 7] illustrates a succession of steps for implementing a method for correcting images acquired by a rolling shutter camera 3, in a rolling shutter type acquisition mode. Such a succession of steps includes a first acquisition phase at steps 700, 710 illustrating a conventional image acquisition and a second image acquisition and processing phase at steps 720, 730, 740, 750, 760 and 770, illustrating a correction of the rolling shutter effect in the acquired images.

[0060] An observer point corresponding to an observer vehicle VO and an observed object OBJ are considered, corresponding to a portion of a vehicle neighboring the observer vehicle VO, for example the wheel of a neighboring vehicle. The observer vehicle VO is moving relative to the observed object OBJ. The camera 3 is mounted on the observer vehicle VO so as to capture a scene surrounding the observer vehicle VO according to a predefined field of view.

[0061] In a step 700, at least one first image IMG1 of the scene acquired by the camera 3 is considered. Such a first image IMG1 is associated with a first acquisition time TL. Such a first acquisition time T1 conventionally corresponds to the moment when the capture of the scene is triggered, namely the moment when the first line of pixels of the sensor of the camera 3 is read, in the context of a “rolling shutter” acquisition. During the acquisition of the first image IMG1, the observer vehicle VO and the observed object OBJ occupy given positions in the environment. In particular, the scene captured on the first acquired image IMG1 includes the observed object OBJ.

[0062] In a step 710, initial data associated with the first acquired image IMG1 are obtained. Such initial data may include data relating to the object OBJ captured in the first acquired image IMG1, for example data relating to a position of the observed object OBJ. In particular, the data relating to the position of the observed object OBJ may be estimated with respect to a reference frame associated with the acquired image, called an image reference frame. Such an image reference frame is in particular two-dimensional. In other words, a pixel or a set of pixels of the first acquired image IMG1 may be associated with a position of the object OBJ observed in the first acquired image IMG1. In other words, a pixel or a set of pixels of the first acquired image IMG1 may be associated with a position of the object OBJ in the first acquired image IMG1.In one embodiment, the data relating to the position of the observed object OBJ can be estimated with respect to a reference frame associated with the real environment in which the observer vehicle VO and the observed object OBJ are located, called a world reference frame. Such a world reference frame is in particular three-dimensional. In other words, three-dimensional coordinates can be associated with a position of the observed object OBJ in the scene.

[0063] The initial data obtained in step 710 may in particular include a first estimated distance Xi between the observed object OBJ and the camera 3. In other words, a depth of the observed object OBJ in the first acquired image IMG1 may be estimated. Such a first distance Xi may be estimated for example from a classifier allowing the detection and tracking of objects in the acquired images. For example, such a first distance Xi may be estimated from the first acquired image IMG1 and other images acquired by the camera at acquisition times preceding the first acquisition time Tl. The first distance Xi can then be estimated from analysis of optical flows and / or segments detected on the images acquired at the first acquisition time Tl or upstream, for example with a Kalman filter. The first distance Xi of the object OBJ associated with the first acquisition time Tl can be obtained from any known estimation method, based on the analysis of the first image IMG1 acquired or of several images successively acquired by the camera 3. In addition, the first distance Xi of the object OBJ can be expressed in the world frame, as illustrated in [Fig.6], such a first distance Xi reflecting an estimated real distance between the observer vehicle VO and the observed object OBJ. Such a first distance Xi can in particular be obtained from the data relating to the position of the observed object OBJ, expressed in the world frame or in the image frame.In the case of data relating to the position of the observed object OBJ expressed in the image frame, the pinhole model can for example be used to deduce the first distance Xb.

[0064] With reference to [Fig.8], a first acquired image IMG1 including the detection of the object OBJ observed at a first estimated distance Xi is illustrated.

[0065] In particular, such a first distance Xb and more generally the data relating to the position of the observed object OBJ, integrate an estimation bias linked to the “rolling shutter” effect previously described. Indeed, as illustrated in FIGS. 6 and 8, the observed object OBJ is detected on the first image IMG1 acquired at the level of the fifth line of pixels 15. Given the “rolling shutter” acquisition mode of the first image IMG1, a time lag exists between the acquisition of the first line of the first image IMG1, implemented at the first acquisition time T1 and the acquisition of the fifth line of pixels 15, during which the observer vehicle VO has moved.

[0066] It is proposed to correct such an estimation bias during a second acquisition phase described in the following steps.

[0067] In steps 720 and 730, a second image and second data are obtained and associated with a second acquisition time T2, subsequent to the first acquisition time TL. In particular, these steps 720, 730 are implemented in a similar manner to the steps 700, 710 previously described. In addition, in step 730, from the second data obtained and the initial data obtained in step 710, an association of an object OBJ detected on both the first image IMG1 and the second image IMG2 acquired can be carried out. In other words, it can be determined by known means that the object detected on the two images IMG1, IMG2 belong to the same subject (eg, the same wheel of the same neighboring vehicle). A second distance X2 can then be estimated between the object OBJ detected on the second image and camera 3.

[0068] The object OBJ as detected in the second image is illustrated in solid lines in [Fig.9]. It can be observed that compared to [Fig.8], the observed object OBJ has moved closer to the camera 3 (the estimated depth of the object OBJ in the second image is less than the estimated depth of the object OBJ in the first image IMG1). Such an evolution reflects for example that the observer vehicle VO is moving forward on the road and is approaching an adjacent level of the neighboring vehicle, parked on the shoulder.

[0069] Like the first distance Xi estimated on the first acquired image IMG1, the second distance X2 presents an estimation bias linked to the “rolling shutter” effect. Such an estimation bias is notably illustrated in [Fig.6]. At the second acquisition time T2, the acquisition of the second image IMG2 begins. More precisely, at the instant corresponding to the second acquisition time T2, the first line 11 is read, corresponding to a first portion of the second image IMG2 (in an upper zone of the second image IMG2) on which the observed object OBJ does not appear. The following lines are then successively read at times offset from the second acquisition time T2, due to the operation of the line-by-line acquisition. More precisely, the n-th line is finished being read at the time:

[0070] T2 + X xn

[0071] where T2 corresponds to the second acquisition time, X corresponds to the reading time per line and n corresponds to the number of the line read.

[0072] As illustrated in Figures 6 and 9, the object OBJ observed during the second acquisition phase associated with the second acquisition time T2 is captured when reading line 17. In other words, the observed object OBJ is in reality captured not at the second acquisition time T2 but at the time:

[0073] T2 + X x (n=7)

[0074] Consequently, the second distance X2 estimated between the object OBJ and the camera 3 is biased in that, between the second acquisition time T2 and the instant at which the object OBJ is actually captured, the observer vehicle VO has moved from a position VO* to a position VO, as illustrated in [Fig.6].

[0075] In a step 740, it is verified whether a first criterion is satisfied in order to allow the correction of the second acquired image IMG2, and more particularly of the object OBJ observed on the second acquired image IMG2. Such a first criterion can be satisfied if an element among the camera 3 and the object OBJ is static (i.e., non-mobile) between the first acquisition time T1 and the second acquisition time T2. More particularly, the first criterion can be satisfied if one element among the camera 3 and the object OBJ is static between the acquisition of the first image IMG1 and the acquisition of the second image IMG2. In other words, in step 740, it is verified that the relative movement existing between the object OBJ and the camera 3 is linked to the movement of only one of the two elements and not of both elements at the same time. For example, in the context of a camera 3 on board a mobile observer vehicle VO, it is verified that the observed object OBJ (and therefore the neighboring vehicle) is stationary in the scene. For this, step 740 can implement any known method for distinguishing the fixed or mobile nature of the object OBJ relative to the camera 3.For example, it is possible to estimate, from the first acquired image IMG1 and the movement characteristics (a priori known) of the observer vehicle VO (or of the object OBJ, in the case where the camera 3 is fixed), a predicted or expected position of the object OBJ in the second image IMG2, or to use a Kalman filter in order to obtain a predicted position of the object OBJ in the second image IMG2. In such an example, if the object OBJ (or a segment or optical flows associated with the object OBJ) is detected at the predicted position, the first criterion is considered satisfied. If the first criterion is not satisfied in step 740, the correction method is not continued on the detected object OBJ. Steps 720 to 740 can then be implemented on other subjects detected on the second image IMG2 and / or on other acquired images. If the first criterion is satisfied in step 740, a step 750 is implemented.

[0076] In step 750, a corrected distance X2>coir relative to the second distance X2 between the observed object OBJ and the camera 3 is determined. The corrected distance X2jCOir can in particular be expressed by: 100771 = 3^ x ( X2 -X, ) +AS,

[0078] where X2>coir is the corrected distance between the object OBJ and the camera 3, Xi is the first distance, X2 is the second distance, dt is the difference between the start of the first acquisition time T1 and the start of the second acquisition time T2, d1 is a line difference between a position of the object OBJ determined on the first image IMG1 and a position of the object OBJ determined on the second image IMG2 and X is the reading time per line.

[0079] As illustrated in [Fig.6], the corrected distance X2jCOIT makes it possible to estimate the distance separating the observer vehicle VO and the object OBJ observed at the second acquisition time T2, taking into account the time shift linked to the acquisition of the second image IMG2 in “rolling shutter”.

[0080] At a step 760, from at least the corrected distance X2jCOIT, a corrected position of the object OBJ is determined on the second acquired image IMG2. In other words, the pixels of the second image IMG2 associated with the detected object OBJ are realigned, so as to correspond to the corrected depth obtained (i.e., to the corrected distance X2>coir). In particular, determining such a corrected position amounts to determining the line of the sensor at which the object OBJ is captured (in the example illustrated in figures 6 and 9, this is line 17).

[0081] Thus, step 770 makes it possible to reduce the estimated position bias of the object OBJ on the second image IMG2 (therefore in the image frame) from the corrected distance X2jCOIT in the world frame.

[0082] In a step 770, the second data associated with the observed object OBJ (obtained in step 730 with a bias) are corrected from the corrected distance X2>coir and the corrected position (in pixels) of the observed object OBJ respectively determined in steps 750 and 760. In step 770, a corrected observed object OBJ* is thus obtained. In other words, the optical flows and / or segments associated with the object OBJ as detected in the second image IMG2, or even a virtual object associated with the object OBJ for its tracking by a classifier or any other tracking method) can be corrected a posteriori of the detection of the object OBJ on the second image IMG2, so as to correct the second acquired image IMG2 with respect to the “rolling shutter” effect. As illustrated in [Fig.9], the corrected object OBJ* is determined. The corrected data may in particular relate to the relative movement actually existing between the object OBJ* and the camera 3 (eg, via corrected optical flows).

[0083] Optionally, such a corrected object OBJ* and the corresponding corrected data can then be exploited or used by or for other processing units, for example by being fed to a Kalman filter, to a classifier etc. in the context of methods of tracking, detection, alert, driving assistance, surveillance etc. For this, the processing circuit proposed for the implementation of the image correction method can be integrated or connected to other processing circuits exploiting the detected objects and the associated data from the images acquired by the camera 3.

Claims

Claims

1. Method for correcting at least one object (OBJ) represented in images acquired (IMG1, IMG2) by a camera (3), said images (IMG1, IMG2) being acquired by a line-by-line reading process of a sensor of the camera (3), called a rolling shutter acquisition process, said method comprising: - (700) obtaining a first image (IMG1) of the object (OBJ), acquired at a first acquisition time (T1), said first image (IMG1) being associated ((710) with initial data including at least a first distance (XJ of the object (OBJ) relative to the camera (3), - (720) acquiring a second image (IMG2) of the object (OBJ), at a second acquisition time (T2) subsequent to the first acquisition time (T1), said second image (IMG2) being associated (730) with at least a second distance (X2) of the object (OBJ) relative to the camera (3),- if (740) one of the camera (3) and the object (OBJ) is static between the first acquisition time (T1) and the second acquisition time (T2), (750) determining a corrected distance (X2jCOir) of the object (OBJ) relative to the camera (3) at the second acquisition time (T2) from at least the second distance (X2), characterized in that said corrected distance (X2jCOIT) depends at least on a reading time per line (X) of the sensor of the camera (3) and is determined as a function further of a line difference (d1) between a first position of the object (OBJ) on the first acquired image (IMG1) and a second position of the object (OBJ) on the second acquired image (IMG2).,

2. Method according to claim 1, wherein said line difference (dl) is related to a relative movement between the object (OBJ) and the camera (3).

3. Method according to one of the preceding claims, in which the corrected distance (X2jCOIT) is determined by: - ​​X2jC oir is the corrected distance, - Xi is the first distance, - X2 is the second distance, - dt is the difference between the start of the first acquisition time and the start of the second acquisition time, - dl is a line difference between a first position of the object on the first acquired image and a second position of the object on the second acquired image, and - X is the reading time per line.

4. Method according to one of the preceding claims further comprising: - (760) determining, from at least the corrected distance (X2jCOIT), a corrected position of the object (OBJ) on the second acquired image (IMG2).

5. Method according to claim 4, in which the corrected position corresponds to an estimate of the line on which the object (OBJ) is located in the second image (IMG2) at the start of the second acquisition time (T2).

6. Method according to one of claims 4 and 5, further comprising: - from the corrected position and the initial position, (770) determining corrected data (OBJ*) relating at least to a relative movement between the object (OBJ) and the camera (3) between the first acquisition time (T1) and the second acquisition time (T2).

7. Device on board an observer vehicle (VO) and connected to a camera (3) configured to acquire images (IMG1, IMG2) of an environment of the observer vehicle (VO), said device including a unit for correcting an object belonging to the environment, said unit being configured to implement the method according to one of claims 1 to 6.

8. Computer program comprising instructions for implementing the method according to one of claims 1 to 6 when this program is executed by a processor (1).

Images