Method and device for enhancing an image of a scene

Abstract

The invention relates to a method (100) for enhancing a current image (IMC) of a scene, captured at a current instant, said method (100) comprising, for at least one target point in said scene appearing in a past image (IMP) of said scene, captured at a past instant in relation to the current instant, a phase (110) for enhancing the current image (IMC), comprising the following steps: - identifying (114) the target point in the current image (IMC); - calculating a speed of the target point according to the positions of the target point in the current (IMC) and past (IMP) images; and - enhancing (124) the current image by adding, in said current image (IMC) and in association with the target point, a speed datum comprising the calculated speed. The invention also relates to a computer program and a device implementing such a method.

Description

DESCRIPTION Title: Method and device for enriching an image of a scene

[0001] The present invention relates to a method for enriching an image of a scene. It also relates to a device, a computer program, an imaging apparatus and a vehicle implementing such a method. It further relates to uses of enriched image(s) obtained by the present invention.

[0002] The field of the invention is the field of enriching digital images, and in particular images of a scene comprising one or more objects. State of the art

[0003] Digital images are used in many fields. In particular, the use of digital images is becoming essential in areas such as object detection, object recognition, and tracking objects in space and / or time. For example, more and more vehicles include one or more cameras to image the environment. These images are then processed within the vehicle to provide driving assistance for said vehicle, which can go as far as partially or fully autonomous driving of said vehicle.

[0004] In this context, it is common for an image to be processed by different modules with a different purpose. For example, in the field of object detection, the same image may undergo a plurality of processing operations, carried out successively or simultaneously, to detect the presence of an object, detect the nature of an object, possibly link this object to a previously detected object, etc. It may also happen that an image is processed in parallel by several modules having the same purpose, for the purpose of redundancy or verification.

[0005] In this context, it may be advantageous to enrich an image of a scene with data that may be useful to several applications, such as those mentioned above as non-limiting examples, or to several processing modules operating simultaneously or in turn, with a view to facilitating, and / or making the processing less consuming in terms of time and computing resources.

[0006] Thus, one aim of the invention is to propose a solution for enriching an image of a scene with one or more data.

[0007] Another aim of the invention is to propose a solution for enriching an image of a scene with data that can be used in various applications. Statement of the invention

[0008] The invention proposes to achieve at least one of the aforementioned aims by a method of enriching an image, called current image, of a scene, captured at an instant, called current instant, said method comprising, for at least one target point of said scene appearing on an image, called past, of said scene captured at an instant past with respect to said current instant, a phase of enriching the current image comprising the following steps: - identification of said target point in the current image; - calculating a speed of said target point as a function of the positions of said target point on said current and past images, and of the time elapsed between said current and past instants; and - enrichment of said current image by said calculated speed, in association with said target point.

[0009] Thus, the invention proposes to enrich an image by adding to said image, speed data associated with a point, and preferably with several points, or even with all the points, of a scene appearing in said image. This speed data is then stored in said image and can be used as many times as desired, or necessary, in various processing operations or by various modules or in the context of various applications. Otherwise In other words, the calculation of the speed data is carried out only once for the image and therefore does not need to be repeated each time this data is used by an application or processing. [OO1O] Thus, the invention makes it possible to propose an enriched image making it possible to reduce the time and computing resources necessary for processing said image, during multiple processing of said image.

[0011] It should be noted that there are currently known techniques for determining speed data associated with an object appearing in an image. These techniques are very time-consuming and resource-intensive because they first require detecting the presence of the object in the image, possibly recognizing it, and then determining a speed associated with said object. Unlike this solution, the invention proposes to enrich an image, and in particular a pixel of the image, with speed data associated with said point of the scene, and therefore with said pixel, appearing in said image, independently of the detection or recognition of an object in said image.

[0012] In the present application, the term "image" means a digital image and in particular a matrix digital image. Each point of the matrix corresponds to a pixel of the image and comprises one or more digital values ​​representing different components of the image for said pixel. For example, in the context of an RGB image, each pixel may comprise three values, each representing one of the RGB components of the digital image. Of course, each pixel may comprise other values ​​representing other quantities, such as brightness, hue, saturation, etc.

[0013] In the present application, a point of a scene corresponds to a pixel of an image of said scene. When the scene is imaged at two different times, the same point of the scene can correspond to two pixels located in different positions on each of the two images.

[0014] For a target point, the current image can be enriched by the speed calculated for said target point, in several ways.

[0015] In one embodiment, the speed may be stored in association with the position of the image pixel corresponding to the target point in a file, matrix, or other container independent of the image.

[0016] According to one embodiment, the speed may be added directly into the image in association with the target point. For example, the speed may be added to the data of the pixel of the image corresponding to the target point. In the non-limiting example of an RGB image of a scene, with three values, {R, G, B} for each pixel of the image representing, each for one of the RGB components, the speed, denoted V, may be added to this data so that the data of the pixel is: {R,G,B,V}.

[0017] According to embodiments, for at least one target point, the speed can be calculated in a reference frame linked to the image sensor used to capture the current image.

[0018] In this case, for said target point, the current image can be enriched with said speed calculated in the frame linked to the sensor.

[0019] In the following, the reference frame associated with the sensor is noted as the UV reference frame. This reference frame is a two-dimensional reference frame defining a plane, and in particular the plane of the image sensor used for image acquisition.

[0020] In this case, the speed of the target point is calculated directly using the positions of the target point in each of the two images expressed in the sensor plane. For example, when calculating the speed of the target point in the image plane, denoted Vuv, then the calculated speed corresponds to a speed vector comprising a first value, denoted Vu, giving the speed in the U direction in the image plane, and a second value, denoted V v , giving the velocity in the V direction in the image plane.

[0021] Alternatively, or in addition, for at least one target point, the velocity can be calculated in a frame linked to the scene.

[0022] In this case, for said target point, the current image can, alternatively or additionally, be enriched with said speed calculated in the reference frame linked to the scene.

[0023] In the following, the frame associated with the scene is noted as the XY frame. This frame is a two-dimensional frame defining a plane corresponding to the image plane in the frame linked to the scene.

[0024] When the target point velocity is calculated in a frame linked to the scene, the target point velocity cannot be calculated directly using the target point positions in each of the current and past frames, expressed in the frame associated with the frame. These positions must first be converted to express them in the frame associated with the scene.

[0025] The speed calculated for the target point can be a speed in the XY image plane of the current image, and denoted VXY. In this case, the speed VXY is a speed vector comprising two values: a first value, denoted Vx, giving the speed in the X direction in the image plane, and a second value, denoted VY, giving the speed in the Y direction in the image plane.

[0026] It should be noted that it is also possible to calculate the speed of the target point in the direction perpendicular to the image plane, and therefore to the XY plane in the frame linked to the scene and to the UV plane in the frame linked to the sensor, from the depths associated with the target point in the current image and in the past image. In this case, the speed calculated for the target point may include a third value giving the speed of said target point in the direction perpendicular to the image plane. For example, in the frame linked to the scene this third value can be noted Vz, and gives the speed of said target point in the Z direction perpendicular to the XY plane. In this case, the speed is a speed vector comprising three values: VXYZ= {VX,VY,VZ}, in the frame linked to the scene.

[0027] In the following, in order to simplify in particular the presentation of the calculations of the transition from the speeds in the plane of the sensor to those in the scene, we assume that the U and V axes of the camera are parallel and oriented in the same direction and we assume that the X, Y axes are parallel to the U and V axes respectively, and oriented in the same direction.

[0028] Of course, this example of implementation of the method is in no way limiting and other relative arrangements between the camera axes and with the axes of the scene reference frame are possible, according to calculations generally involving more coefficients. In the case of angular variation of the scene axes relative to the observer, it may also be useful to take into account the rotation speeds and rotation accelerations to move from the sensor reference frame to that of the scene.

[0029] When the speed of the target point is calculated in a frame of reference linked to the scene, the enrichment phase may further comprise, before the speed calculation step, a step for expressing the positions of the target point in the current image and in the past image, in said frame of reference associated with the scene. Then, the calculation of the speed is carried out as a function of said positions of the target point expressed in said frame of reference associated with the scene.

[0030] The conversion of the position of the target point expressed in the image plane into a reference frame linked to the scene can be carried out using any known technique, provided that the position of the camera that captured the image is known.

[0031] According to a non-limiting exemplary embodiment, the conversion of the position of the target point from the UV reference frame linked to the sensor into an XY reference frame linked to the scene can be carried out in the following manner. By noting (u,v) the coordinates of the target point in the reference frame linked to the sensor, said target point corresponds to 2 angles au and av such that: - according to the X direction: ■ at * f = u ■ at * Z = - x - according to the Y direction: ■ av *f = V ■ av * Z = - y where f is the focal length of the sensor, Z the depth of the target point in the image, and where (x,y) are the coordinates of said target point, the reference frame linked to the scene. Which gives: x = - Z * u / fy = - Z * v / f

[0032] According to embodiments, the enrichment phase may further comprise, for at least one target point, a step of calculating a speed of said target point in the direction perpendicular to the image plane, as a function of: - depths of said target point in the past image and in the current image, - of a duration elapsed between the current and past instants, and - possibly a displacement distance between the capture positions of the current image and the past image.

[0033] In this case, the enrichment phase may further comprise a step of enriching said current image by adding, in said current image and in association with said target point, speed data comprising said calculated speed.

[0034] The speed data in the direction perpendicular to the image plane can be added to the current image individually and independently of the other data. In this case, this speed is a value, possibly positive or negative, indicating the speed of said target point in said direction, noted Z in the frame linked to the scene.

[0035] Alternatively, this velocity data may be added to the velocity data in the image plane, calculated for the target point, if any, for example as described above. In this case, the velocity data may be a three-dimensional velocity vector, as explained above. According to embodiments, the enrichment of the image with the velocity in the image plane and the enrichment of the image with the velocity in the direction perpendicular to the image plane may be carried out within a single enrichment step.

[0036] The depth associated with the target point can be: - measured with a sensor, such as a time-of-flight camera, radar or lidar, etc.; - calculated based on the positions of said target point in the past and current images, for example by triangulation.

[0037] According to embodiments, for at least one target point, the enrichment phase may further comprise the following steps: - calculation of an acceleration of said target point in the plane of the current image as a function of a speed of said target point in each of said current and past images, and of the time elapsed between said current and past instants; and - enrichment of the current image by adding said acceleration in said current image in association with said target point.

[0038] Thus, this acceleration is calculated only once and stored in the image to enrich it so that it does not have to be recalculated later and can be used as many times as desired, or necessary, in various treatments or by various modules or within the framework of various applications.

[0039] When acceleration data is added to the current image in association with a target point, said acceleration data may be added to the data of the image pixel corresponding to said point.

[0040] In the non-limiting example of an RGB image, with three values, {R, G, B}, for each pixel, the acceleration data can be added to this data so that the pixel data is: {R, G, B, A}. If a target point of an image is enriched with a velocity data and an acceleration data, in this case the data representing the pixel corresponding to this target point can be {R, G, B, G, A}.

[0041] Like velocity, the acceleration associated with the target point in the current image can be calculated in the UV frame linked to the image sensor. In this case, the acceleration, denoted Auv, can be a two-dimensional vector comprising two acceleration values: - a first acceleration value, noted Au, giving the acceleration in the U direction, and - a second acceleration value, noted Av, giving the acceleration in the V direction.

[0042] Alternatively, the acceleration associated with the target point in the current frame can be calculated in the XY frame, linked to the scene. In this case, the acceleration, denoted AXY, can be a multi-dimensional vector comprising several acceleration values: - a first acceleration value, noted Ax, giving the acceleration in the X direction, and - a second acceleration value, noted AY, giving the acceleration in the Y direction, and possibly,

[0043] Furthermore, it is possible to obtain the acceleration in the direction perpendicular to the image plane, when the velocity of said target point is known in said perpendicular direction, for the past image and the current image, as described above.

[0044] Thus, according to the invention, for at least one target point, the current image can be enriched: - with speed data expressed in the reference frame linked to the image sensor; and - possibly, with acceleration data expressed in the frame linked to the image sensor.

[0045] Alternatively, or in addition, for at least one target point, the current image can be enriched: - with speed data expressed in the frame linked to the scene; and - possibly, with acceleration data expressed in the frame linked to the scene.

[0046] Whatever the embodiment, the speed data may include: - the speed in the image plane, and - possibly the speed in the direction perpendicular to said image plane

[0047] Whatever the embodiment, the acceleration data may include: - acceleration in the image plane, and - possibly the acceleration in the direction perpendicular to said image plane.

[0048] For a target point, the enrichment phase may comprise a single enrichment step during which the current image is enriched with one or more data chosen from the speed data and the acceleration data described above. Alternatively, the enrichment phase may comprise, for a target point, several enrichment steps, each adding one or more data selected from the velocity data and / or the acceleration data described above, for the target point.

[0049] Identifying a target point from a past image into a current image can be done in several ways.

[0050] According to non-limiting embodiments, the step of identifying the target point of a past image in the current image may comprise a comparison of the target point in the past image to at least one point in the current image, said comparison relating to at least one of the following data associated with said points: - color data, - texture data, - brightness data, - color data, - saturation data, - RGB data. Thus, if the current image includes the target point, said target point is identified in said current image by making a comparison of the pixel representing the target point to one or more pixels in said current image, this comparison being based on any combination of at least one of the data listed above, and more generally on any combination of at least one of the pixel data.

[0051] According to embodiments, the comparison may concern all the points / pixels in the current image, so that the target point of the past image is compared to each point / pixel of the current image.

[0052] Alternatively, the enrichment phase may comprise, prior to the identification step, a step of determining a target area in said current image, the target point being sought in said target area of ​​said image.

[0053] In this case, the target point is compared only to the points of the current image located in said target area, which reduces the number of comparisons and allows the identification of the target point in the current image more quickly and with fewer resources.

[0054] The target area in the current image can be identified in different ways.

[0055] According to particularly advantageous embodiments, the target zone can be identified as a function of speed data associated with the target point in the past image. If the speed data associated with the target point in the past image is available, and in particular the speed Vuv expressed in the frame of reference associated with the image sensor, then it is possible to have the speed in each of the directions in the plane of the image. By knowing the position of the target point in the past image and the time elapsed between the past instant and the current instant, an approximate location of the target point in the current image can be determined.

[0056] The comparison of the target point to the points in the current image may start from said probable location and extend to other points in the current image moving away from said probable position, for example gradually or in discrete steps, for example in one direction or for example in all directions.

[0057] The enrichment phase can be implemented for multiple points, and in particular all points, in the past image.

[0058] Thus, the current image can be enriched with speed data, and possibly with acceleration data, for several points of the scene, and therefore for several pixels of the current image.

[0059] According to embodiments, the method according to the invention may comprise a step of pre-processing the past image, to select the target point(s). In this case, the enrichment phase is implemented for only part of the points of the scene appearing on the past image, namely the points selected during the pre-processing step.

[0060] The point(s) of the scene in the image can be selected according to any logic, or relationship.

[0061] According to embodiments, the pre-processing step may comprise the following steps: - extraction, at each point of the past image, of a spatial gradient, in particular by derivation of said past image, and even more particularly by Sobel filtering; - calculation of a norm of the gradient of each point, in particular the Euclidean norm; - selection, as target point(s), the point(s) whose gradient norm is greater than a predefined threshold. Thus, the points of the past image for which the enrichment phase is carried out are those of interest according to the norm of the spatial gradient. These points generally constitute the limits of the different objects of the scene and provide good information on the evolution of the scene. Thus, the current image is enriched for only a part of the points of the scene of interest, which allows an optimized enrichment of said current image.

[0062] The method according to the invention can be implemented to enrich the images of a stream of images of a scene, said stream of images being provided by the same camera or several cameras whose relative positions are known.

[0063] The image stream may be a video provided by a single camera. In this case, the method according to the invention makes it possible to enrich the images of said video.

[0064] A current image within the image stream can be enriched based on a past image that immediately precedes it in the image stream.

[0065] Alternatively, the past image may not be the one immediately preceding the current image so that it may be separated from the current image by one or more other images.

[0066] Additionally, each image in the image stream can be enriched.

[0067] Alternatively, only one image every N images can be enriched in the image stream, with N>2.

[0068] For at least one current image, the enrichment phase of said current image can be carried out immediately after the acquisition of said current image.

[0069] For at least one current image, the enrichment phase of said current image may be offset in time, relative to the time of acquisition of said current image.

[0070] According to another aspect of the invention, an enriched image of a scene is provided, stored on a storage means, obtained by the method according to the invention.

[0071] The enriched image according to the invention is a digital image, preferably a matrix image.

[0072] The image enriched according to the invention can be captured by any image acquisition means.

[0073] The storage medium can be of any type, portable or not, such as a memory card, a USB key, a computer, a server, a telephone, a camera, etc.

[0074] According to another aspect of the invention, there is proposed a use of enriched images of a scene according to the invention, for the identification of a target point of said scene located in a first enriched image, in a second enriched image of said scene, said identification comprising a comparison of a speed data item associated with said target point in the first image with a speed data item of at least one point in said second enriched image.

[0075] According to embodiments, the identification may take into account the acceleration data, if applicable. In particular, the identification may comprise a comparison of an acceleration data associated with said target point in the first image to an acceleration data of at least one point in said enriched second image.

[0076] Enriched images can be taken: - by the same camera or by two different cameras, - at the same time or at different times.

[0077] Indeed, in an enriched image according to the invention one or more target points comprise speed data, and possibly acceleration data, associated with said target point(s). Thus, this speed data, and possibly this acceleration data, can be used to identify this target point in another enriched image because the speed data, respectively the acceleration data, is additional information which can be used when identifying the target point.

[0078] According to another aspect of the invention, a use of enriched image(s) according to the invention is proposed for at least one of the following applications: - for the detection of an object in a scene at a given moment, - for tracking an object in a scene over time, - for assistance in driving, or for autonomous or semi-autonomous driving of a vehicle.

[0079] Indeed, at least one speed data item, and possibly at least one acceleration data item, associated with a target point can be used to detect an object in this image because it is likely that the points having the same speed, and possibly the same acceleration, belong to the same object. It is therefore possible to discriminate and / or detect the objects on an image enriched according to the invention as a function of the speed data, and possibly the acceleration data.

[0080] Furthermore, at least one velocity data item, and possibly at least one acceleration data item, associated with a target point can be used to find the position of an object on another captured image. This makes it possible to track an object in time and space, whether or not it is visible in said other image.

[0081] In addition, at least one speed data item, and possibly at least one acceleration data item, associated with a target point can be used to estimate, or predict, the probable position of an object in the near future, and in particular at a following instant. This makes it possible to improve the tracking of an object in time and space, and / or to provide assistance in piloting a vehicle or a robot.

[0082] According to another aspect of the invention, there is provided a computer program comprising executable instructions which, when executed by a computing device, implement all the steps of the method according to the invention.

[0083] The computer program can be in any computer language, such as machine language, C, C++, JAVA, Python, etc.

[0084] According to another aspect of the invention, a device is proposed comprising means configured to implement all the steps of the method according to the invention.

[0085] The device may be a processor, a calculator or a graphics card, a computer, a server, and more generally a computing unit configured to implement the method according to the invention.

[0086] For example, the device may be provided with the computer program according to the invention.

[0087] Imaging device comprising: - at least one camera module, and - at least one processing means, in particular a processor or a graphics card, configured to implement all the steps of the method according to the invention to enrich an image captured by said at least one camera module.

[0088] According to non-limiting embodiments, the imaging device according to the invention can be a camera, a Smartphone, a tablet, a virtual reality headset, augmented reality headset, endoscope-type medical imaging device, etc.

[0089] According to another aspect of the present invention, there is provided a vehicle comprising: - at least one camera module, and - at least one processing means, in particular a processor or a graphics card, configured to implement all the steps of the method according to the invention to enrich an image captured by said at least one camera module.

[0090] According to embodiments, the vehicle may be a land vehicle, such as a car, autonomous or not.

[0091] According to embodiments, the vehicle may be a flying vehicle, such as a drone, an airplane, a helicopter, autonomous or not.

[0092] According to embodiments, the vehicle may be a maritime vehicle, such as a boat or a submarine, autonomous or not. Description of figures and embodiments

[0093] Other advantages and characteristics will appear on examining the detailed description of non-limiting embodiments, and the attached drawings in which: - FIGURE 1 is a schematic representation of a non-limiting exemplary embodiment of a method according to the invention for enriching an image; - FIGURE 2 is a schematic representation of a non-limiting exemplary embodiment of a method according to the invention for enriching an image stream; - FIGURE 3 is a schematic representation of a non-limiting exemplary embodiment of a device according to the invention for enriching an image or a stream of images; - FIGURE 4 is a schematic representation of a non-limiting exemplary embodiment of an imaging device according to the invention; and - FIGURE 5 is a schematic representation of a non-limiting exemplary embodiment of a vehicle according to the invention.

[0094] It is understood that the embodiments which will be described below are in no way limiting. In particular, it is possible to imagine variants of the invention comprising only a selection of characteristics described below isolated from the other characteristics described, if this selection of characteristics is sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art. This selection includes at least one preferably functional characteristic without structural details, or with only part of the structural details if it is this part which is only sufficient to confer a technical advantage or to differentiate the invention compared to the state of the prior art.

[0095] In particular, all the variants and embodiments described can be combined with each other if there is no technical obstacle to this combination.

[0096] In the figures and in the rest of the description, the elements common to several figures retain the same reference.

[0097] FIGURE 1 is a schematic representation of a non-limiting exemplary embodiment of a method according to the present invention.

[0098] The method 100 of FIGURE 1 can be used to enrich a digital image, called the current image, and noted IMC, of ​​a scene, captured at a current time, by using another digital image of said scene, called the past image, noted IMP, captured at a past time preceding the current time.

[0099] The current and past images can be captured by the same camera. The camera position can be fixed. Alternatively, the camera can be mobile so that its position changes between the past time at which the past image was captured and the current time at which the current image was captured. These positions are known and are referred to as past position and current position in the following.

[0100] Alternatively, the current and past images can be captured by two cameras, referred to as the past camera and the current camera in the following. The relative positions of the cameras are known.

[0101] The method 100 of FIGURE 1 includes an optional step 102 of pre-processing the past image IMP to select the points of interest. The points not of interest will not be considered in the remainder of the method and will not be selected as target points.

[0102] Points of no interest are, for example, points corresponding to a part of the sky, points in the center of a monotonous and non-changing surface, etc. Points of interest are, for example, points on the outline of an object in the scene, such as a person, or on a part of an object, such as a person's arm, etc.

[0103] The point(s) of the scene in the image can be selected according to any logic, or relationship.

[0104] According to embodiments, the pre-processing step 102 may comprise the following steps: - a step 104 of extracting at each point of said image a spatial gradient by Sobel filtering; - a step 106 of calculating a Euclidean norm of the gradient of each point; and - a step 108 of selection, as target point(s), the point(s) whose gradient norm is greater than a predefined selection threshold.

[0105] The spatial gradient can be calculated on any of the quantities stored for each pixel. Alternatively, the spatial gradient can be calculated on a quantity itself deduced from at least one of the quantities stored for each pixel. For example, the spatial gradient can be calculated on the brightness, the latter being able to correspond to a weighted sum of the R, G and B components for each pixel, in the context of an RGB image.

[0106] The selection threshold can be determined by trial. Alternatively, the selection threshold can be calculated based on Euclidean norms calculated for all points of the past image. For example, the selection threshold may correspond to an average of the Euclidean norms calculated in step 106.

[0107] Thus, the points of the past image for which the enrichment phase will be carried out are those of interest according to the norm of the spatial gradient. As indicated above, these points generally constitute the limits of the different objects of the scene and provide good information on the evolution of the scene. Thus, the current image will be enriched for only a part of the points of the scene of interest, which allows an optimized enrichment of said current image, in terms of time and computing resources. In addition, the enriched image will have an optimized size.

[0108] The method 100 comprises a phase 110 of enriching the current image.

[0109] This enrichment phase is carried out for a point of the scene located in the past image, called the target point. The target point is chosen from the points of the scene located in the past image selected in step 102, or alternatively from all the points of the past image when step 102 is not carried out.

[0110] When several points of the past image are concerned, then the enrichment phase 110 is carried out individually for each of these points, in turn or simultaneously. Preferably, the enrichment phase is carried out for all the points selected during step 102. [YES] The enrichment phase 110 comprises an optional step 112, carried out when the speed in the image plane is known for the target point in the past image. This speed may be known when the past image is itself an enriched image within the meaning of the present invention with a prior image captured before the past image. Alternatively, the speed in the image plane may be calculated by another technique or measured by a sensor.

[0112] Optional step 112 determines, in the current image, a target area in which the target point may be located. To do this, a probable position of the target point in the current image is calculated based on: - the position of the target point in the past image, - the speed of the target point in the plane of the past image, and - the time elapsed between the current time and the past time. Optionally, if the current and past positions (from which the current and past images were acquired) are not the same, they are taken into account in calculating the probable position of the target point. Then, an area is defined around the probable position. This area can be of a predetermined size, for example.

[0113] Preferably, the speed used during step 112 is the speed of the target point, in the image plane, expressed in the sensor frame of reference.

[0114] In a step 114, the pixel that corresponds to the target point in the past image is compared to one or more pixels in the current image. The comparison of two pixels involves a comparison of one or more of the following respective data of these pixels: - color, - tint, - saturation, - texture, - brightness, - etc.

[0115] When step 112 is performed, the comparison may begin with the pixel corresponding to the probable position, or with the pixels located in the target area, identified in the current image during step 112. If a pixel in the current image corresponding to the pixel of the target position in the past image is identified, then the comparison stops. Otherwise, the target area may be enlarged in steps, or progressively, until it potentially extends to the entire current image as long as the target point has not been identified in the current image.

[0116] When step 112 is not performed, the comparison can begin, from the current image, with the pixel of the same position as the pixel from the target point in the past frame, to any other position in the current frame.

[0117] If step 114 does not find the target point in the current image, then the enrichment phase for this target point is stopped. Indeed, it is possible that the target point of the scene appearing in the past image does not appear in the current image, for example because this point has left the camera's field of view. The enrichment phase can be executed for another target point of the past image.

[0118] In the case where step 114 has identified the target point in the current image, the position of said target point in the current image is stored. This position is determined in the UV frame of the sensor and is therefore expressed in said UV frame of the sensor.

[0119] At this stage, the position of the target point, noted PosP = {u P ,v P}, is known in the past image expressed in the frame linked to the sensor, and the position of the target point, noted PosC={u c ,v c}, is known in the current image in the current image frame.

[0120] The enrichment phase 110 may comprise an optional step 116, during which the positions of the target point in the past and current images are converted and expressed in a frame of reference linked to the scene, and no longer in the frame of reference linked to the sensor. The frame of reference linked to the scene is noted XY, as indicated above.

[0121] This conversion requires knowing the depth associated with the target point, in the frame linked to the scene. By noting (u,v) the coordinates of a point in the frame linked to the sensor, said point corresponds to 2 angles au and av such that: - according to the X direction: ■ at * f = u ■ at * z = - x - according to the Y direction: ■ av *f = V ■ av * z = - y where f is the focal length of the sensor, z is the depth of the point in the image, and where (x,y) are the coordinates of said target point, the reference frame linked to the scene. This gives: x = - Z * u / fy = - Z * v / f

[0122] Thus, using the above relationships, it is possible to express each of the positions of the point in the frame linked to the scene so that: PosP = {x P ,y P}, And PosC = {xc,y c}, And

[0123] The depth z of the target point can be determined by any means: for example measured by a sensor, or determined by calculation.

[0124] Thus, the optional step 116 provides the position of the target point, noted p 0S p scene = {Xp,Y P} in the past image expressed the scene reference and the position of the target point, noted PosC scene ={X c ,Y c}, in the current image expressed the scene reference. These positions can be memorized.

[0125] The enrichment phase 110 comprises a step 118 during which the speed of the target point in the plane of the current image is calculated as a function of: - the position of said target point in the current image, - the position of said target point in the past image, and - of the duration D elapsed between the past and current instants; for example using the relation: VitC=(PosC-PosP) / D

[0126] In this step 118, it is possible to calculate the speed of the target point expressed in the UV frame linked to the sensor. To do this, the speed is calculated using the positions PosC and PosP of the target point expressed in the UV frame linked to the sensor, as obtained in step 114. The speed thus calculated is a speed vector, which can be noted VitCuv={VitCu,VitCv}, comprising two values: a first value VitCu giving the speed of the target point in the U direction and a second value VitCv giving the speed of the target point in the V direction.

[0127] Alternatively or in addition, during this step 118, it is possible to calculate the speed of the target point expressed in the XY reference frame linked to the scene. To do this, the speed is calculated using the positions PosC and PosP of the target point expressed in said XY frame linked to the scene, as obtained in step 116. The speed thus calculated is a speed vector, which can be noted VitCxY={VitCx,VitCv}, comprising two values: a first value VitCx giving the speed of the target point in the X direction and a second value VitCy giving the speed of the target point in the V direction.

[0128] Thus, step 118 calculates and provides: - either the speed of the target point, in the image plane, expressed in the frame linked to the scene, - either the speed of the target point, in the image plane, expressed in the frame linked to the sensor, - or both.

[0129] The, or at least one of these, speeds can be added to the data of the pixel corresponding to the target point in the current image, to enrich the current image.

[0130] The enrichment phase 110 may comprise an optional step 120 of calculating a speed, denoted VitCz, in the direction perpendicular to the image plane. Preferably, and without any loss of generality, this speed may be expressed in the frame of reference linked to the scene.

[0131] The speed VitCz can be calculated based on: - a depth, noted ProfC, of ​​the target point in the current image, - a depth, noted ProfP, of the target point in the past image, and - of the duration D elapsed between the past instant and the current instant; for example using the relation: VitCz=(ProfC-ProfP) / D

[0132] For each of the current and past images, the depth of the target point corresponds to the distance between the camera and the target point. As mentioned above, this depth can be measured by a sensor, such as a radar or a lidar or a time-of-flight camera. Alternatively, the depth can be provided by the camera, especially when the camera is a 3D camera. In yet another alternative, the depth can be calculated by known triangulation techniques based on the positions of the target point in the current image IMC and in the past image IMP. This value of velocity VitCz can be added to the data of the pixel corresponding to the target point in the current IMC image, to enrich the current IMC image. Alternatively, the velocity VitCz can be used in association with the velocity vector VitCxv or VitCuv to obtain a three-dimensional velocity vector. This three-dimensional velocity vector can be added to the data of the pixel corresponding to the target point in the current image, to enrich the current image.

[0133] Optionally, the enrichment phase 110 may comprise a step 122 during which an acceleration of the target point may be calculated as a function of: - the speed of the target point in the current image, noted Vite, - the speed of said target point in the past image, noted VitP, and - the duration D elapsed between the past and current instants; for example using the relation: AccC=(VitC-VitP) / D

[0134] The speed Vite of the target point in the current image has been determined previously (during step 118, and possibly step 120, of the enrichment phase 110). The speed VitP of the target point in the past image may be provided in the past image. In particular if the past image is an image enriched according to the invention, the speed VitP may have been stored in the past image and be found in the data of the pixel corresponding to the target point in the past image.

[0135] This step 122 can calculate the acceleration in the frame linked to the image sensor using the speeds expressed in said frame linked to the sensor. Alternatively, or in addition, step 122 can calculate the acceleration in the frame linked to the scene using the speeds expressed in said frame linked to the scene.

[0136] Whatever the reference frame, when the current and past velocities are velocities in the image plane, XY plane or UV plane, then the calculated acceleration will be the acceleration in the image plane. When the current and past velocities are velocities in the direction perpendicular to the image plane, then the calculated acceleration will be the acceleration in said direction. According to yet another alternative, the current and past can be three-dimensional velocity vectors giving the velocity of the target plane in each of the three directions: in this case the calculated acceleration AccC gives the acceleration of the target point in each of these three directions.

[0137] Thus, step 122 provides one or more of the acceleration data indicated above.

[0138] The enrichment phase 110 comprises a step 124 of enriching the current IMC image with: - at least one of the speed data calculated in step 118, and possibly in step 120; and - optionally, with at least one of the acceleration data calculated in step 122.

[0139] Enrichment of the current image can be achieved by writing this or these data directly into the image, in particular with the data of the pixel corresponding to the target point. Alternatively, enrichment of the current image can be achieved by writing this or these data into a separate matrix.

[0140] In particular, for at least target point, the current image can be enriched with speed data giving the speed of the target point in the image plane, expressed in the frame linked to the image sensor.

[0141] Alternatively, or in addition, for at least one target point, the current image can be enriched with speed data giving the speed of the target point in the image plane, expressed in the frame linked to the scene.

[0142] Optionally, the current image can be enriched with speed data giving the speed of the target point in the depth direction, for example in the direction noted Z in the reference frame linked to the scene.

[0143] Optionally, the current image can be enriched with acceleration data, giving the acceleration of the target point, in the image plane, expressed in the frame of the image sensor.

[0144] Optionally, the current image can be enriched with acceleration data, giving the acceleration of the target point, in the image plane, expressed in the frame linked to the scene.

[0145] Optionally, the current image can be enriched with acceleration data giving the acceleration of the target point, in the depth direction, for example in the direction noted Z in the frame linked to the scene.

[0146] The scene-related marker can be any type of marker.

[0147] For example, it is possible to work in a frame of reference associated with the observer. This relative case allows us to know the relative speed of the elements around the observer, and also in particular the observer's own speed in relation to the scene.

[0148] Alternatively, it is possible to work in a frame of reference associated with a fixed point in the scene. In this case, called the absolute case, calculating the various data requires knowing the observer's position, and in particular their speed. This absolute case allows us to know the speed of the objects in the scene, in the scene's frame of reference. This makes it possible to calculate their trajectory in this frame of reference directly, for subsequent decision-making layers and data use.

[0149] Enriching the image with data expressed in a frame of reference linked to the observer can be useful in various applications, in particular in object tracking applications for example. Enriching the image with data expressed in a frame of reference linked to the scene can be useful in these same applications, or other applications, in particular in applications for trajectory calculation, or vehicle control, driving assistance, etc.

[0150] The enrichment phase 110 ends, for the target point considered, and can be repeated to process another target point and enrich the current IMC image for said other target point. And so on.

[0151] At least two target points may be treated in turn. Alternatively, and preferably, at least two target points may be treated simultaneously.

[0152] When all target points have been processed, an enriched current image, noted IMC*, is provided.

[0153] This enriched current image IMC* can then be used, as a past image, to enrich a new current image acquired at a time later than the current time. And so on.

[0154] The method 100 of FIGURE 1, and generally the method according to the invention, can provide: - an image enriched with data expressed in the frame linked to the sensor, and / or - an image enriched with data expressed in the reference frame linked to the scene; and / or - an image enriched with data expressed in the frame linked to the sensor and data expressed in the frame linked to the scene.

[0155] FIGURE 2 is a schematic representation of a non-limiting exemplary embodiment of a method for enriching an image stream according to the present invention.

[0156] The method 200 of FIGURE 2 may be used to enrich an image stream 202 from one or more cameras. In particular, and without loss of generality, the image stream 202 is video from a single camera.

[0157] The image stream 202 is formed from a plurality of images 202i-202 m acquired one after the other, over time, and at a given frequency, for example at a frequency of 24 images per second.

[0158] Image 202i is the first image of stream 202 and is not enriched by the method according to the invention. Image 202i is however used, as a past image, to enrich image 2022, considered as the current image. Image 202i is the image which immediately precedes image 2022, in the stream 202. The processing of the image 2022 by the method according to the invention provides an enriched image noted 2022*.

[0159] Then, the image 202s is processed by the method according to the invention, as a current image. The past image used to enrich the current image 202s can be the image 2022, or preferably the enriched image 2022*. The processing of the image 202s provides an enriched image 202s*. And so on to obtain an enriched image stream 202* formed by the images 202i, 2022*- 202m*.

[0160] In the example shown in FIGURE 2, each image of the image stream 202 is processed, except for the first image 202i.

[0161] Alternatively, only a portion of the images in the image stream 202 may be processed. For example, without loss of generality, the image stream 202 may be enriched every N images, with N>2, or at yet another frequency, constant or variable.

[0162] FIGURE 3 is a schematic representation of a non-limiting exemplary embodiment of an image(s) enhancement device according to the present invention.

[0163] The device 300 of FIGURE 3 can be used to enrich an image, respectively a stream of images.

[0164] The device 300 is configured to implement the method according to the invention, and in particular the method 100 of FIGURE 1 for enriching an image or the method 200 of FIGURE 2.

[0165] The device 300 optionally comprises a module 302 for pre-processing the past image, in order to select the target points which are of interest. In particular, the pre-processing module 302 is configured to implement step 102 of the method 100 of FIGURE 1.

[0166] The device 300 comprises a module 304 for enriching the current IMC image. The enrichment module 304 can be configured to implement the enrichment phase of the method according to the invention and in particular phase 110 of the method 100 of FIGURE 1.

[0167] The module 304 may optionally comprise a module 306 for determining a target area in the current IMC image. In particular, the module 304 may be configured to implement step 112 of the method 100 of FIGURE 1.

[0168] The enrichment module 304 comprises a module 308 for identifying the target point in the current IMC image. In particular, the module 308 can be configured to implement step 114 of the method 100 of FIGURE 1.

[0169] The module 304 may optionally comprise a module 310 for converting coordinates of a target point to express it in a reference frame associated with the scene. In particular, the module 310 may be configured to implement step 116 of the method 100 of FIGURE 1.

[0170] The enrichment module 304 comprises a module 312 for calculating the speed of the target point in the current IMC image. In particular, the module 312 can be configured to implement step 118, and where appropriate the optional step 120, of the method 100 of FIGURE 1.

[0171] The enrichment module 304 may optionally comprise a module 314 for calculating the acceleration of the target point in the current IMC image. In particular, the module 314 may be configured to implement step 122 of the method 100 of FIGURE 1.

[0172] The enrichment module 304 comprises a module 316 for adding, in the current IMC image, at least one speed data item and possibly at least one acceleration data item. In particular, the module 316 can be configured to implement step 124 of the method 100 of FIGURE 1.

[0173] At least one of the modules 302-316 may be an individual module, independent of the other modules. Alternatively, at least two of the modules 302-316 may be integrated into a single module. In particular, all the modules of the device 300 may be integrated into a single module.

[0174] At least one of the modules 302-316 may be a hardware module, such as a processor, a microchip, a graphics card, a calculator, a computer, a server, etc.

[0175] Alternatively, at least one of the modules 302-316 may be a software module executed by at least one computer means.

[0176] According to yet another alternative at least one of the modules 302-316 may be a combination of at least one hardware module with at least one software module.

[0177] The configuration of one, or each, of the modules 302-316 may be a software configuration, such as an application, or a computer program, loaded into said module, and / or a hardware configuration such as a specific hardware architecture.

[0178] FIGURE 4 is a schematic representation of a non-limiting exemplary embodiment of an imaging apparatus according to the present invention.

[0179] The apparatus 400 of FIGURE 4 may be used to capture and enhance an image, or a stream of images, or even a video.

[0180] The apparatus 400 comprises a camera module 402 comprising at least one lens and at least one image sensor, for example a CCD or CMOS sensor. The camera module 402 makes it possible to capture images or a video formed by a stream of images.

[0181] The apparatus 400 further comprises a processing unit 404, which may for example be the device 300 of FIGURE 3, to enrich one or more images captured by the camera module 402.

[0182] The camera module 402 and the processing unit 404 may be integrated within a single unit. Alternatively, the camera module 402 and the processing unit 404 may not be integrated into a single unit and may be independent of each other. In this case, the camera module and the processing unit are in communication, direct or indirect, wired or wireless.

[0183] FIGURE 5 is a schematic representation of a non-limiting exemplary embodiment of a vehicle according to the present invention.

[0184] The vehicle 500 of FIGURE 5 may be any type of land vehicle, such as, for example, a car, a truck, etc.

[0185] The 500 vehicle may or may not be an autonomous vehicle.

[0186] The vehicle 500 is equipped with an apparatus according to the invention, such as for example the apparatus 400 of FIGURE 4. In general, the vehicle 500 can be equipped with: - at least one camera module 402, and - at least one processing unit 404.

[0187] Of course, the camera module 402 and the processing unit 404 can be arranged at any suitable location in the vehicle. In the example shown, the camera module 402 is arranged in the front part of the vehicle and makes it possible to image the scene in front of the vehicle.

[0188] The processing unit 404 may be an individual and independent unit. Alternatively, the processing unit 404 may be integrated into a computer or a management unit (not shown) of the vehicle 500.

[0189] The vehicle 500 may comprise a driving assistance unit 502, receiving the enriched images from the processing unit 404, and providing driving instructions either to the driver of the vehicle, or directly to components of the vehicle, from said enriched images.

[0190] Of course, the invention is not limited to the examples which have just been described.

Claims

CLAIMS 1. Method (100) for enriching an image (IMC), called current image, of a scene, captured at an instant, called current instant, said method (100) comprising, for at least one target point of said scene appearing on an image (IMP), called past, of said scene captured at an instant past relative to said current instant, a phase (110) of enriching the current image (IMC) comprising the following steps: - identification (114) of said target point in the current image (IMC); - calculation of a speed of said target point, in the plane of the current image, as a function of the positions of said target point on said current (IMC) and past (IMP) images, and of the time elapsed between said current and past instants; and - enrichment (124) of said current image by adding, in said current image (IMC) and in association with said target point, speed data comprising said calculated speed.

2. Method (100) according to the preceding claim, characterized in that, for at least one target point, the speed is calculated in a reference frame linked to the image sensor used to capture the current image, the current image being enriched with said speed.

3. Method (100) according to any one of the preceding claims, characterized in that, for at least one target point, the speed is calculated in a reference frame linked to the scene, the current image being enriched with said speed.

4. Method according to the preceding claim, characterized in that the enrichment phase further comprises, before the speed calculation step, a step (116) for expressing the positions of the target point in the current image (IMC) and in the past image (IMP), in the reference frame associated with the scene, the calculation of the speed expressed in the reference frame linked to the scene being carried out as a function of said positions in said reference frame associated with the scene.

5. Method (100) according to any one of the preceding claims, characterized in that the speed comprises the speed in the image plane.

6. Method (100) according to any one of the preceding claims, characterized in that the speed comprises the speed in a direction perpendicular to the image plane.

7. Method (100) according to the preceding claim, characterized in that, for at least one target point, the enrichment phase (110) further comprises the following steps: - calculation (122) of an acceleration of said target point as a function of a speed of said target point in each of said current (IMC) and past (IMP) images, and of the time elapsed between said current and past instants; and - enrichment (124) of the current image (IMC) by adding said acceleration in said current image (IMC) in association with said target point.

8. Method (100) according to any one of the preceding claims, characterized in that the step of identifying (114) the target point in the current image (IMC) comprises a comparison of the target point in the past image with at least one point in the current image (IMC), said comparison relating to at least one of the following data associated with said points: - color data, - texture data, - brightness data, - color data, - saturation data, - RGB data.

9. Method (100) according to any one of the preceding claims, characterized in that the enrichment phase (110) comprises, prior to the identification step (114) comparison, a step (112) determining a target area in said current image, the target point being sought in said target area of ​​said current image.

10. Method (100) according to any one of the preceding claims, characterized in that the enrichment phase (110) is implemented for several points, and in particular all the points, in the past image.

11. Method (100) according to any one of the preceding claims, characterized in that it comprises a step (102) of pre-processing the past image, to select the target point(s), said pre-processing step (102) comprising the following steps: - extraction (104), at each point of said past image (IMP), of a spatial gradient, in particular by derivation of said past image (IMP), and even more particularly by Sobel filtering; - calculation (106) of a norm of the gradient of each point, in particular the Euclidean norm; - selection (108), as target point(s), the point(s) whose gradient norm is greater than a predefined threshold.

12. Method (100) according to any one of the preceding claims, characterized in that it is implemented to enrich the images of an image stream (202) of a scene, said image stream being provided by the same camera or several cameras whose relative positions are known.

13. Enriched image (IMC*) of a scene, stored on a storage means, obtained by the method (100) according to any one of the preceding claims.

14. Use of enriched images of a scene according to the preceding claim, for the identification of a target point of said scene located in a first enriched image of said scene, in a second enriched image of said scene, said identification comprising a comparison of a speed data item associated with said target point in the first image to speed data of at least one point in said second enriched image.

15. Use of enriched image(s) of a scene according to claim 13 for at least one of the following applications: - for the detection of an object in a scene at a given moment, - for tracking an object in a scene over time, - for assistance in driving, or for autonomous or semi-autonomous driving of a vehicle.

16. A computer program comprising executable instructions which, when executed by a computing device, implement all the steps of the method (100) according to any one of claims 1 to 12.

17. Device (300) comprising means configured to implement all the steps of the method according to any one of claims 1 to 12.

18. Imaging apparatus (400) comprising: - at least one camera module (402), and - at least one processing means (300; 404), in particular a processor, configured to implement all the steps of the method (100) according to any one of claims 1 to 12 to enrich an image captured by said at least one camera module (402).

19. Vehicle (500) comprising: - at least one camera module (402), and - at least one processing means (300; 404), in particular a processor or a graphics card, configured to implement all the steps of the method according to the invention to enrich an image captured by said at least one camera module.

Images