Method for calculating information related to relative speed between object and camera, control method for vehicle, computer program, computer-readable recording medium, object motion analysis system, and control system

Abstract

Computer-implemented method for computing information related to the relative velocity between an object (200) and a camera (30) based on two images (I k ,I k+1 ) of the object acquired by the camera. The method comprises: S41) determining a value of the optical flow (F) between the two images (I k ,I k+1 ); and, together with or after determining the value of the optical flow, using the optical flow to determine at least one parameter (s, tx, ty) of a transformation; and S42) computing information (RVD, TtC) related to the relative velocity between the object and the camera based on said at least one parameter (s, tx, ty) of the transformation. Also disclosed are a control method for a vehicle and an object motion analysis system based on the above method, as well as a computer program and a computer-readable recording medium implementing the above method.

Description

TECHNICAL FIELD

[0001] The present disclosure mainly relates to a method for estimating the speed of an object detected in a sequence of images output by one or more cameras. By extension, it also relates to a vehicle control using the relative speed information estimated by this method. BACKGROUND

[0002] When a vehicle is moving, it is essential to safely drive the vehicle to detect independently moving objects moving around the vehicle. It is of course also very important to estimate the relative speed of these objects with respect to the vehicle.

[0003] This relative speed can generally be determined by radars installed on the vehicle.

[0004] However, in the case where the moving objects are not in the acquisition area of the radars of the vehicle, or in the case where one of these radars is faulty, it can be necessary to use an alternative method to obtain information on the relative speed between the vehicle and the moving objects moving around the vehicle.

[0005] There is therefore a need for a method and a device capable of providing information on the relative speed between an object and a vehicle, in particular when this information cannot be provided by the radars of the vehicle. SUMMARY

[0006] The present invention was conceived in view of the above problems of the prior art, and therefore the first object of the present disclosure is to propose a method for calculating information related to the relative speed between an object and an "observer" which can be for example a vehicle, but the observer can more generally be any static observer of the scene, which method does not rely on radars.

[0007] According to the present disclosure, a computer-implemented method for calculating information related to the relative speed between an object and a camera is proposed.

[0008] This method is based on an initial image I k and a final image I k+1 of the object derived from image frames output by the camera, and comprises the following steps:

[0009] S41) determining a value of an optical flow between the initial image and the final image, the optical flow defining for a pixel of the initial image its estimated motion during a transformation of the initial image into the final image; and,

[0010] using the optical flow, together with or after determining the value of the optical flow, to determine at least one parameter of the transformation; and

[0011] S42) calculating information related to the relative speed between the object and the camera based on the at least one parameter of the transformation.

[0012] The information related to the relative speed between the vehicle and the object can be, for example, a value indicating whether the detected object is moving away or moving towards the vehicle; it can be a relative velocity in depth of the detected object with respect to the vehicle; a time to collision with the detected vehicle; or a relative acceleration (in depth) of the detected object with respect to the vehicle; etc.

[0013] In the method according to the disclosure, the information related to the relative speed between the object and the camera can be computed not only based on a single value of the at least one parameter of the transformation, but also based on a plurality of values of the at least one parameter of the transformation. In this case, the plurality of values can be computed by performing step S41 a plurality of times to obtain a plurality of estimates of the parameter(s) of the transformation (step S41 is then performed on a plurality of pairs of images, each pair of images comprising an initial image and a final image); and the information related to the relative speed is computed based on said plurality of estimates of the parameter(s) of the transformation.

[0014] Advantageously, the method only uses very cheap and common sensors: cameras.

[0015] The method only uses a pair of images based on a pair of frames output by the camera to compute the information related to the relative speed between the object and the camera. This pair of frames from which the initial image and the final image are derived typically consists of two frames acquired by the camera at two different points in time (here k refers to the number of the frame in the sequence of images output by the camera).

[0016] Importantly, in the method of the disclosure, the initial image can precede or follow in time the final image representing the object: the order of the two images (with respect to time) is not important.

[0017] The method does not use the entire frames output by the camera, but only the parts of these frames representing the object for which the relative speed is to be estimated.

[0018] Therefore, a preliminary step consists in extracting from the pair of frames two images representing the object in the two frames (“sub-images” or “patches”). These two sub-images or patches are the initial image and the final image on which the method relies.

[0019] According to the disclosure, it has been observed that the information related to the relative speed between the object and the camera can be computed using optical flow in (at least) two steps. As it is well known in the art, the optical flow comprises the coordinates of the motion vector (u, v) defining the motion of a pixel (x, y) in the transformation of the initial image into the final image. Therefore, the optical flow is defined by the function F in the following optical flow equation:

[0020] (u, v) = F(x, y)

[0021] An object of interest (e.g. a vehicle) is detected and its bounding box is determined in each of the consecutive frames using an object detection algorithm. Several such detection algorithms are well known, e.g. SSD, Yolo, AdaBoost, etc. The sub-image identified in this way and representing the object is cropped from the frame using the bounding box that has been determined. Furthermore, recent advances in 3D object detection, e.g. MonoDis, can be used as a previous step of this method. Also, instance segmentation methods such as Mask-RCNN can be used to make a distinction between background and foreground.

[0022] It is assumed that the observed object is rigid and, therefore, the projection view of it in the image is a constrained transformation. In the more general or extreme case, this can be represented by a perspective or affine transformation, but, especially for objects viewed at large distances, the transformation is preferably represented by a similarity transformation that involves only scaling (expansion) and translation (tx, ty).

[0023] Under these assumptions, the apparent motion (optical flow) (u, v) from the previous frame to the current frame of a pixel (x, y) representing the object can be expressed as:

[0024] (x, y)→(x + u, y + v) = s · (x, y) + (tx, ty), or

[0025]

[0026] Note that the scaling and translation in (1) are considered to be the same for all points of the patch. The value of the scaling s is usually close to 1 and the case s > 1 corresponds to an expansion (approaching) of the object, while the case 0 < s < 1 corresponds to a contraction (moving away) of the object.

[0027] Therefore, in the differential optical flow equation, (u, v) can be replaced by a function of s, tx, and ty.

[0028] This leads to different differential optical flow equations; in the case of the Horn-Schunck algorithm, this differential optical flow equation can be expressed as:

[0029]

[0030] In some embodiments, the present disclosure proposes a method of estimating this equation for each point (x, y) within the patch, which provides an over-determined system of equations for the three variables s, tx, and ty.

[0031] By solving the above equations for the whole patch or for a significant part thereof, information about the relative velocity between the object and the camera can be obtained with high accuracy.

[0032] The two steps S41 and S42 of the proposed method can be described as follows.

[0033] In S41, first, the value of the optical flow is determined for the pixels of one of the initial image or the final image; and, together with or after determining the value of the optical flow, the optical flow is used to determine at least one parameter of the transformation.

[0034] Here, the term "used" means that the optical flow is used as an input or as an intermediate variable during the determination of the parameter(s) of the transformation.

[0035] In this step, the parameter(s) of the transformation are determined together with or after determining the value of the optical flow.

[0036] Then, in S42, information about the relative velocity between the object and the camera is computed based on the parameter(s) of the transformation determined in the first step.

[0037] Direct method

[0038] The first embodiment defines a "direct method", which is called "direct method" because the parameter(s) of the transformation are computed in one step (step S412 below).

[0039] In this method, step S41 comprises two sub-steps:

[0040] S411) determining the value of the optical flow between the two images; and,

[0041] The transformed pixel is a pixel obtained by applying the transformation to the corresponding previous pixel,

[0042] The coordinates of the transformed pixel are thus obtained by adding the value of the optical flow F of the corresponding previous pixel to the coordinates of the corresponding previous pixel;

[0043] S412) solving a system of equations expressing the coordinates of the transformed pixel as a function of the coordinates of the corresponding previous pixel using said at least one parameter of the transformation, and thereby determining said at least one parameter of the transformation.

[0044] During step S411, the optical flow can be computed by any known method. For example, the Lucas-Kanade algorithm, the Horn-Schunck algorithm, etc. can be used to compute the optical flow.

[0045] Then, during a step S412, the parameter(s) of the transformation are computed, based on the optical flow. During this step, the system of equations can be solved, for example by robust fitting of a linear model, for example by a RANSAC method or an IRLS method ("Iterative Reweighted Least Squares" method).

[0046] Iterative method

[0047] As an alternative to the direct method, in a second or alternative method, the parameter(s) of the transformation are computed iteratively. In this method, the parameter(s) of the transformation are computed as well as the optical flow.

[0048] The iterative method is defined as follows:

[0049] The transformed pixel is a pixel obtained by applying the transformation to the corresponding previous pixel;

[0050] The coordinates of the transformed pixel are thus obtained by adding the value of the optical flow F of the corresponding previous pixel to the coordinates of the corresponding previous pixel;

[0051] The first equation is an equation expressing the coordinates of the transformed pixel as a function of the coordinates of the corresponding previous pixel using said at least one parameter of the transformation, and

[0052] The second equation is an optical flow equation expressing a relationship between, for the considered pixel, the space-time information about the pixel intensity I at the considered pixel, the space-time information about the pixel intensity at another pixel offset by an estimated offset value from the considered pixel, and said at least one parameter of the transformation;

[0053] The step S41 comprises performing the following initialization step:

[0054] S415) determining initial values of said at least one parameter of the transformation and of the optical flow;

[0055] The step S41 further comprises iteratively performing the following steps:

[0056] S416) determining updated values of said at least one parameter of the transformation, based on the estimated value of the optical flow F for said considered pixel, by solving the system of second equations established for the considered pixel of the initial image;

[0057] S417) determining whether it is necessary to perform a new iteration (of the algorithm); and,

[0058] S418) determining an updated estimated value of the optical flow F for said considered pixel, using the system of first equations for said considered pixel and the value of the parameter(s) of the transformation determined in the step S416.

[0059] Step S418 is preferably a conditional step, which is only performed if it is determined in step S417 that it is necessary to perform another iteration.

[0060] The space-time information can comprise, for example, a space-time derivative of the pixel intensity at the considered pixel.

[0061] In step S417, in some embodiments, it is determined that it is necessary to perform a new iteration of the algorithm when the difference between the updated value of the parameter(s) and the previously calculated value of the parameter(s) exceeds a predetermined threshold.

[0062] Alternatively or additionally, in other embodiments, it is determined that it is necessary to perform a new iteration of the algorithm as soon as (or only if) the maximum number of iterations i max is not exceeded.

[0063] In this second method, contrary to the direct method, the optical flow F is not a freely defined "per-pixel flow field", but is defined by a transformation chosen to model the motion of the sub-image as a whole.

[0064] The optical flow F can thus be expressed in terms of a function. Many different functions can be used.

[0065] For example, in one embodiment, the flow is expressed in terms of an expansion s between the centers of the sub-images in successive frames and a displacement vector (tx, ty). However, any other suitable expression can be chosen to model the optical flow F.

[0066] Advantageously, it has been found that the parameter(s) of the transformation (such as the expansion rate, etc.) identified with this method are very numerically stable - at least for small displacements - even in the presence of noise, attenuation or compression artifacts in the input images.

[0067] In this method, the values of the optical flow F of the pixels of the initial image are estimated step by step, using an algorithm called "biloc algorithm", together with and constrained by the parameter(s) of the transformation.

[0068] The biloc algorithm is particularly suitable in the case where the values of the optical flow are large, i.e. where the motion of the pixels from the initial image to the final image is large. This algorithm can be implemented in a particularly efficient manner when the initial estimate of the optical flow can be derived from a previous frame acquired by the camera.

[0069] In a usual method of optical flow determination, the optical flow is determined based on the displacement between two images, by considering the space-time derivative at a single location in the two images, assuming that the motion is small.

[0070] On the contrary, the second method is a "bilocal" method. The term "bilocal" means that the optical flow equation is now formulated to take into account the space-time information at two different locations in the image (the "considered pixel" and "another pixel" mentioned above).

[0071] The other pixel is obtained by shifting the "considered pixel" by an estimated offset value. This offset value is computed based on the current value of the optical flow; for example, it can be equal to the current value of the optical flow.

[0072] Thanks to this feature, the second method can be used even in case of large displacement from one frame to the next.

[0073] Of course, it is not necessary to establish the equation for all the pixels (or combinations of pixels) of the initial image or of the final image. However, the more the number of pixels used, the higher the accuracy in determining the parameter(s) of the transformation.

[0074] In an embodiment, the second set of equations represents, for each of the considered pixels, a relationship between the space-time information in the initial image at the considered pixel and the space-time information in the final image at the other pixel.

[0075] In an embodiment, wherein the transformation is a similarity defined by the transformation parameters s, tx and ty, the second set of equations is defined as:

[0076] (2d')

[0077] where, for the current iteration i, the bilocal spatial derivative and and the bilocal temporal derivative are defined as:

[0078] (2a)

[0079] (2b)

[0080] (2c)

[0081] where u i , v i are the values of the optical flow F for the current iteration i.

[0082] The computation methods according to the present disclosure (in particular the direct method and the iterative method) can be implemented for a variety of transformations. For example, the transformation can be a perspective transformation, a similarity transformation combined with a translation, an affine transformation, a rotation of a rigid object combined with a translation.

[0083] After having computed information related to the relative speed between the object and the camera, in step S42, this information can be used for various purposes.

[0084] In an embodiment of particular interest, the information is used to control at least one device, for example at least one device of a vehicle.

[0085] For example, the control method can comprise the following steps:

[0086] S10) acquiring, with at least one camera, camera images of the environment of said at least one camera;

[0087] S20) identifying an object in at least one pair of camera images acquired by one of said at least one camera;

[0088] S30) extracting an initial image and a final image of the object, the initial image and the final image being portions of said pair of camera images defined by a bounding box of said object identified;

[0089] S40) computing information related to the relative speed between the object and said at least one camera using one of the methods defined above;

[0090] S50) controlling said at least one vehicle device based on said information related to the relative speed between the object and said at least one camera.

[0091] In a particular embodiment, the proposed control method is determined by computer program instructions.

[0092] It is therefore another object of the present disclosure to propose a computer program stored on a non-transitory computer-readable storage medium and adapted to be executed on a processor, the program comprising instructions adapted, when run on the processor, to perform the steps of the method as defined above.

[0093] The computer program can use any programming language and can be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desired form.

[0094] The processor described above can be any data processing device, for example a personal computer, an electronic control unit configured to be installed in a vehicle, etc.

[0095] The present disclosure also comprises a computer-readable medium having stored therein instructions which, when executed by a processor, cause the processor to execute the computer program described above.

[0096] The computer readable medium can be an entity or device capable of storing the program. For example, the computer readable medium can include a storage means such as a Read Only Memory (ROM), e.g. a Compact Disc Read Only Memory (CD ROM), or a microelectronic circuit ROM, or indeed a magnetic recording means, e.g. a floppy disk or a hard disk.

[0097] Alternatively, the computer readable medium can be an integrated circuit in which the program is incorporated, the circuit being adapted to perform, or to control the performance of, the control method in question.

[0098] Another object of the disclosure is to propose an object motion analysis system. The object motion analysis system comprises an electronic control unit configured to perform one of the above methods based on an initial image and a final image of an object derived from image frames output by a camera:

[0099] S41) determining a value of an optical flow between the initial image and the final image, the optical flow defining, for a pixel of the initial image, an estimated motion of this pixel during a transformation of the initial image into said final image; and,

[0100] using the optical flow to determine at least one parameter of the transformation, together with or after determining the value of the optical flow; and

[0101] S42) computing information related to a relative speed between the object and the camera based on said at least one parameter of the transformation.

[0102] More generally, the invention comprises an object motion analysis system comprising an electronic control unit configured to perform one of the above methods based on an initial image and a final image of an object derived from image frames output by a camera.

[0103] In an embodiment, the electronic control unit is configured to:

[0104] S411) determining a value of an optical flow between the two images; and,

[0105] the transformed pixel being a pixel obtained by applying the transformation to the corresponding previous pixel,

[0106] the coordinates of the transformed pixel being thus obtained by adding the value of the optical flow F of the corresponding previous pixel to the coordinates of the corresponding previous pixel;

[0107] S412) solving a system of equations expressing the coordinates of the transformed pixel as a function of the coordinates of the corresponding previous pixel using said at least one parameter of the transformation, and thereby determining said at least one parameter of the transformation.

[0108] For example, the electronic control unit can be configured to solve the system of equations by robustly fitting a linear model, for example by a RANSAC method or an IRLS method.

[0109] Alternatively, in an embodiment, and based on the definition of the first and second equations as presented above, the electronic control unit is configured to perform the following initialization steps:

[0110] S415) determining initial values of the at least one parameter of the transformation and of the optical flow;

[0111] The electronic control unit is further configured to iteratively perform the following steps:

[0112] S416) determining updated values of the at least one parameter of the transformation based on the estimated value of the optical flow F for the considered pixel and by solving the system of second equations established for the considered pixel of the initial image;

[0113] S417) determining whether it is necessary to perform a new iteration; and

[0114] S418) determining an updated estimated value of the optical flow F for the considered pixel using the system of first equations for the considered pixel and the value(s) of the transformation parameter(s) determined in step S416.

[0115] The transformation can in particular be a perspective transformation, a similarity transformation combined with a translation, an affine transformation, a rotation of a rigid object combined with a translation.

[0116] The updated estimated value of the optical flow F for the considered pixel is preferably determined in step S418 only if it is determined in step 417 that it is necessary to perform another iteration.

[0117] The object motion analysis system can in particular be integrated in a control system, in particular in an automatic driving system for a vehicle.

[0118] In this case, the control system comprises at least one camera configured to acquire images; and the electronic control unit is configured to:

[0119] S10) acquiring, with the at least one camera, camera images of an environment of the at least one camera;

[0120] S20) identifying an object in at least one pair of camera images acquired by one of the camera(s);

[0121] S30) extracting an initial image and a final image of the object, the initial image and the final image being portions of the pair of camera images defined by a bounding box of the identified object;

[0122] S40) calculating information related to the relative speed between the object and the at least one camera using one of the methods defined above; and

[0123] S50) controlling at least one vehicle device of the automatic control system based on said information related to the relative speed between the object and the at least one camera. BRIEF DESCRIPTION OF DRAWINGS

[0124] The present disclosure can be better understood, and its numerous objects, advantages, and benefits will become more apparent to those skilled in the art by reference to the following drawings, in which like reference numerals refer to like elements throughout the several views, and wherein:

[0125] Figure 1 is a schematic view substantially illustrating a car equipped with an automatic driving system according to the present disclosure;

[0126] Figure 2 is a block diagram illustrating the steps of a method for determining information related to the relative speed between a moving object and a camera of a vehicle according to a first embodiment of the method according to the present disclosure; and

[0127] Figure 3 is a block diagram illustrating the steps of a method for determining information related to the relative speed between a moving object and a camera of a vehicle according to a second embodiment of the method according to the present disclosure. DETAILED DESCRIPTION

[0128] An automatic driving system 10 (as an example of a control system according to the present disclosure) will now be described, configured to implement the proposed method for controlling at least one device.

[0129] Figure 1 A car 100 equipped with an automatic driving system 10 forming an exemplary embodiment of a control system according to the present disclosure is schematically illustrated. The car 100 is moving on a road (arrow A); in the opposite direction, another car 200 is coming in the other lane.

[0130] The automatic driving system 10 (or simply, system 10) comprises an object motion analysis system including an electronic control unit 20, a camera 30, and several other sensors not shown.

[0131] The images produced by the camera 30 are sent to the electronic control unit 20 (ECU 20).

[0132] The ECU 20 has the hardware architecture of a computer in general. The ECU 20 comprises a microprocessor 22, a Random Access Memory (RAM) 24, a Read Only Memory (ROM) 26, an interface 28.

[0133] The hardware elements of the ECU 20 are optionally shared with other units of the autonomous driving system 10 and / or other systems of the car 100.

[0134] The interface 28 comprises in particular a haptic display installed in or on the dashboard of the car, as well as various displays.

[0135] The interface 28 comprises a driver interface with a (not shown) display to send information to the driver of the car 100, as well as an interface connection with actuators and other vehicle devices of the car. In particular, the interface 28 comprises a connection with the driving actuators of the car 100, i.e. the engine 32, the steering column 34 and the brakes 36.

[0136] The memory 26 stores a computer program configured to partially assume the driving task by performing lateral and longitudinal control of the vehicle. The program is configured to compute information related to the relative speed between the car 100 and surrounding objects detected in images sent by the camera 30. The program is further configured to output commands to control the driving actuators of the host vehicle during at least some driving periods.

[0137] The program and the memory 26 are respectively an example of a computer program and of a non-transitory computer readable medium according to the present disclosure.

[0138] The Read Only Memory 26 of the ECU 20 does indeed constitute a non-transitory computer readable medium readable by the processor 22 according to the present disclosure. It stores instructions which, when executed by the processor, cause the processor 22 to perform the control method according to the present disclosure.

[0139] More specifically, the program stored in the memory 26 comprises instructions for performing a method for controlling a plurality of devices of a vehicle 100 based on information related to the relative speed between the vehicle and an “Independently Moving Object” (IMO) moving outside the vehicle. In the present embodiment, a vehicle 200 coming in the opposite direction is taken as an example of an IMO outside the vehicle 100. In the present embodiment, the autonomous driving system 10 constitutes an exemplary object motion analysis system according to the present disclosure.

[0140] To perform its function, the system 10 uses the images provided by the cameras 30, processes these images in the ECU 20 and controls the vehicle 100 based on information calculated by the ECU 20 related to the relative speed between the vehicle and the “independently moving object” detected around the vehicle.

[0141] According to the present disclosure, the vehicle 100 can be controlled during driving, in particular according to one of the control methods illustrated in Figure 2 and Figure 3 The control method according to the present disclosure is illustrated in Figure 2 and Figure 3 An exemplary embodiment of the control method according to the present disclosure is illustrated in

[0142] In the control method illustrated in Figure 2 the parameters of the transformation transforming the initial image into the final image are calculated in a direct two-step method. In contrast, in the control method illustrated in Figure 3 the parameters are calculated iteratively.

[0143] The steps of the control method illustrated in Figure 2 and Figure 3 will now be described.

[0144] Step S10-S30: Preparation

[0145] In the control method illustrated in Figure 2 and Figure 3 the control method first comprises the following preparation steps:

[0146] S10) The camera images of the environment of the vehicle are acquired continuously by the cameras 30. For each of these successive acquisitions, the value of the main index k is incremented. In practice, the cameras 30 continuously monitor the scene in front of the car 100 and therefore send image frames to the electronic control unit 20 at a rate of 30 frames per second.

[0147] S20) In each of these images, the electronic control unit identifies the objects present. This identification is made using a neural network and focuses on identifying in particular pedestrians, other vehicles, etc.

[0148] In the present example, an oncoming car 200 is identified in the successive images sent by the cameras 30.

[0149] S30) When an object is identified in an image, the image of the object is extracted from the camera image. To this end, a bounding box is determined for the object in each of these camera images. Then, a sub-image showing the object in the image is cropped from the camera image so as to be defined by the bounding box of the object.

[0150] According to this procedure, two images (Ik and I k+1 where k and k+1 correspond to two successive acquisitions of the camera 30) are thus extracted from the frames acquired by the camera 30 and represent the initial and final images of the object having the meaning of the present disclosure.

[0151] Each of these images forms a matrix of pixel intensities I(x,y).

[0152] Step S40: Calculate information related to relative speed

[0153] S40) Then, in two steps S41 and S42, information related to the relative speed between the object and the camera is calculated.

[0154] Step S41

[0155] In step S41, the optical flow F between the two images I k and I k+1 is determined; then, once the optical flow F has been determined, the parameter(s) of the transformation that transforms the initial image (I k ) into the final image (I k+1 ) is calculated using the optical flow F.

[0156] The optical flow defines, for a pixel (x,y) of the initial image (I k ), the estimated motion (u,v) of these pixels between the initial image (I k ) and the final image (I k+1 ). As mentioned previously, in general terms, the optical flow is defined by a function F that provides (u,v) as a function of (x,y):

[0157] (u,v) = F(x,y)

[0158] In this expression, (x,y) denotes the coordinates of a pixel in the initial image. Generally, the optical flow is modeled by making some assumptions that simplify the calculations.

[0159] In some embodiments, for example, in the embodiments considered herein in connection with Figure 2 and Figure 3 , it is assumed that the transformation of the image is a similarity ratio s combined with a translation (tx,ty). Thus, the position of the transformed pixel is defined by the following equation:

[0160] (1)

[0161] More generally, in the method according to the disclosure, any kind of function can be used to define the transformation of the images. For example, the transformation can be defined as a perspective transformation, an affine transformation, a rotation of a rigid object combined with a translation, etc.

[0162] Thus, in the present embodiment, the optical flow F is defined by the following system of equations (1') which is equivalent to the system (1):

[0163] (1')

[0164] The system (1') can be established for any pixel (x, y) of the initial image for which the optical flow (u, v) can be computed.

[0165] The step 41 can be implemented in different ways. Two preferred implementations of the step S41 will be given in more details below: Figure 2 the "direct method" illustrated, and Figure 3 the "iterative method" illustrated.

[0166]

[0167] In an implementation of the control method, the step S41 comprises two sub-steps.

[0168] In a first sub-step S411, the optical flow F is determined. Here, any known method can be used to compute the optical flow F.

[0169] Then, in a sub-step S412, the parameter(s) of the transformation are determined.

[0170] For example, if the transformation is a similarity combined with a translation defined by the parameters s, tx, ty as conceived above, all the systems (1') established for all the pixels (x, y) of the initial image for which the optical flow can be computed are combined, i.e. one large system of equations is obtained. This system of equations comprises only three unknowns, s, tx and ty.

[0171] During the step S412, this system of equations is solved by any known method. It can for example be solved classically by the least squares method; however, a robust fitting method is preferably used, such as the RANSAC method, the IRLS method, etc.

[0172]

[0173] In the step S41 as described above, the parameter(s) (s, tx, ty) of the transformation are determined based on the initial image (I k ) and the final image (I k+1 ).

[0174] In this implementation of step S41, step S41 comprises an initialization sub-step S415, followed by several sub-steps which are iteratively executed. At each iteration, the index i is incremented, this index i being the index of the local loop nested in step S41.

[0175] First, in the initialization step S415(i=0), initial values are set for the estimated optical flow F.

[0176] In the iterative approach, the optical flow F is constrained (or defined) by the parameter(s) of the transformation. Thus, the initialization of the flow is performed by initializing the parameter(s) of the transformation.

[0177] Thus, the initialization comprises setting initial values for the parameter(s) of the transformation. In the present exemplary embodiment, the following values can be chosen for example:

[0178] (s, tx, ty)0= (1, 0, 0).

[0179] Thus, for each pixel in the sub-image, u0= v0= 0 (here, in the variables (s, tx, ty)0, u0and v0, the subscript refers to i, with i = 0).

[0180] The initialization of the optical flow is not limited to u0= v0= 0. Any reasonable initial value of u0and v0may be used. For example, their values can be initialized by performing the direct method described above.

[0181] Then, in step S416, updated values (s, tx, ty) of the parameter(s) of the transformation are computed i+1 .

[0182] In applying the principle of the "dual local algorithm" previously proposed, step S416 is based on the following computation:

[0183] For the pixels of the initial image I k for which it is possible to be computed, the dual local spatial derivative is introduced, denoted for the current iteration i as and and defined as follows:

[0184] (2a)

[0185] (2b)

[0186] In the above equations, the values u i , v i are the values of the optical flow F for the current iteration (pending values for the index i).

[0187] Moreover, the following dual local temporal derivative is also introduced which is defined as follows:

[0188] (2c)

[0189] The double local temporal derivative can also be expressed in the following simplified form:

[0190]

[0191] The double local spatial derivative and and the double local temporal derivative are examples of space-time information according to the present disclosure.

[0192] On the numerical side, for this implementation, the spatial derivative and is evaluated using bilinear interpolation to perform the evaluation of the space-time information at (x + u i , y + v i ).

[0193] To formulate the double local optical flow, for example based on the Horn-Schunck algorithm, the basic (local) optical flow equation is used to express the conservation of luminance intensity of a pixel across consecutive frames. This leads to:

[0194] (2d1)

[0195] Of course, other criteria than the conservation of luminance intensity of a pixel across frames can be used to compute the double local formulation (which defines the second equation 2d1).

[0196] As an example, it is possible to add the possibility of an additive scaling and a multiplicative scaling on the intensity variation. Moreover, the Horn and Schunck equation above models only first order effects, neglecting the higher order terms in the Taylor expansion (equation 2d2). Also from this perspective, it is possible to add second order and higher order terms in the optical flow equation.

[0197] Given an estimate of the optical flow at the considered pixel (u i , v i ), it is possible to reformulate the above equation (2d1) using the current frame I k and the next frame I k+1 (the next frame I k+1 is "warped" with this initial estimate). Thus, the double local optical flow equation can be formulated as follows, where equation (2d1) is applied to the residual flow (u - u i , v - vi

[0198] (2d)

[0199] The above equation is explained as follows:

[0200] Note: in the following description, the optical flow estimation is denoted without involving the iterative process, as the mathematical description is general and does not depend on the iteration.

[0201] Let us consider the vector and its position at distance in two successive frames k and k+1, observed at intensity images I (here the analysis is given for two frames, but it can be applied to more than two frames): • The brightness conservation rule across successive frames forces

[0202] For small displacements, one can use a Taylor expansion to approximate this equation, so that,

[0203] (2d2)

[0204] When omitting the second order effects, equation (2d2) is equivalent to the 1D equation which is equivalent to the scalar formulation of equation (2d1) involving spatial and temporal information at x. The temporal derivative represents the intensity variation between two successive frames of choice.

[0205] • Moreover, starting from equation (2d2), one can also consider k+1 as the reference and express the observed motion -u from frame k+1 to k, i.e. This gives:

[0206] (2d3)

[0207] From (2d2) and (2d3) we have: This confirms that the gradient remains constant under the brightness conservation assumption.

[0208] • Given an estimate of the actual motion and a residual motion which follows equations (2d2) and (2d3):

[0209]

[0210] (2d4)

[0211] (2d5) ​​

[0212] Equation (2d4) in and Residual motion was established at the location The bilocal relationship between spatial-temporal information.

[0213] In this formula, there exists Smaller implicit assumptions, but for or There is no limit to the size. Equation (2d5) describes frame I. k and I k+1 The equivalent forward / backward twisting and corresponding motion between them.

[0214] Equation (2d4) is equivalent to the scalar formula of equation (2d).

[0215] The above equations can then be simplified by replacing the optical flow (u,v) with an expression that functions as a function of (x,y) and the parameters (one or more) of the transformation. In the proposed embodiment, the transformation is a similarity transformation defined by equation (1) with parameters (s,tx,ty).

[0216] Therefore, the two local optical flow equation (2d) can be expressed by parameters (s, tx, ty) as follows:

[0217] (2d')

[0218] Equation (2d') constitutes a second equation with the meaning of this disclosure.

[0219] These second equations are written for all pixels that can be computed. This results in a set of equations that represent the relationship between the spatial-temporal information of the pixel intensity I at the considered pixel, the spatial-temporal information of the pixel intensity I at another pixel for each of the considered pixels, and the transformation(one or more) parameters (s, tx, ty).

[0220] More specifically, for each of the considered pixels, these second equations are expressed in the considered pixels. Initial image I at the location k Spatial-temporal information in the final image (I k+1 The other pixel in ) The relationship between spatial and temporal information.

[0221] The other pixel is an estimated offset value from the considered pixel offset, which is based on the current value of the optical flow of the considered pixel (u). i ,v i). In the present embodiment, the estimated offset value is equal to the current estimate of the optical flow (u i ,v i ).

[0222] The system of equations (2d') is a linear system (with respect to the parameters (s, tx, ty) ) which is then solved by any appropriate method, thereby providing new (or updated) estimates (s, tx, ty) of the parameters of the transformation s, tx and ty i+1 .

[0223] Then, based on the updated estimates (s, tx, ty) i+1 , and using equation (1'), updated values of the optical flow (u i+1 ,v i+1 ) are computed.

[0224] Then, in step S417, the new estimates of the parameters (s, tx, ty) i+1 are compared to the previous estimates (s, tx, ty) i . If the difference exceeds a predetermined threshold, it is concluded that the algorithm has not yet converged.

[0225] The algorithm therefore jumps to step S418. In step S418, updated values of the optical flow (u i+1 ,v i+1 ) are computed, for each pixel (x, y) of the initial image for which these values can be computed, based on equation (1').

[0226] The algorithm then continues at step S416, as described above.

[0227] On the contrary, if in step S1417 it is determined that the difference between (s, tx, ty) i+1 and (s, tx, ty) i is below a predetermined threshold, it is concluded that the algorithm has sufficiently converged. It is therefore concluded that the values (s, tx, ty) i+1 are the values of the parameter(s) of the transformation that transforms the initial image into the final image.

[0228] Step S42

[0229] The algorithm then continues at step S42.

[0230] In step S42, the values of the parameter(s) of the transformation computed in step S412 are used to compute information related to the relative speed between the object and the camera.

[0231] This information can be very simple. For example, depending on the value of the similarity ratio s, it can be determined whether the detected object is moving away (s < 1) or moving towards (s > 1) the vehicle. This information is information related to the relative speed between the vehicle and the object, and is sometimes sufficient to decide to stop tracking the object (if it moves away), or, on the contrary, to increase the level of attention invested to the object if it becomes closer.

[0232] In fact, the similarity ratio s is a visual expansion parameter: it characterizes how fast the image of the object grows or, on the contrary, shrinks in the image acquired by the camera 30.

[0233] As another information related to the relative speed between the object and the camera that can be extracted from the parameter(s) of the transformation, based on the estimate D of the distance from the vehicle 100 to the object 200 (which can be a rough estimate) and the similarity ratio s, the depth relative velocity RVD of the object with respect to the vehicle can be computed using the following formula:

[0234] RVD = D · (1 - s) / (T1 - T0)

[0235] where T0and T1are the instants at which the initial image I k and the final image I k+1 are acquired.

[0236] As another information related to the relative speed between the object and the camera that can be extracted from the parameter(s) of the transformation, based on the depth relative velocity RVD, the time to collision TtCwith the object (car 200) can be estimated using the following formula:

[0237] TtC= D / RVD

[0238] It is also possible to consider the above equations for RVDand TtCat different time steps T0,..., T n assuming, for example, a constant relative velocity over the time period. This combination leads to a system of equations:

[0239] RVD= D0· (1 - s k ) / (T k - T0), k = 1,..., n

[0240] where s k is the expansion rate at different time steps T k , k = 1,..., n, and are all computed with respect to the initial frame at T0.

[0241] This joint system of homogeneous equations is then directly solved to compute TtC: given as an estimate for D, RVD is computed as:

[0242] RVD = D / TtC.

[0243] This particular formulation stabilizes the numerical output of D and RVD by TtC.

[0244] Moreover, in addition to the relative velocity RVD, the relative acceleration RAC can be considered in the formulation to calculate the time TtC of collision.

[0245] In this case, the velocity is not considered constant; at different time steps T0,..., T n above and given the extension s k under to the object D k can be expressed as:

[0246] D k = D0- RVD(T k - T0) - RAC(T k - T0) 2 / 2 = s k D0

[0247] (The above equation can be considered as a Taylor expansion of the kinematic behavior of the car: it can of course include additional term(s)).

[0248] The above homogeneous system of equations can be solved to determine the values of RVD / D and RAC / D. Since there are two unknowns, at least 3 observations (e.g. k = 1, 2, with 0 used as a reference) must be used to obtain a solution.

[0249] If a proper estimate exists for the distance D0(e.g. by means of the car width and camera intrinsic characteristics), the time TtC of collision can be calculated based on the values of RVD and RAC.

[0250] Final step S50: Control at least one vehicle device

[0251] S50) Finally, the electronic control unit 20 outputs a command to control at least one vehicle device (32, 34, 36) of the car 100 based on said information related to the relative velocity between the object and the camera, e.g. based on the visual extension s, the depth relative velocity RVD of the car 200 and / or the time TtC of collision with the car 200.

[0252] Although embodiments of the method according to the present disclosure are presented only for the acquisition of two images of the object derived from camera images acquired by a (single) camera, it should be understood that these methods are generally implemented iteratively based on a succession of images acquired by one or more cameras at successive time steps.

[0253] Thus, in some embodiments, step S41 can be performed several times based on pairs of initial images and final images. Thereby several values of the parameter(s) of the transformation are obtained. Then, in step S42, information related to the relative velocity between the object and the camera can be calculated based on these different values.

[0254] Further, the above defined method can be performed to calculate information related to the relative velocity between the object and the camera not only based on a single camera, but based on multiple (two or more) cameras. In this case, all cameras must be held in a fixed position relative to each other. Further, the cameras must have overlapping fields of view so that the object appears in the images of the different cameras.

[0255] In this case, step S41 can be performed based on pairs of initial images and final images, wherein the pairs of initial images and final images are based on camera images of different cameras. Thereby several values of the parameter(s) of the transformation are obtained. Then, in step S42, information related to the relative velocity between the object and the camera can be calculated based on these different values.

Claims

1. A computer-implemented method for deriving an initial image (I) of an object based on image frames output from a camera (30) mounted on a vehicle (100). k ) and the final image (I k+1 The method includes calculating information related to the relative velocity between the object (200) and the camera (30). S41) Determine the initial image (I k ) and the final image (I k+1 The value of optical flow (F) between pixels (x, y) of the initial image, wherein the optical flow is defined for each pixel (x, y) in the initial image (I). k ) transformed into the final image (I) k+1 The estimated motion (u,v) during the transformation period of ) ; and, Together with or after determining the value of the optical flow, the optical flow is used to determine the initial image (I) k Transformed into the final image (I) k+1 The transformation of ) includes at least one parameter (s,tx,ty), said at least one parameter (s,tx,ty) including similarity ratio (s) and translation (tx,ty); S42) Based on the similarity ratio (s), obtain the initial image (I). k The time (T0) and the acquisition of the final image (I) k+1 The depth relative velocity (RVD) of the object (200) relative to the vehicle (100) is calculated according to the following formula, based on the time (T1) of the collision between the vehicle (100) and the object (200), and the estimated distance (D) between the vehicle (100) and the object (200), the collision time (TtC) between the vehicle (100) and the object (200) is calculated: RVD = D·(1–s) / (T1–T0); and S50) at least one device (32, 34, 36) controls the vehicle (100) based on the depth relative velocity (RVD) and the collision time (TtC).

2. The computer-implemented method as described in claim 1, wherein, Step S41 includes: S411) Determine the two images (I k ,I k+1 The value of the optical flow between ) and; and, The transformed pixel is obtained by applying the transformation to the corresponding previous pixel; The coordinates of the transformed pixel are therefore obtained by adding the value (u,v) of the optical flow (F) of the corresponding previous pixel to the coordinates (x,y) of the corresponding previous pixel; S412) Solve a system of equations that express the coordinates (x+u,y+v) of the transformed pixel as a function of the coordinates (x,y) of the corresponding previous pixel using at least one parameter (s,tx,ty) of the transformation, and thereby determine the at least one parameter (s,tx,ty) of the transformation.

3. The computer implementation method according to claim 1, wherein: The transformed pixel is obtained by applying the transformation to the corresponding previous pixel. The coordinates of the transformed pixel are therefore obtained by adding the value (u,v) of the optical flow (F) of the corresponding previous pixel to the coordinates (x,y) of the corresponding previous pixel; The first equation (1) is an optical flow equation that uses at least one parameter (s, tx, ty) of the transformation to express the coordinates (x+u, y+v) of the transformed pixel as the coordinates (x, y) of the corresponding previous pixel. The second equation (2d') represents the spatial-temporal information of the pixel intensity I at the considered pixel (x,y). Regarding the deviation from the estimated pixel offset value (u) mentioned above i ,v i Spatial-temporal information of pixel intensity at another pixel. And the optical flow equation relating at least one parameter (s, tx, ty) of the transformation; Step S41 includes performing the following initialization steps: S415) Determine the initial value ((s,tx,ty)0) of at least one parameter of the transformation and the initial value (u0,v0) of the optical flow; Step S41 also includes iteratively performing the following steps: S416) Based on the estimated value (u) of the optical flow (F) for the considered pixel. i ,v i The at least one parameter ((s,tx,ty) of the transformation is determined by solving a set of second equations (2d') established for the considered pixels of the initial image. i+1 The updated value of ); S417) Determine whether it is necessary to perform a new iteration; S418) Using the set of the first equation (1) for the considered pixel and the one or more transformation parameters ((s,tx,ty) determined in step S416) i+1 The value of ) is used to determine the updated estimate (u) of the optical flow (F) for the considered pixel. i+1 ,v i+1 ).

4. The computer-implemented method as described in claim 3, wherein, For each of the considered pixels, the set of the second equations represents the considered pixel. The initial image (I) at the location k The spatial-temporal information in the final image (I) k+1 The other pixel in ) The relationship between spatial and temporal information.

5. The computer-implemented method as described in claim 3, wherein: The transformation is a similarity defined by transformation parameters s, tx, and ty; and The second set of equations is defined as follows: (2d') Among them, the bilocal spatial derivative with respect to the current iteration i and and the two local time derivatives Defined as: (2a) (2b) (2c) Among them, u i v i It is the value of the optical flow F for the current iteration i.

6. The computer-implemented method as described in claim 1, wherein, The transformation is a perspective transformation, a similarity transformation (s) combined with translation (tx, ty), an affine transformation, or a rotation of a rigid object combined with translation.

7. A control method for controlling at least one device (32, 34, 36) of a vehicle (100), the control method comprising the following steps: S10) Using at least one camera (30) mounted on the vehicle (100), acquire camera images of the environment of the at least one camera (30); S20) Identify an object (200) in at least one pair of camera images acquired by one of the at least one camera (30); S30) Extract the initial image of the object (I k ) and the final image (I k+1 The initial image (I) k ) and the final image (I k+1 ) is the portion of the camera image pair defined by the bounding box of the identified object (200); S40) The depth relative velocity (RVD) of the object (200) relative to the vehicle (100) and the collision time (TtC) between the vehicle (100) and the object (200) are calculated using the method according to any one of claims 1 to 6. S50) controls the at least one device (32, 34, 36) based on the depth relative velocity (RVD) and the collision time (TtC).

8. A computer program stored on a computer-readable storage medium and adapted to be executed on a processor, said program comprising instructions which, when executed on said processor, are adapted to perform the steps of the method according to claim 1.

9. A computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the computer program according to claim 8.

10. An object motion analysis system (10) mounted on a vehicle (100), the object motion analysis system (10) including an electronic control unit (20) configured to analyze an initial image (I) of the object derived from image frames output by a camera. k ) and the final image (I k+1 ): S41) Determine the initial image (I k ) and the final image (I k+1 The value of optical flow (F) between pixels (x, y) of the initial image, wherein the optical flow is defined for each pixel (x, y) in the initial image (I). k ) transformed into the final image (I) k+1 The estimated motion (u,v) during the transformation period of ) ; and, Together with or after determining the value of the optical flow (u,v), the optical flow is used to determine the initial image (I). k Transformed into the final image (I) k+1 The transformation of ) includes at least one parameter (s, tx, ty), said at least one parameter (s, tx, ty) comprising a similarity ratio (s) and a translation (tx, ty); and S42) Based on the similarity ratio (s), obtain the initial image (I). k The time (T0) and the acquisition of the final image (I) k+1 The depth relative velocity (RVD) of the object (200) relative to the vehicle (100) is calculated according to the following formula, based on the time (T1) of the collision between the vehicle (100) and the estimated distance (D) from the vehicle (100) to the object (200), and the collision time (TtC) between the vehicle (100) and the object (200) is calculated based on the depth relative velocity (RVD) of the object (200) relative to the vehicle (100) and the estimated distance (D) from the vehicle (100) to the object (200): RVD = D·(1–s) / (T1–T0); and S50) at least one device (32, 34, 36) controls the vehicle (100) based on the depth relative velocity (RVD) and the collision time (TtC).

11. The object motion analysis system (10) according to claim 10, wherein, The electronic control unit (20) is configured as follows: S411) Determine the two images (I k ,I k+1 The value of the optical flow between ) and; and, The transformed pixel is obtained by applying the transformation to the corresponding previous pixel. The coordinates of the transformed pixel are therefore obtained by adding the value (u,v) of the optical flow F of the corresponding previous pixel to the coordinates (x,y) of the corresponding previous pixel; S412) Solve a system of equations that express the coordinates (x+u,y+v) of the transformed pixel as a function of the coordinates (x,y) of the corresponding previous pixel using the at least one parameter (s,tx,ty) of the transformation, and thereby determine the at least one parameter (s,tx,ty) of the transformation.

12. The object motion analysis system (10) according to claim 11, wherein, The electronic control unit is configured to solve the system of equations by robustly fitting a linear model.

13. The object motion analysis system (10) according to claim 10, wherein: The transformed pixel is obtained by applying the transformation to the corresponding previous pixel. The coordinates of the transformed pixel are therefore obtained by adding the value of the optical flow F (u(x,y), v(x,y)) of the corresponding previous pixel to the coordinates (x,y) of the corresponding previous pixel. The first equation (1) is an equation that uses the at least one parameter (s,tx,ty) of the transformation to express the coordinates (x+u,y+v) of the transformed pixel as the coordinates (x,y) of the corresponding previous pixel. The second equation (2d') represents the spatial-temporal information of the pixel intensity at the considered pixel (x,y). Regarding the deviation from the considered pixels to estimate the offset value (u) i ,v i Spatial-temporal information of pixel intensity I at another pixel. And the optical flow equation relating at least one parameter (s, tx, ty) of the transformation; the electronic control unit (20) is configured to perform the following initialization steps: S415) Determine the initial value ((s,tx,ty)0) of at least one parameter of the transformation and the initial value (u0,v0) of the optical flow; The electronic control unit (20) is configured to iteratively perform the following steps: S416) Based on the estimated value (u) of the optical flow F for the considered pixel. i ,v i The at least one parameter ((s,tx,ty) of the transformation is determined by solving a set of second equations (2d') established for the considered pixels of the initial image. i+1 The updated value of ); S417) Determine whether a new iteration is necessary; and S418) Using the set of the first equation (1) for the considered pixel and the one or more transformation parameters ((s,tx,ty) determined in step S416) i+1 The value of ) is used to determine the updated estimate of the optical flow F for the considered pixel (u). i+1 ,v i+1 ).

14. The object motion analysis system (10) according to claim 10, wherein, The transformation is a perspective transformation, a similarity transformation (S) combined with translation (tx,ty), an affine transformation, or a rotation of a rigid object combined with translation.

15. A control system comprising the object motion analysis system (10) according to claim 10, wherein: The control system includes at least one camera (30) mounted on the vehicle (100), the at least one camera (30) being configured to acquire images (I k ,I k+1 ); The electronic control unit is configured as follows: S10) Using the at least one camera (30), acquire camera images of the environment of the at least one camera; S20) Identify an object (200) in at least one pair of camera images acquired by one of the at least one cameras; S30) Extract the initial image of the object (I k ) and the final image (I k+1 The initial image (I) k ) and the final image (I k+1 ) is the portion of the camera image pair defined by the bounding box of the identified object (200); S40) The depth relative velocity (RVD) of the object (200) relative to the vehicle (100) and the collision time (TtC) between the vehicle (100) and the object (200) are calculated using the method according to any one of claims 1 to 6. as well as S50) Control at least one device (32, 34, 36) of the control system based on the depth relative velocity (RVD) and the collision time (TtC).

Images