Improved method for detecting object by means of neural network

Abstract

The invention relates to an improved method for detecting an object (16) by means of a neural network which is provided at an input with at least one raw first optical flow graph, the invention relates to a method for generating a first original optical flow graph representing computerized tracking of a moving object (16) in a scene by analyzing a content difference between a first image captured by an image acquisition device (14) at a first position (P1) at an earlier time and a continuous second image captured by said image acquisition device (14) at a second position (P2) at a current time. The method comprises correcting the original optical flow graph by using self-motion estimation information of the image acquisition device (14).

Description

[Technical field]

[0001] The present invention relates to the field of detecting objects by means of neural networks and more particularly to an improved method for detecting objects by means of neural networks.

[0002] The invention is particularly designed to be implemented by a computer equipped with a vehicle, in particular a motor vehicle, which comprises a driver assistance system. [Background Technology]

[0003] As we all know, motor vehicles are equipped with driving assistance systems, usually abbreviated as ADAS, namely "Advanced Driver Assistance System".

[0004] As is known, such assistance systems comprise at least one image acquisition device which is mounted on the vehicle and which makes it possible to generate a series of images representing the vehicle's surroundings.

[0005] The image acquisition device is, for example, a Lidar (light detection and ranging), a camera or a radar.

[0006] The computer uses the captured images to assist the driver, for example by detecting "objects" such as pedestrians, stationary vehicles or any other object on the road, and by calculating, for example, the time until a collision with the detected object.

[0007] The information given by the images captured by the image acquisition device makes it possible to implement simultaneous localization and mapping (abbreviated SLAM, i.e. simultaneous localization and mapping) so as to simultaneously build and enrich a scene representing the motor vehicle's environment and also to localize the motor vehicle in the scene.

[0008] Therefore, the use of neural networks (also abbreviated CNN, i.e. Convolutional Neural Networks) is well known.

[0009] The neural network is used to perform scene perception functions and to provide information about various objects present in the vehicle environment.

[0010] Neural networks are particularly used to provide semantic, detection, motion, and position information about objects in the scene.

[0011] To this end, the neural network should be trained in a learning mode by providing it with a large number of previously labeled images at the input in order to teach the neural network to recognize objects.

[0012] Once the neural network has been trained, it can be used in detection mode to detect and recognize objects.

[0013] To this end, a previously trained neural network is fed with images, for example images captured by an image acquisition device mounted on a motor vehicle.

[0014] Therefore, it is well known to provide optical flow maps to neural networks.

[0015] Optical flow maps are used to describe and represent computerized tracking of moving objects by analyzing the content differences between consecutive video images.

[0016] The computer can locate reference frames that mark the boundaries, edges, and regions of individual still images.

[0017] Detecting their progression enables computers to track objects in time and space.

[0018] In other words, the optical flow map represents the motion of a moving area or a moving object between a first image captured by an image acquisition device at an earlier time t-1 at a first position and a continuous second image captured by the image acquisition device at a second position at a current time t.

[0019] The motion direction and motion force of a moving object are shown, for example, by a motion vector on an optical flow map.

[0020] In the field of the invention, namely driving assistance, the image acquisition device moves in unison with the motor vehicle carrying it.

[0021] The image acquisition device is attached to a camera reference system that includes a lateral axis, a vertical axis, and a longitudinal axis that extends along a main direction of movement of the vehicle from rear to front.

[0022] The movement of the image acquisition device or its own movement includes six degrees of freedom, namely lateral translation, vertical translation from top to bottom, longitudinal translation, rotation around the lateral axis (called pitch), rotation around the vertical axis (called yaw), and rotation around the longitudinal axis (called roll).

[0023] During the service life of a vehicle, the vehicle and the image acquisition device will move substantially in the main direction of travel of the vehicle on the road, ie in longitudinal translation.

[0024] Therefore, the neural network is trained with optical flow maps, most of which correspond to the movement of longitudinal translation and a few correspond to the other movements mentioned above.

[0025] Therefore, the neural network can effectively detect objects when the vehicle moves in longitudinal translation.

[0026] Conversely, the neural network is less effective at detecting objects when the vehicle's motion includes components other than longitudinal translation, such as sliding corresponding to lateral translation, speed bumps corresponding to top-to-bottom translation and corresponding to pitch, sharp turns corresponding to yaw rotations, potholes corresponding to roll rotations, or strong accelerations of the vehicle corresponding to pitch rotations.

[0027] Therefore, it is observed that the performance of a neural network is limited by the data provided to it. [Summary of the invention]

[0028] The invention aims to propose an improved method for detecting objects with the aid of a neural network, which improves the performance of the neural network, in particular under conditions corresponding to vehicle movements other than longitudinal translational movements in the main driving direction of the vehicle.

[0029] This object, as well as other objects that will become apparent from reading the following description, is achieved by means of an improved method for detecting an object by means of a neural network, said neural network being provided at an input with at least one original first optical flow map representing computerized tracking of a moving object in a scene by analyzing the content difference between a first image captured by an image acquisition device at an earlier moment in a first position and a successive second image captured by said image acquisition device at a current moment in a second position, characterized in that the method comprises at least:

[0030] a first step of estimating three translation parameters and three rotation parameters of the image acquisition device's ego-motion between said first position and said second position, said three translation parameters comprising two secondary translation parameters and one primary translation parameter along a longitudinal axis corresponding to a primary movement of the image acquisition device,

[0031] a second step, which consists in estimating a first rotation matrix making it possible to move from the first position to a first virtual correction position, and a second rotation matrix making it possible to move from the second position to a second virtual correction position, so that the two secondary translation parameters and the three rotation parameters enabling the movement from the first correction position to the second correction position are equal to zero,

[0032] a third normalization step, comprising computing a corrected second optical flow map, the corrected second optical flow map being obtained by applying the first rotation matrix and the second rotation matrix to the original first optical flow map, and the corrected second optical flow map representing a computerized tracking of a moving object between the first corrected position and the second corrected position.

[0033] The method according to the invention therefore makes it possible to obtain corrected optical flow maps which neglect parasitic movements of the image acquisition sensor, such as pitch, roll and yaw movements when the image acquisition sensor is mounted on a motor vehicle.

[0034] Other optional features according to the present invention used alone or in combination:

[0035] - the method comprises a learning step consisting in providing a plurality of corrected optical flow maps to a neural network according to the third normalization step of the method in order to train said neural network. This feature makes it possible to improve the learning performance of a neural network provided with corrected optical flow maps in this way;

[0036] - The method comprises a detection step comprising providing a plurality of corrected optical flow maps to a neural network according to the third normalization step in order to perform object detection. This feature makes it possible to improve the detection performance of a neural network provided with corrected optical flow maps in this way.

[0037] The present invention also relates to a computer configured to implement the above method.

[0038] Furthermore, the invention relates to a motor vehicle comprising a computer of the above-mentioned type and at least one image acquisition device moving in unison with the motor vehicle, the motor vehicle moving mainly along a longitudinal axis. [Drawings]

[0039] Other features and advantages of the present invention will become apparent from reading the following description with reference to the accompanying drawings, which show:

[0040] [ Figure 1 ] is a schematic plan view of a motor vehicle equipped with an image acquisition device in an earlier first position and a current second position;

[0041] [ Figure 2 ] is similar to Figure 1 Schematic diagram showing a motor vehicle in a first correction position and a second correction position according to the method of the present invention;

[0042] [ Figure 3 ] is a schematic diagram of the original first optical flow map;

[0043] [ Figure 4 ] is a schematic diagram of a second optical flow map corrected by the method according to the present invention.

[0044] In the description and claims, the terms transverse, vertical and longitudinal will be used in a non-limiting manner with reference to the transverse axis Xc, the vertical axis Yc and the longitudinal axis Zc of the camera reference system Rc indicated in the drawings, taking into account that the motor vehicle extends longitudinally and moves longitudinally forward. [Specific implementation method]

[0045] Figure 1 A motor vehicle 10 is shown equipped with a computer 12 and an image acquisition device 14 .

[0046] Consider the camera reference frame Rc, which is the reference frame attached to the image acquisition device 14 .

[0047] Consideration is also given to a nominal reference frame (not shown) corresponding to the nominal position of the image acquisition device 14 , ie, the theoretical ideal position that the image acquisition device 14 should occupy on the associated motor vehicle 10 .

[0048] The camera reference system Rc includes an axis Xc extending transversely, an axis Yc extending vertically, and an axis Zc extending longitudinally along the main direction of movement of the motor vehicle 10 from rear to front.

[0049] It should be noted that the image acquisition device 14 is movably connected to the motor vehicle 10 .

[0050] Therefore, the movement of the image acquisition device 14 or its self-movement includes six degrees of freedom, namely the degree of freedom corresponding to the lateral translation along the axis Xc, the degree of freedom corresponding to the vertical translation from top to bottom along the axis Yc, the degree of freedom corresponding to the longitudinal translation along the axis Zc, the degree of freedom around the axis Xc (which corresponds to the rotation called pitch), the degree of freedom around the axis Yc (which corresponds to the rotation called yaw), and the degree of freedom around the axis Zc (which corresponds to the rotation called roll).

[0051] The motor vehicle 10 is equipped with a driver assistance system intended to assist the driver, for example by analysing data provided by the image acquisition device 14 , in particular intended to detect “objects” such as pedestrians, stationary vehicles or any other obstacles on the road.

[0052] To this end, according to the invention, computer 12 of motor vehicle 10 implements an improved method for detecting objects by means of a neural network.

[0053] The neural network is provided with multiple optical flow maps at the input.

[0054] For the sake of clarity, a single optical flow map (referred to as “original first optical flow map C1 ”) will be used hereinafter to describe the operation of the method according to the present invention.

[0055] Figure 3The original first optical flow map C1 shown in represents computerized tracking of a moving object in a scene by analyzing the content difference between two consecutive images captured by the image acquisition device 14.

[0056] refer to Figure 1 , so that the image from which the original first optical flow map C1 can be obtained includes a first image captured by the image acquisition device 14 at an earlier moment in the first position P1, and a continuous second image captured by the image acquisition device 14 at the second position P2 at the current moment.

[0057] The method according to the invention comprises a first step of estimating the ego-motion of the image acquisition device 14 .

[0058] The first step consists in estimating three translation parameters and three rotation parameters of the ego-motion of the image acquisition device 14 between the first position P1 and the second position P2.

[0059] These three translation parameters include a primary translation parameter along the longitudinal axis Zc of the main movement of the image acquisition device 14 , and two secondary translation parameters along the vertical axis Yc and the transverse axis Xc of the camera reference system Rc.

[0060] The estimate of the ego-motion of the image acquisition device 14 can be obtained by various methods, such as the method of simultaneous localization and mapping, abbreviated as SLAM (simultaneous localization and mapping), or with the help of data provided by inertial sensors carried by the motor vehicle 10, or with the help of a geolocation system.

[0061] After the first step, the method carries out a second step consisting in estimating a transformation matrix making it possible to move from the first virtual corrected position P3 to the second virtual corrected position P4 of the image acquisition device 14 .

[0062] As in Figure 2 As can be seen in FIG. 1 , the movement from the first correction position P3 to the second correction position P4 only requires a longitudinal translation along the longitudinal axis Zc, as shown in FIG. Figure 2 As shown by the dotted line Tz in .

[0063] The longitudinal translation along the longitudinal axis Zc corresponds to the main direction of movement of the assembly formed by the motor vehicle 10 and the image acquisition device 14 .

[0064] Therefore, the second step includes estimating a first rotation matrix and a second rotation matrix, wherein the first rotation matrix makes it possible to move from the first position P1 to the first virtual correction position P3, and the second rotation matrix makes it possible to move from the second position P2 to the second virtual correction position P4, thereby enabling the two secondary translation parameters and three rotation parameters of the movement from the first correction position P3 to the second correction position P4 to be equal to zero.

[0065] After the second step, the method performs a third normalization step, which includes calculating Figure 4 The corrected second optical flow map C2 is shown, which is obtained by applying the first rotation matrix and the second rotation matrix to the original first optical flow map C1.

[0066] The corrected second optical flow map C2 represents computerized tracking of moving objects between the first corrected position P3 and the second corrected position P4 of the image acquisition device 14 .

[0067] Therefore, the corrected second optical flow map C2 has the correction Figure 3 and Figure 4 Taking advantage of the optical flow represented by the motion vectors in , the movement from the first correction position P3 to the second correction position P4 is limited to a longitudinal translation along the longitudinal axis of travel Zc of the motor vehicle 10 .

[0068] Exemplary embodiments of the method according to the invention are described below.

[0069] refer to Figure 1 , the image acquisition device 14 draws a curved trajectory moving from the first position P1 to the second position P2, which translates into a rotation of the image acquisition device 14 and the motor vehicle 10 around the vertical axis Yc.

[0070] exist Figure 1 and Figure 2 This curved trajectory is shown with a dashed axis.

[0071] During a first estimation step of the method, the ego-motion of the image acquisition device 14 is estimated.

[0072] Next, during the second step of the method, a first rotation matrix and a second rotation matrix are estimated, the first rotation matrix making it possible to move from the first position P1 to the first virtual correction position P3, and the second rotation matrix making it possible to move from the second position P2 to the second virtual correction position P4, thereby making the two secondary translation parameters and the three rotation parameters of the movement from the first correction position P3 to the second correction position P4 equal to zero.

[0073] As shown in the original first optical flow map C1 Figure 3 As can be seen in FIG. 1 , the plurality of motion vectors represent tracking of the object 16 by analyzing the content difference between a first image captured by the image acquisition device 14 at the first position P1 and a second image captured by the image acquisition device 14 at the second position P2 .

[0074] Figure 4 The corrected second optical flow map C2 computed during the third normalization step of the method is shown.

[0075] It will be noted that the movements caused by the curved trajectory of the motor vehicle 10 are compensated, this movement corresponding to the rotation of the acquisition sensor 14 about the vertical axis Yc.

[0076] This compensation translates into Figure 4 The reduction in the length of the motion vector of , that is, the suppression of the bias caused by the curved trajectory of the motor vehicle 10.

[0077] It should be understood that the method according to the invention is designed to correct the bias caused by any movement other than longitudinal translation along the axis Zc, such as sliding corresponding to lateral translation along the axis Xc, speed bumps corresponding to translation from top to bottom along the axis Yc and / or rotations about the axis Xc, sharp turns corresponding to yaw rotations about the axis Yc, potholes corresponding to roll rotations about the axis Zc, or strong accelerations of the vehicle corresponding to pitch rotations about the axis Xc.

[0078] The method according to the invention can be used in an object detection mode and in a learning mode.

[0079] In the detection mode, the method comprises a detection step, which comprises providing a plurality of corrected optical flow maps to a neural network according to a third normalization step.

[0080] The detection mode makes it possible to improve the performance of the neural network in detecting objects in many situations, especially in the pitch, roll, yaw and other situations mentioned above.

[0081] In the learning mode, the method comprises a learning step, which includes providing a plurality of corrected optical flow maps according to the third normalization step to the neural network in order to train the neural network.

[0082] The learning mode makes it possible to improve the performance of a neural network trained by the method according to the invention.

[0083] Of course, the present invention has been described in the foregoing text by way of example. It should be appreciated that those skilled in the art can produce various variant embodiments of the present invention without thereby departing from the scope of the present invention.

[0084] For example, it is possible to compensate and correct for significant tilt of the image acquisition device 14 in certain elevated vehicles (such as trucks), or to correct for mounting errors of the image acquisition device 14 relative to its theoretical nominal position.

[0085] To this end, after the second step of the method, a suitable transformation is applied between the camera reference system Rc, which is a reference system attached to the image acquisition device 14 , and a nominal reference system corresponding to the nominal position of the image acquisition device 14 .

[0086] Such a transformation consists, for example, in applying an additional rotation matrix to the original first optical flow map C1 during a third step, in order to correct for mounting errors of the image acquisition device 14 relative to its theoretical nominal position.

Claims

1. An improved method for detecting an object (16) by means of a neural network, said neural network being provided at an input with at least one original first optical flow map (C1) representing computerized tracking of a moving object (16) in a scene by analyzing the content difference between a first image captured by an image acquisition device (14) at an earlier moment in a first position (P1) and a successive second image captured by said image acquisition device (14) at a current moment in a second position (P2), characterized in that The method at least comprises: a first step of estimating three translation parameters and three rotation parameters of the self-motion of the image acquisition device (14) between the first position (P1) and the second position (P2), the three translation parameters comprising two secondary translation parameters and one primary translation parameter along the longitudinal axis (Zc) corresponding to the primary movement of the image acquisition device (14), a second step, which consists in estimating a first rotation matrix and a second rotation matrix, the first rotation matrix makes it possible to move from the first position (P1) to a first virtual correction position (P3), the second rotation matrix makes it possible to move from the second position (P2) to a second virtual correction position (P4), so that the two secondary translation parameters and the three rotation parameters enabling the movement from the first correction position (P3) to the second correction position (P4) are equal to zero, a third normalization step, which comprises calculating a corrected second optical flow map (C2), the corrected second optical flow map being obtained by applying the first rotation matrix and the second rotation matrix to the original first optical flow map (C1), And the corrected second optical flow map represents computerized tracking of the moving object (16) between the first corrected position (P3) and the second corrected position (P4).

2. The improved method for detecting an object (16) by means of a neural network according to claim 1, characterized in that The method comprises a learning step, which comprises providing a plurality of corrected optical flow maps (C2) to a neural network according to the third normalization step of the method, in order to train said neural network.

3. The improved method for detecting an object (16) by means of a neural network as claimed in any one of the preceding claims, characterized in that The method comprises a detection step, which comprises providing a plurality of corrected optical flow maps (C2) to a neural network according to the third normalization step in order to perform object detection.

4. A computer (12) configured to implement the method as claimed in any one of the preceding claims.

5. A motor vehicle (10) comprising a computer (12) as claimed in claim 4 and at least one image acquisition device (14) moving in unison with the motor vehicle (10), the motor vehicle (10) moving mainly along a longitudinal axis (Zc).

Images