Method for detecting a surface including surfaces that are not easily visible, for example, transparent, semi-opaque or perforated surfaces such as grids, meshes and nets.
The optical flow detection method effectively identifies and measures distances to transparent or perforated surfaces, addressing the limitations of existing technologies by providing accurate and efficient surface detection for autonomous vehicles.
Patent Information
- Application Number
- FR2023005403
- Authority / Receiving Office
- FR · FR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing methods fail to effectively detect barely visible surfaces such as transparent or perforated surfaces like grids, fences, and nets, which pose a danger to autonomous vehicles, as they lack the capability to differentiate these surfaces from textured ones and require substantial computational resources.
A method utilizing optical flow detection, involving image acquisition, spherical optical flow vector extraction, mathematical modeling, and comparison of adjusted digital data to detect surfaces, employing iterative least squares methods and mathematical models to differentiate between single and multiple surfaces.
The method achieves reliable detection of barely visible surfaces with high accuracy and efficiency, reducing the need for operator supervision and enhancing the safety of autonomous vehicles by accurately identifying and measuring distances to these surfaces.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Title of the invention: Method for detecting a surface including surfaces that are not very visible, for example, transparent, semi-opaque or openwork surfaces such as grids, wire mesh and nets.
[0001] The invention relates to a device and a method for detecting a barely visible surface, such as an openwork surface, in particular of the grid, wire mesh or net type, or a transparent surface, such as a glass surface. The invention also relates to the detection of such a barely visible surface, in particular when this surface is arranged between a device and a textured surface such as a wall or the ground. The invention finds application in all fields, in particular that of autonomous vehicles, marine, underwater, land and / or air vehicles having a camera and / or an optical sensor.
[0002] In the context of avoidance procedures or in a potential collision situation, it is essential for a device, such as an aerial, terrestrial, marine and / or underwater vehicle, to be able to detect surfaces or obstacles and to recognize its positioning with respect to these surfaces or obstacles as well as the relative distances between this device and these surfaces or obstacles, in particular in the context of autonomous navigation.
[0003] Indeed, in the case of a vehicle, when moving indoors or outdoors, certain barely visible surfaces may represent a danger to the integrity of the vehicle. As an indicative example, when an aerial vehicle, such as a drone, flies indoors or outdoors, surfaces such as safety nets, grilles, wire mesh or windows represent a threat that is difficult to identify for most conventional visual detection methods.
[0004] Conventional computer vision and mapping tools fail to detect safety nets, for example. Thus, the moving device cannot take into account the less visible surfaces that are in its path. An operator must therefore anticipate the presence of such obstacles in the environment to avoid collisions or supervise the movement of the device in real time so as to detect any openwork or transparent surfaces in time. Therefore, it is costly in terms of resources and even difficult for the operator to supervise the movement of each device, even those capable of moving autonomously.
[0005] Indeed, visual navigation methods are based on the presence of textured surfaces and a transparent or openwork surface does not have the same characteristics as a textured surface. Some studies, such as “Lidar-based glass detection for improved occupancy grid mapping” by H. Tibebu, J. Roche, V. De Silva, and A. Kondoz (Sensors, vol. 21, no. 7, p. 2263, 2021), “Mapping with reflection detection and utilization of reflection in 3D LIDAR scans ” by X. Zhao, Z. Yang, and S. Schwertfeger (in IEEE International Symposium on Safety, Security, and Rescue Robotics, 2020, pp. 27-33), “Cartographer glass: 2D graph SLAM framework using LIDAR for glass environments” by L. Weerakoon, GS Herr, J. Blunt, M. Yu, and N. Chopra (arXiv:2212.08633, 2022) or “Window detection from mobile LIDAR data” by R. Wang, J. Bach, and FP Ferrie (Workshop on Applications of Computer Vision, IEEE, 2011, pp. 58-65), propose navigation through a Lidar sensor while other documents focus on shape recognition, such as “Window detection in façades ” by H. Ali, C. Seifert, N. Jindal, L. Paletta, and G. Paar (International Conference on Image Analysis and Processing, IEEE, 2007, pp.837-842), on the combined use of several sensors of different nature, as explained in the document “Multi-sensor fusion glass detection for robot navigation and mapping” by H. Wei, X.-e. Li, Y. Shi, B. You, and Y. Xu (WRC Symposium on Advanced Robotics and Automation, IEEE, 2018, pp. 184-188), on image segmentation or direct analysis as presented in the papers “Windows detection using k-means in cielab color space” by M. Recky and F. Leberl (20th International Conference on Pattern Recognition, IEEE, 2010, pp. 356-359) or “Don't hit me! glass detection in real-world scenes” by H. Mei, X. Yang, Y. Wang, Y. Liu, S. He, Q. Zhang, X. Wei, and R.W. Lau (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 3687-3696).
[0006] It is known from the paper “Deep learning of monocular depth, optical flow and ego-motion with geometry guidance for uav navigation in dynamic environments” by F. Mumuni, A. Mumuni, and CK Amuzuvi (Machine Learning with Applications, vol. 10, p. 100416, 2022) to measure optical flow, and extract visual information, by Machine Learning methods to facilitate navigation. However, these methods are limited by the data used to train the neural networks and require substantial computing capacity.
[0007] Thus, none of the documents cited above proposes a viable solution for the detection of surfaces, in particular for transparent and / or openwork surfaces and which can be adapted to a moving device, and particularly an autonomous vehicle.
[0008] The invention aims to overcome all or part of the problems mentioned above by proposing a method for detecting surfaces, in particular surfaces that are not very visible such as openwork surfaces comprising grids, grilles, or nets as well as transparent surfaces such as windows.
[0009] The detection method according to the invention also makes it possible to differentiate such a surface placed between another textured surface and the moving device, the method being based on the detection of optical flow. The invention also relates to a device for detecting any surface, visible or barely visible.
[0010] To this end, the invention relates to a method for detecting at least one surface by an optical sensor moving relative to the at least one surface, the detection method comprising the following steps: - Image acquisition using the optical sensor, - Extraction of a set of spherical optical flow vectors from the acquired images, and further comprising for each optical flow vector of the set of optical flow vectors the following steps: - Calculation of a magnitude data of the spherical optical flow vector, the magnitude data of the spherical optical flow vector being a physical data of the optical flow vector, - Modeling of a digital magnitude data of the optical flow vector by means of a first mathematical model, - Adjustment of the modeled digital magnitude data relative to the magnitude data of the spherical optical flow vector, the first model being a function of the amplitude of the optical flow - Determination of a difference between the calculated spherical optical flow vector magnitude data and the estimated digital magnitude data, the detection method then comprising the following steps: - Grouping the adjusted digital quantity data into a first grouping and a second grouping, the difference between each adjusted digital quantity data and the respective calculated spherical optical flow vector quantity data being positive for the first grouping, and negative for the second grouping, - Modeling of the digital magnitude data of the first grouping and the second grouping according to a second mathematical model, the second mathematical model being a function of an amplitude Aa, of the optical flow, - Comparison of the amplitude of the optical flow between the first grouping and the second grouping, - Detection of at least one surface based on the result of the comparison.
[0011] According to one aspect of the invention, the detection method comprises a step of iteration, for each grouping among the first grouping and the second grouping, - the adjustment of the digital magnitude data modeled according to the second mathematical model with respect to the magnitude data of the spherical optical flow vector, for each optical flow vector of the set of optical flow vectors, - determining a difference between the digital quantity data according to the second mathematical model and the calculated quantity data, for each optical flow vector of the set of optical flow vectors, - grouping according to two groups of data of distinct magnitude according to the sign of the difference, - modeling of the digital magnitude data of each of the two groups according to a predefined mathematical model, - comparison of the amplitude of the optical flow of the two groups of data of distinct quantities, - and detection of at least one surface based on the result of the comparison.
[0012] According to one aspect of the invention, the detection method comprises a step of filtering the digital quantity data following the adjustment step.
[0013] According to one aspect of the invention, the extraction step comprises a step of extracting a set of planar optical flow vectors and a step of backprojecting the set of planar optical flow vectors following the step of extracting the set of planar optical flow vectors.
[0014] According to one aspect of the invention, the first mathematical model is defined according to the following function:
[0015] ^)1 = AMcos^k*6)
[0016] Where / ^represents the amplitude of the optical flow, k represents a correction factor of the movement of the optical sensor relative to the at least one surface and a'i represents a magnitude of the optical flow at an instant i of the movement and at an angle 6 relative to the optical sensor.
[0017] According to one aspect of the invention, the step of adjusting the digital quantity data comprises an approximation step according to the iterative least squares method.
[0018] According to one aspect of the invention, the adjustment step comprises an additional step of canceling the adjustment if the iterative least squares method does not converge after a predefined number of steps or if the difference between the calculated spherical optical flow vector magnitude data and the estimated numerical magnitude data is greater than ten percent of the value of the calculated spherical optical flow vector magnitude data.
[0019] According to one aspect of the invention, the detection method comprises, following the adjustment step, a step of selecting a magnitude data zone, the data zone being defined according to the following equation: GJ(Aa£os3(k*Q) ) < < 1.3( A^os^k*^ )
[0020]
[0021]
[0022]
[0023]
[0024]
[0025]
[0026]
[0027]
[0028]
[0029]
[0030]
[0031] According to one aspect of the invention, the first mathematical model is defined according to the following function: H«)| = Where / ^ represents the amplitude of the spherical optical flow, represents a magnitude of the optical flow at an instant i and at an angle 6 relative to the optical sensor, m represents a coefficient depending on the angle of incidence between the optical sensor and the at least one surface and the orientation deviation between the at least one surface and the normal of the at least one surface and o represents a correction factor for the movement of the optical sensor relative to the at least one surface. According to one aspect of the invention, the step of adjusting the digital quantity data comprises an approximation step according to the direct least squares method. According to one aspect of the invention, the step of detecting at least one surface comprises an additional step of determining at least two surfaces if the difference resulting from the comparison of the first grouping and the second grouping is included in a determination interval, the determination interval being between [0.05; 0.5]. According to one aspect of the invention, the detection method comprises a step of estimating a distance between the optical sensor and the at least one surface, the distance between the optical sensor and the at least one surface being defined according to the following function: D — Uwall ~ Where V represents the speed of the optical sensor relative to the at least one observed surface and Aa!waU represents an amplitude value of the optical flow of the at least one observed surface. The invention also relates to an optical flow sensor configured to detect an optical flow, the optical flow sensor comprising a device for detecting at least one surface, the detection device being configured to implement the steps of the detection method having at least one of the characteristics previously described. The invention will be better understood and other advantages will appear on reading the detailed description of an embodiment given by way of example, a description illustrated by the attached drawing in which: [Fig.l] [Fig.l] represents a schematic view of a method for detecting a surface; [Fig.2] [Fig.2] represents a schematic view of a variant of the method for detecting a surface;
[0032] [Fig.3A] [Fig.3A] represents a representative graph of a detection of a single surface by means of the detection method;
[0033] [Fig.3B] [Fig.3B] represents a representative graph of a detection of two superimposed surfaces by means of the detection method;
[0034] [Fig.4] [Fig.4] represents a schematic view of a device for detecting a inconspicuous surface placed between the detection device and another surface.
[0035] For the sake of clarity, the same elements will bear the same references in the different figures.
[0036] [Fig.l] represents a schematic view of a method 1 for detecting at least one surface, and in particular at least one barely visible surface, from an optical sensor. To do this, the detection method 1 is based on the measurement of an optical flow and therefore requires that the optical sensor, included in the device, moves relative to this at least one surface. Furthermore, the detection method 1 is also capable of detecting such a barely visible surface positioned between the optical sensor and any other surface, for example a textured surface, so that a superposition of the surfaces is observable from the point of view of the optical sensor.
[0037] For the purposes of the invention, a barely visible surface can be defined as a) a surface having a certain transparency. According to a preferred configuration, the barely visible surface is a transparent surface and has a certain visual disparity in its transparency with the presence of visual contrasts, for example the presence of impurities, inclusions, dust or defects on the surface. This surface can also consist of b) an openwork surface such as a net, a grid, a wire mesh.
[0038] For the purposes of the invention, a textured surface may be a surface which, unlike the barely visible surface, has a certain texture. In other words, the textured surface has a certain opacity compared to the barely visible surface. As an indicative example, the textured surface is an opaque surface or a surface having significant visual disparities.
[0039] Optical flow is defined as the apparent movement of objects, surfaces, and contours in a visual scene, caused by the relative movement between an observer, the device, and the observed scene.
[0040] The method according to the invention makes it possible to differentiate several surfaces that are not very visible when they are superimposed without being in contact.
[0041] The method of the invention also makes it possible to identify these less visible surfaces being positioned in front of a textured surface. It is not necessary for the textured surface to be flat.
[0042] The detection method 1 comprises a first step 10 of acquiring images by means of an optical sensor. During the acquisition step 10, the optical sensor takes image captures or photographs at a predefined frequency. Specifically, image captures or photographs can be taken at an acquisition period of between 10 milliseconds and 100 milliseconds.
[0043] Following the acquisition of the images during step 10 by the optical sensor, a second step makes it possible to extract a set of spherical optical flow vectors from the acquired images or photographs. More precisely, it is possible to define points on an acquired image and to follow the displacement of these points on the images successively acquired by the optical sensor. Therefore, an optical flow vector can be defined as the displacement of a targeted point between two successively acquired images. During step 11, the detection method 1 makes it possible to extract from the acquired images a set of spherical optical flow vectors, that is to say a set of spherical optical flow vectors for example in an area of interest of the image. As an indicative example, three aligned spherical optical flow vectors can be extracted per acquired image.This set of spherical optical flow vectors can take the form of a line or a diffuse field. To do this, different methods are known. According to a preferred configuration, it can be considered to use, during the extraction step 11, the LUCAS KANADE algorithm, as described in the document “An iterative image registration technique with an application to stereo vision” by BD Lucas and T. Kanade (IJCAI'81: 7th international joint conference on Artificial intelligence, vol. 2, 1981, pp. 674-679), in conjunction with a corner detector according to the Shi-Thomasi method, presented in the document “Good features to track” by J. Jianbo and C. Tomasi (IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1994, pp. 593-600) to extract the Optical Flow.The method selects certain points on an image that are defined as corners, and attempts to track them in the next image, thus forming spherical optical flow vectors. This process is highly configurable: it is possible to define the number of searched points, their quality, and the number of filter levels, which refine the final measurement of the resulting optical flow, but more searched points require more computing power.
[0044] From then on, a resulting optical flow is extracted, this resulting optical flow is defined according to a set of spherical optical flow vectors between two successive images.
[0045] Furthermore, it may also be envisaged to extract a field of spherical optical flow vectors in order to improve the measurement of the resulting optical flow.
[0046] Then follows, for each spherical optical flow vector of the set of vectors, a step 12 of calculating a magnitude data item of the spherical optical flow vector. The magnitude data item of the spherical optical flow vector is a physical data item of the optical flow vector. Indeed, the optical flow vector can be defined according to its direction and its norm. However, the magnitude data is the norm of the spherical optical flow vector or only an axial component of the spherical optical flow vector or a combination of several components of the spherical optical flow vector. The magnitude data of the vector is therefore a physical data of the optical flow vector. Physical data means data extracted directly from the optical flow vector by physical measurement during the extraction step 11. In addition, the definition of the magnitude data, according to the norm of the vector or according to at least one component of the vector depends solely on a desire to simplify the future numerical modeling of this data as much as possible. Indeed, choosing only one component makes it possible to define only a translational movement and therefore to eliminate any other movement by this hypothesis, simplifying the numerical calculation.Conversely, defining the magnitude data as the norm of the vector makes it possible to take into account any movement of the optical sensor relative to at least one observed surface but increases the complexity of the numerical calculations.
[0047] Preferably, the magnitude data of the spherical optical flow vector is the norm of the vector.
[0048] In other words, for each corner or point analyzed, an optical flow vector can be obtained and a magnitude data of the vector can be estimated. And all of these spherical optical flow vectors are grouped to form a spherical optical flow.
[0049] A set of spherical optical flow vectors can then be obtained and, for each vector, the magnitude data can be obtained. It is then possible to digitally transcribe these optical flow vectors, and to use a digital expression of the optical flow coinciding with the spherical optical flow vectors.
[0050] The detection method 1 comprises, for each optical flow vector of the set, a modeling step 13 by means of a first mathematical model. This mathematical model is a function of an amplitude A(„ of the optical flow. The digital optical flow vector magnitude data is thus mathematically modeled so as to be equivalent to each spherical optical flow vector magnitude data obtained during the calculation step 12.
[0051] The modeling step 13 makes it possible to obtain a digital representation of the spherical optical flow and in particular of the magnitude data of the spherical optical flow for each observed point or corner.
[0052] More precisely, the resulting spherical optical flow can be represented numerically thanks to its measured spherical optical flow magnitude data a>(( 0), with knowledge of the corner considered as well as the movement of the optical sensor between the two image acquisitions.
[0053] Each magnitude data of the optical flow vector can then be calculated according to different methods.
[0054] According to one embodiment, when the optical sensor moves parallel to the at least one surface.
[0055] In other words, according to the first configuration, the optical sensor moves in translation, parallel to the plane which is analyzed, namely the at least one surface. The modeling step 13 then makes it possible to estimate the spherical optical flow measured in the direction of movement of the optical sensor.
[0056] The measured spherical optical flow can be defined according to the formula introduced by Koenderink, JJ, van Doom, AJ Facts on optic flow. Biol. Cybem.56, 247-254, 1987: 100571 =
[0058] By defining the observation angle (relative to the optical axis of the optical sensor, the magnitude data of the measured spherical optical flux can then be defined as:
[0059] |w ; (0) | =
[0060] 1^(0)^=^^^(1^^
[0061] |w.(0) | = ^cos\e)
[0062] Where D represents the distance from the orthogonal between the previously detected corner and the optical sensor. In other words, D represents the distance between the optical sensor and the at least one surface as a function of the defined theoretical flux.
[0063] Where V represents the measurable velocity component relative to the movement of the resulting optical flow between two successively acquired images in the direction parallel to the at least one surface.
[0064] Where 0 represents the angle between the optical sensor and the point considered, namely the observed corner.
[0065] Where ad represents the distance between the optical sensor and the at least one observed surface.
[0066] Where j represents the pointing direction of the optical sensor.
[0067] In the presence of a translational movement parallel to the at least one surface to be detected, the value of the resulting optical flow can be approximated to the curve of the magnitude of the spherical optical flow measured by a function of the type A^OS3^ with Aa, and k being parameters of the model, A(l. represents an amplitude value of the spherical optical flow measured at the corner considered, k is a coefficient value introduced to compensate for the approximations. The addition of this correctional parameter makes it possible to reduce the modeling error without significantly increasing the time of treatment.
[0068] Given the measurement deviations such as translational movement of the optical sensor relative to the surface to be detected, the model advantageously incorporates a corrective factor k.
[0069] The corrective factor k is between a value equal to 0 and a maximum value equal to 2.4. Preferably, the corrective factor k is between 1.1 and 1.33 and ideally, the corrective factor k is equal to 1.19.
[0070] For information purposes, according to the first embodiment and by defining the magnitude data of the optical flow vector as the component of the optical flow vector along the direction of movement of the device carrying the optical sensor, the modeling step 13 may comprise a calculation step 131, represented in [Fig.2], of the numerical magnitude data, preferably the norm of the optical flow vector. According to this first configuration of parallel movement relative to the at least one surface. The component of the optical flow vector along the direction of movement of the device carrying the optical sensor, that is to say, the magnitude data of the optical flow vector is then calculated according to the following function: [0071 ] | | - AMcas? ( k*3 )
[0072] Where Aw represents an amplitude value of the spherical optical flow measured at the corner considered and k represents the corrective factor of the movement of the optical sensor relative to the at least one surface at an instant i during the movement and at an angle 9 relative to the optical sensor.
[0073] In other words, the first mathematical model is defined according to the following function:
[0074] | 0) | = A„;cos^ (k*0)
[0075] Alternatively, when the movement is only in translation relative to the at least one surface and has an angle of incidence between the optical sensor and the at least one surface, the magnitude of the optical flow vector can be calculated, during step 131, according to the following formula:
[0076] । \_
[0077] Where Am represents the amplitude of the measured spherical optical flow, represents a value of the optical flow at an instant i and at an angle θ relative to the optical sensor, m represents a coefficient depending on the angle of incidence between the optical sensor and the at least one surface and the orientation deviation between the at least one surface and the normal of the at least one surface and o represents a correction factor for the movement of the optical sensor relative to the at least one surface.
[0078] Therefore, the first mathematical model is defined according to the following function:
[0079] > / ûAÎ . / r
[0080] This formula thus has the advantage of better representing, in a numerical manner, the magnitude of the resulting optical flow observed previously but requires greater computational resources. Indeed, in the presence of any angle of incidence between the optical sensor and the at least one surface, i.e. a biased viewing angle, the use of a cosine type function is generally not very functional. Indeed, the angle of incidence adds complexity to the model.
[0081] This function also has the advantage of tending towards zero when the iteration increases. Thus, this function follows the theory according to which the optical flow evolves inversely proportional to the distance as stated with the formula of the Koenderink and van Doom formula, and as it is described according to the formula stated above.
[0082] In other words, the step 131 of calculating the optical flow makes it possible to approximate the digital magnitude data of the optical flow, namely the estimated digital magnitude of the optical flow, to the value of the magnitude of the vector of the resulting spherical optical flow analyzed previously.
[0083] Furthermore, any type of numerical model that allows obtaining a “bell-shaped” function with a relatively flat top can be considered. As an indicative example, a trigonometric function based on the sine can be considered, as well as a polynomial function or a Gaussian function.
[0084] As an indicative example, the first mathematical model can be a polynomial function of the type:
[0085] | <4(9)1 = a O4 + b O3 + c O2 + d 0 + e
[0086] where a,b,c,d,e represent parameters of the polynomial.
[0087] Then follows, for each optical flow vector of the set of optical flow vectors, a step 14 of adjusting the estimated digital optical flow vector magnitude data relative to the calculated spherical optical flow vector magnitude data.
[0088] During the adjustment step 14, the aim is to approximate the digital quantity data model to the value of each optical flow vector quantity calculated during the calculation step 12. To do this, the adjustment step 14 uses the least squares method. Indeed, the least squares method optimizes the curve of the optical flow quantity to minimize the sum of the square of the errors on each approximated point, namely each corner.
[0089] According to a preferred aspect, the adjustment step 14 comprises an approximation step 142 according to the iterative least squares method with a limit on the number of iterations. In other words, the adjustment step 14 may comprise an additional step of canceling the adjustment 144 if the iterative least squares method does not converge after a predefined number of iteration steps. As an indicative example, this predefined number of iteration steps may be between 3000 and 8000 and is preferably set to 5000 iterations.
[0090] Alternatively, it may be envisaged that the adjustment step 14 comprises an approximation step 142 according to the direct least squares method. This method has the advantage of being more directive than the iterative least squares method but has the disadvantage of being more expensive in terms of computing power. In addition, the direct least squares method also has the disadvantage of sometimes not converging.
[0091] Alternatively, any numerical optimization method minimizing the residuals between the measured optical flow quantities and the modeled numerical quantities can be considered. As an indicative example, the descending gradient method can also be considered.
[0092] Following the adjustment step 14, the detection method 1 comprises, for each spherical optical flow vector, a step of determining an estimation error 15 between the calculated magnitude data of the spherical optical flow vector and the adjusted digital magnitude data. More precisely, this involves determining the estimation error between the sum of the errors between the calculated magnitude data of the spherical optical flow vector and the adjusted digital magnitude data. The step of determining the estimation error 15 thus makes it possible to determine the estimation error or the distance between the magnitude of the measured spherical optical flow vector and the value of the digital magnitude according to the first mathematical model at the observed point.
[0093] According to a preferred embodiment of the invention, the adjustment of the digital magnitude data can also be eliminated if the difference 15 between the calculated spherical optical flow vector magnitude data and the estimated digital magnitude data is greater than a threshold of ten percent of the value of the calculated spherical optical flow vector magnitude data.
[0094] Following the step of determining the deviation 15, the detection method 1 comprises a step of grouping 16 the adjusted digital quantity data into a first grouping and a second grouping. The deviation between each adjusted digital quantity data and the respective calculated spherical optical flow vector quantity data is positive for the first grouping and negative for the second grouping.
[0095] The grouping step 16 thus makes it possible to separate all of the optical flow digital magnitude data into two groups or clusters, namely the first grouping and the second grouping.
[0096] In other words, during the grouping step 16, it is a matter of generating the hypothesis that at least two surfaces are superimposed in front of the optical sensor and that therefore the digital distribution is sufficiently large to observe at least two surfaces. In other words, the values of magnitudes of spherical vectors are relatively far from each other.
[0097] Following this grouping 16, the detection method 1 comprises a new estimation step carried out on the digital magnitude data of the first grouping and of the second grouping 17 according to a second mathematical model, the second mathematical model being a function of an amplitude AM of the optical flow. It is then a question, during step 18, of estimating, in a manner similar to the estimation step 13, the digital magnitude data of the first grouping according to the second mathematical model and the digital magnitude data of the second grouping according to the second mathematical model. From then on, the digital magnitude data of the first grouping is then calculated according to the following second mathematical model:
[0098] Where represents the magnitude of the optical flow of the first group of optical flows, represents an amplitude value of the spherical optical flow measured at the corner considered for the first grouping and kl represents a first corrective coefficient of the movement of the optical sensor relative to the at least one surface at an instant i and at an angle 0 relative to the optical sensor.
[0099] And, the digital magnitude data of the second grouping is calculated according to the following second mathematical model: ~ A2acos3(k2*6)
[0100] Where represents the numerical magnitude of the optical flow of the second grouping, A2 represents an amplitude value of the spherical optical flow measured at the corner considered for the second grouping and k2 represents a second corrective coefficient of the movement of the optical sensor relative to the at least one surface at an instant i and at an angle 9 relative to the optical sensor.
[0101] Preferably, the first corrective coefficient kl is equal to the second corrective coefficient k2.
[0102] It should be noted that the second mathematical model may be equivalent to the first mathematical model, as indicated above. However, any other mathematical model may be considered as, for example, a second mathematical model based on the following function:
[0103] i / n\| _ 4
[0104] It is then necessary to establish, for each surface whose hypothesis is established, that is to say two surfaces, the coefficient m as a function of the angle of incidence between the optical sensor and the at least one surface and the orientation difference between the at least one surface and the normal of the at least one surface and the correction factor a of the movement of the optical sensor relative to the at least one surface.
[0105] Thus, preferably, the second mathematical model is identical to the first mathematical model. However, it may also be envisaged that the second mathematical model is different from the first mathematical model or even another mathematical model of the same "type", i.e. bell-shaped for example. The term "bell-shaped" defines any function whose graphical representation is a curve whose summit is relatively flat.
[0106] This is followed by a comparison step 18 of the amplitude Aœ of the optical flow between the first grouping and the second grouping. This involves verifying the hypothesis stated previously during the grouping step 16, namely whether at least two surfaces are identifiable. More precisely, this involves comparing the amplitude of the optical flow according to the second digital model for the first grouping with the amplitude of the optical flow according to the second digital model for the second grouping.
[0107] If there is little or no difference, i.e. less than a threshold that can be defined, between the value of the amplitude of the optical flow of the first grouping A{ and the value of the amplitude of the optical flow of the second grouping A2 then the distribution is centered, which reflects the presence of a single surface facing the optical sensor. In other words, the optical flow of the first grouping is identical to the optical flow of the second grouping and there is in reality only one surface facing the optical sensor, invalidating the basic hypothesis.
[0108] There is then a detection 19 of a surface.
[0109] Conversely, if a gap is observable then it is possible to state that either there is no surface facing the optical sensor, or at least two superimposed surfaces are facing the optical sensor.
[0110] In order to be able to know if at least two superimposed surfaces are facing the optical sensor, the comparison step 18 may comprise an additional step of determining at least two surfaces 182. More precisely, highlighting a difference between the value of the amplitude of the optical flow of the first grouping Aj, and the value of the amplitude of the optical flow of the second grouping A2 may sometimes not be sufficient to identify whether it is a situation with at least two surfaces or no surfaces. Indeed, noise may be included in the first grouping or in the second grouping so that a small difference may be observed between the value of the amplitude of the optical flow of the first grouping Aj and the value of the amplitude of the optical flow of the second grouping A2.
[0111] The determination step 182 induces an additional condition, if the comparison of the amplitude A^, of the optical flow between the first grouping and the second grouping is included in a determination interval, then two surfaces are detected.
[0112] The determination interval is defined as being an interval of percentage difference between the value of the amplitude of the optical flow of the first grouping Aj and the value of the amplitude of the optical flow of the second grouping A2. According to one embodiment of the invention, this percentage difference is between [0.05; 0.5]. In other words, if there is more than five percent difference between the value of the amplitude of the optical flow of the first grouping Aj and the value of the amplitude of the optical flow of the second grouping A2 and less than fifty percent difference between the value of the amplitude of the optical flow of the first grouping A} and the value of the amplitude of the optical flow of the second grouping A2 then there is determination of at least two surfaces.
[0113] According to a preferred variant, the determination interval is included in an interval of [0.1;0.4].
[0114] Conversely, if there is less than a five percent difference between the value of the amplitude of the optical flow of the first grouping A^ and the value of the amplitude of the optical flow of the second grouping A2, that is to say that the difference is small, then it is a centered distribution with the presence of noise explaining the small phase shift between the first group of optical flow and the second group of optical flow.
[0115] The detection method 1 then only distinguishes one surface.
[0116] And, if the value of the amplitude of the optical flow of the first grouping Aj and the value of the amplitude of the optical flow of the second grouping A2 are far from each other, then the distribution is heterogeneous even with application of the first model and the second model.
[0117] This then results in an absence of surface or excessively noisy digital quantity data, thus preventing the detection of planes facing the optical sensor; the detection step 19 then does not distinguish any surface facing the optical sensor.
[0118] Furthermore, if the comparison step 18 highlights the fact that more than one surface is detected, i.e. when the distribution is not sufficiently centered to validate the presence of a single surface, it is possible to envisage, by means of the detection method 1, a reiteration of the detection process on each of the two groupings, namely the first grouping and the second grouping. In other words, if more than one surface is detected in step 19, i.e. at least two superimposed surfaces, the detection method 1 can reiterate, respectively for the first grouping and for the second grouping the steps of adjusting 14 the digital quantity data of the second mathematical model with respect to the calculated quantities, determining a difference 15 between the digital quantities according to the second mathematical model and the calculated quantity data, grouping 16 according to two distinct groups of quantity data according to the difference, modeling 17 the digital quantity data according to another mathematical model, comparing 18 the digital quantity data according to this other mathematical model of the two distinct groups of quantity data and detecting 19 at least one surface for each of the two distinct groups of data. And this iterative loop of steps 14 to 19 for each new group of digital data makes it possible to determine the number of surfaces facing the optical sensor.
[0119] In other words, the detection method 1 comprises, following the detection step 19, an iteration step, for each grouping among the first grouping and the second grouping, - adjustment 14 of the digital magnitude data modeled according to the second mathematical model with respect to the magnitude data of the spherical optical flow vector, for each optical flow vector of the set of optical flow vectors, - determining a difference 15 between the digital magnitude data according to the second mathematical model and the calculated magnitude data, for each optical flow vector of the set of optical flow vectors, - grouping 16 according to two groups of data of distinct magnitude according to the sign of the difference, - modeling 17 of the digital quantity data of each of the two groups according to a predefined mathematical model, - comparison 18 of the amplitude of the optical flow of the two groups of data of distinct quantities, - and detection 19 of at least one surface depending on the result of the comparison.
[0120] It may be envisaged, optionally, to process, according to a pre-processing step 10', as shown in [Fig.2], the optical data acquired during the acquisition step 10, in order to extract a central band on each image of 40 to 80 pixels wide. Ideally, the central band is approximately 60 pixels wide. This pre-processing step 10' thus makes it possible to reduce the computational cost in the detection method 1.
[0121] Further, extracting the set of optical flow vectors in step extraction 11 must allow the extraction of a three-dimensional optical flow. Thus, the optical flow sensor can advantageously be a camera, and preferably a “fish eye” type camera or a spherical optical flow sensor.
[0122] Alternatively, the extraction step 11 may comprise a step of extracting a set of planar optical flow vectors 112 by means of a conventional flat retina camera for example, and a step of backprojection 114 of the set of planar optical flow vectors following the step of extracting the set of planar optical flow vectors 112.
[0123] Indeed, the optical sensor can be defined to operate only in two dimensions. From then on, there is extraction of a set of vectors or a vector field.
[0124] And, at this back-projection step 114, the measurement of the optical flow is planar, since it is taken from a two-dimensional image. However, the most useful representation of the resulting optical flow is the spherical representation. The only difference then is the fact that the resulting planar optical flow is projected onto a sphere, in order to obtain a resulting spherical optical flow measurable in radians per second. To do this, a matrix expression of the back-projection, taking into account the intrinsic parameters of the optical sensor such as the pixel size, the focal length or even the resolution, can be applied.This matrix expression of the backprojection is applied to all the planar optical flow vectors detected during the extraction step 112 to thus transform a two-dimensional planar representation of the planar optical flow vectors and therefore of the extracted resulting optical flow into a three-dimensional spherical representation of the spherical optical flow vectors and therefore of the extracted resulting optical flow. Any conventional backprojection matrix expression can be considered. Nevertheless, details on the computation of the backprojection matrix expression can be given in the paper “A general approach for egomotion estimation with omnidirectional images” by R. Vassallo, J. Santos-Victor, and H. Schneebeli (Proceedings of the IEEE Workshop on Omnidirectional Vision 2002. Held in conjunction with ECCV'02, 2002, pp. 97-103 as well as in the book “Robotics, Vision and Control” by P. Corke (Springer, 2017).
[0125] The backprojection step thus allows the calculation of a backprojection matrix at each observed point or corner, thus making it possible to switch from a planar view to a spherical view of the optical flow and of each of the extracted optical flow vectors.
[0126] The detection method 1 may comprise, following the adjustment step 14, a step of removing extreme points 146 from the digital magnitude data of the optical flow vector. In other words, the step of removing the extreme points 146 allows a removal of the aberrant points in the first mathematical model. More precisely, the step of removing the extreme points 146 allows a selection of a band or area of value around the curve defining the digital magnitude data of the optical flow vector by removing any digital data that deviates too far from the corresponding detected resulting optical flow value.
[0127] In other words, during the step of deleting the extreme points 146, it is a matter of deleting from the first mathematical model the magnitude data judged to be too far from the expected result, that is to say the extreme points, reflecting for example erroneous local measurements or problems linked to the optical sensor.
[0128] As an indicative example, this value zone or band may be defined as a band of values between seventy percent and one hundred and thirty percent of the value of the adjusted numerical magnitude of the spherical optical flow. In other words, the step of removing the extreme points 16 may comprise a step of selecting 146, represented in [Fig.2], a magnitude zone, the magnitude zone being defined according to the following optical flow equation: 0.7(A^cos^k^e) ) < {<4^)1 < l.3(A <j;cos3(k*8) )
[0129] Alternatively, the value area or band around the curve may be defined as a band of values between ninety percent and one hundred and ten percent of the value of the refined spherical measured optical flow magnitude.
[0130] In other words, the zone of accepted quantities is then defined according to the following optical flow equation:
[0131] 0.9 (A(ücos^k*e)) <)
[0132] The selection step 148 thus allows filtering of the digital quantities of the vector.
[0133] In addition, it may be envisaged to calculate the distance separating the optical sensor from the at least one surface, if this at least one surface is detected during step 19.
[0134] In a situation where only one surface is identified, it is possible to measure the distance separating the optical sensor from said surface facing it. To do this, the detection method 1 may comprise a step 20 of estimating a distance Dwau between the optical sensor and said detected surface. Indeed, from the formula stated previously, it is possible to link the distance between the optical sensor and the detected surface and the value of the magnitude of the optical flow vector calculated:
[0135] *cos3 (8) = A„jcos3 (k)
[0136] The distance Dwau between the optical sensor and the detected surface is then defined according to the following function:
[0137] J)
[0138] Where V represents the speed of the optical sensor relative to the detected surface and represents an amplitude value of the optical flow, which is equal to the value of the amplitude of the optical flow of the first grouping Aj or even to the value of the amplitude of the optical flow of the second grouping A2^since the distribution is centered.
[0139] In a situation where at least two surfaces are detected, the value of the amplitude of the optical flow which has the lowest value thus defines the greatest distance separating the detected surface from the at least two detected surfaces of the optical sensor. And, the value of the amplitude of the optical flow which has the greatest value thus defines the smallest distance separating the detected surface from the at least two detected surfaces of the optical sensor.
[0140] Advantageously, if two surfaces have been detected during step 19, such as for example an openwork surface located in front of a textured surface, the detection method 1 can allow the estimation of a distance Dnet between the optical sensor and the openwork surface and a distance Dwall between the optical sensor and the textured surface.
[0141] The distance Dnet between the optical sensor and the openwork surface is then defined according to the following function:
[0142] Dnet = ^
[0143] Where V represents the speed of the optical sensor relative to the detected openwork surface and At,^ represents the largest value of the amplitude of the optical flow between the value of the amplitude of the optical flow of the first grouping A| or the value of the amplitude of the optical flow of the second grouping A2
[0144] And, the distance DwaU between the optical sensor and the textured surface is defined according to the following function:
[0145] / ) ,. = -1- ^wall A^
[0146] Where V represents the speed of the optical sensor relative to the detected openwork surface and Aw,ra„ represents the smallest value of the amplitude of the optical flow between the value of the amplitude of the optical flow of the first grouping Aj or the value of the amplitude of the optical flow of the second grouping A2.
[0147] The detection method 1 was applied as indicated previously according to the first configuration. Two case studies were set up, a first situation with an optical sensor placed opposite a textured surface. And, a second situation setting up the same optical sensor and the same textured surface with, between the optical sensor and the textured surface, an openwork surface such as a net.
[0148] [Fig.3A] thus represents the digital result of the detection of a single surface by means of the detection method 1 while [Fig.3B] represents the digital result of the detection of two surfaces, namely the textured surface and the openwork surface, superimposed by means of the detection method 1.
[0149] Thus, it is observable in [Fig.3A] a distribution of the digital quantities of optical flows, which are defined according to this indicative example as the absolute values of the component following the direction of movement of the optical sensor, more centered, demonstrating the presence of a single surface allowing the projection of optical flow in the direction of the optical sensor, namely the textured surface alone, whereas in [Fig.3B], the distribution of the digital quantities of optical flow is much more scattered and shows the presence, by the use of the detection method 1, of two surfaces generating optical flows, namely the textured surface and the superimposed openwork surface.
[0150] It should be noted that each surface detection by means of the detection method 1 results from measurements taken with a single pair of two successive images. Nevertheless, during a passage or a trajectory several detections or acquisitions can be accumulated so as to obtain several times an estimate of digital quantities of optical flows and refine the detection by increasing the number of samples.
[0151] Thus, other experiments were also implemented according to the same parameters as stated previously. For each of these experiments, the number of images per trajectory and the number of usable images per trajectory was also indicated: Vol 1 2 3 4 5 6 7 8 9 Case AAAAAABBB Image 30 30 30 30 30 30 20 20 20 Usable Image 25 30 26 28 25 28 18 20 16 Number of Detections 0 3 2 3 1 1 17 20 16 Percentage of Correct Estimate 100 90 93 89 96 96 94 100 100 Determination of Case AAAAAABBB Flight 10 11 12 13 14 15 Case BBBBBB Image 30 30 30 30 30 30 Usable Image 28 27 29 28 26 26 Number of detections 28 27 29 27 26 26 Percentage of correct estimation 100 100 100 97 100 100 Determination of Case BBBBBB
[0152] Case A represents a situation with an optical sensor arranged opposite a textured surface while case B represents a situation with the same optical sensor and the same textured surface with, between the optical sensor and the textured surface, a net.
[0153] Thus, these results highlight the fact that the detection method 1 has a reliability of the order of 90% by taking only two successive images and still allows detection of a net if several pairs of images are integrated during a pass. The detection method 1 detects the net with a rate of absence of detection, less than 5% and also has an erroneous detection rate, less than 10%. again with only two images used. The use of several detections allows to further increase the precision of the detection method 1 and of the device going to 100%
[0154] The invention also relates to an optical flow sensor 100, shown in [Fig.4], configured to capture images in the direction of a textured surface 110. The optical flow sensor 100 comprises a detection device 120 of an openwork surface 130 between the optical flow sensor and the textured surface 110. The detection device 120 is thus configured to implement the steps of the detection method 1.
[0155] In other words, the detection device 120 comprises: - Image acquisition module by the optical flow sensor 100. The acquisition module makes it possible to implement the acquisition step 10. The acquisition module can also implement the pre-processing step 10' if necessary. - A module for extracting a set of spherical optical flow vectors from the acquired images. - A module for calculating a magnitude data of the 12 vector of spherical optical flow. - A module for adjusting the estimated digital quantity data in relation to the quantity data of the spherical optical flow vector. - A module for determining a difference between the calculated spherical optical flow vector magnitude data and the magnitude data estimated digital. - A module for grouping the adjusted digital quantity data into a first grouping and a second grouping, the difference between each adjusted digital quantity data and the respective calculated spherical optical flow vector quantity data being positive for the first grouping, the difference between each adjusted digital quantity data and the respective calculated spherical optical flow vector quantity data being negative for the second grouping. - A module for estimating the digital magnitude data of the first grouping and the second grouping according to a second mathematical model, the second mathematical model being a function of an amplitude of the optical flow. - And a module for comparing the amplitude Aœ of the optical flow between the first grouping and the second grouping and for detecting at least one surface based on the result of the comparison.
[0156] The detection device may also comprise a processor or an electronic card for analyzing the magnitude data of each optical flow vector to enable real-time in-flight detections.
[0157] Finally, the detection device 120 also comprises a module for estimating the distance between the optical sensor and the at least one surface detected according to step 20.
[0158] The detection of a non-opaque or semi-transparent surface such as a net or a window, by an optical sensor mounted on a device such as a drone for example, is thus made possible thanks to the detection method 1. The nature of the optical sensor can be arbitrary. As an indicative example, the optical sensor can be a low-cost “global shutter” camera. The detection method 1 thus has the advantage of allowing more efficient and safer movement for the device, without an operator needing to first check for the presence of such obstacles in the area. The detection method 1 can also be added to an avoidance algorithm. Furthermore, with a confidence of more than 90% in the direct measurements and nearly 100% in the estimation of the global trajectory, any semi-transparent surface is detected in a few steps with a limited number of images.
Claims
1. Claims Method for detecting (1) at least one surface (110, 130) by optical sensor (120) moving relative to the at least one surface (110, 130), the detection method (1) comprising the following steps: - Acquisition (10) of images by means of the optical sensor (120), - Extraction (11) of a set of spherical optical flow vectors from the acquired images, and further comprising for each optical flow vector of the set of optical flow vectors the following steps: - Calculation of a magnitude data of the spherical optical flow vector (12), the magnitude data of the spherical optical flow vector being a physical data of the optical flow vector, - Modeling of a digital magnitude data of the optical flow vector (13) by means of a first mathematical model, - Adjustment (14) of the modeled digital magnitude data relative to the magnitude data of the spherical optical flow vector, the first model being a function of the amplitude of the optical flow - Determination of a difference (15) between the calculated spherical optical flow vector magnitude data and the estimated digital magnitude data, the detection method (1) then comprising the following steps: - Grouping (16) the adjusted digital quantity data into a first grouping and a second grouping, the difference between each adjusted digital quantity data and the respective calculated spherical optical flow vector quantity data being positive for the first grouping, and negative for the second grouping, - Modeling of the digital magnitude data of the first grouping and the second grouping (17) according to a second mathematical model, the second model ma- thematic being a function of an amplitude Aw of the optical flow, - Comparison (18) of the amplitude Aw of the optical flow between the first grouping and the second grouping, - Detection of the at least one surface (19) as a function of the result of the comparison.
2. Detection method (1) according to claim 1, comprising a step of iteration, for each grouping among the first grouping and the second grouping, - of adjusting (14) the digital quantity data modeled according to the second mathematical model with respect to the quantity data of the spherical optical flow vector, for each optical flow vector of the set of optical flow vectors, - of determining a difference (15) between the digital quantity data according to the second mathematical model and the calculated quantity data, for each optical flow vector of the set of optical flow vectors, - of grouping (16) according to two distinct groups of quantity data according to the sign of the difference, - of modeling (17) the digital quantity data of each of the two groups according to a predefined mathematical model,- comparison (18) of the amplitude Au of the optical flow of the two groups of data of distinct quantities, - and detection (19) of at least one surface according to the result of the comparison.,
3. Detection method (1) according to claim 1 or 2, comprising a step of filtering (142) the digital quantity data following the adjustment step (14).
4. Detection method (1) according to one of claims 1 to 3, wherein the extraction step (11) comprises a step of extracting a set of planar optical flow vectors (112) and a step of backprojecting (114) the set of planar optical flow vectors following the step of extracting the set of optical flow vectors planar (112).
5. Detection method (1) according to one of claims 1 to 4, in which the first mathematical model is defined according to the following function: Where Aw represents the amplitude of the optical flow, k represents a correction factor of the movement of the optical sensor relative to the at least one surface and represents a magnitude of the optical flow at an instant i and at an angle 9 relative to the optical sensor.
6. Detection method (1) according to one of claims 1 to 5, in which the step of adjusting (14) the digital quantity data comprises an approximation step (142) according to the iterative least squares method.
7. The detection method (1) according to claim 6, wherein the adjusting step (14) comprises an additional step of canceling (144) the adjustment if the iterative least squares method does not converge after a predefined number of steps or if the deviation (15) between the calculated spherical optical flow vector magnitude data and the estimated numerical magnitude data is greater than ten percent of the value of the calculated spherical optical flow vector magnitude data.
8. Detection method (1) according to claim 3, comprising, following the adjustment step (14), a step of selecting (146) a magnitude data area, the data area being defined according to the following equation: 0.7(A^vs\k*0) ) < |w^)| < l.3(AOJcos3(k*G) )
9. Detection method (1) according to one of claims 1 to 4, in which the first mathematical model is defined according to the following function: Iw^l - Where A^ represents the amplitude of the spherical optical flow, represents a magnitude of the optical flow at an instant i and at an angle 6 relative to the optical sensor, m represents a coefficient depending on the angle of incidence between the optical sensor and the at least one surface and the deviation in orientation between the at least one surface and the normal of the at least one surface and o represents a correction factor for the movement of the optical sensor relative to the at least one surface.
10. Detection method (1) according to one of claims 1 to 5, in wherein the adjustment step (14) of the digital quantity data comprises an approximation step according to the direct least squares method.
11. Detection method (1) according to claim 10, wherein the step of detecting at least one surface (19) comprises an additional step of determining at least two surfaces (182) if the difference resulting from the comparison (18) of the first grouping and the second grouping is included in a determination interval, the determination interval being between [0.05; 0.5].
12. Detection method (1) according to one of the preceding claims, comprising a step of estimating (20) a distance DwaU between the optical sensor (120) and the at least one surface (110, 130), the distance Dwall between the optical sensor (120) and the at least one surface (110, 130) being defined according to the following function: Dwall = Where V represents the speed of the optical sensor relative to the at least one observed surface (110, 130) and represents an amplitude value of the optical flow of the at least one observed surface (110, 130).
13. Optical flow sensor (100) configured to detect an optical flow, the optical flow sensor (100) comprising a detection device (120) of at least one surface (110, 130), the detection device (120) being configured to implement the steps of the detection method (1) according to any one of the preceding claims.
14. Optical flow sensor (100) according to the preceding claim, wherein the optical flow sensor is a camera, preferably a fisheye type camera.
Citation Information
Patent Citations
Translational Optical Flow Sensor
US20090225300A1
Cited By
Method and system for measuring multiple parameters of quartz pendulous reed
CN121297669A
