METHOD FOR DETECTING OBJECTS IN A THREE-DIMENSIONAL SCENE

Description

[0001] The invention relates to the field of detection and recognition of objects in a scene observed by a three-dimensional image sensor.

[0002] The invention relates specifically to a method for recognizing objects in a scene observed in three dimensions, the method comprising a segmentation of the elements of the scene then a matching of the segments with predefined geometric models.

[0003] The invention applies in the field of computer vision and automatic analysis of 3D scenes for any industrial application that requires the automatic detection and recognition of objects in a scene captured in three dimensions. Possible applications of the invention include in particular industrial robotics, autonomous vehicles, intelligent buildings, video surveillance, video protection but also applications related to virtual reality.

[0004] A general technical problem that the invention seeks to solve concerns the automatic and real-time semantic analysis of a scene captured by a sensor or a three-dimensional camera. More precisely, an objective of the invention consists, from a set of 3D points captured by a camera filming a 3D scene, in modeling the scene, decomposing it into different elements corresponding to the elements of the scene and in characterizing each element semantically and geometrically.

[0005] A more specific technical problem concerns the ability to recognize objects in a three-dimensional scene when these objects are partially occluded. The occlusion of an object can come from another part of the same object or from another object. Part of an object can also be occluded because it is outside the field of view of the camera or 3D sensor.

[0006] The literature contains numerous studies on the recognition of three-dimensional objects. Examples include, but are not limited to, the solutions described in patents or patent applications EP 2549434, US 20160154999, US 9396545, US 20150288947 and US 8885925.

[0007] A general disadvantage of these methods is that they do not offer an effective solution for performing fine recognition of the elements of a 3D scene when some of these elements are partially occluded.

[0008] There are several levels of detail in recognizing 3D objects in a scene. For example, for vehicle class recognition in a road scene, four or more levels of detail can be defined. The first level of recognition concerns the size of a vehicle: two-wheelers, light vehicles, intermediate vehicles, heavy goods vehicles and coaches with two axles, heavy goods vehicles and coaches with three axles or more. The second level of recognition concerns the type of vehicle: bicycle, motorcycle, car, van, truck, bus, heavy goods vehicle. The third level of recognition concerns the vehicle category: road, sports, cruiser or scooter for motorcycles, urban, city, compact, sedan, SUV, Pick-up for cars. The fourth level of recognition concerns the vehicle make and model.

[0009] While prior art 3D object recognition methods are limited to coarse recognition levels, the invention allows recognition to be performed with all levels of fineness.

[0010] Furthermore, it is known that a single shot of a scene, i.e. from a single point of view, is not sufficient to fully model a scene or a subject. Indeed, due to problems of occultation, self-occultation or limitation of the camera's field of vision, it is generally necessary to take several shots of the same scene in order to be able to completely model the scene and perform precise recognition of the elements that compose it.

[0011] Prior art methods generally use multiple acquisitions of the same scene from different viewpoints to produce multiple associated point clouds. The collected point clouds are then grouped and transformed into a single coordinate system to reconstruct the scene from all viewpoints. This technique, known in particular as "3D scanning pipeline", is described in reference [1].

[0012] The invention makes it possible to eliminate the drawbacks of the 3D object recognition methods according to the prior art by proposing a new method based on three main steps. The image acquired by a three-dimensional sensor is first segmented in order to detect the different elements of the scene. Then, a reduced number of candidate poses is determined from geometric characteristics of the detected segments in order to limit the number of operations to be carried out and thus to remain compatible with a real-time application. Finally, a search for the optimal pose among the candidate poses and an optimal geometric model is carried out for each detected segment, taking into account any partial occlusions of the segment.To achieve this result, each candidate geometric model is embedded in the three-dimensional scene according to the candidate pose and a comparison with the detected segment on the one hand and the other segments of the scene on the other hand is carried out.

[0013] The invention thus makes it possible to fully model a scene from a single 3D acquisition of the scene made by a 3D scanner from a single point of view. A 3D scanner measures the positions in a coordinate system of a sampling of 3D points of the surface of a scene. The coordinates of the collected 3D points are generally sparse, that is to say that the surfaces of the elements of the scene are represented in a discrete manner. The invention proposes to match the discrete point cloud provided by the 3D scanner during an observation of a scene and continuous geometric models (generally surface) predefined in advance. To get as close as possible to a continuous observation of the scene, the invention implements a particular technique of embedding the geometric models in the 3D scene. Virtual 3D points are calculated on these models, very close to the 3D points actually acquired by the 3D scanner.In other words, the invention uses the acquisition parameters of the 3D scanner to select on the geometric model the 3D points closest to an object in the scene. The advantage of this technique is to optimize the choice of 3D points on the models for a better matching with the 3D points acquired by the 3D scanner.

[0014] The subject of the invention is a method for recognizing objects in a three-dimensional image, according to claim 1.

[0015] Other characteristics and advantages of the present invention will appear better on reading the description which follows in relation to the appended drawings which represent: There figure 1 , a flowchart detailing the steps of implementing the method for recognizing objects in a 3D scene according to the invention, The figures 2a And 2b, two illustrations of the result of a support element segmentation step applied to an image of a 3D scene, The figures 3a And 3b , two illustrations of the result of a step of segmentation of structuring elements applied to an image of a 3D scene, The figures 4a à 4d , four illustrations, according to four different points of view, of a structuring segment associated with a bounding box, the segment comprising a dominant plane, The figures 5a à 5p , sixteen illustrations of the embedding of a geometric model of the car class in a 3D scene according to sixteen different candidate poses, The figures 6a à 6d , four illustrations, from four different points of view, of the different sets of points used to calculate a matching score between a geometric model and a segment, The figures 7a And 7b, two illustrations, respectively in right view and in top view, of the results of the complete reconstruction of a scene in which each segment is reconstructed by an optimal geometric model embedded according to an optimal pose, The figure 8 , a diagram of a 3D object recognition system according to the invention.

[0016] There figure 1 schematizes, on a flowchart, the main steps of implementing the method of recognizing objects in a 3D scene according to the invention.

[0017] A first step 101 of the method consists of acquiring one or more 3D images capturing a scene observed in three dimensions.

[0018] This acquisition can be carried out using any 3D sensor or 3D camera or equivalent device making it possible to acquire a point cloud representing an observed scene, each point being defined by three spatial coordinates. There are in particular three types of 3D sensors making it possible to carry out the first step 101 of the invention.

[0019] A first example of a 3D sensor is a LIDAR (Light Detection and Ranging) sensor, which is a type of active 3D scanner based on laser remote sensing. To acquire a 3D observation of a scene, the LIDAR sensor emits a laser beam that scans the entire scene. 3D acquisition consists of measuring distances based on the analysis of the properties of light beams reflected by objects in the scene towards the LIDAR sensor. Generally, the distance between the sensor and the surface of the object studied is measured by counting the time required for the round trip of the reflected laser beam pulse. LIDAR sensors have the capacity to process between 10,000 and 100,000 points per second. The data produced by the LIDAR sensor acquisition is a 3D point cloud positioning each of the sampled points relative to the 3D scanner.

[0020] A second example of a 3D sensor is a time-of-flight (or TOF) camera. This type of 3D sensor provides a depth image where each pixel corresponds to the distance between a point in the scene and a specific point.

[0021] A third example of a 3D sensor is a stereoscopic scanner, generally composed of a set of cameras or a pair composed of a camera and a projector, associated with specific processing such as disparity calculations.

[0022] In the following, the 3D acquisition device will be referred to interchangeably as a 3D sensor, 3D scanner, or 3D camera.

[0023] In the first step 101 of the method which is the subject of the invention, the scene is observed from a 3D sensor which produces a 3D point cloud. A 3D point cloud, P = { pi = (xi ,yi , zi ); a ≤ i < N}, is a series of N data each composed of three coordinates p i = (xi , yi , zi ) positioning each of the sampled points relative to the 3D sensor. If the sensor is positioned at the origin of the reference frame, each 3D point can be expressed by the spherical coordinates p i = (ri , ϕ i , θ i ). ri represents the distance from the sensor to point pi . ϕ i and θ j are the elevation and azimuth angles, formed between the line going from the origin of the 3D sensor to point pi and two planes passing through the origin, and being respectively one horizontal and the other vertical. These spherical coordinates make it possible to locate in space each of the points in relation to the sensor, work preliminary and necessary for 3D modeling. Each 3D point analyzed p i is associated with a ray ray i , expressed by spherical angles ray i = (ϕ i , θ i ) or by the three normalized coordinates ray i = (xi / ri , yi / ri , zi / ri ). The ray designates the half-line which has as its origin the position of the 3D sensor and which passes through the point pi . In the case of a LIDAR type sensor, it is the laser beam which scans the scene by passing successively through the different points pi on the surface of the elements on which the laser beam is reflected. In the following, Ray = {ray i ; 0 ≤ i < N} is the set of N rays launched by the 3D sensor and which are associated with the N respective points pi. Without departing from the scope of the invention, each 3D point pi can be expressed indifferently by its Cartesian or spherical coordinates. The rays Ray can be directly produced by the 3D sensor (in particular when it is a LIDAR type sensor) or can be calculated from the coordinates of the points pi and the position of the sensor or the distance between the sensor and the point Pi.

[0024] The method according to the invention also receives as input a set 102 of geometric models associated with object classes. The design of these geometric models is not part of the invention. The geometric models are predetermined and stored in memory, for example in a database, to be used by the invention. The models may depend on the intended application and may correspond to different types of geometric representations. For example, the geometric models 102 may have elementary two-dimensional shapes such as a plane, a disk, a square, a rectangle, a polygon or even a rhombus. They may also have elementary three-dimensional shapes such as a sphere, a cube, a box, a cylinder, a cone. The geometric models may be complex models composed of several elementary models.They can take the form of polygonal meshes, implicit, parametric or even quadric surfaces.

[0025] A geometric model can be a single geometric representation or a composite of multiple geometric representations of different types. An object class can be associated with multiple geometric models. For example, the car class can be associated with multiple polygon meshes representing different makes and models of cars.

[0026] The choice of geometric models depends on the nature of the scene observed and the possible elements that characterize it. For example, models representing elements of a road scene can be: a 2D polygon for the ground (road or sidewalk), a cylinder for poles, a 3D box for walls and buildings, a polygonal mesh for vehicles, people and animals, a model composed of an elementary 2D shape and a 3D box for road signs.

[0027] According to a particular embodiment of the invention, to better optimize the matching between a segment and a geometric model, the geometric model is normalized. Indeed, a 3D model is, a priori, designed in an arbitrary reference frame and scale. The goal of normalization is to define a natural reference frame and a real size intrinsic to the 3D geometric model. The normalization process is broken down, for example, into three phases: a first phase of alignment or normalization of the orientation which consists of orienting the model in its natural pose according to human perception. This first phase can be carried out by transforming the model into a reference frame using the product of two matrices: the rotation matrix fixed by the three alignment axes given in a precise order and the reflection matrix fixed by the directions of these three axes, a second phase of centering or normalization of the position of the model which consists of positioning the model in relation to the origin of the reference frame. In other words, it involves moving a particular center of the geometric model to the origin of its reference frame, a third phase of scaling or normalization of the size of the model which consists of sizing the model according to its real size.

[0028] This normalization process can be performed manually, semi-automatically or fully automatically as illustrated, for example, in the reference publication [2].

[0029] This normalization process can be illustrated with a concrete example of a geometric model consisting of a polygonal mesh corresponding to a car.

[0030] By applying the first alignment phase, we determine the three ordered alignment axes and their directions. The z-axis is the normal of the axis of the rear axle of the car which is the normal of the plane of symmetry of the vehicle, the direction of this axis being from the passenger side to the driver side. The y-axis is the normal of the axis connecting the midpoints of the two axles, the direction of this axis being from the rear of the car to the front of the car. The z-axis is the vector product of the x-axis and the y-axis, the direction of this axis being from the bottom of the car to the top of the car.

[0031] By applying the second centering phase, we determine the center (xc, yc, zc) of the model, where (xc, yc) are the first two coordinates of the 3D point, intersection of the axis of the rear axle and the axis connecting the midpoints of the two axles and zc is given by the low plane circumscribed to the four wheels.

[0032] By applying the third scaling phase, we determine the scale factors (sx, sy, sz) which are given by the ratios between the three dimensions of the box enclosing the model and the three real dimensions of the car along the x, y and z axes.

[0033] From the set of 3D points acquired by the 3D sensor and the predefined geometric models, the method according to the invention is broken down into three main phases: a first phase of 3D segmentation of the scene, a second phase of calculating the candidate 3D poses for each segment and a third phase of calculating an optimal geometric model and an optimal pose for each segment.

[0034] Thus, an objective of the invention is to determine, for each structuring segment of the scene, an optimal geometric model, as well as the optimal transformation allowing the geometric model to be positioned in the same pose as the segment. In this way, the invention allows recognition of the objects in the scene, from a single 3D shot, including detection of the objects and classification of the detected objects by classes of objects associated with the geometric models.

[0035] We now describe an example of carrying out step 103 of segmentation of the 3D scene observed and captured by the set of 3D points acquired in step 101.

[0036] 3D segmentation consists of associating an identifier with each group of 3D points representing a geometric coherence. It is generally done by aggregation according to a spatial proximity of 3D points or according to a similarity of descriptors calculated on the 3D points. 3D segmentation does not allow the semantic classification of 3D points but only the detection of elements present in the scene.

[0037] Filters can be applied to the point cloud before 3D segmentation to remove noise that may be of natural origin such as rain, dust, glare or noise related to sensors or noise related to acquisition processing.

[0038] A 3D scene is often composed of two types of elements: support elements and structural elements. Support elements support structural elements, and structural elements rest on support elements.

[0039] For example, when the scene observed is a road scene, the supporting elements are the ground, in particular roads, sidewalks or cycle paths, and the structuring elements are the infrastructure, i.e. buildings, bridges, low walls, but also vehicles, pedestrians, trees or poles.

[0040] In another example, when the observed scene concerns a building, a house or a room, the supporting elements are the solid structures which form the building and delimit the rooms such as floors, walls, partitions and ceilings, and the structuring elements are the doors, windows, furniture, tables, chairs, cupboards and any object resting or being fixed on a supporting element.

[0041] In yet another example, if the scene observed is a scene composed of different objects placed on a table or on a shelf, in this case, the support element is the tray(s), and the structuring elements are the objects placed on the support elements.

[0042] The structuring elements of the scene are connected to each other by support elements. In other words, a structuring element is always connected, placed on or connected to a support element.

[0043] The segmentation step 103 of the 3D scene exploits the distinction between support elements and structuring elements to segment these two types of elements separately, in order to facilitate the overall segmentation of the scene.

[0044] Supporting elements are most often represented by planes or polygons, and structural elements can be represented by more complex geometric shapes. However, structural elements are always linked to a supporting element.

[0045] For example, in the case of a road scene, the vehicles are geometrically linked by the road.

[0046] If we can identify the 3D points associated with the road and filter them from the 3D point cloud, it is then easier to segment the vehicles as related instances.

[0047] Thus, according to a particular embodiment of the invention, the segmentation step 103 is implemented in three successive sub-steps: a first sub-step of segmentation of the support elements of the scene, a second sub-step of deletion of the 3D points belonging to detected support elements then a third sub-step of segmentation of the other elements of the scene: the structuring elements.

[0048] The first sub-step of segmentation of the support elements of the scene is, for example, implemented using the algorithm described in reference [3].

[0049] The third sub-step of segmentation of the structuring elements of the scene is, for example, implemented by a technique called 3D clustering as described in reference [4].

[0050] In a particular embodiment of the invention, the segmentation step 103 can begin, beforehand, with a phase of filtering the noise impacting the 3D points.

[0051] In another particular embodiment of the invention, the segmentation step 103 may also comprise a step of refining the segmentation obtained.

[0052] At the output of the segmentation step 103, a segmentation mask is obtained making it possible to identify the support elements and the structuring elements in the observed scene.

[0053] The outputs of the segmentation step 103 include the number H of support segments detected, the number S of structuring segments detected, a correspondence vector of size S indicating for each structuring segment, the associated support segment on which it rests or with which it is linked. The segmentation step 103 further provides a segmentation mask which includes, for each 3D point, the identifier of the segment to which it belongs. The segmentation mask can also indicate whether the 3D point corresponds to noise or whether it has not been segmented.

[0054] In the special case where the scene does not include any support elements, the sub-step of segmenting the support elements can be made optional.

[0055] In another particular case, if the scene consists of a single object, then the segmentation step 103 can be made entirely optional or can be limited to a simple step of assigning an identifier to the single object identified in the scene.

[0056] THE figures 2a And 2b represent, respectively in right view and in top view, the result obtained after segmentation of the support elements of a 3D image of a road scene. These two figures represent the set of 3D points captured (according to two different points of view) and in bold, the points corresponding to the segments detected by the segmentation. On the figures 2a And 2b , only floor 200 is segmented.

[0057] THE figures 3a And 3b represent, respectively in right view and in top view, the result obtained after segmentation of the structuring elements on the same 3D image as that of the figures 2a And 2b. On these figures 3a And 3b , the cars 201 are segmented as structural elements resting on the support element constituted by the ground 200.

[0058] An example of implementing step 104 of determining candidate poses is now described. This step consists of determining a set of candidate poses to be applied to each geometric model to transform it into the 3D reference frame of the observed scene so as to make it correspond as best as possible with a segment detected in step 103.

[0059] We recall that a three-dimensional pose of an object corresponds to the point of view of this object taken by the camera. A 3D pose is defined by a transformation in space according to the six degrees of freedom, in other words by three angles of rotation and the translations according to the three axes of the reference frame.

[0060] In a first embodiment, step 104 simply consists of selecting all the poses that can be applied to each geometric model to match it to each segment.

[0061] However, this solution involves a significant number of operations subsequently because the final step of the process (which will be described in more detail later) consists, for each segment, in determining a matching criterion between the segment and each geometric model transformed according to each candidate pose. We therefore see that the higher the number of candidate poses, the greater the number of operations, which can be penalizing for applications that must respect real-time.

[0062] In a second embodiment, step 104 consists of selecting a reduced number of candidate poses based on geometric characteristics of the geometric models and / or segments.

[0063] Aligning and normalizing a geometric model involves calculating an intrinsic 3D pose for the model and bringing the model back to its actual size. As explained earlier, this normalization process can be manual, semi-automatic, or fully automatic.

[0064] Let us assume that a geometric model correctly represents a segment s and that their initial poses are different. One objective of the invention is to find the RT transformation corresponding to the six degrees of freedom to be modified (the three angles of rotation and the translations along the three axes) on the model to estimate the pose of the geometric model in the segment frame.

[0065] If there is an intrinsic pose of the segment that is the same as that of the model then the problem of optimal pose between the segment and the model is solved, size normalization not being necessary because the points of the segment reflect real data and the model is brought to its real size.

[0066] However, it is difficult to find the intrinsic pose of a segment due to problems of self-occultation, occultation by other elements of the scene or limitation of the field of vision but also due to the resolution and precision of the 3D sensor which are limited.

[0067] To solve this problem, a method is proposed to implement step 104 for selecting a reduced set of relevant 3D poses to match a segment to a geometric model. An objective of this step is to minimize the space of six degrees of freedom.

[0068] To do this, we assume that a human generally seeks to observe an object along particular axes, characterizing geometric criteria such as symmetries or principal planes. This approach allows both to find the most natural pose of an object and to align two visually similar objects in the same way. The proposed method consists of determining geometric characteristics (on the segment, on a local context to the segment or on a global context of the scene) which allow to deduce certain information allowing to limit the number of degrees of freedom for the calculation of the optimal pose of the segment. This information also takes into account the geometric model that we seek to match with the segment. They include in particular: one, two or all three coordinates of an intrinsic center of the segment, one, two or all three axes of orientation of the segment, the order of the axes, i.e., what is the x-axis? what is the y-axis? and what is the z-axis?, the directions of the axes.

[0069] Geometric characteristics calculated from the local or global context of the scene can be determined in particular when a structuring segment is linked to at least one support segment. In such a case, the pose of the support segment is linked to the pose of the structuring segment. Thus, by knowing the complete pose of the support segment or some of its degrees of freedom, it is possible to reduce the number of possible pose hypotheses for the structuring segment which is attached to the support segment.

[0070] For example, geometric features calculated from the local or global context may include: the normal of the plane of the support segment associated with the structuring segment which gives an alignment axis of the structuring segment the normal of other planes of support segments

[0071] Geometric characteristics calculated directly on a segment may include: the vertices of a 3D box encompassing the segment, the normal of a reflection symmetry plane of the segment, the direction of a local translation symmetry of the segment, the normal of the dominant plane of the segment, the principal axes resulting from a principal component analysis.

[0072] Geometric characteristics can vary from one object class to another. A preliminary task is to define geometric characteristics for each object class that we want to detect in the 3D scene. We recall that these object classes are represented by a set of geometric models that constitute an input to the method. Knowing the class of the geometric model to be matched with the structuring segment, it is more judicious to use the most relevant characteristics for this class.

[0073] Let's take the example of a point cloud representing a road scene consisting only of a car and a road, the car being on the road. After the segmentation step, the scene is decomposed into two segments, a structuring segment and a support segment. In this case, the structuring segment corresponds to the 3D points belonging to the car and the support segment associated with the structuring segment corresponds to the 3D points belonging to the road. The geometric model to be matched with the structuring segment is a polygonal mesh among a base of 3D models representing different cars.

[0074] First, we can use certain geometric characteristics of the support segment to which the structuring segment is attached. For example, by calculating the normal of the road plane, two rotation angles and a translation along the axis are deduced (three degrees of freedom less).

[0075] Then, we can determine some geometric characteristics of the structuring segment. Since we are trying to match the structuring segment and the geometric model, the geometric characteristics are predefined beforehand according to the car class. The car class has several geometric characteristics such as reflection symmetry, local translation symmetry, two principal planes (the left side and the right side), the 3D bounding box and others. For example, we know that the car wheels must necessarily be located on the ground, which eliminates some pose assumptions of the geometric model of the car class.

[0076] In this example, the dominant plane and the bounding 3D box are predefined as geometric features for the car class.

[0077] This example is illustrated on the figures 4a à 4d which represent, according to four different points of view, a segment extracted from the segmentation obtained on the figures 3a And 3b . The segment shown corresponds to a car. On the figures 4a à 4d , we have represented the bounding box 400 of the segment as well as its dominant plane 401. The dominant plane 401 is the plane of the segment which contains the most points.

[0078] As explained above, we know that the bounding box must rest on the ground, which has already been segmented as a support element. We also know that the dominant plane 401 is parallel to the ground for an element belonging to the car object class. These geometric characteristics thus make it possible to fix the three degrees of freedom in rotation as well as one degree of freedom in translation (along the axis perpendicular to the dominant plane). There therefore only remain translations along two axes in the plane parallel to the dominant plane.

[0079] This first step therefore makes it possible to limit the number of candidate poses to 16.

[0080] By calculating the normal of the dominant plane on the detected structuring segment, the third angle is deduced. At this level, the oriented z-axis and the other two directions are well identified. The rotation ambiguity then lies in the order and directions of the x and y axes.

[0081] Thus, by taking into account the geometric characteristics of the segment and the geometric model, we can reduce the number of relevant pose hypotheses to be applied to the geometric model to make it correspond to the segment.

[0082] Indeed, the number of rotation configurations can be limited to four hypotheses. As for the translation, it is reduced to a 2D translation on the (x, y) plane. Thus, the number of translation configurations can also be limited to four hypotheses by averaging the vertices of the 2D boxes of the segment and the model after projection on the (x, y) plane. For this example, sixteen candidate poses are thus retained corresponding to four rotation configurations and four translation configurations.

[0083] Generally speaking, step 104 can be implemented as follows. First, geometric characteristics are predefined in advance for each class of object that is to be detected in the 3D scene. Then, for each structuring segment, the predefined geometric characteristics are analyzed for the class of the geometric model that is to be matched with the structuring segment. This analysis is done either intrinsically on the structuring segment, or in relation to the context of the scene or to the geometric characteristics of a support segment to which the structuring segment is attached. For example, the constraints linked to the already determined orientations of the support segment make it possible to limit the number of possible poses for the structuring segment attached to this support segment.Similarly, the geometric characteristics of the object class considered for the geometric model also make it possible to limit the number of possible poses for the structuring segment because the geometric model presents certain constraints of symmetry, rotation or translation.

[0084] This analysis makes it possible to considerably reduce the pose hypotheses to be applied to the geometric model which must correspond to the structuring segment. For example, if the optimal pose of the support segment has been previously determined, a limited number of candidate poses for the associated structuring segment can be deduced.

[0085] The last phase of the invention is now described, which concerns the search for the optimal pose and the optimal geometric model to be matched with each segment detected in step 103.

[0086] This phase is performed for each detected segment, for each candidate geometric model and for each pose hypothesis among the poses retained at the end of step 104. The candidate geometric models can be preselected according to the application and the a priori knowledge of the content of the scene. For example, for an autonomous vehicle application, we know that the scene is a shot from a car, we can then limit the candidate geometric models to model classes dedicated to this type of scene.

[0087] This last phase is broken down into several steps, a step 105 of embedding the geometric model in the observed scene, according to the candidate pose and taking into account the occlusions of the segment, a step 106 of calculating a matching score between the embedded geometric model and the segment, these two steps are iterated for all the segments, all the candidate models and all the candidate poses. Finally, for each segment, we search 107 for an optimal score and we retain 108 the geometric model and the associated pose to identify and recognize the segment.

[0088] Step 105 of embedding the geometric model in the observed scene consists of applying the candidate pose to the geometric model to modify its orientation and its position in the frame linked to the camera. The candidate pose can be represented by a projection matrix RT k , m s where s is the segment index, k is the candidate pose index, and m is the geometric model index. The geometric model is denoted by a surface representation G m . The geometric model transformed to be embedded in the scene is denoted by G k , m s The operation performed in step 105 consists of applying the projection matrix to the geometric model and can be formulated using the following relation: G k , m s = RT k , m s ∗ G m

[0089] THE figures 5a à 5p represent the embedding of a geometric model from the car model class, in the segmented 3D scene on the figures 3a And 3b. The model 501 is embedded on a segment 500 according to 16 different candidate poses which were retained after taking into account the geometric characteristics of the segment, the ground (support segment) and the class of objects corresponding to the cars. The limitation of the number of candidate poses to a number equal to 16 was explained previously on the example of figures 4a-4d .

[0090] The sixteen candidate poses correspond to the different possible orientations of the model in the plane parallel to the ground and to the dominant plane of segment 500, each time aligning the geometric model 501 on one of the four upper corners of the bounding box of segment 500.

[0091] Step 106 consists of determining a matching score between the geometric model embedded in the scene in step 105 and the segment. To take into account the phenomena of occultation, self-occultation and limitation of the field of vision of the camera, the score determined in step 106 depends on one or more criteria of proximity or distance between different elements of the scene and the embedded geometric model.

[0092] For this, step 106 includes two sub-steps.

[0093] In a first sub-step, at least one set of 3D points is determined from among four sets of points characteristic of a correct matching between the embedded model and the segment.

[0094] In a second sub-step, a matching score is determined for each set of points determined in the first sub-step.

[0095] The final matching score can be taken as one of the scores calculated for one of the sets of points or as a linear combination of several scores calculated for different sets of points.

[0096] A first set of points consists of the intersection points between the embedded geometric model and the rays associated with the points of the segment. When acquiring the 3D points of the scene, the 3D sensor restores the coordinates of each 3D point as well as the associated ray. Thus, it is possible to trace the rays that start from the origin of the sensor's reference frame and pass through the 3D points. We can thus determine the intersection between these rays and the embedded geometric model. We denote this first set R k , m s .

[0097] For this first set of 3D points, the correspondence criterion between the segment and the embedded model is a proximity criterion between the first set of 3D points and the segment. Indeed, the closer the points of the first set are to the points of the segment, the more this means that the embedded geometric model is close to the segment and positioned according to an optimal pose.

[0098] The matching score calculated based on this first set is a first function f 1 which depends on the proximity between the first set of 3D points and the segment. Depending on the function used, maximization or minimization of the function occurs for an optimal match between the embedded model and the segment.

[0099] For example, to determine the first function f 1 , we first determine, for each ray, the distance between the point of the segment and the point of the first set located on this ray. The points of the first set for which this distance is less than or equal to a predetermined threshold sp are placed in a first subset M k , m s . According to a first example, the function f 1 is a constant function taken equal to the cardinal of the first subset M k , m s . According to a second example, the function f 1 is taken equal to the sum of the distances calculated above for all the rays associated with the points of the segment s. In the first example, the function reaches a maximum for an optimal correspondence between the embedded model and the segment. In the second example, the function reaches a minimum for an optimal correspondence between the embedded model and the segment. Any other function allowing to obtain the same properties can be considered.

[0100] In a particular embodiment of the invention, the points of the first set for which the calculated distance is greater than the predetermined threshold sp are placed in a second subset N k , m s This second subset corresponds to points of the model which are located on the same rays as the points of the segment but at a far distance.

[0101] Another matching score is calculated based on this second subset. This second score is a second function f 2 that depends on the distance between the second subset of 3D points and the segment. Depending on the function used, maximization or minimization of the function occurs for an optimal match between the embedded model and the segment.

[0102] According to a first example, the function f 2 is a constant function taken equal to the cardinal of the second subset N k , m s . According to a second example, the function f 2 is taken equal to the sum of the distances calculated above for all the rays associated with the points of the segment s. In the first example, the function f 2 reaches a minimum for an optimal correspondence between the embedded model and the segment. In the second example, the function reaches a minimum for an optimal correspondence between the embedded model and the segment. Any other function allowing to obtain the same properties can be considered.

[0103] A second set of 3D points consists of the points of the segment for which the associated rays do not intersect the embedded geometric model. We denote this second set V k , m s .

[0104] For this second set of 3D points, the correspondence criterion between the segment and the embedded model is a distance criterion between the second set of 3D points and the embedded geometric model. Indeed, the fewer the points in the second set and the farther they are from the points of the embedded geometric model, the closer it means that the embedded geometric model is to the segment and positioned according to an optimal pose.

[0105] The matching score calculated based on this second set is a third function f 3 that depends on the distance between the second set of 3D points and the embedded geometric model. Depending on the function used, maximization or minimization of the function occurs for an optimal match between the embedded model and the segment.

[0106] According to a first example, the third function f 3 is a constant function taken equal to the cardinal of the second set V k , m s . According to a second example, the function f 3 is taken equal to the sum of the minimum distances between each point of the second set and the set of points of the embedded geometric model. In the first example, the function f 3 reaches a minimum for an optimal correspondence between the embedded model and the segment. In the second example, the function f 3 reaches a minimum for an optimal correspondence between the embedded model and the segment. Any other function allowing to obtain the same properties can be considered.

[0107] A third set consists of the intersection points between the embedded geometric model and the rays of the points in the scene except the segment s. In other words, we are looking here for the intersection between the rays other than those used to determine the first and second sets of points and the embedded geometric model. We denote this third set by N k , m s ¯ .

[0108] For this third set of 3D points, the correspondence criterion between the segment and the embedded model is a distance criterion between the third set of 3D points and the segment. Indeed, the further the points of the third set are from the points of the segment, the more this means that the embedded geometric model is far from the points of the scene except the segment s and consequently, it is close to the segment and positioned according to an optimal pose.

[0109] The matching score calculated based on this third set is a fourth function f 4 which depends on the distance between the third set of 3D points and the segment s. Depending on the function used, maximization or minimization of the function occurs for an optimal match between the embedded model and the segment.

[0110] According to a first example, the fourth function f 4 is a constant function taken equal to the cardinal of the third set N k , m s ¯ . According to a second example, the function f 4 is taken equal to the sum of the minimum distances between each point of the third set and the set of points of the segment s. In the first example, the function f 4 reaches a minimum for an optimal correspondence between the embedded model and the segment. In the second example, the function f 4 reaches a minimum for an optimal correspondence between the embedded model and the segment. Any other function allowing to obtain the same properties can be considered.

[0111] Finally, a fourth set of points is made up of the points of the scene except those of the segment s whose rays do not intersect the embedded geometric model. We denote this fourth set M k , m s ¯ .

[0112] For this fourth set of 3D points, the correspondence criterion between the segment and the embedded model is a proximity criterion between the fourth set of 3D points and the points of the scene other than those belonging to the segment s. Indeed, the closer the points of the fourth set are to the points of the scene, the further they are from the points of the segment s and the more this means that the embedded geometric model is far from the points of the scene except for the segment s and consequently, it is close to the segment and positioned according to an optimal pose.

[0113] The matching score calculated based on this third set is a fifth function f 5 which depends on the proximity between the fourth set of 3D points and the points of the scene except the segment s. Depending on the function used, maximization or minimization of the function occurs for an optimal match between the embedded model and the segment.

[0114] According to an example, the fifth function f 5 is a constant function taken equal to the cardinal of the third set M k , m s ¯ . In this example, the function f 5 reaches a maximum for an optimal match between the embedded model and the segment.

[0115] The matching score calculated in step 106 may be equal to one of the scores calculated for one of the point sets defined above or it may be equal to a linear combination of several of these scores.

[0116] For example, the matching score can be given by the following relation: score P s G k , m s = − w 1 ∗ f 1 + w 2 ∗ f 2 + w 3 ∗ f 3 + w 4 ∗ f 4 − w 5 ∗ f 5 .

[0117] P s< denotes the set of points of the segment s.

[0118] w 1 ,w 2 ,w 3 ,w 4 ,w 5 are predetermined weighting coefficients so as to give more or less weight to each function fi depending on the confidence given to each criterion.

[0119] The weighting coefficients applied to the functions f 1 and f 5 are of opposite signs to the weighting coefficients applied to the functions f 2 , f 3 , f 4 to take into account the direction of optimization of each function. Indeed, as indicated in the examples given above, an optimal correspondence between the embedded model and the segment is obtained for a maximization of the functions f 1 and f 5 and a minimization of the functions f 2 , f 3 , f 4 .

[0120] Without departing from the scope of the invention, the values ​​and signs of the weighting coefficients are adapted according to the precise functions chosen to translate each proximity or distance criterion.

[0121] In a first example of implementation, the matching score is given by the relation: score P s G k , m s = − w 1 ∗ card M k , m s + w 2 ∗ card N k , m s + w 3 ∗ card V k , m s + w 4 ∗ card N k , m s ¯ − w 5 ∗ card M k , m s ¯ . where card() denotes the cardinal function.

[0122] In another exemplary embodiment, the matching score is given by the relation: score P s G k , m s = − w 1 ∗ ∑ q i ∈ M k , m s d 1 q i p i + w 2 ∗ ∑ q i ∈ N k , m s d 2 q i p i + w 3 ∗ ∑ q i ∈ V k , m s d 3 q i + w 4 ∗ ∑ q i ∈ N k , m s ¯ d 4 q i − w 5 ∗ ∑ q i ∈ M k , m s ¯ d 5 q i . With d 1 q i p i = 1 − r i g − r i S p d 2 q i p i = r i g − r i S p − 1 laughed and r i g are the respective distances between the origin of the reference frame and the respective points of the segment s and of the first subset M k , m s (for d 1 ) or the second subset N k , m s (for d 2 ). |, | is the L 1 norm or the L 2 norm or any other function allowing to calculate a distance between two points. d 3 ,d 4 ,d 5 correspond respectively to the functions f 3 ,f 4 , f 5 given in the second example of realization of these functions for each associated set of points.

[0123] THE figures 6a à 6d represent, from four different points of view, the determination of sets of points M k , m s , N k , m s , V k , m s , N k , m s ¯ , M k , m s ¯ , on an example of matching a geometric model of a car with a corresponding segment, the model being positioned on the segment with an optimal pose.

[0124] On the four figures 6a à 6d , the set of points M k , m s is represented by gray spheres, the set of points N k , m s is represented by black spheres, the set of points V k , m s is represented by cones, the set of points N k , m s ¯ is represented by cylinders and the set of points M k , m s ¯ is represented by cubes.

[0125] We can see, in particular on the figure 6c , appear an occultation by another object in the scene.

[0126] When all the matching scores have been calculated for all the geometric models and candidate poses, the optimal score is determined in a step 107, which may be, depending on the functions chosen, the maximum score or the minimum score among all the calculated scores.

[0127] In a step 108, the geometric model and the pose associated with the optimal score are retained, the segment s is then reconstructed and identified.

[0128] The algorithm is continued for all segments detected in segmentation step 103.

[0129] THE figures 7a And 7b represent, respectively in right view and in top view, the results of the complete reconstruction of a scene in which each segment is reconstructed by an optimal geometric model embedded according to an optimal pose.

[0130] The method according to the invention can be implemented as a computer program, the method being applied to an image I previously acquired using a 3D sensor, a 3D scanner, a lidar or a 3D camera.

[0131] The invention may be implemented as a computer program comprising instructions for its execution. The computer program may be recorded on a recording medium readable by a processor. Reference to a computer program that, when executed, performs any of the functions described above, is not limited to an application program running on a single host computer. Rather, the terms computer program and software are used herein in a general sense to refer to any type of computer code (e.g., application software, firmware, microcode, or any other form of computer instruction) that can be used to program one or more processors to implement aspects of the techniques described herein. In particular, the computing means or resources may be distributed (" Cloud computing"), possibly using peer-to-peer technologies. The software code may be executed on any suitable processor (e.g., a microprocessor) or processor core or a set of processors, whether provided in a single computing device or distributed among several computing devices (e.g., as may be accessible in the device environment). The executable code of each program enabling the programmable device to implement the processes according to the invention may be stored, for example, in the hard disk or in read-only memory. Generally, the program(s) may be loaded into one of the storage means of the device before being executed.The central unit can control and direct the execution of the instructions or portions of software code of the program(s) according to the invention, instructions which are stored in the hard disk or in the read-only memory or in the other aforementioned storage elements.

[0132] The results of the method according to the invention can be displayed on a screen or any other human-machine interface.

[0133] The invention can also be implemented by means of an object recognition system SYS, an example of which is described in figure 8 . The SYS system comprises a 3D sensor, a 3D scanner, a lidar or a 3D camera or any equivalent CAM device, for capturing an image of a three-dimensional scene. The system further comprises a BMG database in which several predefined geometric models are saved. The system also comprises a PROC processor and a MEM memory for implementing the object recognition method from the 3D points provided by the CAM scene acquisition device, and from the geometric models contained in the BMG database.

[0134] The processor may be a generic processor, a specific processor, an application-specific integrated circuit (also known as an ASIC for "Application-Specific Integrated Circuit") or an in situ programmable gate array (also known as an FPGA for "Field-Programmable Gate Array"). The system according to the invention may use, to implement the invention, one or more dedicated electronic circuits or a general-purpose circuit. The technique of the invention may be carried out on a reprogrammable computing machine (a processor or a microcontroller for example) executing a program comprising a sequence of instructions, or on a dedicated computing machine (for example a set of logic gates such as an FPGA or an ASIC, or any other hardware module).

[0135] Recognition of objects in the scene can be rendered to a user through an AFF display screen or any other human-machine interface.

[0136] Without departing from the scope of the invention, the SYS system may be implemented differently from the example provided in figure 8 For example, geometric models can be stored on a remote server with which the SYS system is able to communicate. Références

[0137] [1] : "KinectFusion: Real-time dense surface mapping and tracking", Richard A. Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J. Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, Andrew Fitzgibbon. 10th IEEE International Symposium on Mixed and Augmented Reality, 2011. [2] "Alignment of 3D models", Mohamed Chaouch, Anne Verroust-Blondet [3] : "A Fast Ground Segmentation Method for 3D Point Cloud", Phuong Chu, Seoungjae Cho, Sungdae Sim, Kiho Kwak and Kyungeun Cho. Journal of information processing systems, Volume 13, N° 3, 2017. [4] : "Segmentation Based Classification of 3D Urban Point Clouds: A Super-Voxel Based Approach with Evaluation", Ahmad Kamal Aijazi, Paul Checchin and Laurent Trassoudaine. Remote Sensing 2013.

Claims

1. A method for recognising objects in a three-dimensional image, the method being implemented by a system and comprising the following steps: • receiving (101) an image of a three-dimensional scene acquired using a 3D sensor, with the acquired image comprising a plurality of 3D points each being associated with a ray connecting the 3D point to the 3D sensor; • receiving (102) a plurality of geometric models each associated with a class of objects; • determining (103) a segmentation of the scene so as to produce a plurality of segments each corresponding to an element of the scene, with a segment being a group of 3D points representing a geometric coherence; • determining (104) at least one candidate pose for each determined segment and each geometric model; • for each segment, each geometric model and each candidate pose: - embedding (105) the geometric model in the scene according to the candidate pose; - determining at least one set of 3D points from among: a first set consisting of the intersection points between the embedded geometric model and the rays of the points of said segment, a second set consisting of the points of said segment whose rays do not intersect the embedded geometric model, a third set consisting of the intersection points between the embedded geometric model and the rays of the points of the scene apart from said segment, a fourth set consisting of the points of the scene apart from said segment whose rays do not intersect the embedded geometric model; - computing (106) a matching score between the embedded geometric model and said segment, as a function of at least one matching criterion selected from among: a proximity criterion between the first set and said segment, a distancing criterion between the second set and the embedded geometric model, a distancing criterion between the third set and said segment, a proximity criterion between the fourth set and the points of the scene apart from said segment; • determining (107) an optimal score from among the set of computed scores; • selecting (108) the geometric model and the candidate pose corresponding to the optimal score.

2. The method for recognising objects according to claim 1, wherein the first set of points comprises a first subset consisting of the points of the first set that are located at a distance that is less than or equal to a predetermined threshold of the points of said segment belonging to the same respective rays and the proximity criterion between the first set and said segment is a function of a first secondary proximity criterion selected from among the cardinal of the first subset or a first function of the sum of the distances between the points of the first set and the points of said segment belonging to the same respective rays.

3. The method for recognising objects according to claim 2, wherein the first set of points comprises a second subset consisting of the points of the first set that are located at a distance that is greater than said predetermined threshold of the points of said segment belonging to the same respective rays and the proximity criterion between the first set and said segment is a function of a second secondary distancing criterion selected from among the cardinal of the second subset or a function with a sign opposite to the first function.

4. The method for recognising objects according to any of claims 2 or 3, wherein the matching score is computed from a linear combination of at least two matching criteria wherein the weighting coefficients applied to the first secondary proximity criterion and the proximity criterion between the fourth set and the points of the scene apart from said segment have opposite signs to the weighting coefficients applied to the second secondary distancing criterion, to the distancing criterion between the second set and the embedded geometric model and to the distancing criterion between the third set and said segment.

5. The method for recognising objects according to any of the preceding claims, wherein the distancing criterion between the second set and the embedded geometric model is selected from among the cardinal of the second set or a second function of the sum of the minimum distances between each point of the second set and the embedded geometric model.

6. The method for recognising objects according to any of the preceding claims, wherein the distancing criterion between the third set and said segment is selected from among the cardinal of the third set or a third function of the sum of the minimum distances between each point of the third set and said segment.

7. The method for recognising objects according to any of the preceding claims, wherein the proximity criterion between the fourth set and said points of the scene apart from said segment is taken as equal to the cardinal of the fourth set.

8. The method for recognising objects according to any of the preceding claims, wherein the step (104) of determining at least one candidate pose for each segment and each geometric model comprises: • searching for at least one geometric feature of the geometric model; • eliminating at least one pose hypothesis incompatible with the at least one geometric feature of the geometric model.

9. The method for recognising objects according to claim 8, wherein the at least one geometric feature is selected from among a geometric symmetry feature or a geometric orientation feature.

10. The method for recognising objects according to any of the preceding claims, wherein the step (103) of determining a segmentation of the scene comprises at least the following sub-steps of: • determining a first segmentation of a first type of elements, called support elements of the scene, for example, the floors, walls, partitions and ceilings, • removing the 3D points belonging to a segment of the first segmentation, • determining a second segmentation of a second type of elements, called structuring elements of the scene, for example, the doors, windows, furniture, with each structuring element being connected to at least one support element, the support elements supporting the structuring elements, the structuring elements resting on the support elements.

11. The method for recognising objects according to claim 10, wherein the step (104) of determining at least one candidate pose for each segment comprises: • selecting at least one candidate pose for each segment associated with a support element of the scene, • eliminating, for each segment associated with a structuring element connected to said support element, at least one incompatible pose hypothesis from the at least one candidate pose selected for the segment associated with said support element.

12. The method for recognising objects according to any of the preceding claims, comprising a prior step of aligning and scaling the geometric models.

13. A recognition system (SYS) comprising a 3D sensor (CAM) for acquiring an image of a three-dimensional scene, with the acquired image comprising a plurality of 3D points each associated with a ray connecting the 3D point to the 3D sensor, the system (SYS) further comprising a device (PROC, MEM, AFF, BMG) configured to implement a method for recognising objects in a three-dimensional image according to any one of claims 1 to 12.

14. A computer program comprising program code instructions for executing the steps of the method for recognising objects in a three-dimensional image according to any one of claims 1 to 12, when said program is executed on a computer.

15. A processor-readable storage medium storing a program having instructions for executing the method for recognising objects in a three-dimensional image according to any one of claims 1 to 12, when the program is executed by a processor.