OBJECT RECOGNITION METHOD, IMAGE RECOGNITION DEVICE, COMPUTER PROGRAM, AND MEMORY UNIT

Abstract

The invention relates to a method (10) for object recognition (12) based on measurement data (14) of at least one point-based sensor (16) capturing the object (12), whereby the measurement data (14) based on a point cloud (20) with a plurality of points (22) and attributive features (24) is first processed in a first processing step (26) on a point basis and with at least one processing plane (32) by implementing input features (28) of the point cloud (20) as learned features (30) and by obtaining at least information (50) about the relations between the points (22), and the learned features (30) are subsequently transferred in a second processing step (70) on a grid basis and with at least one processing plane (68) to a model grid (74) with a plurality of grid cells (72), after which output data (80) related to the cells are generated. Furthermore, the invention relates to an image recognition device (104), a computer program, and a memory unit.

Description

[Technical Field]

[0001] The present invention relates to an object recognition method according to claim 1. Furthermore the invention relates to an image recognition device, a computer program and a memory unit. [Background technology]

[0002] DE 10 2020206 990 A1 describes a method for processing sensor measurement data, which transfers the measurement data of a first sensor in a first encoder and the measurement data of a second sensor in a second encoder into respective latent spaces, from which features are derived by a first decoder the reconstructed measurement data of the first sensor and by a second decoder the reconstructed measurement data of the second sensor. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] German Patent Application Publication No. 102020206990 Summary of the Invention [Means for solving the problem]

[0004] According to the invention, an object recognition method is proposed having the features according to claim 1, whereby the relationships between points can be captured more accurately and reliably and better taken into account during processing, the feature context of these points can be better taken into account, the information loss during processing can be reduced and the recognition performance can be increased.

[0005] The object may be a vehicle, a living being, a person, a building, and / or an object, among others. Object detection may take into account recognition of at least one object characteristic (object regression), object classification, and / or object tracking.

[0006] The point-based sensor may output measurement data in the form of at least one point cloud. The measurement data may be provided by at least two such sensors. The point-based sensor may be a camera, in particular a stereo camera or a mono camera (preferably including depth information and / or with the application of image processing algorithms), a Time of Flight camera, a lidar sensor, an ultrasonic sensor, a microphone, or a radar sensor.

[0007] The first processing step may transform input features into learned features through multiple processing planes. The first processing step may apply PointNet, Pointnet++, graph neural networks, continuous convolution, kernel point convolution, or other neural networks with point clouds as input and output.

[0008] A second processing step may transfer the learned features to a two-dimensional model grid, for example based on a bird's eye view (BEV). If there is only one point of the point cloud within a grid cell, the learned feature of this point may be the feature of this grid cell. If there are multiple points of the point cloud within a grid cell, the learned features of these points of this grid cell may be aggregated as the feature of this grid cell. This aggregation may be done by applying a pooling algorithm or PointNet.

[0009] The model grid may be defined by a set grid resolution. The higher the grid resolution, the more grid cells there are per unit space or unit area. The smaller the grid resolution, the higher the probability of object recognition. The larger the grid resolution, the more accurately the object may be characterized.

[0010] In a preferred embodiment of the invention, it is advantageous if the input features are contained in an input feature vector assigned to each point and the learned features are contained in a latent feature vector assigned to this point. The input features can be passed to the first processing step in a random order, without relying on their order.

[0011] Advantageously, a preferred form of the invention is one in which the input feature vector has a different dimensionality to the latent feature vector, which may have a higher or lower dimensionality than the input feature vector.

[0012] In a preferred embodiment of the present invention, the input characteristics of each of the points advantageously include information about the spatial location of each of the points, properties of each of the points, and / or points neighboring each of the points. The spatial location may be represented by coordinates in a three-dimensional coordinate system. The properties may be backscattered signal strength or input strength, reflection cross section, elevation angle, and / or radial velocity. The information about points neighboring each of the points may include the number of neighboring points within a set perimeter.

[0013] In a preferred embodiment of the present invention, the first processing step applies a trained artificial neural network, which may be implemented as deep learning, and the processing plane may be a hidden layer within the artificial neural network.

[0014] The second processing step may apply a trained artificial neural network, and the learned features of the first processing step may be subsequently used in the second processing step. The training of the network in the second processing step may or may not depend on the training of the network in the first processing step.

[0015] A preferred embodiment of the invention is advantageous in that from the output data relating to the cells, after at least one further processing step, output data relating to the object are generated in order to calculate an oriented bounding box of the object. The oriented bounding box may be an oriented rectangular parallelepiped bounding box. The oriented bounding box may have at least one box parameter assigned to this object. The box parameter may be a pose, at least one dimension, an object type class, and / or a presence probability. The object type class may characterize the attribution to the object.

[0016] The oriented envelope shape can be characterized relatively accurately by a point-based first processing step, after which a grid-related second processing step allows for improved object recognition probability and lower false recognition rates.

[0017] The output data relating to the object may include a list with object hypotheses, for each object hypothesis object properties may be calculated, particularly the object type class and the oriented envelope shape.

[0018] The box parameters of the oriented envelope shape can be calculated depending on the characteristics of the grid cells. Furthermore, an image recognition device is proposed, which comprises at least one point-based sensor for providing measurement data about an object and a processing unit adapted to implement a method having at least one of the above-mentioned features, whereby the computing power of the processing unit can be reduced and the image recognition device can be made cheaper.

[0019] In a preferred embodiment of the invention, it is advantageous if the point-based sensor is adapted to output at least one point cloud as measurement data. The point-based sensor may be a camera, in particular a stereo camera or a mono camera (preferably with the application of image processing algorithms), a ToF camera, a lidar sensor, an ultrasonic sensor, a microphone or a radar sensor.

[0020] The image recognition device may be assigned to a driver assistance system and / or an autonomous or partially autonomous vehicle, a robot, in particular a robotic lawnmower, a perimeter monitoring device, in particular a traffic monitoring device, or a vehicle, in particular a car, a truck, or a two-wheeler, preferably a bicycle.

[0021] Image recognition devices can be used in automated assembly systems, for example, to detect parts and their orientation for gripping point determination. Image recognition devices can be used in automated lawn mowers, for example, to detect objects, especially obstacles. Image recognition devices can be used in automatic access control, for example, for person detection and person identification for automatic door opening. Image recognition devices can be used in perimeter surveillance systems, preferably for monitoring squares or buildings, for example, for the detection, inspection, and classification of hazardous materials. Image recognition devices can be used in traffic surveillance systems, especially with stationary radar sensor equipment. Image recognition devices can be used in driver assistance systems to detect and classify road users, for example, bicycles or other two-wheeled vehicles.

[0022] Furthermore, a computer program is proposed which is machine-readable and has instructions executable on at least one computer, the execution of which causes a method having at least one of the characteristics presented above to proceed.Furthermore, a memory unit is proposed which is made machine-readable and accessible by at least one computer, and in which said computer program is stored.

[0023] Further advantages and advantageous forms of the invention are apparent from the description and figures of the figures. The present invention will be described in detail below with reference to the drawings. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is an exemplary block diagram of an object recognition method in accordance with a particular embodiment of the present invention; [Figure 2] FIG. 1 illustrates the structure of the graph convolution of an artificial neural network in the first processing step. [Figure 3] 1 illustrates an image recognition device in a particular embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0025] 1 shows an exemplary block diagram of an object recognition method in a particular embodiment of the present invention. The object recognition method 10 of an object 12 utilizes measurement data 14 from at least one point-based sensor 16 capturing the object 12. The sensor may be a radar sensor 18. The measurement data 14 includes a point cloud 20 having a plurality of points 22 and attributed features 24. In a first processing step 26, the features 24 as input features 28 of the point cloud 20 are implemented as learned features 30. The first processing step 26 includes at least one processing plane 32.

[0026] The first processing step 26 is point-based. The input features 28 for each of the points 22 may include information about the spatial location of each of the points 22, the properties of each of the points 22, and / or the neighboring points 22 of each of the points 22, and may be implemented as an input feature vector 34. The spatial location may be represented by coordinates in a three-dimensional coordinate system. The properties may be backscattered or input signal strength, cross section, elevation angle, and / or radial velocity. The information about the neighboring points 22 of each of the points 22 may include the number of neighboring points 22 within a set perimeter. The input features 28 may be implemented in the first processing step 26 randomly, without relying on their order.

[0027] The processing surface 32 can be implemented by a trained artificial neural network 36, here, for example, a graph neural network 38, which is illustrated by way of example in FIG. 2 and will be described in more detail below. In a first step 40, the graph neural network 38 constructs a graph 42 based on the points 22. For this purpose, points 22 that are within a set distance of each other, e.g., 3 meters, are connected by edges 44. These points 22 represent nodes 46 of the graph 42. In a second step 48, a message 50 is generated for every edge 44 of the graph 42. The message 50 consists of the relative positions 52 of the nodes 46 of one edge 44 relative to each other and the features 54 adjacent to the starting node 55. Thus, the trained features 30 contain information about the relationships between the points 22. These messages 50 are processed by a multilayer perceptron 56 to extract new features 58. Each layer of the multilayer perceptron 56 shares parameters 59 for every message 50.

[0028] In a third step 60, features 64 calculated from the generated messages 50 are extracted by max pooling 62 as learned features 30 for the origin node 55. Then, in a calculation step 66, the difference between the old and new information is calculated (skip connection) and in a second step 48 is again connected as new information to node 46, i.e., point 22.

[0029] Multiple processing planes 68 can be passed through in the first processing step 26. Instead of the graph neural network 38, PointNet, Pointnet++, continuous convolution, kernel point convolution or other neural networks with point clouds as input and output can also be applied.

[0030] Returning to FIG. 1 , in a second processing step 70, which is grid-based and has at least one processing plane 68, the learned features 30 are transferred to a model grid 74 having multiple grid cells 72. A pillar feature network 76 is then used to project the learned features 30, arranged together in a latent feature vector 77, onto the two-dimensional model grid 74. To this end, all points 22 within one grid cell 72 are grouped into a pillar 78. The learned features 30 of each point 22 are individually embedded using a fully connected neural network. If multiple points 22 belong to the same pillar 78, average pooling is applied to all points 22 within the pillar 78 to obtain a fixed-length feature vector. Instead of the pillar feature network 76, other methods may be used to transfer the feature vectors of the points to the model grid 74, such as directly assigning the points to grid cells 72 and then aggregating all feature vectors belonging to the same grid cell 72 using, for example, average pooling, max pooling, or an attention mechanism.

[0031] These features can then be further processed as cell-related output data 80 via a third processing step 82, inter alia, by a two-dimensional convolutional neural network 84 acting as a backbone. For example, a backbone consisting of a residual network and a feature pyramid network can be used, which extracts features for various resolutions of the two-dimensional model grid 74.

[0032] In a fourth processing step 86, a class head estimates for each grid cell 72, via a further two-dimensional convolutional neural network 88, an object probability 90 between 0 and 1 and box parameters 92 relating to the oriented envelope shape of the object 12, in particular its position, length, width, height, and / or orientation. To detect different object types, several of these class heads can be used, each responsible for estimating one object type class, i.e., object types with similar characteristics, such as trucks and buses. These class heads use feature maps 94 with appropriate resolutions corresponding to the object type to be detected, i.e., feature maps 94 with higher resolutions are used for small objects, such as pedestrians, and for larger objects, such as trucks.

[0033] Because an object may span multiple grid cells 72, in a fifth processing step 96, the object hypotheses 98 generated in the fourth processing step 86 are filtered. This is done, among other things, by non-maximum suppression 100 (NMS). In this case, for each object, spatially overlapping object hypotheses 98 are filtered to the object hypothesis with the highest object probability. The filtered object hypotheses, in the form of oriented envelope shapes 102, form the object-related output data 80 of the method 10.

[0034] The object related output data 80 is, for example, a list with object hypotheses. For each object hypothesis, object properties may be calculated, in particular object type classification, object position, and box parameters, in particular length, width, height, and / or orientation of the oriented envelope shape 102 that encloses the object.

[0035] FIG. 3 illustrates an image recognition device in a specific embodiment of the present invention. In FIG. 3a), an image recognition device 104 is shown including a processing unit 106 that implements an object recognition method. The image recognition device 104 can be used in an automated assembly system 108, for example, to detect parts and their orientation for grasping point determination. In FIG. 3b), the image recognition device 104 can be used in an automated lawn mower 110, for example, to detect objects 12, especially obstacles. In FIG. 3c), the image recognition device 104 can be used in automated access control, for example, for person detection and identification for automatic door opening. In FIG. 3d), the image recognition device 104 can be used in a perimeter monitoring system 114, preferably for monitoring a square or building, for example, for detecting, inspecting, and classifying dangerous objects. In FIG. 3e), the image recognition device 104 can be used in a traffic monitoring system 116, especially with at least one stationary radar sensor 18. The image recognition device 104 in FIG. 3f) can be used in a driver assistance system 118 to detect and classify road users, such as bicycles 120 or other two-wheeled vehicles.

Claims

1. A method (10) for object recognition of an object (12) based on measurement data (14) of at least one point-based sensor (16) capturing the object (12), comprising: a point-based first processing step (26) using an artificial neural network (36) having at least one hidden layer (32) for processing the measurement data (14) based on a point cloud (20) having a plurality of points (22), the first processing step (26) processing input features (28) of the point cloud (20) to output learned features (30) containing information (50) about relationships between the points (22); a grid-based second processing step (70) using an artificial neural network (38) having at least one hidden layer (68) to process and transform the learned features (30) onto a model grid (74) having a plurality of grid cells (72), and thereafter generate output data (80) related to the grid cells (72); An object recognition method (10) characterized by performing the following steps.

2. 2. The object recognition method (10) of claim 1, wherein the input features (28) are contained in an input feature vector (34) assigned to the point (22), and the learned features (30) are contained in a latent feature vector (77) assigned to the point (22).

3. 3. The method (10) of claim 2, wherein the input feature vector (34) has a different dimension than the latent feature vector (77).

4. 2. The object recognition method (10) of claim 1, wherein the input features (28) of the point (22) include information (50) about the spatial location of the point (22), properties of the point (22), and / or neighboring points of the point (22).

5. 2. The method (10) of claim 1, wherein said first processing step (26) applies a trained artificial neural network (36).

6. 2. The object recognition method (10) of claim 1, wherein from the cell-related output data (80), through at least one further processing step (82, 86, 96), object-related output data (80) is generated for calculating an oriented envelope shape (102) of the object (12).

7. 10. An image recognition device (104) having at least one point-based sensor (16) that provides measurement data (14) about an object (12) and a processing unit (106) adapted to perform the method (10) of claim 1.

8. 8. The image recognition device (104) of claim 7, wherein the point-based sensor (16) is adapted to output at least one point cloud (20) as measurement data (14).

9. A computer program product having machine-readable, computer-executable instructions, the execution of which causes the method (10) of claim 1 to proceed.

10. A memory unit made machine-readable and accessible by at least one computer and having stored thereon the computer program of claim 9.

Images