OBJECT RECOGNITION METHOD, IMAGE RECOGNITION DEVICE, COMPUTER PROGRAM, AND MEMORY UNIT

Abstract

The invention relates to a method (10) for object recognition (12) based on measurement data (14) of at least one point-based sensor (16) capturing the object (12), whereby the measurement data (14) based on a point cloud (20) with a plurality of points (22) and attributive features (24) is first processed in a first processing step (26) on a point basis and with at least one processing plane (32) by implementing input features (28) of the point cloud (20) as learned features (30) and by obtaining at least information (50) about the relations between the points (22), and the learned features (30) are subsequently transferred in a second processing step (70) on a grid basis and with at least one processing plane (68) to a model grid (74) with a plurality of grid cells (72), after which output data (80) related to the cells are generated. Furthermore, the invention relates to an image recognition device (104), a computer program, and a memory unit.

Description

[Technical field]

[0001] The invention relates to a method for object recognition according to claim 1. Furthermore the invention relates to an image recognition device, a computer program and a memory unit. [Background technology]

[0002] DE 10 2020206 990 A1 describes a method for processing measurement data of sensors, which transfers the measurement data of a first sensor in a first encoder and the measurement data of a second sensor in a second encoder into respective latent spaces, from features in which a first decoder derives the reconstructed measurement data of the first sensor and a second decoder derives the reconstructed measurement data of the second sensor. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] DE 102020206990 Summary of the Invention [Means for solving the problem]

[0004] According to the invention, a method for object recognition is proposed having the features according to claim 1, whereby the associations between points can be captured more accurately, more reliably and better taken into account during processing. The feature context of these points can be better taken into account. Information loss during processing can be reduced and the recognition performance can be increased.

[0005] The object may be a vehicle, a living being, a person, a building, and / or an object, among others. Object detection may take into account at least one object characteristic recognition (object regression), object classification, and / or object tracking.

[0006] The point-based sensor may output measurement data in the form of at least one point cloud. The measurement data may be provided by at least two such sensors. The point-based sensor may be a camera, in particular a stereo camera or a mono camera (preferably including depth information and / or with the application of image processing algorithms), a ToF camera, a lidar sensor, an ultrasonic sensor, a microphone, or a radar sensor.

[0007] The first processing step may pass the input features through multiple processing planes to transform them into learned features. The first processing step may apply PointNet, Pointnet++, graph neural networks, successive convolutions, kernel point convolutions, or other neural networks with point clouds as input and output.

[0008] A second processing step may transfer the learned features to a two-dimensional model grid, for example based on a bird's eye view (BEV). If there is only one point of the point cloud in one grid cell, the learned features of this point may be the feature of this grid cell. If there are multiple points of the point cloud in one grid cell, the learned features of these points of this grid cell may be aggregated as the feature of this grid cell. This aggregation may be done by applying a pooling algorithm or PointNet.

[0009] The model grid may be defined by a set grid resolution. The higher the grid resolution, the more grid cells there are per unit space or area. The smaller the grid resolution, the higher the probability of object recognition. The larger the grid resolution, the more accurately the object may be characterized.

[0010] In a preferred embodiment of the invention, it is advantageous if the input features are contained in an input feature vector assigned to each one of the points and the learned features are contained in a latent feature vector assigned to this point. The input features can be passed to the first processing step in a random manner, without relying on their order.

[0011] Advantageously, a preferred form of the invention is where the input feature vector has a different dimensionality to the latent feature vector. The latent feature vector may have a higher or lower dimensionality than the input feature vector.

[0012] In a preferred embodiment of the invention, it is advantageous if the input characteristics of each of the points include information about the spatial location of each of the points, properties of each of the points, and / or neighboring points of each of the points. The spatial location may be represented by coordinates in a three-dimensional coordinate system. The properties may be backscattered signal strength or input strength, reflection cross section, elevation angle, and / or radial velocity. The information about the neighboring points of each of the points may include the number of neighboring points within a set perimeter.

[0013] In a preferred embodiment of the invention, the first processing step applies a trained artificial neural network. This training can be implemented as deep learning. The processing surface can be a hidden layer in the artificial neural network.

[0014] The second processing step may apply a trained artificial neural network. The trained features of the first processing step may be subsequently used in the second processing step. The training of the network in the second processing step may or may not be dependent on the training of the network in the first processing step.

[0015] A preferred embodiment of the invention is advantageous, in which from the output data related to the cells, via at least one further processing step, output data related to the object are generated for calculating an oriented bounding box of the object. The oriented bounding box may be an oriented rectangular parallelepiped. The oriented bounding box may have at least one box parameter assigned to this object. The box parameter may be a pose, at least one dimension, an object type class and / or a presence probability. The object type class may characterize the attribution to the object.

[0016] The oriented envelope shape can be characterized relatively accurately by a point-based first processing step, after which a grid-related second processing step allows for improved object recognition probability and lower false recognition rates.

[0017] The output data relating to the objects may include a list with object hypotheses, for each object hypothesis object properties may be calculated, particularly the object type class and the oriented envelope shape.

[0018] The box parameters of the oriented envelope shape can be calculated depending on the characteristics of the grid cells. Furthermore, an image recognition device is proposed, which comprises at least one point-based sensor providing measurement data about the object and a processing unit adapted to implement a method having at least one of the above listed features, whereby the computing power of the processing unit can be reduced and the image recognition device can be produced more cheaply.

[0019] In a preferred embodiment of the invention, it is advantageous if the point-based sensor is adapted to output at least one point cloud as measurement data. The point-based sensor may be a camera, in particular a stereo camera or a mono camera (preferably with the application of an image processing algorithm), a ToF camera, a lidar sensor, an ultrasonic sensor, a microphone or a radar sensor.

[0020] The image recognition device may be assigned to a driver assistance system and / or an autonomous or partially autonomous vehicle. The image recognition device may be assigned to a robot, in particular a robotic lawnmower, a surroundings monitoring device, in particular a traffic monitoring device, or a vehicle, in particular a car, a truck, or a two-wheeler, preferably a bicycle.

[0021] Image recognition devices can be used in automated assembly installations, for example to detect parts and their orientation for gripping point determination. Image recognition devices can be used in automatic lawnmowers, for example to detect objects, especially obstacles. Image recognition devices can be used in automatic access control, for example for person detection and person identification for automatic door opening. Image recognition devices can be used in perimeter monitoring installations, preferably for monitoring squares or buildings, for example for detection, inspection and classification of hazardous materials. Image recognition devices can be used in traffic monitoring installations, especially with stationary radar sensor equipment. Image recognition devices can be used in driver assistance systems to detect and classify road users, for example bicycles or other two-wheeled vehicles.

[0022] Furthermore, a computer program is proposed, comprising machine-readable and executable instructions on at least one computer, the execution of which causes the progression of a method having at least one of the characteristics presented above.Furthermore, a memory unit is proposed, made machine-readable and accessible by at least one computer, and in which said computer program is stored.

[0023] Further advantages and advantageous configurations of the invention are apparent from the description and figures of the figures. The present invention will be described in detail below with reference to the drawings. [Brief description of the drawings]

[0024] [Figure 1] FIG. 2 is an exemplary block diagram of an object recognition method in accordance with a particular embodiment of the present invention; [Diagram 2] FIG. 2 shows the structure of the graph convolution of an artificial neural network in the first processing step. [Diagram 3] FIG. 2 illustrates an image recognition device in a particular embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] 1 shows an exemplary block diagram of an object recognition method in a particular embodiment of the present invention. The object recognition method 10 of an object 12 utilizes measurement data 14 of at least one point-based sensor 16 capturing the object 12. The sensor may be a radar sensor 18. The measurement data 14 comprises a point cloud 20 having a plurality of points 22 and attributed features 24. In a first processing step 26, the features 24 as input features 28 of the point cloud 20 are implemented as learned features 30. The first processing step 26 comprises at least one processing plane 32.

[0026] The first processing step 26 is point-based. The input features 28 of each of the points 22 may include information about the spatial location of each of the points 22, the properties of each of the points 22, and / or the neighboring points 22 of each of the points 22, and may be implemented as an input feature vector 34. The spatial location may be represented by coordinates in a three-dimensional coordinate system. The properties may be backscattered signal strength or input strength, cross section, elevation angle, and / or radial velocity. The information about the neighboring points 22 of each of the points 22 may include the number of neighboring points 22 within a set perimeter. The input features 28 may be implemented in the first processing step 26 in a random manner, without relying on their order.

[0027] The processing surface 32 can be applied with a trained artificial neural network 36, here for example a graph neural network 38, which is illustrated by way of example in FIG. 2 and will be explained in more detail below. In a first step 40, the graph neural network 38 builds a graph 42 on the basis of the points 22, for which the points 22 that are within a set distance from each other, for example 3 meters, are connected by edges 44. These points 22 represent the nodes 46 of the graph 42. In a second step 48, messages 50 are generated for every edge 44 of the graph 42, which messages 50 consist of the relative positions 52 of the nodes 46 of an edge 44 relative to each other and the features 54 adjacent to the vicinity of the origin node 55. The trained features 30 thus contain information about the relations between the points 22. These messages 50 are processed by a multi-layer perceptron 56 in order to extract new features 58. Each layer of the multi-layer perceptron 56 then shares parameters 59 for every message 50.

[0028] In a third step 60, features 64 computed from the generated messages 50 are extracted by max pooling 62 as learned features 30 for the origin node 55. Then in a calculation step 66, the difference between the old and new information is calculated (skip connection) and in a second step 48 is again connected as new information to node 46, i.e. point 22.

[0029] A number of processing planes 68 can be passed in the first processing step 26. Instead of the graph neural network 38, also PointNet, Pointnet++, continuous convolution, kernel point convolution or other neural networks having point clouds as input and as output can be applied.

[0030] Returning to FIG. 1, in a second processing step 70, which is grid-based and has at least one processing plane 68, the learned features 30 are transferred to a model grid 74 with multiple grid cells 72. A Pillar Feature Network 76 is then used to project the learned features 30, arranged together in a latent feature vector 77, onto the two-dimensional model grid 74. For this purpose, all points 22 within one grid cell 72 are grouped into a pillar 78. The learned features 30 of each point 22 are embedded individually by a fully connected neural network. If multiple points 22 belong to the same pillar 78, average pooling of all points 22 within the pillar 78 is applied to obtain a fixed-length feature vector. Instead of the Pillar Feature Network 76, other methods may be used to transfer the feature vectors of the points to the model grid 74, for example by directly assigning the points to grid cells 72 and then aggregating all feature vectors belonging to the same grid cell 72, for example by passing them through average pooling, max pooling, or an attention mechanism.

[0031] These features can then be further processed as cell-related output data 80 via a third processing step 82, notably by a two-dimensional convolutional neural network 84 acting as a backbone. For example, a backbone consisting of a residual network and a feature pyramid network can be used, which extracts features for different resolutions of the two-dimensional model grid 74.

[0032] In a fourth processing step 86, the Class Heads estimate for each grid cell 72, via a further 2D convolutional neural network 88, an object probability 90 between 0 and 1 and box parameters 92 for the oriented envelope shape of the object 12, in particular the position, length, width, height and / or orientation. To detect different object types, several of these Class Heads can be used, each responsible for estimating one object type class, i.e. object types with similar characteristics, for example trucks and buses. These Class Heads use feature maps 94 with appropriate resolutions according to the object types to be detected, i.e. for small objects, for example pedestrians, feature maps 94 with higher resolutions are used for large objects, for example trucks.

[0033] Since an object may span multiple grid cells 72, in a fifth processing step 96 the object hypotheses 98 generated in the fourth processing step 86 are filtered. This is done, among other things, by non-maximal suppression 100 (NMS). In this case, for each object, the spatially overlapping object hypotheses 98 are filtered to the object hypothesis with the highest object probability. The filtered object hypotheses in the form of oriented envelope shapes 102 form the object related output data 80 of the method 10.

[0034] The object related output data 80 is, for example, a list with object hypotheses. For each object hypothesis, object properties may be calculated, in particular the object type classification, the object position, and box parameters, in particular the length, width, height, and / or orientation of the oriented envelope shape 102 that encloses the object.

[0035] FIG. 3 shows an image recognition device in a special embodiment of the invention. In FIG. 3a) an image recognition device 104 is shown which includes a processing unit 106 for implementing the object recognition method. The image recognition device 104 can be used in an automated assembly installation 108, for example, for detecting parts and their orientation for gripping point determination. The image recognition device 104 in FIG. 3b) can be used in an automatic lawnmower 110, for example, for detecting objects 12, especially obstacles. The image recognition device 104 in FIG. 3c) can be used in automatic access control, for example for person detection and person identification for automatic door opening. The image recognition device 104 in FIG. 3d) can be used in a perimeter monitoring installation 114, preferably for monitoring a square or a building, for example for detection, inspection and classification of dangerous objects. The image recognition device 104 in FIG. 3e) can be used in a traffic monitoring installation 116, especially with at least one stationary radar sensor 18. The image recognition device 104 in FIG. 3f) may be used in a driver assistance system 118 to detect and classify road users, such as bicycles 120 or other two-wheeled vehicles.

Claims

1. A method (10) for object recognition of an object (12) based on measurement data (14) of at least one point-based sensor (16) capturing said object (12), said measurement data (14) being based on a point cloud (20) having a plurality of points (22) and attributive features (24), comprising: firstly, in a first processing step (26) having at least one processing plane (32) on a point basis, input features (28) of said point cloud (20) are implemented as learned features (30) and at least information (50) about relations between said points (22) is obtained; and The learned features (30) are subsequently processed in a second processing step (70) having at least one processing plane (68) on a grid basis by transferring them to a model grid (74) having a plurality of grid cells (72), and then generating output data (80) related to the cells.

2. 2. The method (10) of claim 1, wherein the input features (28) are contained in an input feature vector (34) assigned to each of the points (22), and the learned features (30) are contained in a latent feature vector (77) assigned to the points (22).

3. 3. The method (10) of claim 2, wherein the input feature vector (34) has a different dimension to the latent feature vector (77).

4. The method (10) for object recognition according to any one of claims 1 to 3, characterized in that the input features (28) of each of the points (22) comprise information (50) about the spatial position of each of the points (22), about properties of each of the points (22), and / or about points adjacent to each of the points (22).

5. Method (10) for recognizing objects according to any one of the preceding claims, characterized in that said first processing step (26) comprises applying a trained artificial neural network (36).

6. The method (10) for recognizing an object according to any one of claims 1 to 5, characterized in that from the cell-related output data (80), via at least one further processing step (82, 86, 96), object-related output data (80) is generated for computing an oriented envelope shape (102) of the object (12).

7. An image recognition device (104) having at least one point-based sensor (16) providing measurement data (14) about an object (12) and a processing unit (106) adapted to perform the method (10) according to any one of claims 1 to 6.

8. 8. The image recognition device (104) of claim 7, wherein the point-based sensor (16) is adapted to output at least one point cloud (20) as measurement data (14).

9. A computer program comprising at least one machine-readable and computer-executable instruction, the execution of which causes the method (10) of any one of claims 1 to 6 to proceed.

10. A memory unit produced in a machine-readable and accessible manner by at least one computer and having stored thereon the computer program of claim 9.

Images