Apparatus and method for object detection

Abstract

A method and apparatus for object detection. A method for object detection using a multi-modal sensor input has the steps of: detecting first and second sensor data in an image section by means of a first and second sensor device and forming respective first and second feature vectors; providing an arrangement, in particular a grid, having a predetermined first plurality of suggested locations in the image section for object search; extracting feature vectors from the first and second feature vectors in the surroundings of the suggested locations, correspondingly fusing the first and second feature vectors at the suggested locations, generating a corresponding estimated bounding box for each of the suggested locations in the surroundings of the suggested locations, calculating a corresponding confidence level for the bounding boxes; reducing the first plurality of suggested locations to less than the first plurality of second number of locations according to the corresponding calculated confidence level; an object is identified based on an object search at a second number of locations.

Description

Technical Field

[0001] The present invention relates to a device and a method for object detection. Background Art

[0002] Object detection is a branch of computer vision in which attempts are made to identify individual objects in sensor data. In this context, the sensor data is segmented into regions that form meaningful units, which are then investigated with regard to their characteristics in order to be able to assign image regions to classes of objects. In the case of object detection, for example, an image is segmented into smaller image segments (windows) of a certain size, and classification algorithms are then applied to these windows.

[0003] Such reliable object detection is of particular relevance in the field of autonomous driving of motor vehicles, wherein the surroundings of the motor vehicle are detected by means of sensor devices such as cameras, lidar, radar or the like.

[0004] In recent years, the Transformer architecture was first developed for the field of language processing and has subsequently also been successfully applied to two-dimensional and three-dimensional object detection. Transformer-based object detectors are based on so-called object queries, i.e. high-dimensional vectors, wherein each of these object queries can be understood as an object candidate that can detect at most one object.

[0005] An object query (also called an object search) is essentially a feature vector in a latent or embedding space that encodes the information needed to predict a classified 2D or 3D bounding box.

[0006] Typically, each object query is associated with a reference point or anchor, based on which a bounding box is predicted. Currently, the initialization of these object queries is intensively studied. Here, two basic schemes are distinguished, namely training a fixed distribution of feature vectors through training or initializing according to the current sensor input.

[0007] Object detection with initialization by the trained distribution requires multiple object queries in order to cover the entire object detection grid, and may miss objects if multiple objects are arranged in a tight space if the object query cannot cover all objects.

[0008] Object detection with initialization from current sensor input solves this problem by placing object queries at locations where objects are expected based on the predictions of the previous stage with the help of a trainable network.

[0009] X.Bai et al. describe object detection with initialization using current sensor inputs from a lidar and a camera in TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers (Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 2022pp.1080-1089), where camera-based features are used to refine the predictions of the initial set of lidar data. However, this method fails to optimally utilize the sensor input from the camera because the feature vector for object query is initialized using the sensor input from the lidar, and the sensor input from the camera is only considered sequentially for refinement.

[0010] DE 10 2009 006 113 A1 discloses a device and a method for providing a representation of the environment of a vehicle, comprising at least one first sensor device and at least one second sensor device and an evaluation device, wherein the sensor devices provide information about objects recognized in the environment of the vehicle in the form of sensor objects, wherein the sensor objects represent objects recognized by the corresponding sensor devices and the sensor objects include at least one existence probability of the represented object as an attribute, and the sensor objects recognized by the at least one first sensor device and by the at least one second sensor device (3) undergo object fusion, in which a fused object is generated, which is assigned at least one existence probability as an attribute, wherein the existence probability of the fused object is fused based on the existence probability of the sensor objects, wherein the fusion of the existence probability of one of the sensor objects is respectively performed according to the corresponding sensor device that provides the corresponding sensor object.

[0011] DE 11 2018 000 899 T5 discloses a method for recognizing objects from a 3D point cloud and a 2D image, comprising the steps of determining a first set of 3D proposals using Euclidean clustering of the 3D point cloud, determining a second set of 3D proposals from the 3D point cloud based on a neural 3D convolutional network, merging the first set of 3D proposals and the second set of 3D proposals to determine a set of 3D candidates, projecting the first set of 3D proposals onto a 2D image, determining a first set of 2D proposals based on the image using a neural 2D convolutional network, merging the projected first set of 3D proposals with the first set of 2D proposals to determine a set of 2D candidates, and merging the set of 3D candidates with the set of 2D candidates.

[0012] DE 10 2019 127 282 A1 discloses a system for analyzing a three-dimensional surrounding environment, comprising: a sensor unit configured to provide a point cloud representing a three-dimensional surrounding environment, an RGB camera sensor for providing at least one RGB image having RGB color information of the three-dimensional surrounding environment, a first processing unit for fusing the point cloud from the sensor unit with the RGB color information of the RGB image from the RGB camera sensor into an RGB point cloud, and a second processing unit for implementing at least one of the following tasks: an object detection task, a semantic segmentation task, or a classification task related to the three-dimensional surrounding environment, wherein the first processing unit is configured to set the second processing unit with the fused RGB point cloud, the second processing unit comprises an encoder, a first decoder for the object detection task, a second decoder for the semantic segmentation task, and a third decoder for the classification task, wherein the encoder is configured to receive the point cloud as input data, extract features required for implementing the object detection task, the semantic segmentation task, and the classification task from the input data based on a deep neural network, and provide the extracted features to the first decoder, the second decoder, and the third decoder, and wherein the first decoder, the second decoder, and the third decoder each have a neural network dedicated to their corresponding tasks. Summary of the invention

[0013] A device for object detection, in particular when using multimodal sensor input, and a method for object detection, in particular when using multimodal sensor input, are proposed.

[0014] The initial estimation of object features can first be performed in a defined arrangement, such as a grid, for a large number of initial object queries by associating sensor inputs from a first sensor device, such as a lidar, and from a second sensor device, such as a camera. Subsequently, the number of object queries can be reduced and the object queries can be arranged (platzierung) according to the statistical analysis of the formed fused feature vectors. This enables a significant reduction in the initialization overhead (overheads) of typically 90% compared to the prior art (TransFusion). The first and second sensor devices can belong to different sensor types (e.g. camera-lidar, lidar-radar, radar, camera), so-called multimodal sensor inputs, or different sensors of the same sensor type (e.g. camera-camera, lidar-lidar, radar-radar). "First and second sensor devices" are understood as a minimum amount. The described device and method can also be supplemented with other sensor devices to be applied accordingly (be it multimodal or not). In an alternative configuration, the method for reducing the location for object queries can be used for a system with one sensor.

[0015] According to a preferred embodiment, the output device is designed to output an object detection representation as a function of the object search result.

[0016] According to a further preferred embodiment, the first sensor device comprises a lidar sensor device and the second sensor device comprises a camera sensor device.

[0017] According to a further preferred embodiment, the grid can be expanded two-dimensionally in a plane and each suggested location for the object search can be assigned the same fixed height in order to configure the grid three-dimensionally.

[0018] According to a further preferred embodiment, the decoder is a transformer decoder having a regression head and a classification head.

[0019] According to a further preferred embodiment, corresponding predicted features for the object search can be predicted using a regression head and a classification head.

[0020] According to a further preferred embodiment, the initial position on the grid of the object query can be offset by a corresponding predicted or estimated offset, with the aim of moving closer to the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The invention is explained in more detail below with reference to an exemplary embodiment illustrated in the schematic drawings.

[0022] Here it is shown:

[0023] Figure 1 : a schematic block diagram for illustrating an apparatus for object detection using a multimodal sensor input according to a first embodiment of the present invention; and

[0024] Figure 2 : A schematic flow chart for illustrating a method for object detection using multimodal sensor input according to a second embodiment of the present invention.

[0025] The accompanying drawings are intended to further understand the embodiments of the present invention. They illustrate the embodiments and are used to illustrate the principles and concepts of the present invention in conjunction with the description. Other embodiments and many of the advantages mentioned are derived with reference to the accompanying drawings. The elements in the accompanying drawings are not necessarily shown to scale with each other.

[0026] Unless otherwise indicated, identical elements, features and components having the same function and the same effect are each provided with the same reference symbols in the figures. DETAILED DESCRIPTION

[0027] Figure 1 A schematic block diagram is shown for explaining a device for object detection using multimodal sensor input according to a first specific embodiment of the present invention.

[0028] A device for detecting objects in the surroundings of a carrier device, in particular a motor vehicle, using multimodal sensor input according to a first specific embodiment has a sensor element SE, a processing element ST, a transducer decoder TD and an output device DA.

[0029] In the present embodiment, these assemblies or components are arranged on a common carrier device, for example a motor vehicle, although they may also be configured in other ways.

[0030] The sensor component SE comprises a first sensor device S1, BS1, which comprises a lidar sensor device for detecting first sensor data SD1 in an image segment BA of the surroundings and further comprises a first converter device BS1 for forming a corresponding first feature vector F1 from the first sensor data SD1. Alternatively, the first feature vector F1 can also be formed in the processing component ST.

[0031] The sensor component SE further comprises a second sensor device S2, BS2, which comprises a camera sensor device S2 for detecting second sensor data SD2 in an image segment BA of the surroundings, and further comprises a second converter device BS2 for forming a corresponding second feature vector (M2) from the second sensor data SD2. Alternatively, the second feature vector F2 can also be formed in the processing component ST.

[0032] The processing unit ST includes a processing device, such as a computer. The processing device ST uses the outputs of the first sensor device S1, BS1 and the second sensor device S2, BS2 to provide a suggested location l' for object search. ij .

[0033] For this purpose, the processing device ST provides an arrangement, in particular a grid G, which has a first plurality of, in particular a predetermined number of, suggested locations l for the object search in the image segment BA. ij , which can, for example, cover the image area BA substantially uniformly, but can also have a different distribution. Typically, the grid points l ij The number is about 3000 to 5000.

[0034] For example, the grid G ​​is generated in such a way that the grid G ​​is spread out two-dimensionally in the x,y plane (bird's eye view) over the image segment BA and for each proposed location l for the object search ij The same fixed height is assigned in order to configure the grid (G) three-dimensionally and thus determine a first plurality of suggested locations l for the object search in the image segment BA. ij .

[0035] The processing means calculates the projections in order to determine which areas of the feature arrangements F1, F2 are closest to the proposed location l ij This means for the camera sensor data S2 that the proposed location l ij According to the projection, the proposed location l for object search is extracted from F1 and F2. ij The feature vectors F1', F2' in the surrounding environment are fused, for example by concatenation.

[0036] The processing device ST is further configured to, at a suggested location l for the object search, ij The corresponding predicted bounding box b is generated for object search in the surrounding environment ij . Bounding box b ij The location of the object can be determined, for example, by its corresponding geometric center point. ij and the confidence level is determined by the regression and classification heads SV.

[0037] For example, the decoder TD is a trained transformer decoder equipped with a regression head and a classification head, and the corresponding predicted bounding box b for object search ij is predicted using both regression and classification heads.

[0038] The processing device ST then transforms the predicted bounding box b for object search into the corresponding confidence level obtained by the detection head SV from the corresponding fused feature vector. ij to a predetermined second plurality that is significantly smaller than the first plurality.

[0039] Typically, the reduced predicted bounding box b ij The quantity is about 200-900.

[0040] Optionally, each predicted bounding box b for object search is determined by ij Center: Using regression head (SV) to make the initial proposal location l ij With the corresponding predicted or estimated offset Δ ij The suggested locations for object search that are staggered ij 'Offset. Then, the staggered suggested location l for object search is obtained from the associated first and second feature vectors F1, F2. ij 'The new fused feature vector of the surrounding environment.

[0041] The trained transformer decoder TD receives the corresponding optionally staggered reduced locations l ij or ij ' as the initial location for object query, and optionally receives the bounding box b ij , and output the corresponding object search result QO.

[0042] The output device DA is used to output the object detection representation PO, for example on a screen, based on the object search result QO. Further internal processing, for example for activating actuators without the output device DA, is also possible.

[0043] Figure 2 is a schematic flow chart for explaining a method for object detection using multimodal sensor input according to a second embodiment of the present invention.

[0044] In a step S1 , first sensor data SD1 are acquired in an image segment BA by means of a first sensor device S1 , BS1 , and a corresponding first feature vector F1 is formed.

[0045] In a step S2 , second sensor data SD2 are acquired in the image section BA by means of a second sensor device S2 , BS2 , and a corresponding second feature vector M2 is formed.

[0046] In this case, the image section BA can respectively represent only a portion of the detection range of the first and second sensor device in which the two detection ranges overlap.

[0047] In step S3, an arrangement, for example a grid G, is provided, which has a first plurality of predefined suggested locations l for the object search in the image segment BA. ij In one embodiment, the arrangement covers the image section BA substantially uniformly, but different patterns of arrangement of the suggestion locations are also possible, for example with a clustering (ie smaller distances between the locations) in the center of the image.

[0048] In step S4, from the suggested locations l for object search ij The first and second feature vectors F1 and F2 in the surrounding environment extract feature vectors F1' and F2', and merge them into the proposed location I respectively. ij The corresponding feature vector at a specific location of , in particular by splicing.

[0049] In step S5, the associated fused feature vectors F1', F2' are used to find the suggested location l for the object search. ij In the surrounding environment, generate the corresponding estimated bounding box b for object search ij and a confidence level. The confidence level here represents a confidence value for the bounding box (e.g. its location and size). This confidence level can be determined with the aid of the trained model or network used. The formation of the bounding box and the confidence level is preferably as described in Figure 1 Proceed as described.

[0050] In step S6, for the further object search, the number of locations is reduced from the initially suggested locations to a second predetermined number that is smaller, in particular significantly smaller, than the first number, depending on the corresponding confidence level of the bounding box determined from the associated fused feature vectors F1, F2. The locations can be determined by the bounding box with the highest confidence level, in particular in that they correspond to the center of the bounding box with the highest confidence level. The latter step not only reduces the number of locations, but also shifts them in position, depending on the position of the bounding box.

[0051] In step S7, the reduced estimated location l' is received. ij, as the initial location for the object query - object search QI, and in particular outputs the corresponding object search result QO with the help of the trained decoder TD and regression head / classification head DA.

[0052] Although the present invention has been described on the basis of preferred embodiments, it is not limited thereto but can be modified in many ways.

[0053] In a special alternative configuration, the corresponding method can also be performed with only one sensor source. In this case, for object recognition, sensor data in an image segment can be detected by means of a sensor device and a corresponding feature vector can be formed. Feature vectors can be extracted from the formed feature vectors in the surroundings of one of the plurality of proposed locations. Based on the extracted feature vectors, a corresponding estimated bounding box can be calculated for each location of the proposed location in the surroundings of the proposed location, and a corresponding confidence level of the estimated bounding box can be formed. Based on the corresponding calculated confidence level, the plurality of proposed locations can be reduced to a number of locations that is less than the plurality of proposed locations. The object can then be identified by an object search at a second number of locations.

[0054] In particular, the invention is not limited to a lidar sensor device in combination with a camera sensor device, but can be applied to any desired sensor combination. The possible fusion of the sensor data is designed in different ways.

[0055] The initial first plurality of suggested locations l for object search ij The number of can be arbitrarily configured in a directionally applicable manner, the second plurality of reduced, suggested, displaced locations l' ij The same is true for the dimensions of the feature vector.

[0056] Finally, the invention is not limited to transformer decoders but can be applied to any object detector.

[0057] The first plurality of suggested locations l for the object search can be determined with the aid of a three-dimensional grid or an arbitrary three-dimensional or two-dimensional arrangement. ij .

[0058] The invention can be used in the automotive sector, for example in driver assistance systems or in automated driving functions, but also in robotics or safety technology or other areas in which object detection is relevant.

Claims

1. A device for detecting objects in the surroundings of a carrier device, such as a motor vehicle, using multimodal sensor input, the device comprising: a first sensor device (S1, BS1) for detecting first sensor data (SD1) in an image section (BA) of the surroundings, a second sensor device (S2, BS2) for detecting second sensor data (SD2) in an image section (BA) of the surroundings, A processing device (ST) for providing a first plurality of suggested locations (100) for searching for an object in the image segment (BA) ij ) arrangement (G), in, The processing device (ST) is configured to, from the suggested locations (l ij ), extracting feature vectors (F1', F2') from a first feature vector (F1) formed by the first sensor data and a second feature vector (F2) formed by the second sensor data in an environment of the image sensor, and constructing a method for fusing the first and second feature vectors (F1, F2); wherein the processing device (ST) is configured to, at the suggested location (l ij ) in the surrounding environment to generate the corresponding predicted bounding box (b) for object search ij ), and generating a confidence level for the corresponding bounding box; The processing device (ST) is designed to, according to the corresponding confidence level determined from the associated fused feature vectors (F1, F2), calculate the first plurality of predicted locations (1 for the object search). ij ) is reduced to a second number of estimated locations (I') which is less than the first plurality of locations. ij ); Therein, the processing device (ST) is configured to perform an object search (QI) at the estimated location and to output a resultant object search result (QO).

2. The device according to claim 1, comprising output means (DA) for outputting an object detection representation (PO) based on the object search result (QO).

3. The device according to claim 1 or 2, wherein: The first sensor device (S1) and the second sensor device (S2) respectively include one of a lidar sensor device, a radar sensor device or a camera sensor device. In particular, the first sensor device (S1) and the second sensor device (S2) represent two different lidar sensor devices, radar sensor devices or camera sensor devices.

4. The device according to any one of the preceding claims, wherein: The arrangement (G) is a grid that can be expanded two-dimensionally in a plane and for each proposed location (l ij ) can be assigned the same fixed height in order to configure the grid (G) three-dimensionally.

5. Apparatus according to any one of the preceding claims, wherein: The arrangement for the object search accesses a decoder (TD), which is in particular a transformer decoder having a regression head (DA) and a classification head (DA).

6. The device according to claim 5, wherein: The corresponding predicted bounding box (b ij ) can be predicted using the regression head.

7. The device according to claim 5, wherein: Each predicted bounding box (b ij ) can be used with the corresponding prediction offset (Δ ij ) and the corresponding suggested locations for object search (l ij )stagger.

8. A method for object detection, in particular when using multimodal sensor input, comprising the steps of: Acquiring first sensor data (SD1) in an image section (BA) by means of a first sensor device (S1, BS1) and forming a corresponding first feature vector (F1); detecting second sensor data (SD2) in the image section (BA) by means of a second sensor device (S2, BS2) and forming a corresponding second feature vector (M2); An arrangement, in particular a grid (G), is provided, which has a first plurality of predefined suggested locations (1 ) for an object search in the image segment (BA). ij ); From the proposed location (l ij ) in the surrounding environment of the first feature vector and the second feature vector (F1, F2) extract feature vectors (F1', F2'), at the proposed location (l ij ) where the first and second eigenvectors are fused correspondingly, According to the fused first feature vector and the second feature vector, at the suggested location (l ij ) in the surrounding environment of each of the proposed locations to generate a corresponding estimated bounding box (b ij ), and calculating a corresponding confidence level for the bounding box; The first plurality of suggested locations (l ij ) is reduced to a second number of locations (l') which is less than the first plurality of locations ij );and An object is identified based on a search for the object at the second number of locations.

9. The method according to claim 8, comprising the step of outputting an object detection representation (PO) based on the object search result (QO).

10. The method according to claim 8 or 9, wherein: The first sensor device (S1) and the second sensor device (S2) respectively include one of a lidar sensor device, a radar sensor device or a camera sensor device. In particular, the first sensor device (S1) and the second sensor device (S2) represent two different lidar sensor devices, radar sensor devices or camera sensor devices.

11. The method according to any one of claims 8 to 10, wherein: The arrangement is a grid (G) which is spread out two-dimensionally in a plane and which, for each proposed location (l ij ) are assigned the same fixed height in order to configure the grid (G) three-dimensionally.

12. The method according to any one of claims 8 to 11, wherein: The object search is performed with the aid of a trained decoder (TD), in particular a transformer decoder having a regression head (DA) and a classification head (DA).

13. The method according to claim 12, wherein: The corresponding predicted bounding box (b ij ) is predicted using the regression head.

14. The method according to claim 12, wherein: The position of a respective location from the second number of locations is ascertained from a bounding box of the surroundings from the first plurality of suggested locations, in particular determined by a center of the bounding box.

15. A method for object detection, comprising the steps of: detecting sensor data in the image section by means of a sensor device and forming a corresponding feature vector; extracting feature vectors from feature vectors formed in the surroundings of a plurality of proposed locations, generating a corresponding estimated bounding box for each of the proposed locations in the surroundings of the proposed locations according to the extracted feature vectors, and calculating a corresponding confidence level for the estimated bounding box; reducing the plurality of suggested locations to a number of locations less than the plurality of suggested locations based on corresponding calculated confidence levels; and An object is identified based on a search for the object at the second number of locations.

Images