Method for partitioning a distributed detection of three-dimensional objects

Abstract

Method for partitioning a distributed detection of three-dimensional objects from acquired data, wherein a neural network (200) is used which is distributed across at least one sensor computing node (310) and one aggregation computing node (320), each sensor computing node (310) is assigned at least one sensor which acquires data from its environment, the sensor computing nodes (310) forward evaluated data to the aggregation computing node (320), and the partitioning is carried out on a common three-dimensional representation.

Description

[0001] The invention relates to a method for partitioning a distributed detection of three-dimensional objects and a neural network suitable for carrying out the method. State of the art

[0002] The method presented here is particularly applicable in the field of automated driving. Automated driving requires the detection or perception of the vehicle's surroundings. Cameras are typically used as sensors for this purpose, with which the surroundings are recorded in order to detect relevant objects in the environment from the recorded data, particularly image data. Known methods for detecting three-dimensional objects from multi-sensor / multi-view data transform abstract representations of data from each individual sensor into a common or unified three-dimensional representation, which is then collected or aggregated and used to predict three-dimensional objects.

[0003] These methods typically run on a single machine or on a single compute node. With a growing number of sensors involved, methods that perform three-dimensional multi-sensor / multi-view object detection on a single compute node face significant limitations due to the volume of incoming raw sensor data, a lack of redundancy, and inflexible scalability.

[0004] A well-known method for distributed execution across multiple compute nodes, primarily used to reduce latency in edge devices and cloud deployments, is distributed DNN (deep neural network) inference. DNN model partitioning typically occurs at a low level of abstraction, e.g., by splitting the sensor-space input data of a single sensor based on a dynamic load balancing mechanism and processing each part on a different compute node.

[0005] This can mean, for example, that an image is cut into different patches, and each patch is processed on a separate compute node with a certain degree of inter-computational communication to ensure accurate results. Furthermore, rudimentary distributed inference strategies exist for classification methods of multi-sensor / multi-view data, but none for three-dimensional object detection, where low-level sensor space features are aggregated across multiple compute nodes before being transformed into a three-dimensional representation.

[0006] Applying low-level DNN partitioning and multi-node distribution mechanisms known from the state of the art enables scaling, but the sensor space where the partitioning takes place lacks sufficient semantic meaning. Meaningful semantic redundancy and scalability are not achieved, making correct system design difficult. Disclosure of the invention

[0007] Against this background, a method according to claim 1 and a neural network according to claim 9 are presented. Embodiments emerge from the dependent claims and from the description.

[0008] A method for partitioning a distributed detection of three-dimensional objects from acquired sensor data is presented, wherein a neural network is used which is distributed across at least one sensor computing node and one aggregation computing node, each sensor computing node is assigned at least one sensor which acquires data from its environment, the sensor computing nodes forward evaluated data to the aggregation computing node, and the partitioning is performed on the common or unified three-dimensional representation.

[0009] A neural network is a network of artificial neurons and has a natural biological model. Natural neural networks represent a network of neurons in the nervous system. A deep neural network (DNN) mimics the functioning of the human brain.

[0010] The neural network presented here is designed, for example, as a deep neural network and is configured to carry out a method of the type described here.

[0011] A method for partitioning a distributed execution of three-dimensional object detection is presented that provides a direct link between partitioning and spatial regions, thus delivering a higher, more human-understandable meaning. This enables better, more targeted, robust, and scalable system-level designs based on three-dimensional geometry. It also allows for bandwidth savings.

[0012] It is proposed to base the method for detecting three-dimensional objects directly on their common or unified three-dimensional representation (230 in Fig. 1), in which the individual sensor representations are aggregated, partitioned, and distributed. Thus, three-dimensional representations that are structurally similar to the real three-dimensional world and thus have a human-understandable geometric meaning at a higher level of abstraction (high level) are transferred between the compute nodes. In practical implementation, this leads to a partitioning into sensor compute nodes that process sensor data and transform it into a three-dimensional representation. All three-dimensional representations are then sent to an aggregation compute node, which calculates the common or unified three-dimensional representation. Exploiting the geometric and mathematical quantities or properties of the three-dimensional representation offers advantages with regard to partitioning, such as: A) Easy system scalability is achieved both for additional sensors per sensor compute node and for additional sensor compute nodes connected to the aggregation compute node. In both cases, the partitioning, including the interfaces, of the rest of the distributed system remains unaffected. B) An explicit resilience design, when multiple overlapping sensors are given across different compute nodes and the same area in the three-dimensional world is considered, is enabled. C) Fixed partitioning of the algorithm or load and fixed-size data blocks transferred between compute nodes, instead of dynamic load balancing or variable-size data blocks, enable real-time processing. D) Implicit bandwidth savings are achieved because the three-dimensional representation is smaller than raw sensor data or features at a low abstraction level or can be made smaller than such. E) Explicit bandwidth savings between compute nodes are achieved by aggregating the three-dimensional representations from all sensors on the respective sensor compute node and / or by transmitting only the parts of the three-dimensional representation within the field of view of the contributing sensors. F) Training DNN-based three-dimensional object detection methods as a whole (end-to-end) is possible without having to consider the proposed partitioning methods at training time. Partitioning can be applied retrospectively.

[0013] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

[0014] It is understood that the features mentioned above and those to be explained below can be used not only in the combination specified in each case, but also in other combinations or on their own, without departing from the scope of the present invention. Short description of the drawings Fig. 1 shows a block diagram of a basic processing chain for a method for detecting three-dimensional objects. Fig. 2 shows a block diagram of a partitioning for a distributed execution of a method for detecting three-dimensional objects. Fig. 3 shows a block diagram of a partitioning for a distributed execution of a method for detecting three-dimensional objects with a pre-aggregation of the three-dimensional representations on a sensor computing node. Fig. Figure 4 shows an exemplary three-dimensional representation using a grid from a bird's eye view. Embodiments of the invention

[0015] The invention is illustrated schematically in the drawings using embodiments and is described in detail below with reference to the drawings.

[0016] Fig. Figure 1 shows a block diagram of a basic processing chain for a method for detecting three-dimensional objects. The diagram shows sensor input data 110, three-dimensional objects 120, a neural network 200 for detecting three-dimensional objects, a sensor feature encoder 210, a unit 200 for transforming sensor features into a three-dimensional representation, an aggregation 230 of the three-dimensional representations from multiple sensors, and a decoder 240 for three-dimensional representations.

[0017] What follows is a detailed description based on the exemplary use case of automated driving with a sensor configuration consisting of multiple cameras capturing the world from different angles or views, using a DNN-based three-dimensional object detector.

[0018] The method is not limited to this specific use case. It also applies to sensors that are not cameras, such as lidar, radar, ultrasound, etc. The method is also applicable to sensor configurations designed for other applications that aim to detect the environment, such as in robotics in a more general sense or in surveillance systems with spatially distributed sensors. The method is particularly applicable to sensors whose fields of view overlap.

[0019] It should also be noted that transmitted data blocks can have a fixed size.

[0020] The basic signal and processing chain of a non-distributed three-dimensional object detection method is described in Fig. 1. One or more camera images as sensor input data 110 are input to the three-dimensional object detection network 200, which predicts three-dimensional objects 120 as output, e.g., an enclosing box. The sensor input data 110 is each provided by a sensor, e.g., a camera.

[0021] Each camera image is encoded into low-level image-space features using the sensor feature encoder 210 and subsequently transformed from image space into a three-dimensional spatial representation using the unit 220. Then, three-dimensional per-camera representations from multiple cameras are aggregated into a common or unified three-dimensional representation 230, which is subsequently decoded into a unified list of three-dimensional objects using the decoder 240. Beyond a list of three-dimensional objects, the three-dimensional representation can also be the basis for other detection tasks and object representations, such as segmentation, instance masking, or spatial occupancy, where the proposed method is also applicable.

[0022] The proposed partitioning strategy for a distributed execution of the three-dimensional object detection method is presented in Fig. 2 shown.

[0023] Fig. Figure 2 shows a neural network 150 for the detection of three-dimensional objects, which is distributed across sensor compute nodes 310, namely compute node A 310a and compute node B 310b, and an aggregation compute node 320. Optionally, a sensor compute node 310 can also assume the role of the aggregation compute node 320 simultaneously. There can also be more than one aggregation compute node 320, for example, to provide redundancy. The number of sensor compute nodes 310 is arbitrary and can vary freely. On the hardware side, a compute node can be represented, for example, by a chip, a dedicated computing accelerator, or a complete computer system. Partitioning is proposed after transforming from the sensor space into the three-dimensional representation by means of the unit 220, and the three-dimensional representation data 115 are sent from the sensor computing node 310 to the aggregation computing node 320.

[0024] If multiple cameras are assigned to a sensor compute node 310b, as in Fig. 3 (compute node B), the three-dimensional representations 230 for these cameras may already be pre-aggregated on the sensor compute node 310b before being sent to the aggregation compute node 320 to save bandwidth. The number of sensors per sensor compute node 310 is arbitrary and may vary from one sensor compute node 310 to another. Sensor types / modalities may also be arbitrarily mixed, even on a sensor compute node 310. From a mathematical perspective, pre-aggregation requires only one associative aggregation function, such as sum, average, minimum, or maximum, which are common choices.

[0025] The unified three-dimensional representation 230 may have less memory requirements than raw sensor data 110 or lower level features.

[0026] It may further be provided that only parts of the unified three-dimensional representation 230 are transmitted to the aggregation computing node 230.

[0027] To leverage some of the advantages of the proposed partitioning strategy, certain other extensions and properties of the three-dimensional representation 220 and aggregation functions become essential. The following explanations and details assume, by way of example, that the three-dimensional spatial representations by unit 220 are a projection into a two-dimensional Euclidean grid from a bird's-eye view, as shown in Fig. 4. However, the concept can also be applied to other three-dimensional spatial representations, such as various projections, voxels, and polar representations.

[0028] Fig.4 shows an exemplary three-dimensional representation 400 of a vehicle environment in a bird's eye view with a number of cells 402. The displayed extent, the number of cells and the objects, in this case vehicles 404, are arbitrary.

[0029] The advantages mentioned above are discussed in more detail below: A) The numerical value range of the individual elements in the bird's-eye view grid is invariant to the number n of cameras involved. How this can be achieved depends on the aggregation function used. The minimum or maximum function, for example, is inherently invariant for this. If the average of the representations R i is used, a weight w i , which represents the number of sensors involved in this grid in bird's eye view, along with each R i sent so that: Aggregated=∑iRi / ∑iwi is invariant in the numerical scaling to n. All valid R i on a sensor can also be summed there. Such aggregation benefits the sensor / compute node failure resilience (B) as well as the scalability of the system if more compute nodes or sensors are added. B) Since the geometric structure of the three-dimensional representation is similar to the relevant part of the three-dimensional world, system resilience and redundancy can be directly designed to meet certain criteria in an obvious and understandable way. By processing all data from a sensor on a sensor compute node, the field of view and redundancy considerations are directly linked to the sensor compute nodes. Combined with a suitable aggregation function, as described under point A, redundancy and thus increased resilience can be added to the system without changes to the overall partitioning, interfaces, and system design. For example, if the area in front of the ego vehicle of a If the automated driving system should be redundantly covered with independent cameras and compute nodes, this criterion can be easily reflected in the system design using the proposed partitioning strategy: There are two groups, each with an appropriate number of cameras, with both groups covering the relevant field of view and thus providing sensor redundancy. Each group is assigned to its own sensor compute node, whose outputs are connected to one or more aggregation compute nodes. This ensures coverage in the event of problems with both a sensor and a compute node.

[0030] Low-level partitioning mechanisms do not offer such easy traceability due to the distributed sensor data of each individual sensor across compute nodes. Thus, a compute node failure can affect not only known three-dimensional regions, but any set of outputs, or all outputs. It should also be noted that the proposed partitioning allows for multiple instances of the aggregation compute node to be distributed across independent hardware to increase resilience, if necessary. Sensor compute node outputs simply need to be sent to all aggregation compute nodes in parallel. C) Known distributed DNN inference methods for edge and cloud computing use cases, e.g., consisting of smartphone applications and cloud servers, attempt to reduce latency through adaptive strategies, whereby variable portions of data are sent from the edge devices to the cloud. While this offers flexibility in dynamically changing systems, e.g., by varying data bandwidth or computing power, it is not feasible for real-time applications. Thus, the proposed partitioning is based on fixed-size data blocks, where the size of the transferred data and the computational steps to be executed on a specific compute node are known and specified in advance. This is the basis for real-time applications where predictability is required. D) In ​​automated driving applications, raw sensor data, especially camera data, tends to be significantly large due to their high resolution. Consequently, low-level features are also very large, as their size increases linearly with sensor resolution. Thus, known partitioning strategies that transmit high-resolution raw sensor data or low-level features require significant bandwidth between compute nodes. In contrast, the size of the bird's-eye view grid is independent of sensor resolution and can be designed according to the needs of the object detection functionality, making it smaller than raw data or low-level data. This leads to reduced bandwidth requirements between compute nodes. E) If a suitable aggregation function is used, as described in A, all bird's-eye view grids of the cameras assigned to a sensor node can be aggregated in advance, and the amount of data transferred to the aggregation node is reduced and remains fixed, e.g., independent of the number of cameras assigned to the sensor node. Furthermore, the bird's-eye view representation in device memory typically allows efficient access to specific patches, e.g., a rectangular region representing a specific area of ​​the world. Thus, only the patch covered by the field of view of the participating cameras needs to be transferred to the aggregation compute node, along with a small amount of metadata describing the patch's location within the complete bird's-eye view grid. More elaborate and wider-cropped regions than rectangular regions are possible, but must be considered in light of the increased amount of required metadata describing their shape. If real-time capability is important and for sensor configurations with a fixed field of view within the three-dimensional representation, the metadata can be communicated to the aggregation compute node prior to online execution to avoid additional bandwidth consumption. F) Model training of DNN-based three-dimensional object detection methods is typically already a non-trivial task. The proposed partitioning strategy has no retroactive influence on DNN training and therefore prevents increased complexity.

Claims

[1] Method for partitioning a distributed detection of three-dimensional objects from acquired data, wherein a neural network (200) is used which is distributed over at least one sensor computing node (310) and one aggregation computing node (320), each sensor computing node (310) is assigned at least one sensor which collects data from its environment, the sensor computing nodes (310) forward evaluated data to the aggregation computing node (320), and the partitioning is done on a common three-dimensional representation. [2] Method according to claim 1, wherein a plurality of sensors are connected to at least one sensor computing node (310). [3] Method according to claim 1 or 2, which is carried out in a motor vehicle which is arranged for automated operation. [4] A method according to claim 1 or 2, which is carried out in a field selected from a group consisting of: robotics, surveillance systems. [5] Method according to one of claims 1 to 4, wherein the sensors are selected from a group consisting of: camera, lidar sensor, radar sensor, ultrasonic sensor. [6] Method according to one of claims 1 to 5, in which sensors are used whose fields of view overlap. [7] Method according to one of claims 1 to 6, wherein the detection is carried out according to the partitioning carried out. [8] Method according to one of claims 1 to 7, wherein the detection is carried out in real time. [9] Method according to one of claims 1 to 8, wherein the detected objects are shown in a representation (400) in a bird's eye view. [10] Method according to one of claims 1 to 9, which is carried out in a deep neural network. [11] Neural network which is distributed over at least one sensor computing node (310) and one aggregation computing node (320) and is arranged to carry out a method according to one of claims 1 to 10. [12] Neural network according to claim 11, which is designed as a deep neural network.

Images