Methods for fusing sensor data in the context of artificial neural networks

Abstract

The present invention relates to a method and system (10) for fusing data from at least one sensor (1). The method comprises: a) receiving input sensor data, the input sensor data comprising: a step (S1) including a first representation (401,701) including a first region (101,601) of the scene and a second representation (502,702) including a second region (102,602) of the scene, the first and second regions overlapping each other but not being identical; b) determining (S2) a first feature map (1301) having a first height and width based on the first representation (401, 701), and determining (S3) a second feature map (1302) having a second height and width based on the second representation (502, 702); c) computing (S4) a first output feature map (1321) using a first convolution of the first feature map (1301), and computing (S5) a second output feature map (1322) using a second convolution of the second feature map (1302); d) computing a fused feature map (1330) by element-wise summing of the first and second output feature maps (1321, 1322), where the positions of the first and second regions are taken into account relative to one another such that elements of overlapping regions are summed; and e) outputting (S8) a fused feature map (1330). The method is very run-time efficient and can be used to fuse data from one or more surrounding environment sensors for a vehicle's ADAS / AD systems.

Description

[Technical Field]

[0001] The present invention relates to a method and system for fusing sensor data in the context of artificial neural networks, for example in environment-based ADAS / AD systems for vehicles. [Background technology]

[0002] The resolution of environmental sensors (especially camera sensors) for ADAS / AD systems is constantly increasing. This makes it possible to detect small objects, detect sub-objects, and, for example, read small characters at a great distance. The disadvantage of high resolution is the significantly higher computing power required to process the correspondingly large sensor data. Therefore, various resolution levels of the sensor data are often used for processing. For example, a large range or high resolution is often required in the center of the image, while not in the peripheral areas (e.g., similar to the human eye).

[0003] Patent document 1 describes a camera device for imaging the surrounding environment for a vehicle, comprising an image sensor device for capturing pixel images and a processing device configured to combine adjacent pixels of the pixel images in an adapted pixel image. By combining pixel values ​​of adjacent pixels in the form of a 2x2 image pyramid or an nxn image pyramid, different adapted pixel images can be generated at different resolutions.

[0004] US Pat. No. 5,629,999 and US Pat. No. 5,629,999 describe driver assistance systems that take images with variable resolution.

[0005] Patent Document 4 describes a camera device for capturing the surrounding area of ​​a vehicle, which includes a wide-angle optical element and a high-resolution image capturing sensor. For the images of the image sequence, pixel binning can be used to capture an image of the entire capture area with reduced resolution, or a partial area of ​​the capture area with full resolution.

[0006] Artificial neural network technologies are increasingly being used in environmental sensor-based ADAS / AD systems to better recognize, classify, and at least partially understand road users and scenes. Deep neural networks, such as convolutional neural networks (CNNs), have clear advantages over traditional approaches, which rely on trained classifiers, such as support vector machines or AdaBoost, to extract features that are relatively hand-designed (e.g., histograms of oriented gradients (HOGs), local binary patterns (LBPs), Gabor filters, etc.). In the case of (multi-layer) CNNs, feature extraction is performed by machine (deep) learning algorithms, which significantly increase the dimensionality and depth of the feature space, ultimately resulting in significantly improved performance, e.g., higher detection rates.

[0007] A particular challenge is the process of integrating sensor data with different and even overlapping capture areas and different resolutions.

[0008] Patent Document 5 describes a CNN-based method for learning parameters of an object detector. An object region is estimated in a camera image, and excerpts of this region are generated from various image mirror levels. The excerpts, for example, have the same height, are padded on the sides with "zero padding" (zero regions), and are concatenated. This form of concatenation can be roughly described as a collage, where excerpts of the same height are "pasted together." The generated composite image is thus a composite of regions from the same original camera image at various resolution levels. The CNN is trained so that the object detector can detect objects based on the composite image, thereby enabling detection of even more distant objects. The advantage of such an approach, as opposed to processing each image region separately one after the other using a CNN, is that the weights only need to be applied once for the composite image. The disadvantage in this case is that adjacent image regions in the composite image are considered, in particular, independently of each other, using object detectors by CNNs: objects present in overlapping regions, which may be incompletely included in the image regions, must be identified in a non-trivial way as belonging to one and the same object. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] German Patent Application Publication No. 102015208889 [Patent Document 2] U.S. Patent No. 10,742,907 [Patent Document 3] U.S. Patent No. 10,757,330 [Patent Document 4] U.S. Patent No. 1,0798,319 [Patent Document 5] European Patent Application Publication No. 3686798 [Patent Document 6] German Patent Application Publication No. 102020204840 Summary of the Invention [Problem to be solved by the invention]

[0010] It is an object of the present invention to provide an improved sensor data fusion method in the context of an artificial neural network that efficiently fuses input sensor data having different capture areas and different resolutions and provides them for further processing. [Means for solving the problem]

[0011] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. a) Capture a large image area; b) capturing a relevant image region, e.g., a remote object in the center of the image, at high resolution; The present invention relates to an efficient implementation of object detection based on input data of at least one image capture sensor.

[0012] Before explaining the solution, the following considerations are made.

[0013] Using multiple levels of an image pyramid in a neural network, the low-resolution overview image and the high-resolution central extract image can be processed separately by two independent inferences (two CNNs trained for each). This implies a large computational / runtime cost: in particular, the weights of the trained CNN need to be multiplied anew for each different image, and features from different pyramid levels are not considered jointly.

[0014] Alternatively, the process of Patent Document 5 can be performed on a combined image from various resolution levels. Thus, a combined image from various sub-images / resolution levels is generated, and inference or a trained CNN is performed on this. This is somewhat more efficient because each weight needs to be applied only once for all images, rather than anew for each sub-image. However, other disadvantages remain, such as the lack of combination of features from various resolution levels.

[0015] Methods for fusing sensor data include: a) receiving input sensor data, the input sensor data comprising: a first representation including a first region of the scene; and - a second representation including a second region of the scene, the first and second regions overlapping each other but not being identical; b) determining a first feature map having a first height and width based on the first representation, and determining a second feature map having a second height and width based on the second representation; c) computing a first output feature map using a first convolution of the first feature map, and computing a second output feature map using a second convolution of the second feature map; d) computing a fused feature map by element-wise summing the first and second output feature maps, where the positions of the first and second regions are taken into account relative to each other so that elements (of the first and second output feature maps) of overlapping regions are summed; e) outputting a fused feature map.

[0016] The representation may be, for example, a two-dimensional representation of the scene, as captured by a sensor, or may be, for example, a grid, a map or an image. A point cloud or depth map is an example of a three-dimensional representation that may be captured, for example, by a lidar sensor or a stereo camera, and may be converted to a two-dimensional representation for many purposes, for example, by planar cross-section or projection. The feature maps may be determined from the representation or other (existing) feature maps by convolution or convolution layers / convolution kernels. The height and width of the feature map are operationally related to the height and width of the underlying representation (or input feature map).

[0017] The relative positions of the first and second regions are specifically taken into account in order to add appropriate elements of the first and second output feature maps for fusion. The position of the overlap region is determined, for example, by a starting value (x s ,y s ) In the overlap region, the elements of the first and second output feature maps are added. Outside the overlap region, the elements of the output feature maps that cover this region can be moved to the fused feature map. If neither of the output feature maps covers a region of the fused feature map, this region can be filled with zeros.

[0018] The method can be implemented, for example, in the context of an artificial neural network, preferably a convolutional neural network (CNN). ADAS / AD functions often (especially on the perception side) use at least one artificial neural network or CNN that is trained using machine learning methods to assign sensor input data to output data related to the ADAS / AD function. ADAS stands for Advanced Driver Assistance Systems and AD stands for Automated Driving.

[0019] The trained artificial neural network can be implemented in a processor of an ADAS / AD controller in the vehicle. The processor can be configured to evaluate the sensor data with the trained artificial neural network (inference). The processor can include a hardware accelerator for the artificial neural network.

[0020] The processor or inference can be configured to detect or further determine ADAS / AD-relevant information, for example, from input sensor data of one or more environmental sensors. The relevant information is, for example, object and / or environmental information for an ADAS / AD system or an ADAS / AD controller. The ADAS / AD-relevant object and / or environmental information represents input variables important to the ADAS / AD system, such as objects, landmarks, road signs, traffic participants, distances to objects, relative speeds of objects, etc. Examples of functions for detecting relevant information are lane detection, object detection, depth perception (3D estimation of image components), semantic detection, traffic sign detection, etc.

[0021] In one embodiment, the first and second output feature maps have the same height and width in the overlap region. In other words, adjacent elements in the overlap region of these output feature maps are equidistant from each other in real space. This is therefore a possible case since the first and second feature maps already have the same height and width in the overlap region. For example, the first and second regions (also) have the same height and width.

[0022] According to one example embodiment, the height and width of the fused feature map are determined by a rectangle that encloses (exactly contains) the first and second output feature maps.

[0023] In one embodiment, the first region is an overview region of the scene and the second region is a subregion of the overview region of the scene. The overview region included in the first representation may correspond to the entire region, i.e., the maximum capture area of ​​the sensor. The subregion of the scene included in the second representation may correspond to a Region of Interest (ROI) that is also included in the first representation.

[0024] According to one example embodiment, the first representation has a first resolution and the second representation has a second resolution. For example, the second resolution is higher than the first resolution. The resolution of the second representation may correspond to the maximum resolution of the sensor. For example, the higher resolution may provide more detail regarding a subregion or region of interest that is the content of the second representation. The resolution of the representation may correspond to the accuracy or data depth, for example the minimum distance between two adjacent data points of the sensor.

[0025] In one embodiment, after the height and width of the fused feature map are determined by a rectangle that encloses (exactly contains) the first and second output feature maps, the first and / or second output feature maps can be scaled or adjusted so that the first and / or second output feature maps reach the width and height of the fused feature map, while maintaining the positions of the first and second output feature maps relative to each other. The overlapping regions are in the same position in both adjusted output feature maps. The newly added regions of each (adjusted) output feature map due to the scaled-up operation are filled with zeros (zero padding). Both adjusted output feature maps can then be added element-by-element.

[0026] According to one example embodiment, a template output feature map is created whose width and height are derived from the height and width of the first and second output feature maps and the location of the overlap region (see the previous paragraph, bounding rectangle). The template output feature map is filled with zeros.

[0027] For the adjusted first output feature map, elements from the first output feature map in the region covered by the first output feature map are carried over, and starting values ​​can be used to indicate the vertical and horizontal position of the first output feature map within the template output feature map. Correspondingly, an adjusted second output feature map is constructed, and both adjusted output feature maps can then be added element-wise.

[0028] In one embodiment, for the special case where the second output feature map includes the entire overlap region (i.e., the subregion of the first output feature map that includes the overlap region), the adjustment for the different heights and widths of the second output feature map can be omitted. In this case, no adjustment is made to the first output feature map because the fused feature map has the same height and width as the first output feature map. In this case, element-by-element addition of the second output feature map with the first output feature map may be performed only in the overlap region based on a preferred starting value. The starting value predetermines the base point within the first output feature map (i.e., in the overlap region) from which elements of the second output feature map are added to elements of the first output feature map to generate the fused feature map.

[0029] In one embodiment, the feature maps have a depth that depends on the resolution of the representation: a high resolution representation (e.g., an excerpt image) corresponds to a feature map with a high depth, e.g., the feature map contains more channels. For example, the processor may include a hardware accelerator for an artificial neural network that can further process a stack of sensor channel data "packets" during a clock or computation period. Stacks of sensor data or representations or features (maps) may be provided in the hardware accelerator as stacked sensor channel data packets.

[0030] According to one example embodiment, detection of ADAS / AD related features is based on a fused feature map.

[0031] In one embodiment, the method is implemented in a hardware accelerator for artificial neural networks or CNNs.

[0032] According to one example embodiment, the fused feature map is generated in an encoder of an artificial neural network or CNN that is configured or trained to determine ADAS / AD-related information.

[0033] In one embodiment, an artificial neural network or CNN configured or trained to determine ADAS / AD-related information comprises multiple decoders for various ADAS / AD detection functions. In one embodiment, the representation (of the scene) includes or contains image data from an image capture sensor, which may comprise one or more representative elements from the group of monocular cameras, stereo cameras, satellite cameras, individual cameras in omnidirectional systems, lidar sensors, laser scanners or other three-dimensional cameras, particularly with wide-angle capture areas (e.g., 100° or more) and large peak resolutions (e.g., 5 megapixels or more).

[0034] According to one example embodiment, the first and second representations include image data from at least one image capture sensor.

[0035] In one embodiment, the (only) image capture sensor is a monocular camera. The first and second representations may also be provided by the (same) image capture sensor. The first representation (or first image) may correspond to an overview image with reduced resolution captured at a wide angle, and the second representation (or second image) may correspond to a partial image with higher resolution.

[0036] According to one example embodiment, the first and second images correspond to different image pyramid levels of the image captured by the image capture sensor. Depending on the resolution, the input sensor data, i.e., the input image data, may be or may be encoded into multiple channels, e.g., each channel having the same height and width. In this case, within each channel, the spatial relationship of the pixels involved may be maintained. For further details in this regard, reference is made to US Pat. No. 6,229,699, the contents of which are incorporated herein in their entirety.

[0037] In one embodiment, the one or more image capture sensors are two monocular cameras with overlapping capture areas. The two monocular cameras may be components of a stereo camera. The two monocular cameras may have different aperture angles and / or resolutions ("hybrid stereo camera"). Both monocular cameras may be satellite cameras mounted independently on the vehicle.

[0038] According to one example embodiment, multiple cameras of an omnidirectional camera system are used as image capture sensors. For example, four monocular cameras with fisheye lenses (capture angles of, for example, 180° or more) can capture the entire surroundings of the vehicle. Every two adjacent cameras have an overlapping area of ​​about 90°. In this case, a fused feature map of the 360° surroundings of the vehicle can be created from four separate images (four representations).

[0039] A further aspect of the invention relates to a system or apparatus for fusing sensor data, the apparatus comprising an input interface, a data processing unit and an output interface. An input interface is configured to receive input sensor data, the input sensor data including first and second representations, the first representation including or containing a first region of the scene. The second representation contains a second region of the scene. The first and second regions overlap each other. The first and second regions are not identical. The data processing unit is configured to perform the following steps b) to d). b) determining a first feature map having a first height and width based on the first representation, and determining a second feature map having a second height and width based on the second representation; c) computing a first output feature map using a first convolution of the first feature map and a second output feature map using a second convolution of the second feature map. d) computing a fused feature map by element-wise summing the first and second output feature maps, wherein the element-wise summing takes into account the positions of the first and second regions relative to each other, such that elements (of the first and second output feature maps) of overlapping regions are summed. An output interface is configured to output the fused feature map. The output may be to a downstream ADAS / AD system or to downstream layers of a CNN or further artificial neural network for a "big" ADAS / AD.

[0040] According to one example embodiment, the system includes a CNN hardware accelerator, wherein the input interface, the data processing unit, and the output interface are implemented in the CNN hardware accelerator.

[0041] In one embodiment, a system includes a convolutional neural network having an encoder, wherein the input interface, the data processing unit, and the output interface are implemented in the encoder such that the encoder is configured to generate a fused feature map.

[0042] According to one example embodiment, a convolutional neural network includes multiple decoders configured to implement various ADAS / AD detection functions based at least on the fused feature map. Thus, multiple decoders of the CNN can use input sensor data encoded by a common encoder. The various ADAS / AD detection functions may be, for example, semantic segmentation of one or more representations, free space detection, lane detection, object detection, or object classification.

[0043] In one embodiment, the system includes an ADAS / AD controller configured to implement the ADAS / AD sensing function based at least on a result of the ADAS / AD sensing function.

[0044] The system may include at least one sensor, such as one or more camera sensors, radar sensors, lidar sensors, ultrasonic sensors, localization sensors, and / or V2X systems (Vehicle to X, i.e., telematics systems).

[0045] A further aspect of the invention relates to a vehicle comprising at least one sensor and a corresponding system for fusing sensor data.

[0046] The system or data processing unit may comprise, among others, a microcontroller or microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural / AI processing unit (NPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., and software for performing the corresponding method steps.

[0047] According to one embodiment, the system or data processing unit is implemented in a hardware-based sensor data pre-processing stage (eg, an Image Signal Processor (ISP)).

[0048] The invention also relates to a computer program element or program product which, when configured to run a data fusion program on a processor of the system, instructs the processor to perform a method for fusing input sensor data corresponding thereto.

[0049] The present invention also relates to a computer-readable storage medium having such program elements stored thereon.

[0050] Accordingly, the invention may be implemented in digital electronic circuitry, in computer hardware, firmware, or software.

[0051] In the following, exemplary embodiments and drawings will be described in the context of the present invention. [Brief explanation of the drawings]

[0052] [Figure 1] FIG. 1 shows a system for fusing data from at least one sensor. [Figure 2] FIG. 2 shows generally the extents and positions of first and second capture areas of a sensor or of two different sensors, from which first and second representations of a scene can be determined. [Figure 3] FIG. 3 shows the overall image with high resolution. [Figure 4] FIG. 4 shows an overview or overview image with reduced resolution. [Figure 5] FIG. 5 shows the central excerpt image with high resolution. [Figure 6] FIG. 6 shows an alternative arrangement of the first (general) capture area and the second central capture area. [Figure 7] FIG. 7 shows an example of how the corresponding digital image would appear as a grayscale image. [Figure 8] FIG. 8 shows a basic method for fusing such images. [Figure 9] FIG. 9 shows a second alternative fusion method. [Figure 10] FIG. 10 illustrates a third advantageous fusion method. [Figure 11] FIG. 11 shows the concatenation of two feature maps that are then processed (and thereby fused) by a convolution kernel. [Figure 12] FIG. 12 shows an alternative flow in which two feature maps are processed by two separate convolution kernels followed by element-wise addition. [Figure 13] Figure 13 shows the flow for fusing two feature maps with different widths and heights. [Figure 14] FIG. 14 shows a possible method flow. DETAILED DESCRIPTION OF THE INVENTION

[0053] FIG. 1 shows schematically a system 10 for fusing data of at least one sensor 1, comprising an input interface 12, a data processing unit 14 having a fusion module 16, and an output interface 18 for outputting fused data to a further unit 20.

[0054] An example of the sensor 1 is a monocular camera sensor having wide-angle optics and a high-resolution image capture sensor, such as a CCD or CMOS sensor. Further examples of the sensor 1 may be a radar sensor, a lidar sensor, an ultrasonic sensor, a localization sensor, or a V2X system. In many cases, the sensors have different resolutions and / or capture areas. Data preprocessing is useful for fusion, allowing for the fusion of features from the sensor data.

[0055] An example embodiment described in more detail below relates to the processing of a first image of a camera sensor and a second image of the camera sensor, the second image comprising (only) a partial region of the first image and having a higher resolution compared to the resolution of the first image. Based on the image data of the camera sensor, a number of ADAS or AD functions, such as lane detection, lane keeping assist, traffic sign recognition, speed limit assist, traffic participant detection, collision warning, emergency braking assist, following distance control, construction site assist, highway pilot, cruising chauffeur function and / or autopilot, can be provided by the ADAS / AD controller, for example to a further unit 20.

[0056] The overall system 10, 20 may include an artificial neural network, e.g., a CNN. To enable the artificial neural network to process image data in real time, e.g., in a vehicle, the overall system 10, 20 may include a hardware accelerator for the artificial neural network. Such a hardware component can essentially dedicatedly accelerate a software-implemented neural network so that the neural network can operate in real time.

[0057] The data processing unit 14 can process image data in a "stacked" format, i.e., it can read and process a stack of multiple input channels within a computational cycle (clock cycle). In a specific example, the data processing unit 14 can read four image channels with a resolution of 576x320 pixels. Fusing at least two image channels provides an advantage for subsequent CNN detection in that the already fused channel information or feature map can be processed by the CNN, rather than having to process the channels individually by their corresponding CNNs. Such fusion can be performed by the fusion module 16. Details of fusion are described in more detail below with reference to the following drawings.

[0058] The fusion can be performed in an encoder of the CNN. The fused data can then be processed by one or more decoders of the CNN to obtain detection or other ADAS / AD-related information. In such a configuration, the encoder can be represented by block 10 and the one or more decoders can be represented by block 20 in FIG. 1. The CNN will comprise blocks 10 and 20, hence the notation "whole system."

[0059] Figure 2 shows schematically the extents and positions of a first capture area 101 and a second capture area 102 of a sensor or two different sensors, from which a first and second representation of a scene can be determined. In the case of a camera sensor, this corresponds to a first image capture area 101 from which an overview or overall image can be captured as the first representation, and a second image capture area 102, e.g., a central image area for the second representation, containing an excerpt of the first image capture area 101. Figures 3-5 show an example in which images can be captured using a camera sensor.

[0060] 3 shows a schematic overview or full image 300 with high resolution. A scene is captured including nearby and more distant traffic participants (304 and 303) on a road 305 or roadway passing by a house 306. A camera sensor can capture such an overview image with maximum width, height, and resolution (or number of pixels). However, processing this large amount of data (e.g., in the 5-10 megapixel range) is generally not possible in real time in AD or ADAS systems, resulting in further processing of reduced image data.

[0061] 4 shows a schematic representation of an overview or summary image 401 with reduced resolution. When the resolution is halved, the number of pixels drops to one-fourth. The summary image 401 with reduced resolution is referred to below as a wfov (wide field of view) image. Nearby traffic participants 404 (vehicles) can be detected from the wide field of view image even in the case of reduced resolution. On the other hand, distant traffic participants 403 (pedestrians) cannot be detected from this wide field of view image due to the limited resolution.

[0062] 5 shows a schematic representation of a central excerpt image with high (or highest) resolution 502. The excerpt image with high resolution 502 will be referred to as the central image in the following. The central image allows for high-resolution detection of a remote pedestrian 503, while a nearby vehicle 504 is not included or is only partially included in the capture area of ​​the central image 502.

[0063] 6 shows an alternative arrangement of a first (overview) capture area 601 and a central capture area 602. This central capture area 602 is located "below," meaning that it starts vertically at the same height as the overall capture area 601. The start values ​​(x0, y0) can indicate the position of the central capture area 602 in the horizontal and vertical directions within the overall or overview capture area.

[0064] Figure 7 shows an example of how the corresponding digital image appears as a grayscale image. At the bottom, a first image, a wide-field image 701 captured by the vehicle's front camera, can be seen. The vehicle is traveling toward a road intersection. A large, possibly multi-lane road runs perpendicular to the direction of travel. A bicycle path runs parallel to the large road. Traffic signals control the right-of-way of traffic participants. Buildings and trees demarcate the road and sidewalks.

[0065] The central excerpt 702 is shown faded in the wide-field image 701 to illustrate that this excerpt exactly matches this excerpt 702 of the first image 701 as a second image (central image) 7020 with higher resolution. The second image 7020 is shown on top, where it is easier for a human observer to recognize that the traffic light is showing a red light to the egocentric vehicle, that the bus has just crossed the intersection from left to right, and other details of the captured scene. Based on the higher resolution of the second image 7020, even more distant objects or traffic participants can be robustly detected by image processing.

[0066] The image pyramid may have, for example, 2304x1280 pixels at the highest level for the second (center) image, 1152x640 pixels at the second level, 576x320 at the third level, 288x160 at the fourth level, 144x80 pixels at the fifth level, etc. The image pyramid for the first (wide field) image will naturally have more pixels for the same resolution (i.e., for the same levels as the center image).

[0067] Since the wide-field image and the central image are generally derived from different pyramid levels, the central image is adapted to the resolution of the wide-field image through a resolution reduction operation. In this case, the number of channels in the feature map of the central image is generally increased (the amount of information per pixel is increased). The resolution reduction operation is, for example, striding or pooling. In the case of striding, only every second (or fourth, or nth) pixel is read. In the case of pooling, multiple pixels are summarized into one pixel, for example, in max pooling, the maximum value of a pixel pool (e.g., two pixels or 2x2 pixels) is taken over.

[0068] Assume the level 5 overview image is 400x150 pixels, and the level 5 center image is located horizontally at x0=133 pixels from the left edge of the overview image and extends vertically to y0=80 pixels from the bottom edge of the overview image. Assume each pixel corresponds to one element of the output feature map. To adjust the second output feature map, we need to add 133 zeros per row (one zero per pixel) to the left, 70 zeros per column to the top, and 133 zeros per row to the right, so that the channels of the adjusted second output feature map can be added, element by element, to the channels of the first output feature map. The starting values ​​x0,y0 are determined from the position of the (second) representation of the subregion within the (first) representation of the overview region. The starting values ​​x0,y0 indicate the horizontal and vertical shift or extension.

[0069] FIG. 8 shows a schematic diagram of a basic method for fusing such images (eg, first or wide field image 701 and second or central image 7020 of FIG. 7).

[0070] The wide-field image is sent as input image data to the first convolutional layer c1 of an artificial neural network (e.g., CNN). The central image is sent as input image data to the second convolutional layer c2 of the CNN. Each convolutional layer has an activation function and optional pooling.

[0071] The central image is padded with a "large" zero-padding ZP region so that its height and width match those of the wide-FOV image, thereby maintaining the spatial relationship. Based on FIG. 7 , this can be thought of as padding the region 701 outside the central extract image 702 for the central image 7020 (i.e., the unbleached, and therefore darker, regions from the wide-FOV image 701 in the lower part of FIG. 7 ) with zeros. The high resolution of the central image 7020 results in a high depth of the (second) feature map generated by the second convolutional layer c2. The height and width of the second feature map correspond to those of the central extract image 702 for the wide-FOV image 701. Here, the adjustment of the different heights and widths of the first and second feature maps is achieved by the zero-padding ZP of the second feature map.

[0072] The features of the wide-field image and the central image are concatenated. The concatenated features are sent to the third convolutional layer c3, which generates a fused feature map.

[0073] When convolving with the second feature map filled with zero padding ZP, many multiplications by 0 are required. These calculations of "0" multiplications of the zero padding ZP regions in the convolutional layer c3 are unnecessary and therefore not advantageous. On the other hand, it is not always possible to truncate these regions, for example, because known CNN accelerators do not allow spatial control of the application region of the convolution kernel.

[0074] However, the advantage is that the depths of the two feature maps can be different. Concatenation combines the two feature maps "deeply together." This is particularly advantageous when the central image has a higher resolution than the wide-field image, and therefore more information can be extracted from the central image. In this respect, the method is relatively flexible.

[0075] Figure 9 shows a schematic representation of an alternative second method, in which the wide-field features and the central features are combined via appropriate element-wise addition (+) (instead of concatenation cc of both feature maps), before the central image is adapted for height and width using zero padding ZP after feature extraction by the second convolutional layer c2. The feature map with the element-wise added features is sent to the third convolutional layer c3.

[0076] Also, the method accepts a performance penalty because addition combines features with different semantic meanings, and it is not advantageous for the tensors to have the same dimensions. The advantage is that adding zero (in the zero-padded ZP domain) requires significantly less computation time than multiplying by zero.

[0077] Both of these methods offer advantages and disadvantages, and it would be desirable to take advantage of the advantages of each, which is possible in a skilled combination.

[0078] FIG. 10 illustrates an advantageous method in a schematic manner. Based on the first alternative shown in Figure 8, i.e., combining features by concatenation, we will now describe a mathematical decomposition by c3, which obsoletes the multiplication by zeros in the zero-padding ZP region, which is not necessarily required. Convolutional layer c n ,o n 3D tensor FM with feature stacks (channels) n where n is a natural number. For conventional 2D convolution, the following applies:

[0079]

number

[0080]

number

[0081] Subsequent concatenation using convolutional layers (see Figure 8) yields two reduced convolutions c 3A and c 3B and then element-wise addition (+) is performed.

[0082]

number

[0083]

number

[0084] Both reduced convolutions c 3A and c 3B The adaptation of different heights and widths of the feature maps generated from is done before the element-wise addition (+). Convolution kernel c3 is 3A and c 3B By decomposing into 3B is applied to the reduced size of the central image with high runtime efficiency. This element-wise addition (+) is runtime-neutral on currently available accelerators for artificial neural networks.

[0085] Zero-padding ZP with subsequent addition is equivalent to summing the median feature values ​​of the adapted starting points. Alternatively, the median feature map may be written to a larger region that is previously initialized with zeros. In this case, zero-padding ZP is performed indirectly.

[0086] The activation function / pooling after c3 cannot be decomposed and is applied after addition. In particular, for large padded regions of zeros, no convolution operations are computed.

[0087] Overall, this embodiment offers the following particular advantages: a) For example, for distant objects, utilizing regions of interest with high resolution, using large field of view / capture area of ​​the sensor, and integrated feature considerations of different (image) pyramid levels for optimal overall performance. b) At the same time, it is a highly runtime efficient implementation.

[0088] In Figures 11-13 the method is again illustrated differently.

[0089] Figure 11 shows a schematic diagram of the concatenation of two feature maps 1101, 1102 processed by a convolution kernel 1110, resulting in an outputtable fused feature map 1130. Unlike the similar situation in Figure 8, here both feature maps 1101, 1102 have the same width w and height h. They are shown simply as two rectangular surfaces. Concatenation refers to combining "depths," with the second feature map 1102 shown schematically as being spatially behind the first feature map. Here, the convolution kernels 1110 are shown with opposing hatching for comparison, indicating that the first part, i.e., the "first convolution 2D kernel" shown with light hatching, scans the first feature map 1101, and the second (shown with dark hatching) convolution 2D kernel scans the second feature map 1102. The result is the fused output feature map 1130. The fused feature map 1130 is no longer separable with respect to the first and second feature maps 1101, 1102 after convolution.

[0090] 12 shows a schematic diagram of an alternative flow for fusing two feature maps of the same width w, height h, and depth d. The depth d of the feature maps may correspond to the number of channels or may be resolution dependent. Here, a first feature map 1201 is scanned by a first convolutional two-dimensional kernel 1211 to generate a first output feature map 1221, and a second feature map 1202 is scanned by a second convolutional two-dimensional kernel 1212 to generate a second output feature map 1222. The convolutional two-dimensional kernels 1211; 1212 may have dimensions of, for example, 3x3x the number of input channels, to generate an output layer. The depth of the output feature map can be defined by the number of convolutional two-dimensional kernels 1211; 1212. The fused feature map 1230 can be computed from both output feature maps 1221, 1222 by element-wise addition (+). The flow here, i.e., performing two separate convolutions for each feature map and then simply adding them together, is equivalent to the flow in Figure 11, where two feature maps are concatenated and then a convolution is performed.

[0091] FIG. 13 shows a schematic flow for fusing two feature maps of different widths and heights, corresponding to the flow described in FIG.

[0092] The first feature map 1301 (computed from the wide-field image) has a large width w and height h but a small depth d, while the second feature map 1302 (computed from the high-resolution central extract image) has a small width w and height h but a large depth d. The first convolutional 2D kernel 1311 scans the first feature map 1301, thereby generating a first output feature map 1321 with an expanded depth d. The second convolutional 2D kernel 1312 scans the second feature map, thereby generating a second output feature map 1322 (a diagonally hatched diced region). The depth d of the second output feature map is the same as that of the first output feature map. To perform the fusion of the first and second output feature maps 1321, 1322, it is advantageous to consider the position of the subregions within the overview region. Therefore, the height and width of the second output feature map 1322 are expanded to match those of the first output feature map 1321. The starting values ​​of the width and height for adjustment are, for example, information on the position of the central region 602 or 702 of the common outline region 601 or 701 from FIG. 6 or FIG. 7, for example, in the form of starting values ​​x0, y0, or the width starting value x of the feature map derived therefrom. s and the starting height value y s The second output feature map 1322 can be determined in the form: The missing regions (left, right, and top) in the second output feature map 1322 are filled with zeros (zero padding). The second output feature map adjusted in this way can be fused with the first output feature map 1321 by simple element-wise addition. Such a fused feature map 1330 is shown at the bottom of Figure 13.

[0093] FIG. 14 shows a schematic representation of a possible method flow.

[0094] In a first step S1, input data of at least one sensor is received. The input sensor data may be generated, for example, by two forward-facing ADAS sensors of the vehicle, e.g., a radar and a lidar with partially overlapping capture areas. The lidar sensor may have a wide capture area (e.g., a large aperture angle of 100° or 120°), which generates a first representation of the scene. The radar sensor may capture only a (central) sub-area of ​​the scene (e.g., a small capture angle of 90° or 60°), but may instead detect more distant objects, which generates a second representation of the scene.

[0095] To enable fusion of input data from the lidar and radar sensors, the sensor raw data may be mapped to a representation, which shows a bird's-eye view onto the roadway plane ahead of the vehicle. The representation or a feature map determined thereon may also be created, for example in the form of an occupancy grid. In the overlap region, there is lidar and radar data, in the side edge regions there is only lidar data, and in the further forward region there is only radar data.

[0096] In a second step S2, a first feature map is determined from the input data. From the (first) representation of the lidar sensor, a first feature map can be generated having a first height and width (or roadway plane depth and roadway plane width in a bird's eye view).

[0097] In a third step S3, a second feature map is determined from the input data. From the (second) representation of the radar sensor's capture area, a second feature map can be generated having a second height and width, where the width of the second feature map is smaller than the width of the first feature map and the height of the second feature map (distance in the direction of travel) is greater than the height of the first feature map.

[0098] In a fourth step S4, a first output feature map is determined based on the first feature map, where the first output feature map is calculated using a first convolution of the first feature map.

[0099] In a fifth step S5, a second output feature map is determined based on the second feature map, the second output feature map being calculated using a second convolution of the second feature map, the width and height of which are limited to the height and width of the second feature map.

[0100] In a sixth step S6, an adjustment of the different sizes of the first and second output feature maps is performed, in particular an adaptation of the height and / or width.

[0101] According to a first modification, the height of the first output feature map can be expanded to match the height of the second output feature map. The width of the second output feature map can be expanded to match the width of the first output feature map. The newly added area of ​​each (adjusted) output feature map due to the expansion is filled with zeros (zero padding).

[0102] According to the second variant, a template output feature map is first created, and its width and height are derived from the heights and widths of the first and second output feature maps and the location of the overlapping region. The template output feature map is then filled with zeros. In this case, the template output feature map has the width of the first output feature map and the height of the second output feature map.

[0103] The adjusted first output feature map inherits elements from the first output feature map in the region it covers, and uses starting values ​​that indicate the position of the first output feature map vertically and horizontally within the template output feature map.

[0104] The lidar output feature map, for example, extends across the entire width of the template output feature map, while the far region is empty. Thus, in the vertical direction, the starting value y sis predetermined, from which the template output feature map is "filled in." Similarly, based on the template output feature map pre-filled with zeros, an adjusted second output feature map is generated by inserting elements of the second output feature map from a preferred starting position. For example, the radar output feature map is first calculated by dividing the horizontal starting point x s and extends vertically across the entire height.

[0105] In the seventh step S7, the adjusted first and second output feature maps are fused by element-wise addition. By adjusting the height and width, element-wise addition of both output feature maps is readily possible for common CNN accelerators. The result is a fused feature map.

[0106] In the special case where the second output feature map includes the entire overlap region (i.e., the subregion of the first output feature map that includes the overlap region itself (see FIG. 13)), the adjustment of the different heights and widths of the second output feature map can be omitted, and in this case, the element-by-element addition of the second output feature map with the first output feature map is performed only in the overlap region based on the preferred starting point value, where the height and width of the fused feature map are the same as the height and width of the first output feature map (see FIG. 13).

[0107] In an eighth step S8, the fused feature map is output. The present application relates to the invention described in the claims, but also includes the following as other aspects. 1. In the method of fusing sensor data: a) receiving input sensor data, said input sensor data comprising: - the first representation (401,701) containing the first region of the scene (101,601) and - a step (S1) including a second representation (502, 702) including a second region (102, 602) of said scene, said first and said second regions overlapping each other but not being identical; b) determining (S2) a first feature map (1301) having a first height and width based on the first representation (401, 701), and determining (S3) a second feature map (1302) having a second height and width based on the second representation (502, 702); c) a step (S4) of computing a first output feature map (1321) using a first convolution of the first feature map (1301), and a step (S5) of computing a second output feature map (1322) using a second convolution of the second feature map (1302); d) calculating a fused feature map (1330) by element-wise summing the first and second output feature maps (1321, 1322), where the positions of the first and second regions are taken into account relative to each other so that elements of overlapping regions are summed; and e) outputting (S8) the fused feature map (1330). 2. 2. The method according to claim 1, wherein the first and second output feature maps (1321, 1322) have the same height and width in the overlapping region. 3. 3. The method according to claim 1 or 2, wherein the height and width of the fused feature map (1330) are determined by a rectangle that encloses the first and second output feature maps (1321, 1322). 4. 4. The method according to any one of 1 to 3 above, wherein the first region (101, 601) is an outline region of the scene, and the second region (502, 702) is a sub-region of the outline region of the scene. 5. 5. The method according to any one of 1 to 4 above, wherein the first representation has a first resolution and the second representation has a second resolution, the second resolution being higher than the first resolution. 6. The method according to any one of 3 to 5 above, wherein the first and / or second output feature maps (1321, 1322) are expanded so that the first and / or second output feature maps (1321, 1322) reach the width and height of the fused feature map (1330) and the positions of the first and second output feature maps (1321, 1322) are maintained relative to each other, and the newly added area of ​​each adjusted output feature map by the expansion is filled with zeros (ZP). 7. A template output feature map is first created, whose width and height are obtained from the height and width of the first and second output feature maps (1321, 1322) and the location of the overlapping region, and the template output feature map is filled with zeros; For the adjusted first output feature map, elements from the first output feature map (1321) in the area covered by the first output feature map (1321) are inherited; 6. A method according to any one of 1 to 5 above, wherein for the adjusted second output feature map, elements from the second output feature map (1322) in the area covered by the second output feature map (1322) are carried over. 8. 6. The method according to claim 4 or 5, wherein the second output feature map (1322) includes the entire overlap region, and the fused feature map (1330) is calculated by adding element-by-element of the second output feature map (1322) to the first output feature map (1321) only in the overlap region based on a preferred starting point value. 9. 9. The method according to any one of 1 to 8 above, wherein said feature maps (1301, 1302, 1321, 1322) each have a depth that depends on the resolution of said representations (401; 502; 701; 702). 10. 10. The method according to any one of 1 to 9 above, wherein determining the ADAS / AD related information is performed based on the fused feature map (1330). 11. 11. The method according to any one of claims 1 to 10, implemented in a hardware accelerator for artificial neural networks. 12. 12. The method according to any one of claims 1 to 11, wherein the fused feature map is generated in an encoder of an artificial neural network configured to determine ADAS / AD related information. 13. 13. The method of claim 12, wherein the artificial neural network configured to determine ADAS / AD-related information comprises a plurality of decoders for different ADAS / AD detection functions. 14. A system (10) for fusing sensor data, comprising an input interface (12), a data processing unit (14) and an output interface (18): a) the input interface (12) is configured to receive input sensor data, the input sensor data comprising: - the first representation (401,701) containing the first region of the scene (101,601) and - a second representation (502, 702) including a second region (102, 602) of said scene, said first and said second regions overlapping each other but not identical; The data processing unit (14) b) configured to determine a first feature map (1301) having a first height and width based on the first representation (401, 701) and to determine a second feature map (1302) having a second height and width based on the second representation (502, 702); c) configured to compute a first output feature map (1321) using a first convolution of the first feature map (1301) and to compute a second output feature map (1322) using a second convolution of the second feature map (1302); d) configured to calculate a fused feature map by element-wise summing the first and second output feature maps (1321, 1322), the positions of the first and second regions being taken into account relative to each other so that elements of overlapping regions are summed; e) The system (10), wherein the output interface (18) is configured to output the fused feature map (1330). 15. 15. The system of claim 14, wherein the system (10) comprises a CNN hardware accelerator, and the input interface (12), the data processing unit (14) and the output interface (18) are implemented in the CNN hardware accelerator. 16. 16. The system of claim 14 or 15, wherein the system (10) comprises a convolutional neural network having an encoder, the input interface (12), the data processing unit (14) and the output interface (18) being implemented in the encoder such that the encoder is configured to generate the fused feature map. 17. 17. The system of claim 16, wherein the convolutional neural network comprises a plurality of decoders configured to implement various ADAS / AD detection functions based at least on the fused feature map. 18. 18. The system of claim 17, comprising an ADAS / AD controller, the ADAS / AD controller configured to implement an ADAS / AD detection function based at least on a result of the ADAS / AD detection function. [Explanation of symbols]

[0108] 1 sensor 10 Systems 12 Input Interface 14 Data Processing Unit 16 Fusion Model 18 Output Interface 20 Control Unit 101 Overview area 102 Partial area 300 high resolution overview images 303 Pedestrians or further distant traffic participants 304 Vehicle or nearby traffic participants 305 Road or roadway 306 Houses 401 Reduced Resolution Overview Image 403 Pedestrian (undetectable) 404 vehicles 502 high resolution central extract images 503 Pedestrians 504 Vehicles (undetectable or completely undetectable) 601 Overview area 602 Partial area 701 Reduced Resolution Overview Images 702 Capture area of ​​excerpt image with high resolution 7020 High resolution (center) excerpt image 1101 First feature map 1102 Second feature map 1110 Convolution Kernel 1130 Fusion Feature Map 1201 First feature map 1202 Second feature map 1211 First Convolution 2D Kernel 1212 Second Convolution 2D Kernel 1221 First output feature map 1222 Second output feature map 1230 Fusion Feature Map 1301 First feature map 1302 Second feature map 1311 First Convolution 2D Kernel 1312 Second Convolution 2D Kernel 1321 First output feature map 1322 Second output feature map 1330 Fusion Feature Map x0 horizontal starting point y0 Vertical start or extension value wfov Overview image with reduced resolution center High resolution (center) excerpt image

number

Claims

1. A method for fusing sensor data using a data processing unit (14), comprising: The method comprises the following steps: a) receiving input sensor data using an input interface (12) of the data processing unit (14) of the vehicle, the input sensor data comprising: a first representation (401, 701) generated using a lidar sensor, comprising a first region (101, 601) of the scene, which is an overview region of the scene; and - a step (S1) including a second representation (502, 702) generated by means of a radar sensor, the second representation including a second area (102, 602) of the scene, the second area being a sub-area of ​​the overview area of ​​the scene, the first and second areas overlapping each other but not being identical; b) determining (S2) a first feature map (1301) having a first height and width based on the first representation (401, 701), and determining (S3) a second feature map (1302) having a second height and width based on the second representation (502, 702); c) a step (S4) of computing a first output feature map (1321) using a first convolution of the first feature map (1301), and a step (S5) of computing a second output feature map (1322) using a second convolution of the second feature map (1302); d) computing a fused feature map (1330) by element-wise addition of the first and second output feature maps (1321, 1322), the positions of the first and second regions being taken into account relative to one another so that elements of overlapping regions are added, the fused feature map (1330) being generated in an encoder of an artificial neural network configured to determine ADAS / AD related information, the artificial neural network comprising a number of decoders for different ADAS / AD detection functions; e) outputting the fused feature map (1330) (S8); f) determining ADAS / AD related information based on said fused feature map (1330); A method comprising:

2. The method of claim 1 , wherein the first and second output feature maps (1321, 1322) have the same height and width in the overlapping region.

3. The method of claim 1 , wherein the height and width of the fused feature map (1330) are determined by a rectangle that encloses the first and second output feature maps (1321, 1322).

4. The method of claim 1 , wherein the first representation has a first resolution and the second representation has a second resolution, the second resolution being greater than the first resolution.

5. 5. The method according to claim 1, wherein the first and / or second output feature maps (1321, 1322) are enlarged so that the first and / or second output feature maps (1321, 1322) reach the width and height of the fused feature map (1330) and the positions of the first and second output feature maps (1321, 1322) are maintained relative to each other, and the newly added areas of the adjusted first and second output feature maps (1321, 1322) are filled with zeros (ZP).

6. A template output feature map is first created, whose width and height are obtained from the height and width of the first and second output feature maps (1321, 1322) and the location of the overlap region, and the template output feature map is filled with zeros; For the adjusted first output feature map, elements from the first output feature map (1321) in the region covered by the first output feature map (1321) are carried over; 3. The method of claim 1, wherein for the adjusted second output feature map, elements from the second output feature map (1322) in the region covered by the second output feature map (1322) are carried over.

7. 5. The method of claim 1, wherein the second output feature map (1322) includes the entire overlap region, and the fused feature map (1330) is computed by element-wise adding the second output feature map (1322) to the first output feature map (1321) only in the overlap region based on a preferred starting point value.

8. 3. The method of claim 1, wherein the first feature map (1301), the second feature map (1302), the first output feature map (1321), and the second output feature map (1322) each have a depth that depends on the resolution of the first representation (401, 701) and the second representation (502, 702).

9. 3. The method of claim 1 or 2, implemented in a hardware accelerator for artificial neural networks.

10. A system (10) for fusing sensor data using a data processing unit (14) comprising an input interface (12), a data processing unit (14) and an output interface (18), comprising: a) the input interface (12) is configured to receive input sensor data, the input sensor data comprising: a first representation (401, 701) generated using a lidar sensor, comprising a first region (101, 601) of the scene, which is an overview region of the scene; and a second representation (502, 702) generated using a radar sensor, comprising a second region (102, 602) of the scene, the second region being a subregion of the overview region of the scene, said first and second regions overlapping each other but not being identical; The data processing unit (14) b) determining a first feature map (1301) having a first height and width based on the first representation (401, 701) and determining a second feature map (1302) having a second height and width based on the second representation (502, 702); c) configured to compute a first output feature map (1321) using a first convolution of the first feature map (1301) and to compute a second output feature map (1322) using a second convolution of the second feature map (1302); d) configured to compute a fused feature map by element-wise addition of the first and second output feature maps (1321, 1322), the positions of the first and second regions being taken into account relative to one another so that elements of overlapping regions are added, the fused feature map (1330) being generated in an encoder of an artificial neural network configured to determine ADAS / AD related information, the artificial neural network comprising a plurality of decoders for different ADAS / AD detection functions; e) the output interface (18) outputs the fused feature map (1330); and f) A system (10) configured to determine ADAS / AD related information based on said fused feature map (1330).

11. 11. The system of claim 10, wherein the system (10) comprises a CNN hardware accelerator, and the input interface (12), the data processing unit (14), and the output interface (18) are implemented in the CNN hardware accelerator.

12. 12. The system of claim 10, wherein the system comprises a convolutional neural network having an encoder, the input interface, the data processing unit, and the output interface being implemented in the encoder such that the encoder is configured to generate the fused feature map.

13. 13. The system of claim 12, wherein the convolutional neural network comprises a plurality of decoders configured to implement various ADAS / AD detection functions based at least on the fused feature map.

14. 14. The system of claim 13, comprising an ADAS / AD controller, the ADAS / AD controller configured to implement an ADAS / AD detection function based at least on a result of the ADAS / AD detection function.

Images