Methods for fusing image data in the context of artificial neural networks

Abstract

The present invention relates to a method and system (10) for fusing image data of at least one image capture sensor (1). The method comprises: a) receiving input image data, the input image data including a first image (401, 701) including a first region (101, 601) of a scene and a second image (502, 702) including a second region (102, 602) of the scene, the first and second regions overlapping each other but not being identical; b) determining (S2) a first feature map (1301) having a first height and width based on a first image (401, 701), and determining (S3) a second feature map (1302) having a second height and width based on a second image (502, 702); c) computing (S4) a first output feature map (1321) using a first convolution of the first feature map (1301), and computing (S5) a second output feature map (1322) using a second convolution of the second feature map (1302); d) computing a fused feature map (1330) by element-wise summing of the first and second output feature maps (1321, 1322), where the positions of the first and second regions are taken into account relative to one another such that elements of overlapping regions are summed; and e) outputting (S8) a fused feature map (1330). The method is very run-time efficient and can be used to fuse image data from one or more image capture sensors (1) for a vehicle's ADAS / AD system.

Description

[Technical Field]

[0001] The present invention relates to a method and system for fusing image data in the context of artificial neural networks, for example in environment-based ADAS / AD systems for vehicles. [Background technology]

[0002] The resolution of imaging environment sensors (especially camera sensors) for ADAS / AD systems is constantly increasing. This makes it possible to detect small objects, detect sub-objects, and, for example, read small characters at a distance. The disadvantage of high resolution is the significantly higher computing power required to process the correspondingly large image data. Therefore, various resolution levels of image data are often used for processing. For example, a large range or high resolution is often required in the center of the image, while not in the peripheral areas (e.g., similar to the human eye).

[0003] Patent document 1 describes a camera device for imaging the surrounding environment for a vehicle, comprising an image sensor device for capturing pixel images and a processing device configured to combine adjacent pixels of the pixel images in an adapted pixel image. By combining pixel values ​​of adjacent pixels in the form of a 2x2 image pyramid or an nxn image pyramid, different adapted pixel images can be generated at different resolutions.

[0004] US Pat. No. 5,629,999 and US Pat. No. 5,629,999 describe driver assistance systems that take images with variable resolution.

[0005] Patent Document 4 describes a camera device for capturing the surrounding area of ​​a vehicle, which includes a wide-angle optical element and a high-resolution image capturing sensor. For the images of the image sequence, pixel binning can be used to capture an image of the entire capture area with reduced resolution, or a partial area of ​​the capture area with full resolution.

[0006] Artificial neural network technologies are increasingly being used in environmental sensor-based ADAS / AD systems to better recognize, classify, and at least partially understand road users and scenes. Deep neural networks, such as convolutional neural networks (CNNs), have clear advantages over traditional approaches, which rely on trained classifiers, such as support vector machines or AdaBoost, to extract features that are relatively hand-designed (e.g., histograms of oriented gradients (HOGs), local binary patterns (LBPs), Gabor filters, etc.). In the case of (multi-layer) CNNs, feature extraction is performed by machine (deep) learning algorithms, which significantly increase the dimensionality and depth of the feature space, ultimately resulting in significantly improved performance, e.g., higher detection rates.

[0007] A particular challenge is the process of integrating sensor data with different and even overlapping capture areas and different resolutions.

[0008] Patent Document 5 describes a CNN-based method for learning parameters of an object detector. An object region is estimated in a camera image, and excerpts of this region are generated from various image mirror levels. The excerpts, for example, have the same height, are padded on the sides with "zero padding" (zero regions), and are concatenated. This form of concatenation can be roughly described as a collage, where excerpts of the same height are "pasted together." The generated composite image is thus a composite of regions from the same original camera image at various resolution levels. The CNN is trained so that the object detector can detect objects based on the composite image, thereby enabling detection of even more distant objects. The advantage of such an approach, as opposed to processing each image region separately one after the other using a CNN, is that the weights only need to be applied once for the composite image. The disadvantage in this case is that adjacent image regions in the composite image are considered, in particular, independently of each other, using object detectors by CNNs: objects present in overlapping regions, which may be incompletely included in the image regions, must be identified in a non-trivial way as belonging to one and the same object. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] German Patent Application Publication No. 102015208889 [Patent Document 2] U.S. Patent No. 10,742,907 [Patent Document 3] U.S. Patent No. 10,757,330 [Patent Document 4] U.S. Patent No. 1,0798,319 [Patent Document 5] European Patent Application Publication No. 3686798 [Patent Document 6] German Patent Application Publication No. 102020204840 Summary of the Invention [Problem to be solved by the invention]

[0010] It is an object of the present invention to provide an improved image data fusion method in the context of an artificial neural network that efficiently fuses input image data from various partially overlapping capture regions and provides them for further processing. [Means for solving the problem]

[0011] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. a) Capture a large image area; b) capturing a relevant image region, e.g., a remote object in the center of the image, at high resolution; The present invention relates to an efficient implementation of object detection based on input data of at least one image capture sensor.

[0012] Before explaining the solution, the following considerations are made.

[0013] Using multiple levels of an image pyramid in a neural network, the low-resolution overview image and the high-resolution central extract image can be processed separately by two independent inferences (two CNNs trained for each). This implies a large computational / runtime cost: in particular, the weights of the trained CNN need to be multiplied anew for different images, and features from different pyramid levels are not considered jointly.

[0014] Alternatively, the process of Patent Document 5 can be performed on a combined image from various resolution levels. Thus, a combined image from various sub-images / resolution levels is generated, and inference or a trained CNN is performed on this. This is somewhat more efficient because each weight needs to be applied only once for all images, rather than anew for each sub-image. However, other disadvantages remain, such as the lack of combination of features from various resolution levels.

[0015] A method for fusing image data of at least one image capture sensor includes: a) receiving input image data, the input image data comprising: a first image (or a first representation) including or containing a first region of the scene; and - including a second image that includes or contains a second region of the scene, the first and second regions, while overlapping one another, not being identical; b) determining a first feature map having a first height and width based on the first image, and determining a second feature map having a second height and width based on the second image; c) computing a first output feature map using a first convolution of the first feature map, and computing a second output feature map using a second convolution of the second feature map; d) computing a fused feature map by element-wise summing the first and second output feature maps, where the positions of the first and second regions are taken into account relative to each other so that elements (of the first and second output feature maps) of overlapping regions are summed; e) outputting a fused feature map.

[0016] An image may be, for example, a two-dimensional representation of a scene captured by an image capture sensor. A point cloud or depth map is an example of a three-dimensional image or representation that may be captured, for example, by a lidar sensor or a stereo camera as an image capture sensor. For many purposes, the three-dimensional representation may be converted into a two-dimensional image, for example, by planar cross-section or projection. The feature map may be determined from an image or other (existing) feature map by convolution or a convolution layer / convolutional layer / convolution kernel. The height and width of the feature map are operationally related to the height and width of the underlying image (or input feature map).

[0017] The relative positions of the first and second regions are specifically taken into account in order to add appropriate elements of the first and second output feature maps for fusion. The position of the overlap region is determined, for example, by a starting value (x s ,y s ) In the overlap region, the elements of the first and second output feature maps are added. Outside the overlap region, the elements of the output feature maps that cover this region can be moved to the fused feature map. If neither of the output feature maps covers a region of the fused feature map, this region can be filled with zeros.

[0018] The method can be implemented, for example, in the context of an artificial neural network, preferably a convolutional neural network (CNN). ADAS / AD functions often (especially on the perception side) use at least one artificial neural network or CNN that is trained using machine learning methods to assign image input data to output data related to the ADAS / AD function. ADAS and AD stand for Advanced Driver Assistance Systems and Automated Driving, respectively. The trained artificial neural network can be implemented in a processor of an ADAS / AD control unit in the vehicle. The processor may be configured to evaluate image data by the trained artificial neural network (inference). The processor may include a hardware accelerator for the artificial neural network.

[0019] The processor or inference can be configured to detect or further determine ADAS / AD-relevant information, for example, from input image data of one or more image capture sensors. The relevant information can be, for example, object and / or surrounding environment information for an ADAS / AD system or an ADAS / AD controller. The ADAS / AD-relevant object and / or surrounding environment information represents input variables important to the ADAS / AD system, such as objects, landmarks, road signs, traffic participants, distances to objects, relative speeds of objects, etc. Examples of functions for detecting relevant information are lane detection, object detection, depth perception (3D estimation of image components), semantic detection, traffic sign detection, etc.

[0020] In one embodiment, the first and second images are captured by the same image capture sensor, which may be an upstream step in the method. In particular, the first and second images may be captured by the image capture sensor simultaneously or in immediate succession.

[0021] In one embodiment, the (only) image capture sensor is a monocular camera, and the first representation (or first image) may correspond to an overview image with reduced resolution captured at a wide angle, and the second representation (or second image) may correspond to a partial image with higher resolution.

[0022] According to one example embodiment, the first and second images correspond to different image pyramid levels of the (original) image captured by the image capture sensor.

[0023] Depending on the resolution, the input image data may be or may be encoded into multiple channels, for example, each channel having the same height and width, where the spatial relationship of the pixels contained within each channel may be maintained. For further details in this regard, reference is made to U.S. Patent No. 6,249,999, the contents of which are incorporated herein in their entirety.

[0024] In one embodiment, the first region is an overview region of the scene and the second region is a subregion of the overview region of the scene. The overview region included in the first image may correspond to the entire region, i.e., the maximum capture area of ​​the image capture sensor. The subregion of the scene included in the second image may correspond to a Region of Interest (ROI) that is also included in the first image.

[0025] According to one example embodiment, the first image has a first resolution and the second image has a second resolution. For example, the second resolution is higher than the first resolution. The resolution of the second image may correspond to the maximum resolution of the image capture sensor. For example, the higher resolution may provide more detail regarding a subregion or region of interest that is the content of the second image. Image resolution may correspond to the accuracy or data depth, for example, the smallest distance between two adjacent picture elements (pixels) of an image capture sensor.

[0026] In one embodiment, two monocular cameras with overlapping capture areas are used as image capture sensors. The two monocular cameras may be components of a stereo camera. The two monocular cameras may have different aperture angles and / or resolutions ("hybrid stereo camera"). Both monocular cameras may be satellite cameras mounted independently on the vehicle.

[0027] According to one example embodiment, multiple cameras of an omnidirectional camera system are used as image capture sensors. For example, four monocular cameras with fisheye lenses (capture angles of, for example, 180° or more) can capture the entire surroundings of the vehicle. Every two adjacent cameras have an overlapping area of ​​about 90°. In this case, a fused feature map of the 360° surroundings of the vehicle can be created from four separate images (four representations).

[0028] In one embodiment, the first and second output feature maps have the same height and width in the overlap region. In other words, adjacent elements in the overlap region of these output feature maps are equidistant from each other in real space. This is therefore a possible case since the first and second feature maps already have the same height and width in the overlap region. For example, the first and second regions or the first and second images (also) have the same height and width.

[0029] According to one example embodiment, the height and width of the fused feature map is determined by a rectangle that encloses (exactly contains) the first and second output feature maps.

[0030] In one embodiment, after the height and width of the fused feature map are determined by a rectangle that encloses (exactly contains) the first and second output feature maps, the first and / or second output feature maps can be scaled or adjusted so that the first and / or second output feature maps reach the width and height of the fused feature map, while maintaining the positions of the first and second output feature maps relative to each other. The overlapping regions are in the same position in both adjusted output feature maps. The newly added regions of each (adjusted) output feature map due to the scaled-up operation are filled with zeros (zero padding). Both adjusted output feature maps can then be added element-by-element.

[0031] According to one example embodiment, a template output feature map is created whose width and height are derived from the height and width of the first and second output feature maps and the location of the overlap region (see the previous paragraph, bounding rectangle). The template output feature map is filled with zeros. For the adjusted first output feature map, elements from the first output feature map in the region covered by the first output feature map are carried over, and starting values ​​can be used to indicate the vertical and horizontal position of the first output feature map within the template output feature map. Correspondingly, an adjusted second output feature map is constructed, and both adjusted output feature maps can then be added element-wise.

[0032] In one embodiment, for the special case where the second output feature map includes the entire overlap region (i.e., the subregion of the first output feature map that includes the overlap region), the adjustment for the different heights and widths of the second output feature map can be omitted. In this case, no adjustment is made to the first output feature map because the fused feature map has the same height and width as the first output feature map. In this case, element-by-element addition of the second output feature map with the first output feature map may be performed only in the overlap region based on a preferred starting value. The starting value predetermines the base point within the first output feature map (i.e., in the overlap region) from which elements of the second output feature map are added to elements of the first output feature map to generate the fused feature map.

[0033] In one embodiment, the feature map has a depth that depends on the resolution of the (underlying) image: a high-resolution image (e.g., an excerpt image) results in a feature map with a high depth, e.g., the feature map contains more channels. For example, the processor may include a hardware accelerator for an artificial neural network that can further process stacks of image channel data "packets" during a clock or computation period. Stacks of image data or features (maps) may be provided in the hardware accelerator as stacked image channel data packets.

[0034] According to one example embodiment, detection of ADAS / AD related features is based on a fused feature map.

[0035] In one embodiment, the method is implemented in a hardware accelerator for artificial neural networks or CNNs.

[0036] According to one example embodiment, the fused feature map is generated in an encoder of an artificial neural network or CNN that is configured or trained to determine ADAS / AD-related information.

[0037] In one embodiment, an artificial neural network or CNN configured or trained to determine ADAS / AD related information comprises multiple decoders for various ADAS / AD detection functions.

[0038] A further aspect of the invention relates to a system or apparatus for fusing image data of at least one image capture sensor, the apparatus comprising an input interface, a data processing unit and an output interface. An input interface is configured to receive input image data, the input image data including first and second images, the first image including or containing a first region of a scene. The second image contains a second region of the scene. The first and second regions overlap each other. The first and second regions are not identical. The data processing unit is configured to perform the following steps b) to d). b) determining a first feature map having a first height and width based on the first image, and determining a second feature map having a second height and width based on the second image; c) computing a first output feature map using a first convolution of the first feature map and a second output feature map using a second convolution of the second feature map. d) computing a fused feature map by element-wise summing the first and second output feature maps, the element-wise summing taking into account the positions of the first and second regions relative to each other, such that elements (of the first and second output feature maps) of overlapping regions are summed. An output interface is configured to output the fused feature map. The output may be to a downstream ADAS / AD system or to downstream layers of a CNN or further artificial neural network for a "big" ADAS / AD.

[0039] According to one example embodiment, the system includes a CNN hardware accelerator, wherein the input interface, the data processing unit, and the output interface are implemented in the CNN hardware accelerator.

[0040] In one embodiment, a system includes a convolutional neural network having an encoder, wherein the input interface, the data processing unit, and the output interface are implemented in the encoder such that the encoder is configured to generate a fused feature map.

[0041] According to one example embodiment, a convolutional neural network includes multiple decoders configured to implement various ADAS / AD detection functions based at least on the fused feature map. Thus, multiple decoders of the CNN can use input image data encoded by a common encoder. The various ADAS / AD detection functions can be, for example, semantic segmentation of an image or image data, free space detection, lane detection, object detection, or object classification.

[0042] In one embodiment, the system includes an ADAS / AD controller configured to implement the ADAS / AD sensing function based at least on a result of the ADAS / AD sensing function.

[0043] The system may include at least one image capture sensor or sensors, such as a monocular camera, a stereo camera, a satellite camera, individual cameras of an omnidirectional system, a lidar sensor, a laser scanner or other three-dimensional camera, especially with a wide-angle capture area (e.g., 100° or more) and a large peak resolution (e.g., 5 megapixels or more).

[0044] A further aspect of the invention relates to a vehicle comprising at least one image capture sensor and a corresponding system for fusing image data.

[0045] The system or data processing unit may comprise, among others, a microcontroller or microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural / AI processing unit (NPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., and software for performing the corresponding method steps.

[0046] According to one embodiment, the system or data processing unit is implemented in a hardware-based image data pre-processing stage (eg, an Image Signal Processor (ISP)).

[0047] The invention also relates to a computer program element or program product which, when configured to run an image data fusion program on a processor of a system, instructs the processor to perform a method for fusing corresponding input image data.

[0048] The present invention also relates to a computer-readable storage medium having such program elements stored thereon.

[0049] Accordingly, the invention may be implemented in digital electronic circuitry, in computer hardware, firmware, or software.

[0050] In the following, exemplary embodiments and drawings will be described in the context of the present invention. [Brief explanation of the drawings]

[0051] [Figure 1] FIG. 1 illustrates a system for fusing image data from at least one image capture sensor. [Figure 2] FIG. 2 shows generally the extents and positions of first and second capture areas of an image capture sensor or two different image capture sensors from which first and second images of a scene can be determined. [Figure 3] FIG. 3 shows the overall image with high resolution. [Figure 4] FIG. 4 shows an overview or overview image with reduced resolution. [Figure 5] FIG. 5 shows the central excerpt image with high resolution. [Figure 6] FIG. 6 shows an alternative arrangement of the first (general) capture area and the second central capture area. [Figure 7] FIG. 7 shows an example of how the corresponding digital image would appear as a grayscale image. [Figure 8] FIG. 8 shows a basic method for fusing such images. [Figure 9] FIG. 9 shows a second alternative fusion method. [Figure 10] FIG. 10 illustrates a third advantageous fusion method. [Figure 11] FIG. 11 shows the concatenation of two feature maps that are then processed (and thereby fused) by a convolution kernel. [Figure 12] FIG. 12 shows an alternative flow in which two feature maps are processed by two separate convolution kernels followed by element-wise addition. [Figure 13] Figure 13 shows the flow for fusing two feature maps with different widths and heights. [Figure 14] FIG. 14 shows a possible method flow. DETAILED DESCRIPTION OF THE INVENTION

[0052] FIG. 1 shows schematically a system 10 for fusing data of at least one sensor 1, comprising an input interface 12, a data processing unit 14 having a fusion module 16, and an output interface 18 for outputting the fused data to a further unit 20. An example of an image capture sensor 1 is a monocular camera sensor having wide-angle optics and a high-resolution image capture sensor, such as a CCD or CMOS sensor. In many cases, the image data or image capture sensors have different resolutions and / or capture areas. Image data pre-processing is useful for fusion, allowing for the fusion of features from image data from one or more image capture sensors. An example embodiment described in more detail below relates to the processing of a first image of a camera sensor and a second image of that camera sensor, the second image comprising (only) a partial region of the first image and having a higher resolution compared to the resolution of the first image. Based on the image data of the camera sensor, a number of ADAS or AD functions, such as lane detection, lane keeping assist, traffic sign recognition, speed limit assist, traffic participant detection, collision warning, emergency braking assist, following distance control, construction site assist, highway pilot, cruising chauffeur function and / or autopilot, can be provided by the ADAS / AD controller, for example to a further unit 20. The overall system 10, 20 may include an artificial neural network, e.g., a CNN. To enable the artificial neural network to process image data in real time, e.g., in a vehicle, the overall system 10, 20 may include a hardware accelerator for the artificial neural network. Such a hardware component can essentially dedicatedly accelerate a software-implemented neural network so that the neural network can operate in real time.

[0053] The data processing unit 14 can process image data in a "stacked" format, i.e., it can read and process a stack of multiple input channels within a computational cycle (clock cycle). In a specific example, the data processing unit 14 can read four image channels with a resolution of 576x320 pixels. Fusing at least two image channels provides an advantage for subsequent CNN detection in that the already fused channel information or feature map can be processed by the CNN, rather than having to process the channels individually by their corresponding CNNs. Such fusion can be performed by the fusion module 16. Details of fusion are described in more detail below with reference to the following drawings.

[0054] The fusion can be performed in an encoder of the CNN. The fused data can then be processed by one or more decoders of the CNN to obtain detection or other ADAS / AD-related information. In such a configuration, the encoder can be represented by block 10 and the one or more decoders can be represented by block 20 in FIG. 1. The CNN will comprise blocks 10 and 20, hence the notation "whole system."

[0055] 2 shows a schematic representation of the ranges and positions of a first capture area 101 and a second capture area 102 of an image capture sensor 1 or two different image capture sensors from which first and second images of a scene can be determined. From the first image capture area 101, an overview or overall image can be captured as a first image, and from the second image capture area 102, e.g., a central image area, a second image including a portion of the first image capture area 101 can be captured. Figures 3-5 show an example in which images can be captured using image capture sensors (or camera sensors).

[0056] 3 shows a schematic overview or full image 300 with high resolution. A scene is captured including nearby and more distant traffic participants (304 and 303) on a road 305 or roadway passing by a house 306. A camera sensor can capture such an overview image with maximum width, height, and resolution (or number of pixels). However, processing this large amount of data (e.g., in the 5-10 megapixel range) is generally not possible in real time in AD or ADAS systems, resulting in further processing of reduced image data.

[0057] 4 shows a schematic representation of an overview or summary image 401 with reduced resolution. When the resolution is halved, the number of pixels drops to one-fourth. The summary image 401 with reduced resolution is referred to below as a wfov (wide field of view) image. Nearby traffic participants 404 (vehicles) can be detected from the wide field of view image even in the case of reduced resolution. On the other hand, distant traffic participants 403 (pedestrians) cannot be detected from this wide field of view image due to the limited resolution.

[0058] 5 shows a schematic representation of a central excerpt image with high (or highest) resolution 502. The excerpt image with high resolution 502 will be referred to as the central image in the following. The central image allows for high-resolution detection of a remote pedestrian 503, while a nearby vehicle 504 is not included or is only partially included in the capture area of ​​the central image 502.

[0059] 6 shows an alternative arrangement of a first (overview) capture area 601 and a central capture area 602. This central capture area 602 is located "below," meaning that it starts vertically at the same height as the overall capture area 601. The start values ​​(x0, y0) can indicate the position of the central capture area 602 in the horizontal and vertical directions within the overall or overview capture area.

[0060] FIG. 7 shows an example of how a corresponding digital image appears as a grayscale image. At the bottom, a wide-field-of-view image 701 captured by a vehicle's front camera can be seen as a first image. The vehicle is traveling toward a road intersection. A large, possibly multi-lane road runs perpendicular to the direction of travel. A bicycle path runs parallel to the large road. Traffic lights control the right-of-way for traffic participants. Buildings and trees demarcate the road and sidewalks. The central excerpt 702 is shown faded in the wide-field-of-view image 701 to illustrate that this excerpt exactly matches this excerpt 702 of the first image 701 as a second (central) image 7020 with higher resolution. The second image 7020 is shown at the top; here, it is easier for a human observer to recognize that the traffic light is showing a red light to the vehicle, that a bus has just crossed the intersection from left to right, and other details of the captured scene. Due to the high resolution of the second image 7020, even more distant objects or traffic participants can be robustly detected by image processing. The image pyramid may, for example, have 2304x1280 pixels at the highest level of the second (central) image, 1152x640 pixels at the second level, 576x320 at the third level, 288x160 at the fourth level, 144x80 pixels at the fifth level, etc. The image pyramid of the first (wide field of view) image will naturally have more pixels at the same resolution (i.e., at the same levels as the central image).

[0061] Since the wide-field image and the central image are generally derived from different pyramid levels, the central image is adapted to the resolution of the wide-field image through a resolution reduction operation. In this case, the number of channels in the feature map of the central image is generally increased (the amount of information per pixel is increased). The resolution reduction operation is, for example, striding or pooling. In the case of striding, only every second (or fourth, or nth) pixel is read. In the case of pooling, multiple pixels are summarized into one pixel, for example, in max pooling, the maximum value of a pixel pool (e.g., two pixels or 2x2 pixels) is taken over.

[0062] Assume the level 5 overview image is 400x150 pixels, and the level 5 center image is located horizontally at x0=133 pixels from the left edge of the overview image and extends vertically to y0=80 pixels from the bottom edge of the overview image. Assume each pixel corresponds to one element of the output feature map. To adjust the second output feature map, we need to add 133 zeros per row (one zero per pixel) to the left, 70 zeros per column to the top, and 133 zeros per row to the right. This allows us to add the channels of the first output feature map element-by-element to the channels of the adjusted second output feature map. The starting values ​​x0,y0 are determined from the position of the subregion's (second) image within the overview region's (first) image. The starting values ​​x0,y0 indicate the horizontal and vertical shift or extension.

[0063] FIG. 8 shows a schematic diagram of a basic method for fusing such images (eg, first or wide field image 701 and second or central image 7020 of FIG. 7). The wide-field image is sent as input image data to the first convolutional layer c1 of an artificial neural network (e.g., CNN). The central image is sent as input image data to the second convolutional layer c2 of the CNN. Each convolutional layer has an activation function and optional pooling. The central image is padded with a "large" zero-padding ZP region so that its height and width match those of the wide-FOV image, thereby maintaining the spatial relationship. Based on FIG. 7 , this can be thought of as padding the region 701 outside the central extract image 702 for the central image 7020 (i.e., the unbleached, and therefore darker, regions from the wide-FOV image 701 in the lower part of FIG. 7 ) with zeros. The high resolution of the central image 7020 results in a high depth of the (second) feature map generated by the second convolutional layer c2. The height and width of the second feature map correspond to those of the central extract image 702 for the wide-FOV image 701. Here, the adjustment of the different heights and widths of the first and second feature maps is achieved by the zero-padding ZP of the second feature map. The features of the wide-field image and the central image are concatenated. The concatenated features are sent to the third convolutional layer c3, which generates a fused feature map. When convolving with the second (filled with zero padding ZP) feature map, many multiplications by 0 are required. These calculations of "0" multiplications of the zero padding ZP regions in the convolutional layer c3 are unnecessary and therefore not advantageous. On the other hand, it is not possible to truncate these regions, for example, because known CNN accelerators do not allow spatial control of the application region of the convolution kernel. However, the advantage is that the depths of the two feature maps can be different. Concatenation combines the two feature maps "deeply together." This is particularly advantageous when the central image has a higher resolution than the wide-field image, and therefore more information can be extracted from the central image. In this respect, the method is relatively flexible.

[0064] Figure 9 shows a schematic representation of an alternative second method, in which the wide-field features and the central features are combined via appropriate element-wise addition (+) (instead of concatenation cc of both feature maps), before the central image is adapted for height and width using zero padding ZP after feature extraction by the second convolutional layer c2. The feature map with the element-wise added features is sent to the third convolutional layer c3. Also, the method accepts a performance penalty because addition combines features with different semantic meanings, and it is not advantageous for the tensors to have the same dimensions. The advantage is that adding zero (in the zero-padded ZP domain) requires significantly less computation time than multiplying by zero.

[0065] Both of these methods offer advantages and disadvantages, and it would be desirable to take advantage of the advantages of each, which is possible in a skilled combination.

[0066] FIG. 10 illustrates an advantageous method in a schematic manner. Based on the first alternative shown in Figure 8, i.e., combining features by concatenation, we will now describe a mathematical decomposition by c3, which obsoletes the multiplication by zeros in the zero-padding ZP region, which is not necessarily required. Convolutional layer C n O n 3D tensor FM with feature stacks (channels) n where n is a natural number. For conventional 2D convolution, the following applies:

[0067]

number

[0068]

number

[0069] Subsequent concatenation using convolutional layers (see Figure 8) produces two reduced convolutions C 3A and C 3B and then element-wise addition (+) is performed.

[0070]

number

[0071]

number

[0072] Both reduced convolutions C 3A and C 3B The adaptation of different heights and widths of the feature maps generated from is done before the element-wise addition (+). Convolution kernel C3 is C 3A and C 3B By decomposing it into 3B is applied to the reduced size of the central image with high runtime efficiency. This element-wise addition (+) is runtime-neutral on currently available accelerators for artificial neural networks.

[0073] Zero-padding ZP with subsequent addition is equivalent to summing the median feature values ​​of the adapted starting points. Alternatively, the median feature map may be written to a larger region that is previously initialized with zeros. In this case, zero-padding ZP is performed indirectly.

[0074] The activation function / pooling after c3 cannot be decomposed and is applied after addition. In particular, for large padded regions of zeros, no convolution operations are computed.

[0075] Overall, this embodiment offers the following particular advantages: a) For example, for distant objects, utilizing a region of interest with high resolution, using a large field of view / capture area of ​​the image capture sensor, and integrated feature considerations of various (image) pyramid levels for optimal overall performance. b) At the same time, it is a highly runtime efficient implementation.

[0076] In Figures 11-13 the method is again illustrated differently.

[0077] Figure 11 shows a schematic diagram of the concatenation of two feature maps 1101, 1102 processed by a convolution kernel 1110, resulting in an outputtable fused feature map 1130. Unlike the similar situation in Figure 8, here both feature maps 1101, 1102 have the same width w and height h. They are shown simply as two rectangular surfaces. Concatenation refers to combining depths together, and is shown diagrammatically as the second feature map 1102 being spatially behind the first feature map 1101. Here, the convolution kernels 1110 are shown with opposing hatching for comparison, indicating that the first part, i.e., the "first convolution 2D kernel" shown with light hatching, scans the first feature map 1101, and the second (shown with dark hatching) convolution 2D kernel scans the second feature map 1102. The result is the fused output feature map 1130. The fused feature map 1130 is no longer separable with respect to the first and second feature maps 1101, 1102 after convolution.

[0078] 12 shows a schematic diagram of an alternative flow for fusing two feature maps of the same width w, height h, and depth d. The depth d of the feature maps may correspond to the number of channels or may depend on the resolution of the underlying image. Here, a first feature map 1201 is scanned by a first convolutional two-dimensional kernel 1211 to generate a first output feature map 1221, and a second feature map 1202 is scanned by a second convolutional two-dimensional kernel 1212 to generate a second output feature map 1222. The convolutional two-dimensional kernels 1211; 1212 may have dimensions of, for example, 3x3x the number of input channels, to generate an output layer. The depth of the output feature map can be defined by the number of convolutional two-dimensional kernels 1211; 1212. The fused feature map 1230 can be computed from both output feature maps 1221, 1222 by element-wise addition (+). The flow here, i.e., performing two separate convolutions for each feature map and then simply adding them together, is equivalent to the flow in Figure 11, where two feature maps are concatenated and then a convolution is performed.

[0079] FIG. 13 shows a schematic flow for fusing two feature maps of different widths and heights, corresponding to the flow described in FIG.

[0080] The first feature map 1301 (computed from the wide-field image) has a large width w and height h but a small depth d, while the second feature map 1302 (computed from the high-resolution central extract image) has a small width w and height h but a large depth d.

[0081] A first convolutional 2D kernel 1311 scans the first feature map 1301, producing a first output feature map 1321 with an expanded depth d. A second convolutional 2D kernel 1312 scans the second feature map, producing a second output feature map 1322 (the diagonally hatched diced region). The depth d of the second output feature map is the same as the depth of the first output feature map.

[0082] To perform the fusion of the first and second output feature maps 1321, 1322, it is advantageous to take into account the position of the subregions within the overview region. Thus, the height and width of the second output feature map 1322 are enlarged to match the height and width of the first output feature map 1321. The starting values ​​of the width and height for adjustment can be obtained, for example, from information on the position of the central region 602 or 702 of the common overview region 601 or 701, e.g., in the form of starting values ​​x0, y0, or the width starting value x of the feature map derived therefrom, e.g., from Fig. 6 or Fig. 7. s and the starting height value y s It can be determined by the following format:

[0083] The missing regions (left, right, and top) in the second output feature map 1322 are filled with zeros (zero padding). The second output feature map adjusted in this way can be fused with the first output feature map 1321 by simple element-wise addition. Such a fused feature map 1330 is shown at the bottom of Figure 13.

[0084] FIG. 14 shows a schematic representation of a possible method flow. In a first step S1, input data of at least one image capture sensor is received. The input sensor data may be generated, for example, by the vehicle's ADAS sensors facing two driving directions, e.g., one telephoto camera and one lidar with partially overlapping capture areas. The lidar sensor may have a wide capture area (e.g., a large aperture angle of 100° or 120°), which generates a first image or representation of the scene. The telephoto camera may capture only a (central) subarea of ​​the scene (e.g., a small capture angle of 50°), but may instead detect more distant objects, which generates a second representation of the scene. To enable the lidar and telephoto camera sensor input data to be fused, the sensor raw data may be mapped to images that show a bird's-eye view of the roadway plane ahead of the vehicle. In the overlapping region, both lidar data and telephoto camera data exist, in the side edge regions, only lidar data exists, and in the further forward region, only telephoto camera data exists.

[0085] In a second step S2, a first feature map is determined from the input data. From the (first) image of the lidar sensor, a first feature map having a first height and width (or roadway plane depth and roadway plane width in a bird's-eye view) can be generated. In a third step S3, a second feature map is determined from the input data. A second feature map having a second height and width can be generated from a (second) image of the capture area of ​​the telephoto camera sensor, where the width of the second feature map is smaller than the width of the first feature map and the height of the second feature map (distance in the direction of travel) is greater than the height of the first feature map.

[0086] In a fourth step S4, a first output feature map is determined based on the first feature map, where the first output feature map is calculated using a first convolution of the first feature map.

[0087] In a fifth step S5, a second output feature map is determined based on the second feature map, the second output feature map being calculated using a second convolution of the second feature map, the width and height of which are limited to the height and width of the second feature map.

[0088] In a sixth step S6, an adjustment of the different sizes of the first and second output feature maps is performed, in particular an adaptation of the height and / or width.

[0089] According to a first modification, the height of the first output feature map can be expanded to match the height of the second output feature map. The width of the second output feature map can be expanded to match the width of the first output feature map. The newly added area of ​​each (adjusted) output feature map due to the expansion is filled with zeros (zero padding).

[0090] According to the second variant, a template output feature map is first created, and its width and height are derived from the heights and widths of the first and second output feature maps and the location of the overlapping region. The template output feature map is then filled with zeros. In this case, the template output feature map has the width of the first output feature map and the height of the second output feature map.

[0091] The adjusted first output feature map inherits elements from the first output feature map in the region it covers. Also, a starting value can be used to indicate the position of the first output feature map vertically and horizontally within the template output feature map. For example, the lidar output feature map extends across the entire width of the template output feature map, while the remote region is empty. Thus, in the vertical direction, the starting value y s is predetermined, from which the template output feature map is "filled in."

[0092] Similarly, based on the template output feature map pre-filled with zeros, an adjusted second output feature map is generated by inserting elements of the second output feature map from a suitable starting point position. For example, the radar output feature map is first generated by inserting elements of the second output feature map from a starting point position x in the horizontal direction. s and extends vertically across the entire height.

[0093] In the seventh step S7, the adjusted first and second output feature maps are fused by element-wise addition. By adjusting the height and width, element-wise addition of both output feature maps is readily possible for common CNN accelerators. The result is a fused feature map.

[0094] In the special case where the second output feature map includes the entire overlap region (i.e., includes the subregion of the first output feature map that includes the overlap region (see FIG. 13)), the adjustment of the different heights and widths of the second output feature map can be omitted. In this case, the element-by-element addition of the second output feature map with the first output feature map is performed using the preferred starting value x s ,ys Here, the height and width of the fused feature map are the same as the height and width of the first output feature map (see FIG. 13).

[0095] In an eighth step S8, the fused feature map is output. The present application relates to the invention described in the claims, but also includes the following as other aspects. 1. A method for fusing image data of at least one image capture sensor (1), comprising: a) receiving input image data, the input image data comprising: - the first image (401,701) containing the first region of the scene (101,601) and - a step (S1) including a second image (502, 702) including a second region (102, 602) of said scene, said first and said second regions overlapping each other but not identical; b) determining (S2) a first feature map (1301) having a first height and width based on the first image (401, 701), and determining (S3) a second feature map (1302) having a second height and width based on the second image (502, 702); c) a step (S4) of computing a first output feature map (1321) using a first convolution of the first feature map (1301), and a step (S5) of computing a second output feature map (1322) using a second convolution of the second feature map (1302); d) calculating a fused feature map (1330) by element-wise summing the first and second output feature maps (1321, 1322), where the positions of the first and second regions are taken into account relative to each other so that elements of overlapping regions are summed; and e) outputting (S8) the fused feature map (1330). 2. 2. The method of claim 1, wherein the first and second images are captured by the same image capture sensor. 3. 3. The method according to claim 1 or 2, wherein the first image (401, 701) and the second image (502, 702) correspond to different levels of an image pyramid of the image capture sensor. 4. 4. The method according to any one of 1 to 3 above, wherein the first region (101, 601) is an outline region of the scene, and the second region (502, 702) is a sub-region of the outline region of the scene. 5. 5. The method according to any one of 1 to 4 above, wherein the first image has a first resolution and the second image has a second resolution, the second resolution being higher than the first resolution. 6. 6. A method according to any one of claims 1, 4 or 5, wherein two monocular cameras with overlapping capture areas are used as image capture sensors. 7. 6. The method according to any one of 1, 4 or 5 above, wherein multiple cameras of an omnidirectional camera system are used as image capture sensors. 8. 8. The method according to any one of 1 to 7 above, wherein the first and second output feature maps (1321, 1322) have the same height and width in the overlapping region. 9. 9. The method according to any one of 1 to 8 above, wherein the height and width of the fused feature map (1330) are determined by a rectangle that surrounds the first and second output feature maps (1321, 1322). 10. 10. The method according to any one of 1 to 9 above, wherein the feature maps (1301, 1302, 1321, 1322) each have a depth that depends on the resolution of the images (401; 502; 701; 702). 11. 11. The method according to any one of claims 1 to 10, wherein the fused feature map is generated in an encoder of an artificial neural network configured to determine ADAS / AD related information. 12. 12. The method of claim 11, wherein the artificial neural network configured to determine ADAS / AD related information comprises a plurality of decoders for different ADAS / AD detection functions. 13. A system (10) for fusing image data of at least one image capture sensor, comprising an input interface (12), a data processing unit (14) and an output interface (18): a) the input interface (12) is configured to receive input image data, the input image data comprising: - the first image (401,701) containing the first region of the scene (101,601) and - a second image (502, 702) including a second region (102, 602) of said scene, said first and said second regions overlapping each other but not identical; The data processing unit (14) b) determining a first feature map (1301) having a first height and width based on the first image (401, 701) and determining a second feature map (1302) having a second height and width based on the second image (502, 702); c) configured to compute a first output feature map (1321) using a first convolution of the first feature map (1301) and to compute a second output feature map (1322) using a second convolution of the second feature map (1302); d) configured to calculate a fused feature map by element-wise summing the first and second output feature maps (1321, 1322), the positions of the first and second regions being taken into account relative to each other so that elements of overlapping regions are summed; e) The system (10), wherein the output interface (18) is configured to output the fused feature map (1330). 14. 14. The system of claim 13, wherein the system (10) comprises a convolutional neural network having an encoder and at least one decoder, the input interface (12), the data processing unit (14) and the output interface (18) being implemented in the encoder such that the encoder is configured to generate the fused feature map, and the at least one decoder is configured to implement an ADAS / AD detection function based at least on the fused feature map. 15. A vehicle comprising at least one image capture sensor (1) and a system according to claim 13 or 14. [Explanation of symbols]

[0096] 1 Image capture sensor 10 Systems 12 Input Interface 14 Data Processing Unit 16 Fusion Model 18 Output Interface 20 Control Unit 101 Overview area 102 Partial area 300 high resolution overview images 303 Pedestrians or further distant traffic participants 304 Vehicle or nearby traffic participants 305 Road or roadway 306 Houses 401 Reduced Resolution Overview Images 403 Pedestrian (undetectable) 404 vehicles 502 high resolution central extract images 503 Pedestrians 504 Vehicles (undetectable or completely undetectable) 601 Overview area 602 Partial area 701 Reduced Resolution Overview Images 702 Capture area of ​​excerpt image with high resolution 7020 High resolution (center) excerpt image 1101 First feature map 1102 Second feature map 1110 Convolution Kernel 1130 Fusion Feature Map 1201 First feature map 1202 Second feature map 1211 First Convolution 2D Kernel 1212 Second Convolution 2D Kernel 1221 First output feature map 1222 Second output feature map 1230 Fusion Feature Map 1301 First feature map 1302 Second feature map 1311 First Convolution 2D Kernel 1312 Second Convolution 2D Kernel 1321 First output feature map 1322 Second output feature map 1330 Fusion Feature Map x0 horizontal starting point y0 Vertical start or extension value wfov Overview image with reduced resolution center High resolution (center) excerpt image

number

Claims

1. A method for fusing image data of at least one image capture sensor (1), comprising: a) receiving input image data, said input image data comprising: a first image (401,701) containing a first region (101,601) of the scene, and a) a step (S1) including a second image (502, 702) including a second region (102, 602) of the scene, the first and second regions overlapping each other but not being identical; b) a step (S2) of determining a first feature map (1301) having a first height and width based on the first image (401, 701), and a step (S3) of determining a second feature map (1302) having a second height and width based on the second image (502, 702); c) computing a first output feature map (1321) using a first convolution of the first feature map (1301) (S4), and computing a second output feature map (1322) using a second convolution of the second feature map (1302) (S5); d) computing a fused feature map (1330) by element-wise summing the first and second output feature maps (1321, 1322), where the positions of the first and second regions are taken into account relative to each other so that elements of overlapping regions are summed; and e) outputting (S8) the fused feature map (1330); The method, wherein the first and second images are captured by the same image capture sensor.

2. The method of claim 1 , wherein the first image (401, 701) and the second image (502, 702) correspond to different levels of an image pyramid of the image capture sensor.

3. 2. The method of claim 1, wherein the first region (101, 601) is an overview region of the scene and the second region (502, 702) is a sub-region of the overview region of the scene.

4. The method of claim 1 , wherein the first image has a first resolution and the second image has a second resolution, the second resolution being greater than the first resolution.

5. The method of claim 1 , wherein the first and second output feature maps (1321, 1322) have the same height and width in the overlapping region.

6. The method of claim 1 , wherein the height and width of the fused feature map (1330) are determined by a rectangle that encloses the first and second output feature maps (1321, 1322).

7. 2. The method of claim 1, wherein the feature maps (1301, 1302, 1321, 1322) each have a depth that depends on the resolution of the image (401; 502; 701; 702).

8. The method of claim 1 , wherein the fusion feature map is generated in an encoder of an artificial neural network configured to determine ADAS / AD-related information.

9. 9. The method of claim 8, wherein the artificial neural network configured to determine ADAS / AD related information comprises a plurality of decoders for different ADAS / AD detection functions.

10. A system (10) for fusing image data of at least one image capture sensor, comprising an input interface (12), a data processing unit (14) and an output interface (18): a) the input interface (12) is configured to receive input image data, the input image data comprising: a first image (401,701) containing a first region (101,601) of the scene, and a second image (502, 702) including a second region (102, 602) of said scene, said first and said second regions overlapping each other but not being identical; The data processing unit (14) b) determining a first feature map (1301) having a first height and width based on the first image (401, 701) and determining a second feature map (1302) having a second height and width based on the second image (502, 702); c) configured to compute a first output feature map (1321) using a first convolution of the first feature map (1301) and to compute a second output feature map (1322) using a second convolution of the second feature map (1302); d) configured to compute a fused feature map by element-wise addition of the first and second output feature maps (1321, 1322), the positions of the first and second regions being taken into account relative to one another so that elements of overlapping regions are added; e) the output interface (18) is configured to output the fused feature map (1330); The system (10), wherein the first and second images are captured by the same image capture sensor.

11. 11. The system of claim 10, wherein the system comprises a convolutional neural network having an encoder and at least one decoder, the input interface, the data processing unit, and the output interface being implemented in the encoder such that the encoder is configured to generate the fused feature map, and the at least one decoder is configured to implement an ADAS / AD detection function based at least on the fused feature map.

12. A vehicle comprising at least one image capture sensor (1) and a system according to claim 10 or 11.

Images