Methods for fusing sensor data in the context of artificial neural networks
By employing two independent CNNs to process overview and central image regions and fusing feature maps based on relative positions, the method addresses the challenge of integrating sensor data with varying resolutions, enhancing object detection efficiency in ADAS/AD systems.
Patent Information
- Application Number
- JP2024527772
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-03
- Filing Date
- 2022-11-03
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2042-11-03
AI Technical Summary
Existing methods struggle to efficiently fuse sensor data with different capture regions and resolutions in artificial neural networks, particularly in ADAS/AD systems, leading to increased computational costs and challenges in identifying overlapping objects.
The method involves separating overview and central image regions using two independent CNNs, and then fusing feature maps by considering the relative positions of overlapping regions, allowing efficient processing with reduced computational costs.
This approach enables efficient fusion of sensor data with different resolutions, improving object detection in ADAS/AD systems by reducing computational overhead and enhancing the detection of both nearby and distant objects.
Smart Images

Figure 2024544963000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a method and system for fusing sensor data in the context of artificial neural networks, for example in environment-based ADAS / AD systems for vehicles. [Background technology]
[0002] In ambient sensors (especially camera sensors) for ADAS / AD systems, the resolution is constantly increasing. This makes it possible to detect small objects, to detect sub-objects and, for example, to read small characters at a great distance. The disadvantage of high resolution is the significantly higher computational power required to process the correspondingly large sensor data. Therefore, different resolution levels of the sensor data are often used for processing. A large range or high resolution is, for example, often required in the image center, while not required in the peripheral areas (e.g. similar to the human eye).
[0003] Patent document 1 describes a camera device for imaging the surroundings of a vehicle, comprising an image sensor device for capturing pixel images and a processing device configured to combine adjacent pixels of the pixel images in an adapted pixel image. By combining pixel values of adjacent pixels in the form of a 2x2 image pyramid or an nxn image pyramid, different adapted pixel images can be generated at different resolutions.
[0004] US Pat. No. 5,399,433 and US Pat. No. 5,499,443 describe driving assistance systems which take images with variable resolution.
[0005] Patent document 4 describes a camera device for capturing the surrounding area of a vehicle, comprising a wide-angle optical element and a high-resolution image capture sensor. For the images of the image sequence, it is possible to capture the entire capture area image with reduced resolution using pixel binning, or a partial area of the capture area with full resolution.
[0006] Techniques using artificial neural networks are increasingly being used in ADAS / AD systems based on ambient sensors, which allow better recognition, classification and at least partial understanding of road users and scenes. In this regard, deep neural networks, e.g. Convolutional neural networks (CNNs, in German, e.g. "convolution-based neural networks" or "convolutional networks"), have clear advantages over conventional approaches, which use rather manually designed features (Histogram of oriented gradients (HOG), Local Binary Patterns (LBP), Gabor filters, etc.) by means of trained classifiers, e.g. Support Vector Machines or AdaBoost. In the case of (multi-layer) CNNs, feature extraction is performed by machine (deep) learning using algorithms, which greatly increase the dimensionality and depth of the feature space, ultimately resulting in significantly improved performance, e.g. higher detection rates.
[0007] A particular challenge is the process of integrating sensor data with different and even overlapping capture areas and different resolutions.
[0008] Patent document 5 describes a CNN-based method for learning the parameters of an object detector. An object region is estimated in a camera image and excerpts of this region are generated from different image miramid levels. The excerpts, for example, have the same height, are filled laterally with "zero padding" (zero regions) and are concatenated (in English, concatenated). This form of concatenation can be roughly described as a collage-like form, where excerpts of the same height are "pasted together". The generated composite image is thus a composite of a region of the same original camera image from different resolution levels. The CNN is trained so that the object detector detects objects based on the composite image, and is thus also able to detect objects at greater distances. The advantage of such an approach, versus processing each image region separately one after the other using a CNN, is that the weights only need to be applied once for the composite image. The disadvantage in this case is that image regions in the composite image are taken into account adjacent, in particular, independent of each other, by CNNs using object detectors: objects present in overlapping regions, which may be incompletely included in the image regions, need to be identified in a non-trivial way as belonging to one and the same object. [Prior art documents] [Patent documents]
[0009] [Patent Document 1] DE 102015208889 A1 [Patent Document 2] U.S. Pat. No. 1,074,2907 [Patent Document 3] U.S. Pat. No. 1,075,730 [Patent Document 4] U.S. Pat. No. 1,079,8319 [Patent Document 5] European Patent Application Publication No. 3686798 [Patent Document 6] DE 102020204840 Summary of the Invention [Problem to be solved by the invention]
[0010] The objective of the present invention is to provide an improved sensor data fusion method in the context of an artificial neural network, which efficiently fuses input sensor data having different capture areas and different resolutions and supplies them for further processing. [Means for solving the problem]
[0011] One aspect of the present invention is a method for producing a composition comprising the steps of: a) Capture a large image area; b) capturing a relevant image region, e.g., a remote object in the center of the image, at high resolution; The present invention relates to an efficient implementation of object detection based on input data of at least one image capture sensor.
[0012] Before describing the present solution, the following considerations are made.
[0013] Using multiple levels of an image pyramid in the neural network, the overview image with low resolution and the central extract image with high resolution can be processed separately by two independent inferences (two CNNs trained for each). This implies large computational / runtime costs: in particular, the trained CNN weights need to be multiplied anew for each different image, and features from different pyramid levels are not considered jointly.
[0014] Alternatively, the process of, for example, US Patent No. 5,999,463 can be performed on the combined image from different resolution levels. Thus, a combined image from different sub-images / resolution levels is generated and inference or trained CNN is performed on it. This is somewhat more efficient because each weight needs to be applied only once for all images, rather than anew for each sub-image. However, other disadvantages remain, such as the lack of combination of features from different resolution levels.
[0015] Methods for fusing sensor data include: a) receiving input sensor data, the input sensor data comprising: a first representation including a first region of the scene; and - a second representation including a second region of the scene, the first and second regions, while overlapping with one another, are not identical; b) determining a first feature map having a first height and width based on the first representation, and determining a second feature map having a second height and width based on the second representation; c) computing a first output feature map using a first convolution of the first feature map, and computing a second output feature map using a second convolution of the second feature map; d) computing a fused feature map by element-wise summing the first and second output feature maps, where the positions of the first and second regions are taken into account relative to each other such that elements (of the first and second output feature maps) of overlapping regions are summed; e) outputting a fused feature map.
[0016] The representation may for example be a two-dimensional representation of the scene captured by a sensor. The representation may for example be a grid, a map or an image. A point cloud or depth map is an example of a three-dimensional representation that may be captured, for example, by a lidar sensor or a stereo camera, and may be converted to a two-dimensional representation for many purposes, for example by planar cross-sections or projections. The feature maps may be determined from the representation or from other (existing) feature maps by convolution or convolution layers / convolutional layers / convolutional kernels. The height and width of a feature map are computationally related to the height and width of the underlying representation (or input feature map).
[0017] The mutual positions of the first and second regions are specifically taken into account in order to add the appropriate elements of the first and second output feature maps for fusion. The position of the overlap region is determined, for example, by a starting value (x s ,y s ) In the overlap region, the elements of the first and second output feature maps are added. Outside the overlap region, the elements of the output feature maps that cover this region can be transferred to the fused feature map. If neither of the output feature maps covers a region of the fused feature map, this region can be filled with zeros.
[0018] The method can be carried out, for example, in the context of an artificial neural network, preferably a convolutional neural network (CNN). ADAS / AD functions often (especially on the perception side) use at least one artificial neural network or CNN that is trained using machine learning methods to assign sensor input data to output data related to the ADAS / AD function. ADAS stands for Advanced Driver Assistance Systems and AD stands for Automated Driving (in English, respectively).
[0019] The trained artificial neural network can be implemented in a processor of an ADAS / AD controller in the vehicle. The processor can be configured to evaluate the sensor data by the trained artificial neural network (inference). The processor can include a hardware accelerator for the artificial neural network.
[0020] The processor or inference can be configured to detect or further determine ADAS / AD relevant information, for example, from input sensor data of one or more environmental sensors. The relevant information is, for example, object and / or environmental information for the ADAS / AD system or the ADAS / AD controller. The ADAS / AD relevant object and / or environmental information represents input variables important for the ADAS / AD system, for example, objects, landmarks, road signs, traffic participants, distances to objects, relative speeds of objects, etc. Examples of functions for detecting relevant information are lane detection, object detection, depth perception (3D estimation of image components), semantic detection, traffic sign detection, etc.
[0021] In one embodiment, the first and second output feature maps have the same height and width in the overlap region. In other words, adjacent elements in the overlap region of these output feature maps are equidistant from each other in real space. This is therefore a possible case since the first and second feature maps already have the same height and width in the overlap region. For example, the first and second regions (also) have the same height and width.
[0022] According to one example embodiment, the height and width of the fused feature map is determined by a rectangle that encloses (exactly contains) the first and second output feature maps.
[0023] In one embodiment, the first region is an overview region of the scene and the second region is a sub-region of the overview region of the scene. The overview region included in the first representation may correspond to the entire region, i.e. the maximum capture area of the sensor. The sub-region of the scene included in the second representation may correspond to a Region of Interest (ROI) that is also included in the first representation.
[0024] According to one example embodiment, the first representation has a first resolution and the second representation has a second resolution, e.g., the second resolution is higher than the first resolution. The resolution of the second representation may correspond to the maximum resolution of the sensor. For example, the higher resolution may provide more details regarding a sub-region or region of interest, which is the content of the second representation. The resolution of the representation may correspond to the accuracy or data depth, for example the minimum distance between two adjacent data points of a sensor.
[0025] In one embodiment, after the height and width of the fused feature map are determined by a rectangle that encloses (exactly contains) the first and second output feature maps, the first and / or second output feature maps can be scaled or adjusted such that the positions of the first and second output feature maps are maintained relative to each other so that the first and / or second output feature maps reach the width and height of the fused feature map. The overlapping regions are in the same position in both adjusted output feature maps. The newly added regions of each (adjusted) output feature map due to the scale are filled with zeros (zero padding). Both adjusted output feature maps can then be added element-wise.
[0026] According to one example embodiment, a template output feature map is created whose width and height are derived from the height and width of the first and second output feature maps and the location of the overlap region (see previous paragraph, enclosing rectangle). The template output feature map is filled with zeros.
[0027] For the adjusted first output feature map, elements from the first output feature map in the area covered by the first output feature map are carried over, and a starting value can be used to indicate the position of the first output feature map in the vertical and horizontal directions within the template output feature map. Correspondingly, an adjusted second output feature map is constructed, and both adjusted output feature maps can then be added element-wise.
[0028] In one embodiment for the special case where the second output feature map includes the entire overlap region (i.e., the subregion of the first output feature map that includes the overlap region itself), the adjustment of the different height and width of the second output feature map can be omitted. In this case, no adjustment is made to the first output feature map either, since the fused feature map has the same height and width as the first output feature map. In this case, the element-by-element addition of the second output feature map with the first output feature map may be performed only in the overlap region, based on a suitable starting value. The starting value predetermines from what point in the first output feature map (i.e., in the overlap region) the elements of the second output feature map are added to the elements of the first output feature map to generate the fused feature map.
[0029] In one embodiment, the feature map has a depth that depends on the resolution of the representation: a high resolution representation (e.g., an excerpt image) corresponds to a feature map with a high depth, e.g., the feature map contains more channels. For example, the processor may comprise a hardware accelerator for an artificial neural network, which can further process a stack of sensor channel data "packets" during a clock or computation period. A stack of sensor data or representations or features (maps) may be provided in the hardware accelerator as stacked sensor channel data packets.
[0030] According to one example embodiment, detection of ADAS / AD related features is based on a fusion feature map.
[0031] In one embodiment, the method is implemented in a hardware accelerator for artificial neural networks or CNNs.
[0032] According to one example embodiment, the fusion feature map is generated in an encoder of an artificial neural network or CNN that is configured or trained to determine the ADAS / AD related information.
[0033] In one embodiment, an artificial neural network or CNN configured or trained to determine ADAS / AD related information comprises multiple decoders for various ADAS / AD detection functions. In one embodiment, the representation (of the scene) includes or contains image data of an image capture sensor, which may comprise one or more representative elements from the following group: monocular cameras, stereo cameras, satellite cameras, individual cameras of omnidirectional systems, lidar sensors, laser scanners or other three-dimensional cameras, especially with a wide-angle capture area (e.g., 100° or more) and a large maximum resolution (e.g., 5 megapixels or more).
[0034] According to one example embodiment, the first and second representations include image data of at least one image capture sensor.
[0035] In one embodiment, the (only) image capture sensor is a monocular camera. The first and second representations may be provided by the (same) image capture sensor. The first representation (or first image) may correspond to an overview image with reduced resolution, captured at a wide angle, and the second representation (or second image) may correspond to a partial image with high resolution.
[0036] According to one example embodiment, the first and second images correspond to different image pyramid levels of an image captured by an image capture sensor. The input sensor data, i.e., the input image data, may be or may be encoded into multiple channels depending on the resolution, for example each channel having the same height and width. In this case, within each channel, the spatial relationships of the pixels involved may be maintained. For further details in this regard, reference is made to US Pat. No. 6,399,633, the contents of which are incorporated herein in their entirety.
[0037] In one embodiment, two monocular cameras with overlapping capture areas are used as the image capture sensor or sensors. The two monocular cameras may be components of a stereo camera. The two monocular cameras may have different aperture angles and / or resolutions ("hybrid stereo camera"). Both monocular cameras may be satellite cameras mounted on the vehicle independently of each other.
[0038] According to one example embodiment, multiple cameras of an omnidirectional camera system are used as image capture sensors. For example, four monocular cameras with fisheye lenses (capture angle is, for example, 180° or more) can capture the vehicle's surroundings in their entirety. Every two adjacent cameras have an overlapping area of about 90°. In this case, a fusion feature map of the vehicle's 360° surroundings can be created from four separate images (four representations).
[0039] A further aspect of the invention relates to a system or apparatus for fusing sensor data, the apparatus comprising an input interface, a data processing unit and an output interface. The input interface is configured to receive input sensor data, the input sensor data including first and second representations, the first representation including or containing a first region of the scene. The second representation contains a second region of the scene. The first and second regions overlap each other. The first and second regions are not identical. The data processing unit is configured to carry out the following steps b) to d). b) determining a first feature map having a first height and width based on the first representation, and determining a second feature map having a second height and width based on the second representation. c) computing a first output feature map using a first convolution of the first feature map, and computing a second output feature map using a second convolution of the second feature map. d) computing a fused feature map by element-wise summing the first and second output feature maps, the element-wise summing taking into account the positions of the first and second regions relative to each other, such that elements (of the first and second output feature maps) of overlapping regions are summed. An output interface is configured to output the fused feature map. The output may be provided to a downstream ADAS / AD system or may be provided to downstream layers of a CNN or further artificial neural network for a "big" ADAS / AD.
[0040] According to one example embodiment, the system includes a CNN hardware accelerator, the input interface, the data processing unit and the output interface being implemented in the CNN hardware accelerator.
[0041] In one embodiment, the system includes a convolutional neural network having an encoder, the input interface, the data processing unit, and the output interface being implemented in the encoder such that the encoder is configured to generate a fused feature map.
[0042] According to an example embodiment, the convolutional neural network comprises a plurality of decoders, which are configured to implement various ADAS / AD detection functions based on at least the fused feature map. Thus, the multiple decoders of the CNN can use input sensor data encoded by a common encoder. The various ADAS / AD detection functions are, for example, semantic segmentation of one or more representations, free space detection, lane detection, object detection or object classification.
[0043] In one embodiment, the system includes an ADAS / AD controller configured to implement the ADAS / AD sensing function based at least on a result of the ADAS / AD sensing function.
[0044] The system may include at least one sensor, such as one or more camera sensors, radar sensors, lidar sensors, ultrasonic sensors, localization sensors, and / or a V2X system (Vehicle to X, i.e., telematics system).
[0045] A further aspect of the invention relates to a vehicle comprising at least one sensor and a corresponding system for fusing sensor data.
[0046] The system or data processing unit may comprise, inter alia, a microcontroller or microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural / AI processing unit (NPU), a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), etc., together with software for performing the corresponding method steps.
[0047] According to one embodiment, the system or data processing unit is implemented in a hardware-based sensor data pre-processing stage (eg, an Image Signal Processor (ISP)).
[0048] The invention also relates to a computer program element or program product which, when the processor of the system is configured to execute a data fusion program, instructs the processor to execute a method for fusing input sensor data corresponding thereto.
[0049] The invention also relates to a computer-readable storage medium having such program elements stored thereon.
[0050] Accordingly, the invention may be implemented in digital electronic circuitry, in computer hardware, firmware, or software.
[0051] In the following, exemplary embodiments and drawings will be described in the context of the present invention. [Brief description of the drawings]
[0052] [Figure 1] FIG. 1 illustrates a system for fusing data from at least one sensor. [Diagram 2] FIG. 2 shows generally the extents and positions of first and second capture areas of a sensor, or of two different sensors, from which first and second representations of a scene can be determined. [Diagram 3] FIG. 3 shows the overall image with high resolution. [Figure 4] FIG. 4 shows an overview or overview image with reduced resolution. [Diagram 5] FIG. 5 shows a central excerpt image with high resolution. [Figure 6] FIG. 6 shows an alternative arrangement of the first (general) capture area and the second central capture area. [Figure 7] FIG. 7 shows an example of how the corresponding digital image would appear as a grayscale image. [Figure 8] FIG. 8 shows a basic method for fusing such images. [Figure 9] FIG. 9 shows an alternative second fusion method. [Figure 10] FIG. 10 illustrates a third advantageous fusion method. [Figure 11] FIG. 11 shows the concatenation of two feature maps that are then processed (and fused) by a convolution kernel. [Figure 12] FIG. 12 shows an alternative flow in which two feature maps are processed by two separate convolution kernels followed by element-wise addition. [Figure 13] Figure 13 shows the flow of fusing two feature maps of different widths and heights. [Figure 14] FIG. 14 shows a possible method flow. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0053] FIG. 1 shows diagrammatically a system 10 for fusing data of at least one sensor 1 comprising an input interface 12, a data processing unit 14 with a fusion module 16 and an output interface 18 for outputting the fused data to a further unit 20.
[0054] An example of the sensor 1 is a monocular camera sensor having wide-angle optics and a high-resolution image capture sensor, such as a CCD or CMOS sensor. Further examples of the sensor 1 may be a radar sensor, a lidar sensor, an ultrasonic sensor, a self-localization sensor or a V2X system. In many cases, the sensors have different resolutions and / or capture areas. For fusion, data pre-processing is useful, which allows fusing features from the sensor data.
[0055] An example embodiment described in more detail below relates to processing a first image of a camera sensor and a second image of the camera sensor, the second image comprising (only) a partial region of the first image and having a higher resolution compared to the resolution of the first image. Based on the image data of the camera sensor a number of ADAS or AD functions, such as lane detection, lane keeping assist, traffic sign recognition, speed limit assist, traffic participant detection, collision warning, emergency braking assist, distance control, construction site assist, highway pilot, cruising chauffeur function and / or autopilot, can be provided by the ADAS / AD control device, for example to a further unit 20.
[0056] The overall system 10, 20 may include an artificial neural network, e.g., a CNN. To allow the artificial neural network to process image data in real time, e.g., in a vehicle, the overall system 10, 20 may include a hardware accelerator for the artificial neural network. Such a hardware component can essentially accelerate a software-implemented neural network specifically so that the neural network can operate in real time.
[0057] The data processing unit 14 can process image data in a "stacked" format, i.e., can read and process a stack of multiple input channels within a computation period (clock period). In a specific example, the data processing unit 14 can read four image channels with a resolution of 576x320 pixels. Fusing at least two image channels provides an advantage for subsequent CNN detection that the already fused channel information or feature map can be processed by the CNN, without the need to process the channels individually by their corresponding CNNs. Such fusion can be performed by the fusion module 16. Details of fusion are described in more detail below with reference to the following figures.
[0058] The fusion may be performed in an encoder of the CNN. The fused data may then be processed by one or more decoders of the CNN to obtain detection or other ADAS / AD related information. In such a configuration, the encoder may be represented by block 10 and the one or more decoders may be represented by block 20 in FIG. 1. The CNN would comprise blocks 10 and 20, hence the notation "whole system."
[0059] Figure 2 shows diagrammatically the extent and location of a first capture area 101 and a second capture area 102 of a sensor or of two different sensors, from which a first and a second representation of a scene can be determined. In the case of a camera sensor, this corresponds to a first image capture area 101, from which an overview or overall image can be captured as the first representation, and a second image capture area 102, for example a central image area for the second representation, including an excerpt of the first image capture area 101. Figures 3-5 show an example in which images can be captured using a camera sensor.
[0060] Fig. 3 shows a schematic overview or overall image 300 with high resolution. A scene is captured including nearby and more distant traffic participants (304 and 303) on a road 305 or roadway passing by a house 306. A camera sensor can capture such an overall image with maximum width, height and resolution (or number of pixels). However, processing this large amount of data (e.g. in the range of 5-10 megapixels) is generally not possible in real time in AD or ADAS systems, leading to further processing of reduced image data.
[0061] Fig. 4 shows a schematic overview or overview image 401 with reduced resolution. When the resolution is halved, the number of pixels drops by a factor of four. The overview image 401 with reduced resolution is referred to in the following as a wfov (wide field of view) image. Nearby traffic participants 404 (vehicles) can be detected from the wide field of view image even in the case of reduced resolution. On the other hand, distant traffic participants 403 (pedestrians) are not detectable from this wide field of view image due to the limited resolution.
[0062] 5 shows diagrammatically a central excerpt image with high (or highest) resolution 502. The excerpt image with high resolution 502 is referred to as the central image in the following. The central image allows for high resolution detection of remote pedestrians 503, while nearby vehicles 504 are not included or are only partially included in the capture area of the central image 502.
[0063] 6 shows an alternative arrangement of a first (overview) capture area 601 and a central capture area 602. This central capture area 602 is located "below", i.e. starts vertically at the same height as the overall capture area 601. The start values (x0,y0) can indicate the position of the central capture area 602 in the horizontal and vertical directions within the overall or overview capture area.
[0064] Fig. 7 shows an example of how the corresponding digital image appears as a grayscale image. At the bottom, as a first image, a wide field of view image 701 captured by the vehicle's front camera can be seen. The vehicle is driving towards a road junction. Perpendicular to the direction of travel, a large, possibly multi-lane road runs. Parallel to the large road, a bicycle lane runs. Traffic lights control the right of way of traffic participants. Buildings and trees demarcate the road and the footpath.
[0065] The central excerpt 702 is shown in faded color in the wide-field image 701 to illustrate that this excerpt exactly corresponds to this excerpt 702 of the first image 701 as a second image (central image) 7020 with a higher resolution. The second image 7020 is shown on top, where it is easier for a human observer to recognize that the traffic light is showing a red light to the ego vehicle, that the bus has just crossed the intersection from left to right, and further details of the captured scene. Based on the higher resolution of the second image 7020, even more distant objects or traffic participants can be robustly detected by image processing.
[0066] The image pyramid may have, for example, 2304x1280 pixels at the highest level for the second (central) image, 1152x640 pixels at the second level, 576x320 at the third level, 288x160 at the fourth level, 144x80 pixels at the fifth level, etc. The image pyramid for the first (wide field) image will of course have many more pixels for the same resolution (i.e. for the same levels as the central image).
[0067] Since the wide-field image and the central image are generally derived from different pyramid levels, the central image is adapted to the resolution of the wide-field image by a resolution reduction operation. In this case, the number of channels in the feature map of the central image is generally increased (the amount of information per pixel is increased). The resolution reduction operation is, for example, stride or pooling. In the case of stride, only every second (or fourth or nth) pixel is read. In the case of pooling, multiple pixels are summarized into one pixel, for example, in the case of max pooling, the maximum value of a pixel pool (e.g., of two pixels or 2x2 pixels) is taken over.
[0068] Assume that the level 5 overview image has 400x150 pixels, and the level 5 central image is located horizontally at x0=133 pixels from the left edge of the overview image and extends vertically to y0=80 pixels from the bottom edge of the overview image. Assume that each pixel corresponds to one element of the output feature map. Then, to adjust the second output feature map, we need to add 133 zeros per row (one zero for each pixel) on the left side, 70 zeros per column on the top side, and 133 zeros per row on the right side, so that we can add, element by element, the channels of the first output feature map to the channels of the adjusted second output feature map. The starting point values x0,y0 are determined from the position of the (second) representation of the subregion within the (first) representation of the overview region. The starting point values x0,y0 indicate the horizontal and vertical shift or extension.
[0069] FIG. 8 shows a schematic diagram of a basic method for fusing such images (eg, the first or widefield image 701 and the second or central image 7020 of FIG. 7).
[0070] The wide-field image is sent as input image data to the first convolutional layer c1 of an artificial neural network (e.g., CNN). The central image is sent as input image data to the second convolutional layer c2 of the CNN. Each convolutional layer has an activation function and optional pooling.
[0071] The central image is padded with a "large" zero padding ZP region so that its height and width match those of the wide-field image, thereby maintaining the spatial relationship. Based on FIG. 7, it can be thought of as padding with zeros the regions 701 other than the central extract image 702 for the central image 7020 (i.e., the regions from the wide-field image 701 in the lower part of FIG. 7 that are shown as non-bleached and therefore darker). The high resolution of the central image 7020 leads to a high depth of the (second) feature map generated by the second convolutional layer c2. The height and width of the second feature map correspond to the height and width of the central extract image 702 of the wide-field image 701. Here, the adjustment of the different heights and widths of the first and second feature maps is performed by the zero padding ZP of the second feature map.
[0072] The features of the wide-field image and the central image are concatenated. The concatenated features are sent to the third convolutional layer c3, which creates a fused feature map.
[0073] During convolution with the second feature map filled with zero padding ZP, many multiplications with 0 are required. These calculations of "0" multiplications of the zero padding ZP regions in the convolutional layer c3 are unnecessary and therefore not advantageous. On the other hand, it is not always possible to truncate these regions, since, for example, known CNN accelerators do not allow spatial control of the application region of the convolution kernel.
[0074] However, the advantage is that the depths of both feature maps can be different. Concatenation combines both feature maps "deeply together". This is particularly advantageous when the central image has a higher resolution than the wide-field image, and therefore more information can be extracted from the central image. In this respect, the method is relatively flexible.
[0075] Fig. 9 shows a schematic of an alternative second method, where the wide-field features and the central features are combined via appropriate element-wise addition (+) (instead of concatenation cc of both feature maps), before the central image is adapted in height and width using zero padding ZP after feature extraction by the second convolutional layer c2. The feature map with the element-wise added features is fed to the third convolutional layer c3.
[0076] Also, the method accepts a performance penalty because addition combines features with different semantic meanings, and it is not advantageous for the tensors to have the same dimensions. The advantage is that adding zeros (in the zero-padded ZP domain) requires significantly less computation time than multiplying zeros.
[0077] Both of the above methods offer advantages and disadvantages, and it would be desirable to take advantage of the advantages of each, which is possible in a skilled combination.
[0078] FIG. 10 illustrates a schematic diagram of an advantageous method. Based on the first alternative method shown in FIG. 8, i.e. combining features by concatenation, in the following we describe a mathematical decomposition by c3, which obsoletes the multiplication by zeros in the zero-padding ZP domain, which is not necessarily required. Convolutional layer c n o n 3D tensor FM with feature stacks (channels) n where n is a natural number. For conventional 2D convolution, the following applies:
[0079]
number
[0080]
number
[0081] Subsequent concatenation using convolution layers (see Figure 8) produces two reduced convolutions c 3A and c 3B and then element-wise addition (+) is performed.
[0082]
number
[0083]
number
[0084] Both reduced convolutions c 3A and c 3B Adaptation for different heights and widths of the feature maps generated from is done before the element-wise addition (+). Convolution kernel c3 is 3A and c 3B By decomposing it into 3B is applied to the reduced size of the central image with high runtime efficiency. This element-wise addition (+) is runtime-neutral on currently available accelerators for artificial neural networks.
[0085] Zero padding ZP with subsequent addition is equivalent to summing the median feature values of the adapted starting points. Alternatively, the median feature map may be written to a larger region that has been initialized with zeros before. In this case, zero padding ZP is done indirectly.
[0086] The activation function / pooling after c3 cannot be decomposed and is applied after addition. In particular, for large padding regions of zeros, no convolution operations are computed.
[0087] Overall, this embodiment offers the following particular advantages: a) For example, for distant objects, utilizing regions of interest with high resolution, using a large viewing angle / capture area of the sensor, and integrated feature considerations of different (image) pyramid levels for optimal overall performance. b) At the same time, it is a highly runtime efficient implementation.
[0088] In Figures 11 to 13 the method is again illustrated differently.
[0089] Figure 11 shows a schematic of the concatenation of two feature maps 1101, 1102 processed by a convolution kernel 1110, resulting in a fused feature map 1130 that can be output. Unlike the similar situation in Figure 8, here both feature maps 1101, 1102 have the same width w and height h. They are shown simply as two rectangular planes. Concatenation means combining together "depth", and the second feature map 1102 is shown diagrammatically as being located spatially behind the first feature map. Here, the convolution kernels 1110 are shown with opposing hatching for comparison, to indicate that a first part, i.e., the “first convolution 2D kernel” shown with light hatching, scans the first feature map 1101, and a second (shown with dark hatching) convolution 2D kernel scans the second feature map 1102. The result is the fused output feature map 1130. The fused feature map 1130 is no longer separable with respect to the first and second feature maps 1101, 1102 after convolution.
[0090] 12 shows a schematic of an alternative flow for fusing two feature maps of the same width w, height h and depth d. The depth d of the feature maps may correspond to the number of channels or may be resolution dependent. Here, a first feature map 1201 is scanned by a first convolutional 2D kernel 1211 to generate a first output feature map 1221, and a second feature map 1202 is scanned by a second convolutional 2D kernel 1212 to generate a second output feature map 1222. The convolutional 2D kernels 1211;1212 may have dimensions of, for example, 3x3x the number of input channels to generate an output layer. The depth of the output feature map can be defined by the number of convolutional 2D kernels 1211;1212. The fused feature map 1230 can be computed from both output feature maps 1221, 1222 by element-wise addition (+). The flow here, i.e. performing two separate convolutions for each feature map and then simply adding them together, is equivalent to the flow in Figure 11, where two feature maps are concatenated and then a convolution is performed.
[0091] FIG. 13 illustrates a schematic flow for fusing two feature maps of different widths and heights, corresponding to the flow described in FIG.
[0092] A first feature map 1301 (computed from a wide field of view image) has a large width w and height h but a small depth d, while a second feature map 1302 (computed from a high resolution central extract image) has a small width w and height h but a large depth d. The first convolutional 2D kernel 1311 scans the first feature map 1301, which generates a first output feature map 1321 with an expanded depth d. The second convolutional 2D kernel 1312 scans the second feature map, which generates a second output feature map 1322 (a diagonal hatched diced region). The depth d of the second output feature map is the same as that of the first output feature map. To perform the fusion of the first and second output feature maps 1321, 1322, it is advantageous to take into account the location of the subregions within the overview region. Thus, the height and width of the second output feature map 1322 are expanded to match the height and width of the first output feature map 1321. The starting values of the width and height for adjustment are, for example, information on the position of the central region 602 or 702 of the common outline region 601 or 701 from FIG. 6 or FIG. 7, for example, in the form of starting values x0, y0, or the width starting value x of the feature map derived therefrom. s and the starting height value y s The second output feature map 1322 can be determined in the form of: The missing regions (left, right and top) in the second output feature map 1322 are filled with zeros (zero padding). The second output feature map adjusted in this way can be fused with the first output feature map 1321 by simple element-wise addition. Such a fused feature map 1330 is shown at the bottom of FIG.
[0093] FIG. 14 shows a schematic diagram of a possible method flow.
[0094] In a first step S1, input data of at least one sensor is received. The input sensor data may be generated, for example, by two forward-facing ADAS sensors of the vehicle, for example by a radar and a lidar with partially overlapping capture areas. The lidar sensor may have a wide capture area (e.g. a large aperture angle of 100° or 120°), which generates a first representation of the scene. The radar sensor captures only a (central) sub-area of the scene (e.g. a small capture angle of 90° or 60°), but may instead detect more distant objects, which generates a second representation of the scene.
[0095] To enable fusion of the lidar and radar sensor input data, the sensor raw data may be mapped to a representation, where the images show a bird's-eye view onto the roadway plane ahead of the vehicle. The representation or a feature map determined on the basis thereof may also be created, for example in the form of an occupancy grid. In the overlap region, there is lidar and radar data, in the side edge regions there is only lidar data, and in the further forward region there is only radar data.
[0096] In a second step S2, a first feature map is determined from the input data. From the (first) representation of the lidar sensor, a first feature map can be generated having a first height and width (or roadway plane depth and roadway plane width in a bird's-eye view).
[0097] In a third step S3, a second feature map is determined from the input data. From the (second) representation of the radar sensor's capture area, a second feature map can be generated having a second height and width, where the width of the second feature map is smaller than the width of the first feature map and the height of the second feature map (distance in the direction of travel) is larger than the height of the first feature map.
[0098] In a fourth step S4, a first output feature map is determined based on the first feature map, where the first output feature map is computed by means of a first convolution of the first feature map.
[0099] In a fifth step S5, a second output feature map is determined based on the second feature map, the second output feature map being computed using a second convolution of the second feature map, the second convolution being limited in width and height to the height and width of the second feature map.
[0100] In a sixth step S6, an adjustment of the different sizes of the first and second output feature maps is performed, in particular an adaptation of the height and / or width.
[0101] Also, according to the first modified example, the height of the first output feature map can be expanded to match the height of the second output feature map. The width of the second output feature map can be expanded to match the width of the first output feature map. The area of each (adjusted) output feature map newly added by the expansion is filled with 0 (zero padding).
[0102] According to the second variant, first, a template output feature map is created, whose width and height are obtained from the height and width of the first and second output feature maps and the location of the overlapping area. The template output feature map is filled with zeros. In this case, the template output feature map has the width of the first output feature map and the height of the second output feature map.
[0103] The adjusted first output feature map inherits elements from the first output feature map in the area it covers, and a starting point value can be used to indicate the position of the first output feature map vertically and horizontally within the template output feature map.
[0104] The lidar output feature map, for example, extends across the entire width of the template output feature map, while the remote regions are empty. Thus, in the vertical direction, the starting point value y sis predefined, from which the template output feature map is "filled in." Similarly, based on the template output feature map pre-filled with zeros, an adjusted second output feature map is generated by inserting elements of the second output feature map from a preferred starting position. For example, the radar output feature map is first calculated by dividing the horizontal starting point x s , and extend vertically across the entire height.
[0105] In a seventh step S7, the adjusted first and second output feature maps are fused by element-wise addition. By adjusting the height and width, element-wise addition of both output feature maps is immediately possible for a typical CNN accelerator. The result is a fused feature map.
[0106] In the special case where the second output feature map contains the entire overlap region (i.e., contains the subregion of the first output feature map itself that contains the overlap region (see FIG. 13)), the adjustment of the different heights and widths of the second output feature map can be omitted, in which case the element-wise addition of the second output feature map with the first output feature map is performed only in the overlap region based on the preferred starting point value, where the height and width of the fused feature map are the same as the height and width of the first output feature map (see FIG. 13).
[0107] In an eighth step S8, the fused feature map is output. [Explanation of symbols]
[0108] 1 Sensor 10. System 12 Input Interface 14 Data Processing Unit 16 Fusion Model 18 Output Interface 20 Control Unit 101 Overview area 102 Partial area 300 high resolution overview images 303 Pedestrians or further remote traffic participants 304 Vehicles or nearby traffic participants 305 Roads or roadways 306 Houses 401 Reduced Resolution Overview Image 403 Pedestrian (undetectable) 404 vehicles 502 high resolution central extract images 503 Pedestrians 504 Vehicles (undetectable or completely undetectable) 601 Overview area 602 Partial area 701 Reduced Resolution Overview Image 702 Capture area of excerpt image with high resolution 7020 High resolution (center) excerpt image 1101 First feature map 1102 Second feature map 1110 Convolution Kernel 1130 Fusion Feature Map 1201 First feature map 1202 Second feature map 1211 First Convolution 2D Kernel 1212 Second Convolution 2D Kernel 1221 First output feature map 1222 Second output feature map 1230 Fusion Feature Map 1301 First feature map 1302 Second feature map 1311 First Convolution 2D Kernel 1312 Second Convolution 2D Kernel 1321 First output feature map 1322 Second output feature map 1330 Fusion Feature Map x0 Horizontal starting point value y0 Vertical start or extension value wfov Overview image with reduced resolution center High resolution (center) excerpt image
number
Claims
1. In a method for fusing sensor data: a) receiving input sensor data, the input sensor data comprising: a first representation (401,701) containing a first region of the scene (101,601), and - a step (S1) including a second representation (502, 702) including a second region (102, 602) of said scene, said first and said second regions overlapping each other but not being identical; b) determining (S2) a first feature map (1301) having a first height and width based on the first representation (401, 701), and determining (S3) a second feature map (1302) having a second height and width based on the second representation (502, 702); c) computing (S4) a first output feature map (1321) using a first convolution of the first feature map (1301), and computing (S5) a second output feature map (1322) using a second convolution of the second feature map (1302); d) computing a fused feature map (1330) by element-wise summing of the first and second output feature maps (1321, 1322), the positions of the first and second regions being taken into account relative to one another such that elements of overlapping regions are summed; and e) outputting (S8) the fused feature map (1330).
2. The method of claim 1 , wherein the first and second output feature maps (1321, 1322) have the same height and width in overlapping regions.
3. The method of claim 1 or 2, wherein the height and width of the fused feature map (1330) is determined by a rectangle enclosing the first and second output feature maps (1321, 1322).
4. The method according to any one of claims 1 to 3, wherein said first region (101, 601) is an overview region of said scene and said second region (502, 702) is a sub-region of said overview region of said scene.
5. The method of any one of claims 1 to 4, wherein the first representation has a first resolution and the second representation has a second resolution, the second resolution being higher than the first resolution.
6. The method according to any one of claims 3 to 5, wherein the first and / or the second output feature map (1321, 1322) are enlarged such that the first and / or the second output feature map (1321, 1322) reach the width and height of the fused feature map (1330) and the positions of the first and the second output feature maps (1321, 1322) are maintained relative to each other, and the area of each adjusted output feature map newly added by the enlargement is filled with zeros (ZP).
7. A template output feature map is first created, whose width and height are obtained from the height and width of the first and second output feature maps (1321, 1322) and the location of the overlapping region, and the template output feature map is filled with zeros; For the adjusted first output feature map, elements from the first output feature map (1321) in the area covered by the first output feature map (1321) are inherited; 6. The method of claim 1, wherein for the adjusted second output feature map, elements from the second output feature map (1322) in the area covered by the second output feature map (1322) are carried over.
8. 6. The method of claim 4 or 5, wherein the second output feature map (1322) includes the entire overlap region, and the fused feature map (1330) is computed by element-wise addition of the second output feature map (1322) to the first output feature map (1321) only in the overlap region based on a preferred starting point value.
9. The method according to any one of the preceding claims, wherein said feature maps (1301, 1302, 1321, 1322) each have a depth that depends on the resolution of said representation (401; 502; 701; 702).
10. The method according to any one of claims 1 to 9, wherein determining ADAS / AD related information is based on the fused feature map (1330).
11. The method of any one of claims 1 to 10 implemented in a hardware accelerator for artificial neural networks.
12. The method according to any one of claims 1 to 11, wherein the fusion feature map is generated in an encoder of an artificial neural network configured to determine ADAS / AD related information.
13. The method of claim 12 , wherein the artificial neural network configured to determine ADAS / AD related information comprises a plurality of decoders for different ADAS / AD detection functions.
14. A system (10) for fusing sensor data comprising an input interface (12), a data processing unit (14) and an output interface (18): a) the input interface (12) is configured to receive input sensor data, the input sensor data being a first representation (401,701) containing a first region of the scene (101,601), and - a second representation (502, 702) including a second region (102, 602) of said scene, said first and said second regions overlapping each other but not being identical; The data processing unit (14) b) configured to determine a first feature map (1301) having a first height and width based on the first representation (401, 701) and to determine a second feature map (1302) having a second height and width based on the second representation (502, 702); c) configured to compute a first output feature map (1321) using a first convolution of the first feature map (1301) and to compute a second output feature map (1322) using a second convolution of the second feature map (1302); d) configured to compute a fusion feature map by element-wise addition of the first and second output feature maps (1321, 1322), the positions of the first and second regions being taken into account relative to one another such that elements of overlapping regions are added; e) the system (10), wherein the output interface (18) is configured to output the fused feature map (1330).
15. 15. The system of claim 14, wherein the system (10) comprises a CNN hardware accelerator, and the input interface (12), the data processing unit (14) and the output interface (18) are implemented in the CNN hardware accelerator.
16. 16. The system of claim 14 or 15, wherein the system (10) comprises a convolutional neural network having an encoder, the input interface (12), the data processing unit (14) and the output interface (18) being implemented in the encoder such that the encoder is configured to generate the fused feature map.
17. 17. The system of claim 16, wherein the convolutional neural network comprises a plurality of decoders configured to implement various ADAS / AD detection functions based on at least the fused feature map.
18. 20. The system of claim 17, comprising an ADAS / AD controller, the ADAS / AD controller configured to implement an ADAS / AD detection function based at least on a result of the ADAS / AD detection function.
Citation Information
Patent Citations
Information processing device, and information processing method
JP2019096006A
Target detection program, target detection device, and target detection method
JP2020071793A
Information processing device, information processing method, and program
WO2021033591A1
Camera device and method for imaging an environment for a motor vehicle
DE102015208889A1
Processing of multi-channel image data from an image acquisition device by an image data processor
DE102020204840A1
Cited By
MEC-PA-based bearing surface feature extraction method and system, and storage medium
CN120673079A
