Method and apparatus for semantic segmentation of image data

Abstract

A method for performing computationally-efficient semantic segmentation of image data of an imaging sensor with the aid of an artificial neural network, in particular a convolutional neural network, wherein the artificial neural network has an encoder path and a decoder path, wherein the encoder path transitions into the decoder path, wherein the transition is achieved via a discriminant path, wherein the following steps are performed in the discriminant path: decomposing an input tensor according to a decomposition function into at least one first slice tensor and at least one second slice tensor, wherein the input tensor comes from the encoder path; concatenating at least one first slice tensor with at least one second slice tensor according to a concatenation function to obtain a concatenated tensor; and outputting a class tensor to the decoder path of the neural network.

Description

Technical Field

[0001] The invention proceeds from a method and a device for semantic segmentation of image data. The subject matter of the invention is also a computer program. Background Art

[0002] "Fully Convolutional Models for Semantic Segmentation by Evan Shelhamer, Jonathan Long, Trevor Darrell (PAMI: Transactions on Pattern Analysis and Machine Intelligence, 2016)" discloses an extension of convolutional neural networks. Convolutional neural networks are powerful artificial neural networks for processing visual data, which can produce a hierarchy of semantic features of visual data. The document discloses a scheme using a "fully convolutional network" that can accept input data of any size and can output outputs of corresponding size by efficiently inferring features.

[0003] Olaf Ronneberger, Philipp Fischer, Thomas Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation (MICCAI: Medical Image Computing and Computer-Assisted Intervention, Springer, LNCS, Vol. 9351) discloses an artificial neural network architecture and a training strategy for a network based on the use of extended (augmented) training data in order to use existing annotated examples more efficiently. The network architecture includes a "contraction path" (encoder path) for detecting the context of the input data, and symmetrically therewith an "expansion path" (decoder path) that enables precise localization of the detected image features. The artificial neural network can be trained using a relatively small amount of training data. Summary of the invention

[0004] Artificial neural networks for semantic segmentation of image data, in particular for localization and classification of objects in image data, in particular so-called convolutional neural networks (CNNs), have high requirements for computing resources. The need for computing resources is further increased dramatically by adding decoding components or upsampling components, because by adding decoding components or upsampling components, the image data is restored to the original resolution after the semantic analysis in the encoder component. In some embodiments, this may lead to an exponential increase in the need for computing resources.

[0005] In addition to the increased demand for computing resources, pixel-based semantic segmentation of image data when using artificial neural networks (especially CNNs) also requires more memory bandwidth, memory access and storage space during the training phase and application of the network.

[0006] The disadvantage of this additional demand for computing resources, memory bandwidth, memory access, and storage space is exacerbated whenever the application is not implemented on a special computing unit with high memory and distributed computing, such as a graphics processing unit cluster (GPU cluster), but runs on an embedded computing unit, such as embedded hardware.

[0007] In this context, the present invention provides a method, an artificial neural network, a device, a computer program and a machine-readable storage medium for semantic segmentation of image data of an imaging sensor.

[0008] In this context, image data can be understood as data from an imaging sensor. Firstly, this can be understood as data from a video sensor (i.e., a camera). Due to the similarity of the data, data from radar sensors, ultrasonic sensors, lidar sensors, etc. can also be processed as image data with the aid of the present invention. Therefore, in terms of the present invention, radar sensors, ultrasonic sensors, lidar sensors, etc. can be understood as imaging sensors.

[0009] Of particular importance for the present invention are image data of imaging sensors or the like which are suitable for use in vehicles (ie automotive image sensors).

[0010] In this context, semantic segmentation is to be understood as the processing of image data with the aim of determining not only the semantic class of the objects contained in the image, but also the location of the objects in the image. It should be taken into account that global information in the image can lead to conclusions about the semantic class of the object, while local information in the image can lead to conclusions about the location of the object in the image.

[0011] One aspect of the invention is a method for semantic segmentation of image data by means of an artificial neural network, in particular a convolutional neural network (CNN). The artificial neural network has an encoder path for determining semantic categories in the image data and a decoder path for locating the determined categories in the image data. In the transition between the encoder path and the decoder path there is a discriminative path (Discriminative Pfad). The method comprises the following steps in the discriminative path:

[0012] An input tensor is decomposed into at least one first slice tensor and at least one second slice tensor according to a decomposition function, wherein the input tensor comes from an encoder path.

[0013] At least one first slice tensor is concatenated with at least one second slice tensor according to a first concatenation function to obtain a category tensor.

[0014] At least one second slice tensor is concatenated with the category tensor according to a second concatenation function to obtain a decoder tensor.

[0015] Output the decoder tensor to the decoder path of the neural network.

[0016] In this context, an artificial neural network is to be understood as a network of artificial neurons which is used for processing information, for example for processing image data (in particular for localizing and classifying objects in the image data).

[0017] In this context, a convolutional neural network (CNN) is understood to be a class of artificial neural networks that is considered "state of the art" in the field of classification. The basic structure of a CNN consists of an arbitrary sequence of convolutional and pooling layers, which are terminated by one or more fully connected layers. The respective layers consist of artificial neurons.

[0018] In this context, an encoder path is to be understood as a path from the processing of the image data up to the classification of objects in the image data.

[0019] In this context, a decoder path is understood to be a path that follows the encoder path and which proceeds from the classification to restore the original image data in order to locate the classified object.

[0020] In this context, the discriminant path is understood to be the transition region between the encoder path of the artificial neural network and the decoder path of the artificial neural network. This relates to the deepest section of the architecture or structure of an artificial neural network, in particular an artificial neural network according to a convolutional neural network (CNN) architecture.

[0021] In this article, a tensor is understood to be a representation of data during processing in an artificial neural network. A dataset includes the processing state of image data and the associated feature maps. A tensor of level l at the i-th step in an artificial neural network is usually represented as It has n rows, m columns, and f feature maps.

[0022] The input tensor is a data representation before being processed by the method of the present invention. The input tensor according to the present invention comes from the encoder path of an artificial neural network (especially an artificial neural network according to a convolutional neural network architecture). That is, the input tensor is the result of the raw image data processing in the encoder path of the artificial neural network.

[0023] The slice tensor is the data representation after the decomposition step of the method according to the present invention.

[0024] The category tensor is a data representation of the semantic category identified or obtained in the image data to be processed. The category tensor can represent the processing result in the encoder path.

[0025] The decoder tensor is a data representation of the image data to be processed, which can be used as an input data set for the decoder path of an artificial neural network (especially according to a convolutional neural network architecture).

[0026] In this context, a decomposition function is understood to be any function that is suitable for selecting appropriate quantities from the set of input tensors and feature maps. This can be done by factoring, exponentially, etc.

[0027] In this context, a concatenation function may be understood as any function that is suitable for concatenating at least one first slice tensor with at least one second slice tensor. This may be done by concatenation, summing, replacement, duplication, and the like.

[0028] The advantages of the method according to the invention reside in those steps of the method which are carried out in the transition region between the encoder path and the decoder path, ie in the decision path.

[0029] The application of the steps of the method can improve the accuracy of positioning without producing significant "gradient dispersion" effects in the discrimination path.

[0030] "Gradient vanishing" is understood to be an effect that can occur during the training of artificial neural networks, i.e. the disappearance of changes in parameters. In the worst case, this effect can lead to a stagnation in the change or improvement of the training parameters.

[0031] Furthermore, the application of the steps of the method is not hardware-dependent. Thus, the method can be easily implemented not only on embedded hardware (so-called embedded hardware) but also on GPU clusters.

[0032] According to one embodiment of the method according to the invention, the decomposition function in the decomposition step is configured such that only a subset of the feature maps of the input tensor is selected to form the at least one first slice tensor.

[0033] In this article, feature maps are understood to be the output of a layer (level) of an artificial neural network. A CNN usually involves the result of processing first by a convolutional layer and then by a corresponding pooling layer, and the CNN can be used as input data for the next layer or (if provided) a fully concatenated layer.

[0034] According to one embodiment of the method according to the invention, the concatenation function (merge) in the concatenation step is designed in such a way that the dimension of the input tensor remains unchanged.

[0035] This embodiment has the advantage that fewer computing resources can be used with respect to the computing resources to be used and thus the computing resources can be reused with more favorable information of different layers of the artificial neural network, in particular a convolutional neural network (CNN).

[0036] According to one embodiment of the method according to the present invention, the method has a preceding receiving step, wherein in the receiving step, an input tensor and a decomposition function are received.

[0037] This implementation of the method has the following advantages: compared with a conventional parallel implementation of an artificial neural network, it makes the artificial neural network more flexible at a lower cost and can respond more finely to the corresponding input tensor at the corresponding layer (level).

[0038] According to one embodiment of the method according to the invention, in the decomposition step, a first function of the artificial neural network is used for at least one first slice tensor and a second function of the artificial neural network is used for at least one second slice tensor.

[0039] In this article, the function of an artificial neural network can be understood as any function of the neuron layer of the artificial neural network. This can be a convolution or a configuration of convolution blocks: multiple applications of convolution, depthwise convolution, compression, residual, dense, inception, activation, normalization, pooling.

[0040] In this article, Inception should be understood as an architectural variant of artificial neural networks (especially convolutional neural networks), first seen in Going Deep with Convolutions by Szegedy et al. (In the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9, 2015).

[0041] According to one embodiment of the method according to the present invention, the decomposition function in the decomposition step is configured such that the decomposition function includes a quantity to be calculated of the feature map and a corresponding function of the artificial neural network or the computation graph for calculating at least one first slice tensor and at least one second slice tensor.

[0042] This embodiment of the method according to the invention has the advantage that different functions of the artificial neural network can be applied to at least one first slice tensor and at least one second slice tensor in a simple manner, thereby making the artificial neural network more flexible at lower cost and more responsive to the corresponding input tensors compared to a conventional parallel implementation of the artificial neural network.

[0043] Another aspect of the present invention is an artificial neural network for localization and classification of image data, wherein the artificial neural network has an encoder path for classification of image data and a decoder path for localization of image data. In this case, the encoder path transitions into the decoder path. This transition is achieved via a decision path. The network is configured to implement the steps of the method according to the present invention in the decision path.

[0044] In one embodiment of the artificial neural network, the artificial neural network can be configured as a convolutional neural network.

[0045] A further aspect of the invention is a device which is configured to carry out the steps of the method according to the invention.

[0046] A further aspect of the invention is a computer program which is configured to carry out the steps of the method according to the invention.

[0047] Another aspect of the present invention is a machine-readable storage medium on which an artificial neural network according to the present invention and a computer program according to the present invention are stored. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The details and embodiments of the present invention are explained in more detail below with reference to the drawings.

[0049] The accompanying drawings show:

[0050] Figure 1 shows a diagram of a so-called fully convolutional network (FCN) in which the present invention is implemented;

[0051] Figure 2 A block diagram showing a portion of one embodiment of an artificial neural network according to the present invention;

[0052] Figure 3 A block diagram showing a portion of another embodiment of an artificial neural network according to the present invention;

[0053] Figure 4 A flow chart of the method according to the invention is shown. DETAILED DESCRIPTION

[0054] Figure 1 A diagram of an artificial neural network according to a so-called Fully Convolutional Network (FCN) architecture is shown, in which the present invention is implemented.

[0055] A processing path 111 of input data (which may be data from an imaging sensor) extends from left to right through the illustrated artificial neural network.

[0056] The left part of the network shows the encoder path 110. The right part of the network shows the decoder path 120. In the transition region between the encoder path 110 and the decoder path 120, there is a discriminant path 130 at the deepest position of the network.

[0057] By using the decomposition function, a slice tensor 112 is generated in the encoder path 110, which is provided to the decoder path 120. This is shown by the dashed and solid arrows.

[0058] Each layer ends or begins with an input tensor 113, which is usually produced by a compression step (pooling).

[0059] The discriminant path 130 also starts with an input tensor 133, which comes from the encoder path 110. In the network shown, it also passes through a compression step (pooling). In principle, it is conceivable that the input tensor 133 in the encoder path 110 can be provided to the discriminant path 130 as an input tensor 133 by using another function of the artificial neural network.

[0060] In the discriminant path 130, the decomposition function is also applied to the input tensor 133 in order to obtain a slice tensor 132. Subsequently, at least one class tensor 134 is generated by using the function of the artificial neural network.

[0061] Subsequently, the class tensor 134 is concatenated with the at least one slice tensor 132 of the discriminant path 130 by using at least one concatenation function to obtain a decoder tensor 135 .

[0062] Subsequently, the decoder tensor 135 is provided to the decoder path 120 . There, the decoder tensor 135 is first up-converted into the input tensor 123 of the decoder path 120 .

[0063] In the embodiment shown, the input tensor 123 is concatenated with at least one tensor of the encoder path 110 by using a concatenation function to obtain a concatenated tensor 126. In the embodiment shown, this concerns the last encoder tensor of the corresponding layer of the encoder path 110.

[0064] Layers are considered to correspond if the resolutions of the tensors can be used with each other substantially accordingly (i.e. without requiring extensive conversions).

[0065] At least one function of the artificial neural network is applied to the concatenated tensor 126 in order to obtain a correction tensor 127 .

[0066] By using at least one function of an artificial neural network (in order to obtain the correction tensor 127), both the coarse-grained features from the encoder path 110 and the fine-grained features from the decoder path 120 are concatenated with each other.

[0067] In a further process, the correction tensor 127 is concatenated with the slice tensor 112 (so-called skip tensor) of the corresponding layer of the encoder path 110 by using at least one further concatenation function in order to obtain a result tensor 128 .

[0068] The input tensor 123 of the decoder path 120 is refined with the aid of the correction tensor 127 using a concatenation function.

[0069] The resulting tensor 128 is provided as the input tensor 123 of the decoder path 120 to the next higher layer in the decoder path 120 .

[0070] In each layer, the steps used in the decoder path 120 are enclosed by means of boxes. The steps within the boxes constitute a so-called correction module 121.

[0071] By means of the correction module 121, not only the coarse-grained features from the encoder path 110 but also the fine-grained features from the decoder path 120 are concatenated with each other by means of a step using a function of an artificial neural network (in order to obtain a correction tensor 127). In addition, by means of the step of concatenating the correction tensor 127, the concatenated tensor 126 is refined with the aid of the correction tensor 127 in order to obtain a result tensor 128 or to generate an input tensor 123 of the encoder path 120 of the next higher layer.

[0072] Figure 2 A block diagram of a part of an embodiment of an artificial neural network according to the invention is shown. According to this embodiment, the artificial neural network or the method according to the invention is run in the so-called “tensor mode” 200 .

[0073] Here, in the decision path 130, in the i-th step of the artificial neural network, an input tensor x having the number of rows n, the number of columns m and the number of feature maps f is shown. i ∈R n×m××f 133. In addition, there is a decomposition function (slice) 220. According to the decomposition function (slice) 220, the input tensor 133 is decomposed into at least one first slice tensor 230 and at least one second slice tensor 250. Here, the decomposition can be performed according to any decomposition function (slice) 220. In addition, it is conceivable that the decomposition is performed according to a splitting factor, according to an exponent, etc.

[0074] For this purpose, it is provided that at least one first slice tensor 230 is concatenated with the class tensor 134 in the discrimination path by means of a concatenation function to form a decoder tensor 135 in order to be associated there with a coarse, abstract feature representation.

[0075] The at least one second slice tensor 250 is provided to a concatenation function (merge) 260 together with the at least one first slice tensor 230 to generate the concatenated tensor 132. Any rule suitable for concatenating the first slice tensor 230 with the second slice tensor 250 can be used as the concatenation function (merge) 260. In addition, concatenation, summing, replacement, duplication, etc. can be envisioned.

[0076] The concatenated tensor 132 is arranged to be developed into a class tensor 134 in the decision path 130. Here, the further development can be realized by using a function of an artificial neural network.

[0077] Figure 3 A block diagram of a part of another embodiment of an artificial neural network according to the invention is shown. According to this embodiment, the artificial neural network or the method according to the invention is operated in a so-called “function mode” 300 .

[0078] The input data of the function mode 300 includes, in addition to the input tensor 133, a decomposition function (slice) 320. The decomposition function (slice) 320 is applied to the input tensor 133 in order to obtain a first slice tensor 230 and a second slice tensor 250. Furthermore, unlike the "tensor mode" 200, any functions of the artificial neural network 321, 322 are applied to the first slice tensor 230 and the second slice tensor 250. Furthermore, multiple applications of convolution, residual, density, inception, activation, normalization, pooling, etc. can be envisioned. Different functions of the artificial neural network 321, 322 can be applied to the first slice tensor 230 and the second slice tensor 250.

[0079] For this purpose, it is provided that at least one first slice tensor 230 is concatenated with the class tensor 134 in the discrimination path by means of a concatenation function to form a decoder tensor 135 in order to be associated there with a coarse, abstract feature representation.

[0080] At least one second slice tensor 250 is supplied together with at least one first slice tensor 230 to a concatenation function (merge) 260 in order to generate the concatenated tensor 132. Any rule that is suitable for concatenating the first slice tensor 230 with the second slice tensor 250 can be used as the concatenation function (merge) 260. In addition, concatenation, summing, replacement, duplication, etc. are conceivable.

[0081] The cascade tensor 132 is provided for development into a class tensor 134 in the decision path 130. In this case, the further development can be realized by using functions of an artificial neural network.

[0082] Figure 4 A flow chart of the method according to the invention is shown.

[0083] In step 410 , the input tensor 133 is decomposed into at least one first slice tensor 230 and at least one second slice tensor 250 according to the decomposition functions 220 , 320 , wherein the input tensor 133 comes from the encoder path 110 of the artificial neural network.

[0084] This means that the input tensor may be a data representation of the image data 111 to be processed after processing in the encoder path 110 of the artificial neural network.

[0085] In step 420 , at least one first slice tensor 230 is concatenated with at least one second slice tensor 250 according to a first concatenation function 260 to generate at least one concatenated tensor 132 .

[0086] At least one cascade tensor 132 is further developed in the decision path 130 to form a class tensor 134. In this case, the further development can be realized by using functions of an artificial neural network.

[0087] In step 430 , at least one first slice tensor 230 is concatenated with the class tensor 134 in a discriminant path to form a decoder tensor 135 by means of a second concatenation function.

[0088] In step 440, the decoder tensor 135 is output to the decoder path 120 of the artificial neural network for further processing by the artificial neural network.

[0089] The present invention is preferably suitable for use in automotive systems (in particular in conjunction with driver assistance systems up to partially automated or fully automated driving).

[0090] Of particular interest here is the processing of image data or image streams representing the surroundings of the vehicle.

[0091] These image data or image streams can be detected by an imaging sensor of the vehicle. In this case, the detection is performed by means of a single sensor. It is also conceivable to fuse the image data of multiple sensors, if necessary, the image data of multiple sensors with different detection sensors (e.g. video sensors, radar sensors, ultrasonic sensors, lidar sensors).

[0092] In this context, the ascertainment of free space (free space detection) and the semantic differentiation between foreground and background in the image data or image stream are particularly important.

[0093] By using the artificial neural network according to the invention, these features can be determined by processing the image data or image stream. Based on this information, the control system for the longitudinal and lateral control of the vehicle can be controlled accordingly, so that the vehicle responds appropriately to the detection of these features in the image data.

[0094] Another field of application of the invention can be seen as the accurate pre-marking (pre-labeling) of image data or image data streams for camera-based vehicle control systems.

[0095] In this case, the identifier (label) to be assigned represents the class of the object to be detected in the image data or the image stream.

[0096] The invention can also be used in all fields where accurate pixel-based object recognition (pixel-wise prediction) with the aid of artificial neural networks is required (e.g. automotive, robotics, health, surveillance, etc.). Here, for example, optical flow, depth from monochrome image data, digital, edge detection, key cards, object detection, etc. can be mentioned.

Claims

1. A method (400) for performing computationally efficient semantic segmentation of image data (111) of an imaging sensor by means of an artificial neural network, wherein: The artificial neural network comprises an encoder path (110) for classification of image data and a decoder path (120) for localization of image data, wherein the encoder path (110) transitions into the decoder path (120), wherein the transition is realized via a discriminant path (130), wherein the following steps are performed in the discriminant path (130): decomposing (410) an input tensor (113) according to a decomposition function (220, 320) into at least one first slice tensor (112, 230) and at least one second slice tensor (112, 250), wherein the input tensor (113) comes from the encoder path (110); decomposing (410) an input tensor (113) according to a first concatenation function (260) into at least one first slice tensor (112, 230) and at least one second slice tensor (112, 250) , concatenating (420) the at least one first slice tensor (112, 230) with the at least one second slice tensor (112, 250) to obtain at least one cascade tensor (132); concatenating (430) the at least one first slice tensor (112, 230) with a category tensor (134) according to a second concatenation function to obtain a decoder tensor (135), wherein the category tensor comes from the at least one cascade tensor (132), wherein the cascade tensor (132) is developed into the category tensor (134) by using a function of an artificial neural network; and outputting (440) the decoder tensor (135) to a decoder path (120) of the neural network.

2. The method (400) according to claim 1, wherein: In the step of decomposing (410), the decomposition function (220, 320) is designed in such a way that only a subset of the feature maps (114) of the input tensor (112) is selected to form the at least one first slice tensor (230).

3. The method (400) according to claim 1 or 2, wherein: The first splicing function (260) and / or the second splicing function are configured in such a way that the dimension of the input tensor (113) remains unchanged.

4. The method (400) according to claim 1 or 2, comprising a previous step of receiving, wherein: In the receiving step, the input tensor (113) and the decomposition function (220, 320) are received.

5. The method (400) according to claim 1 or 2, wherein: In the step of decomposing (410), a first function (321) of a neural network is applied to the at least one first slice tensor (230), and a second function (322) of a neural network is applied to the at least one second slice tensor (250).

6. The method (400) according to claim 5, wherein: The decomposition function (320, 321, 322) is designed such that it includes a quantity to be calculated of a feature map and a corresponding function (321, 322) of an artificial neural network for calculating the at least one first slice tensor (230) and the at least one second slice tensor (250).

7. The method (400) of claim 1, wherein: The artificial neural network is a convolutional neural network.

8. A device for semantic segmentation of image data (111) of an imaging sensor, the device comprising an artificial neural network, wherein: The artificial neural network has an encoder path for classification of image data and a decoder path for localization of image data, wherein the encoder path (110) transitions into the decoder path (120), wherein the transition is effected via a decision path (130), wherein the artificial neural network is configured to carry out the steps of the method (400) according to any one of claims 1 to 7 in the decision path (130).

9. The device according to claim 8, wherein: The artificial neural network is a convolutional neural network. 10 . A computer program product comprising a computer program, the computer program being configured to cause a computer to carry out all the steps of the method ( 400 ) according to claim 1 , when the computer program product is run by the computer. 11 . A machine-readable storage medium having a computer program stored thereon, the computer program comprising instructions for causing a computer to execute the method according to claim 1 when the computer program is executed by the computer.

Images