Method and apparatus for classifying pixels of an image

Abstract

The present disclosure relates to methods and devices for classifying pixels of an image. A method for classifying pixels of an image is provided. An image comprising a plurality of pixels is captured by a sensor device. A probability value for each pixel is estimated using a neural network, each probability value indicating a probability that the corresponding pixel is associated with one of a plurality of predetermined categories. One of the categories is assigned to each pixel of the image based on the corresponding probability value to create a predicted segmentation map. To train the neural network, a loss function is generated by associating the predicted segmentation map with ground truth labels. In addition, an edge detection algorithm is applied to at least one of the predicted segmentation map and the ground truth labels, wherein the edge detection algorithm predicts boundaries between objects. The step of generating the loss function is based on the result of the edge detection algorithm.

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus for performing semantic segmentation by classifying a plurality of pixels of an image. Background Art

[0002] For autonomous driving and driver assistance systems, it is important to have reliable perception of the vehicle's environment. An important foundation for environmental perception is semantic segmentation, i.e., classifying objects in the environment and determining, for example, the areas in an image of the environment that are occupied by specific object types or object classes.

[0003] For semantic segmentation, a number of predefined categories are typically defined for objects that may appear in an environment image. For example, the predefined categories may include categories for other vehicles, pedestrians, roads, buildings, and so on. Semantic segmentation can also be performed on the pixels of the environment image. That is, each pixel in the image is assigned to one of the predefined categories.

[0004] Currently, convolutional neural networks (CNNs) are primarily used for semantic segmentation tasks. CNNs are a special type of neural network that is configured to mimic the human visual processing system, from the eye to the visual cortex. A CNN consists of any number of consecutive layers that are trained together to perform a given task—in this case, classifying the objects represented by the pixels of an image, where each pixel is assigned to one of a predetermined set of categories.

[0005] In order to train any neural network, a reliable loss function is necessary to achieve the desired certainty in the network's output. The loss function involves some comparison between the output, or prediction, of the neural network and the desired output, also known as the ground truth labeling. During neural network training, regression is performed by assigning large loss values to unexpected output data and minimizing the overall loss.

[0006] In the case of semantic segmentation of a large number of pixels within an image, the total loss is usually estimated as the average of the loss values of all pixels. Therefore, individual pixels contribute only a small proportion to the total loss. As a result, areas of the image that are more challenging to classify (such as object boundaries) are weighted according to their size rather than their importance. Therefore, when running in a live vehicle system, the trained neural network may incorrectly classify pixels of such small objects and object boundaries.

[0007] Therefore, there is a need for a method and apparatus that can provide reliable semantic segmentation for small objects and object boundaries. Summary of the Invention

[0008] The present disclosure provides a computer-implemented method, a computer system and a non-transitory computer-readable medium. Implementation options are given in the dependent claims, the description and the drawings.

[0009] In one aspect, the present disclosure relates to a computer-implemented method for classifying pixels of an image. According to the method, an image captured by a sensor device is received, wherein the image comprises a plurality of pixels. A probability value for each pixel is estimated using a neural network implemented on a processing device, wherein each probability value indicates a probability that the corresponding pixel is associated with one of a plurality of predetermined categories. One of the plurality of predetermined categories is assigned to each pixel of the image based on the corresponding probability value of the each pixel to create a predicted segmentation map of the image. To train the neural network, a loss function is generated by associating the predicted segmentation map with ground truth labels. In addition, an edge detection algorithm is applied to at least one of the predicted segmentation map and the ground truth labels, wherein the edge detection algorithm predicts boundaries between objects in the predicted segmentation map and the ground truth labels. The step of generating the loss function is based on the results of the edge detection algorithm.

[0010] The sensor device may include a camera and / or a lidar system. Thus, the term "image" generally refers to the output of the sensor device and may include a two-dimensional or three-dimensional matrix containing a corresponding plurality of pixels.

[0011] A number of predefined categories refer to the types of objects expected to be seen in an image. Each category can be represented by an indicator that can be assigned to each pixel in the image. For automotive applications, these categories may include "other vehicles," "pedestrians," "road," "buildings," and so on.

[0012] The ground truth labeling refers to an image where the type of object is known for each pixel. In other words, the ground truth labeling represents the desired output of the neural network and the overall method. Therefore, during neural network training, the neural network's predictions are compared to the ground truth labeling, and the deviation between the predictions and the ground truth labeling is quantified via a loss function, thereby minimizing the loss function.

[0013] Edge detection algorithms, such as those based on the Sobel operator, are used to extract boundaries between objects belonging to different categories. Since the results of the edge detection algorithm are incorporated into the definition or generation of the loss function, the contribution of object boundaries to the loss function is greatly enhanced. Consequently, the contribution of small objects is also enhanced because the fraction of their pixels belonging to boundaries relative to their total number of pixels is greater than that of large objects.

[0014] In summary, since the results of the edge detection algorithm are taken into account when generating the loss function, the neural network is trained with a focus on small objects and object boundaries within the image. Consequently, the method according to the present disclosure improves the overall semantic segmentation or classification of pixels within the image. For automotive applications, this method helps prevent small objects and object boundaries from being overlooked or incorrectly evaluated. Therefore, this method can improve safety when semantic segmentation is used within the framework of autonomous driving or driver assistance systems.

[0015] The method may include one or more of the following features:

[0016] The edge detection algorithm may be applied only to the predicted segmentation map. Alternatively, the edge detection algorithm may be applied only to the ground truth labels, or the edge detection algorithm may be applied to both the predicted segmentation map and the ground truth labels. The results of applying the edge detection algorithm to the predicted segmentation map and the results of applying the edge detection algorithm to the ground truth labels are combined by selecting the maximum of the respective results for each pixel.

[0017] The result of the edge detection algorithm can be applied to the result of associating the predicted segmentation map with the ground truth labels to generate a loss function. In addition, the result of the edge detection algorithm can be a mask of pixels covering the predicted boundary. The mask can include a corresponding element for each pixel, and a loss matrix including an element for each pixel can be calculated by associating the predicted segmentation map with the ground truth labels, and when generating the loss function, each element of the mask can be multiplied by the corresponding element of the loss matrix for each pixel. Elements of the mask for pixels outside the predicted boundary can be assigned a value close to zero.

[0018] The edge detection algorithm may include a Sobel operator including two predetermined convolution kernels and two additional kernels generated by rotating the two predetermined convolution kernels. The edge detection algorithm may include a resulting convolution kernel for increasing the width of predicted boundaries between objects. The additional convolution kernels may be bivariate Gaussian kernels.

[0019] Depending on the embodiment, the edge detection algorithm may be applied only to the predicted segmentation map. Alternatively, the edge detection algorithm may be applied only to the ground truth labels. By taking into account either the predicted segmentation map or the ground truth labels when applying the edge detection algorithm, the amount of computation required to generate the loss function based on the results of the edge detection algorithm is reduced. If the predicted segmentation map is taken into account, misclassified "hallucination" objects may be suppressed during the training of the neural network since their weights within the loss function may be increased and they may not have corresponding objects within the ground truth labels. Therefore, minimizing the total loss during the training of the neural network may result in the suppression of hallucination objects. On the other hand, if the edge detection algorithm only takes into account the ground truth labels, it may be ensured that the most important objects are considered in the loss function, i.e., independent of their predicted classification via the neural network.

[0020] According to a further embodiment, the edge detection algorithm can be applied only to both the predicted segmentation map and the ground truth labels. For this embodiment, the advantages of both previously described embodiments are valid, namely, suppressing "hallucinated" objects and correctly accounting for the most important objects. The same edge detection algorithm can be applied to both the predicted segmentation map and the ground truth labels to generate the final result of the edge detection algorithm. By considering the ground truth labels in addition to the predicted segmentation map when performing edge detection, even objects that were missed by the neural network base model and therefore missed in the predicted segmentation map can be emphasized.

[0021] The results of applying the edge detection algorithm to the predicted segmentation map and the results of applying the edge detection algorithm to the ground truth labels can be merged by selecting the maximum of the respective results for each pixel. In other words, the final result of edge detection can be generated by applying a pixel-wise maximization operator to the individual results of applying the edge detection algorithm to the prediction and ground truth. Compared to arithmetic merging, the final result of edge detection can be improved.

[0022] The result of the edge detection algorithm can be applied to the result of associating the predicted segmentation map with the ground truth label to generate the loss function. In detail, a "preliminary loss" can be estimated by the relationship between the predicted segmentation map and the ground truth label, for example, by applying a cross entropy loss function. Thereafter, the result of the edge detection algorithm can be applied to the preliminary loss to estimate the final loss function so as to increase the contribution of small objects and object boundaries. In other words, the contribution of the boundaries between objects can be enhanced because all pixels that do not belong to the boundary are suppressed.

[0023] The result of the edge detection algorithm can be a mask of pixels covering the predicted boundary. The mask can include corresponding elements for each pixel, and a loss matrix including elements for each pixel can be calculated by associating the predicted segmentation map with the ground truth label, and when generating the loss function, each element of the mask can be multiplied by the corresponding element of the loss matrix for each pixel. On the other hand, the elements of the mask for pixels outside the detection boundary can be assigned values close to zero. In detail, the mask can include values between 0 and 1. If a pixel belongs to the boundary between objects, the mask element of the pixel can be assigned a value of 1 or close to 1, while pixels that do not belong to the boundary between objects can be assigned a small value, such as 0.1. Such a small value close to zero ensures that all pixels are taken into account when generating the loss function. However, the contribution of pixels that do not belong to the boundary between objects can be strongly suppressed.

[0024] According to a further embodiment, the edge detection algorithm may include a Sobel operator comprising two predetermined convolution kernels and two additional kernels generated by rotating the two predetermined convolution kernels. The Sobel operator itself is a well-known means for edge detection and typically has two predefined and constant convolution kernels. However, it has been shown that the standard Sobel kernel does not provide good results for detecting diagonal edges or boundaries between objects. Therefore, two additional convolution kernels may be included in the edge detection algorithm, which are rotated versions of the original Sobel kernel. For example, the original Sobel kernel may be rotated by + / - 45°. In this way, the results of the edge detection algorithm may be improved compared to applying only the original Sobel operator.

[0025] In addition, the edge detection algorithm may include a resulting convolution kernel for increasing the width of the predicted boundaries between objects. The additional convolution kernel may be a bivariate Gaussian kernel. The additional kernel may also be referred to as a padding kernel, as it may add, for example, three pixels of padding before the original line representing the boundaries between objects, and three pixels of padding after the original line representing the boundaries between objects. By increasing the width of the boundaries between objects, successful learning of neural networks may be supported, as it has been shown that if an edge detection algorithm can be performed without padding, the result of the edge detection algorithm may provide boundaries between objects that are too narrow. A bivariate Gaussian kernel may be a convenient representation of such a padding kernel. In detail, the bivariate Gaussian kernel used by the method may be normalized so that the average value of all its elements may be one.

[0026] In another aspect, the present disclosure relates to a system for classifying pixels of an image. The system includes a sensor device configured to capture an image comprising a plurality of pixels, and a processing device. The processing device is configured to receive an image from the sensor device, implement a neural network for estimating a probability value for each pixel, wherein each probability value indicates a probability that the corresponding pixel is associated with one of a plurality of predetermined categories. The processing device is further configured to assign one of the plurality of predetermined categories to each pixel of the image based on the respective probability value of each pixel to create a predicted segmentation map for the image, generate a loss function for training the neural network by associating the predicted segmentation map with ground truth labels, and apply an edge detection algorithm to at least one of the predicted segmentation map and the ground truth labels, wherein the edge detection algorithm predicts a boundary between objects in the predicted segmentation map and the ground truth labels. The step of generating the loss function is based on a result of the edge detection algorithm.

[0027] As used herein, the terms processing device, processing unit, and module may refer to, be part of, or include an application specific integrated circuit (ASIC), an electronic circuit, a combinational logic circuit, a field programmable gate array (FPGA), a processor (shared, dedicated, or grouped) that executes code, other suitable components that provide the aforementioned functionality, or some or all of the above (such as in a system-on-chip). The term module may include memory (shared, dedicated, or grouped) that stores code executed by a processor.

[0028] In summary, the system according to the present disclosure includes a sensor device (e.g., a camera and / or a lidar system) and a processing device configured to perform the steps for the corresponding method as described above. Therefore, the benefits, advantages, and disclosures described above for the method are also valid for the system according to the present disclosure.

[0029] In another aspect, the present disclosure relates to a computer system configured to perform several or all of the steps of the computer-implemented method described herein.

[0030] The computer system may include a processing unit, at least one storage unit, and at least one non-volatile data memory. The non-volatile data memory and / or the storage unit may include a computer program for instructing a computer to perform several or all steps or aspects of the computer-implemented method described herein.

[0031] In another aspect, the present disclosure relates to a non-transitory computer-readable medium containing instructions for performing some or all of the steps or aspects of the computer-implemented methods described herein. The computer-readable medium can be configured as: an optical medium such as a compact disc (CD) or a digital versatile disc (DVD); a magnetic medium such as a hard disk drive (HDD); a solid-state drive (SSD); a read-only memory (ROM); a flash memory; etc. In addition, the computer-readable medium can be configured as a data storage unit accessible via a data connection such as an Internet connection. The computer-readable medium can be, for example, an online data repository or cloud storage.

[0032] The present disclosure also relates to a computer program for instructing a computer to execute several or all steps or aspects of the computer-implemented method described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Example implementations and functionality of the present disclosure are described herein with reference to the following figures, which are schematically shown:

[0034] Figure 1 shows a high-level diagram of a system according to the present disclosure,

[0035] Figure 2 The results of semantic segmentation based on background technology are shown, where the left side is the ground truth and the right side is the network prediction.

[0036] Figure 3 shows a flow chart of a method according to the present disclosure,

[0037] Figures 4a to 4c shows the edge detection results and their application to ground truth labeling,

[0038] Figure 5 and Figure 6 Comparison is made between the corresponding semantic segmentation based on background technology and the method according to the present disclosure, wherein, Figure 5 and Figure 6 The upper part is the method according to the background technology, the lower part is the method according to the present disclosure, and

[0039] Figures 7a to 7c Visualizations of the validation of semantic segmentation are shown, including bounding intersection-over-union and its complement.

[0040] Reference Number List

[0041] 10 System

[0042] 11 Cameras and sensor devices

[0043] 13 images

[0044] 15 Processing Equipment

[0045] 17 Semantic Segmentation of Images

[0046] 21 Small Objects

[0047] 23 Boundaries between objects

[0048] 25 Network Prediction of Small Objects (Background Art)

[0049] 27 Network Prediction of Object Boundaries (Background Art)

[0050] 31 Methods

[0051] 33 Convolutional Neural Networks

[0052] 35 Predicted segmentation map

[0053] 37 Loss Function

[0054] 39 ground truth labels

[0055] 41 Edge Detection Algorithm

[0056] 43 Prediction Mask

[0057] 44 ground truth masks

[0058] 45 Maximization Operator

[0059] 47 Final or merge mask

[0060] 50 elements

[0061] 51 Surrounding area

[0062] 53 strokes

[0063] 55 Traffic Light

[0064] 57 pedestrians DETAILED DESCRIPTION

[0065] Figure 1 A high-level diagram of a system 10 for classifying pixels of an image, i.e., for performing semantic segmentation of an image, is schematically shown. The system includes a camera 11 as a sensor device and a processing device 15. The camera 11 is mounted on a vehicle and is capable of capturing an image 13 of the vehicle's environment (e.g., in front of the vehicle). Alternatively, a lidar system can be used to capture the image 13, in which case the image 13 can be a three-dimensional matrix of pixels.

[0066] The image 13 captured by the camera 11 is used as input to a processing device 15 which is configured to generate a convolutional neural network 33 (see Figure 3), for performing semantic segmentation of the image 13. For semantic segmentation, the processing device 15 uses the following Figure 3 The output of the processing device 15 , ie the semantic segmentation 17 of the image 13 , is provided as input to other devices of the vehicle, which are for example part of a driver assistance system.

[0067] Figure 2 1 shows the result of semantic segmentation according to the background art. That is, the area within the image 13 captured by the camera 11 is assigned to a predetermined type of object or object category, for example, assigned to one of the object categories "other passenger car", "truck", "road", "sidewalk", "pedestrian", etc. The area occupied by a specific object type is Figure 2 are represented by a unique color.

[0068] exist Figure 2 On the left side of , we show the ground truth labels for two different scenes. For these scenes, we know which region belongs to which object type. Figure 2 The ground truth labels shown on the left are used to train a convolutional neural network that provides semantic segmentation for image 13. Figure 2 The network predictions shown on the right. In detail, semantic segmentation or classification of objects is performed based on the pixels of the corresponding image 13. For example, each image 13 includes r = 2048 rows and c = 1024 columns, i.e., almost 2.1 million pixels. For each pixel, the convolutional neural network provides a prediction about the object class, i.e., it assigns an indicator of a specific object class to each pixel. Thus, as Figure 2 Each region shown on the left and having a specific color contains multiple pixels that are classified by the convolutional neural network as a specific type of object.

[0069] like Figure 2 As shown on the left side of FIG, the corresponding ground truth labeling of image 13 includes many small objects 21 and boundaries 23 between objects. Figure 2 As shown on the right side of , the corresponding network prediction according to the background art cannot correctly classify the small object 21 and the boundary 23. In detail, in the area represented by 25, the small object 21 is missing or almost missing. In addition, as Figure 2 The clear boundaries 23 between objects shown on the left are also not correctly identified in the network predictions, as shown in 27. The clear boundaries 23 between objects appear to be destroyed in the output of the convolutional neural network.

[0070] The cause of errors or unsatisfactory classifications predicted by networks according to the background art is related to the loss function commonly used to train convolutional neural networks. A robust loss function is generally necessary for any neural network learning or training process. The loss function typically includes some comparison of the predicted output of the neural network with the expected output (i.e., the ground truth label). Typically, regression is performed by assigning large loss values to unexpected values within the network prediction and thereafter minimizing the total loss during the training of the neural network.

[0071] To evaluate the output or predictions of a neural network, the so-called cross entropy function is usually used as the loss function and is defined as

[0072]

[0073] Among them, y i represents the output or prediction of the neural network, and represents the desired or ground truth label. The total loss is then defined as the average of all pixels within the image 13. Since for an image with 2048 rows and 1024 columns and thus almost 2.1 million pixels, the contribution of a single pixel to the loss function or the total loss is approximated by 1 / (2.1*10 6 ) weighting. As a result, small objects comprising a relatively small number of pixels and boundaries between objects are not correctly represented in loss functions according to the background art, in which all pixels have the same weight for their contribution to the loss function. Therefore, it is desirable to have a loss function for training convolutional neural networks in which the representation of small objects and boundaries between objects is improved.

[0074] Figure 3 1 shows a flow chart of a method 31 according to the present disclosure. The method 31 is provided for Figure 1 ) is used to classify the pixels of an image 13 captured by a processor 15. That is, an image 13 comprising a plurality of pixels is provided as input to the method, i.e. as a pixel classification system by a processor 15 (see Figure 1 ) generates the input of the convolutional neural network 33. The convolutional neural network 33 (see Figure 3 ) includes multiple layers based on the underlying model.

[0075] The neural network 33 estimates a probability value for each pixel. Each probability value indicates the probability of the corresponding pixel being associated with one of a plurality of predetermined categories. Based on the corresponding probability value of each pixel, each pixel of the image 13 is assigned to one of the plurality of categories, i.e. by selecting the category with the highest probability value. The predetermined categories include predetermined types of objects visible to the camera 11, i.e. other vehicles, roads, sidewalks, pedestrians, etc. As output, the convolutional neural network 33 creates a predicted segmentation map 35, which includes the predicted category of each pixel. In the example Figure 3 The predicted segmentation map 35 shown and Figure 5 and Figure 6 Similar predicted segmentation maps are shown, with different types of objects represented by different colors.

[0076] In order to train the neural network 33, the predicted segmentation map 35 is correlated with the ground truth labels 39 (i.e., the desired output of the convolutional neural network 33). For the ground truth labels 39, the correct assignment to the corresponding class or object type is known for each pixel of the image 13. The predicted segmentation map 35 is correlated with the ground truth labels 39 via a loss function 37, which is based on the above Figure 2 During the training of the convolutional neural network 33, the total loss provided by the loss function 37 will be minimized.

[0077] As mentioned above, a disadvantage of the commonly used cross-entropy based loss function is that all pixels contribute equally, which leads to underestimation of small objects and object boundaries when estimating the loss function and therefore leads to incorrect or missing classification of small objects and object boundaries.

[0078] To overcome this drawback, the method comprises providing a modified version 37 of the loss function, wherein small objects and object boundaries are set with larger weights in order to increase their contribution to the overall loss.

[0079] In order to increase the contribution of small objects and object boundaries to the loss function 37, an edge detection algorithm 41 is applied to the predicted segmentation map 35 and the ground truth labels 39. When applied to the predicted segmentation map 35, the output of the edge detection algorithm 41 is a predicted mask 43, and when applied to the ground truth labels 39, the output of the edge detection algorithm 41 is a ground truth map 44. Within the masks 43, 44, all boundaries between objects are Figure 3 is highlighted in because the mask assigns much higher weights to pixels that are detected as boundaries or edges than to pixels that do not belong to boundaries.

[0080] The edge detection algorithm 41 is based on the so-called Sobel operator, which is able to extract the boundaries between the predicted classes. The standard Sobel operator consists of two predefined and constant convolution kernels:

[0081]

[0082] By using these kernels, color gradients in the image 13 can be detected.

[0083] However, it has been shown that the standard Sobel operator based on a predefined constant convolution kernel is insufficient to successfully weight the loss function 37. In detail, the Sobel kernel defined in (2) shows little performance when trying to detect diagonal edges. Therefore, two additional convolution kernels are added, which are rotations of the original Sobel kernel by + / - 45°.

[0084] In addition, pixels that do not belong to the detected edge are suppressed by setting their values in the mask to near zero. As a result, almost no loss value is assigned to the main part of the predicted segmentation map 35 to create the masks 43, 44. However, since a prediction is still to be provided for all pixels of the image 13, all pixels in the corresponding mask 43 or 44 have at least a small value, such as 0.1, in order to account for the loss value of all pixels.

[0085] Furthermore, it turns out that the resulting edges provided by the edge detection algorithm 41 are so far insufficient to support successful learning or training of the convolutional network 33. Therefore, a corresponding convolution kernel is applied, which is a bivariate Gaussian kernel with a predefined size. In detail, a 3×3 padding kernel is used given by

[0086]

[0087] When performing convolution within the edge detection algorithm 41, the additional kernel adds three pixels of padding before the original line or boundary between the corresponding objects and three pixels of padding after it. Thus, the width of the boundary represented in the masks 43 and 44 is increased. In practice, an additional padding kernel of approximately 30×30 pixels is used. In contrast to a classic Gaussian kernel, which is normalized so that the sum of all its elements equals 1, the method according to the present disclosure normalizes the bivariate Gaussian kernel so that the mean of its elements is unity.

[0088] After generating the respective masks 43, 44 based on a modified or extended Sobel operator including an additional kernel, the two masks 43, 44 are merged by using a pixel-level maximization operator at 45. That is, for each pixel, the larger value of the respective masks 43, 44 is selected so as to generate a final or merged mask 47. For each pixel, the final mask 47 is included in the generation of the loss function 37. That is, for each pixel, the contribution to the loss is calculated according to the cross entropy function as defined in (1), so that a "preliminary loss" or loss matrix is generated, in which each element includes the contribution of the corresponding pixel. Thereafter, the final mask 47 is applied to the preliminary loss or loss matrix. In detail, for each pixel, the preliminary loss is multiplied by the corresponding element of the final mask 47. Therefore, the contribution of object boundaries and small objects to the loss function 37 is greater than that of areas that do not belong to the boundaries between detected objects.

[0089] according to Figure 3 In the illustrated embodiment, the method includes: generating a prediction mask 43 that the edge detection algorithm 41 applies to the predicted segmentation map 35; generating a ground-truth mask 44 that the edge detection algorithm 41 applies to the ground-truth labels 39; and generating a final mask 47 that combines masks 43 and 44 and applies them when generating the loss function 37. However, in another embodiment, the prediction mask 43 is only applied to the ground-truth labels 39. That is, the final mask 47 is the same as the prediction mask 43 of the present embodiment because the merging of the two masks 43 and 44 is not performed. In another alternative embodiment, only the ground-truth mask 44 is used to generate the final mask 47, that is, the prediction mask 43 is omitted and the merging of the two masks 43 and 44 is not performed. In other words, in this alternative embodiment, the ground-truth mask 44 is directly applied to the loss function 37, that is, to the preliminary loss.

[0090] Figure 4a An enlarged example of the prediction mask 43 is shown. Figure 4b Shows the Figure 4a The prediction mask 43 is applied to the result of the ground truth labeling 39. By using the prediction mask 43, misclassified or "hallucinated" objects can be suppressed. Figure 4b Such hallucinated objects can be identified in the lower left corner of . Since the weight or contribution of this hallucinated object is increased due to the prediction mask 43 when calculating the total loss, these objects will be suppressed by the trained neural network after the loss function is minimized during training. In addition, Figure 4b Also represented are the masked ground truth labels for embodiments of the method for which the prediction mask 43 is considered only for generating the loss function 37 .

[0091] In contrast, Figure 4cThe masked ground truth labels are shown where the merged final mask 47 has been applied to the original ground truth labels 39. That is, Figure 4c Indicates Figure 3 Mask ground truth labeling of the embodiment shown, where a final mask 47 is generated by merging two masks 43, 44. By additionally taking into account the ground truth mask 44, it is ensured that the most important parts of the image are properly considered. Figure 4c As can be seen in the upper right corner of , the contribution of the boundary belonging to the pink element 50 is strongly enhanced because the ground truth mask 44 is additionally considered. The weight of the pink element 50 is almost missing in the predicted mask 43, as shown in Figure 4a The upper right corner can be recognized.

[0092] Notice, Figure 4a The prediction mask 43 and the merged final mask 47 shown in Figure 3 ) are applied to or multiplied by the ground truth labels 39 for illustrative purposes only, i.e., to demonstrate the effect of the masks 43, 47 on the objects within the image 13. However, in order to generate the loss function 37 and estimate the total loss, the masks are applied to (or multiplied by) the “preliminary loss,” i.e., the loss matrix calculated based on the cross entropy function, but are not directly applied to the ground truth labels 39.

[0093] Figure 5 and Figure 6 The method according to the background technology ( Figure 5 and Figure 6 ) performed by and by the method according to the present disclosure ( Figure 5 and Figure 6 As can be seen in the surrounding area 51, the boundaries between objects are clearer in the segmentation result according to the method of the present disclosure. In addition, according to the method of the background art, some small objects are almost completely classified in a wrong way. For example, in the segmentation result according to the background art, it is impossible to identify Figure 6 The pole on the right side of the image is clearly visible when segmentation is performed by the method according to the present disclosure. The same is true for the traffic light as shown at 55. In addition, the recognition of the pedestrian 57 is improved by the method according to the present disclosure, as shown in FIG. Figure 6 as shown on the right side of .

[0094] like Figure 5 and Figure 6 The improved segmentation result shown in the corresponding lower part of is due to the fact that when generating the loss function 37, the edge detection algorithm 41 (see Figure 3) is incorporated into the method 31. Since the results of the edge detection algorithm 41 are applied to the ground truth labels 39 before associating the predicted segmentation map 35 with the ground truth labels 39 via the loss function 37, the contribution of object boundaries and small objects to the loss function 37 is enhanced during the training process of the neural network 33. Since small objects and object boundaries are Figure 5 and Figure 6 The semantic segmentation results shown in (corresponding lower part) can be better identified, so if the method according to the present disclosure is applied, for example, in combination with a driver assistance system and autonomous driving, the safety of the vehicle can be improved.

[0095] For detailed verification of segmentation results, the so-called border intersection over union (border IoU (border intersection over union)) has been estimated. For example, in "RangeNet++:Fast and accurate LiDAR semantic segmentation" by Milioto, A et al., 2019 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), pp.4213-4220. IEEE, 2019, border IoU is defined and applied. The intersection over union (IoU) is commonly used to describe the similarity of sets, vectors, and objects. In semantic segmentation, IoU is often used as a metric for evaluating labeling performance, which associates the true positives predicted by the network with the sum of the true positive, false positive, and false negative predictions of the neural network. For border IoU, this evaluation key is only applied to the boundaries between objects.

[0096] This is Figures 7a to 7c Visible in. Figure 7a The camera 11 (see FIG. Figure 1 )The original image captured 13. Figure 7b The boundary IoU is shown, where the evaluated pixels of the image 13 are masked with their respective colors assigned to the respective object classes. Figure 7c In the figure, the so-called intra-IoU is shown, which is evaluated as Figure 7b For almost all object categories, all metrics (i.e., “normal” IoU, interior IoU, and especially boundary IoU) provide better values for semantic segmentation based on the method according to the present disclosure compared to semantic segmentation based on the method according to the background art.

Claims

1. A computer-implemented method (31) for classifying pixels of an image (13), the computer-implemented method (31) comprising: Receiving an image (13) captured by a sensor device (11), the image (13) comprising a plurality of pixels; estimating a probability value for each pixel using a neural network (33) implemented on the processing device (12), wherein each probability value indicates a probability that the corresponding pixel is associated with one of a plurality of predetermined classes; assigning one of the plurality of predetermined classes to each pixel of the image (13) based on the corresponding probability value of each pixel to create a predicted segmentation map (35) of the image (13); generating a loss function (37) for training the neural network (33) by correlating the predicted segmentation map (35) with ground truth labels (39); and applying an edge detection algorithm (41) to at least one of the predicted segmentation map (35) and the ground truth labels (39), the edge detection algorithm (41) predicting boundaries between objects in the predicted segmentation map (35) and objects in the ground truth labels (39), wherein the step of generating the loss function (37) is based on the result of the edge detection algorithm (41), wherein the result of the edge detection algorithm (41) is a mask (43, 44, 47) of pixels covering the predicted boundary, wherein the mask (43, 44, 47) comprises a corresponding element for each pixel, a loss matrix comprising a matrix element for each pixel is calculated by associating the predicted segmentation map (35) with the ground truth label (39), and when generating the loss function, each element of the mask (43, 44, 47) is multiplied by the corresponding matrix element of the loss matrix for each pixel, wherein the edge detection algorithm (41) is applied to the predicted segmentation map (35) and the ground truth labels (39), and wherein the result of applying the edge detection algorithm (41) to the predicted segmentation map (35) and the result of applying the edge detection algorithm (41) to the ground truth labeling (39) are merged by selecting the maximum value of the respective results for each pixel.

2. The computer-implemented method (31) of claim 1, wherein: The results of the edge detection algorithm (41) are applied to the results of associating the predicted segmentation map (35) with the ground truth labels (39) to generate the loss function (37).

3. The computer-implemented method (31) of claim 1, wherein: Elements of the mask (43, 44, 47) for pixels outside the prediction boundaries are assigned values close to zero.

4. The computer-implemented method (31) of claim 1, wherein: The edge detection algorithm (41) includes a Sobel operator including two predetermined convolution kernels and two additional kernels generated by rotating the two predetermined convolution kernels.

5. The computer-implemented method (31) of claim 4, wherein: The edge detection algorithm (41) includes a resulting convolution kernel for increasing the width of predicted boundaries between objects.

6. The computer-implemented method (31) of claim 5, wherein: The additional kernel is a bivariate Gaussian kernel.

7. A system (10) for classifying pixels of an image (13), the system (10) comprising: a sensor device (11) configured to capture an image (13) comprising a plurality of pixels; as well as A processing device (12), the processing device being configured to: receiving the image (13) from the sensor device (11); implementing a neural network (33) for estimating a probability value for each pixel, wherein each probability value indicates a probability that the corresponding pixel is associated with one of a plurality of predetermined categories; assigning one of the plurality of predetermined classes to each pixel of the image (13) based on the corresponding probability value of each pixel to create a predicted segmentation map (35) of the image (13); generating a loss function (37) for training the neural network (33) by correlating the predicted segmentation map (35) with ground truth labels (39); and applying an edge detection algorithm (41) to at least one of the predicted segmentation map (35) and the ground truth labels (39), the edge detection algorithm (41) predicting boundaries between objects in the predicted segmentation map (35) and objects in the ground truth labels (39), wherein the step of generating the loss function (37) is based on the result of the edge detection algorithm (41), wherein the result of the edge detection algorithm (41) is a mask (43, 44, 47) of pixels covering the predicted boundary, wherein the mask (43, 44, 47) comprises a corresponding element for each pixel, a loss matrix comprising a matrix element for each pixel is calculated by associating the predicted segmentation map (35) with the ground truth label (39), and when generating the loss function, each element of the mask (43, 44, 47) is multiplied by the corresponding matrix element of the loss matrix for each pixel, wherein the edge detection algorithm (41) is applied to the predicted segmentation map (35) and the ground truth labels (39), and wherein the result of applying the edge detection algorithm (41) to the predicted segmentation map (35) and the result of applying the edge detection algorithm (41) to the ground truth labeling (39) are merged by selecting the maximum value of the respective results for each pixel.

8. A computer system configured to perform the computer-implemented method according to any one of claims 1 to 6.

9. A non-transitory computer-readable medium comprising instructions for executing the computer-implemented method according to any one of claims 1 to 6.

Images