Image segmentation

Abstract

In one aspect, hierarchical image segmentation is applied to an image formed of a plurality of pixels by classifying the pixels according to a hierarchical classification scheme, where at least some of the pixels are classified by a parent classifier with respect to a set of parent classes, each of the parent classes being associated with a subset of child classes, each of the parent classes corresponds to a class of visible structures and each of the subclasses associated with each of the parent classes corresponds to a different type of visible structures within the class, and each of the pixels is also classified by at least one child-level classifier with respect to one of the subsets of the child classes, where each of the parent classes corresponds to a class of visible structures and each of the subsets of child classes associated with each of the parent classes corresponds to a different type of visible structures within the class.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] The present invention is a divisional application with an application date of June 6, 2019, application number 201980031032.9, and invention name “Image Segmentation”. The parent application is a PCT national phase application, with an international application date of June 6, 2019, and application number “PCT / EP2019 / 064826”, claiming the rights and interests of UK patent application GB.1809345.0 filed on June 7, 2018, all contents and substance of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to image segmentation. Background Art

[0004] An autonomous vehicle, also called a self-driving vehicle, is a vehicle that has a sensor system for monitoring its external environment and a control system that is able to use these sensors to autonomously make and implement driving decisions. This includes, among other things, the ability to automatically adjust the vehicle's speed and direction of travel based on input from the sensor system. A fully autonomous or "driverless" vehicle has sufficient decision-making capabilities to operate without any input from a human driver. However, the term "autonomous vehicle" as used herein also applies to semi-autonomous vehicles, which have more limited autonomous decision-making capabilities and therefore still require some degree of supervision from a human driver.

[0005] Autonomous decision making is a key capability of an automated vehicle. This is true regardless of its level of autonomy, but as the level of autonomy increases, so does the complexity of the decisions it needs to be able to make safely, to the point where a fully automated vehicle requires an exceptionally robust decision engine that can be relied upon to safely handle any driving situation it encounters, no matter how unexpected.

[0006] One factor that limits the decision-making capabilities of autonomous vehicles is their ability to interpret sensor inputs, such as captured images of their surroundings, even when those sensor inputs are imperfect. Summary of the invention

[0007] The present disclosure provides a hierarchical image segmentation (pixel-level classification) scheme that can be implemented as a suitably trained convolutional neural network (CNN) (or any other suitable image segmentation system). The scheme allows the network to provide some information about a pixel even if it is not sure what the full classification of the pixel is. For example, a pixel may be part of a centerline road marking, but the network may not be sure about this. Despite this, the network can still determine that the pixel is part of a road (as opposed to a tree). The hierarchical structure allows the network to determine that a pixel is a road even if it is not sure about the fine-grained distinction, and the hierarchical structure of the segmentation scheme provides a method to express this result.

[0008] In the context of autonomous vehicle decision making, one aspect of robustness is the provision of one or more default actions to be taken in a “worst case scenario” where the autonomous vehicle determines that the overall quality of its sensor inputs has degraded to a level where it cannot continue safely. For example, the default action might be to come to a complete stop in the shortest time that can be safely achieved, or (in the case of a semi-autonomous vehicle) to trigger an alert to the human driver and revert to manual control. However, such extreme actions should not be taken when they can be safely avoided, and an important capability of a useful autonomous vehicle is the ability to safely implement autonomous decisions based on imperfect sensor inputs (e.g., captured images) without resorting to unnecessary default actions.

[0009] In this case, when a hierarchical classification scheme is used as the basis for decision making, this can greatly reduce the number of scenarios where default actions are required compared to traditional image classification techniques, because certain types of decisions can be safely taken based on higher-level classifications of certain pixels, even if fine-grained (lower-level) classifications of those pixels cannot be obtained with sufficient certainty.

[0010] However, it should be noted that the present invention can be applied to any image segmentation problem where it is desirable to learn a semantic hierarchy of visual structure at the pixel level in contexts other than autonomous driving.

[0011] A first aspect of the present invention provides a computer-implemented method for classifying pixels of an image, the method comprising: receiving an image formed by multiple pixels; and classifying the pixels according to a hierarchical classification scheme, wherein at least some of the pixels are classified by a parent classifier with respect to a set of parent classes, each parent class in the set of parent classes is associated with a subset of child classes, and each of the pixels is also classified by at least one child classifier with respect to one of the subsets of child classes, wherein each parent class corresponds to a category of visible structures, and each subset of child classes associated with each parent class corresponds to a different type of visible structure in the category.

[0012] In general, it is easier to classify at a higher level because even for lower quality images or image regions (e.g., for distant low-resolution structures, or in the presence of noise, distortion, blur, etc.), it may not be possible to classify with sufficient certainty at least one of the associated subsets of the subclasses, but pixels may be classified with sufficient certainty with respect to the set of parent classes. However, for higher quality images / image regions, it may be possible to classify with sufficient certainty with respect to a subset of the subclasses, and in this case, the classification may provide additional granularity that complements the classification with respect to the set of parent classes.

[0013] In the described embodiment of the invention, this hierarchical classification is used as the basis for decision making in an autonomous vehicle to exploit the fact that certain types of decisions can be safely made based on higher level classifications of certain pixels even when low level classifications of those pixels cannot be made with sufficient certainty. That is, certain decisions can be safely made based on a particular set of pixels corresponding to a particular class of road structure (i.e., based on a successful high level classification) without having to determine the specific type of road structure in that class at that point in time (i.e., despite the failure of the lower level classification). If the road type can indeed be identified, this additional granularity can be fed into the decision making process as needed or desired. For example, in the case of a road structure that is initially far away, it can be classified as a "road" rather than a specific type of road ("lane", "road marking", etc.), and this level of classification is sufficient for making certain types of decisions. As the vehicle drives toward the road structure, there will be a point for those pixels where the road type can be identified with sufficient certainty, and this information can in turn be used by the decision engine when useful or needed.

[0014] That is, in an embodiment of the present invention, the method may include executing a decision process (e.g., an automatic vehicle control process) based on pixel classification by a controller. The controller may detect a failure of a child classifier to classify at least one pixel during the execution of the decision process, and make a decision based on the classification of the pixel by a parent classifier.

[0015] Another benefit of the hierarchical classification scheme is that it allows for training a segmentation convolutional neural network (or other trainable image segmentation system) from partially annotated (labeled) data. For example, some training images may only show the distinction between road and non-road, while other training images may only have labels for lane markings and road edges, as well as labels for road / non-road areas only. This is because the hierarchy is constructed in such a way that it is possible to provide a learning mechanism that can obtain error signals at various levels of the hierarchy while taking into account unannotated regions at any level of its hierarchy. With respect to the annotated training images, training is performed to minimize the error signal generated over the entire hierarchy as a whole.

[0016] In an embodiment of the invention, unannotated regions may be accounted for using what are referred to herein as "masks", where any region of an image that is not annotated at a given level of the hierarchy with respect to a given set of categories does not contribute to the corresponding error signal defined at that level of the hierarchy, such that a classifier cannot be "penalized" during training for the way it chooses to classify pixels at that level of the hierarchy with respect to that category. This is illustrated by the example in the following table:

[0017]

[0018]

[0019] In an embodiment, the method may include performing, by the controller, a decision process based on pixel classification.

[0020] The method may include, during the execution of the decision, detecting a failure of the child classifier to classify at least one of the pixels, and making a decision based on the classification of the pixel by the parent classifier.

[0021] The decision making process may be an automated vehicle control process.

[0022] The child classifier may determine at least one child classification value that classifies the pixel with respect to a subclass of a subset of the subclass, and the parent classifier may determine at least one parent classification value that classifies the pixel with respect to a parent class associated with the subset of the child classes.

[0023] The child classifier may determine at least one conditional classification value associated with the child class conditioned on the parent class for the pixel, and determine the child classification value based on the conditional classification value and the parent classification value.

[0024] The child classifier can determine the child classification value by multiplying the parent classification value with the conditional classification value.

[0025] The parent classification value and the child classification value may be values ​​indicating the probability that the pixel belongs to the parent class and the child class, respectively.

[0026] The conditional classification value may be a value indicating the conditional probability that the pixel belongs to the child class under the condition that the pixel belongs to the parent class.

[0027] The method may include: generating, by a feature extractor, a plurality of pixel feature vectors based on the pixel and a set of pixels related to the pixel, each pixel feature vector representing a pixel, wherein the pixel is classified based on the pixel feature vector of the pixel.

[0028] The set of related pixels may be formed from pixels in a portion of the image surrounding the pixel.

[0029] The feature extractor may be formed by one or more layers of a neural network.

[0030] The feature extractor can be formed by multiple layers in a convolutional neural network.

[0031] Each feature vector can be processed according to the learned feature vector modification parameters to generate: a modified feature vector of the parent class set and a modified feature vector of each subset of the subclass, the modified feature vector of the parent class set being used to classify the pixel represented by the modified feature vector of the parent class set with respect to the parent class set; the modified feature vector of each subset of the subclass being used to classify the pixel with respect to the subset of the subclass.

[0032] The modified feature vector for each subset of the subclass may be combined with the modified feature vector for the set of parent classes to determine a combined feature vector for the subset of the subclass, which is used to classify the pixel with respect to the subset of the subclass.

[0033] The sub-classifier can determine the sub-classification value for a subset of each sub-class.

[0034] The parent classifier and the child classifier may be probabilistic classifiers that determine a probability distribution over a set of parent classes and a subset of child classes, respectively.

[0035] Each of the pixels may be classified by at least one of: a first child classifier for a first subset of child classes associated with a first class in the parent class, and a second child classifier for a second subset of child classes associated with a second class in the parent class.

[0036] Each of at least some of the pixels can be classified by a parent classifier, a child classifier, and at least one other child classifier about a set of other subclasses, which is a child of the child classifier because each of the set of other subclasses is a child of a subclass in a subset of subclasses within the hierarchical classification scheme.

[0037] Each of at least some of the pixels may be classified by a parent classifier, a child classifier, and at least one other parent classifier with respect to a set of other parent classes that is a parent of the parent classifier, as each parent class is a child of a parent class in the set of other parent classes.

[0038] A second aspect of the present invention provides a computer-implemented method for training an image pixel classifier, the method comprising: receiving an image formed by a plurality of pixels; and receiving label data for labeling at least some of the pixels according to a hierarchical classification scheme, wherein each of the pixels is labeled as belonging to one of a set of parent classes, each parent class in the parent class is associated with a subset of child classes, and each of the pixels is also labeled as belonging to one of a subset of child classes associated with the parent class to which the pixel belongs, wherein each parent class corresponds to a category of visible structures, and each of the subsets of child classes associated with each parent class corresponds to a different type of visible structure in the category; and using the pixel data and the label data to train a parent classifier to classify image pixels with respect to the set of parent classes, and training at least one child classifier to classify image pixels with respect to a subset of child classes associated with one of the parent classes.

[0039] In an embodiment, the classifier may be trained by classifying each pixel with respect to a set of parent classes and each subset of child classes using the classifier, determining respective error signals for the set of parent classes and each subset of child classes, respectively, and adjusting parameters of the classifier to minimize the error signal.

[0040] The error signal for the parent class set may provide a measure of the difference between the label data for the parent class set and the results of pixel classification performed on the parent class set.

[0041] The error signal for each subset of subclasses can provide a measure of the difference between the label data for the subset of subclasses and the results of pixel classification performed on the subset of subclasses, wherein any pixel that is not labeled for the subset of subclasses does not affect the error signal for the subset of subclasses.

[0042] In this case, any pixels that are not labeled with respect to the subset of the subclass may still be classified with respect to the subset of the subclass, but the results do not affect the error signal for the subset of the subclass.

[0043] This method may be performed for a plurality of images, wherein at least one of the plurality of images is not labeled at all with respect to at least one of the subsets of the subclass, so that the error signal for the image and the subset of the subclass is always a zero error signal.

[0044] The data of a pixel may include a plurality of pixel feature vectors generated by a feature extractor based on the pixel and a set of pixels related to the pixel, each pixel feature vector representing one pixel.

[0045] The parameters of the feature extractor can be tuned during training along with the parameters of the classifier to minimize the error signal.

[0046] Each feature vector can be processed according to the feature vector modification parameters to generate: a modified feature vector of the parent class set and a modified feature vector of each subset of the subclass, wherein the modified feature vector of the parent class set is used to classify the pixel represented by the modified feature vector of the parent class set with respect to the parent class set; and the modified feature vector of each subset of the subclass is used to classify the pixel with respect to the subset of the subclass.

[0047] The feature vector modification parameters may be adjusted during training together with the parameters of the feature extractor and classifier in order to minimize the error signal.

[0048] The modified feature vector for each subset of the subclass may be combined with the modified feature vector for the set of parent classes to determine a combined feature vector for the subset of the subclass, which is used to classify the pixel with respect to the subset.

[0049] The modified feature vectors can be combined by concatenation.

[0050] The error signals can be combined into a cost function that is minimized during training.

[0051] Another aspect of the present invention provides a hierarchical classifier embodied as a computer system, comprising one or more processors configured to: process a set of inputs to determine a feature vector for each input; modify the feature vector of each input to generate a modified parent feature vector; classify each input associated with at least one parent class based on the modified parent feature vector; modify the feature vector of each input to generate a modified child feature vector; determine a combined feature vector for each input by combining the child feature vector of the input with the parent feature vector of the input; and classify each input with respect to a set of child classes, the set of child classes being children of a parent class in a classification hierarchy, based on the combined feature vector.

[0052] In an embodiment, each input may be a pixel of an image.

[0053] Another aspect provides a method for training the above-mentioned hierarchical classifier to learn a parameter set by the following steps, the steps being: receiving a training input set with respective training labels, at least some of the training inputs in the training input set being labeled as belonging to a subclass and belonging to one of a parent class; using a hierarchical classifier to classify each input associated with: (i) at least one parent class, and (ii) a set of subclasses; determining an error signal for at least one parent class by comparing the result of (i) with the parent class label; determining an error signal for the set of subclasses by comparing the result of (ii) with the subclass label; and adjusting parameters to minimize the determined error signal.

[0054] Another aspect provides a hierarchical image segmentation system configured to implement any of the steps or functions disclosed herein.

[0055] Another aspect provides a computer program comprising computer readable instructions stored on a non-transitory storage medium, and the computer readable instructions are configured to, when executed on one or more processors, cause the one or more processors to implement any steps or functions disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] For a better understanding of the invention and to show how embodiments of the invention may be implemented, reference is made to the following drawings by way of example, in which:

[0057] Figure 1 A highly schematic block diagram of a system for training a road detection component based on machine learning is shown;

[0058] Figure 2 A highly schematic block diagram of an automated vehicle is shown;

[0059] Figure 3 shows a highly schematic block diagram of a vehicle for capturing street scene images for use in training;

[0060] Figure 4 An example of a hierarchical road structure classification scheme is shown;

[0061] Figure 5 A schematic block diagram showing a traditional neural network structure;

[0062] Figure 6 An example of processing performed within a convolutional neural network is shown;

[0063] Fig. 7A and 7B shows an example of the convolution process applied to an input volume in a convolutional neural network;

[0064] Figure 8 shows a schematic block diagram of an example image segmentation network;

[0065] Fig. 9 A schematic block diagram of an example image segmentation network combining pooling connections and skip connections is shown;

[0066] Fig. 10A and 10B A schematic block diagram of a hierarchical image segmentation network in an embodiment of the present invention is shown;

[0067] Fig.11 shows a hierarchical image segmentation network during the training phase;

[0068] Fig. 12A and 12B The principle is shown by way of example, according to which a cost function for training a hierarchical segmentation network can be determined;

[0069] Fig.13A and 13B An extension is shown in which a modified version of the pixel feature vector is determined and combined. DETAILED DESCRIPTION

[0070] In the field of autonomous driving, a road or other structure detection component (also called a machine vision component) may be required, which, given visual input, can determine real-world structure, such as road or lane structure, i.e., which part of the image is the road surface, which part of the image constitutes a lane on the road, etc. In particular, the structure detection is fed into the decision making by the autonomous vehicle decision engine.

[0071] In the described embodiment of the invention, structure detection is achieved using a hierarchical classification scheme applied to image pixels individually, and in turn used as the basis for such decisions.

[0072] The benefit of a hierarchical classification scheme is that even if a particular pixel cannot be classified with sufficient certainty with respect to a lower level classification in the hierarchy, for certain types of decisions, the decision engine can still "fallback" on a higher level classification when a failure of a lower level classification further down the layer is detected. That is, while in extreme cases a default action may be performed in response to a failure to classify a certain set of pixels with sufficient certainty, in other cases the decision engine can still safely make certain types of decisions by selecting one of a range of options based on a higher level classification, certain types of decisions can be made with sufficient certainty based on the set of pixels, while other types of decisions that require (one or more) lower level classifications can be safely deferred.

[0073] In the context of image processing, classification refers to the task of identifying structure within an image. Classification can be performed at the image level, that is, the image as a whole is classified with respect to a set of image structure classes. Classification can also be performed at the level of structure within the image by individually classifying the pixels of the image according to which structure they form part of, which is also referred to herein as "segmentation" or pixel-level classification.

[0074] Preferably, this is achieved through machine learning using a convolutional neural network trained using training images. These training images are like the images seen from the camera of the autonomous vehicle, but they have been annotated with the information that the neural network needs to learn. For example, they will have annotations marking which pixels on the image are road surface and / or which pixels on the image belong to lanes. At training time, the network is shown thousands (preferably tens of thousands) of these annotated images and learns to itself which features of the image indicate that a pixel is part of a lane, or road surface, etc. At runtime, the network can then make its own decisions using images it has never seen before. The training images can be manually annotated for training. The training images can be, for example, frames taken from a collection of video images.

[0075] Figure 1 A highly schematic functional block diagram of a training system for training the structure detection component 102 based on annotated street scene images (also referred to herein as road images) is shown. That is, the street scene images have associated image annotation data. Figure 1 In the figure, a street scene image is labeled as 104, and its corresponding image annotation data is labeled as 106. The annotation data 106 marks the location of (one or more) certain structures (e.g., roads, lanes, intersections, non-drivable areas, etc.) within the image 104, as well as possible objects within the image (e.g., other vehicles, pedestrians, street signs or other infrastructure, etc.).

[0076] The images can be divided into a training dataset, a test dataset, and a validation dataset, which are labeled 108a, 108b, and 108c, respectively.

[0077] The detection component 102 takes the street scene image labeled U as input image data and generates an output V=f(V), where f is a function defined by a set of model parameters of the detection component 102.

[0078] The detection component 102 is trained based on the training image 108a so that its output f(U) matches the corresponding annotation data. This is a recursive process in which the input component 112 of the training system 110 systematically inputs the image data of the training image 108b to the structure detector 102, and the training component 114 of the training system 110 adjusts the model parameters in an attempt to optimize an objective function (also called a cost function or an error function) that provides a measure of the difference between the output F(U) and the corresponding annotation data 106 for each training image (cost or error).

[0079] The detection component 102 may be, for example, a convolutional neural network, where the model parameters are the weights between neurons.

[0080] The test data 108b is used to minimize overfitting, which refers to the fact that beyond a certain point, the accuracy of the detection component 102 on the training data set 108a is improved, which is not conducive to its ability to generalize images not encountered during training. Overfitting can be identified as a point where improving the accuracy of the detection component 102 on the training data 108 will reduce (or not improve) its accuracy on the test data, where accuracy is measured according to the objective function. The purpose of training is to minimize the objective function to the extent that the objective function is minimum when not overfitting. Training can, for example, be based on back propagation (backward propagation of errors), based on the calculation of the gradient of the objective function about the model parameters. In this case, the training component 112 can be referred to as a back propagation component.

[0081] If desired, the validation data set 108c may be used to provide a final assessment of the performance of the detection assembly.

[0082] Figure 2 A highly schematic block diagram of an automated vehicle 200 is shown, which is shown to include an instance of a trained detection component 102 having an input connected to an image capture device 202 of the vehicle 200 and an output connected to an automated vehicle controller 204. In use, the trained structure detection component 102 of the automated vehicle 200 detects structures within images captured by the image capture device 102 in real time according to its training, and the automated vehicle controller 204 controls the speed and direction of the vehicle based on the results, with no or limited input from any human. In this example, the vehicle 200 is a car, but it can be any form of vehicle.

[0083] In the examples described below, structure detector 102 is trained to detect road structure (and optionally other structures as well), and structure detector 102 may be referred to herein as a road detection component or a road detector. However, the same techniques may be equally applicable to other forms of visual structure and may be applied beyond autonomous driving.

[0084] The road detection component 102 and the automatic vehicle controller 204 are functional components of the automatic vehicle 200, which represent certain high-level functions implemented within the automatic vehicle 200. These components can be implemented in hardware or software or a combination of both. For software implementation, the functions discussed are implemented by one or more processors (not shown) of the automatic vehicle 100, which can be a general processing unit such as a CPU and / or a dedicated processing unit such as a GPU. When the machine-readable instructions stored in the memory are executed on one or more processors, the machine-readable instructions stored in the memory enable these functions to be implemented. For hardware implementation, dedicated hardware (e.g., an application-specific integrated circuit (ASIC) and / or a field-programmable gate array (FPGA)) can be used to implement the functions discussed.

[0085] The image capture device 202 may be a three-dimensional (3D) image capture device that can capture 3D image data. That is, in addition to information about its position within the image position of the camera, there is also depth information about the visual structure. For example, this can be provided using stereo imaging, LIDAR, time-of-flight measurement, etc. For example, the image capture device 202 may be a stereo image capture device having a pair of stereoscopically arranged image capture units (cameras). The image capture units each capture a two-dimensional image, but the arrangement of these cameras is such that depth information can be extracted from the pairs of two-dimensional (2D) images captured simultaneously by the cameras, thereby providing three-dimensional (3D) imaging. However, it will be appreciated that other forms of 3D imaging that can provide depth information for a two-dimensional array of pixels may be used in the present context. Although in Figure 2 Only one image capture device 202 is shown in FIG. 1 , but the automated vehicle may include multiple such devices, such as forward and rearward image capture devices.

[0086] Note however that for the purposes of the present invention, 3D images are not required. The present invention can be implemented using only 2D images, so a single camera is sufficient. The described hierarchical training and classification scheme can have many uses that do not require 3D information.

[0087] It should be understood that the above is a highly simplified description of certain automated vehicle functions. The general principles of automated vehicles are known and will not be described in detail.

[0088] Figure 3 A simplified block diagram of a vehicle 300 is shown, which can be used to capture road images to be annotated, that is, as described above with reference to Figure 1 The images may be captured as frames of short video clips recorded as vehicle 300 travels along the road. Vehicle 300 may be referred to as a training vehicle, as it is used to compare it with the vehicle 300. Figure 2300 is a convenient shorthand for distinguishing the automated vehicle 200 from the training vehicle 300. The training vehicle 300 is shown as including an image capture device 302, which may be a forward or rearward facing image capture device, and which is coupled to a processor 304. The processor 304 receives the captured images from the image capture device 302 and stores them in a memory 306, from which the captured images may be retrieved for use in the manner described below.

[0089] As described above, the captured images may then be used to train the road detection component 102 .

[0090] The road detection component 102 is trained to perform pixel-by-pixel image classification, i.e., to classify each pixel of each image separately. This is sometimes referred to in the art as image segmentation, as opposed to image classification, which is about classifying the image as a whole. In this regard, it is noted that, unless otherwise noted, references to classification in the following description refer to pixel-level classification (i.e., segmentation), not image-level classification.

[0091] In the embodiment of the invention described below, the road detection component 102 is trained to classify such images according to a hierarchical classification scheme with at least two levels of road structure categories. According to the hierarchical classification scheme, a single pixel can be classified simultaneously with respect to multiple road structure categories at different levels of the hierarchy. Each category at each level except the lowest level is directly related to a subset of one or more categories at its directly next level, the higher level categories being called parent categories, and the one or more categories directly related to them being called children of the parent category. Each parent category corresponds to a category of visible structure, and its (one or more) subcategories correspond to (different) types of visible structures within the category. For schemes with three or more levels, each category at any level except the highest level is a child of the parent category at the level above it. Any category at any level except the lowest level can be the parent of one or more categories below that level, so categories at levels except the highest and lowest levels can be both parent categories and child categories. Therefore, at the top of the hierarchy, the highest level categories correspond to broad categories of road structure, and as one moves down the hierarchy, these categories correspond to increasingly specific categories / types. Another term to note is that a class that is both a parent and a child class corresponds to a specific type of road structure class within the (broader) road structure class to which its parent class corresponds, but also corresponds to a (narrower) road structure class, with each of its own child classes corresponding to a different type of road structure within that narrower road structure class. A class that is itself a child of a parent is called a grandchild of that parent.

[0092] As the automated vehicle 200 travels, the image capture device 202 repeatedly captures images of the vehicle's immediate surroundings. The captured images are provided to the road detection component 102, and segmentation is applied to each of those images in real time. The results are used as the basis for one or more decision processes implemented by the decision engine of the controller 204, where the controller 204 uses knowledge of the surrounding road structure relative to where the vehicle 100 is currently located to autonomously make driving decisions.

[0093] As an example, Figure 4 An example of a hierarchical road structure classification scheme with a first, second and third level is shown, the first, second and third levels are respectively referred to as levels A, B and C, and are so labeled in the figure. Level A is the highest level, level C is the lowest level, and level B is an intermediate level between levels A and C. It will be appreciated that this is merely an example, and the hierarchical classification scheme may have any number of levels more than one. There are two types of roads at level A and in this case there are mutually exclusive categories of road structure, namely "drivable" (A1) (road) and "non-drivable" (A2). Therefore, any pixel within the image can be classified as drivable (i.e., corresponding to a point in an area where the vehicle 200 can travel) or non-drivable (i.e., corresponding to a point in an area where the vehicle 200 cannot travel). Road category A1 has two subcategories at level B: "markings" (B2) for pixels that are part of a road marking area and "lanes" (B1) corresponding to points on a road lane. Lane category B1 then has three subcategories at level C: "ego lane" (C1) for pixels that are part of the lane in which vehicle 200 is currently traveling (ego lane), "non-ego lane" (C2) for pixels that are not part of the ego lane but are part of a different lane in which vehicle 200 can travel (i.e., a lane in which vehicle 200 is allowed to change lanes if safe). Line marking category B2 has two subcategories at level C: "edge" (C3) for pixels that are part of a line marking area at the edge of the road, "middle" (C4) for pixels that are part of a line marking area in the middle of the road (i.e., not at the edge).

[0094] In general, the subclassification should be exhaustive. It is possible to train multiple overlapping layers in the same network, which may be useful for some applications.

[0095] The non-drivable category A2 is shown as having two subcategories at level B: "sidewalk" / "pedestrian path" (B3) and "shadow" (B4) corresponding to a shadow area of ​​the road (or similar restricted area) where vehicles are not allowed to travel.

[0096] Note, however, that in general, a parent class can have any number of subclasses.

[0097] according to Figure 4 In the scheme of , pixels that are part of the ego lane can be assigned to categories C1 (ego lane), B1 (lane), and A1 (drivable) at the same time, or in the case of probability (or other score-based) classification, pixels that are part of the ego lane are assigned high probabilities (or other scores) for these categories. However, the advantage of this scheme is that, taking this pixel as an example, the road detector 102 can reasonably determine that it belongs to the drivable area and lane (category A1 and B1), but at the same time is not sure whether it is a ego lane or a non-ego lane. In that case, it is still possible to classify pixels associated with category A1 and category B1 with a high enough certainty, which may be a sufficient level of knowledge for at least some types of decisions made by the controller 204, even if it is currently unable to classify pixels associated with categories C1 and C2 with sufficient certainty. That is, although the road detector 102 currently fails to classify pixels associated with categories C1 / C2 lower in the hierarchy, its successful classification of pixels associated with categories A1 / A2 and B1 / B2 can provide a sufficient basis for at least some types of decisions made by the controller 204.

[0098] In other words, the benefit of the hierarchical classification scheme is that even if the road detection component 102 is unable to classify a particular pixel with sufficient certainty with respect to a lower level category, it is still able to "fall back" at a higher level classification for certain types of decisions upon detecting a failure of a lower level classification further down the layer. That is, although in extreme cases the controller 204 performs a default action (e.g., stops as quickly as safely possible) in response to the road detection component 102 being unable to classify a particular set of pixels with sufficient certainty, in other cases where the road detection component 102 is unable to classify certain pixels with sufficient certainty with respect to one or more lower level categories, the controller 204 can still safely make certain types of decisions by selecting one of a series of options based on the higher level classification that can be made with sufficient certainty with respect to the set of pixels, while other types of decisions that require (one or more) lower level classifications can be safely postponed. The hierarchical classification scheme thus provides a "middle ground" for the controller 204, where when the controller 204 knows that the lower level classification of particular pixels has failed but the higher level classification of those pixels is still reliable, the controller 204 can continue to operate safely by adjusting its decisions accordingly without having to resort to a default operating mode that fails completely.

[0099] An example is when different parts of a system require different levels of detailed segmentation.

[0100] Another example might be that the system might take some information from vision and some from the map, depending on the level of detail of the visual detection. For example, extract the lanes from the segmentation if they exist, and if not, use the road segmentation and then divide it by the number of lanes indicated by the map.

[0101] Although each parent class above has two or three subclasses, a given parent class may have any desired number of subclasses. In fact, any other or alternative road structure categories may be selected in any manner that respects the underlying principles of hierarchical classification.

[0102] It will be appreciated that the classification scheme may be implemented in conjunction with any segmentation method that may be used to classify pixels with respect to road structure classes within a hierarchy.The described method may generally be implemented with any form of machine learning component that may learn a hierarchy.

[0103] The following describes a preferred embodiment of the present invention, which uses a convolutional neural network (CNN) as the basis for road structure detection. First, some useful context about CNN will be described, but it is noted that this is not exhaustive and is provided by way of example only. CNN can be implemented with a variety of different architectures and components, and it should be understood that the methods described herein can be implemented using any suitable CNN architecture, or indeed any machine learning architecture capable of learning pixel embeddings.

[0104] In order to achieve the purpose of the hierarchical classification scheme provided in this paper, it is important to provide a learning mechanism that can obtain error signals at various levels of the hierarchy. This is independent of the exact structure of the CNN. In fact, different styles of CNN feature extractors with significantly different structures can be used in combination with a general purpose hierarchical backend that performs hierarchical classification based on the pixel feature vectors extracted by the feature extractor. The feature vectors provided by the feature extractor are manipulated as described below to perform hierarchical classification.

[0105] Figure 5 A schematic block diagram is shown representing a general structure for an artificial neural network structure 500, which is shown as having a plurality of processing layers 502 (input layer), 504, 506 (first and second hidden layers), and 508 (output layer). Figure 5 Two hidden layers are shown in Figure 1, but a neural network can have any number of hidden layers (or none at all). Each layer is shown as including a number of processing nodes, also called "neurons." Each node is represented as a circle containing an integer indicating the index of the node in its layer. Figure 5A feed-forward architecture is depicted in which each node of the input layer 502 has an input for receiving an input value, which is defined as a component of the input vector received at the input layer 502. Each node of the first hidden layer 504 has an input, each input connected to the output of a different node of the input layer 502 to receive a calculated value therefrom (output value of the input layer 502, input value to the first hidden layer 504); each node of the second hidden layer 506 has an input, each input connected to the input of a different node of the first hidden layer 504 to receive a calculated value therefrom (output value of the first hidden layer 504, input value to the second hidden layer 506); and each node of the output layer 508 has an input, each input connected to the output of a different node of the second hidden layer 506 (output value of the second hidden layer 506, input value to the output layer 508); and an output configured to provide a value (output value of the output layer 508), which is a component of the output vector at the output layer 508. A neural network layer in which each node is connected to every node in the previous layer can be called a "fully connected" layer. In traditional neural network architectures, all layers are fully connected. The input layer 502 does not itself compute new values, but is a convenient way to represent the ability of a neural network to receive a set of input values ​​(input vectors) to be processed within the network. In this way, the output value of each node of the input layer 502 matches its input value. Each node in the remaining layers 504-508 calculates its output value by processing the input values ​​received from the nodes in the previous layer. The output value at each node is calculated based on the weighted sum of its inputs received from the nodes to which it is connected, which can be expressed mathematically as follows:

[0106]

[0107] where v i represents the output value of node i in a given layer, u j represents the jth input received at this node, w i，j Indicates that the input value u is applied at this node j The weights of , and g represents the function (activation function) applied at the node. The weights w across nodes and layers i，j Constitute the model parameters, which are referenced above during training Figure 1 The outlined approach is modified. A "bias term" can be incorporated into the above by fixing u0 = 1, so that w i，0The bias term is sometimes written as a separate term in the equation, and is also adjusted during training. As mentioned above, when trained correctly in this way, the neural network is able to generalize its training to examples it has never encountered before. The power of neural networks comes from the use of nonlinear activation functions g in one or more layers, as this can fully utilize the full potential of the neural network's architecture.

[0108] Neural networks are known per se, and therefore the general principles of neural networks will not be described in further detail here.

[0109] As described above, in this example, a specific convolutional neural network (CNN) architecture is used in the road detection component 102.

[0110] refer to Figure 6 , the inputs and outputs of each layer of a CNN are conceptualized as "volumes" (i.e., three-dimensional arrays) in a discrete three-dimensional space, each of which is formed by a stack of two-dimensional arrays. Each two-dimensional array in the stack is also referred to herein as a "data layer" and in some cases a "feature map". Therefore, it should be noted that the term "layer" may be used herein to refer to a processing layer of a neural network, and also to a data layer in a stack of data layers that constitute an input volume or an output volume. This will be clear in the context.

[0111] By way of example, Figure 6 A sequence of five such volumes 602, 604, 606, 608, and 610 is shown, which may be generated by a series of convolution operations, pooling operations, and nonlinear transformations as described below. For reference, the two data layers within the first volume 602 are labeled 602a and 602b, respectively, and the two data layers within the fifth volume 610 are labeled 610a and 610b, respectively.

[0112] Each value in the volume can be assigned a discrete position in three dimensions (index triplet), denoted as (x, y, z), where z is the index (distance along the stack) of the data layer to which the value belongs, and (x, y) is the index defining its position within that data layer (i.e., in the plane of the data layer). Here, U represents the input volume of a given layer, and V represents the output volume of a given layer, noting that the input volume U of a layer can include or be derived from the output volume V of another layer. x，y，z and v x,y，z Represents the value of the position (x, y, z) in volume U and V, respectively, in K z and L z represents the data layer at depth z in volumes U and V such that u x，y，z and v x，y，z The data layer K z and L zThe value at position (x, y) in .

[0113] In turn, a three-dimensional position (x, y, x) can be assigned to each node in a given processing layer of the CNN, where the node at position (x, y, z) in that processing layer will compute the value v of the output volume V x，y，z In this regard it is important to note that although the term data layer refers to a two-dimensional data structure, the term processing layer, when considered from this perspective, refers to a three-dimensional node structure.

[0114] Each volume has a width and height (extent in the x and y directions, respectively) and depth (extent in the z direction). The depth is the number of data layers in the volume. The width and height can vary with z (i.e. be different at different data layers) since different data layers can contain different numbers of values, but it may be more convenient to use padding or similar techniques if necessary to ensure that each volume has the same width and height at all depths.

[0115] A CNN differs from a traditional neural network architecture in that it has processing layers that are not fully connected. Instead, a processing layer is provided that is only partially connected to (one or more) other processing layers. In particular, each node in a convolutional layer is only connected to a local 3D region of (one or more) processing layers, from which each node receives input, and on which the node performs convolution with respect to a filter, as described in further detail later. As described below, the local partial connections from one layer to the next respect the location (x, y) of the values ​​within their respective bodies, so that the (x, y) location information is retained in the CNN at least to some extent as the data passes through the network.

[0116] To illustrate this in the context of image processing, consider the following: volume 602 is an image received at the input layer of a CNN. For a color image with three color channels (e.g., red, green, and blue (RGB)), the image volume 602 has three depths, z=0, 1, 2, corresponding to the three color channels, respectively, and the value at (x, y) in the data layer z is the pixel value of the corresponding color channel at position (x, y) in the image. That is, the output value v of the input layer is x，y，0 , v x，y，1 and v x,y，2 are the pixel values ​​of the three color channels at the position (x, y) in the image.

[0117] Note that while it is useful to think of an image as a three-dimensional structure in this way, in the absence of physical depth information associated with the pixels, this is still a 2D image. Note also that the term "depth" may be used herein in relation to images to refer both to the number of color channels in the context of CNN processing, and to the physical pixel depth, i.e., the distance from the device that captured it in the case of a captured 3D image. What is meant will be clear in context.

[0118] Fig. 7A and 7B The example shows the depth Z applied at the convolution layer. U The input volume U is used to generate the depth Z V The principle of the convolution operation on the output volume V of . This involves applying a set of Z V The filter is expressed as:

[0119] F={F z |z=0,...,Z V -1}.

[0120] That is, the z dimension of the output volume V corresponds to the set of filters applied at the layer in question, and the depth of the output layer V matches the number of filters applied at the convolutional layer that computed V. By using the corresponding filter F z Applying a set of local convolutions to the input volume U allows us to compute each data layer L of the output volume V z For this purpose, in the case where the output volume V is computed by convolution, this direction z is referred to herein as the filter direction.

[0121] A node at any given position (x, y, z) in a convolutional layer applies an applicable filter F in the following way z Applied to its input. Filter F z is defined as a three-dimensional array of weights with size X×Y×Z, where X, Y, and Z are integers representing the width, height, and depth of the filter. The filter can be mathematically written as:

[0122]

[0123] where w z，x′，y′，z′ is the filter F at position (x′, y′, z′) z The different filters applied in a given convolutional layer can have different widths X and / or heights Y, but each filter is typically chosen so that its depth Z is consistent with the depth Z of the input volume U to which it is applied. U That is, usually each filter F zThe value v of the output volume V at position (x, y, z) is calculated by the node at position (x, y, z) in the convolutional layer. x，y，z , as the filter F z The convolution with a local portion of the input volume U of size X×Y×Z, within the “perception window” of the filter at that location (x, y, z), is defined as:

[0124]

[0125] Where W x，y，z (U) is the perception window W at (x, y, z) x，y，z This localization aspect can be equivalently stated as follows: the convolutional layer that generates V is partially connected to the layer (another layer) that outputs U, because the node at each position (x, y, z) in the convolutional layer is only connected to the nodes in other processing layers within the perception window for that position (x, y, z), that is, the output W. x，y，z Note that this does not exclude that the node is additionally partially or fully connected to one or more additional processing layers, for example via one or more skip connections (see below) from (one or more) earlier layers and / or one or more feedback connections from (one or more) later layers.

[0126] In the above, * denotes convolution and S is an integer called the "stride" with which the filter F is applied. z As mentioned above, v x，y，z The data layer L representing V at position (x, y) z The value in the data layer L z Corresponding to the filter F z The stride S defines the "resolution" of the convolution operation, as it defines how much overlapping parts of the input volume U are considered. Figure 7B , the convolution operation can be conceptualized as “sliding” the filter’s perception window in the x-direction and y-direction by S by each unit increment, as Fig. 7A The example in Figure 7B In Figure 7B In the manner shown, two filters F0 and F1 are applied to an input volume U of depth 3 with a stride S = 1 to produce an output volume U of depth 2, where the data layers L0 and L1 correspond to the filters F0 and F1, respectively. However, different strides S can be applied to different filters F z (And even different strides S1, S2 can be applied to the x and y directions.) The convolution is the summation of the values ​​of the input volume U within the perceptual window weighted by the filter weights:

[0127] v′ x，y，z =F z *W x，y，z (U)=Σ x′，y′，z′ w z，x′，y′，z′ u SX+x′ ,s y+y′，z′ .

[0128] It can be seen that this is equivalent to the weighted sum computed by the nodes in a traditional neural network, but restricted to the perceptual window of the filter. It is this restriction that means that the (x, y) position in the formation is preserved, at least to some extent, when applying consecutive convolutions in the network. Another difference from traditional neural network architectures is that for a given depth z in the convolutional layer, each node applies the same filter and therefore the same set of weights (whereas in traditional networks each node applies its own set of weights). In the context of structure recognition, this is based on the assumption that the structure to be recognized is equally likely to appear anywhere in the image. In cases where this assumption is not needed, this restriction can be lifted and different nodes at the same depth z but at different (x, y) positions within the processing layer can be allowed to apply different filters.

[0129] The data layer L given z is generated by convolution z is called the “feature map” of filter z. Each filter corresponds to a specific structural feature, and in the context of image processing, the feature map for that filter is actually a map of where that structural feature appears in the image.

[0130] When convolutions are applied directly to an image in this way, each filter acts as a low-level structure detector in that “activation” (i.e., large output values) occurs when the pixels within the filter’s receptive window form certain structures (i.e., structures that match a particular filter). However, when convolutions are applied to a volume that is itself the result of an earlier convolution in the network, each convolution is performed on a set of feature maps about different features, so activations further into the network occur when specific combinations of lower-level features are present in the receptive field. Thus, with each successive convolution, the network is detecting the presence of higher and higher-level structural features that correspond to specific combinations of features from previous convolutions. Thus, in earlier layers, the network is effectively performing lower-level structure detection, but the network gradually moves toward higher-level semantic understanding of structure in later layers. The filter weights are learned from annotated examples in training, which is how the network learns what structure to look for.

[0131] As we all know, convolution can be used in combination with other operations. For example, pooling and nonlinear transformations are typical operations used in combination. As known in the art, pooling is a form of dimensionality reduction.

[0132] In order to exploit the full potential of its architecture, nonlinear transformations are introduced into the network. One example of an effective nonlinear function that can be used is rectification (ReLu), which returns zero for negative input values ​​and returns the input value for positive input values. However, there are different ways of introducing nonlinearity into a network.

[0133] The nonlinear function g can be conveniently used as an activation function on the convolutional processing layer to provide an output volume V′ given by:

[0134] V′=g(V)=(v′ x，y，z )

[0135] where v′ x，y，z is the value of V′ at position (x, y, z), and (v′ x，y，z ) represents a 3D array of such values.

[0136] Another non-linear activation function that can be used is the softmax function (g=softmax):

[0137]

[0138] That is, the value of V′ at position (x, y, z) is the value v at the matching position (x, y, z) in V x，y，z The value of V′ at position (x, y, z) is normalized over the depth of V. The result of this normalization is that the value of V′ at any given (x, y) position sums to 1 over the depth dimension z:

[0139] ∑ z v′ x，y，z =1

[0140] This in turn means that the set of values ​​of V′ at (x, y) over the entire depth is:

[0141] {v′ x，y，z |z=0,...,Z V}

[0142] can be interpreted as 0, ..., Z V The probability distribution on can be interpreted as a category, so that v′ x，y，z can be interpreted as the original image (denoted as P x，y ) belongs to category A. z The probability of is:

[0143] v′ x，y，z =Pr(P x，y ∈A z) In other words, the feature map provided at the output layer provides a probability distribution over the input image at the pixel level, and each point in the feature map is a softmax probability output with respect to the corresponding pixel and image structure category (referred to as a softmax feature map in this article, or more generally, a classification feature map), thereby achieving the desired segmentation.

[0144] Therefore, a convolutional layer with a softmax activation function may be referred to as a probabilistic (softmax) classifier or a softmax layer in this article.

[0145] CNN-based segmentation:

[0146] Now refer to Figure 8 Describe how to apply these principles to the problem of image segmentation based on road structure. Figure 8 A convolutional neural network with five processing layers 802 (input layer), 804, 806, 808 (hidden layers), and 810 (output layer) is shown. According to the above principle, these processing layers are represented by volumes. The nodes within these volumes are not shown individually, but correspond to various points within these volumes.

[0147] An image (not shown) is received at the input layer 802 and passed to the first hidden layer 804 for processing. Each of the hidden layers 804, 806, and 808 applies a convolution with a ReLu activation (or other nonlinear activation) to the input volume (not shown) from the previous layer. The output layer 810 also applies a convolution to the input volume (not shown) from the last hidden layer 808 but with a softmax activation, so that it acts as a softmax classifier as described above. Each node of the softmax layer 810 at (x, y) is connected to a 1x1xZ volume of nodes at (x, y) in the last layer 808 of the feature extractor 808, where Z is the depth of the layer 808. The output values ​​of this volume, labeled 809, constitute a semantically rich feature vector for the pixel at (x, y), which the softmax classifier 810 uses to classify the pixel.

[0148] As the term is used herein, the input layer and hidden layers 802-808 are an example of a CNN feature extractor that work together to extract features from the input image for classification by the softmax layer 810. Figure 8800. In each convolutional layer 804-810, nodes are partially connected to nodes in the previous layer, and convolution is applied to the outputs of those nodes as described above. For example, the points labeled 824, 826, 828, and 820 in layers 804, 806, 808, and 810 correspond to nodes, each of which is connected to a subset of nodes in the previous layer defined by the receptive field of the correlation filter and represented as 812, 814, 816, and 818, respectively.

[0149] The network is trained using annotated images so that the output of the softmax layer 810 at position (x, y) throughout its depth provides a probability distribution "running along" the z dimension of the softmax layer 810 over the set of desired image structure classes. Figure 8 In , the set of categories is represented as {A1, ..., AN}, where AN is the depth of the softmax layer. Therefore, the complete set of nodes in the softmax layer 810 provides a probability distribution over the desired image structure category at each point (pixel) in the original image, thereby achieving the desired segmentation.

[0150] Fig. 9 A CNN feature extractor 800 is shown in which pooling is combined with one or more upsampling layers (902, 904 in this example) and a "skip connection" 900. An upsampling layer is a layer that upsamples the input volume U in order to increase its (x, y) size (and simultaneously reduce its depth if necessary). Upsampling cannot recover (x, y) position information from a volume in which information has been lost, and the function of the skip connection is to provide such position information directly from (one or more) earlier layers to the (one or more) upsampling layers at the end of the network, "skipping" the intermediate layers that discarded the (x, y) position information. This information is incorporated into the upsampling process in order to ultimately combine the pixel-level position information received from (one or more) earlier layers with the semantic understanding of the image structure provided by the later layers. A variety of possible upsampling functions can be used, one example of which is a transpose convolution.

[0151] In this example, the final layer 904 of the feature extractor 800 has (x, y) dimensions that match the (x, y) dimensions of the input layer and, therefore, the input image, such that each (x, y) position within the final layer 904 corresponds to a pixel at position (x, y) in the original image.

[0152] CNN-based hierarchical segmentation:

[0153] CNN-based segmentation is known per se and is therefore described here in further detail only to the extent that it helps to understand how CNN-based segmentation can be applied to the hierarchical image classification scheme according to the present invention.

[0154] Fig. 10A and 10B A schematic block diagram of an exemplary hierarchical image segmentation network 1000 for use in the structure detection component 102 is shown, which classifies an image at the pixel level according to a hierarchical classification scheme of the type described above.

[0155] The hierarchical image segmentation network 1000 is shown as including the above reference Figure 8 and Fig. 9 A CNN feature extractor 800 of the type described. Fig. 10B The internal structure of the CNN feature extractor is completely omitted, and Fig. 10A Only the input layer 802 and the final layer 1012 (corresponding to Figure 8 Layer 810 and Fig. 9 904 in the figure).

[0156] In order to provide hierarchical classification, multiple softmax layers are provided, each of which is based on Figure 8 and 9 That is, each softmax layer is a convolutional layer with softmax activation, and the softmax layer is partially connected to the final hidden layer 1012 of the CNN feature extractor 800. Fig. 10A Only three softmax classification layers 1002 (level A), 1004 and 1005 (level B) are shown; Fig. 10B An additional softmax classification layer 1008 (level C) is shown. However, it will be appreciated that any plural number of softmax classification layers may be provided, as will become apparent below, with the number being dictated by the desired hierarchical classification scheme.

[0157] like Fig. 10AAs shown, in each softmax classifier 1002, 1004, 1006, the nodes at positions (x, y) along the z-axis work together to provide a probability distribution for the corresponding pixel at (x, y) in the original image (corresponding to the node at position (x, y) in the input layer 802) on the corresponding set of image structure categories extending along the z-axis of the softmax layer. In each softmax classifier 1002, 1004, 1006, the node at (x, y) in each data layer is shown as a local 1x1xZ body 1004 connected to the node at (x, y) and along the z-axis of the final layer 101 of the CNN feature extractor 800.

[0158] Therefore, each softmax node at position (x, y) along the z-axis in each softmax layer 1002, 1004 and 1006 receives a set of output values ​​from the local 1x1xZ volume of node 1004 in the final layer 1002 of the CNN extractor 800 connected thereto. As described above, those outputs constitute the semantic feature vector (feature set) of the pixel at position (x, y) in the image. It is sometimes referred to as pixel embedding. The feature set is a semantically rich representation of the pixel at (x, y), which takes into account the relevant pixels in the image in the area (area or volume) around the pixel. The classification node uses this feature set of pixel (x, y) to classify the pixel with respect to the image structure category in question. Partial connection makes it possible that, given any classification node at a specific (x, y) position in a given softmax layer, the partial connection to the classification node can be traced back to the block (patch) (limited area or volume) around the pixel at (x, y) in the original image through the feature extractor 800. This block corresponds to a node of the input layer 802 to which the classification node is indirectly connected via an intermediate node of the feature extractor 800 (and possibly directly via a skip connection). In that case, the set of features about the pixel constitutes a semantically rich representation of the pixel that takes into account the context provided by the surrounding image blocks.

[0159] The benefit of using this type of CNN feature extractor architecture is that it allows for very efficient creation and representation of feature sets for different patches by avoiding duplicate processing and duplication of values ​​(much more efficient than treating each patch as a separate image to be classified, which would be another way to build a pixel-level classifier).

[0160] The hierarchical aspect of classification is adapted by training many softmax classifiers to compute the conditional probability distribution. Fig. 10A and 10BIn the example of , the first softmax classifier 1002 is a parent classifier (at "level A"), and the first softmax classifier 1002 is about two image structure categories A1 and A2 (e.g., "driveable" and "non-driveable" ( Figure 4 The second softmax classifier 1004 is a child classifier (at "level B") that classifies pixels into categories B1 and B2 (e.g., "lane" and "marking" ( Figure 4 The third softmax classifier 1006 is a child classifier (also at "level B"), which classifies pixels into categories B3 and B4 (e.g., "sidewalk" and "shadow") that are children of category A2 (non-drivable). Figure 4 )) to classify the pixel in categories B3 and B4). Fig. 10B The fourth softmax classifier 1008 shown in FIG. 1 is a child classifier (at “level C”) that classifies categories C1 and C2 (e.g., “self-lane” and “non-self-lane”) that are children of category B1 (lane). Figure 4 Note that in this example, category B1 is at least both a parent category (as the parent of C1 and C2) and a child category (as the child of category A1).

[0161] A subclass classifier is trained to compute, for each pixel at position (x, y) in the original image, a conditional probability distribution over a subset of subclasses, where each of these subclasses has a common parent class. The conditional probability distribution provides the probability for each of the subsets of subclasses that a pixel belongs to that subclass, given that the subclasses all belong to the common parent class (i.e., conditionally, that the subclasses all belong to the common parent class). Figure 4 For example, a softmax classifier is trained to classify a pixel at (x, y) as follows:

[0162]

[0163] wait.

[0164] In the above:

[0165] Pr(C) is the pixel P x,y The shorthand notation for the probability of belonging to category C, that is, the full notation for Pr(C) introduced above is Pr(P x,y ∈C); and

[0166] Pr(C|D) is thex,y If pixel P belongs to category D x,y The shorthand notation for the probability of belonging to category C, that is, the full notation for Pr(C|D) introduced above is Pr(P x,y ∈C|P x,y ∈D).

[0167] This in turn allows the calculation of the given pixel P according to Bayes' theorem x,y The absolute probability of belonging to a subclass is as follows:

[0168] Pr(b1)=Pr(B1|A1)×Pr(A1)

[0169] Pr(B2)=Pr(B2|A1)×Pr(A1)

[0170] Pr(B3)=Pr(B3|A2)×Pr(A2)

[0171] Pr(B4)=Pr(B4|A2)×Pr(A2)

[0172] Pr(C1)=Pr(C1|B1)×Pr(B1)=Pr(C1|B1)×Pr(B1|A1)×Pr(A1)

[0173] Pr(C2)=Pr(C2|B1)×Pr(B1)=Pr(C2|B1)×Pr(B1|A1)×Pr(A1)

[0174] etc.

[0175] This is in Fig. 10A and 10B , where the output of each child classifier is shown as input to the relevant multiplication component, and the child classifier also receives the absolute probability of the corresponding absolute parent category probability as input. Fig. 10B , each sub-level softmax classifier can be called a conditional probability (softmax) classifier, which together with its associated multiplication component constitutes an absolute probability (softmax) classifier, which classifies pixels with respect to sub-level categories according to absolute probabilities. The absolute softmax classifiers including conditional softmax classifiers 1014, 1015, and 1018 are respectively Fig. 10B They are marked as 1014, 1015 and 1018 respectively.

[0176] Layered training

[0177] Temporarily return to Figure 1As described above, the goal of training is to use a corpus of annotated (labeled) training data to teach (in this case) the road detection component 102 the function F(U) so that it can be applied to inputs U that have never been encountered before.

[0178] In this case, the training data is annotated street scene images that are labeled according to the hierarchical classification scheme to be learned. In this way, each pixel can be labeled as belonging to multiple categories according to the hierarchy to provide a hierarchical collection of "ground truth" maps that represent the expected output at each softmax classification layer 1002, ..., 1008. Thus, for example:

[0179] A ground truth feature map of a given image and an A1 / A2 (drivable / non-drivable) classifier will mark the drivable and non-drivable areas in the image respectively;

[0180] The ground truth feature map of the image and the B1 / B2 (lane / marking) classifier will mark the lanes and marking areas of the drivable area of ​​the image respectively;

[0181] The ground truth feature map of the image and the C1 / C2 (ego lane / non-ego lane) classifier will mark the ego lane and non-ego lane regions of the lane region of the image respectively;

[0182] ·etc.

[0183] Fig.11 It shows that overall Figure 1 The training system described in , which is used to train a segmentation network 1000 based on a ground truth feature map. As indicated, the CNN feature extractor 800 together with a plurality of hierarchical softmax classifiers 1002, ..., 1008 constitute the image segmentation network 1000 to be trained. The image segmentation network has a whole set of trainable parameters, which is formed by the trainable parameters of the CNN feature extractor 800 together with the trainable parameters of each softmax classifier 1002, ..., 1008.

[0184] exist Fig.11 , the ground truth feature maps of each training image 1116 applied to softmax classifiers 1102 (A1 / A2), 1104 (B1 / B2), 1106 (C1 / C2) and 1108 (C3 / C4) are labeled as 1122, 1124, 1126 and 1128, respectively.

[0185] During segmentation training, training images 1116 are passed to feature extractor 800, which processes image 1118 according to its current parameters. The resulting output of feature extractor 800 (i.e., the corresponding pixel feature set) is fed to each softmax classifier 1002, ..., 1008, which in turn processes its received feature set according to its current parameters to generate a corresponding softmax feature map as described above.

[0186] Segmentation training is based on back propagation, which is performed on the entire set of trainable parameters of the segmentation network 1000, that is, the network 1000 as a whole. Training is performed in cycles. In each training cycle, a set of training images is passed through the network 1000, and its performance is evaluated by determining the extent to which each softmax classifier 1002, ..., 1008 can replicate the ground truth feature map in its softmax output, and based on the evaluation of the performance, the entire set of trainable parameters is adjusted for the next training cycle (until the process terminates).

[0187] As referenced above Figure 1 As briefly discussed, back propagation is performed with respect to a cost function. The cost function is designed to "penalize" the network 1000 for pixel classifications that deviate significantly from the corresponding ground truth features 1002, ..., 1008. That is, it provides an overall measure (cost or error) of the difference between the softmax outputs of the softmax classifiers 1002, ..., 1008 and the corresponding ground truth maps 1122, ..., 1128. The goal of training is to minimize the cost function before overfitting occurs, i.e., in this case, to minimize the overall difference between the softmax feature maps generated from the training images and the corresponding ground truth feature maps.

[0188] In this case, the cost function is evaluated over the entire set of softmax outputs from the set of softmax classifiers 1002, ..., 1008, i.e., those softmax feature maps are evaluated as a whole. The cost function evaluation component 1102 is shown as having inputs to receive the softmax outputs from the set of softmax classifiers 1002, ..., 1008 and the corresponding ground truth feature maps. From these, the cost function evaluation component 1002 calculates the gradients (partial derivatives) of the cost function with respect to the entire set of trainable parameters, which are then back-propagated by the back-propagation component 1114 (corresponding to Figure 1The gradient (partial derivative) is used by the training component 112 in the embodiment to adjust the entire set of trainable parameters (i.e., the trainable parameters of the feature extractor 800 and the trainable parameters of the softmax classifiers 1002, ..., 1008) according to the back-propagation algorithm. Back-propagation itself is well known in the art and will not be described in detail here.

[0189] Note that the entire set of trainable parameters on which backpropagation is performed does not necessarily include every single parameter that might be trained, but rather refers to all parameters on components of the network 1000 being trained that have been designated for segmentation training. For example, some parameters that are theoretically trainable may be pre-trained parameters that are "frozen" during segmentation training. For example, a CNN may be pre-trained to perform image-level classification, and then at least some of its "pre-trained" layers may be merged into a segmentation (i.e., pixel-level classification) network, since at least part of the feature recognition "skills" learned in the image-level classification task may also be applied to image segmentation. This is a form of so-called transfer learning. Parameters frozen in this manner are examples of parameters that do not form part of the entire set of trainable parameters, as that term is used herein.

[0190] Masking in Layered Training

[0191] Another component of training is referred to in this paper as "masking".

[0192] Masking refers to the fact that the cost function is defined so that a child classifier is not penalized for any classification of pixels outside of its parent class. That is, the cost function is defined so that a child classifier cannot be penalized for how it classifies pixels that do not belong to its parent class. For example, the "lane" / "road marking" (B1 / B2) classifier 1004 cannot be penalized for how it classifies pixels that are marked as "non-drivable" (A2) and therefore not marked with respect to any "drivable" (A1) subclass. Similarly, the "center marking" / "edge marking" (C1 / C2) classifier is not penalized for how it classifies pixels that are not marked as "road marking" (e.g., lane pixels and non-drivable pixels). This is in Fig.11 as well as Fig. 12A and Fig. 12B This is shown by way of example in , where shading is used to show image regions that do not affect the cost function for a given classifier.

[0193] The cost function is formed by combining the error signals defined for classifiers 1002, 1004, 1006, and 1008, respectively. Each error signal encodes a measure of the difference between the classification output of the associated classifier and the corresponding label at the applicable level of the hierarchy. Thus, in this example, the error function is defined with respect to the applicable ground truth value as follows:

[0194] The error signal of the drivable / non-drivable classifier 1002 provides a measure of the difference between the classification of each pixel determined by the classifier and the drivable / non-drivable (A1 / A2) pixel label at level A of the hierarchy.

[0195] The error signal of the lane / road marking classifier 1004 provides a measure of the difference between the classification of each pixel determined by the classifier and the lane / road marking pixel label (B1 / B2 - children of class A1) at level B of the hierarchy. Only pixels marked as drivable (A1) can cause this error signal, and it is irrelevant to pixels not marked as belonging to the drivable class A1.

[0196] The error signal of the self / non-self lane classifier 1006 provides a measure of the difference between the classification of each pixel determined by this classifier and the self / non-self pixel labels (C1 / C2 - children of class B1) at level C. Only pixels labeled as lane (B1) can cause this error signal and it is irrelevant to pixels not labeled as belonging to lane class B1.

[0197] The error signal of the center mark / edge mark classifier 1008 provides a measure of the difference between the classification of each pixel determined by this classifier and the center mark / edge mark pixel label (C3 / C4 - children of class B2) at level C. Only pixels marked as marked (B2) can cause this error signal, and it is irrelevant to pixels that are not marked as belonging to the marked class B2.

[0198] ·etc.

[0199] These error signals are combined into an overall cost function that is minimized during training.

[0200] In this example an image is shown where every pixel is labeled as road or non-road, although this is not required – some pixels may be completely unlabeled, in which case they will not affect the overall cost function at all.

[0201] It will be appreciated that the principles of hierarchical segmentation training described above can also be applied to machine learning models other than CNNs, such as other forms of neural networks or entirely different models.

[0202] 13 shows an optional extension of the network, in which feature vector modification components 1202, 1204, 1205 and 1208 are incorporated for classes A1 / A2, B1 / B2, B3 / B4 and C1 / C2, respectively. Each of these is partially connected to the final layer of the feature extractor 800 and receives feature vectors extracted for image pixels from the feature extractor 800. Each of these components 1202-1208 is a trainable component of the network, which can be formed, for example, by a small number of other convolutional layers.

[0203] The idea is that each of these components can freely modify the feature vector extracted for each pixel to "optimize" it for the purpose of performing the corresponding classification task. That is, to customize it to the applicable classification task. Therefore, the feature vector modification components 1202-1208 modify the feature vector of each pixel according to the respective feature vector modification parameter sets learned in training to generate respective modified feature vectors for use by the classifiers 1002-1008, respectively. As described above, these parameters are learned together with other trainable parameters.

[0204] The inventors have also recognized that the knowledge gained in tailoring the feature vector to the task of classifying pixels associated with a set of parent classes is also useful when it comes to those pixels associated with the children of those parent classes. Thus, for example, the knowledge gained by learning to tailor the feature vector to the drivable / non-drivable (A1 / A2) classification is useful for both the lane / line classification (B1 / B2-children of A1) and the sidewalk / shadow area classification (B3 / B4-children of A2). Similarly, the knowledge gained by learning to tailor the feature vector to the lane / line classification (B1 / B2) is also useful when it comes to the center lane / edge lane classification task (C3 / C4-children of B2), and so on. On the other hand, the more fine-grained knowledge learned when tailoring the feature vector to the lane / line classification task (B1 / B2) is not expected to be particularly useful when performing the coarser drivable / non-drivable (A1 / A2) classification task. That is, in this case, the knowledge gained further up the hierarchy will also be useful at the lower levels of the hierarchy, but not vice versa.

[0205] This is exploited by selectively combining modified feature vectors through cascading. Specifically, the modified feature vectors determined for a given set of parent classes are cascaded with the modified feature vectors determined for each subset of child classes to generate a combined (concatenated) feature vector for each subset of child classes.

[0206] This is Fig.13A and 13BThis is illustrated by way of example in which it is shown how to determine a modified version of the feature vector f for a given pixel determined by the feature extractor 800 and then concatenate as follows ( Indicates cascade):

[0207] For the set of categories p = {A1, A2}, f → f p

[0208] For the category set q1 = {B1, B2}, f → f q1 (Subclass of A1∈p)

[0209] · It is used to classify pixels associated with q1

[0210] For the category set q2 = {B3, B4}, f → f q2 (Subclass of A2∈p)

[0211] · It is used to classify pixels associated with q2

[0212] For the category set r1 = {C1, C2}, f → f r1 (Subclass of B1∈q1)

[0213] · It is used to classify pixels associated with r1

[0214] For the category set r2 = {C3, C4}, f → f r2 (Subclass of B2∈q1)

[0215] · It is used to classify pixels associated with r2

[0216] For the category set r3 = {C5, C6}, f → f r3 (Subclass of B3∈q2)

[0217] · It is used to classify pixels associated with r3

[0218] For the category set r4 = {C7, C8}, f → f r4 (Subclass of B4∈q2)

[0219] · It is used to classify pixels associated with r4

[0220] ·etc.

[0221] In the sense defined above, the back-propagation component 1114 and the feature vector modification components 1202-1208 are functional components of the system, which in a software implementation are implemented together with the segmentation network 1000 on one or more processors (e.g., CPU and / or GPU) of a training system, which can be, for example, a backend or other offline training system.

[0222] It should be understood that the above embodiments are described by way of example only. In view of the teachings presented herein, other embodiments and applications of the present invention will be apparent to those skilled in the art. The present invention is not limited by the described embodiments, but only by the appended claims.

Claims

1. A computer-implemented method for classifying pixels of an image, the method comprising: receiving an image formed by a plurality of pixels; classifying the pixels according to a hierarchical classification scheme, wherein at least some of the pixels are classified by a parent classifier with respect to a set of parent classes, each of the parent classes being associated with a subset of child classes, and each of the pixels is also classified by at least one child classifier with respect to one of the subsets of child classes, wherein each of the parent classes corresponds to a class of visible structure, and each of the subsets of child classes associated with each of the parent classes corresponds to a different type of visible structure within the class; and A decision process is performed by a controller based on the pixel classification; the decision process comprises: detecting a failure of the child classifier to classify at least one of the pixels, and A decision is made by the parent classifier based on the classification of the pixel.

2. The method according to claim 1, wherein: The decision making process is an automatic vehicle control process.

3. The method according to claim 1 or 2, wherein: The child classifier determines at least one child classification value that classifies the pixel with respect to a subclass of a subset of the subclasses, and the parent classifier determines at least one parent classification value that classifies the pixel with respect to the parent class associated with the subset of the subclasses.

4. The method according to claim 3, wherein: The child classifier determines at least one conditional classification value associated with the child class conditioned on the parent class for the pixel, and determines the child classification value based on the conditional classification value and the parent classification value.

5. The method according to claim 4, wherein: The sub-level classifier determines the sub-classification value by multiplying the parent classification value by the conditional classification value.

6. A method according to any preceding claim, wherein: The parent classification value and the child classification value are values ​​representing the probability that the pixel belongs to the parent class and the child class, respectively, wherein the conditional classification value is a value representing the conditional probability that the pixel belongs to the child class under the condition that the pixel belongs to the parent class.

7. A method according to any preceding claim, comprising: A plurality of pixel feature vectors are generated by a feature extractor based on the pixel and a set of pixels related to the pixel, each pixel feature vector representing one of the pixels, wherein the pixel is classified based on the pixel feature vector of the pixel.

8. The method according to claim 7, wherein: The set of correlated pixels is formed by pixels in a portion of the image surrounding the pixel.

9. A method according to any preceding claim, wherein: The sub-level classifier determines a sub-classification value for each of the subset of sub-classes.

10. A method according to any preceding claim, wherein: The parent classifier and the child classifier are probabilistic classifiers, and the parent classifier and the child classifier determine probability distributions on the parent class set and a subset of the child classes, respectively.

11. A method according to any preceding claim, wherein: Each of the pixels is classified by at least one of: a first child classifier for a first subset of child classes associated with a first class in the parent class, and a second child classifier for a second subset of child classes associated with a second class in the parent class.

12. A method according to any preceding claim, wherein: Each of at least some of the pixels is classified by the parent classifier, the child classifier, and at least one other child classifier from a set of other subclasses, the other child classifier being a child of the child classifier because each of the set of other subclasses is a child of a subclass within a subset of the subclasses within the hierarchical classification scheme.

13. A method according to any preceding claim, wherein: Each pixel of at least some of the pixels is classified by the parent classifier, the child classifier and at least one other parent classifier about a set of other parent classes, wherein the other parent classifier is a parent of the parent classifier because each parent class of the parent class is a child of a parent class in the set of other parent classes.

14. A hierarchical image segmentation system comprising: an input layer configured to receive an image formed by a plurality of pixels; and a plurality of classifiers configured to classify the pixels according to a hierarchical classification scheme, wherein at least some of the pixels are classified by a parent classifier of the plurality of classifiers with respect to a set of parent classes, each of the parent classes being associated with a subset of child classes, and each of the pixels is further classified by at least one child classifier of the plurality of classifiers with respect to one of the subset of child classes, wherein each of the parent classes corresponds to a class of visible structures, and each of the subset of child classes associated with each of the parent classes corresponds to a different type of visible structure within the class; and A controller configured to perform a decision process based on the pixel classification; the decision process includes detecting a failure of the child classifier to classify at least one of the pixels, and making a decision by the parent classifier based on the classification of the pixels.

15. A computer program comprising computer-readable instructions stored on a non-temporary storage medium and configured to, when executed on one or more processors, cause the one or more processors to implement the method of any one of claims 1-13, wherein the classifier is embodied in the computer-readable instructions for execution on the one or more processors.

Images