Electronic device, system and method for determining a semantic grid of a vehicle environment

Abstract

The invention relates to an electronic device (1) for determining a semantic grid of an environment of a vehicle. The electronic device is configured to: receive first image data from an optical sensor, the first image data comprising a 2D image of the environment; perform semantic segmentation of the 2D image and project the resulting semantic image into at least one predetermined semantic plane; receive an occupancy grid representing an allocentric bird's-eye view of the environment, wherein the control device further comprises: a neural network configured to determine the semantic grid by fusing the occupancy grid with at least one predetermined semantic plane. The invention further relates to systems and methods.

Description

Technical Field

[0001] The present disclosure relates to electronic devices, systems and methods for determining a semantic mesh of a vehicle's environment, particularly based on 2D image data from a monocular camera. Background Art

[0002] Modern vehicles include object detection algorithms for collision warning or avoidance and other active safety applications. In particular, autonomous vehicles need to accurately perceive and understand their surroundings in order to navigate safely and successfully. The limited capabilities of sensors, obstructions, complexity and uncertainty of the environment make this a challenging task.

[0003] The use of several types of sensors as well as hybrid active and passive sensors have been proposed. Introducing architectures with active sensors (lidar, radar, ...) and passive sensors (cameras, inertial sensors) can improve the perception of the scene.

[0004] For example, passive sensors such as cameras are cheap and provide dense and rich appearance information. However, the image information they provide is very sensitive to environmental changes, and current computer vision algorithms suffer from performance degradation when processing this image information.

[0005] On the other hand, active sensors such as lasers provide sparse depth and heading measurements. Point clouds generated by lasers or images recorded by cameras have been widely used to detect generic objects in urban environments.

[0006] Deep learning is arguably the dominant approach for autonomous and assisted systems in smart vehicles. The high capacity of deep networks, as adopted by most major industrial players in this field, allows them to create high-level semantic predictions from low-level sensor data (RGB images from monocular cameras, 3D images from stereo cameras, LIDAR, etc.). The downside is that handling uncertainty in a principled way with deep models alone remains a daunting task. On the other hand, Bayesian techniques have a built-in ability to manage uncertainty due to a long history of sensor fusion applications.

[0007] A widely used approach to understanding the surrounding environment is to construct an occupancy grid. An occupancy grid is a spatial 2D map of the environment that can also model areas containing moving objects, see for example:

[0008] L. Rummelhard, A. Negre, and C. Laugier, “Conditional monte carlo dense occupancy tracker,” in ITSC. IEEE, 2015, pp. 2485–2490.

[0009] The cells of these grids contain the probabilities of the cell states. One of the advantages of occupancy maps is that they are dense and provide information about free space, which is important for vehicles and mobile robots. Another advantage is that they do not depend on the type of sensor. Due to the generative nature of the model, different types of sensors can be easily integrated by adapting the observation model. In contrast to discriminative methods, no retraining is required.

[0010] However, while these models provide an accurate picture of the scene with possible obstacles, they do not provide the semantics of individual units which is important for decision making.

[0011] Furthermore, various studies have been conducted in the field of semantic segmentation of RGB images.

[0012] For example, it is proposed to improve the perception capabilities of intelligent vehicles in complex urban environments by using meta-knowledge obtained from constructed semantic context images applied on an evidence grid constructed from stereo vision, see:

[0013] Bernades et al., “Stereo Vision for Dynamic Urban Environment Perception Using Semantic Context in Evidential Grid,” IEEE 18th International Conference on Intelligent Transportation Systems, 2015.

[0014] In addition, WO2017214595 (A1) discloses a system and method for performing three-dimensional semantic parsing of indoor space.

[0015] A method and a system for computationally performing a scene analysis of a semantic traffic space based on an adaptive space-time ray method are known from JP2016115334 (A).

[0016] CN 107066507(A) relates to a semantic graph construction method based on a cloud robot hybrid cloud architecture. Summary of the invention

[0017] Currently, it remains desirable to provide electronic devices, systems, and methods for determining a semantic grid of a vehicle's environment that represents an occupancy grid containing semantic information.

[0018] Therefore, according to an embodiment of the present disclosure, an electronic device for determining a semantic grid of an environment of a vehicle (ie, a semantic grid representing the environment) is provided.

[0019] The electronic device is configured to:

[0020] - receiving first image data of an optical sensor, the first image data comprising a 2D image of an environment,

[0021] - performing semantic segmentation of the 2D image and projecting the resulting semantic image into at least one predetermined semantic plane,

[0022] - receiving an occupancy grid representing an allocentric bird's-eye view of the environment, wherein

[0023] The electronic device further comprises:

[0024] - A neural network configured to determine a semantic grid by fusing the occupancy grid with at least one predetermined semantic plane.

[0025] By providing the electronic device, an electronic device can be obtained that determines (or estimates) an occupancy grid containing detailed semantic information (i.e., a semantic grid according to the present disclosure). Semantic features can include categories such as, for example, roads, cars, pedestrians, sidewalks, buildings, cars, vegetation, etc. To this end, information from multiple sensors (lidar, radar, odometer, and monocular RGB camera) can be utilized and fused. Ideally, the following hybrid approach can be utilized: i) high-capacity deep neural networks, and ii) Bayesian filtering, which can model uncertainty in a unique way.

[0026] The relationship between the segmented image and the semantic plane can be determined based on the projective geometry.

[0027] The semantic plane is desirably predetermined in the sense that its geometric properties (ie its position and / or orientation in space) are predetermined. The "content" of the semantic plane is desirably the projected segmented 2D image (ie making the predetermined plane a "semantic" plane).

[0028] The semantic image may be projected to the semantic plane by transforming the coordinate system of the semantic image so that it matches the coordinate system of the occupancy grid.

[0029] Therefore, the projected semantic image and occupancy grid can have compatible spatial coordinates.

[0030] For example, the at least one predetermined semantic plane may be parallel to the allocentric bird's eye view and / or parallel to the ground plane at a predetermined distance from the ground plane.

[0031] At least one predetermined semantic plane may overlap with the occupancy grid in the allocentric bird's-eye view.

[0032] Therefore, the projected semantic image and the occupancy grid can cover the same area of ​​the environment.

[0033] The resulting semantic image may be projected into a plurality of predetermined parallel semantic planes, each predetermined parallel semantic plane having a different predetermined distance from the ground plane. For example, the plurality of parallel semantic planes may form a layered 3D image.

[0034] 2D images can be segmented by assigning a semantic label to each image pixel. In this way, segments can be generated in the image where the pixels in each segment have the same label.

[0035] At least one semantic plane may comprise a plurality of (semantic) plane cells, each plane cell comprising a semantic label. In other words, since a 2D image may comprise a plurality of cells, a semantic plane (ie a projected segmented 2D image) may also comprise (semantic) cells.

[0036] Each of the semantic planes may comprise a plurality of (semantic) plane units, wherein each pixel of the semantic image may be assigned to a plane unit in at least one of the semantic planes.

[0037] The occupancy grid may include a plurality of occupancy grid cells, each occupancy grid cell indicating an occupancy state, such as an idle cell, a statically occupied cell, a dynamically occupied cell, and an unknown cell.

[0038] Hence, the received occupancy grid may be a non-semantic occupancy grid.The received occupancy grid may be estimated based on a 3D point cloud of the environment processed by a Bayesian particle filter.

[0039] A 3D point cloud may be generated by an active sensor (eg, LiDAR) that includes multiple scan areas distributed over the environment.

[0040] The neural network may be further configured to fuse the occupancy grid with the plurality of predetermined semantic planes by estimating at least one of the planar cells in the semantic plane that matches the occupancy grid cells.

[0041] The neural network can be further configured as:

[0042] fusing the occupancy grid with a predetermined semantic plane by selecting a plane cell of the semantic plane based on the grid states of the overlapping occupancy grid cells, and

[0043] The overlapping occupied grid cells are assigned the semantic labels of the selected plane cells.

[0044] As described above, the projected semantic image (i.e., the semantic plane) and the spatial coordinates of the occupancy grid can have compatible spatial coordinates. The aim is to train a learned mapping that integrates the two representations into a segmented occupancy grid (i.e., the semantic plane). The basic assumption is that objects with height h < D are visible in the 2D image and are within the range of the occupancy grid. Then, if the distance between the semantic planes is small enough, at least one of the points in these planes will have the correct label for the object, and the learned mapping can obtain the integration with the occupancy grid.

[0045] The electronic device may further include:

[0046] A further neural network configured to perform semantic segmentation of the 2D image.

[0047] The neural network may be a convolutional-deconvolutional geometric fusion network.

[0048] The semantic grid may include a plurality of cells. For example, each cell may be assigned semantic information, which may include, for example, semantic categories and the probability of each category.

[0049] The present disclosure further relates to a system for determining a semantic grid of the environment of a vehicle (in particular for a vehicle or a mobile robot system), comprising:

[0050] - The electronic device as described above,

[0051] - An optical sensor (e.g., a passive optical sensor) configured to generate first image data including a 2D image of the environment, and,

[0052] - An active sensor (desirably a scanning sensor, e.g., LIDAR) configured to generate second data including a 3D point cloud of the environment representing a plurality of scanned regions distributed over the environment (in particular outside the vehicle).

[0053] The present disclosure further relates to a vehicle comprising the system as described above. However, the present disclosure may also relate to any robot system comprising the system as described above.

[0054] The optical sensor may be a monocular camera or a set of monocular cameras. In the latter case, a set of monocular cameras may, for example, jointly produce a panoramic image of the environment.

[0055] The system may further include an odometry device configured to measure the movement of the vehicle and / or generate odometry data of the vehicle.

[0056] Finally, the present disclosure also relates to a method for determining a semantic grid of the environment of a vehicle. The method includes the following steps:

[0057] - receiving first image data of the optical sensor, the first image data comprising a 2D image of the environment,

[0058] - performing semantic segmentation of the 2D image and projecting the resulting semantic image into at least one predetermined semantic plane,

[0059] - receiving an occupancy grid representing an allocentric bird's-eye view of an environment, wherein,

[0060] - The neural network determines a semantic grid by fusing the occupancy grid with at least one predetermined semantic plane.

[0061] The method may include further method steps corresponding to the functionality of the electronic device as described above.

[0062] The described method is applicable to any kind of robotic systems, not only vehicles.

[0063] Unless described to the contrary, it is intended that the elements described above and within the specification be combined.

[0064] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

[0065] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A block diagram showing a system having an electronic device according to an embodiment of the present disclosure;

[0067] Figure 2 A schematic diagram showing a bird's eye view sensed by a system diagram according to an embodiment of the present disclosure;

[0068] Figure 3 A schematic flow chart showing steps of a method for determining a semantic grid according to an embodiment of the present disclosure; and

[0069] Figure 4 Schematically illustrates the alignment of a projection view with a bird's eye view (only three planes are shown) according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0070] Reference will now be made in detail to exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Throughout the drawings, the same reference numerals will be used wherever possible to refer to the same or like parts.

[0071] Figure 1A block diagram of a system 10 having an electronic device 1 according to an embodiment of the present disclosure is shown. The system may have various functions, for example, may be a robotic system. For example, the system 10 may be movable, for example, having drivable wheels, and the system 10 may have a device for acquiring an object, for example, at least one gripper. The system 10 may further be integrated in a vehicle 100 or constitute a vehicle 100.

[0072] The electronic device 1 executes a computer vision algorithm for determining a semantic mesh of the system's environment. Desirably, this step includes detecting the presence and location of an object in the sensed scene and determining the semantic category of the object, i.e., whether the object is a road, a car, a pedestrian, a sidewalk, a building, a car, vegetation, etc. In addition, when, for example, vehicles and other objects are detected, the application software can use the object detection information to provide warnings or take appropriate measures.

[0073] The electronic device 1 may additionally perform other functions in the system 10 and / or the vehicle 100. For example, the electronic device may also be used as a general ECU (electronic control unit) for the system. The electronic device 1 may include electronic circuits, processors (shared, dedicated, or grouped), combinatorial logic circuits, memories for executing one or more software programs, and / or other suitable components that provide the described functionality. In other words, the device 1 may be a computer device. The device 1 may be connected to a memory 2, which may store data, such as a computer program, which, when executed, performs a method according to the present disclosure.

[0074] The electronic device 1 is further connected to a (passive) optical sensor 3, in particular a digital camera. The digital camera 3 is configured so that it can record the scene in front of the vehicle 100 and in particular output digital data providing information about the appearance (color, e.g. RGB) of the scene. The camera 3 is desirably a monocular camera configured to generate first image data comprising a 2D image of the environment. A set of monocular cameras generating a panoramic 2D image may also be provided.

[0075] The electronic device 1 is further connected to an active sensor 4, in particular a LiDAR sensor, a laser sensor, an ultrasonic sensor or a radar sensor. The active sensor may be a radar or an active optical sensor, in particular a laser sensor and / or a LiDAR sensor. The active sensor may be configured to generate second data, the second data comprising a 3D point cloud representing the environment of a plurality of scan areas distributed over the environment.

[0076] The outputs of the (passive) sensors 3 and active sensors 4 are sent to the electronic device 1. Desirably, the outputs are sent instantaneously (ie in real time or quasi real time). Thus, the electronic device can also identify the sensed object in real time or quasi real time.

[0077] The system 10 may additionally be connected to an external server 20, so as to form together a system 30. The server 20 may be used to provide and eventually update the algorithms executed by the electronic device 1. The device 1 may be connected to the server. For example, the electronic device 1 may be connected to the server 20 via a wireless connection. Alternatively or additionally, the electronic device 1 may be connected to the server 20 via a fixed connection (e.g., via a cable).

[0078] Figure 2 A schematic scenario of a bird's eye view sensed by a system diagram according to an embodiment of the present disclosure is shown.

[0079] Vehicle 100 scans the scene in front of it using a digital camera 3 having a field of view 13 and a LiDAR sensor 4 having a field of view 14 .

[0080] Camera sensors can detect objects such as pedestrians, cars (as target vehicle 101), traffic lights, traffic signs, etc. Digital cameras can provide relatively dense data (pixel space), the color and appearance of objects. This information is needed to understand the scene around the vehicle, that is, to generate semantic information. However, this sensor does not provide reliable depth information and is only a 2D sensor.

[0081] LiDAR sensors can detect objects and provide distance measurements to these objects. LiDAR sensors are desirable because they are able to provide depth information, i.e., a 3D point cloud of the environment representing an occupancy grid. The occupancy grid is particularly useful for indicating which (3D) areas of the environment are occupied by objects. However, the data generated by the LiDAR sensor is less useful for obtaining semantic information about objects in the environment. A separate scan point 24 can be provided for each 1 / 2° in the entire field of view of the sensor. For example, an object 101 in front of the main vehicle 100 can be detected. Therefore, multiple scan points 24 are returned, which identify the distance of the object 101 from the main vehicle 100. However, the detection lacks any semantic information, i.e., in this example, the detected object 101 is a vehicle.

[0082] Figure 3 A schematic flow chart showing steps of a method for determining a semantic grid according to an embodiment of the present disclosure.

[0083] In brief, the method fuses the 3D point cloud input l from LIDAR (step S6) with the 2D image input x from the RGB camera (step S1) to obtain a semantic mesh y from a system-centric bird’s-eye view (step S11).

[0084] In more detail, the LIDAR input l (step S6) is geometrically and temporally integrated with a Bayesian particle filter (step S8) to produce an occupancy grid o (step S9). This grid has the following advantages: it maintains a 360-degree view of the environment and encodes the complete LIDAR observation history through temporal integration. However, the semantic information of the grid is low because the state of each cell (e.g., empty; occupied; in motion) encodes possible values ​​but does not contain fine-grained information about the semantic category of the occupied cell. This information can be obtained from the output s of the deep network that takes the RGB image x as input (step S3). However, these two representations are of different types (3D projected geometry vs. bird's-eye view) and need to be fused.

[0085] In the following subsections, the different parts of the method are described: i) semantic segmentation of the RGB input image (steps S1 to S5), ii) occupancy grid estimation from LIDAR input (steps S6 to S9); and iii) learning the fusion of these two representations with a deep neural network (steps S10, S11). Note that Figure 3 The steps shown in may represent operation steps (eg, segmentation of an RGB image) or steps of providing data (eg, an RGB image).

[0086] A. Semantic Segmentation of Monocular RGB Input

[0087] A deep neural network (step 2) takes a monocular RGB input x (step S1) and predicts an estimate of the segmentation s (step S3). High-capacity neural networks are able to estimate high-level semantic information from low-level input data when trained on a large enough database. It is expected to follow standard practices: the model is first pre-trained to classify on a large predefined dataset and then fine-tuned on the target dataset for segmentation.

[0088] The main challenge in using deep networks for image segmentation is to preserve resolution, i.e., to provide segmentations at the same spatial resolution as the input. The difficulty lies in the fact that internally, convolutional neural networks perform downsampling and pooling between consecutive layers to increase the spatial support of the convolutional filters in subsequent layers.

[0089] It is expected that semantic information s can be obtained using a SegNet variant, see for example:

[0090] V. Badrinarayanan, A. Kendall, and R. Cipolla, "SegNet: A Deep Convolutional Encoder / Decoder Architecture for Image Segmentation," in IEEE PAMI, vol. 39, no. 12, pp. 2481–2495, 2017.

[0091] SegNet’s accuracy is not the highest among other reported results, but its runtime / accuracy tradeoff is very favorable. SegNet is an encoder-decoder network, and its details are given in the following sections.

[0092] We use parameters from a previously trained version of the VGG16 architecture trained for object recognition. Pixels are classified using a soft-max layer. Labels are, for example, road, car, sidewalk, vegetation, pedestrian, bicycle, building, sign, fence, and unknown.

[0093] B. Bayesian Occupancy Grid

[0094] The Bayesian particle filter (see step S8) processes the LIDAR data (see step S6) and the odometry information (see step S7) from the vehicle motion to reliably estimate the allocentric bird's-eye view in the form of an occupancy grid. The occupancy grid desirably contains a 360-degree view of the car's surroundings and integrates information from the observation history through temporal filtering; however, the occupancy grid does not include fine-grained semantic categories.

[0095] In particular, Bayesian filtering methods are used to predict the dynamic state and occupancy of cells in an occupancy grid map. This is achieved by deriving the occupancy probability given the sensor measurements of each cell and the previous state of the cell. The probabilities can be obtained using CMCDOT (Conditional Monte Carlo Dense Occupancy Tracker), which is a Bayesian filtering method used to infer the probability of dynamic and grid occupancy in recent Bayesian filtering methods, see for example:

[0096] Rummelhard et al., “Conditional Monte Carlo Dense Occupancy Tracker,” in ITSC.IEEE, 2015, pp. 2485–2490.

[0097] First, CMCDOT introduced idle states, static occupied states, dynamic occupied states, and unknown states, instead of defining each cell as occupied or not. The idle state represents the probability that there is no object of any type in the cell. However, it should be noted that it does not necessarily mean an area that a vehicle can navigate to. For example, a sidewalk has a high probability of being idle, but driving to the idle area is not allowed due to the consequences of driving to the idle area. A statically occupied cell refers to the probability that the cell is occupied by an obstacle. Although an obstacle may be part of a permanent static structure such as a building, it may also be part of a temporary static object (such as a parked car). A dynamically occupied cell shows the probability that the cell is occupied by a dynamic object. This state also includes information about the velocity of the cell contained in the particles in the cell. Only the particles in the dynamic cell area have information related to velocity, which significantly reduces the computational complexity.

[0098] The first step of the Bayesian filter is to predict the current state probability from the previous state probability. To achieve this, transition probabilities are defined. They represent the probability that the state at the previous time will transform into another state at the current time. For example, a statically occupied cell will remain a statically occupied cell with a transition probability of 0:99, or will become a dynamically occupied cell with a transition probability of 0:01. The transition probabilities also include speed information.

[0099] In the next step, the updated probabilities are evaluated. Probabilistic sensor models can be used for observation models, see for example:

[0100] In S. Thrun, W. Burgard, and D. Fox, Probabilistic Robotics, MIT Press, 2005. It should be noted that due to the flexibility of the observation model, the Bayesian filter can be used with a variety of sensors, which is one of its main advantages. In this exemplary embodiment, for example, LIDAR sensor data I is used.

[0101] After the evaluation step, the state distribution is estimated. Applying particle resampling assigns new particles to new dynamically occupied regions and focuses on important areas. After particle resampling, the iteration continues with the prediction step if necessary. Although the occupancy and dynamic information about the cell state can be accurately obtained via the Bayesian filtering method, this cannot always be used to plan the next action of the vehicle. As mentioned above, the same state can refer to different semantic categories. For example, an idle cell can be a road or a sidewalk, which will lead to different behaviors. Therefore, semantic information is adopted to enrich the occupancy grid.

[0102] Four occupancy grid states o (see step S9) are used as input to the network to estimate the semantic grid. The four states of the occupancy grid represent, for example, free cells, statically occupied cells, dynamically occupied cells, and unknown cells.

[0103] C. Fusion of geometric and semantic information

[0104] Fusing the geometric information from the occupancy grid o obtained from the LIDAR input l (see step S9) with the semantic information s obtained from the projected RGB input x (see step S3) is an important problem that can be solved by direct geometric computation only if depth information is available together with the RGB input: a point in the projected image x or s may potentially correspond to several 3D points in the scene, and thus to several points in the grid o. Stereo depth may provide sufficient constraints. An alternative solution is proposed, in which the fusion process is learned, avoiding the computation of a disparity map from stereo.

[0105] Directly learning the fusion in a black-box strategy by providing both representations as input to a neural network would be suboptimal. The network would need to learn multiple mappings, namely (i) the geometric transformation between the projected and bird's-eye views, (ii) resolving point-to-line correspondences due to the lack of depth information, and (iii) fusing the occupancy grid state with the semantic categories from the segmentation. While (ii) and (iii) are unavoidable in this approach, (i) can be solved directly (up to point-to-line correspondences).

[0106] For this reason, an intermediate representation p is introduced (see step S5) that transforms the projected input s into a bird's-eye view representation compatible with the coordinate system of the occupancy grid o, as Figure 4 shown.

[0107] Figure 4 Schematically illustrates the alignment of a projection view with a bird's eye view (only three planes are shown) according to an embodiment of the present disclosure.

[0108] Each cell in p takes the value of a letter of the semantic category, i.e., the same letter used to segment the image s. The goal is to create a representation where a cell with a given spatial coordinate (x; y) corresponds to a grid cell with the same coordinates in o. In order to model the ambiguity caused by the lack of depth information, p = {pi} is a hierarchical 3D graph organized as a collection of D planes pi parallel to the ground. Here, i indexes different heights. Next, the time index t has been removed, since the fusion is calculated separately for each time instant t.

[0109] The distance between the ground and each plane pi is di. Assume that the D planes are at distance δd from each other, so the distance between the i-th plane and the ground is iδd. If the camera is calibrated, the relationship between the image and the plane will be direct and can be derived from projective geometry. For each plane in plane pi, for any point on that plane, first find the coordinates in the image plane by using the coordinate conversion from the plane to the image

[0110]

[0111] Transformation Find the coordinates of the point

[0112] The corresponding pixel in the image plane can be found as

[0113]

[0114] where K is the camera calibration matrix. and are the pixel coordinates of the projection of the j-th point in the i-th plane. For a given pixel in the segmented image s, its semantic label is assigned to a set of points in p, each corresponding to a different depth value (and thus a different height value).

[0115] As described above, the spatial coordinates of representations p and o are compatible. The aim is to train a learned mapping that integrates the two representations into the segmented occupancy grid o. The basic assumption is that objects with height h < D are visible in the RGB image and are within the range of the occupancy grid. Then, if δd is small enough, at least one point in the projection plane p will have the correct object label, and the learned mapping can achieve integration with the occupancy grid.

[0116] D. Joint Dimensionality Reduction and Fusion

[0117] Both the occupancy grid o and the intermediate representation p are fed into a deep neural network (see step S10), which outputs a semantic grid y (see step S11). The semantic grid is a 3D tensor of the same spatial dimensions as the input tensors o and p. Thus, the model needs to maintain the resolution, which would be a non-trivial task if any classification resolution-reducing pooling were used in the intermediate layers of the network. Similarly to what is achieved for semantic segmentation, a convolutional-deconvolutional network from the SegNet variant can be used, which includes skip connections introduced initially, such as the following:

[0118] In IEEE PAMI, 2017, Vol. 39, No. 12, pp. 2481 - 2495, V. Badrinarayanan, A. Kendall, and R. Cipolla, "SegNet: A Deep Convolutional Encoder / Decoder Architecture for Image Segmentation".

[0119] The input data consists of D + 4 planes: 4 planes corresponding to the occupancy grid o and D planes corresponding to the intermediate representation p calculated from the segmented RGB image. The latter is categorical data, as each value in the alphabet corresponds to a semantic class label. By definition, the data is unordered, which makes it inefficient to learn directly from it, as the learned mapping would need to be highly (and unnecessarily) complex. Thus, the number of data planes can be increased by encoding each of the D planes from p in a 1-in-K encoding ("hot-one"). This will create well-behaved data, but the drawback is a significant increase in the size of the input representation: the input tensor now has DxC + 4 planes, where C is the number of semantic classes.

[0120] Directly learning this representation is computationally inefficient and leads to unreasonably large networks. A dimensionality reduction layer can be added at the input of the encoder, which as a first operation reduces the input planes to a lower number. The number of planes is a hyperparameter which is set equal to C. It is implemented as a 1x1 convolution which has the same effect as a point-wise nonlinearity with spatially shared parameters. Dimensionality reduction and fusion are trained jointly end-to-end.

[0121] The rest of the encoder part has, for example, 13 convolutional layers similar to the famous VGG16 network. Each encoder layer applies convolution and obtains a set of features. They are then batch normalized and an element-wise rectified linear nonlinear (ReLU) is performed. Maxpooling is used to reduce the size of the image. The index of the maximum value is retained and passed to the decoder process, which speeds up the decoding process. The decoder has, for example, 13 layers. Convolutions are performed again with batch normalization and ReLU. To upsample the image, the indices stored in the encoder layer are used. A multi-class softmax classifier is applied in the last layer to estimate the classification probability of the pixels. A cross entropy loss is used and the loss is taken as the sum of all pixels in a batch.

[0122] Throughout the specification including the claims, the term "comprising a" shall be understood as being synonymous with "comprising at least one" unless otherwise stated. In addition, any ranges set forth in the specification including the claims shall be understood to be inclusive of their end values ​​unless otherwise stated. Specific values ​​of the elements described shall be understood to be within acceptable manufacturing or industrial tolerances known to those skilled in the art, and any use of the terms "substantially" and / or "approximately" and / or "typically" shall be understood to be within acceptable tolerances.

[0123] Although the present disclosure has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the present disclosure.

[0124] It is intended that the specification and examples be considered as exemplary only, with the true scope of the disclosure being indicated by the following claims.

Claims

1. An electronic device for determining a semantic grid of an environment of a vehicle, The electronic device is configured as follows: receiving first image data from an optical sensor, the first image data comprising a 2D image of the environment, performing semantic segmentation of the 2D image and projecting the resulting semantic image into at least one predetermined semantic plane, receiving an occupancy grid representing an allocentric bird's-eye view of the environment, wherein The electronic device further comprises: A neural network configured to determine a semantic grid by fusing the occupancy grid with the at least one predetermined semantic plane, wherein The at least one predetermined semantic plane is parallel to the allocentric bird's eye view and / or parallel to the ground plane at a predetermined distance from the ground plane, and / or The at least one predetermined semantic plane overlaps with the occupancy grid in the allocentric bird's eye view, and wherein, The resulting semantic image is projected into a plurality of predetermined parallel semantic planes, each parallel semantic plane having a different predetermined distance from the ground plane.

2. The electronic device according to claim 1, wherein: The semantic image is projected onto the semantic plane by transforming the coordinate system of the semantic image so that it matches the coordinate system of the occupancy grid.

3. The electronic device according to claim 1 or 2, wherein: The 2D image is segmented by assigning a semantic label to each image pixel.

4. The electronic device according to claim 1 or 2, wherein: The at least one semantic plane includes a plurality of plane units, each plane unit includes a semantic label, and / or Each of the semantic planes includes a plurality of plane units, wherein each pixel of the semantic image is assigned to a plane unit in at least one of the semantic planes.

5. The electronic device according to claim 1 or 2, wherein: The occupancy grid includes a plurality of occupancy grid cells, each occupancy grid cell indicating an occupancy state.

6. The electronic device according to claim 5, wherein: The neural network is configured as: The occupancy grid is fused with the plurality of predetermined semantic planes by estimating at least one of the planar cells in the semantic plane that matches the occupancy grid cells.

7. The electronic device according to claim 6, wherein: The neural network is configured as: fusing the occupancy grid with the predetermined semantic plane by selecting a plane element of the semantic plane based on the occupancy status of overlapping occupancy grid elements; and The overlapping occupied grid cells are assigned the semantic labels of the selected plane cells.

8. The electronic device according to claim 1 or 2, wherein: The electronic device further comprises: A further neural network is configured to perform semantic segmentation of the 2D image.

9. The electronic device according to claim 1 or 2, wherein: The neural network is a convolution-deconvolution geometric fusion network.

10. The electronic device according to claim 1 or 2, wherein: The semantic grid includes a plurality of cells, each of which is assigned semantic information, the semantic information specifically including a semantic category and a probability of each category.

11. A system for determining a semantic grid of an environment of a vehicle, comprising: The electronic device according to any one of claims 1 to 10, an optical sensor configured to generate first image data comprising a 2D image of the environment, and An active sensor is configured to generate second data comprising a 3D point cloud of the environment representing a plurality of scan areas distributed over the environment.

12. The system according to claim 11, wherein: The optical sensor is a monocular camera or a group of monocular cameras.

13. A method for determining a semantic grid of an environment of a vehicle, comprising the steps of: receiving first image data from an optical sensor, the first image data comprising a 2D image of the environment, performing semantic segmentation of the 2D image and projecting the resulting semantic image into at least one predetermined semantic plane, receiving an occupancy grid representing an allocentric bird's-eye view of the environment, wherein The neural network determines a semantic grid by fusing the occupancy grid with the at least one predetermined semantic plane, wherein The at least one predetermined semantic plane is parallel to the allocentric bird's eye view and / or parallel to the ground plane at a predetermined distance from the ground plane, and / or The at least one predetermined semantic plane overlaps with the occupancy grid in the allocentric bird's eye view, and wherein, The resulting semantic image is projected into a plurality of predetermined parallel semantic planes, each parallel semantic plane having a different predetermined distance from the ground plane.

Images