End-to-end parametric road layout prediction with inexpensive supervision

Abstract

A computer-implemented method for predicting road layouts is provided. The method includes, by a first processor-based element, segmenting an RGB image to output pixel-level semantic segmentation results of the RGB image in the perspective view for both visible and occluded pixels in the perspective view based on contextual cues (510). The method further includes, by a second processor-based element, learning a mapping from the pixel-level semantic segmentation results for the perspective view RGB image to a top view of the RGB image using a road plane assumption (520). The method also includes, by a third processor-based element, generating an occlusion-aware parametric road layout prediction for road layout-related attributes in the top view (530).

Description

[Technical field]

[0001] Related Application Information This application claims priority to U.S. Patent Application No. 17 / 521,193, filed November 8, 2021, U.S. Provisional Patent Application No. 63 / 111,677, filed November 10, 2020, and U.S. Provisional Patent Application No. 63 / 113,945, filed November 15, 2020, which are incorporated by reference in their entireties herein. [Background technology]

[0002] The present invention relates to artificial intelligence, and more particularly to end-to-end parametric road layout prediction with low-cost supervision. 2. Description of Related Art

[0003] In real-world applications such as autonomous driving and path planning, understanding road layouts in a top-view with viewpoint input is crucial. Recent studies have proposed to take RGB images as input and provide pixel-level semantic predictions in top-view. However, such methods usually require pixel-level annotations, which can be very expensive. Also, their representations are not compact and lack the ability to infer occlusion relations. A top-view representation can be more beneficial when occlusion relations are desirable, for example, when two objects cannot occupy the same position in the top view but may occlude each other in the perspective view. Therefore, improved methods for understanding road layouts in top-view are desirable. Summary of the Invention

[0004] According to an aspect of the invention, there is provided a computer-implemented method for road layout prediction, the method including segmenting, by a first processor-based element, an RGB image in a perspective view for both visible and occluded pixels in the perspective view based on contextual cues to output pixel-level semantic segmentation results of the RGB image. The method also includes learning, by a second processor-based element, a mapping of the RGB image to a top view from the pixel-level semantic segmentation results for the RGB image of the perspective view using a road plane assumption. The method further includes generating, by a third processor-based element, an occlusion-aware parametric road layout prediction for road-layout related attributes in the top view.

[0005] According to another aspect of the present invention, a computer program product for road layout prediction is provided. The computer program product includes a non-transitory computer-readable storage medium having program instructions embodied therein. The program instructions are executable by the computer to cause the computer to perform a method. The method includes segmenting, by a first processor-based element of the computer, an RGB image in a perspective view for both visible and occluded pixels in the perspective view based on contextual cues to output a pixel-level semantic segmentation result of the RGB image in the perspective view. The method also includes learning, by a second processor-based element of the computer, a mapping of the RGB image to a top view from the pixel-level semantic segmentation result for the RGB image of the perspective view using a road plane assumption. The method further includes generating, by a third processor-based element of the computer, an occlusion-aware parametric road layout prediction for road-layout-related attributes in the top view.

[0006] According to yet another aspect of the present invention, there is provided a computer processing system for road layout prediction, the computer processing system including a memory device for storing program code, and a processor device for executing program code for segmenting an RGB image in a perspective view to output pixel-level semantic segmentation results of the RGB image for both visible and occluded pixels in the perspective view based on contextual cues, the processor device also executing program code for learning a mapping of the RGB image to a top view from the pixel-level semantic segmentation results for the RGB image of the perspective view using a road plane assumption, and the processor device further executing program code for generating an occlusion-aware parametric road layout prediction for road-layout related attributes in the top view.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. [Brief description of the drawings]

[0008] The present disclosure provides details in the following description of preferred embodiments with reference to the following figures.

[0009] [Figure 1] FIG. 2 is a block diagram illustrating an exemplary computing device in accordance with an embodiment of the present invention.

[0010] [Diagram 2] FIG. 2 is a block diagram illustrating an exemplary end-to-end model, in accordance with an embodiment of the present invention.

[0011] [Diagram 3] 1 is a block diagram illustrating an exemplary system according to an embodiment of the present invention.

[0012] [Figure 4]1 is a block diagram illustrating an exemplary environment in which the present invention may be applied, in accordance with an embodiment of the present invention.

[0013] [Diagram 5] 1 is a flow diagram illustrating an example method for road layout prediction, according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Embodiments of the present invention are directed to end-to-end parametric road layout prediction with low-cost supervision.

[0015] The present invention targets end-to-end trainable top-view layout estimation with perspective images as input. There are two major challenges in the problem. First, the traditional approach, direct RGB method, does not perform well on distance-related attributes and occluded regions. Second, existing SOTA methods have strong assumptions such as the entire video sequence being given, and strong supervision requirements such as pixel-wise semantic annotations and dense depth supervision. Therefore, the present invention proposes to tackle these two above problems with a novel model that can effectively leverage limited human annotations only in parametric form in the top view and output meaningful intermediate representations such as pixel-level occlusion-aware semantic segmentation in the perspective view and semantic in the top view.

[0016] An embodiment of the present invention first proposes an end-to-end model that takes a perspective image as input and outputs a compact parametric prediction of the road layout in the top view. More importantly, the end-to-end model can output pixel-level semantic representations in both perspective and top views without human annotation. In an embodiment, the end-to-end model includes three modules. The first module takes RGB as input and outputs the results of semantic segmentation in the perspective view in the perspective view. The second module inputs the semantics of the first perspective view and learns to map it to the top view. Finally, the third module receives the output from the second module and provides a parametric prediction of road layout related attributes in the top view. The outputs of all three modules correspond to occlusions. We use unitize deep supervision, which injects losses into the first and second modules as well as the last module to guide the training process. Note that our deep supervision is "cheap", meaning that human annotation is only needed at the last step or in a parametric way in the top view. Experimentally, it takes only a few seconds to annotate one image. Interestingly, with such cheap parametric supervision, we are able to obtain a dense pixel-level supervision signal for Module 1 and Module 2. Moreover, since the human annotation focuses only on the road layout, we are also able to effortlessly and automatically recover layout-relevant pixels that are originally occluded by the foreground in the perspective view, i.e. road, crosswalk, sidewalk, lane boundaries.

[0017] 1 is a block diagram illustrating an exemplary computing device 100 configured to perform road layout prediction, in accordance with an embodiment of the present invention.

[0018] Computing device 100 may be embodied as any type of computing or computing device capable of performing the functions described herein, such as, but not limited to, a computer, a server, a rack-based server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and / or a user electronic device. Additionally or alternatively, computing device 100 may be embodied as one or more compute sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically decomposed computing device. As shown in FIG. 1, computing device 100 illustratively includes a processor 110, an input / output subsystem 120, a memory 130, a data storage device 140, and a communication subsystem 150, and / or other components and devices typically found in a server or similar computing device. Of course, computing device 100 may include other or additional components (e.g., various input / output devices) as typically found in a server computer in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated into or otherwise form part of another component. For example, memory 130, or portions thereof, may be incorporated into processor 110 in some embodiments.

[0019] Processor 110 may be embodied as any type of processor capable of performing the functions described herein, and may be embodied as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuitry.

[0020] Memory 130 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, memory 130 may store various data and software used during operation of computing device 100, such as an operating system, applications, programs, libraries, and drivers. Memory 130 is communicatively coupled to processor 610 via I / O subsystem 120 and may be embodied as circuits and / or components to facilitate input / output operations with processor 110, memory 130, and other components of computing device 100. For example, I / O subsystem 120 may be embodied as or may otherwise include a memory controller hub, an input / output control hub, a platform controller hub, integrated control circuits, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems to facilitate input / output operations. In some embodiments, I / O subsystem 120 may form part of a system-on-chip (SOC) and be integrated with processor 110, memory 130, and other components of computing device 100 on a single integrated circuit chip.

[0021] The data storage 140 may be embodied as any type of device or device configured for short-term or long-term storage of data, such as, for example, memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. The data storage 140 may store program code for road layout prediction. The communication subsystem 150 of the computing device 100 may be embodied as any network interface controller or other communication circuitry, device, or collection thereof that may enable communication between the computing device 100 and other remote devices over a network. The communication subsystem 150 may be configured to effectuate such communication using any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., Ethernet, InfiniBand, Bluetooth, Wi-Fi, WiMAX, etc.).

[0022] As shown, computing device 100 may also include one or more peripheral devices 160. Peripheral devices 160 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 160 may include a display, a touch screen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0023] Of course, computing device 100 may include other elements (not shown) or omit certain elements, as would be readily appreciated by one of ordinary skill in the art. For example, various other input and / or output devices may be included in computing device 100 depending on the particular implementation of the same, as would be readily appreciated by one of ordinary skill in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Additionally, additional processors, controllers, memories, and the like may be utilized in a variety of configurations. These and other variations of processing system 100 will be readily contemplated by those of ordinary skill in the art in view of the teachings of the present invention provided herein.

[0024] As employed herein, the term "hardware processor subsystem" or "hardware processor" may refer to a processor, memory (RAM, cache, etc.), software (memory management software, etc.), or combinations thereof, working together to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution units, etc.). The one or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor, or a computing element-based controller (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memories, etc.). In some embodiments, the hardware processor subsystem may include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.), which may be on-board or off-board, or may be dedicated for use by the hardware processor subsystem.

[0025] In some embodiments, the hardware processor subsystem may include and execute one or more software elements, which may include an operating system and / or one or more applications and / or specific code to achieve a particular result.

[0026] In other embodiments, the hardware processor subsystem may include dedicated, dedicated circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application specific integrated circuits (ASICs), FPGAs, and / or PLAs.

[0027] These and other variations of hardware processor subsystems are contemplated in accordance with embodiments of the present invention.

[0028] FIG. 2 is a block diagram illustrating an exemplary end-to-end model 200, in accordance with an embodiment of the present invention.

[0029] Basically, the intermediate outputs are 211 and 240. This is where deep supervision is needed. There are three modules: 210, 260 (215, 230), and 250. Among them, 215 and 230 are combined to transform pixel-level perspective representation into top view for hallucination.

[0030] Thus, the end-to-end model 200 includes three modules. The first module 210, also referred to herein as the strabismus prediction module or strabismus module for short, predicts the semantic class of each pixel, including unseen or occluded regions. The second module 260, formed by modules 215 and 230, also referred to herein as the top-view semantics (TS) module, learns to map the strabismus prediction to the top-view with a planar assumption. The refinement module 230 learns to refine unseen and noisy regions of the initial bevinit (birdseye view initialization) obtained in module 220, outputting the refined bevinit 240. The third module 250, also referred to herein as the parametric predictor module or parametric module for short, provides a parametric representation for the generated top-view map (parametric representation from the refined bevinit). Note that human annotations are only obtained for the third module 250, while the supervision of the first two modules 210 and 260 is obtained automatically. Compared to existing works, the end-to-end model 200 intelligently exploits two intermediate representations without laborious and ambiguous annotations.

[0031] Therefore, we first propose an end-to-end model that takes a perspective image as input and outputs a compact parametric prediction of the road layout in the top view. More importantly, our end-to-end model is able to output pixel-level semantic representations in both perspective and top views without the need for corresponding human annotations.

[0032] As mentioned above, the end-to-end model 200 includes three modules. The first module 210 takes RGB 201 as input and outputs semantic segmentation results in a perspective view. The second module 260 inputs the initial viewpoint semantics and learns to map the initial viewpoint semantics to the top view. Finally, the third module 250 receives the output from the second module 260 and provides parametric predictions of road layout related attributes in the top view. To guide the training process, we propose unitized deep supervision that injects losses not only in the third module 230 but also in the first module 210 and the second module 260. It should be noted that our deep supervision is also inexpensive, which means that human annotation is only needed at the last step or in a parametric form in the top view. Experimentally, it takes only a few seconds to annotate one image. Interestingly, with such cheap parametric supervision, we can obtain dense pixel-level supervision signals for the first module 210 and the second module 220. Moreover, since the human annotation only focuses on the road layout, we can also automatically recover with minimal effort the layout-related pixels that are originally hidden in the foreground in the perspective view, i.e., road, crosswalk, sidewalk, and lane boundaries.

[0033] FIG. 3 is a block diagram illustrating an exemplary system 300 in accordance with an embodiment of the present invention.

[0034] The system 300 receives an RGB image 301. The system 300 includes a perspective module 310 (a first module), a feature transformation module 320 (part of a second module), a hallucination module 330 (part of the second module), and a parametric prediction module 340 (a third module). The feature transformation module 320 and the hallucination module 330 form the TD module 220. The system 300 outputs (i) pixel-level occlusion-aware semantic predictions 351 in perspective views from the perspective module 310, (ii) pixel-level semantics 352 in top views from the hallucination module 330, and (iii) parametric attribute predictions 353 in top views from the parametric prediction module 340. These elements are described in more detail herein.

[0035] FIG. 4 is a block diagram illustrating an exemplary environment 400 in which the present invention may be applied, according to an embodiment of the present invention.

[0036] In the environment 400, a user 488 is located in a scene with multiple objects 799, each with its own position and trajectory. The user 488 is driving a vehicle 472 (e.g., a car, truck, motorcycle, etc.) having an ADAS 477.

[0037] The ADAS477 computes occlusion-aware parametric road layout predictions.

[0038] Vehicle control decisions are made in response to the occlusion-aware parametric road layout prediction, such that the ADAS 477 can control the steering, braking, and acceleration systems, for example, but not limited to, as actions corresponding to the decisions.

[0039] Thus, in an ADAS context, steering, acceleration / braking, friction (or lack thereof), yaw rate, lighting (hazards, flashing high beams, etc.), tire pressure, turn signaling, etc. can all be efficiently utilized in optimization decisions according to the present invention.

[0040] The system of the present invention (e.g., system 400) can interface with the user through one or more systems of the vehicle 472 that the user is operating. For example, the system of the present invention can provide the user information via system 472A of the vehicle 472 (e.g., a display system, a speaker system, and / or some other system). Additionally, the system of the present invention (e.g., system 400) can interface with the vehicle 472 itself (e.g., through one or more systems of the vehicle 472, including but not limited to a steering system, a braking system, an acceleration system, a steering system, a lighting (blinker, headlight) system, etc.) to control the vehicle and cause the vehicle 472 to perform one or more actions. In this manner, the user or the vehicle 472 itself can navigate around these objects 499 and avoid potential collisions therebetween. The provision of information and / or control of the vehicle can be considered actions determined according to embodiments of the present invention.

[0041] FIG. 5 is a flow diagram illustrating an exemplary method 500 for road layout prediction, in accordance with an embodiment of the present invention.

[0042] In block 510, a first module segments the RGB image and outputs pixel-level semantic segmentation results of the RGB image in the perspective view for both visible and occluded pixels in the perspective view based on contextual cues.

[0043] In block 520, a second module learns a mapping from pixel-level semantic segmentation results for the perspective RGB image to the top view of the RGB image using the road plane assumption. The pixel-level semantic segmentation results are occlusion inferred to indicate items hidden in the perspective view of the RGB image. The top view of the RGB image is occluded to indicate what is hidden in the perspective view of the RGB image.

[0044] At block 530, a third module generates an occlusion-aware parametric road layout prediction for the top view road layout related attributes. In an embodiment, the occlusion-aware parametric road layout prediction includes predicted boundaries such as, but not limited to, road boundaries, crosswalk boundaries, sidewalk boundaries, lane boundaries, and collidable object (people, other vehicles, buildings, etc.) boundaries, where one or more members of the group are originally occluded by respective foreground elements in the perspective view.

[0045] At block 540, in response to the occlusion-aware parametric road layout prediction, control a vehicle system of the vehicle for accident avoidance. The vehicle system may be any one or more of a braking system, an acceleration system, a stability system, a steering system, etc.

[0046] Next, the framework of the present invention will be described based on an embodiment of the present invention.

[0047] The model contains three modules: the first Perspective Semantics (PS) module 310 inputs an RGB image and outputs occlusion-inferred pixel-level semantics (OSP) in the perspective view; the second Top-View Semantics (TS) module 330 projects the OSP to the top view and learns to hallucinate / complete pixel-level top-view semantics for out-of-view as well as noisy regions, which we call Hallucinated Semantics in Top View (HST); and the last Top-View Parametric Prediction module 340 analyzes the HST and provides predictions on road layout-related attributes in the top view.

[0048] Next, a full model according to an embodiment of the present invention will be described.

[0049] Here, we focus on the model structure and assume that each module has supervision. It is emphasized that only parametric annotations are required. We assume that there is detailed pixel-level supervision / annotation, which is actually not human annotation but the generative process of the present invention. First, N r A dataset of samples

number

number

number

number

[0050] The full model is defined as follows:

number

number

[0051] Next, a perspective view semantics module according to an embodiment of the present invention will be described.

[0052] The goal of the perspective semantics module 310 is to provide pixel-by-pixel occlusion-reasoned semantics for a given perspective view on RGB, i.e., OSP. Note that compared to traditional semantic segmentation models that predict semantics only from visible pixels, the perspective semantics module 310 focuses on predicting both visible and occluded background classes. Therefore, we need to train the module to predict meanings based on contextual cues, rather than relying entirely on visible information.

[0053] Such a desired ground truth x p Assuming that is available in the dataset, we follow the structure of HRNetV2-W18 as the semantic segmentation backbone since it achieves a very good trade-off between accuracy and efficiency. Formally,

number

number

[0054] The PS module 310 is defined as follows:

number

[0055] Next, a description will be given of the top view semantics module 210 according to an embodiment of the present invention.

[0056] The second module, the top-view semantics module 260, takes the OSP as input and learns to explicitly project the semantics in the perspective views onto the top view.

number

number

[0057] In view of these challenges, we first take advantage of the fact that roads form an approximately flat surface to facilitate projection without depth estimation. This step can be considered as a transformation module. Secondly, we provide a hallucination module 330 to deal with sparse / noisy semantics and imperfect predictions for far and near regions in the top view.

[0058] Specifically, we assume known camera intrinsics and extrinsics with respect to the ground plane. This is a mild assumption since they can be obtained in advance via calibration. Therefore, it is well known that we can compute a homography that maps each pixel of the oblique view to the BEV view. This gives us the transformation module.

[0059] After the transformation module 320 that maps the OSP to the top view, the hallucination module 330 then learns to predict unseen distant regions as well as recover noisy semantics using the contextual information of the top view. A shallow, e.g., 5-layer encoder and decoder is used for the hallucination module 330. Note that the input and output of the hallucination module 330 are of the same size, i.e., h×w×(C+1).

[0060] In summary, the TS module is defined as follows:

number

[0061] Next, a top view parametric prediction module 340 according to an embodiment of the present invention will be described.

[0062] Given the Hallucinated Semantics in Top-view (HST), the next step is to predict the layout attributes. Briefly, the Top-view Parametric Prediction (TPP) module 340 maps the HST x to scene model parameters Θ, which are divided into three groups, i.e., Θ for each of the 14 binaries of the scene model. b , Θ for multi-class m , and Θ for 10 continuous attributes cBinary attributes include information such as whether a road is one-way. The number of lanes to the left of the vehicle is an example of a multi-class attribute, while the distance to a side road on the right might be an example of a continuous attribute.

[0063] The TTP module 340 is defined as follows:

number

[0064] The TTP module 340 can also utilize simulated data: the current design can be easily extended to a hybrid version that leverages the rich and large annotation sets obtained from simulated data during training.

[0065] Next, model training according to an embodiment of the present invention will be described.

[0066] For each module, we assume that supervision is fully available. We explain how to collect such a dataset when we have access to human annotations only for the parametric layout Θ, and show how to utilize deep supervision during training.

[0067] The complete loss function L is defined as follows:

number

[0068] Next, the loss function in the top view parametric prediction module according to an embodiment of the present invention will be described. Since Θ and I are already available, the loss function for parametric prediction is defined as follows:

number

[0069] A top view semantics module according to an embodiment of the present invention will now be described.

[0070] Unlike straightforward design in parametric space, the second module requires pixel-wise supervision in the top view. For this reason, we propose to utilize a rendering function that generates pixel-wise semantics from the parametric annotations. Specifically, for each Θ, we render an x-map.

[0071] Next, we explain the loss function on the top view semantics module.

[0072] The loss function of the TS module is defined as follows:

number

number

[0073] Next, a perspective view semantics module according to an embodiment of the present invention will be described.

[0074] As long as we have a top-view semantics x, we can map it back to the oblique view using the camera parameters in the same way as in the planar assumption. Again, this is achieved by computing the homography between the oblique view and the BEV view.

[0075] Backprojection can indeed recover occluded regions and align semantics well with respect to perspective input images.

[0076] Next, a loss function on the oblique view semantics module according to an embodiment of the present invention will be described.

[0077] Similarly, the loss function for the PS module is:

number

number

number

[0078] The present invention may be a system, method, and / or computer program product at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0079] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0080] The computer readable program instructions described herein can be downloaded from a computer readable storage medium into each computing / processing device, or can be downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can be comprised of copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in the computer readable storage medium in the respective computing / processing device.

[0081] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or object oriented programming languages ​​such as SMALLTALK®, C++, traditional procedural programming languages ​​such as the "C" programming language, or similar programming languages. The computer readable program instructions may be executed completely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), and the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can utilize state information of computer readable program instructions to personalize the electronic circuitry and execute the computer readable program instructions to carry out aspects of the invention.

[0082] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0083] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device to manufacture a machine such that the instructions, executed by the processor of the computer or other programmable data processing device, create means for performing the functions / operations specified in the block or blocks of the flowcharts and / or block diagrams. These computer readable program instructions can also be stored on a computer readable storage medium that can direct a computer, programmable data processing device, and / or other device to function in a particular manner, such that the instructions stored on the computer readable storage medium constitute an article of manufacture having stored instructions that implement aspects of the functions / operations specified in the block or blocks of the flowcharts and / or block diagrams.

[0084] The computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to cause a sequence of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the function / operation specified in the block or blocks of the flowcharts or block diagrams.

[0085] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously or the blocks may be executed in the reverse order depending on the functionality involved. It is also noted that each block of the block diagrams and / or flowchart diagrams, as well as combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special purpose hardware-based system that executes the specified functions or operations or executes a combination of special purpose hardware and computer instructions.

[0086] In the specification, references to "one embodiment" or "one embodiment" of the invention, as well as other variations, mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the invention. Thus, the appearances of the phrase "in one embodiment" or "in one embodiment" in various places throughout this specification, as well as any other variations, do not necessarily all refer to the same embodiment.

[0087] For example, in the case of "A / B," the use of any of the following " / ," "and / or," "at least one of," such as "A and / or B," "at least one of A and B," will be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), the selection of only the first and third listed alternatives (A and C), the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This can be expanded as many times as the number of items listed, as can be readily understood by one of ordinary skill in the present and related arts.

[0088] The foregoing is understood in all respects to be illustrative and illustrative, but not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description, but from the claims which are interpreted in accordance with the full breadth permitted by the patent laws. It will be understood that the embodiments shown and described herein are merely exemplary of the invention, and that those skilled in the art could make various modifications without departing from the scope and spirit of the invention. Those skilled in the art could make various other feature combinations without departing from the scope and spirit of the invention. Having thus described aspects of the invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is as set forth in the appended claims.

Claims

1. 1. A computer-implemented method for road layout prediction, comprising: performing pixel-level semantic segmentation including occlusion inference on a perspective view image of a road to generate a segmentation result that includes semantics of both visible and occluded pixels in the perspective view; mapping the segmentation results, including the semantics of both visible and occluded pixels in the perspective view, to a top view using a road plane assumption indicating that the road forms an approximately flat surface; and generating a parametric prediction of road layout-related attributes for an unoccluded state of the road based on the segmentation result, which includes semantics of both visible and occluded pixels mapped to the top view.

2. 10. The computer-implemented method of claim 1, A computer-implemented method, wherein the step of generating a parametric prediction includes receiving pixel-level human annotations only for the road-layout related attributes in the top view, and the steps of generating a segmentation result and mapping to the top view are performed without human annotation.

3. 2. The computer-implemented method of claim 1, A computer-implemented method, wherein the steps of generating segmentation results and mapping to a top view use a pixel-level supervisory signal.

4. 10. The computer-implemented method of claim 1, A computer-implemented method, wherein the parametric prediction of road layout related attributes includes predicted boundaries selected from the group consisting of road boundaries, crosswalk boundaries, footpath boundaries, and lane boundaries, one or more members of the group being originally occluded by respective foreground elements in the perspective view.

5. 10. The computer-implemented method of claim 1, A computer-implemented method for processing the segmentation result, which includes semantics of both visible and occluded pixels mapped to the top view, to recover semantics of noisy regions.

6. 10. The computer-implemented method of claim 1, The computer-implemented method further comprising controlling a vehicle system of a vehicle traveling on the road for accident avoidance in response to the parametric prediction of the road layout-related attributes.

7. 7. The computer-implemented method of claim 6, further comprising: A computer-implemented method, wherein the vehicle system is selected from the group consisting of a braking system, an acceleration system, a stability system, and a steering system.

8. 6. The computer-implemented method of claim 5, A computer-implemented method for generating an oblique view that represents a state in which the road occlusion has been restored by mapping the segmentation result of the top view, which has been processed to restore the semantics of the noisy regions, onto an oblique view.

9. 10. The computer-implemented method of claim 1, The method is a computer-implemented method performed by an advanced driver assistance system.

10. On the computer, performing pixel-level semantic segmentation including occlusion inference on a perspective view image of a road to generate a segmentation result that includes semantics of both visible and occluded pixels in the perspective view; mapping the segmentation results, including the semantics of both visible and occluded pixels in the perspective view, to a top view using a road plane assumption indicating that the road forms an approximately flat surface; and generating a parametric prediction of road-layout-related attributes for an unoccluded state of the road based on the segmentation results, including semantics of both visible and occluded pixels mapped to the top view.

11. The program according to claim 10, The step of generating a parametric prediction includes receiving pixel-level human annotations only for the road-layout related attributes in the top view, and the steps of generating the segmentation result and mapping to the top view are performed without human annotation.

12. The program according to claim 10, The steps of generating the segmentation result and mapping to the top view use a pixel-level supervisory signal.

13. The program according to claim 10, The parametric prediction of road layout related attributes includes predicted boundaries selected from the group consisting of road boundaries, crosswalk boundaries, sidewalk boundaries, and lane boundaries, one or more members of the group being originally occluded by respective foreground elements in the perspective view.

14. The program according to claim 10, further comprising: A program that processes the segmentation result, which includes the semantics of both visible and occluded pixels mapped to the top view, to recover semantics of noisy regions.

15. The program according to claim 10, The method further comprises causing the computer to control a vehicle system of a vehicle traveling on the road to avoid an accident in response to the parametric prediction of the road layout related attributes.

16. The program according to claim 15, The vehicle system is selected from the group consisting of a braking system, an acceleration system, a stability system, and a steering system.

17. The program according to claim 14, A program that generates a perspective view that represents a state in which the road occlusion has been restored by mapping the segmentation result of the top view, which has been processed to restore the semantics of the noisy region, onto a perspective view.

18. 1. A computer processing system for road layout prediction, comprising: a memory device for storing program code; a processor unit for executing the program code, the program code comprising: performing pixel-level semantic segmentation including occlusion inference on a perspective view image of a road to generate a segmentation result that includes semantics of both visible and occluded pixels in the perspective view; mapping the segmentation results, including the semantics of both visible and occluded pixels in the perspective view, to a top view using a road plane assumption indicating that the road forms an approximately flat surface; and generating a parametric prediction of road layout-related attributes for an unoccluded state of the road based on the segmentation result, which includes semantics of both visible and occluded pixels mapped to the top view.

Images