Method and system for topology detection

Abstract

A system and method includes receiving an image of a first lane of a roadway, the image being captured by a first sensor; defining the first lane as a group of TV ordered points; representing the first lane by a combination of TV ordered points and a predefined number of types of lane components C; generating images, each image representing one of the lane components of the type; and combining the generated images to generate a topological representation of the first lane.

Description

Background Art

[0001] An autonomous vehicle is a motor vehicle capable of performing one or more essential driving functions without input from a human driver, typically including Level 2 or higher capabilities as generally described in the J3016 standard of the Society of Automotive Engineers International (SAE International), and in some embodiments includes a self-driving truck that includes sensors, devices, and systems that can act together to generate sensor data indicative of various parameter values ​​related to the vehicle's position, speed, operating characteristics, and state, including data generated in response to various objects, situations, and environments encountered by the autonomous vehicle during its operation.

[0002] Autonomous vehicles can rely on sensors such as cameras, lidar, radar, inertial measurement units (IMUs), etc. to understand the road and the rest of the world around the vehicle without user interaction. Accurate understanding and modeling of the road on which the autonomous vehicle operates is very important, for example, so that the vehicle can use sensor readings (i.e., sensor data) to safely navigate on the road. Accurate modeling or estimation of the road may be critical for perception (computer vision), control, mapping, and other functions. Without a true understanding of the local environment in which it operates, the autonomous vehicle may have difficulty staying in its lane and have other problems such as steering, navigation, and object avoidance.

[0003] Therefore, there is a need for efficient and robust systems and methods for accurately and effectively determining and modeling road topology for operation of autonomous vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Features and advantages of example embodiments and the manner in which the same are achieved will become more apparent with reference to the following detailed description taken in conjunction with the accompanying drawings.

[0005] Figure 1 is an illustrative block diagram of a control system that may be deployed in a vehicle according to an example embodiment;

[0006] Figure 2A – 2C is an illustrative depiction of an exterior view of a semi-truck according to an example embodiment;

[0007] Figure 3 is an illustrative depiction of a road including multiple lanes in which an autonomous vehicle may operate, according to an example embodiment;

[0008] Figure 4 is an illustrative depiction of a set of defined lane line geometries according to an example embodiment;

[0009] Figure 5A- 5C is an illustrative depiction of road lanes according to an example embodiment, including each lane represented by N ordered points;

[0010] Figure 6A 6D is an illustrative example of a road lane represented by a combination of lane components at N ordered points according to an example embodiment;

[0011] Figure 7 is an illustrative block diagram of a system according to an example embodiment;

[0012] Figure 8 is an illustrative flow chart of a process according to an example embodiment; and

[0013] Figure 9 is an illustrative block diagram of a computing system according to an example embodiment.

[0014] Throughout the drawings and detailed description, unless otherwise described, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The relative sizes and depictions of these elements may be exaggerated or adjusted for clarity, illustration, and / or convenience. DETAILED DESCRIPTION

[0015] In the following description, specific details are set forth in order to provide a thorough understanding of each example embodiment. It should be understood that various modifications to the embodiments will become readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit or scope of the present disclosure. In addition, in the following description, many details are set forth for the purpose of explanation. However, it should be understood by those of ordinary skill in the art that the embodiments may be practiced without using these specific details. In other examples, well-known structures and methods are not shown or described to avoid obscuring the description with unnecessary details. Therefore, the present disclosure is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed herein.

[0016] For convenience and ease of description, various terms will be used herein. For example, the term "semi-trailer truck" will be used to refer to a vehicle in which the systems of the example embodiments can be used. The terms "semi-trailer truck," "truck," "tractor," "vehicle," and "semi-trailer" are used interchangeably herein. Additionally, it will be apparent to those skilled in the art upon reading this disclosure that embodiments of the present invention can be used in conjunction with other types of vehicles. In general, embodiments can be used in conjunction with any vehicle that tows a trailer or transports freight over long distances with the desired results.

[0017] Figure 1A control system 100 is shown according to an example embodiment that may be deployed in and include an autonomous vehicle (AV), such as, but not limited to, an autonomous vehicle. Figure 2A - Semi-trailer truck 200 depicted in 2C. Figure 1 , control system 100 may include sensors 110 that collect data and information that is provided to computer system 140 to perform operations, including, for example, control operations of components of the vehicle through gateway 180. According to some embodiments, gateway 180 is configured to allow computer system 140 to control different components from different manufacturers.

[0018] The computer system 140 may be configured with one or more central processing units (CPUs) 142 to perform processing, including processing that implements features of embodiments of the present invention as described elsewhere herein, and to receive sensor data from the sensors 110 for generating control signals to control one or more actuators or other controllers associated with systems of the vehicle in which the control system 100 is deployed (e.g., actuators or controllers that allow control of the throttle 184, steering system 186, brakes 188, and / or other devices and systems). Generally, the control system 100 may be configured to operate a vehicle (e.g., a semi-truck 200) in an autonomous (or semi-autonomous) operating mode.

[0019] For example, the control system 100 can be operated to capture images from one or more cameras 112 mounted at various locations on the semi-trailer truck 200 and perform processing (e.g., image processing) on ​​these captured images to identify objects near or in the path of the semi-trailer truck 200. In some aspects, one or more lidar 114 and radar 116 sensors can be positioned on the vehicle to sense or detect the presence and size of objects near or in the path of the semi-trailer truck 200. Other sensors can also be positioned or mounted at various locations on the semi-trailer truck 200 to capture other information, such as positioning data. For example, the sensors can include one or more satellite positioning sensors and / or inertial navigation systems such as GNSS / IMU 118. The Global Navigation Satellite System (GNSS) is a space-based satellite system that provides location information (longitude, latitude, altitude) and time information anywhere on or near the Earth in all weather conditions to devices called GNSS receivers. GPS is the most widely used GNSS system in the world and is used interchangeably with GNSS herein. An inertial measurement unit ("IMU") is an inertial navigation system. Typically, an inertial navigation system ("INS") measures and integrates the heading, position, velocity, and acceleration of a moving object. The INS integrates the measured data, with the GNSS used to correct for integration errors in the INS heading calculation. Any number of different types of GNSS / IMU 118 sensors may be used in conjunction with features of the present invention.

[0020] The data collected by each of the sensors 110 may be processed by the computer system 140 to generate control signals that may be used to control the operation of the semi-truck 200. For example, the image and position information may be processed to identify or detect objects around or in the path of the semi-truck 200, and control signals may be transmitted through the controller 182 to adjust the throttle 184, steering 186, and / or brakes 188 as needed to safely operate the semi-truck 200 in an autonomous or semi-autonomous manner. Note that while Figure 1 Illustrative example sensors, actuators, and other vehicle systems and devices are shown in FIG, but those skilled in the art, after reading this disclosure, will understand that other sensors, actuators, and systems consistent with the present disclosure may also be included in system 100. For example, in some embodiments, an actuator may also be provided that provides a mechanism that allows for control of the speed of a vehicle (e.g., semi-truck 200).

[0021] The control system 100 may include a computer system 140 (e.g., a computer server) configured to provide a computing environment in which one or more software, firmware, and control applications (e.g., items 160-182) may be executed to perform at least some of the processes described herein. In some embodiments, the computer system 140 includes components deployed on a vehicle (e.g., deployed on a vehicle such as a Figure 2C The computer system 140 is shown in a system rack 240 within the sleeper compartment 212 of the semi-truck. The computer system 140 can communicate with other computer systems (not shown) that may be located locally and / or remotely from the semi-truck 200 (e.g., the computer system 140 can communicate with one or more remote ground-based or cloud-based computer systems via a wireless communication network connection).

[0022] According to various embodiments described herein, the computer system 140 can be implemented as a server. In some embodiments, the computer system 140 can be configured using any of a number of computing systems, environments, and / or configurations, such as, but not limited to, a personal computer system, a cloud platform, a server computer system, a thin client, a fat client, a handheld or laptop computer device, a tablet computer, a smartphone, a database, a multiprocessor system, a microprocessor-based system, a set-top box, a programmable consumer electronics product, a network PC, a small computer system, a mainframe computer system, a distributed cloud computing environment, etc., which may include any of the above systems or devices, etc.

[0023] Different software applications or components may be executed by the computer system 140 and the control system 100. For example, as shown at active learning component 160, an application that performs active learning machine processing may be provided to process images captured by one or more cameras 112 and information obtained by the lidar 114. For example, a deep learning segmentation model 162 may be used to process image data to identify objects of interest (e.g., other vehicles, construction signs, etc.) in the captured images. In some aspects of the present disclosure, deep learning segmentation may be used to identify lane points within a lidar scan. As an example, the system may use an intensity-based voxel filter to identify lane points within a lidar scan. The lidar data may be processed by a machine learning application 164 to draw or identify bounding boxes on the image data to identify objects of interest located by the lidar sensor.

[0024] The information output from the machine learning applications can be provided as input to the object fusion 168 and visual map fusion 170 software components, which can perform processing to predict the actions of other road users and fuse the local vehicle pose with the global map geometry in real time, thereby enabling immediate map corrections. The output from the machine learning applications can be supplemented with information from the radar 116 and the map positioning 166 application data (as well as positioning data). In some aspects, these applications allow the control system 100 to be less dependent on maps and more capable of handling changing road environments. Additionally, by instantly correcting any map errors, the control system 100 can promote safer, more scalable, and more efficient operations compared to alternative map-centric approaches.

[0025] This information is provided to the prediction and planning application 172, which provides input to the trajectory planning 174 component, allowing the trajectory generation system 176 to generate a trajectory in real time based on the interactions and predicted interactions between the semi-trailer truck 200 and other relevant vehicles in the truck's operating environment. In some embodiments, for example, the control system 100 generates a sixty-second plan time horizon, analyzing the relevant participants and available trajectories. The plan that best meets multiple criteria (including safety, comfort, and route preference) can be selected, and any relevant control inputs required to implement the plan are provided to the controller 182 to control the movement of the semi-trailer truck 200.

[0026] In some embodiments, unless otherwise indicated, these disclosed applications or components (and other components or processes described herein) may be implemented in hardware, in a computer program executed by a processor, in firmware, or in a combination thereof. In some cases, a computer program may be embodied on a computer-readable medium, such as a storage medium or a storage device. For example, a computer program, code, or instructions may reside in random access memory ("RAM"), flash memory, read-only memory ("ROM"), erasable programmable read-only memory ("EPROM"), electrically erasable programmable read-only memory ("EEPROM"), registers, a hard disk, a removable disk, a compact disk read-only memory ("CD-ROM"), or any other form of non-transitory storage medium known in the art.

[0027] A non-transitory storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. In an alternative embodiment, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application specific integrated circuit ("ASIC"). In an alternative embodiment, the processor and the storage medium may reside as discrete components. For example, Figure 1An example computer system 140 is shown that may represent or be integrated with any of the components, etc., disclosed below. Figure 1 It is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the systems and methods disclosed herein.Computer system 140 is capable of implementing and / or performing any of the functionality disclosed herein.

[0028] Computer system 140 may be described in the general context of computer system executable instructions, such as program modules, executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer system 140 may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules may be located in both local computer system storage media and remote computer system storage media, including non-transitory memory storage devices.

[0029] refer to Figure 1 , the computer system 140 is shown in the form of a general-purpose computing device. Components of the computer system 140 may include, but are not limited to, one or more processors (e.g., a CPU 142 and a GPU 144), a communication interface 146, one or more input / output interfaces 148, and one or more storage devices 150. Although not shown, the computer system 140 may also include a system bus that couples various system components including system memory to the CPU 142. In some embodiments, the input / output (I / O) interface 148 may also include a network interface. For example, in some embodiments, some or all of the components of the control system 100 may communicate via a controller area network ("CAN") bus or the like (which interconnects various components within a vehicle in which the control system 100 is deployed and associated).

[0030] In some embodiments, the storage device 150 may include various types and forms of non-transitory computer-readable media. Such media can be any available media accessible to the computer system / server, and the media may include volatile and non-volatile media, removable and non-removable media. In one embodiment, the system memory implements the processes represented by the flowcharts of the other figures of this document. The system memory may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. As another example, the storage device 150 can read and write non-removable, non-volatile magnetic media (not shown and commonly referred to as "hard drives"). Although not shown, the storage device 150 may include one or more removable non-volatile disk drives, such as disk drives, tape drives, or optical drives. In such cases, each drive can be connected to the bus via one or more data media interfaces. The storage device 150 may include at least one program product having a set (e.g., at least one) of program modules, codes, and / or instructions configured to perform the functions of the various embodiments of the present application.

[0031] Figure 2A -2C is an illustrative depiction of an exterior view of a semi-trailer truck 200 that may be associated with or used in accordance with the example embodiments. The semi-trailer truck 200 shown is for illustration purposes only. Therefore, those skilled in the art will understand after reading this disclosure that the embodiments may be used in conjunction with a variety of different types of vehicles and are not limited to Figure 2A – Vehicle of the type shown in 2C. Figure 2A The example semi-trailer truck 200 shown in Figure 2C is a type of truck configuration commonly found in North America and includes an engine 206, a steering axle 214, and two drive axles 216 located in front of a cab 202. A trailer (not shown) can typically be attached to the semi-trailer truck 200 via a fifth-wheel trailer coupling disposed on a frame 218 and positioned above the drive axles 216. A sleeper car 212 can be positioned behind the cab 202, such as Figure 2A and 2C shown. Figure 2A Figure 2C further illustrates a plurality of sensors positioned at various locations on the semi-truck 200. For example, one or more sensors may be mounted on a sensor rack 220 on top of the cab 202. Sensors may also be mounted on the side mirrors 210 and other locations on the semi-truck. Sensors may be mounted on the bumper 204 and on the sides and other locations of the cab 202. For example, a rear-facing radar 236 is shown mounted on the rear of the semi-truck. Figure 2A202. The embodiments may be used with other configurations of trucks and other vehicles, such as semi-trucks with cab-over or cab-forward configurations, etc. In general, and without limiting the embodiments of the present disclosure, the features of the present invention may be used with desirable results on vehicles that carry freight over long distances, such as long-distance semi-truck routes.

[0032] Figure 2B is a front view of a semi-truck 200 and illustrates various sensors and sensor locations. A sensor rack 220 may secure and position several sensors above the windshield 208, including a long-range lidar 222, a long-range camera 224, a GPS antenna 234, and a mid-range forward-facing camera 226. The side mirrors 210 may provide mounting locations for a rear-facing camera 228 and a mid-range lidar 230. The forward-facing radar 232 may be mounted on the bumper 204. Other sensors, including those shown and some not shown, may be mounted elsewhere on the semi-truck 200. Thus, Figure 2A The locations and installations depicted in –2C are for illustration purposes only.

[0033] Now refer to Figure 2C , shows a partial view of a semi-truck 200 depicting some aspects of the interior of the cab 202 and the sleeper compartment 212. In some embodiments, Figure 1 Portions of the control system 100 may be deployed in a system rack 240 in the sleeper car 212, allowing easy access to components of the control system 100 for maintenance and operation.

[0034] Certain aspects of the present disclosure relate to a method and system that provides a framework or architecture for implementing a system in an autonomous vehicle (AV) (e.g., similar to a vehicle). Figure 1 and 2A -2C) generates an accurate representation of the topology of a road including one or more lanes in real time while operating (e.g., driving). Aspects of the present disclosure generally provide a framework for accurately and efficiently determining the topology of lanes of a road near an AV, including, for example, different lane configurations.

[0035] As used herein, the topology of a road refers to a map of the structure or configuration of the road, where some embodiments of the present disclosure may focus on the topology of the road in front of the subject AV (ie, the ego vehicle).

[0036] Figure 3 is an illustrative depiction of a road including multiple lanes in which an AV (or other) vehicle may operate, according to an example embodiment. Figure 3 A road 300 is shown including multiple lanes to accommodate vehicular traffic. Figure 3In the example of , five (5) lanes of traffic are depicted, including lanes 305, 310, 315, 320, and 325, with the direction of traffic flow indicated. Figure 3 In some other figures herein, a road may be represented as a single line, where the single line representation may correspond to the centerline of a traffic lane (i.e., the average value between the boundaries of the traffic lane). In some cases herein, a traffic lane may also be referred to simply as a lane. The lanes of a road may have different configurations. For example, road 300 includes lanes of different configurations, where lanes 305, 310, and 315 may each be characterized as a straight lane, and lane 320 may be characterized as a split lane because it splits or divides into another lane 325 at 330 (e.g., an exit or exit ramp).

[0037] In some aspects, such as Figure 3 As shown in the example lane section of FIG, a lane can be characterized as having a specific lane line geometry (ie, a structural shape or configuration). In some embodiments herein, the lane line geometry of a road lane can be represented by a limited set of predefined different shapes. Figure 4 is an illustrative line drawing depiction of a set of predefined lane line geometries 400 that may be used to represent various lane configurations in some embodiments herein. The example set of predefined lane line geometries 400 includes seven (7) different lane line geometries. Figure 4 In the direction of traffic flow indicated in , the seven (7) lane line geometries of group 400 include straight lane line geometry 405, merge left lane line geometry 410, split right lane line geometry 415, merge left then split right lane line geometry 420, merge right lane line geometry 425, split left lane line geometry 430 and merge right then split left lane line geometry 435.

[0038] In some embodiments, a set of predefined lane geometries 400 can be used, individually or in combination, to describe or otherwise represent substantially all road configurations that an AV might actually use or encounter. That is, in some operational contexts and use cases of AVs herein, real-world lane configurations can be fully represented by a set of predefined lane geometries 400.

[0039] According to some embodiments of the present disclosure, each lane line geometry (i.e., shape or configuration) in a set of predefined lane line geometries 400 can be characterized or defined by a combination of a predefined number C of types of lane components. In an example embodiment, a set of predefined lane line geometries 400 can be fully defined by a combination of three (i.e., C=3) predefined types of lane components. In this embodiment, the three (3) types of lane components can include a continuous lane component, a merged lane component, and a split lane component. In some aspects, a set of predefined lane line geometries 400 can be fully defined by the superposition (i.e., combination) of the three (3) types of lane components (i.e., continuous lane component, merged lane component, and split lane component). Therefore, the combination of the three (3) types of lane components disclosed herein can be used to characterize or define road lanes for substantially all road topologies, because substantially all road topologies can be represented by a set of predefined lane line geometries 400.

[0040] In some aspects, the disclosed framework includes a set of predefined lane line geometries and a predefined number of lane component types that can be combined to define a road lane that can be deployed on an AV to enable a computing system on the AV to characterize lanes of interest in its vicinity in real time as the AV travels on the road. In cases where a road includes more than one lane and multiple lanes are of interest to the AV, the concepts and processes herein for determining the topology of a single lane can be applied to each lane of a multi-lane road, and an aggregation of multiple topologies (one for each lane) can be determined to generate a multi-lane road (e.g., Figure 3 , the overall or overall topology of the road 300).

[0041] In some embodiments, the systems and processes herein can utilize aspects of deep learning to detect the geometry of road lanes. Specifically, in some embodiments herein, aspects and techniques of landmark detection via neural networks can be used to detect the geometry of roads herein in a flexible and adaptable manner. For example, as the vehicle traverses the road, images captured by an onboard camera of an AV can be provided as input to a neural network configured to characterize or define the lanes of the road as a set of N ordered points (i.e., keypoints / landmarks).

[0042] Figure 5A is an illustrative depiction of a road 500, for example, as captured in an image acquired by a camera on an AV as the vehicle travels on the road. Figure 5AIn the example of Figure 5C, the road includes two (2) lanes, lane 505 (i.e., a continuous lane component) and lane 510 (i.e., a merged lane component). According to other aspects of the present disclosure, the road can be configured to include split lane components, wherein the road can include one or more of the three (3) types of lane components disclosed herein in various combinations. Applying the landmark detection process to the image of road 500 may cause lanes 505 and 510 to be each defined by a set of N ordered points (i.e., keypoints or landmarks), such as Figure 5B and 5C Depicted separately. Figure 5B and 5C In the example shown, N = 8 key points or landmarks, such as Figure 5B Representative key points or landmarks 515, 520 and 525 and Figure 5B , where each of the key points or landmarks corresponds to approximately the same location in each lane (e.g., 515-530; 520-535; and 525-540). In some embodiments, the value of N can be set based on, for example, rules, algorithms, the operating state of the AV and its surroundings (environment), implementation constraints (e.g., available computing resources, etc.), and other factors.

[0043] In some embodiments, each set of N ordered points (i.e., landmarks) is vertically distributed in the image. In addition, each landmark can represent each of the three (or other number of defined) types of lane components disclosed herein (e.g., continuous lane components, merged lane components, and split lane components). That is, each of the three (3) types of lane components will have N points to describe it, so a single lane in this document can be defined by N×C points (C=3).

[0044] Figure 6A - 6D is an illustrative example of a road lane represented by a combination of three lane component types at N ordered points defining the lane, according to an example embodiment. Figure 6A The three lane component types for lane 605 at each of the landmarks are shown, with the lanes not splitting or merging. Figure 6A - Graphical element representation of three lane component types in 6D. For a strictly straight topology of the road 605, all landmarks in the landmarks are placed on top of each other. Figure 6B A road 610 is shown having a topology of merging lanes, with the locations of the three lane component types depicted at each of the landmarks. Figure 6C A road 615 having a split lane topology is shown, with the locations of the three lane component types at each of the landmarks shown. Figure 6D Further illustrated is a road 620 having a topology of merging lanes and then splitting lanes, wherein this topology is also effectively and adequately represented by the positions of the three lane component types disclosed herein at each of the determined landmarks.

[0045] In this way, Figure 6A 6D shows how aspects disclosed herein can efficiently and flexibly represent the topology of various different road lane configurations in an efficient manner. For example, in some aspects, the topology of lanes can be represented or defined with a minimum number of parameters N×C.

[0046] Figure 7 is an illustrative block diagram of a system according to an example embodiment. In some aspects, Figure 7 700 is a high-level system architecture for determining the topology of a single lane of a road. As shown, an input image from an onboard camera on an AV is received or otherwise obtained 705. The image may generally include a forward-looking view of the AV's immediately adjacent road.

[0047] An input image 705 is provided to or received by, for example, a U-Net style (i.e., convolutional neural network) image segmentation network 710. In some embodiments, the present disclosure utilizes deep learning features to (1) interpret "objects as points" to detect key points or landmarks of lanes within the input image, and (2) generate a desired output 715 comprising N×C (e.g., C=3) images using a neural network, wherein the output images can be used to define the topology (i.e., structure) of the lanes detected within the input image 705. In some aspects, the desired output comprising N×C (e.g., C=3) images is a result of the problem formulation used in some embodiments herein (i.e., the manner in which lanes herein can be represented or defined by C (e.g., 3) types of lane components and N ordered landmarks).

[0048] In some embodiments, the neural network 710 may include a U-Net style implementation with a ResNet-50 backbone convolutional neural network to generate the desired output image. However, embodiments of the present disclosure are not limited to the use of U-Net and ResNet-50 conventional networks, and other deep learning techniques and processes may be used as long as they are compatible with other aspects of this document. In some aspects, each output image contains a single keypoint or landmark, as described above. That is, each output image in the output 715 corresponds to a single keypoint or landmark. Given that the desired output is N ordered points, Figure 7The order of the images in output 715 corresponds to the order of the keypoints or landmarks established or determined for the lanes in the input image. Thus, output 715 includes three different output types of images (i.e., continuous lane component image 720, merged lane component image 725, and split lane component image 730) for each keypoint describing a merging lane, describing a continuous lane, and describing a split lane, respectively.

[0049] Recalling the principle of lane component superposition introduced above, the image of the key points can be used to determine the continuous lane component position, merged lane component position and split lane component position of each key point or landmark, and then the positions can be combined to generate or determine the topology of a given lane.

[0050] In some embodiments, the systems and methods disclosed herein for determining the topology of a single lane in an image can be extended to determine the topology of additional (i.e., other) lanes in the image. That is, the methods disclosed herein for representing lanes as specific lane component types, as well as the utilization and application of deep learning image segmentation and landmark detection techniques and neural networks, can be applied to all lanes in the image to generate an overall or complete topology of a road including one or more lanes.

[0051] Figure 8 is an illustrative flow chart of an example of a road topology detection process 800 according to an example embodiment. In some embodiments, the framework or architecture disclosed herein (e.g., Figure 7 , architecture 700) can be used to implement some aspects of process 800. In some cases, certain aspects of process 800 are discussed in detail elsewhere in this disclosure and are described in detail in the following. Figure 8 The following discussion may not be repeated.

[0052] At operation 805, a system, device, or apparatus implementing process 800 receives an image comprising a first lane of a road. According to some example embodiments herein, process 800 may be performed by an AV while the vehicle is traveling on a road, wherein the image is captured or otherwise acquired by one or more onboard cameras located on the AV, and the image is received by a computer processing unit, device, system, component, or module on the vehicle for analysis and processing. In some aspects, as disclosed herein above, each lane of the road may be represented by a set of predefined lane line geometries.

[0053] At operation 810, the first lane in the image received at operation 805 may be defined by a set of N ordered (key) points. As disclosed herein above (e.g., Figure 7), deep learning techniques (including certain types of neural networks) can be used to detect a set of N ordered keypoints or landmarks in the first lane in the image, where the N ordered keypoints define the first lane.

[0054] Continuing to operation 815, each of the N ordered points defining the first lane can be represented by a combination of a predefined number of types of lane components C. As disclosed in some example embodiments herein, the predefined number (C) of types of lane components can be three (3), including a continuous lane component, a merging lane component, and a split lane component. In some aspects, as mentioned above, each lane line geometry in a set of predefined lane line geometries can be fully defined by a combination of a predefined number of types of lane components C. Reference Figure 4 For example, a set of predefined lane line geometries may include seven (7) lane line geometries, each of which may be defined by a combination of three predefined number of types of lane components, such as Figure 6A –6D further demonstrated.

[0055] At operation 820, a first lane is defined as a set of N ordered points, and the aspect of representing the first lane by a combination of a predefined number of types of lane components C can be used to generate (N×C) images, where each image represents a lane component of the type of lane component of one of the N ordered points. For example, Figure 7 Output 715 disclosed in shows aspects of this operation.

[0056] Continuing to operation 825, the images generated at operation 820 may be used to generate a topological representation of the first lane. In some aspects, the images generated at operation 820 may be combined (eg, by overlaying) to generate a topological representation of the first lane.

[0057] In some cases, process 800 may be performed for each lane of a multi-lane road to generate a complete topological representation of the road that includes all of the lanes of the road. For example, process 800 may be performed for a first lane of the road and repeatedly (e.g., iteratively) performed for each of one or more additional lanes of the road, followed by, or at least partially in parallel with, processing of the first lane, generating a complete topological representation of the road that includes the first lane and the one or more additional lanes.

[0058] Figure 9 A computing system 900 is shown according to an example embodiment that can be used with the architecture or framework disclosed herein (e.g., Figure 1 , computer 140; Figure 7 , architecture 700) and processes (e.g. Figure 8 ) in any one of . Figure 9 9 is a block diagram of a server node 900 embodying an event processor according to some embodiments. Computing system 900 may include a general-purpose computing device and may execute program code to perform any of the functions described herein. According to some embodiments, computing system 900 may include other elements not shown.

[0059] The computing system 900 includes a processing unit 910 operatively coupled to a communication device 920, a data storage device 930, one or more input devices 940, one or more output devices 950, and a memory 960. The communication device 920 can facilitate communication with external devices (such as an external network, a data storage device, or other data source). The input device 940 can include, for example, a keyboard, a keypad, a mouse or other pointing device, a microphone, a knob or switch, an infrared (IR) port, a docking station, and / or a touch screen. For example, the input device 940 can be used to enter information into the computing system 900 (e.g., a manual request for a specific adjustment of an AV operation or function). The output device 950 can include, for example, a display (e.g., a screen), a speaker, and / or a printer.

[0060] Data storage device 930 may include any suitable persistent storage device, including a combination of magnetic storage devices (e.g., magnetic tape, hard drives, and flash memory), optical storage devices, read-only memory (ROM) devices, etc., and memory 960 may include random access memory (RAM).

[0061] The application servers 932 may each include program code executed by the processor 910 to cause the computing system 900 to perform one or more of the processes described herein. The embodiments are not limited to the execution of these processes by a single computing device. The data storage device 930 may also store data and other program code necessary to provide additional functionality and / or operation of the computing system 900, such as device drivers, operating system files, etc. The topology detection engine 934 may include program code executed by the processor 910 to generate a topological representation of one or more lanes of a road traversed by the AV, as disclosed in various embodiments herein. The topology detection engine 934 may reference a database management system DBMS 936 to retrieve an image of the road being navigated to determine the topology of the road lanes, as disclosed herein. The results generated by the topology detection engine 934 may be stored in a DBMS node 936.

[0062] As can be understood based on the foregoing description, the above-mentioned examples of the present disclosure can be implemented using computer programming or engineering techniques including computer software, firmware, hardware or any combination or subset thereof. According to the examples discussed in the present disclosure, any such result program with computer readable code can be embodied or provided in one or more non-transitory computer-readable media, thereby making a computer program product, that is, a product. For example, a non-transitory computer-readable medium can be, but is not limited to, a fixed drive, a floppy disk, an optical disk, a tape, a flash memory, an external drive, a semiconductor memory such as a read-only memory (ROM), a random access memory (RAM) and / or any other non-transitory transmission and / or receiving medium such as the Internet, cloud storage, the Internet of Things (IoT) or other communication networks or links. Products containing computer code can be made and / or used by executing code directly from a medium, by copying code from a medium to another medium, or by transmitting code via a network.

[0063] A computer program (also referred to as a program, software, software application, "application" or code) may include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, cloud storage, Internet of Things and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receives machine instructions as machine-readable signals. However, "machine-readable medium" and "computer-readable medium" do not include transient signals. The term "machine-readable signal" refers to any signal that can be used to provide machine instructions and / or any other kind of data to a programmable processor.

[0064] The above description and illustration of the method herein should not be considered to imply a fixed order for performing the method steps. On the contrary, the method steps can be performed in any feasible order, including performing at least some steps simultaneously. Although the present disclosure has been described in conjunction with specific examples, it should be understood that various changes, substitutions and modifications that are obvious to those skilled in the art can be made to the disclosed embodiments without departing from the spirit and scope of the present disclosure set forth in the appended claims.

Claims

1. A vehicle computing system comprising: a memory storing computer instructions; a data storage device storing data associated with operation of the vehicle, including data captured by at least a first sensor; as well as a processor communicatively coupled to the memory to execute the instructions and, during operation of the vehicle, capable of: receiving an image of a first lane of a road, the image captured by the first sensor; Define the first lane as a set of N ordered points; For each of the N ordered points, representing the first lane by a combination of a predefined number of types of lane components C; generating an image based on associating one or more of the lane components C of the type with one or more of the N ordered points; as well as The generated images are combined to generate a topological representation of the first lane. 2 . The vehicle computing system of claim 1 , wherein each lane of the road is represented by a set of predefined lane line geometries. 3 . The vehicle computing system of claim 2 , wherein each lane line geometry in the set of predefined lane line geometries is completely defined by a combination of the predefined number of types of lane components C. 4 .

4. A vehicle computing system according to claim 3, wherein the set of predefined lane line geometries includes at least seven (7) lane line geometries defined by combinations of the predefined number of types of lane components, where C is equal to at least 3 components, and (N×C) images are generated based on associating one or more lane components of the types of lane components C with one or more ordered points of the N ordered points. 5 . The vehicle computing system of claim 1 , wherein the first sensor is an onboard camera on the vehicle. 6 . The vehicle computing system of claim 1 , wherein the predefined number of types of lane components C include continuous lane components, merging lane components, and split lane components.

7. The vehicle computing system of claim 1 , wherein the processor is further capable of: receiving an image of one or more additional lanes of the road, the image of the one or more additional lanes captured by the first sensor; defining each of the one or more additional lanes as a set of N ordered points; For each of the N ordered points, representing the one or more additional lanes by a combination of the predefined number of types of lane components C; generating an image based on associating one or more of the lane components C of the type with one or more of the N ordered points of each of the one or more additional lanes; combining the generated images of each of the one or more further lanes to generate a topological representation of each of the one or more further lanes; and The generated topological representation of each of the one or more additional lanes is combined with the generated topological representation of the first lane to generate a complete topological representation of the road including the first lane and the one or more additional lanes.

8. A method comprising: receiving an image of a first lane of a road, the image captured by a first sensor; Define the first lane as a set of N ordered points; For each of the N ordered points, representing the first lane by a combination of a predefined number of types of lane components C; generating an image based on associating one or more of the lane components C of the type with one or more of the N ordered points; and The generated images are combined to generate a topological representation of the first lane.

9. The method of claim 8, wherein each lane of the road is represented by a set of predefined lane line geometries. 10 . The method of claim 9 , wherein each lane line geometry in the set of predefined lane line geometries is completely defined by a combination of lane components C of the predefined number of types.

11. A method according to claim 10, wherein the set of predefined lane line geometries includes at least seven (7) lane line geometries defined by combinations of lane components of the predefined number of types, where C is equal to at least 3 components, and (N×C) images are generated based on associating one or more lane components of the type of lane components C with one or more ordered points of the N ordered points.

12. The method of claim 8, wherein the first sensor is a camera mounted on a vehicle.

13. The method according to claim 8, wherein the predefined number of types of lane components C include continuous lane components, merged lane components and split lane components.

14. The method of claim 8, further comprising: receiving an image of one or more additional lanes of the road, the image of the one or more additional lanes captured by the first sensor; defining each of the one or more additional lanes as a set of N ordered points; For each of the N ordered points, representing the one or more additional lanes by a combination of a predefined number of types of lane components C; generating an image based on associating one or more of the lane components C of the type with one or more of the N ordered points of each of the one or more additional lanes; combining the generated images of each of the one or more further lanes to generate a topological representation of each of the one or more further lanes; and The generated topological representation of each of the one or more additional lanes is combined with the generated topological representation of the first lane to generate a complete topological representation of the road including the first lane and the one or more additional lanes.

15. A non-transitory medium having processor-executable instructions stored thereon, the medium comprising: instructions for receiving an image of a first lane of a road, the image captured by a first sensor; Define the first lane as a set of N ordered points; for representing, for each ordered point of the N ordered points, an instruction for the first lane by a combination of a predefined number of types of lane components C; instructions for generating an image based on associating one or more of the lane components C of the type with one or more of the N ordered points; as well as Instructions for combining the generated images to generate a topological representation of the first lane.

16. The non-transitory medium of claim 15, wherein each lane of the road is represented by a set of predefined lane line geometries.

17. A non-transitory medium according to claim 16, wherein each lane line geometry in the set of predefined lane line geometries is completely defined by a combination of the predefined number of types of lane components C, and (N×C) images are generated based on associating one or more lane components of the types of lane components C with one or more ordered points in the N ordered points.

18. The non-transitory medium of claim 15, wherein the first sensor is an onboard camera on a vehicle.

19. The non-transitory medium of claim 15, wherein the predefined number of types of lane components C include continuous lane components, merged lane components, and split lane components.

20. The non-transitory medium of claim 15, further comprising: instructions for receiving an image of one or more additional lanes of the road, the image of the one or more additional lanes captured by the first sensor; instructions for defining each of the one or more additional lanes as a set of N ordered points; for representing, for each of the N ordered points, instructions for the one or more additional lanes by a combination of lane components C of the predefined number of types; instructions for generating an image based on associating one or more of the lane components C of the type with one or more of the N ordered points for each of the one or more additional lanes; instructions for combining the generated images of each of the one or more additional lanes to generate a topological representation of each of the one or more additional lanes; as well as Instructions for combining the generated topological representation of each of the one or more additional lanes with the generated topological representation of the first lane to generate a complete topological representation of the road including the first lane and the one or more additional lanes.

Images