METHOD AND DEVICE FOR A CONTEXT-APPROPRIATE, CROWD-SOURCE, SPARKLING, HIGH-RESOLUTION MAP

Abstract

A method for controlling a vehicle in question, wherein the vehicle in question includes a first control system, an auxiliary vehicle communication system, a GPS sensor, a spatial monitoring system and a navigation system that uses an in-vehicle navigation map, wherein the method comprises: Capture, via the spatial monitoring system, a 3D sensor representation of a field of view in the vicinity of the vehicle in question and an associated GPS position; Execute a feature extraction routine to extract properties from the 3D sensor representation; Performing a semantic segmentation of the extracted features from the 3D sensor representation; Performing simultaneous localization and mapping of the extracted features from the 3D sensor representation and the associated semantic segmentation of the extracted features from the 3D sensor representation; Performing a context extraction from the simultaneous localization and assignment of the extracted features; Updating the vehicle navigation map based on context extraction; Generating a simple map display based on the updated vehicle navigation map; Transferring the simple map display to a second control, where the second control is an off-board control; Execute, via the second control, from sparse map stitching to update a basic navigation map based on the simple map display; and Updating the vehicle navigation map based on the base navigation map where the execution, via the second control, from sparse map stitching to updating a cloud-based navigation map based on the simple map display includes the execution of a one-time relocation scheme, where performing the simultaneous localization and mapping of the extracted features from the 3D sensor representation and the associated semantic segmentation includes adding semantic segmentation for simultaneous localization and mapping (SLAM) to the vehicle navigation map.

Description

INTRODUCTION

[0001] The present disclosure relates to navigation systems for vehicles, including systems and methods for creating and detailing digitally stored maps and vehicle control based thereon.

[0002] Vehicles configured for autonomous operation are capable of fully or partially controlling one or more vehicle functions such as acceleration, deceleration, braking, and / or steering, with a certain degree of autonomy other than an operator. An autonomous vehicle may be equipped with sensors to monitor its environment, including, for example, radar devices, lidar devices, and imaging cameras. An autonomous vehicle may be capable of communicating outside the vehicle via vehicle-to-vehicle (V2V) and other communications, which may be referred to as vehicle-to-anything (V2X) communication, which may include infrastructure, smart road, and other systems. An autonomous vehicle may be equipped with a navigation system, which may include a positioning system such as a Global Positioning System (GPS) sensor.

[0003] An in-vehicle navigation system may rely on a digitally stored navigation map, which may be a high-definition (HD) map. The operation of an autonomous vehicle can be enhanced by improved precision and up-to-date map details, including the presence and location of dynamic, semi-dynamic, and static objects that may be present in a travel path.

[0004] It is desirable to have a navigation map accessible to a vehicle-based navigation system that contains accurate information on intersections, traffic signs, traffic lights, lane markings, sidewalks, pedestrian crossings, etc. It is also desirable to have a navigation map that can be updated regularly, while being aware of the requirements and limitations that may arise from wireless communication protocols such as 3G, 4G, 5G, etc. It is also desirable to have a navigation map that can be updated regularly using information from other vehicles.

[0005] DE 10 2017 211 607 A1 describes a method for verifying a digital map in a highly automated vehicle (HAF). First, a digital map, preferably a high-precision map, is made available in the vehicle's driver assistance system. The current vehicle position is then determined and located in the map. Subsequently, target feature properties of features in the vehicle's surroundings are provided. These features are detected by vehicle sensors, recording actual feature properties. The determined actual feature properties are then compared with the target feature properties to determine a difference value. The digital map is verified based on this comparison. If the difference value reaches or exceeds a specified threshold, the map is classified as out-of-date. If the difference value remains below the threshold, the map is considered up-to-date.The invention also includes a corresponding system and a computer program for implementing the method.

[0006] DE 10 2014 223 363 A1 describes the localization of a motor vehicle in a stationary reference map. Positions are assigned to objects in the reference map. While the vehicle is moving, radar pulses are emitted at successive points along a trajectory, followed by an angle- and time-resolved measurement of the radar impulse response. From this impulse response, the positions of objects in the vehicle's surroundings are determined, creating an environmental map. The vehicle's position is determined by comparing this environmental map with the reference map, with the reference map being continuously updated with the newly determined positions. The method is characterized by the fact that an object classification is performed for the detected objects in the environmental map and / or the reference map.The vehicle's position is determined taking this classification into account, which improves the accuracy of localization.

[0007] DE 10 2008 012 661 A1 describes a device for updating a vehicle's digital map. This device uses a multitude of sensors that record various aspects of the current traffic situation, vehicle movement, and road conditions. The collected measured values ​​are transmitted to a central unit, where they are evaluated. Based on these evaluations, the central unit creates update data that is sent to other vehicles to update their digital maps. This ensures a continuously updated and precise map of the surroundings for all participating vehicles. DESCRIPTION

[0008] The object of the invention is to improve the safety and efficiency of autonomous navigation. This object is achieved by the subject matter according to claim 1. Further developments can be found in the subclaims.

[0009] An object vehicle is described and includes an on-board controller, an auxiliary vehicle communication system, a GPS sensor, a spatial monitoring system, and a navigation system using an on-board navigation map.A method for controlling the vehicle includes acquiring a 3D sensor representation of a field of view proximal to the subject vehicle and an associated GPS position via the spatial monitoring system, executing a feature extraction routine for extracting features from the 3D sensor representation, performing semantic segmentation of the extracted features from the 3D sensor representation, performing simultaneous localization and mapping (SLAM) of the extracted features from the 3D sensor representation and the associated semantic segmentation, performing context extraction from the simultaneous localization and mapping of the extracted features, and updating the vehicle navigation map based on the context extraction.A simple map representation is generated based on the updated vehicle navigation map and transmitted to a second controller, the second controller being an off-board controller. The second controller performs sparse map stitching to update a basic navigation map based on the simple map representation. The vehicle navigation map is updated based on the off-board navigation map.

[0010] One aspect of the disclosure involves controlling the operation of the subject vehicle based on the updated vehicle navigation map.

[0011] Another aspect of the disclosure includes an autonomous operating system, wherein controlling operation of the subject vehicle based on the updated vehicle navigation map includes controlling the autonomous operating systems based on the updated vehicle navigation map.

[0012] Another aspect of the disclosure involves generating a simple map representation based on the updated vehicle navigation map that includes a set of features extracted from the environment defined by their feature descriptors, semantic labels, and 3D positions along with their viewpoints.

[0013] Another aspect of the disclosure involves executing a progressive relocation scheme to perform sparse map stitching to update the cloud-based navigation map based on the simple map representation.

[0014] Another aspect of the disclosure includes executing a one-time relocation scheme to perform sparse map stitching to update the cloud-based navigation map based on the simple map representation.

[0015] Another aspect of the disclosure involves adding semantic segmentation to the SLAM map to perform simultaneous localization and mapping of the extracted features from the 3D sensor representation and the associated semantic segmentation.

[0016] Another aspect of the revelation is that the off-board control is cloud-based.

[0017] Another aspect of the disclosure involves off-board control, which is a remote back-office control.

[0018] The above features and advantages, as well as other features and advantages of the present teachings, will become apparent from the following detailed description of some of the preferred embodiments and other embodiments for carrying out the present teachings, as defined in the appended claims, when used in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE CHARACTERS

[0019] One or more embodiments will now be described by way of example with reference to the accompanying figures, in which: Fig. 1 schematically illustrates an embodiment of an autonomous vehicle including a Global Position System (GPS) sensor, a navigation system, a telematics device, and a spatial monitoring system according to the disclosure. Fig. 2 schematically illustrates a system architecture used by a subject vehicle, including a vehicle crowd scanning system to generate and dynamically update a context-aware, sparse HD navigation map augmented with semi-dynamic and static objects according to the disclosure. Fig. 3 schematically shows details associated with a semantic segmentation routine that generates semantic labels and positions of semi-dynamic objects detected by a spatial monitoring system according to the disclosure. Fig. 4 schematically illustrates a one-time relocation routine involving the localization of a single keyframe from a vehicle navigation map to a base navigation map according to the disclosure. Fig. 5 schematically shows a progressive relocalization routine that includes the localization of the keyframes collected from the respective vehicle using a fusion of BoVWs (Bag of Visual Words), feature mapping, and PnP solver modules according to the disclosure.

[0020] It should be understood that the accompanying figures are not necessarily to scale and represent a somewhat simplified representation of various preferred features of the present disclosure as disclosed herein, including, for example, specific dimensions, orientations, positions, and shapes. Details associated with such features will be determined in part by the particular intended application and environment of use. DETAILED DESCRIPTION

[0021] The components of the disclosed embodiments, as described and illustrated herein, can be arranged and configured in a variety of different configurations. The following detailed description is therefore not intended to limit the scope of the disclosure, as claimed, but is merely representative of possible embodiments. Moreover, while numerous specific details are set forth in the following description to provide a thorough understanding of the embodiments disclosed herein, some embodiments may be practiced without some of these details. Moreover, for the sake of clarity, certain technical materials understood by those skilled in the art have not been described in detail to avoid unnecessarily obscuring the disclosure.

[0022] With reference to the figures, which are for the purpose of illustrating certain exemplary embodiments and not for the purpose of limiting the same, Fig. 1 schematically illustrates an embodiment of a subject vehicle 10 equipped with a navigation system 55. In one embodiment, and as described herein, the subject vehicle 10 includes a Global Position System (GPS) sensor 50, a telematics device 60, a spatial monitoring system 65, a human-machine interface (HMI) system 75, one or more controllers 15, and an autonomous operating system 45 arranged to provide a degree of autonomous vehicle operation. An on-board navigation map 56 is stored on a vehicle-based controller in communication with the navigation system 55.

[0023] The vehicle 10 in question includes a telematics device 60 that includes a wireless telematics communication system capable of communicating with other vehicles, including communicating with a communication network system having wireless and wired communication capabilities. The telematics device 60 is capable of performing inter-vehicle communication, including vehicle-to-vehicle (V2V) communication and / or vehicle-to-infrastructure (V2X) communication, which may also include communicating with an infrastructure monitor, e.g., a traffic camera. Alternatively or additionally, the telematics device 60 includes a wireless telematics communication system capable of wirelessly communicating with a portable device, e.g., a mobile phone, a satellite phone, or other telephony device.In one embodiment, the portable device is loaded with a software application including a wireless protocol for communicating with the telematics device 60, and the portable device performs the auxiliary vehicle communication, including communication with an external controller 95 via a communication network 90 including a satellite 80, an antenna 85, and / or another communication mode. Alternatively or additionally, the telematics device 60 performs communication with the auxiliary vehicle directly by communicating with the external controller 95 via the communication network 90. ​​A base navigation map 96 is stored in one or more memory devices of the off-board controller 95.The base navigation map 96 is a high-resolution (HD) map that has an accuracy on the order of meters in terms of position and on the order of centimeters in terms of object positions. The off-board control 95 may be a cloud-based computer system or a back-office computer system connected to a service provider. The term "cloud" and related terms can be defined as a model for ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned via virtualization and released with minimal management effort or interaction with the service provider, and then scaled accordingly. A cloud model can be composed of various features (e.g.,On-demand self-service, broad network access, resource pooling, rapid elasticity, metered service, etc.), service models (e.g., Software as a Service (“SaaS”), Platform as a Service (“PaaS”), Infrastructure as a Service (“IaaS”), and deployment models (e.g., private cloud, community cloud, public cloud, hybrid cloud, etc.).

[0024] The vehicle spatial monitoring system 65 includes a spatial monitoring controller in conjunction with a plurality of sensing devices. The vehicle spatial monitoring system 65 dynamically monitors an area proximate the subject vehicle 10 and generates digital representations of observed or otherwise detected distant objects. The spatial monitoring system 65 can determine a linear range, relative velocity, and trajectory for each adjacent distant object. The sensing devices of the spatial monitoring system 65 can include, by non-limiting description, front corner sensors, rear corner sensors, rear angle sensors, side sensors, a front radar sensor, and a camera in one embodiment, although the disclosure is not so limited.The placement of the aforementioned sensors enables the spatial monitoring system 65 to monitor traffic flow, including neighboring vehicles and other objects around the vehicle in question. The data generated by the spatial monitoring system 65 can be used by a lane marker detection processor (not shown) to estimate the roadway. The sensing devices of the vehicle spatial monitoring system 65 can further include object-locating sensing devices, including range sensors such as FM-CW (Frequency Modulated Continuous Wave) radar, pulse and FSK (Frequency Shift Keying) radar, and LIDAR (Light Detection and Ranging) devices, as well as ultrasonic devices that rely on effects such as Doppler-effect measurements to locate forward objects.Possible object location devices include charge-coupled devices (CCD) or complementary metal-oxide-semiconductor (CMOS) video image sensors and other camera / video image processors that use digital photographic techniques to "view" forward and / or rear objects, including one or more object vehicles. Such sensor systems are used to detect and locate objects in automotive applications and are usable with autonomous operating systems, including, for example, adaptive cruise control, autonomous braking, autonomous steering, and side object detection. The term "image" is used herein to refer to a representation of a field of view (FOV) captured by a scanning device of the spatial monitoring system 65, which may include a pixel-by-pixel image captured by a camera, a radar scan, a lidar scan, etc.The term “image” can also be referred to as “sensor representation,” which means that the outputs of sensors such as lidars and radars are scanned, while the output of a camera is an image.

[0025] The sensing devices associated with the spatial monitoring system 65 are preferably positioned within the subject vehicle 10 in relatively unobstructed positions and are capable of 3D sensing the associated field of view (FOV). The sensing device may be arranged to capture an image of an FOV located in front of the subject vehicle 10 in one embodiment. Alternatively or additionally, the sensing device may be arranged to capture an image of an FOV located adjacent to the subject vehicle 10 and / or behind the subject vehicle 10 in one embodiment. An image may include a plurality of RGB or grayscale pixels captured at a predetermined resolution in one embodiment. Each of these sensors provides an estimate of the actual position or state of an object, the estimate including an estimated position and standard deviation.Therefore, the sensory detection and measurement of object positions and conditions can instead be referred to as "estimates." The properties of these sensors may complement each other in that some may be more reliable than others at estimating certain parameters. The sensing devices may have different operating ranges and angular coverage, capable of estimating various parameters within their operating ranges. For example, radar sensors can estimate the range, range rate, and azimuth position of an object, but are typically not robust at estimating the size of a detected object. A camera with an image processor is more robust at estimating an object's shape and azimuth position, but may be less efficient at estimating an object's range and range rate.Scanning-type lidar sensors are efficient and accurate at estimating range and azimuth position, but typically cannot estimate the range rate and therefore may not be as accurate at detecting or recognizing new objects. Ultrasonic sensors are capable of estimating range, but may be less able to estimate or calculate the range rate and azimuth position. The performance of each of the aforementioned sensor technologies is affected by varying environmental conditions. For example, some of the sensor devices may exhibit parametric drift during operation, although overlapping sensor coverage areas create opportunities for sensor data fusion.Sensor data fusion involves combining sensor data or data derived from sensor data from different sources that observe a common field of view so that the resulting information is more accurate and precise than would be possible using these sources individually.

[0026] The HMI system 75 enables human-machine interaction to control the operation of an infotainment system, the GPS sensor 50, the vehicle navigation system 55, a remote service center, and the like. The HMI system 75 monitors operator requests and provides information to the operator, including the status of vehicle systems, service, and maintenance information. The HMI system 75 communicates with and / or controls the operation of a variety of in-vehicle operator interface devices. The HMI system 75 may also communicate with one or more devices that monitor biometric data associated with the vehicle operator, including, for example, gaze position, posture, and head position tracking.The HMI system 75 is depicted as a unitary device for ease of description, but may be implemented as a plurality of controllers and associated sensor devices in one embodiment of the system described herein. The in-vehicle operator interface device(s) may include devices capable of transmitting a message urging the operator to take action and may include an electronic visual display module, e.g., a liquid crystal display (LCD) device, a heads-up display (HUD), an audio feedback device, a wearable device, and a haptic seat.

[0027] The autonomous operating system 45 is arranged to enable autonomous vehicle operation. The autonomous operating system 45 includes a controller and one or more subsystems, which may include an autonomous steering system, an adaptive cruise control system, an autonomous braking / collision avoidance system, and / or other systems configured to control autonomous vehicle operation separately from or in conjunction with operator requests. Autonomous operating commands may be generated to control the autonomous steering system, the adaptive cruise control system, the autonomous braking / collision avoidance system, and / or the other systems. Vehicle operation includes operation in one of the drive modes in response to desired commands, which may include operator requests and / or autonomous vehicle requests.Vehicle operation, including autonomous vehicle operation, includes acceleration, braking, steering, steady-state operation, coasting, and idling. Operator requests can be generated based on operator inputs to an accelerator pedal, a brake pedal, a steering wheel, a transmission range selector, and a cruise control system. Vehicle acceleration includes a tip-in event, which is a request to increase vehicle speed—i.e., to accelerate the vehicle. A tip-in event can arise as an operator request for acceleration or as an autonomous vehicle request for acceleration.A non-limiting example of an autonomous vehicle request to accelerate may occur when a sensor for an adaptive cruise control system indicates that a vehicle can achieve a desired vehicle speed because an obstacle has been removed from a travel lane, such as might occur when a slow-moving vehicle exits a limited-access highway. Braking involves an operator request to reduce vehicle speed. Steady-state operation involves vehicle operation with the vehicle currently moving at a speed without an operator request to brake or accelerate, where the vehicle speed is determined based on the current vehicle speed and vehicle torque, vehicle wind and rolling resistance, powertrain inertial drag, or drag torque.Coasting involves vehicle operation where the vehicle speed is above a minimum threshold speed and the operator request to the accelerator pedal occurs at a point less than necessary to maintain the current vehicle speed. Idling involves vehicle operation where the vehicle speed is at or near zero. The autonomous operating system 45 includes a set of instructions executable to determine a trajectory for the subject vehicle 10 and to determine current and / or upcoming road conditions and traffic conditions based on the trajectory for the subject vehicle 10.

[0028] The term "controller" and related terms such as control module, module, controller, control unit, processor, and similar terms refer to one or more combinations of application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), electronic circuits, central processing units (CPUs), e.g., microprocessors, and associated non-volatile memory components in the form of storage and memory devices (read-only, programmable read-only, random access, hard disk, etc.). The non-volatile memory component is capable of storing machine-readable instructions in the form of one or more software or firmware programs or routines, combinational logic circuits, input / output circuits and devices, signal conditioning and buffer circuits, and other components that can be accessed by one or more processors to provide a described functionality.Input / output circuitry and devices include analog-to-digital converters and associated devices that monitor inputs from sensors, where these inputs are monitored at a preset sampling frequency or in response to a triggering event. Software, firmware, programs, instructions, control routines, code, algorithms, and similar terms refer to controller-executable sets of instructions, including calibrations and lookup tables. Each controller executes one or more control routines to provide the desired functions. Routines can be executed at regular intervals, e.g., every 100 microseconds during operation. Alternatively, routines can be executed in response to the occurrence of a triggering event.Communication between the controllers and communication between the controllers, actuators, and / or sensors can be via a direct wired point-to-point connection, a networked communication bus connection, a wireless connection, or any other suitable communication link. Communication includes the exchange of data signals in a suitable form, such as electrical signals via a conductive medium, electromagnetic signals via air, optical signals via fiber optic cables, and the like. The data signals can include discrete, analog, or digitized analog signals representing inputs from sensors, actuator commands, and communication between controllers. The term "signal" refers to a physically detectable indicator that conveys information and has a suitable waveform (e.g.electrical, optical, magnetic, mechanical or electromagnetic) such as DC, AC, sine wave, triangular wave, square wave, vibration and the like, which is capable of traveling through a medium.

[0029] The term "model" refers to processor-based or processor-executable code and associated calibration that simulates a physical existence of a device or physical process. As used herein, the terms "dynamic" and "dynamic" describe steps or processes that execute in real time and are characterized by monitoring or otherwise determining states of parameters and periodically or regularly updating the states of the parameters during execution of a routine or between iterations of execution of the routine. The terms "calibration," "calibrating," and related terms refer to a result or process that compares an actual or standard measurement associated with a device to a sensed or observed measurement or a predetermined position.A calibration as described herein may be reduced to a storable parametric table, a plurality of executable equations, or any other suitable form that may be used as part of a measurement or control routine.

[0030] A parameter is defined as a measurable quantity representing a physical property of a device or other element that can be detected using one or more sensors and / or a physical model. A parameter can have a discrete value, e.g., either "1" or "0," or it can be continuously variable.

[0031] The Fig. 2, Fig. 3, Fig. 4 and Fig. 5 schematically illustrate details related to the dynamic construction and updating of the vehicle navigation map 56 used by the in-vehicle navigation system 55. This concept involves using a crowdsourced vehicle sensor system to create and update a context-aware, sparse HD map augmented with semi-dynamic and static objects, enabling an augmented 3D view for vehicles over extended periods of time. Crowdsourcing involves sharing sparse data with the cloud and then accurately, robustly, and efficiently merging the data from multiple vehicles into a single sparse HD map representation. This includes a robust semantic segmentation approach to distinguish between dynamic, semi-dynamic, and static objects in the environment.

[0032] Fig. Figure 2 schematically illustrates a system architecture 200 that may be used in an embodiment of the subject vehicle 10 described with reference to Fig. 1, wherein the system architecture 200 utilizes a crowd-scan vehicle scanning system to generate and dynamically update a context-aware, sparse HD navigation map augmented with semi-dynamic and static objects. One goal of the novel system architecture 200 is to provide enhanced 3D vision for the subject vehicle 10 over large time periods, thereby improving controllability during operation of one or more subsystems of the autonomous operating system 45.

[0033] The system architecture 200 may be implemented in the vehicle using the various systems described above and the vehicle controller 15. The system architecture 200 includes a first segment 210 that generates a simple map representation 224 that may be communicated to a second segment 230. In one embodiment, the first segment 210 may be executed in the vehicle and the second segment 230 may be executed by the off-board controller 95, which may be a cloud-based computer system or a back-office computer system in conjunction with a service provider.

[0034] The first segment 210 involves acquiring a 3D image of a field of view proximal to the subject vehicle 10 via a scanning device of the spatial monitoring system 65 and simultaneously acquiring a corresponding GPS location via the GPS sensor 50 (212). A feature extraction program is executed to extract features from the 3D image 214. Feature extraction involves locating points of interest in the 3D image, such as corners or edges, and calculating or otherwise determining their feature descriptors using surrounding pixels in the 3D image.

[0035] The features extracted from the 3D image are subjected to semantic segmentation 216, which involves performing simultaneous localization and mapping (SLAM) of the extracted features from the 3D image and the associated semantic segmentation 218. An embodiment of a semantic segmentation routine is described with reference to Fig. 3 described.

[0036] Context extraction from the simultaneous localization and mapping of the extracted features is performed 220, which involves determining the object class to which each extracted feature belongs, e.g., a road surface, a building, a tree, a traffic sign, a traffic light, etc.

[0037] The navigation map 56 in the vehicle is updated based on the context extraction 222.

[0038] Based on the updated navigation map 224 in the vehicle, which contains stable features in the environment as well as their 3D positions, descriptors, semantic labels, and vantage points, a simple map representation is generated. Furthermore, generating a simple map representation based on the updated vehicle navigation map includes the execution of feature selection and a utility prioritization function.

[0039] The simple map display is transmitted to the off-board controller 95 (226, 232), which executes the second segment 230.

[0040] The first segment 210 can be implemented simultaneously in the controllers of one or more other vehicles, as schematically represented by vehicle 10' with an analog first segment 210' including analog steps from 212' to 226'. Thus, multiple vehicles can provide information to the second segment 230 in the form of simple map representations 224.

[0041] The second segment 230 can be executed by the off-board control 95, wherein the input is provided by one or more vehicles 10, 10', which transmit the simple map display 224, 224' thereto.

[0042] The second segment 230 contains the basic navigation map 96 (236) and update(s) (234) in the form of the simple map representations 224.

[0043] The base navigation map 96 (236) and the update(s) (234) are provided as inputs to a stitching module (238) that performs stitching operations using sparse map stitching representations to update relevant segments of the base navigation map 96 (240), and the updated base navigation map 96' is subsequently transmitted to the plurality of vehicles 10, 10' (242). The sparse map stitching module (238) may include a one-time relocation routine that is executed with reference to Fig. 4. Alternatively or additionally, the sparse map stitching module (238) may include a progressive relocation routine described with reference to Fig. 5 is described in detail.

[0044] A strictly generated map, such as the simple map representation 224, includes features and corresponding 3D positions at which the respective features were collected, also called key frames.

[0045] Fig. Figure 3 schematically shows details related to a semantic segmentation routine 300 that generates semantic labels and positions of semi-dynamic objects (360) based on stereo frame inputs 302 from the in-vehicle spatial monitoring system 65 and the semantic segmentation of extracted features from the 3D image 304 resulting from step (218) of Fig. 2. The semantic segmentation routine 300 executes a first routine to perform semantic labeling on a sparse strictness feature map (310), which generates inputs for a segmentation-driven semi-dynamic object detection routine (340), which includes generating the semantic labels and positions of semi-dynamic objects (360). The semantic segmentation semantically tags each perceived feature with the class of the object to which it belongs, e.g., a vehicle, a roadway, a building or other device, a pedestrian, etc.

[0046] This includes performing a simultaneous location and mapping (SLAM) process (321) to determine a 3D world position and performing an ORB keypoint detection process on the stereo frame inputs 302 (322), resulting in an ORB feature extraction (323) that generates an ORB descriptor. ORB (Oriented FAST and Rotated Brief) is a term that describes a fusion of the FAST keypoint detector and the BRIEF descriptor. FAST is a term that represents "features from accelerated segmentation testing" (FAST) and is a corner detection method that can be used to extract feature points and later track and map objects in image processing. BRIEF (Binary Robust Independent Elementary Features) is a general-purpose feature descriptor routine that can be used for object detection and 3D reconstruction.

[0047] The semantic segmentation of the features extracted from the 3D image 304 is input into a semantic image segmentation routine, e.g., DeepLab (324), the result of which is subjected to a 2D keypoint to 3D map point correspondence routine (325) to generate a semantic label.

[0048] The 3D world position from the SLAM process (321), the ORB descriptor from the ORB feature extraction (323), and the semantic label from the 2D keypoint-to-3D map point correspondence routine (325) are provided to a map point step (330) that generates a semantic label consensus (335) and provided to the inputs of a segment-driven semi-dynamic object detection routine (340) for generating the semantic labels and positions of semi-dynamic objects (360).

[0049] The segmentation-driven semi-dynamic object detection routine (340) performs the simultaneous localization and mapping (SLAM) (345) of the extracted features from the 3D image and the associated semantic segmentation using the semantic segmentation of the extracted features from the 3D image 304 and the semantic label consensus (335).This includes locating the features in the 3D map and performing a difference operation (350), which includes the steps of projecting features from the map 96 into the current scene, performing a feature match between the features of the map and the current scene, identifying new features in the current scene and identifying features not present in the current scene, performing a semantic label filter to identify semi-dynamic objects and static objects, and generating the semantic labels and positions of semi-dynamic objects (360). The difference operation (350) uses the base navigation map and the current 3D scene to determine which features have been moved, i.e., whether new features have been added or removed.With the help of the semantic labels that are attached to the features, the context of these objects can be inferred.

[0050] With further reference to Fig. 4 is made with continued reference to elements of Fig. 2 describes the one-time relocalization routine 400, which involves localizing a single keyframe from the vehicle navigation map 56 to the base navigation map 96, including the development of a transformation matrix. Data structures associated with each keyframe include an identifier (kpID), a pose (kpPose), and a list of map points (kpMPs), where each map point includes an identifier, a feature descriptor, a semantic label, an object class, a reference keyframe, and a world position.

[0051] The transformation matrix can then be used to transform the remaining keyframes from the vehicle navigation map 56 into the base navigation map 96. The base navigation map 96 is transferred to a storage device (402) and subjected to map reconstruction (410). At the same time, a new map segment, which may be in the form of the simple map representations 224, is loaded into a storage device (404).

[0052] The map reconstruction (410) includes executing a keyframe database and a co-visibility graph in conjunction with a full keyframe reconstruction to perform a map-level reconstruction of the base navigation map 96, referred to as the reconstructed base navigation map 412.

[0053] Coincidentally, the simple map representations 224 are subjected to a partial reconstruction (420) which includes determining a keyframe Bag of Visual Words (BoVW) vector and finding two-dimensional (2D) projections of the map points associated with the simple map representations 224 to determine a new map segment 422.

[0054] The reconstructed base navigation map 412 output from the map reconstruction step (410) and the new map segment 422 output from the partial reconstruction step (420) are provided as inputs to a one-shot stitching module 430. The one-shot stitching module 430 includes a relocation step 440, a keyframe addition step 450, and a global transformation step 460, which combine to form a robust, rigorously generated, stitched base navigation map 96.

[0055] The relocalization step 440 includes the execution of a BoVW keyword matching routine, a map point matching routine, a PnP (perspective-n-point) solver routine, a RANSAC routine, and a pose optimization routine from which a transformation matrix 445 is generated.

[0056] The BoVW keyword matching routine involves matching keywords between the reconstructed base navigation map 412 and the new map segment 422.

[0057] The map-point matching routine involves feature matching of keypoints based on their descriptors and their 3D positions.

[0058] The PnP solver routine refers to estimating the pose of the camera, ie, the position and perspective of the sensor of the in-vehicle spatial monitoring system 65, based on the 3D points in an image and the corresponding 2D projections in an image.

[0059] The RANSAC (Random Sample Consensus) routine detects and removes outliers in the dataset through an iterative process that estimates parameters in an associated mathematical model from a set of observed data.

[0060] The pose optimization routine involves using 2D keypoints observed by the sensor together with 3D coordinates in the map to optimize the calculated pose of the camera, radar sensor, or lidar sensor.

[0061] The keyframe addition step 450 uses the transformation matrix 445 generated by the relocation step 440 to transform map points into the base navigation map 96, match map points with those in the base navigation map 96, and remove redundant map points.

[0062] The global transformation step 460 transforms the next keyframe into the base navigation map 96 to generate an updated version of the base navigation map 96.

[0063] With further reference to Fig. 5, with further reference to elements of Fig.3, the progressive relocalization routine 500 involves localizing the keyframes collected from the subject vehicle 10 using a fusion of BoVWs (Bag of Visual Words), feature mapping, and PnP solver modules. This involves transferring the base navigation map 96 to a storage device (502) and subjecting it to map reconstruction (510). The map reconstruction (510) involves executing a keyframe database and a co-visibility graph in conjunction with a complete keyframe reconstruction to perform a map-level reconstruction of the base navigation map 96, referred to as the reconstructed base navigation map 512.Coincidentally, the simple map representations 224 are subjected to a partial reconstruction (520) which includes determining a keyframe Bag of Visual Words (BoVW) vector and finding two-dimensional (2D) projections of the map points associated with the simple map representations 224 to determine a new map segment 522.

[0064] The reconstructed base navigation map 512 output from the map reconstruction step (510) and the new map segment 522 output from the partial reconstruction step (520) are provided as inputs to a progressive relocation stitching module 530. The one-shot stitching module 530 includes a relocation step 540 and a keyframe addition step 550, which combine to produce a robust, rigorously generated, stitched base navigation map 96. Simultaneously, a new map segment, such as may be in the form of the simple map representations 224, is loaded into a storage device (504).

[0065] The relocalization step 540 includes the execution of a BoVW keyword matching routine, a map point matching routine, a PnP (perspective-n-point) solver routine, a RANSAC routine, and a pose optimization routine from which a transformation matrix 545 is generated.

[0066] The BoVW keyword matching routine involves matching keywords between the reconstructed base navigation map 512 and the new map segment 522.

[0067] The map point matching routine involves matching corresponding features from another frame of the same vehicle or matching corresponding frames from different vehicles of the same geographical location.

[0068] The PnP solver routine refers to estimating the pose of the camera, ie, the position and perspective of the sensor of the in-vehicle spatial monitoring system 65, based on the 3D points in an image and the corresponding 2D projections in an image.

[0069] The RANSAC (Random Sample Consensus) routine detects and removes outliers in the dataset through an iterative process that estimates parameters in an associated mathematical model from a set of observed data.

[0070] The keyframe addition step 550 uses the transformation matrix 545 generated by the relocation step 540 to transform map points into the base navigation map 96, match map points with those in the base navigation map 96, and remove redundant map points. The keyframe addition step 550 further constructs a BoVW vector, which it adds to a keyframe database, and then adds the keyframe to the base navigation map 96 to generate an updated version of the base navigation map 96.

[0071] In this way, the updated base navigation map 96' is subsequently communicated to the plurality of vehicles 10, 10' based on the simple map display, which includes updating the vehicle navigation map 56 based on the cloud-based navigation map 96.

[0072] The concepts described herein utilize application programming interfaces (APIs) to facilitate the uploading and downloading of data between the subject vehicle 10 and the off-board controller 95. One API is a vehicle-to-cloud API, which involves uploading data from the subject vehicle 10 to the off-board controller 95. These include: GetMapSegment, which includes the GPS position of the subject vehicle 10; GetMapUpdates, which includes the GPS / SLAM position; and GetSemiDynamicObjects, which includes the GPS / SLAM position. The vehicle-to-cloud API also involves uploading data from the off-board controller 95 to the subject vehicle 10. These include: UploadMapSegment, which includes the corresponding map segment; and UpdateMapSegment, which includes the map update, with associated functions.

[0073] A second API is a Cloud-to-Vehicle API, which includes downloading data from the off-board controller 95 to the subject vehicle 10. This includes: GetMapSegmentUpdates, which includes the GPS position and vehicle ID of the subject vehicle 10; and GetSemiDynamicObjects, which also includes the GPS position and vehicle ID of the subject vehicle 10. The Cloud-to-Vehicle API also includes uploading data from the off-board controller 95 to the subject vehicle 10. This includes: UploadMapUpdates, which includes the GPS position of the subject vehicle 10; and UpdateMapSegment, which includes the map segment and the update.

[0074] The concepts described herein provide a robust semantic segmentation approach for distinguishing between dynamic, semi-dynamic, and static objects in the environment. It also utilizes semantically labeled environmental features embedded in the sparse feature maps to detect temporal changes in the environment. In one embodiment, the concepts described herein can outperform other sampling systems by crowdsourcing sparse data to the cloud via V2X from a multitude of vehicles and then accurately, robustly, and efficiently stitching the data from multiple vehicles into a single sparse HD map representation. In this way, efficiencies in wireless bandwidth usage, map collection time, and sampling in the presence of sparse data can be achieved.

[0075] The concepts utilize visual simultaneous localization and mapping (SLAM) and semantic segmentation tools, taking into account the accuracy of vision SLAM, semantic segmentation, and the capacity / latency of V2X communication technologies. Such operation can enable operation within the limitations of LTE / 4G communication technology. Furthermore, the cost-saving measure removes non-essential map components without which the map's integrity remains intact.

[0076] The teachings may be described herein with respect to functional and / or logical block components and / or various processing steps. It should be noted that such block components may consist of hardware, software, and / or firmware components configured to perform the specified functions. Embodiments may also be implemented in cloud computing environments. The flowchart and block diagrams in the flowcharts illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical functions.It should also be understood that each block of the block diagrams and / or flowchart representations, and combinations of blocks in the block diagrams and / or flowchart representations, may be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. These computer program instructions may also be stored in a computer-readable medium that can direct a controller or other programmable data processing apparatus to function in a particular manner such that the instructions stored on the computer-readable medium produce an article of manufacture, including instructions for implementing the function / act specified in the flowchart and / or block diagram or blocks.

[0077] The detailed description and the drawings or figures are supportive and descriptive of the present teachings, but the scope of the present teachings is defined solely by the claims. While some of the preferred embodiments and other embodiments for carrying out the present teachings have been described in detail, various alternative designs and embodiments exist for practicing the present teachings defined in the appended claims.

Claims

[1] A method of controlling a subject vehicle, the subject vehicle including a first controller, an auxiliary vehicle communication system, a GPS sensor, a spatial monitoring system, and a navigation system using an in-vehicle navigation map, the method comprising: Capturing, via the spatial monitoring system, a 3D sensor representation of a field of view in the vicinity of the vehicle in question and an associated GPS position; Executing a feature extraction routine to extract features from the 3D sensor representation; Performing semantic segmentation of the extracted features from the 3D sensor representation; Performing simultaneous localization and mapping of the extracted features from the 3D sensor representation and the associated semantic segmentation of the extracted features from the 3D sensor representation; Performing context extraction from the simultaneous localization and mapping of the extracted features; Updating the vehicle navigation map based on context extraction; Generating a simple map display based on the updated vehicle navigation map; transmitting the simple map representation to a second controller, wherein the second controller is an off-board controller; Executing, via the second control, sparse map stitching to update a base navigation map based on the simple map display; and Updating the vehicle navigation map based on the base navigation map wherein performing, via the second controller, sparse map stitching to update a cloud-based navigation map based on the simple map representation comprises performing a one-time relocation scheme, wherein performing the simultaneous localization and mapping of the extracted features from the 3D sensor representation and the associated semantic segmentation comprises adding a semantic segmentation for simultaneous localization and mapping (SLAM) to the vehicle navigation map. [2] The method of claim 1, further comprising controlling operation of the subject vehicle based on the updated vehicle navigation map. [3] The method of claim 2, wherein the subject vehicle further comprises an autonomous operating system, and wherein controlling operation of the subject vehicle based on the updated vehicle navigation map comprises controlling the autonomous operating systems based on the updated vehicle navigation map. [4] The method of claim 1, wherein generating a simple map representation based on the updated vehicle navigation map comprises performing feature selection and utility prioritization capability. [5] The method of claim 1, wherein performing, via the second controller, sparse map stitching to update a cloud-based navigation map based on the simple map representation comprises performing a progressive relocation scheme. [6] The method of claim 1, wherein performing, via the second controller, sparse map stitching to update a cloud-based navigation map based on the simple map representation comprises performing sparse map stitching using a sparse representation of a map segment. [7] The method of claim 1, wherein the off-board control is cloud-based. [8] The method of claim 1, wherein the off-board controller comprises a remotely located back-office controller.

Images