Vehicle location and mapping in low-light environments

Abstract

A localization and mapping system and method for a motor vehicle is disclosed that includes at least one camera configured to capture images of an environment surrounding the motor vehicle, at least one sensor configured to capture location information for objects surrounding the motor vehicle, and a controller configured to receive images captured by the at least one camera and location information captured by the at least one sensor, the controller enhances the captured images utilizing a neural network and combines the enhanced images with the location information to locate the vehicle within the mapped environment.

Description

Technical Field

[0001] The present disclosure relates to a system and method for localizing and mapping the environment around a vehicle.

[0002] Background Autonomous and semi-autonomous vehicles continuously collect and update information to determine the position and orientation of the vehicle. In some cases, the vehicle operates in an unfamiliar and unknown low-light environment. The unknown low-light environment poses challenges to vehicle sensors and localization algorithms. Images taken in low-light environments may not provide clear pictures of the surrounding environment that are useful for vehicle localization and orientation.

[0003] The description of the background art presented herein is intended to generally introduce the context of the present disclosure. As far as described in this background art section, the achievements of the inventors recited herein, as well as aspects of this document that might not otherwise be suitable as prior art at the time of filing, are not recognized as prior art that explicitly or implicitly opposes the present disclosure.

[0004] Summary A localization and mapping system for a motor vehicle according to an exemplary embodiment disclosed includes, among other things, at least one camera configured to acquire an image of the environment around the motor vehicle, at least one sensor configured to acquire position information regarding an object around the motor vehicle, and a controller. The controller receives the image captured by the at least one camera and the position information acquired by the at least one sensor, enhances the captured image using a neural network to generate an enhanced image, combines the enhanced image with the position information, and is configured to localize the vehicle based on the combined enhanced image and position information.

[0005] In another exemplary embodiment of the aforementioned localization and mapping system, the neural network includes a database having low-light images and corresponding ground truth images.

[0006] In another exemplary embodiment of the aforementioned localization and mapping systems, the controller is configured to apply weighting coefficients to a loss function applied to a sequence of images in order to enhance the images.

[0007] In another exemplary embodiment of one of the aforementioned localization and mapping systems, weighting coefficients are applied to the pixel-wise mean squared error loss to enhance the captured image.

[0008] In another exemplary embodiment of the aforementioned localization and mapping systems, the weighting coefficients are biased toward the application of enhancements based on a more recent comparison between the captured image and the ground truth image.

[0009] In another exemplary embodiment of the aforementioned localization and mapping systems, the enhanced output image is combined with location information to generate a map of the environment surrounding the motorized vehicle.

[0010] In any other exemplary embodiment of the aforementioned localization and mapping systems, at least one sensor includes at least one radar sensing device mounted on a motorized vehicle.

[0011] In any other exemplary embodiment of the aforementioned localization and mapping systems, at least one sensor includes at least one ultrasonic sensor mounted on the vehicle.

[0012] Another exemplary embodiment of any of the aforementioned localization and mapping systems further includes at least one sensor that generates information about vehicle operating parameters, and a controller is further configured to combine the information about vehicle operating parameters with enhanced images in order to localize the vehicle.

[0013] In any other exemplary embodiment of the aforementioned localization and mapping systems, the controller is configured to implement a SLAM (simultaneous localization and mapping) algorithm using enhanced images, information about the locations of objects around the vehicle, and information about vehicle operation parameters to localize and map the environment around the motorized vehicle.

[0014] A method for localizing and mapping a vehicle in a low-light environment, according to another disclosed exemplary embodiment, includes, among other things, acquiring a sequence of images of the environment surrounding the vehicle; enhancing the low-light images using a low-light image enhancement model to generate a sequence of enhanced images; and generating a map of the environment surrounding the vehicle based on the sequence of enhanced images.

[0015] Another exemplary embodiment of the method described above further includes generating a low-light image enhancement model to enhance the low-light image using a neural network that receives a comparison of the low-light image and the ground truth image.

[0016] Another exemplary embodiment of any of the methods described above further includes applying a weighting factor that biases the application of the correction factor towards the correction factor formed from the more recent image.

[0017] In another exemplary embodiment of any of the methods described above, the weighting coefficients apply greater importance to the most recent image and disregard past images.

[0018] Another exemplary embodiment of any of the methods described above further includes obtaining information from at least one sensor configured to obtain location information about objects around a motorized vehicle, and combining the location information with enhanced images to generate a map of the environment around the vehicle.

[0019] Another exemplary embodiment of any of the methods described above further includes receiving at least one sensor that generates information about vehicle operating parameters, and combining the information about vehicle operating parameters with enhanced images to locate the vehicle.

[0020] While various different examples may have specific components as shown in the drawings, embodiments of this disclosure are not limited to these specific combinations. It is possible to use components or features from one embodiment of multiple embodiments in combination with features or components from another embodiment of multiple embodiments.

[0021] The features disclosed herein and other features are best understood from the following specification and drawings, the following being a brief description of the drawings. [Brief explanation of the drawing]

[0022] [Figure 1] This is a schematic diagram of a vehicle equipped with a location identification and mapping system. [Figure 2] This is a schematic diagram showing the captured input image and the enhanced output image. [Figure 3] This is a schematic diagram illustrating the enhancement of input images using a neural network. [Figure 4] This is a flowchart illustrating an exemplary method for generating a model to enhance low-light images. [Figure 5] This is a flowchart illustrating an exemplary method for localizing and mapping the environment surrounding a vehicle using enhanced image sequences.

[0023] Detailed Description Referring to FIG. 1, vehicle 20 is schematically shown, and vehicle 20 includes a system 25 for localizing and mapping the surrounding environment and the position and orientation of the vehicle in this surrounding environment. Vehicles include more and more autonomous and / or semi-autonomous driver assistance functions. Driver assistance functions utilize environmental information to operate. The disclosed exemplary embodiment of system 25 provides useful information for localizing and mapping the environment around the vehicle without requiring a significant increase in processing power, by enhancing a sequence of images acquired in low light situations.

[0024] Exemplary vehicle 20 includes a camera 24 and at least one other sensing device. In the disclosed example, the sensing device includes a radar device 22 disposed at various different locations around the vehicle. Camera 24 and radar device 22 provide information to controller 28. Although radar device 22 is shown as an example, other sensing devices can also be utilized within the spirit and scope of the present disclosure.

[0025] Images captured by camera 24 may not provide the maximum useful information in a low light environment. Exemplary controller 28 includes an algorithm for enhancing the captured sequence of low light images. The enhanced images are then utilized by a localization and mapping algorithm, such as, for example, a simultaneous localization and mapping (SLAM) algorithm 34. SLAM 34 uses this enhanced image, along with information captured from other vehicle sensors and information collection systems and devices, to generate mapping information.

[0026] The controller 28 may be part of the overall vehicle controller and / or may be a dedicated controller for the exemplary system 25. The controller 28 is configured to process information received from the radar device 22, camera 24, global positioning system device 30 and information from various vehicle operating systems schematically shown by reference numeral 26 to determine the position and orientation of the vehicle 20 within the surrounding environment.

[0027] The SLAM algorithm 34 is executed by the controller 28 of the vehicle 20. The controller 28 is schematically shown and includes at least one processor and a memory device 36. The controller 28 may be a hardware device for executing software, in particular software stored in the memory 36. The processor may be a custom-made or commercially available processor, a central processing unit (CPU), an auxiliary processor one of several processors associated with a computing device, a semiconductor-based microprocessor (in the form of a microchip or chipset), or generally any device for executing software instructions.

[0028] The memory 36 may include one or a combination of volatile memory elements (e.g., random access memory (RAM such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements. Furthermore, the memory 36 may incorporate electronic, magnetic, optical, and / or other types of storage media. It should be noted that the memory may have a distributed architecture, in which case multiple different components are located geographically separated from each other but are accessible by the processor.

[0029] The software in memory 36 may contain one or more separate programs, each of which contains an ordered list of executable instructions for implementing disclosed logical functions and operations. System components embodied as software can also be interpreted as source programs, executable programs (object code), scripts, or any other entity containing a set of instructions to be executed. If a program is constructed as a source program, it is translated through a compiler, assembler, interpreter, or similar, which may or may not be contained in memory.

[0030] Input / output devices (not shown) that may be coupled to the system I / O interface may include, but are not limited to, input devices such as keyboards, mice, scanners, microphones, cameras, proximity devices, etc. Furthermore, input / output devices may also include output devices such as, but are not limited to, printers, displays, etc. Finally, input / output devices may further include devices that communicate as both input and output units, such as, but are not limited to, modulators / demodulators (modems; for accessing other devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc.

[0031] The controller 28 can control the vehicle system to autonomously control the vehicle 20 or to provide driver assistance functions to assist the operator of the vehicle 20.

[0032] Referring to Figure 2 while continuing to refer to Figure 1, an exemplary low-light image 38 is shown, which is input to a neural network model 32 for enhancement. The neural network model 32 utilizes visual enhancement techniques to adjust the image on a pixel-by-pixel basis. The resulting enhanced output image 40 provides greater contrast and sharpness to objects within the image and in the vicinity of the vehicle 20. The neural network model 32 is constructed using past and current images, along with a corresponding ground truth image. The functions and features required to enhance the low-light image are learned by using matched pairs of the low-light image and the ground truth image, and then applied to the current image to obtain the enhanced output image.

[0033] Referring to Figure 3, while continuing to refer to Figures 1 and 2, a convolutional neural network 42 is used to generate a model 32 for enhancing the captured low-light image. The input to the convolutional neural network 42 includes a matched pair of low-light image 44 and ground truth image 46. The disclosed image enhancement is described in the following section:

number

[0034] In the above equation, L total This is the loss function that should be minimized by the convolutional neural network.

number

number

[0035] In the above equation, W and H are the width and height of the input image,

number

number

number

number

number

number

[0036] The loss function described above does not weight any of the images. The illustrated enhancement disclosed is given by the following equation.

number

[0037] L is up to the last numbered image n.

number

[0038] Referring to Figure 4 while continuing to refer to Figures 1 to 3, the generation of the neural network model 32 is schematically shown. Low-light images 44 and the corresponding ground truth images 46 are input to the convolutional neural network 42 in order to train and develop the neural network model indicated by reference numeral 32 for enhancing low-light images. The convolutional neural network applies a loss function along with weight coefficients to continuously update and improve the model 32. Continuous improvement of the model is possible during the initial training process and can continue throughout the operation of the vehicle.

[0039] Referring to Figure 5, continuing with Figure 4, the disclosed exemplary method fuses images from camera 24 with information from GPS 30, radar equipment, and vehicle sensors 26 to generate and locate the vehicle 20 in a self-generated map. During low-light conditions, low-light images 58 from camera 24 are processed by a neural network model 32 to generate an enhanced image 60. The enhanced image 60 is communicated to a SLAM algorithm 34 to generate a localized map 64 and the orientation and position of the vehicle as indicated by reference numeral 62 within this map 64. The vehicle orientation, position, and direction of travel, along with any other positional and dynamic vehicle orientation information, are required for the navigation and operation of driver assistance functions.

[0040] Therefore, the disclosed exemplary localization and mapping system enhances low-light imagery to provide useful information for guiding and positioning vehicles in unfamiliar low-light environments.

[0041] While various non-limiting embodiments are shown having certain components or steps, the embodiments of this disclosure are not limited to any particular combination thereof. It is possible to use some components or features from any non-limiting embodiment in combination with features or components from any other non-limiting embodiment.

[0042] It should be understood that, across multiple drawings, similar reference numerals identify corresponding or similar elements. While specific component arrangements are disclosed and illustrated in these exemplary embodiments, it should be understood that other arrangements may also benefit from the teachings of this disclosure.

[0043] The foregoing description should be interpreted as illustrative and not in any restrictive sense. Those skilled in the art will understand that certain modifications may be included within the scope of this disclosure. For these reasons, the following claims should be studied to define the true scope and content of this disclosure.

Claims

1. A location identification and mapping system for motorized vehicles, wherein the location identification and mapping system is At least one camera configured to acquire images of the environment surrounding the motor vehicle, At least one sensor configured to acquire positional information of objects surrounding the motorized vehicle, Controller and Includes, The controller is configured to receive an image captured by the at least one camera and location information acquired by the at least one sensor, enhance the captured image using a neural network to generate an enhanced image, combine the enhanced image with the location information, and determine the location of the motorized vehicle based on the combined enhanced image and location information. The neural network includes a database having low-light images and corresponding ground truth images. The controller is configured to apply weighting coefficients to a loss function applied to a sequence of images in order to enhance the images. The loss function includes the mean squared error loss in pixels between the low-light image and the ground truth image, The weighting coefficients are biased toward the application of enhancements based on a more recent comparison between the captured image and the ground truth image. Location identification and mapping system.

2. The location identification and mapping system according to claim 1, wherein the enhanced image is combined with the location information to generate a map of the environment surrounding the motorized vehicle.

3. The location identification and mapping system according to claim 1, wherein the at least one sensor includes at least one radar sensing device attached to the motor vehicle.

4. The location identification and mapping system according to claim 1, wherein the at least one sensor includes at least one ultrasonic sensor attached to the motor vehicle.

Images