Processor-implemented method and system for generating a high definition (HD) map for high precision position estimation and map maintenance

Abstract

A processor-implemented method and system for generating a high-definition (HD) map for high-precision position estimation and map maintenance is provided. The method includes mapping an input red-green-blue (RGB) data into a latent space vector representing an image location and an orientation in a three-dimensional (3D) space. The method further includes mapping the image location and the orientation of the input RGB data to a high-dimensional synthesized RGB image. The method further includes acting as a discriminator in a generative adversarial setup by utilizing real-time RGB image data and the high dimensional synthesized RGB image based on the image location for determining whether the HD map needs to be updated. The method further includes operating as a convolution neural network (CNN) binary classifier, for determining whether the input RGB image should be included in a stack for an offline model retraining.

Description

PROCESSOR-IMPLEMENTED METHOD AND SYSTEM FOR GENERATING A HIGH DEFINITION (HD) MAP FOR HIGH PRECISION POSITION ESTIMATION AND MAP MAINTENANCECROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the provisional patent application titled “PROCESSOR-IMPLEMENTED METHOD AND SYSTEM FOR GENERATING A HIGH DEFINITION (HD) MAP FOR HIGH PRECISION POSITION ESTIMATION AND MAP MAINTENANCE”, with application number 63 / 722,204, filed in the United States Patent and Trademark Office on November 19, 2024. The specification of the above referenced patent application is incorporated herein by reference in its entirety.BACKGROUNDTechnical Field

[0002] The embodiments herein generally relate to the field of high-definition maps, and more particularly relates to processor-implemented method and system for generating a high definition (HD) map for high precision position estimation and map maintenance.Description of the Related Art

[0003] Typically, the conventional way of formatting and storing HD Maps is to store information, features, and data from 3D LiDAR scanners and RGB images. These maps are highly structured and expandable in terms of size and data included. The other problem is the map updates, comparing already existing map features and data to new data and then updating the features.

[0004] Hence there is a need to generate machine learning based HD map for high precision position estimation and map maintenance.

[0005] The above-mentioned shortcomings, disadvantages and problems are addressed herein, and which will be understood by reading and studying the following specification.SUMMARY

[0006] This summary is provided to introduce a selection of concepts in a simplified form that are further disclosed in the detailed description. This summary is not intended to determine the scope of the claimed subject matter.

[0007] In an aspect, a processor-implemented method for generating a high definition (HD) map for high precision position estimation and map maintenance is provided. The method includes mapping, by a localization encoder module, an input red green blue (RGB) data into a latent space vector representing an image location and an orientation in a three-dimensional (3D) space, using a deep neural network including of one or more layers of Convolutional Long Short- Term Memory (ConvLSTM) layers. The method further includes mapping, by a reconstruction decoder module, the image location, and the orientation of the input RGB data to a high dimensional synthesized RGB image, by using the deep neural network including one or more transposed ConvLSTM layers. The method further includes acting, by an updating module including the deep neural network model, as a discriminator in a generative adversarial setup by utilizing a real-time RGB image data and the high dimensional synthesized RGB image based on the image location for determining whether the HD map needs to be updated. The method further includes operating, by a discriminator module, as a convolution neural network (CNN) binary classifier, for determining whether the input RGB image should be included in a stack for an offline model retraining.

[0008] According to an embodiment, the method further includes localization, by a map module, one or more Al maps for a high precision position estimation and the HD map maintenance by tilling the generated HD map into one or more small tiles, training a neural network model using the small HD map tiles on a pose regression task, retraining the HD map by creating a new and separate checkpoint for every map tile and communicating to a vehicle one or more VBL model parameters to use in pose regression. The Al maps comprises a new format for HD Maps, that does not comprise of structured visual and geometrical dataset in the form of a database, but a framework of Al models that cooperate to provide the same functionality of an HD Map directly without the need of an ever expanding memory footprint for the data.

[0009] According to an embodiment, the method further includes visualization, by a visualization module, of Al maps for the high precision position estimation and the HD map maintenance by training a decoder network from a variational autoencoder (VAE) trained on each maptile to retrieve original map features from a smaller space of 6 DoF- pose and proposing an additional use of generative adversarial networks (GANs) to generate the visuals of the map and assigning the generative model with the task of generating views for arbitrary poses within the map tile.

[0010] According to an embodiment, by training the VAE on the HD-Map data during the initial stages, the HD-map learns to encode essential spatial information in a compact latent space, facilitating efficient processing and storage.

[0011] According to an embodiment, the method further includes utilizing by a VAE decoder a mean output from the encoder to synthesize one or more views like images captured using a vehicle camera.

[0012] According to an embodiment, the synthetic view generation enables a system to predict the environmental conditions appearing from the vehicle's perspective, aiding in navigation and decision-making processes.

[0013] According to an embodiment, the decision-making process is essential for ensuring that relevant data is utilized to update the Al Maps, optimize resource allocation, and maintain map accuracy over time. The decision-making process for map updates includes that a generative adversarial setup of two models; a generator and a discriminator. The generator is used to generate the synthetic visual representation of the surrounding, and then the discriminator is used to decide whether the synthetic visuals match the real-time visual input from cameras. If the two are visually similar enough, the decision is that this area of the map does not need updates. The discriminator model could give area-specific recommendations for updates, and with enough changes that prevents the reality and the synthetic scenes from being similar, this real-time visual will be used for later updating the Al map. This implies using Al model training methods to update the Al map pre-trained models’ weights by methods of deep learning model training.

[0014] According to an embodiment, actual map updates are performed by training the encoder-decoder setup as a denoising process for the VAE.

[0015] In another aspect, a system for generating a high definition (HD) map for high precision position estimation and map maintenance is provided. The system includes a processor is configured to fetch and execute computer-readable instructions stored in the memory. The system further includes a memory configured to store one or more computer-readable instructions or routines in a non-transitory computer-readable storage medium, which may be fetched and executedto create or share data packets over a network service. The system further includes a processing engine configured to generate a HD map for high-precision position estimation and map maintenance including at least one of a localization encoder module, a reconstruction decoder module, an updating module, and a discriminator module. The system further includes a localization encoder module for mapping an input red green blue (RGB) data into a latent space vector representing an image location and an orientation in a three-dimensional (3D) space, using a deep neural network including of one or more layers of ConvLSTM layers. The system further includes a reconstruction decoder module for mapping the image location and the orientation of the input RGB data to a high dimensional synthesized RGB image, by using the deep neural network including one or more transposed ConvLSTM layers. The system further includes an updating module including the deep neural network model for acting as a discriminator in a generative adversarial setup by utilizing a real-time RGB image data and the high dimensional synthesized RGB image based on the image location for determining whether the HD map needs to be updated. The system further includes a discriminator module for operating as a convolution neural network (CNN) binary classifier, for determining whether the input RGB image should be included in a stack for an offline model retraining.

[0016] According to an embodiment, the system further includes a map module configured for tilling the generated HD map into one or more small tiles, training a neural network model using the small HD map tiles on a pose regression task, retraining the HD map by creating a new and separate checkpoint for every map tile and communicating to a vehicle one or more VBL model parameters to use it in pose regression.

[0017] According to an embodiment, the system further includes a visualization module configured for training a decoder network using a variational autoencoder (VAE) trained on each map tile to retrieve original map features from a smaller space of 6 DoF- pose and proposing an additional use of generative adversarial networks (GANs) to generate the visuals of the map and assigning the generative model with the task of generating one or more views for arbitrary poses within the map tile.

[0018] According to an embodiment, the VAEs is configured for learning efficient representations of input data and training the VAE on the HD-Map data during the initial stages enables the HD-map to learn to encode essential spatial information in a compact latent space, facilitating efficient processing and storage.

[0019] According to an embodiment, the system further includes a VAE decoder that utilizes a mean output from the encoder to synthesize one or more views like images captured using a vehicle's cameras.

[0020] According to an embodiment, the visualization module performs actual map updates by training the encoder-decoder setup as a denoising process for the VAE.

[0021] The present technology offers several advantages over other known alternatives in the realm of autonomous vehicle navigation and mapping technology. By utilizing a generative variational auto-encoder (VAE) within a GAN framework, the algorithm optimizes storage space by replacing HD-Map data with a smaller, more efficient Al Map. The present technology reduces storage requirements while maintaining essential navigational information, enhancing overall system efficiency. The use of crowdsourced data and offline re-training for Al Map updates eliminates the need for dedicated mobile mappers, significantly reducing operational costs associated with map maintenance. Additionally, leveraging existing GAN training resources minimizes development expenses by avoiding the need to build systems from scratch. The proposed framework simplifies the map update process by employing a CNN binary classifier as the discriminator model. The present technology efficiently determines relevant data for model retraining, streamlining the update process and reducing complexity compared to traditional methods. Through the integration of visual-based localization (VBL) techniques and semantic segmentation models, this invention improves the accuracy of pose estimation and map updates. By focusing on relevant image features and changes, the system enhances the precision of autonomous vehicle navigation, mitigating potential errors and improving overall performance. The present technology allows for the seamless integration of new data points into Al Maps without increasing storage requirements. This adaptability ensures that the maps remain up-to-date with evolving environments and road conditions, enhancing the reliability and safety of autonomous driving experiences. The present technology provides a setup and process of considering an Al model as a feasible alternative for an HD Map. The claim is mainly that whatever functionality the HD Map provides, that process and these models can provide the same functionality with less memory space and an easier way to update and maintain the map and its outputs.

[0022] It is to be understood that the aspects and embodiments of the disclosure described above may be used in any combination with each other. Several of the aspects and embodiments may be combined to form a further embodiment of the disclosure.

[0023] The foregoing summary is illustrative only and is not intended to be in any way limiting. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description.

[0024] These and other objects and advantages will become more apparent when reference is made to the following description and accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The other objects, features and advantages will occur to those skilled in the art from the following description of the preferred embodiment and the accompanying drawings in which:

[0026] FIG. 1 illustrates an exemplary block diagram of a system of machine learning based high-definition (HD) map for high precision position estimation and map maintenance, in accordance with an embodiment of the present technology.

[0027] FIG. 2A depicts a block diagram of an encoder used to generate artificial intelligence (Al) HD Maps, in accordance with an embodiment.

[0028] FIG. 2B depicts a block diagram of a decoder used to generate Al HD Maps, in accordance with an embodiment.

[0029] FIG. 2C depicts a block diagram of a discriminator used to generate Al HD Maps, in accordance with an embodiment.

[0030] FIG. 3A illustrates a flow diagram depicting a method for localization of Al maps for high precision position estimation and HD map maintenance, in accordance with an embodiment.

[0031] FIG. 3B illustrates a flow diagram depicting a method for visualization of Al maps for high precision position estimation and HD map maintenance, in accordance with an embodiment.

[0032] FIG. 4 illustrates a flowchart of a processor-implemented method for generating a high definition (HD) map for high precision position estimation and map maintenance, in accordance with an embodiment.

[0033] FIG. 5 illustrates an exemplary computer system in which or with which embodiments of the present disclosure may be implemented.

[0034] Although the specific features of the embodiments herein are shown in some drawings and not in others. This is done for convenience only as each feature may be combined with any or all the other features in accordance with the embodiments herein.DETAILED DESCRIPTION OF THE DRAWINGS

[0035] The detailed description of various exemplary embodiments of the disclosure is described herein with reference to the accompanying drawings. It should be noted that the embodiments are described herein in such details as to clearly communicate the disclosure. However, the number of details provided herein is not intended to limit the anticipated variations of embodiments; on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present disclosure as defined by the appended claims.

[0036] It is also to be understood that various arrangements may be devised that, although not explicitly described or shown herein, embody the principles of the present disclosure. Moreover, all statements herein reciting principles, aspects, and embodiments of the present disclosure, as well as specific examples, are intended to encompass equivalents thereof.

[0037] While the disclosure is susceptible to various modifications and alternative forms, specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below. It should be understood, however that it is not intended to limit the disclosure to the forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0038] The various embodiments of the present technology provide an efficient technique for the generation of machine learning based HD map for high-precision position estimation and map maintenance.

[0039] The present technology addresses the issue of a sustainable method of formatting an HD Map in a format that can directly be used to achieve the same functionality as a HD Map but without the need of a huge memory footprint. The methodology is built on the fact that a selfsupervised training method for an Al model can easily replace the problem of map updates after- the-fact processing-hungry methods with an online method that is well integrated within the map main features of vehicle localization and environment visualization. The present technology represents a pioneering advancement in autonomous vehicle navigation technology, targeting multi-pie crucial objectives. By leveraging cutting-edge machine learning algorithms, particularly visual-based localization (VBL), the innovation significantly enhances the precision of vehicle positioning systems. The HD map format used in this technology enables transforming all HD Map data from sitting data in a data-base format, to a live and updatable orchestra of Al models. These models serve as the actual HD map with abilities to expand features beyond localization, visualization, and map updates. Moreover, this invention addresses the challenge of storage requirements associated with high-definition (HD) maps, proposing novel techniques to reduce storage footprint while retaining essential navigational data. This optimization not only streamlines resource utilization but also facilitates seamless integration of HD maps into autonomous vehicle systems. Furthermore, this technology introduces perception-based localization techniques, empowering vehicles to orient themselves within their surroundings intuitively. By leveraging visual cues and machine learning, the present technology enhances adaptability and responsiveness, enabling vehicles to navigate diverse environments with confidence. A key feature of the present system is its ability to reconstruct visual representations of map scenery, achieved through generative deep neural network (GAN) models. This capability not only enhances situational awareness for autonomous vehicles but also fosters seamless interaction with human operators and other road users. In essence, the system represents a holistic innovation poised to revolutionize autonomous vehicle navigation and mapping technology, paving the way for safer, more efficient, and more immersive autonomous driving experiences.

[0040] FIG. 1 illustrates an exemplary block diagram of a system 100 for machine learningbased HD map for high precision position estimation and map maintenance, in accordance with an embodiment of the present technology. The system 100 includes a processor 102, a memory 104, an interface 106, a processing engine 108, and a database 122. The processors 102 and the memory 104 may be communicably coupled to the one or more other processors. The one or more processor(s) may be implemented as one or more microprocessors, microcomputers, microcontrollers, edge or fog microcontrollers, digital signal processors, central processing units, logic circuitries, and / or any devices that process data based on operational instructions. Among other capabilities, one or more processor(s) may be configured to fetch and execute computer-readable instructions stored in the memory 104 of the system 100. The processor 102 is configured to fetch and execute computer-readable instructions stored in the memory 104. The memory 104 may be configured to store one or more computer-readable instructions or routines in a non-transitorycomputer-readable storage medium, which may be fetched and executed to create or share data packets over a network service. The memory 104 may include any non-transitory storage device including, for example, volatile memory such as random-access memory (RAM), or non-volatile memory such as erasable programmable read-only memory (EPROM), flash memory, and the like. The memory 104 may be configured to store one or more computer-readable instructions or routines in a non-transitory computer-readable storage medium, fetched and executed to create or share data packets over a network service.

[0041] The processing engine 108 is configured to generate HD map for high precision position estimation and map maintenance. The processing engine 108 includes a localization encoder module 110, a reconstruction decoder module 112, an updating module 114 and a discriminator module 116, a map module 118 and a visualization module 120. The localization encoder module 110 is a deep neural network consisting of layers of Convolutional Long Short-Term Memory (ConvLSTM) layers that maps the input RGB data into a latent space vector that represents the image location and orientation in 3D space. The reconstruction decoder module 112 is a deep neural network model of transposed ConvLSTM layers that maps a position and orientation to a higher dimensional synthesized RGB image. The updating module 114 is a deep neural network model that act as a discriminator in a generative adversarial setup with the other two models. This model should take in the real-time RGP image data and the synthesized image based on the location, and then deduce whether the model needs to be updated or not. The actual map updates are performed by simply training the encoder-decoder setup as a denoising process for a variational autoencoder. The discriminator module 116 operates as a CNN binary classifier, responsible for determining whether input images should be included in a stack for offline model retraining. This decision-making process is essential for ensuring that only relevant data is utilized to update the Al Maps, optimizing resource allocation, and maintaining map accuracy over time. Additionally, the discriminator model's potential future integration with a semantic segmentation approach represents an innovative strategy for identifying changed areas within images. By employing such techniques, the system can pinpoint specific regions requiring updates, enhancing the precision and efficiency of map maintenance. The decision-making process for map updates includes that a generative adversarial setup of two models; a generator and a discriminator. The generator is used to generate the synthetic visual representation of the surrounding, and then the discriminator is used to decide whether the synthetic visuals match the real-time visual input fromcameras. If the two are visually similar enough, the decision is that this area of the map does not need updates. The discriminator model could give area-specific recommendations for updates, and with enough changes that prevents the reality and the synthetic scenes from being similar, this real-time visual will be used for later updating the Al map. This implies using Al model training methods to update the Al map pre-trained models’ weights by methods of deep learning model training.

[0042] The system further includes a map module 118 configured for tilling the generated HD map into one or more small tiles, training a neural network model using the small HD map tiles on a pose regression task, retraining the HD map by creating a new and separate checkpoint for every map tile and communicating to a vehicle one or more VBL model parameters to use it in pose regression. The system further includes a visualization module 120 configured for training a decoder network using a variational autoencoder (VAE) trained on each map tile to retrieve original map features from a smaller space of 6 DoF- pose and proposing an additional use of generative adversarial networks (GANs) to generate the visuals of the map and assigning the generative model with the task of generating one or more views for arbitrary poses within the map tile. The visualization module 120 performs actual map updates by training the encoder-decoder setup as a denoising process for the VAE.

[0043] According to an embodiment, the VAEs are configured for learning efficient representations of input data, and training the VAE on the HD-Map data during the initial stages enables the HD-map to learn to encode essential spatial information in a compact latent space, facilitating efficient processing and storage.

[0044] According to an embodiment, the system further includes a VAE decoder that utilizes a mean output from the encoder to synthesize one or more views like images captured using a vehicle's cameras.

[0045] According to an embodiment, the system performs localization of Al maps for high- precision position estimation, and HD map maintenance. The localization includes tilling the HD map into small tiles for example 100m road sections, training a neural network model using the small HD map tiles on a pose regression task, retraining the model, creating a new and separate checkpoint for every map tile, and communicating to the vehicle the VBL model parameters (which will be much lighter) to use it in pose regression process. According to another embodiment, the system performs visualization of Al maps for high-precision position estimation andHD map maintenance. The visualization includes training a decoder network from a variational autoencoder VAE trained on each map tile to retrieve original map features from the smaller space of 6 DoF- pose, proposing the additional use of generative adversarial networks GANs to generate the visuals of the map and assigning the generative model with the task of generating views for arbitrary poses within the map tile where it is used in two different ways including map visualization and the synthetic view vision-based navigation approach.

[0046] FIG. 2A depicts a block diagram of a VAE encoder 202 used to generate HD Maps, in accordance with an embodiment. Variational auto-encoder 202 (VAEs) is adept at learning efficient representations of input data. By training the VAE on HD-Map data during the initial stages, the VAE 202 learns to encode essential spatial information in a compact latent space, facilitating efficient processing and storage.

[0047] FIG. 2B depicts a block diagram of a VAE decoder 204 used to generate Al HD Maps, in accordance with an embodiment. The VAE decoder 204 utilizes the mean output from the VAE encoder 202 to synthesize views similar to what the vehicle cameras would capture. This synthetic view generation enables the system to predict how the environment would appear from the vehicle's perspective, aiding in navigation and decision-making processes.

[0048] FIG. 2C depicts a block diagram of a discriminator 206 used to generate Al HD Maps, in accordance with an embodiment. The discriminator 206 operates as a CNN binary classifier, responsible for determining whether input images should be included in a stack for offline model retraining. This decision-making process is essential for ensuring that only relevant data is utilized to update the Al Maps, optimizing resource allocation, and maintaining map accuracy over time.

[0049] According to an embodiment, the main features of the proposed Al HD Maps are Al models that perform the functions of an HD Map without the need to store map sensor data like LiDAR point cloud or Images. These functions are visualization and localization. 6-DoF Vehicle Localization Encoder model is a deep neural network consisting of layers of ConvLSTM layers that maps the input RGB data into a latent space vector that represents the image location and orientation in 3D space. 3D environment reconstruction decoder model is a deep neural network of transposed ConvLSTM layers that maps a position and orientation to a higher dimensional synthesized RGB image. Al map updating model is a deep neural network model that act as a discriminator in a generative adversarial setup with the other two models. This model should takein the real-time RGP image data and the synthesized image based on the location, and then deduce whether the model needs to be updated or not. Actual map updates are performed by simply training the encoder-decoder setup as a denoising process for a variational autoencoder.

[0050] FIG. 3A illustrates a flow diagram 300 depicting a method for localization of Al maps for high precision position estimation and HD map maintenance. At step 302, the HD map is tiled into small tiles. At step 304, a neural network model is trained using the small HD map tiles on a pose regression task. At step 306, the model is retrained, creating a new and separate checkpoint for every map tile. At step 308, the vehicle the VBL model parameters is communicated to use it in pose regression.

[0051] In one aspect a method of localization of Al maps for high precision position estimation and HD map maintenance is provided. The method includes tilling the HD map into small tiles. The method also includes training a neural network model using the small HD map tiles on a pose regression task. The method also includes retraining the model, creating a new and separate checkpoint for every map tile. The method also includes communicating to the vehicle the VBL model parameters to use it in pose regression.

[0052] FIG. 3B illustrates a flow diagram 310 depicting a method for visualization of Al maps for high precision position estimation and HD map maintenance. At step 312, a decoder network is trained from a variational auto encoder VAE trained on each map tile to retrieve original map features from the smaller space of 6 DoF- pose. At step 314, the additional use of generative adversarial networks GANs is proposed to generate the visuals of the map. At step 316, the generative model is assigned with the task of generating views for arbitrary poses within the map tile.

[0053] FIG. 4 illustrates a flowchart of a processor-implemented method for generating a high-definition (HD) map for high-precision position estimation and map maintenance, in accordance with an embodiment. At step 402, input red green blue (RGB) data is mapped, by a localization encoder module, into a latent space vector representing an image location and an orientation in a three-dimensional (3D) space, using a deep neural network including of one or more layers of ConvLSTM layers. At step 404, the image location, and the orientation of the input RGB data is mapped, by a reconstruction decoder module, to a high-dimensional synthesized RGB image, by using the deep neural network including one or more transposed ConvLSTM layers. At step 406, an updating module including the deep neural network model acts as a discriminator ina generative adversarial setup by utilizing real-time RGB image data and the high dimensional synthesized RGB image based on the image location for determining whether the HD map needs to be updated. The real-time RGB image from the vehicle camera is provided to the discriminator model. The environment visualization model is then used to generate a synthesized RGB image based on the location to envision what the vehicle should be receiving from the camera. Both images are then presented to pre-trained discriminator model to provide a binary output whether the new real-time image contains changes that should be marked for later use for HD Map update, or no changes are detected between real-time and synthesized RGB images.

[0054] At step 408, a discriminator module is operated as a convolution neural network (CNN) binary classifier, for determining whether the input RGB image should be included in a stack for an offline model retraining. The discriminator module determines the above based on real and synthesized views for the same location. The discriminator module compares the real vs. the synthesized RGB images to detect whether there are major changes that deserve to be included in the features learned by the HD Map.

[0055] According to an embodiment, the method further includes localization, by a map module, one or more Al maps for a high-precision position estimation, and the HD map maintenance by tilling the generated HD map into one or more small tiles, training a neural network model using the small HD map tiles on a pose regression task, retraining the HD map by creating a new and separate checkpoint for every map tile and communicating to a vehicle one or more VBL model parameters to use in pose regression. In some embodiments, once the real-time RGB image is marked as an image with changes that need to be used for HD Map update. The RGB image is labeled with the location estimation and stacked to a dataset to be used for further optimizing the weights of the VBL localization model and the view synthesis generative model. Once enough images are collected and labeled with their location and orientation, this dataset is used to further optimize the model weights using a conventional backpropagation-based neural network model training algorithm. The model corresponds to a certain tile of the map, which is why the image location is used to classify each image to belong to a certain dataset to be used to update the HD Map model that works in this tile. Further, a tile may be georeferenced using certain coordinates to maintain an efficient size of the model to be only in service of a certain area of the map.

[0056] According to an embodiment, the method further includes visualization, by a visualization module, of Al maps for the high precision position estimation and the HD map maintenance by training a decoder network from a variational autoencoder (VAE) trained on each map tile to retrieve original map features from a smaller space of 6 DoF- pose and proposing an additional use of generative adversarial networks (GANs) to generate the visuals of the map and assigning the generative model with the task of generating views for arbitrary poses within the map tile. In some embodiments, identifying the location estimate for the synthesized RGB image to be generated is based on the last estimated position of the vehicle. The pose estimate is then provided to the generative model to start generating the synthesized view based on the images the model was previously trained on. The model subsequently starts to take both its learned features and the current pose estimate to generate an RGB image that represents what the vehicle “should” be seeing in this location of the environment.

[0057] According to an embodiment, by training the VAE on the HD-Map data during the initial stages, the HD-map learns to encode essential spatial information in a compact latent space, facilitating efficient processing and storage.

[0058] According to an embodiment, the method further includes utilizing by a VAE decoder a mean output from the encoder to synthesize one or more views like images captured using a vehicle camera.

[0059] According to an embodiment, the synthetic view generation enables a system to predict the environmental conditions appearing from the vehicle's perspective, aiding in navigation and decision-making processes.

[0060] According to an embodiment, the decision-making process is essential for ensuring that relevant data is utilized to update the Al Maps, optimize resource allocation, and maintain map accuracy over time.

[0061] According to an embodiment, actual map updates are performed by training the encoder-decoder setup as a denoising process for the VAE.

[0062] FIG. 5 illustrates an exemplary computer system 500 in which or with which embodiments of the present disclosure may be implemented. The computer system 500 may include an external storage device 510, a bus 520, a main memory 530, a read-only memory 540, a mass storage device 550, a communication port(s) 560, and a processor 570. A person skilled in the artwill appreciate that the computer system 500 may include more than one processor and communication ports. The processor 570 may include various modules associated with embodiments of the present disclosure. The communication port(s) 560 may be any of an RS-232 port for use with a modem-based dialup connection, a 10 / 100 Ethernet port, a Gigabit or 10 Gigabit port using copper or fiber, a serial port, a parallel port, or other existing or future ports. The communication ports(s) 560 may be chosen depending on a network, such as a Local Area Network (LAN), Wide Area Network (WAN), or any network to which the computer system 100 connects.

[0063] In an embodiment, the main memory 530 may be Random Access Memory (RAM), or any other dynamic storage device commonly known in the art. The read-only memory 540 may be any static storage device(s) e.g., but not limited to, a Programmable Read Only Memory (PROM) chip for storing static information e.g., start-up or basic input / output system (BIOS) instructions for the processor 570. The mass storage device 550 may be any current or future mass storage solution, which can be used to store information and / or instructions. Exemplary mass storage solutions include, but are not limited to, Parallel Advanced Technology Attachment (PATA) or Serial Advanced Technology Attachment (SATA) hard disk drives or solid-state drives (internal or external, e.g., having Universal Serial Bus (USB) and / or Firewire interfaces).

[0064] In an embodiment, the bus 520 may communicatively couple the processor(s) 570 with the other memory, storage, and communication blocks. The bus 520 may be, e.g. a Peripheral Component Interconnect PCI) / PCI Extended (PCI-X) bus, Small Computer System Interface (SCSI), USB, or the like, for connecting expansion cards, drives, and other subsystems as well as other buses, such a front side bus (FSB), which connects the processor 570 to the computer system 500.

[0065] Various embodiments of the present technology uses GAN framework to train two major models. The first model is the generator, which is a generative variational auto-encoder (VAE) model and the second one is the discriminator, which is responsible for map updates. The use of a generative variational auto-encoder (VAE) serves the objectives of efficient representation learning, pose estimation and uncertainty modeling and synthetic view generation. VAEs are adept at learning efficient representations of input data. By training the VAE on HD-Map data during the initial stages, it learns to encode essential spatial information in a compact latent space, facilitating efficient processing and storage. The VAE encoder part extracts valuable informationfrom RGB images, outputting two latent space variables, the 6-DoF pose and the aleatoric uncertainty in pose estimation. This information is crucial for accurately localizing the vehicle within its environment and quantifying the certainty of the estimated pose. During inference, the VAE decoder utilizes the mean output from the encoder to synthesize views like what the vehicle's cameras would capture. This synthetic view generation enables the system to predict how the environment would appear from the vehicle's perspective, aiding in navigation and decision-making processes. The discriminator operates as a CNN binary classifier, responsible for determining whether input images should be included in a stack for offline model retraining. This decisionmaking process is essential for ensuring that only relevant data is utilized to update the Al Maps, optimizing resource allocation, and maintaining map accuracy over time. Additionally, the discriminator model's potential future integration with a semantic segmentation approach represents an innovative strategy for identifying changed areas within images. By employing such techniques, the system can pinpoint specific regions requiring updates, enhancing the precision and efficiency of map maintenance.

[0066] According to an embodiment, a visual-based positioning model is implemented to enhance positioning accuracy which involves using smaller tiles in the Calgary dataset and incorporating additional data collected from the same area of the tile. The training process is continued for the Calgary Downtown model to further enhance attitude estimation which includes utilizing data from both left and right cameras in the Calgary dataset and the DeepLoc dataset. Pose estimation is improved uncertainty by continuing to use the KL-divergence loss function with modifications mentioned in the CoordiNet paper. Additionally, exploring the possibility of using a separate model for pose uncertainty estimation. A generative model is developed for HD-Map visual synthesis to enhance visual reconstruction accuracy. This may involve using smaller tiles in the Calgary dataset, incorporating additional data, and utilizing a larger model with more layers. The current trainable parameters is considered for both the generative model and pose regression model (181,955 and 54,285,414 respectively) to ensure scalability and efficiency in future iterations of the invention.

[0067] The present technology offers several advantages over other known alternatives in the realm of autonomous vehicle navigation and mapping technology. By utilizing a generative variational auto-encoder (VAE) within a GAN framework, the algorithm optimizes storage space by replacing HD-Map data with a smaller, more efficient Al Map. The present technology reducesstorage requirements while maintaining essential navigational information, enhancing overall system efficiency. The use of crowdsourced data and offline re-training for Al Map updates eliminates the need for dedicated mobile mappers, significantly reducing operational costs associated with map maintenance. Additionally, leveraging existing GAN training resources minimizes development expenses by avoiding the need to build systems from scratch. The proposed framework simplifies the map update process by employing a CNN binary classifier as the discriminator model. The present technology efficiently determines relevant data for model retraining, streamlining the update process and reducing complexity compared to traditional methods. Through the integration of visual-based localization (VBL) techniques and semantic segmentation models, this invention improves the accuracy of pose estimation and map updates. By focusing on relevant image features and changes, the system enhances the precision of autonomous vehicle navigation, mitigating potential errors and improving overall performance. The present technology allows for the seamless integration of new data points into Al Maps without increasing storage requirements. This adaptability ensures that the maps remain up-to-date with evolving environments and road conditions, enhancing the reliability and safety of autonomous driving experiences. The present technology provides a setup and process of considering an Al model as a feasible alternative for an HD Map. The claim is mainly that whatever functionality the HD Map provides, that process and these models can provide the same functionality with less memory space and an easier way to update and maintain the map and its outputs.

[0068] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and / or adapt for various applications such as specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modifications. However, all such modifications are deemed to be within the scope of the claims.

[0069] The scope of the embodiments herein will be ascertained by the claims to be submitted at the time of filing a complete specification.

Claims

CLAIMSWhat is claimed is:

1. A processor-implemented method for generating a high definition (HD) map for high precision position estimation and map maintenance, the method comprising: mapping, by a localization encoder module, input red green blue (RGB) data into a latent space vector representing an image location and orientation in a three-dimensional (3D) space, using a deep neural network comprising one or more convolutional long short-term memory (ConvLSTM) layers; mapping, by a reconstruction decoder module, the image location and orientation of the input RGB data to a high-dimensional synthesized RGB image using the deep neural network comprising one or more transposed ConvLSTM layers; acting, by an updating module comprising the deep neural network model, as a discriminator in a generative adversarial setup by utilizing real-time RGB image data and the highdimensional synthesized RGB image based on the image location to determine whether the HD map requires updating; and operating, by a discriminator module, as a convolution neural network (CNN) binary classifier to determine whether the input RGB image should be included in a stack for offline model retraining.

2. The processor-implemented method of claim 1, further comprising localizing, by a map module, one or more Al maps for high-precision position estimation and HD map maintenance by: tiling the generated HD map into one or more small tiles; training a neural network model using the small HD map tiles for a pose regression task; retraining the HD map by creating a new and separate checkpoint for each map tile; and communicating to a vehicle one or more VBL model parameters to be used in pose regression, wherein the Al maps comprise a new HD map format that does not include a structured visual and geometrical dataset in the form of a database, but instead a framework of Al models cooperating to provide the same functionality without the need for an ever-expanding memory footprint.

3. The processor-implemented method of claim 1, further comprising visualizing, by a visualization module, the Al maps for high-precision position estimation and HD map maintenance by: training a decoder network using a variational autoencoder (VAE) trained on each map tile to retrieve original map features from a reduced 6 DoF pose space; and employing generative adversarial networks (GANs) to generate visual representations of the map and assigning the generative model the task of generating views for arbitrary poses within each map tile.

4. The processor-implemented method of claim 3, wherein training the VAE on HD map data during the initial stages enables the HD map to learn to encode essential spatial information into a compact latent space, thereby facilitating efficient processing and storage.

5. The processor-implemented method of claim 1, further comprising utilizing, by a VAE decoder, a mean output from the encoder to synthesize one or more views similar to images captured by a vehicle mounted camera.

6. The processor-implemented method of claim 5, wherein the synthetic view generation allows the system to predict environmental conditions from the vehicle's perspective, aiding navigation and decision-making processes.

7. The processor-implemented method of claim 6, wherein the decision-making process ensures that relevant data is used to update the Al maps, optimize resource allocation, and maintain long-term map accuracy.

8. The processor-implemented method of claim 1, wherein actual map updates are performed by training the encoder-decoder setup as a denoising variational autoencoder (VAE) process.

9. A system for generating a high-definition (HD) map for high-precision position estimation and map maintenance, the system comprising:a processor configured to fetch and execute computer-readable instructions stored in memory; a memory configured to store one or more computer-readable instructions in a non-tran- sitory computer-readable storage medium, which could be fetched and executed to create or share data packets over a network; a processing engine configured to generate a HD map for high-precision position estimation and map maintenance, the engine comprising at least: a localization encoder module configured to map input red green blue (RGB) data into a latent space vector representing an image location and orientation in three-dimensional (3D) space using a deep neural network comprising one or more ConvLSTM layers; a reconstruction decoder module configured to map the image location and orientation of the input RGB data to a high-dimensional synthesized RGB image using one or more transposed ConvLSTM layers; an updating module comprising a deep neural network model configured to act as a discriminator in a generative adversarial setup using real-time RGB image data and the synthesized RGB image based on the image location to determine whether the HD map requires updating; and a discriminator module configured as a convolution neural network (CNN) binary classifier to determine whether the input RGB image should be included in a stack for offline model retraining.

10. The system of claim 9, further comprising a map module configured to: tiling the generated HD map into one or more small tiles; training a neural network model using the small HD map tiles for a pose regression task; retraining the HD map by creating a new and separate checkpoint for each map tile; and communicating to a vehicle one or more VBL model parameters to be used in pose regression.

11. The system of claim 9, further comprising a visualization module configured to:training a decoder network using a variational autoencoder (VAE) trained on each map tile to retrieve original map features from a reduced 6 DoF pose space; and employing generative adversarial networks (GANs) to generate visual representations of the map and assigning the generative model the task of generating views for arbitrary poses within each map tile.

12. The system of claim 11, wherein the VAEs are configured to learn efficient representations of input data, and wherein training the VAE on HD map data during initial stages enables the HD map to encode essential spatial information into a compact latent space for efficient processing and storage.

13. The system of claim 9, wherein the system further comprises a VAE decoder configured to utilize a mean output from the encoder to synthesize one or more views similar to images captured by a vehicle mounted camera.

14. The system of claim 9, wherein the visualization module performs map updates by training the encoder-decoder setup as a denoising VAE process.

Images