Methods and systems for learning neural networks to determine vehicle attitude in an environment

Abstract

A method and system are provided for learning a neural network and determining vehicle attitude in an environment. A first processor performs a first feature extraction on sensor-based image data to provide a first feature map. The first processor also performs a second feature extraction on aerial image data to provide a second feature map. The two feature maps are correlated to provide a correlation result. The first processor uses the correlation result and ground-based data to learn a neural network, wherein each of the first and second feature extractions is learned to extract a subset of features from the corresponding image data. The geotagged second feature map can then be retrieved by an onboard processor of the vehicle, which, together with sensor-based data processed by the onboard processor through a network trained by the first processor, determines the vehicle's attitude.

Description

Technical Field

[0001] The technical field generally relates to the determination of the attitude of a vehicle navigating through an environment. More specifically, the present invention relates to methods and systems for learning neural networks to determine the position of a vehicle in an environment in order to improve the real-time localization of the vehicle. Background Technology

[0002] In recent years, due to the increasing demand for passenger transport systems, camera-based navigation systems have become crucial. These systems provide location information for vehicles moving through their environment. This location information is used to navigate the vehicle or to provide information about traffic conditions in a specific area. However, the data obtained from such sensor-based navigation systems sometimes lacks the accuracy required for reliable vehicle navigation in certain situations. The requirement for accurate location information is critical for navigation purposes and becomes even more critical for navigation scenarios involving vehicles navigating through densely populated areas (e.g., large cities), where multiple different features and objects in the surrounding environment must be distinguished. Furthermore, accurate location information is also needed in environments without specific or typical objects, based on which sensor-based localization can be performed. To address the localization problem, in-vehicle sensor-based systems (including, for example, camera systems) are trained using manually annotated features, such as traffic signs and lane edges, to infer pose from these features and match the mapped features to recover the position. However, the manual annotation of the corresponding features required for this approach increases the preparation work for such systems.

[0003] Therefore, it is desirable to provide improved identification of relevant features that avoids such manual feature annotation, to distinguish such relevant features from features that provide less reliable information for navigation purposes, and to provide a self-learning process for improving the distinction, so as to improve the determination of the attitude of moving objects (e.g., vehicles). Furthermore, other desirable features and characteristics of the invention will become apparent from the following detailed description and appended claims, taken in conjunction with the accompanying drawings and the foregoing technical and background information. Summary of the Invention

[0004] A computer implementation method for learning a neural network to determine vehicle attitude in an environment is provided. The method includes acquiring sensor data of the environment by a first processor to provide a sensor-based image. The method further includes performing a first feature extraction by the first processor, wherein features are extracted from the sensor-based image to provide a first feature map. The method also includes acquiring aerial image data of the environment by the first processor. The method further includes performing a second feature extraction by the first processor, wherein features are extracted from the acquired aerial image data to provide a second feature map. The method also includes correlating the first feature map with the second feature map by the first processor to provide a correlation result. The method further includes acquiring ground truth data by the first processor. Furthermore, the method includes learning a neural network by the first processor using the correlation result and the ground truth data, wherein learning the neural network includes learning the first feature extraction to extract a subset of features from the sensor-based image, and learning the second feature extraction to extract a subset of features from the acquired aerial image data.

[0005] In an exemplary embodiment, the first processor rotates the first feature map before associating the first feature map with the second feature map.

[0006] In an exemplary embodiment, before associating the first feature map with the second feature map, the first processor applies interpolation to the second feature map, such as spatial interpolation, cubic interpolation, bilinear interpolation, or any other interpolation process.

[0007] In an exemplary embodiment, the first processor learns the neural network by determining the predicted coordinates in the relevant results using a soft argmax function, determining the ground truth coordinates on the relevant results based on ground truth data, and determining a loss value based on the predicted coordinates and the ground truth coordinates.

[0008] In an exemplary embodiment, the first processor learns the neural network by determining the ground-based true Gaussian distribution based on ground-based real data, and by determining a loss value based on the ground-based true Gaussian distribution and related results.

[0009] In an exemplary embodiment, the first processor backpropagates loss values ​​to learn a neural network, based on which the first processor extracts partial features from a sensor-based image.

[0010] In an exemplary embodiment, the first processor backpropagates loss values ​​to learn a neural network, based on which the first processor extracts partial features from the acquired aerial image data.

[0011] In an exemplary embodiment, the first processor reduces the loss value during the iteration process, during which the loss value is repeatedly determined and backpropagated to learn the neural network.

[0012] In an exemplary embodiment, the first processor determines a prioritized feature group of partial features extracted from the sensor-based image, and also determines a de-prioritized feature group of partial features extracted from the sensor-based image. The prioritized feature group is assigned a higher impact on reducing the loss value than the de-prioritized feature group.

[0013] In an exemplary embodiment, the first processor extracts partial features from a sensor-based image by extracting only features from a prioritized feature group. Furthermore, the first processor extracts partial features from acquired aerial image data by extracting only features from a prioritized feature group.

[0014] In an exemplary embodiment, a first processor learns a neural network until convergence is achieved to a specified precision, thereby providing the learned neural network.

[0015] In an exemplary embodiment, the first processor determines a second feature map, such as a geotagged second feature map, encodes the second feature map after reaching convergence with a specified accuracy, and stores the second feature map on a storage medium.

[0016] In an exemplary embodiment, a second processor is provided that uses a learned neural network to determine the vehicle's posture. The second processor is an onboard processor for the vehicle.

[0017] In an exemplary embodiment, the second processor determines the vehicle's attitude using a learned neural network by: retrieving a second feature map from a storage medium; obtaining sensor data of the environment through which the vehicle navigates to provide a sensor-based image; extracting features from the sensor-based image to provide a first feature map; processing the first feature map using the learned neural network; correlating the first feature map with the second feature map to provide a correlation result; and determining the vehicle's attitude based on the correlation result.

[0018] In an exemplary embodiment, the second processor applies a particle filter to the correlation results determined by the second processor. The particle filter provides a probability distribution within the correlation results, where the probability distribution indicates the expected position of the vehicle. The second processor determines the attitude of the vehicle based on the probability distribution.

[0019] In an exemplary embodiment, a probability distribution is provided within the relevant result by assigning a probability value to each pixel within a portion of the relevant result.

[0020] In an exemplary embodiment, the second processor determines the vehicle's position and orientation based on a probability distribution.

[0021] In an exemplary embodiment, aerial imagery data is provided by an external entity spatially separate from the vehicle.

[0022] In an exemplary embodiment, aerial imagery data is provided by a satellite system.

[0023] A system for determining the position of a vehicle in an environment is provided. The system includes a first processor and a second processor. The first processor acquires sensor data of the environment to provide a sensor-based image and performs a first feature extraction, wherein features are extracted from the sensor-based image to provide a first feature map. The first processor further acquires aerial image data of the environment and performs a second feature extraction, wherein features are extracted from the acquired aerial image data to provide a second feature map. The first processor correlates the first feature map with the second feature map to provide a correlation result. The first processor further acquires ground truth data. The first processor uses the correlation result and the ground truth data to learn a neural network, wherein learning the neural network includes learning the first feature extraction to extract a subset of features from the sensor-based image and learning the second feature extraction to extract a subset of features from the acquired aerial image data. The second processor is an onboard processor of the vehicle and uses the neural network learned on the first processor to determine the vehicle's attitude. Attached Figure Description

[0024] Exemplary embodiments will now be described in conjunction with the following figures, wherein the same numerals denote the same elements, and wherein:

[0025] Figure 1 The vehicle according to the embodiment has a processor that performs a method for determining the vehicle's attitude;

[0026] Figure 2 This is an illustration of a system for determining vehicle attitude according to an embodiment;

[0027] Figure 3 This is a visual example of a method for learning a neural network to determine vehicle posture according to an embodiment.

[0028] Figure 4 A schematic diagram of the software architecture of a system for learning a neural network to determine vehicle attitude, according to an embodiment, is shown.

[0029] Figure 5 This is a block diagram illustrating a detailed example of the method steps of a method for learning a neural network to determine vehicle attitude according to an embodiment.

[0030] Figure 6 This is a block diagram illustrating a detailed example of the method steps for determining a vehicle attitude according to an embodiment.

[0031] Figure 7 Two block diagrams are shown, illustrating the learning process and runtime operation of the method for determining vehicle attitude according to an embodiment. Detailed Implementation

[0032] The following detailed description is exemplary in nature only and is not intended to limit application and use. Furthermore, it is not intended to be bound by any express or implied theory set forth in the foregoing technical fields, background art, summary of the invention, or the following detailed description. As used herein, the terms module and / or system refer individually or in any combination of any hardware, software, firmware, electronic control components, processing logic, and / or processor devices, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped), and memory executing one or more software or firmware programs, combinational logic circuits, and / or other suitable components providing the described functionality.

[0033] Embodiments of this disclosure can be described herein based on functional and / or logical block components and various processing steps. It should be understood that such block components can be implemented by any number of hardware, software, and / or firmware components configured to perform specified functions. For example, embodiments of this disclosure may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices. Furthermore, those skilled in the art will understand that embodiments of this disclosure can be practiced in conjunction with any number of systems, and the systems described herein are merely exemplary embodiments of this disclosure.

[0034] For the sake of brevity, conventional techniques relating to signal processing, data transmission, signaling, control, and other functional aspects of the system (as well as the various operating components of the system) may not be described in detail herein. Furthermore, the connecting lines shown in the various figures included herein are intended to represent exemplary functional relationships and / or physical connections between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in the embodiments of this disclosure.

[0035] refer to Figure 1 The image illustrates a vehicle 10 according to various embodiments. The vehicle 10 typically includes a body 14, front wheels 16, and rear wheels 18. The vehicle 10 typically includes a propulsion system 20, a drivetrain 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34, and a communication system 36. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric motor such as a traction motor, and / or a fuel cell propulsion system.

[0036] Sensor system 28 includes one or more sensing devices 40a-40n that sense the observable state of the external environment 50 and / or internal environment of vehicle 10, such as autonomous vehicle 10. The one or more sensing devices 40a-40n may, for example, sense objects in the external environment 50 of vehicle 10. Sensing devices 40a-40n may include, but are not limited to, radar, lidar, GPS, cameras, particularly optical cameras and / or thermal cameras, ultrasonic sensors, temperature sensors, and / or other sensors. For example, sensing device 40a may include a camera configured to receive object and / or image data from environment 50 (i.e., the surrounding environment of vehicle 10). Image data may include information about at least one object located in environment 50 of vehicle 10. Actuator system 30 includes one or more actuator devices 42a-42n that control one or more vehicle features, such as, but not limited to, sensing devices 40a-40n, propulsion system 20, transmission system 22, steering system 24, and braking system 26.

[0037] The communication system 36 is configured to communicate wirelessly with other entities 48, such as, but not limited to, the first processor 60, which may be a back-end system, a satellite navigation system, other vehicles (“V2V” communication), infrastructure (“V2I” communication), a remote system, a personal device, or a combination thereof.

[0038] The first processor 60 can be configured to execute method steps for learning a neural network, based on which the pose of the vehicle 10 can be determined. Specifically, a second geotagged feature map and the learned neural network can be provided by the first processor 60 to a second processor 44 of the vehicle, where the second processor 44 is an onboard processor of the vehicle. The onboard processor 44 can then use the learned neural network and the second geotagged feature map to determine the pose of the vehicle 10. In other words, the first processor 60 in the background can perform a learning or training process to learn the neural network, and the second processor 44 on the vehicle 10 can use the learned neural network to determine the vehicle's pose.

[0039] In an exemplary embodiment, the communication system 36 is configured to receive information from the first processor 60. Furthermore, the communication system 36 can receive satellite information from other entities 48, such as satellite systems. The communication system 36 may be configured to receive aerial imagery, ground real-time information, or camera images and / or data corresponding to these images from aircraft or spacecraft.

[0040] In an exemplary embodiment, communication system 36 is a wireless communication system configured to communicate using the IEEE 802.11 standard or via a wireless local area network (WLAN) using cellular data communication. However, additional or alternative communication methods, such as Dedicated Short Range Communication (DSRC) channels, are also contemplated within the scope of this disclosure. A DSRC channel refers to a one-way or two-way short-to-medium range wireless communication channel specifically designed for automotive use, along with a set of corresponding protocols and standards.

[0041] In various embodiments, data storage device 32 stores a defined or predetermined map of the navigable environment 50. Specifically, data storage device 32 can provide map data representing the environment 50 of vehicle 10, particularly the environment through which the vehicle is currently navigating. Furthermore, data storage device 32 can provide images, such as stored camera images or aerial images of the environment 50. The map data can be continuously updated, i.e., updated in real time, to provide map and / or image data corresponding to the environment 50 through which vehicle 10 is currently traveling. In various embodiments, the defined maps and / or images can be predefined and obtained from a remote entity 48 (e.g., a remote data server or first processor 60). Specifically, data storage device 32 stores algorithms to implement neural networks, such as a learned neural network obtained from the first processor. Additionally, data storage device 32 can store a second feature map received from the first processor 60, such as a geotagged second feature map. For example, the defined maps and / or images can be collected by a remote system and transmitted to the autonomous vehicle 10 (wirelessly and / or via a wired connection) and stored in data storage device 32. Data storage device 32 may have stored image data of environment 50 and / or feature data of objects in environment 50. For example, the stored image data may include the appearance of objects and object features assigned to specified locations in environment 50. Object features described herein may correspond to trees, vehicles, sidewalks, traffic signs, streets and street intersections, crosswalks, intersections, grass-covered areas near streets, etc. It should be understood that data storage device 32 may be part of controller 34, separate from controller 34, or part of a system separate from vehicle 10.

[0042] The controller 34 includes a second processor 44 and a computer-readable storage device or medium 46. Each of the first processor 60 and the second processor 44 can be any custom or commercially available processor, central processing unit (CPU), graphics processing unit (GPU), auxiliary processor among a plurality of processors associated with the controller 34, semiconductor-based microprocessor (in the form of a microchip or chipset), macroprocessor, any combination thereof, or any device typically used to execute instructions. For example, the computer-readable storage device or medium 46 may include volatile and non-volatile memory in read-only memory (ROM), random access memory (RAM), and persistent active memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operational variables when the first processor 60 and / or the second processor 44 is powered off. The computer-readable storage device or medium 46 can be implemented using any of many known storage devices, such as PROM (programmable read-only memory), EPROM (electric PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combined storage device capable of storing data, some of which represents executable instructions, which the controller 34 uses when controlling the vehicle 10, particularly to determine the attitude of the vehicle 10, i.e., the position and orientation of the vehicle 10 in the environment.

[0043] The instructions may include one or more separate programs, each comprising an ordered list of executable instructions for implementing logical functions. When executed by the first processor 60 and / or the second processor 44, these instructions receive and process signals from the sensor system 28, perform logic, calculations, methods, and / or algorithms, particularly machine learning algorithms, for determining the attitude of the vehicle 10 within the environment 50, for automatically controlling components of the autonomous vehicle 10, and for generating control signals to the actuator system 30 based on the logic, calculations, methods, and / or algorithms to automatically control components of the autonomous vehicle 10. Although in Figure 1 Only one controller 34 is shown, but embodiments of the autonomous vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication media and cooperate to process sensor signals, perform logic, calculations, methods and / or algorithms, and generate control signals to automatically control the features of the vehicle 10.

[0044] In various embodiments, the first processor 60 implements machine learning techniques, such as using ground truth to determine one or more loss values ​​based on feature maps and image backpropagation to improve feature extraction or generalization.

[0045] In various embodiments, the second processor 44 implements methods and systems for determining the attitude, such as position and / or orientation, of the vehicle 10, as will be discussed in more detail herein. Specifically, the second processor 44 uses a neural network learned on the first processor 60 and geotagged aerial imagery data provided by the first processor 60 to implement methods for determining the attitude of the vehicle 10 in the environment 50. The neural network learned on the first processor can be further improved by additional learning techniques provided by the onboard processor 44 of the vehicle 10.

[0046] In an exemplary embodiment, the method and system include training a deep learning neural network (DLNN) on a first processor 60 in an end-to-end manner to directly predict the ground truth pose, and including applying a particle filter on a second processor 44 to match the learned features and estimate the pose. The described method and system are capable of improving localization in regions lacking mapped semantic and vehicle-origin features, thereby improving localization accuracy and reducing costs by eliminating the need to annotate semantic features present in the acquired images. The deep learning neural network executed by the first processor 60 is further trained in an end-to-end manner to extract optimal features for localization based on a prioritization scheme, without requiring any annotation of the features. The prioritization scheme may include distinguishing between prioritized and de-prioritized features, wherein prioritized features have a more reliable response for localization purposes.

[0047] In other words, this method and system automatically identify non-semantic and semantic features when training a detector without annotated data, avoiding the need for an expensive annotation process. These features are automatically selected and learned during the end-to-end process to optimize localization performance while limiting map storage. This includes learning to ignore or de-prioritize transient objects in aerial and camera imagery, such as shadows, people, trees, or other vehicles in the environment. Furthermore, map storage space can be limited by using an autoencoder, and data collection can be facilitated by providing aerial imagery, camera images, and ground-based data from real-time kinematics.

[0048] The method and system enable first-day driving by using aerial imagery instead of vehicle-sourced data, and support ambiguous scenarios where the exact vehicle attitude cannot be fully determined, such as on a straight road without lateral information, by providing uncertainty indicators.

[0049] Now for reference Figure 2 This shows the method used to determine vehicle 10 ( Figure 1The vehicle 10 includes a system 70 for attitude control in environment 50. The vehicle 10 includes sensors with a camera 40a and / or other sensing devices 40n (e.g., radar or lidar) that generate sensor data of the environment 50 through which the vehicle 10 navigates. The camera 40a generates a sensor-based image 110, such as a camera image 110, which includes objects and / or object features, such as trees, other vehicles, sidewalks, etc. The sensor-based image is then provided to a first feature extractor 120. The first feature extractor 120 performs first feature extraction and generates a first feature map based on the sensor-based image 110. The first feature map can include various features of the environment 50 of the vehicle 10. Features in the first feature map can include object features of objects in environment 50, such as the lateral boundaries of roads or intersections, sidewalks, etc. The first feature map can include fewer features compared to the number of features in the camera image prior to feature extraction. The steps of the above process can be performed on the so-called camera side, which will refer to... Figure 5 To describe in more detail.

[0050] Vehicle 10 ( Figure 1 The vehicle 10 also receives aerial imagery data of its environment 50 via communication system 36. This aerial imagery data can be provided as or based on aerial imagery 210 captured by an external entity 48 located outside the vehicle 10. The external entity 48 can be an aircraft or spacecraft, such as a satellite, that has already captured aerial imagery 210 of the environment 50. Aerial imagery 210 can include objects and / or object features, such as trees, other vehicles, sidewalks, etc. Communication system 36 provides the aerial imagery data to a second feature extractor 220. The second feature extractor 220 performs second feature extraction and generates a second feature map based on the aerial imagery data. The second feature map can include various features of the environment 50 of the vehicle 10. Features in the second feature map can include object features of objects in the environment 50, such as the lateral boundaries of roads or intersections, sidewalks, etc. The second feature map can include fewer features compared to the number of features in the aerial imagery prior to feature extraction. The steps of the above process can be performed on the so-called aerial imagery side, which will refer to Figure 5 To describe in more detail.

[0051] Correlator 310 correlates the first feature map with the second feature map to provide correlation results that form a so-called correlation surface. The correlation results may include correlated feature data, such as image feature data of the first and second feature maps.

[0052] In one example, filter 320 is provided, in which a filtering process is performed on the relevant results, such as referencing... Figure 6Further details are provided. The output of the filtering process can be fused with other existing positioning strategies that pass additional input 360, such as Support Vector Machine (SVM) data, Global Positioning System (GPS) data, yaw data, etc. In this way, the attitude distribution 340 of the vehicle 10 in the environment is obtained, which can then be further refined based on ground-based real data and a training process that improves the feature extraction and selection of the first and second feature maps. The filtering process can be part of runtime operations, where a learned neural network is used to determine the vehicle attitude.

[0053] Ground-based real-time kinematics 330 is provided and used, along with relevant results, to determine the loss value indicating the localization loss. The loss value can be based on the vehicle's predicted xy coordinates from the relevant results and the vehicle's ground-based real-x coordinates determined from the ground-based real-time kinematics data. gt -y gt The deviation between coordinates is used to determine the loss value. The loss value is used to learn the neural network, based on which the first feature extraction in the first feature extractor 120 and the second feature extraction in the second feature extractor 220 are improved. Specifically, the extractors learn to adjust their neural network weights to extract features that best reduce the localization loss, i.e., the loss value, which is repeatedly determined and backpropagated to learn the neural network, thus providing the in-loop localization process 350, where the loss is iteratively reduced until training converges to a point where cross-validation performance is satisfactory. In this case, it can be said that the loss value has reached a predetermined or minimum threshold loss value, or has reached convergence at a specified accuracy threshold. If so, the feature extractors 120, 220 have learned to recognize noise, such as shadows, people, trees, and / or cars, which makes the corresponding extracted and generalized features unaffected, e.g., "0", so that such noise in the image does not contribute to the calculation of relevant results.

[0054] Figure 3 A detailed example of a method for learning neural networks to determine vehicle pose, as described herein, is shown, highlighting the learning principles applied in this case to improve the accuracy of pose determination. Specifically, Figure 3 This illustrates how image features from sensor-based image 1300 and aerial image 2300 can be processed to learn a neural network. In this example, image features may correspond to objects such as other vehicles 51, trees, people 52, and objects related to the road 53 in the environment 50 represented within the captured images 1300 and 2300. Image 1300 is composed of vehicles (e.g., Figure 1The image data is obtained from onboard sensors of the vehicle (described in the text) and includes onboard camera views of surrounding objects 51, 53. Image data is obtained from onboard sensors such as cameras, radar, and lidar. Image 2300 is obtained by an independent entity providing an aerial image of the vehicle's surrounding environment 50. However, it should be understood that aerial images can also be retrieved from storage media. Aerial features in aerial image 2300 can be generalized in a single-channel or multi-channel configuration. To provide normalized correlation, i.e., the correlation result 3100 between sensor-based image 1300 and aerial image 2300, feature extraction and generalization are performed on images 1300 and 2300, wherein features of images 1300 and 2300 are extracted and selected. Specifically, features in sensor-based image 1300 are extracted and generalized, resulting in a first feature map 1500. Extraction and generalization include removing noise, such as cars and / or shadows, from image 1300. Furthermore, features in aerial image 2300 are also extracted and generalized, resulting in a second feature map 2500. The extraction and generalization on both sides involves removing noise corresponding to features such as cars, trees, and / or shadows from images 1300 and 2300. During this extraction step, some features are highlighted, resulting in one set of features, i.e., features considered more important, being prioritized over another set of features considered less important. Before correlating the first feature map 1500 with the second feature map 2500, the first feature map 1500 is rotated based on data corresponding to vehicle orientation (i.e., yaw) to provide a rotated first feature map 1600. In the correlation, the rotated first feature map 1600 slides over or covers the second feature map 2500 to provide a correlation result 3100, which is also referred to herein as the correlation surface 3100. Based on the correlation surface 3100 and ground truth data, a vehicle position data map 3500 can be determined. From the highest peak region 3510 in the correlation surface 3100, the predicted position of the vehicle, e.g., with xy coordinates [10,3], can be inferred. However, the ground truth data provides xy coordinates different from the predicted xy coordinates. gt -y gt - Coordinates [10,2]. Based on this bias, the loss used to learn the neural network can be determined, which improves feature extraction and generalization over multiple iterations and ultimately leads to the true coordinates [10,2]. In particular, the relevant surface 3100 can be transformed into a backpropagable loss that can be differentiated in two different ways.

[0055] The first approach is to use a two-dimensional soft argmax function to generate an xy coordinate. The loss, i.e., mean squared error (MSE), can be determined relative to the true ground pose position on the relevant surface 3100, where the x-coordinate can be based on the x-coordinate of a specific region (e.g., the intersection shown in images 1300 and 2300) obtained from a precise positioning device (e.g., real-time kinematics). gt -ygt The ground truth is calculated using the global location coordinates of the ground truth latitude / longitude / azimuth. Since the soft argmax in two-dimensional space is differentiable, feature extraction and generalization learn to adjust the neural network weights to extract features and select those that best reduce localization loss through backpropagation using stochastic gradient descent. This learning is applicable to both the aerial image side and the vehicle sensor side, and is implemented in a neural network based on which the vehicle 10 is determined during runtime operations that can be executed on the onboard processor 44. Figure 1 ) posture.

[0056] For example, if a scene contains road edges, lanes, sidewalks, and shadows, the extracted features might initially refer to things like grass or mailboxes, and the selection / generalization module might primarily select grass, and possibly some other objects. Since other scenes in the dataset might not have mailboxes or grass, the localization loss for these areas would be significant. However, after training iterations, the neural network learns to extract more robust features across all scenes in the dataset, perhaps road edges and sidewalks, as well as general road shapes, because these features are more common and frequent than mailboxes. The neural network also learns the typical appearance of shadows and transient objects such as cars, trees, and people. Generalization and selection can be trained to negate certain features considered noise, such as shadows, trees, people, and / or cars, and highlight more reliable features, such as road shapes or sidewalks. After this training converges to a point (convergence point or accuracy) where cross-validation performance is satisfactory, the neural network is able to identify noise such as shadows and / or cars, rendering individual generalized features inconsequential, for example, "0," so that these features do not contribute to the computation of the relevant surface 3100. In this case, the neural network's learning has converged to the specified accuracy. In this way, the neural network learns a first feature extraction to extract a subset of features from sensor-based images, and learns a second feature extraction to extract a subset of features from the acquired aerial image data.

[0057] Therefore, the neural network is automatically trained to select important features and learns to automatically select robust features and de-prioritize non-robust features. In this way, priority feature groups and non-priority feature groups can be established, where priority feature groups are assigned a higher impact on reducing loss values ​​than non-priority feature groups. The network learns not only semantic features such as traffic signs, lane edges, and road edges, but also non-semantic features that appear in the form of dense features, such as road shape, sidewalks, structures, alleys, and lanes with atypical shapes.

[0058] A second approach to transforming the relevant surface 3100 into a backward-propagating loss involves preparing the ground's true pose as a ground-true Gaussian distribution surface with the same dimensions as the inferred relevant surface 3100 calculated as described above by correlating a rotated first feature map 1600 with a second feature map 2500, and performing a pixel loss on the inferred relevant surface 3100 relative to the ground-true Gaussian distribution surface. This second approach has the effect of guiding the neural network to learn more focused features. This method enables the neural network to focus on specific parts of the road shape, rather than the scattered road shape.

[0059] Figure 4 As shown Figure 3 The diagram shows a schematic of the software architecture of a system for learning neural networks. The learning process can be performed on the first processor 60 ( Figure 1 It is executed on the first processor 60. In particular, the first processor 60 includes different processing modules, which will be described below, with a focus on the learning principles of the methods applied to learning neural networks.

[0060] In an exemplary embodiment, sensor data 1100 of the environment 50 through which the vehicle navigates is obtained from one or more sensors of the vehicle, and is provided to a first image acquisition module 1200 to generate a sensor-based image 1300. The sensor-based image 1300 is provided to a first feature extraction module 1400. The first feature extraction module 1400 extracts a subset of features from the sensor-based image 1300 to provide a first feature map 1500.

[0061] In an exemplary embodiment, aerial image data 2100 of the environment is obtained and provided to a second image acquisition module 2200 that generates an aerial image 2300 based on the aerial image data 2100. The aerial image 2300 is provided to a second feature extraction module 2400. The second feature extraction module 2400 extracts a subset of features from the aerial image 2300 to provide a second feature map 2500.

[0062] In an exemplary embodiment, the correlation module 3000 correlates the first feature map 1500 with the second feature map 2500 to provide a correlation result 3100.

[0063] In an exemplary embodiment, the ground reality determination module 3200 determines ground reality data 3300 and then provides it to the loss determination module 3400. The loss value 3600 is determined by the loss determination module 3400, as referenced above. Figure 3The loss value 3600 is used to learn the neural network, wherein learning the neural network includes learning a first feature extraction module 1400 to extract a subset of features from a sensor-based image 1300, and learning a second feature extraction module 2400 to extract a subset of features from an acquired aerial image 2300.

[0064] Figure 5 This is a block diagram of an exemplary method for learning neural networks to determine the posture of a vehicle (e.g., a battery electric vehicle (BEV)). Figure 5 The method can be derived from Figure 1 The method is executed by the first processor 60 shown. The method includes a camera side 100, an aerial side 200, and related parts 300, the interaction of which will be described in more detail below with reference to the learning process.

[0065] In an exemplary embodiment, the camera side 100 of the method includes capturing an image 110 of the vehicle environment using a camera and / or other sensing devices. This camera image 110 is used as input to the camera side 100 and, in this exemplary embodiment, has three channels, for example, for red, green, and blue. The camera image is provided to a first processor 60 and / or a second processor 44, and specifically undergoes convolutional feature extraction 111 in a BEV convolutional feature extractor, and optionally concatenates 112 with a homography-transformed RGB (red, green, blue) image as features. The features extracted from the convolutional feature extraction 111 are aligned with the BEV by training against a ground-based three-channel RGB aerial patch 211 of an aerial image 210 received from the airside 200. A shallow convolutional layer in the BEV feature generalization module reduces the D-channel features 114 to single-channel features, which are pre-trained 115 relative to a grayscale version of the BEV before being fine-tuned 116 in subsequent steps, where a BEV inference alignment loss L2 is provided. The shadow remediation module detects pixels in the image where shadows may exist (117) based on statistical anomaly detection on the ground real patch 211 in the air, and masks the corresponding pixels (118), providing a per-pixel loss L1 for BEV alignment. The result from camera side 100 is provided to a first feature map, which is rotated (119) before being provided to the relevant part 300. Rotation (119) aligns the image feature viewpoint of the camera image features with the image feature viewpoint of the air image features. Therefore, rotation (119) can take into account the vehicle's orientation, such as obtaining the ego yaw (218) based on the air image (210) on air image side 200. Figure 5 As can be seen, the BEV module includes a convolutional feature extractor and a generalization module.

[0066] In one embodiment, the airside 200 of the method includes capturing an aerial image 210 of the vehicle environment, wherein the aerial image 210 is composed of external entities positioned separately from the vehicle. Figure 5 (Not shown in the image) is provided. For example, aerial image 210 can be provided from an aircraft or satellite, but can also be stored on a storage medium. In this example, aerial image 210 has three channels, for example, for red, green, and blue. The first processor 60 then performs the steps of aerial side 200 described herein. Aerial image 210 is subjected to convolutional feature extraction 212 in an aerial convolutional feature extractor, and the resulting features are passed through another set of shallow convolutional layers in an aerial feature generalization module 213. In one example, if reduced storage usage is required, the resulting features are further passed 214 to an autoencoder. The result of aerial side 200 is a second feature map output as a single-channel generalized feature map. This generalized feature map is pre-trained 216 against grayscale and dual-interpolated versions of the original three-channel RGB aerial image 210, where an aerial feature pre-training loss L4 is provided. If an optional autoencoder is used, the extracted features constitute an n-depth feature map, where the total size of the n-depth feature map is smaller than the single-feature generalized feature map. Otherwise, the single-channel generalized feature map is stored 215 on the vehicle. Furthermore, before associating the first feature map with the second feature map in the relevant section 300, a 217 bilinear interpolation is performed on the single-channel generalized feature map (i.e., the second feature map).

[0067] In the correlation module of the correlation section 300, the first feature map (i.e., the single-channel generalized feature map from the camera side 100) and the second feature map (i.e., the single-channel generalized feature map from the aerial side 200) are correlated 311 to form a correlation surface, which is also referred to herein as the correlation result. In the example, before correlation 311, the two feature maps can be normalized between the values ​​"-1" and "1".

[0068] In an exemplary embodiment, the learning neural network accompanies 312 in using a two-dimensional spatial soft argmax function to determine predicted coordinates in the relevant results and determines ground truth coordinates based on ground truth data. Based on the predicted coordinates and ground truth coordinates, particularly based on the deviation between these coordinates in the relevant results, the loss value of the localization attitude loss L3 is determined 351. L3 may be referred to as the ground truth localization loss. Since the two-dimensional spatial soft argmax function is differentiable, the feature extraction module (for camera side 100 and air side 200) and the generalization module (for camera side 100 and air side 200) can learn to adjust 252 their neural network weights using backpropagation of stochastic gradient descent to extract and select features that best reduce the localization loss, i.e., considering the total loss value of weighted losses L1, L2, L3, and L4. Further loss values ​​can be provided ( Figure 5(Not shown in the image). For example, an additional loss can be provided, which helps the feature extraction neural network find a better convergence point, namely, an aerial feature backward loss. During the training of the feature extractor neural network, it includes a matching loss on bird's-eye view features inferred from the vehicle camera side 100, so that the aerial feature extractor learns not only from relevant results and the true pose on the ground, but also from what is seen from the onboard sensors. In this way, this additional loss helps the extractor learn to converge to a better solution, i.e., converge until a specified accuracy is reached.

[0069] Alternatively, the loss value for determining the positioning attitude loss L3 can be accompanied by determining the true Gaussian distribution of the ground based on real ground data. The loss value L3 is then determined based on the true Gaussian distribution of the ground and related results.

[0070] This means that the total loss value indicating the localization loss can be used to learn the neural network on the camera side 100 and the aerial side 200, and the convolutional feature extraction module and the generalization module extract and select features based on this neural network.

[0071] Figure 6 This is a block diagram illustrating a detailed example of the method steps of a method for determining vehicle attitude using a filtering process according to an embodiment. Emphasis is placed on vehicle 10 ( Figure 1 The attitude determination runtime operations. For the learning steps performed on camera side 100 and air side 200, please refer to... Figure 5 During operation, map features on the airside 200 can be retrieved from a data storage device, such as data storage device 32 of vehicle 10, 215. These map features can be decoded 221 and interpolated 217 to produce a two-dimensional feature matrix used as a second feature map. Camera image 110 is also passed through a feature extractor, and extraction 111 is applied to it to create a BEV two-dimensional feature matrix used as a first feature map after rotation 119, based on the vehicle yaw estimated in step 218. As described herein, the two feature maps are correlated 311, and the correlated surface is passed 320 to a particle filter that uses the correlated surface as an observation model. The state variables of the particle filter are represented by xy coordinates and yaw angle, respectively, providing the position and orientation of vehicle 10 in the environment.

[0072] In an exemplary embodiment, the filtering process may include multiple initial particles, such as 500 particles, randomly scattered around the GPS location of vehicle 10. Each particle is assigned a weight reflecting the magnitude or intensity of each pixel location on the relevant surface. Through iteration, where each particle is perturbed by a random amount in each iteration, particles with higher weights have a greater probability of survival than particles with lower weights. The particles converge around a region of highest peak concentration on the relevant surface, and the particle distribution around this region can be calculated as an attitude distribution, i.e., the average xy coordinates and orientation, and the covariance of each. This attitude distribution can be used alone as the final output localization or attitude determination of the autonomous vehicle 10. Furthermore, as referenced above… Figure 2 The attitude distribution can optionally be fused with other existing localization strategies. By using particle filters, blurred scenes such as long straight roads without longitudinal cues can be used to provide a useful attitude distribution that can provide lateral control of vehicle 10, enabling continued autonomous vehicle operation even in the absence of longitudinal cues.

[0073] Figure 7 Two block diagrams are shown, illustrating a training process 400 for learning a neural network and a runtime operation 500 for a method for determining vehicle attitude according to an embodiment. Figure 7 The left figure shows the training process, while Figure 7 The right-hand figure illustrates the runtime operation of this method. Note that the steps discussed below can be derived from... Figure 1 The processors 44 and 60 shown execute the training process. In particular, the training process 400 is executed on a first processor 60 that is separate from the vehicle 10, such as a remote processor in the background, while the runtime operation 500 can be executed by a second processor 44, such as the on-board processor of the vehicle 10.

[0074] The training process 400 is shown. Figure 7 The left figure includes steps 410, such as acquiring sensor-based image data from a camera and obtaining vehicle pose information through data collection or crowdsourcing. In step 420, the camera-based image data is fed to a convolutional feature extractor and filtered based on existing techniques with current or initial accuracy. To improve the accuracy of these extractions and filtering, the training process applies neural networks, i.e., machine learning techniques, which are learned based on a loss determined after correlation, as referenced... Figure 3 and 5Specifically, step 440 uses bias correction and preprocessing based on the loss determined in the correlation of feature maps from the camera image obtained in step 410 and the aerial image obtained in step 430. This bias correction is improved, for example, by learning feature extraction and generalization patterns for preparing the correlated first and second feature maps, i.e., by training cross-validation in step 450. In step 460, it is determined whether cross-validation has achieved the desired accuracy. This determination can be made relative to specific requirements of vehicle pose determination. If it is determined that cross-validation has achieved the desired accuracy and meets the requirements, such that existing or initial accuracy can be improved, then in step 470, the corresponding features that can be prioritized based on assigned weights are stored as aerial features. For example, weights indicating the prioritization of each feature can be stored. However, if it is determined that cross-validation has not achieved the desired accuracy and therefore does not meet the requirements, resulting in no improvement in existing or initial accuracy, then in step 480, flags are generated for at least some regions indicating that the loss must be further reduced, possibly by initiating further iterations starting from step 410.

[0075] Figure 7 The right figure illustrates runtime operation 500 of a method for determining vehicle position, including step 510 of loading and retrieving stored weights established during training process 400. This may include loading the stored weights when a region with stored abstract features is nearby, i.e., when the vehicle approaches a region where stored abstract features are detected. In step 520, the process includes obtaining a generalized feature map via a forward-passing camera feed from the vehicle's camera. In step 530, the position and orientation of vehicle 10, i.e., vehicle pose, are determined, and uncertainty is calculated using a particle filter. Uncertainty calculation may be performed before triggering abstract localization of vehicle 10. In step 540, map features may be updated during runtime operation 400, and the uncertainty value, i.e., the increment, is sent to a background, such as a first processor 60. In step 550, the obtained estimated localization or pose can then be used directly to determine the vehicle's position and orientation, or it can be fused with an existing localization strategy, such as SVM, etc.

[0076] In summary, the methods and systems described herein avoid any type of manual feature annotation, instead relying on the acquired ground-based true attitude of the vehicle and its time-synchronized onboard sensor data, i.e., camera images. Using geotagged aerial imagery, a feature extraction neural network is learned in an offline process 400 executed by a first processor 60, and then the extracted features are used during onboard runtime operation 500, in which camera-side features are processed on the vehicle, specifically by a second processor 44, from the raw camera feed. The first processor 60 has already processed the aerial feature side.

[0077] While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be understood that numerous variations exist. It should also be understood that the one or more exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of this disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient roadmap for implementing one or more exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. A computer-implemented method for learning a neural network to determine a vehicle pose in an environment, comprising: obtaining, by a first processor, sensor data of the environment to provide a sensor-based image; performing, by the first processor, a first feature extraction, wherein features are extracted from the sensor-based image to provide a first feature map; obtaining, by the first processor, aerial image data of the environment; performing, by the first processor, a second feature extraction, wherein features are extracted from the obtained aerial image data to provide a second feature map; correlating, by the first processor, the first feature map with the second feature map to provide a correlation result; obtaining, by the first processor, ground truth data; and learning, by the first processor, a neural network using the correlation result and the ground truth data, wherein learning the neural network includes learning the first feature extraction to extract a portion of features from the sensor-based image and learning the second feature extraction to extract a portion of features from the obtained aerial image data, wherein learning the neural network includes: determining, using a soft argmax function, predicted coordinates in the correlation result; determining, based on the ground truth data, ground truth coordinates; and determining, based on the predicted coordinates and the ground truth coordinates, a loss value. learning the neural network includes:

2. The method of claim 1, wherein, determining, based on the ground truth data, a ground truth Gaussian distribution; and wherein the loss value is determined based on the ground truth Gaussian distribution and the correlation result.

3. The method of claim 1, further comprising: reducing, by the first processor, the loss value over an iteration process in which the loss value is repeatedly determined and backpropagated to learn the neural network.

4. The method of claim 3, further comprising: determining, by the first processor, a prioritized feature group of the portion of features extracted from the sensor-based image; and determining, by the first processor, a deprioritized feature group of the portion of features extracted from the sensor-based image; wherein the prioritized feature group is assigned a higher impact on reducing the loss value than the deprioritized feature group.

5. The method of claim 1, further comprising: learning, by the first processor, the neural network until a specified precision of convergence is reached, thereby providing a learned neural network.

6. The method of claim 5, further comprising: determining, by the first processor, a second feature map after the specified precision of convergence is reached; and encoding, by the first processor, the second feature map and storing the second feature map on a storage medium.

7. The method of claim 6, further comprising: determining, by a second processor, a pose of the vehicle using the learned neural network, wherein the second processor is an on-board processor of the vehicle.

8. The method of claim 7, determining, by the second processor, the pose of the vehicle using the learned neural network includes: retrieving, by the second processor, the second feature map from the storage medium; obtaining, by the second processor, sensor data of the environment through which the vehicle is navigating to provide a sensor-based image; wherein, extracting, by the second processor, features from the sensor-based image to provide a first feature map; processing, by the second processor, the first feature map using the learned neural network; correlating, by the second processor, the first feature map with the second feature map to provide a correlation result; ​ ​ ​ determine a pose of the vehicle based on the correlation result.

9. A system for determining a position of a vehicle in an environment, comprising: a first processor configured to: obtain sensor data of the environment to provide a sensor-based image; perform a first feature extraction in which features are extracted from the sensor-based image to provide a first feature map; obtain aerial image data of the environment; perform a second feature extraction in which features are extracted from the obtained aerial image data to provide a second feature map; correlate the first feature map with the second feature map to provide a correlation result; obtain ground truth data; learn a neural network using the correlation result and the ground truth data, wherein learning the neural network comprises learning the first feature extraction to extract a portion of features from the sensor-based image and learning the second feature extraction to extract a portion of features from the obtained aerial image data; and a second processor on the vehicle configured to determine a pose of the vehicle using the learned neural network, wherein the learned neural network comprises: determining predicted coordinates in the correlation result using a soft argmax function; determining ground truth coordinates based on the ground truth data; and determining a loss value based on the predicted coordinates and the ground truth coordinates.

Images