Passenger car cross-modal positioning method and system based on data driving

Abstract

This invention relates to a data-driven cross-modal positioning method and system for passenger vehicles. The method includes: M1. Image preprocessing: acquiring a single-frame image captured by a forward-looking camera and preprocessing the image; M2. Map rendering: extracting OSM map data for a 64m×64m area centered on the initial position provided by GNSS; M3. Depth information estimation: using the camera intrinsic parameter matrix and combining it with a monocular depth estimation network, performing dense depth estimation on the preprocessed image to generate a depth map aligned with the image resolution, providing geometric constraints for subsequent 3D spatial feature projection; M4. Image feature extraction branch. This invention not only effectively solves the limitations of traditional methods that rely on multiple consecutive images or high-precision maps by organically combining single-frame images, OSM maps, and depth information through an innovative input data construction method, but also achieves significant improvements in positioning accuracy, computational efficiency, and scene robustness.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a data-driven cross-modal positioning method and system for passenger vehicles. Background Technology

[0002] In recent years, deep learning has been revolutionizing positioning technology. It overcomes the limitations of traditional methods that rely on human features and physical models, achieving smarter and more robust positioning by learning the inherent patterns in data. Simply put, the core value of this technology lies in the fact that when traditional physical methods such as GPS or base station ranging fail, deep learning enables machines to infer their location by "seeing" images, "hearing" signals, or "understanding" verbal descriptions.

[0003] Traditional positioning methods, such as GPS, are prone to failure in indoor or urban canyon environments because they rely on direct signal interaction with external devices such as satellites. Deep learning, on the other hand, excels in its powerful automatic feature extraction and pattern recognition capabilities. It utilizes feature self-learning: eliminating the need for manually designed features (such as the formula for WiFi signal strength versus distance), and automatically learns complex, difficult-to-define patterns from massive amounts of data. It also enables multimodal data fusion: deep learning can effectively fuse different types of input, such as combining visual images and textual descriptions for positioning, which is difficult to achieve with traditional methods. Summary of the Invention

[0004] In view of the above problems, the present invention provides a data-driven cross-modal positioning method and system for passenger vehicles. It not only effectively solves the limitations of traditional methods that rely on multiple consecutive images or high-precision maps by organically combining single-frame images, OSM maps and depth information through an innovative input data construction method, but also achieves significant improvements in positioning accuracy, computational efficiency and scene robustness.

[0005] To achieve the above and other related objectives, the present invention provides the following technical solution: A data-driven cross-modal localization method for passenger vehicles, the method comprising: M1. Image preprocessing: Acquire a single-frame image captured by the front-view camera and preprocess the image. M2. Map rendering: Using the initial position provided by GNSS as the center, OSM map data of a 64m×64m area is extracted. M3. Depth information estimation: Using the camera intrinsic parameter matrix and combined with a monocular depth estimation network, dense depth estimation is performed on the preprocessed image to generate a depth map aligned with the image resolution, providing geometric constraints for subsequent 3D spatial feature projection. M4. Image feature extraction branch uses a residual network combined with a visual Transformer structure to extract multi-scale global and local features of a 512px image; The M5 map feature extraction branch uses a lightweight convolutional neural network to encode the three-channel semantic index map and extract the topological and semantic layout features of local regions. M6. Cross-modal fusion introduces a cross-attention mechanism, using image features as query vectors and map features as keys and values ​​to achieve soft alignment between image observations and map priors; M7. The regression head maps the fused features into three-dimensional vectors through a multilayer perceptron, corresponding to the X coordinate, Y coordinate, and orientation angle Yaw in the OSM map coordinate system.

[0006] Furthermore, in step M1, the preprocessing of the image includes: M11. Distortion correction is performed using camera calibration parameters to eliminate radial and tangential distortion of the lens; M12. Calculates the pitch angle based on gravity sensor or vehicle attitude information, performs pitch correction on the image, and keeps the horizon level. M13. Scale the corrected images to a uniform resolution of 512×512 pixels to fit the input requirements of the neural network.

[0007] Furthermore, in step M2, the OSM map data of the 64m×64m area is extracted by rendering the road, building outlines and vegetation features into a three-channel semantic index map, where different channels represent the road centerline, road edge and building footprint, and the pixel value represents the semantic category index.

[0008] Furthermore, in step M3, the depth estimation network includes an image feature extraction branch, a map feature extraction branch, a cross-modal fusion module, and a regression head.

[0009] Furthermore, in step M4, the extraction of multi-scale global and local features of the 512px image using a residual network combined with a visual Transformer structure includes: M41. The local feature extraction branch uses an improved residual network as its backbone; M42. Global feature extraction branch, using a lightweight visual Transformer; M43. Multi-scale feature fusion integrates the global features output by the Transformer with the multi-scale local features output by the residual network in a top-down and bottom-up bidirectional manner.

[0010] Furthermore, the improved residual network serves as the backbone for an input image of 512px×512px. After four residual stages, it outputs feature maps at four scales with resolutions of 128px, 64px, 32px, and 16px, respectively. Each residual block employs a pre-activation structure and introduces a channel attention mechanism to enhance the response of key local features.

[0011] Furthermore, the lightweight visual Transformer flattens the 16px feature map output by the residual network into a sequence, inputs it into the Transformer encoder, and calculates the dependency relationship between global tokens through a multi-head self-attention mechanism to capture long-distance contextual information.

[0012] Furthermore, in the cross-modal fusion, a cross-attention mechanism is introduced. The global features of the Transformer are used as the query vector, and the local features of the residual network are used as the key and value. By calculating the attention weight of the global context to the local details, global semantic information is accurately injected into the local features, realizing deep coupling between the global and local. After feature fusion, the fused feature map is gradually upsampled through sub-pixel convolution and bilinear interpolation to restore it to the same 512px resolution as the input image, and the final feature map containing rich multi-scale information is output.

[0013] Furthermore, in step M7, the loss function of the regression head is designed to employ a multi-task joint loss function for model training, including the L1 loss for position regression, the periodic loss for orientation angle regression, and the depth consistency constraint loss, to ensure the model's prediction accuracy in translation and rotation. To achieve the above and other related objectives, the present invention also provides a data-driven cross-modal positioning system for passenger vehicles, including a computer device programmed or configured to perform the steps of the data-driven cross-modal positioning method for passenger vehicles.

[0014] The present invention has the following positive effects: This invention, through an innovative input data construction method, organically combines single-frame images, OSM maps, and depth information, effectively overcoming the limitations of traditional methods that rely on multiple consecutive images or high-precision maps. Experimental results show that this method achieves significant improvements in positioning accuracy, computational efficiency, and scene robustness, especially performing exceptionally well in GNSS-constrained scenarios such as urban canyons, providing a low-cost, high-precision global positioning solution for autonomous driving. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the model architecture of the present invention; Figure 3 This is a schematic diagram of the map feature extraction branch of the present invention; Figure 4 This is a schematic diagram illustrating the model training and deployment of the present invention. Detailed Implementation

[0016] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0017] Example: Figure 1 or Figure 2 or Figure 3 or Figure 4 As shown, a data-driven cross-modal localization method for passenger vehicles includes: M1. Image preprocessing: Acquire a single-frame image captured by the front-view camera and preprocess the image. M2. Map rendering: Using the initial position provided by GNSS as the center, OSM map data of a 64m×64m area is extracted. M3. Depth information estimation: Using the camera intrinsic parameter matrix and combined with a monocular depth estimation network, dense depth estimation is performed on the preprocessed image to generate a depth map aligned with the image resolution, providing geometric constraints for subsequent 3D spatial feature projection. M4. Image feature extraction branch uses a residual network combined with a visual Transformer structure to extract multi-scale global and local features of a 512px image; The M5 map feature extraction branch uses a lightweight convolutional neural network to encode the three-channel semantic index map and extract the topological and semantic layout features of local regions. M6. Cross-modal fusion introduces a cross-attention mechanism, using image features as query vectors and map features as keys and values ​​to achieve soft alignment between image observations and map priors; M7. The regression head maps the fused features into three-dimensional vectors through a multilayer perceptron, corresponding to the X coordinate, Y coordinate, and orientation angle Yaw in the OSM map coordinate system.

[0018] In this embodiment, HDMap is stored in a vector format, and the element types are mainly divided into three different shape representations: Point (Pole, Trunk), Line (LaneBoundary, MapBoundary, StopLine), and Polygon (CrossWalk, Arrow).

[0019] This module encodes vector elements of different shapes, types, and individuals into the input required by the learning network. The main encoded information includes the element's position, shape, type, and instance information. In this embodiment, step M1, the preprocessing of the image includes: M11. Distortion correction is performed using camera calibration parameters to eliminate radial and tangential distortion of the lens; M12. Calculates the pitch angle based on gravity sensor or vehicle attitude information, performs pitch correction on the image, and keeps the horizon level. M13. Scale the corrected images to a uniform resolution of 512×512 pixels to fit the input requirements of the neural network.

[0020] In this embodiment, in step M2, the OSM map data of the 64m×64m area is extracted by extracting the features of roads, building outlines and vegetation, and then rendering it into a three-channel semantic index map. The different channels represent the road centerline, road edge and building footprint, respectively, and the pixel value represents the index of the semantic category.

[0021] In this embodiment, in step M3, the depth estimation network includes an image feature extraction branch, a map feature extraction branch, a cross-modal fusion module, and a regression head.

[0022] In this embodiment, step M4, which involves using a residual network combined with a visual Transformer structure to extract multi-scale global and local features of the 512px image, includes: M41. The local feature extraction branch uses an improved residual network as its backbone; M42. Global feature extraction branch, using a lightweight visual Transformer; M43. Multi-scale feature fusion integrates the global features output by the Transformer with the multi-scale local features output by the residual network in a top-down and bottom-up bidirectional manner.

[0023] In this embodiment, the improved residual network serves as the backbone for an input image of 512px×512px. After four residual stages, it outputs feature maps at four scales with resolutions of 128px, 64px, 32px, and 16px, respectively. Each residual block employs a pre-activation structure and introduces a channel attention mechanism to enhance the response of key local features.

[0024] In this embodiment, the lightweight visual Transformer flattens the 16px feature map output by the residual network into a sequence, inputs it into the Transformer encoder, and calculates the dependency relationship between global tokens through a multi-head self-attention mechanism to capture long-distance contextual information.

[0025] In this embodiment, a cross-attention mechanism is introduced in the cross-modal fusion. The global features of the Transformer are used as the query vector, and the local features of the residual network are used as the key and value. By calculating the attention weight of the global context to the local details, the global semantic information is accurately injected into the local features, realizing the deep coupling between the global and local. After feature fusion, the fused feature map is gradually upsampled through sub-pixel convolution and bilinear interpolation to restore it to the same 512px resolution as the input image, and the final feature map containing rich multi-scale information is output.

[0026] In this embodiment, in step M7, the loss function of the regression head is designed to use a multi-task joint loss function for model training, including the L1 loss for position regression, the periodic loss for orientation angle regression, and the depth consistency constraint loss, to ensure the model's prediction accuracy in translation and rotation. In this embodiment, the present invention provides a data-driven cross-modal positioning system for passenger vehicles, including a computer device programmed or configured to perform the steps of the data-driven cross-modal positioning method for passenger vehicles.

[0027] The present invention provides a computer-readable storage medium storing a computer program programmed or configured to perform the aforementioned data-driven cross-modal positioning method for passenger vehicles.

[0028] Any references to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0029] In summary, this invention not only effectively solves the limitations of traditional methods that rely on multiple consecutive images or high-precision maps by organically combining single-frame images, OSM maps, and depth information through an innovative input data construction method, but also achieves significant improvements in positioning accuracy, computational efficiency, and scene robustness.

[0030] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data-driven cross-modal positioning method for passenger vehicles, characterized in that, The method includes: M1. Image preprocessing: Acquire a single-frame image captured by the front-view camera and preprocess the image. M2. Map rendering: Using the initial position provided by GNSS as the center, OSM map data of a 64m×64m area is extracted. M3. Depth information estimation: Using the camera intrinsic parameter matrix and combined with a monocular depth estimation network, dense depth estimation is performed on the preprocessed image to generate a depth map aligned with the image resolution, providing geometric constraints for subsequent 3D spatial feature projection. M4. Image feature extraction branch uses a residual network combined with a visual Transformer structure to extract multi-scale global and local features of a 512px image; The M5 map feature extraction branch uses a lightweight convolutional neural network to encode the three-channel semantic index map and extract the topological and semantic layout features of local regions. M6. Cross-modal fusion introduces a cross-attention mechanism, using image features as query vectors and map features as keys and values ​​to achieve soft alignment between image observations and map priors; M7. The regression head maps the fused features into three-dimensional vectors through a multilayer perceptron, corresponding to the X coordinate, Y coordinate, and orientation angle Yaw in the OSM map coordinate system.

2. The data-driven cross-modal positioning method for passenger vehicles according to claim 1, characterized in that, In step M1, the preprocessing of the image includes: M11. Distortion correction is performed using camera calibration parameters to eliminate radial and tangential distortion of the lens; M12. Calculates the pitch angle based on gravity sensor or vehicle attitude information, performs pitch correction on the image, and keeps the horizon level. M13. Scale the corrected images to a uniform resolution of 512×512 pixels to fit the input requirements of the neural network.

3. The data-driven cross-modal positioning method for passenger vehicles according to claim 1, characterized in that, In step M2, the OSM map data of the 64m×64m area is extracted by rendering the road, building outlines and vegetation features into a three-channel semantic index map. The different channels represent the road centerline, road edge and building footprint, respectively, and the pixel value represents the semantic category index.

4. The data-driven cross-modal positioning method for passenger vehicles according to claim 1, characterized in that, In step M3, the depth estimation network includes an image feature extraction branch, a map feature extraction branch, a cross-modal fusion module, and a regression head.

5. The data-driven cross-modal positioning method for passenger vehicles according to claim 1, characterized in that, In step M4, the extraction of multi-scale global and local features of the 512px image using a residual network combined with a visual Transformer structure includes: M41. The local feature extraction branch uses an improved residual network as its backbone; M42. Global feature extraction branch, using a lightweight visual Transformer; M43. Multi-scale feature fusion integrates the global features output by the Transformer with the multi-scale local features output by the residual network in a top-down and bottom-up bidirectional manner.

6. The data-driven cross-modal positioning method for passenger vehicles according to claim 5, characterized in that: The improved residual network serves as the backbone, taking a 512px×512px image as input. After four residual stages, it outputs feature maps at four scales with resolutions of 128px, 64px, 32px, and 16px, respectively. Each residual block employs a pre-activation structure and introduces a channel attention mechanism to enhance the response of key local features.

7. The data-driven cross-modal positioning method for passenger vehicles according to claim 5, characterized in that: The lightweight visual Transformer flattens the 16px feature map output by the residual network into a sequence, inputs it into the Transformer encoder, and calculates the dependency relationship between global tokens through a multi-head self-attention mechanism to capture long-distance contextual information.

8. The data-driven cross-modal positioning method for passenger vehicles according to claim 1, characterized in that: In the cross-modal fusion, a cross-attention mechanism is introduced. The global features of the Transformer are used as the query vector, and the local features of the residual network are used as the key and value. By calculating the attention weight of the global context to the local details, the global semantic information is accurately injected into the local features, realizing the deep coupling between the global and local. After feature fusion, the fused feature map is gradually upsampled through sub-pixel convolution and bilinear interpolation to restore it to the same 512px resolution as the input image, and the final feature map containing rich multi-scale information is output.

9. The data-driven cross-modal positioning method for passenger vehicles according to claim 1, characterized in that, In step M7, the loss function of the regression head is designed to use a multi-task joint loss function for model training, including L1 loss for position regression, periodic loss for orientation angle regression, and depth consistency constraint loss, to ensure the model's prediction accuracy in translation and rotation.

10. A data-driven cross-modal positioning system for passenger vehicles, comprising computer equipment, characterized in that, The computer device is programmed or configured to perform the steps of the data-driven cross-modal positioning method for passenger vehicles as described in any one of claims 1 to 9.

Images