Map visual positioning method and system based on deep neural network matching
Through deep neural network extraction and matching of semantic features of urban road images and maps, the problem of inaccurate positioning in traditional navigation systems in urban environments is solved, and high-precision sub-meter-level positioning is achieved, suitable for fields such as autonomous driving and augmented reality. .
Patent Information
- Application Number
- CN202411022219.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-22
AI Technical Summary
Traditional navigation systems are inaccurately positioned in urban environments, making it difficult to make full use of the semantic information of urban roads, resulting in insufficient positioning accuracy and reliability, especially in areas such as autonomous driving and augmented reality, which is difficult to meet high requirements.
Using the image visual positioning method based on deep neural networks, by building a deep neural network that locates images, including image network, map network and BEV network, extracts semantic features of images and maps, generates BEV representations, and matches them to estimate the camera pose probability distribution. , determine the position and orientation of the image on the map.
It realizes sub-meter-level high-precision positioning, far exceeding the traditional methods, reduces storage and computing costs, and is suitable for autonomous driving, intelligent transportation and augmented reality devices, providing a more efficient and economical solution.
Smart Images

Figure CN119006593A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision and machine learning, and specifically relates to a map vision positioning method and system based on deep neural network matching. Background Art
[0002] With the increasing demand for urban intelligent navigation and positioning technology, especially in the fields of autonomous driving, intelligent transportation systems and augmented reality. Traditional navigation systems often rely on GPS technology, but in urban environments, GPS signals may be blocked by buildings, trees, etc., resulting in inaccurate positioning. Therefore, more accurate and robust urban positioning solutions have been sought to meet the higher requirements of emerging technologies and applications for positioning accuracy and reliability. With the development of deep learning technology, image processing and computer vision have become new ways to solve urban positioning challenges. Traditional positioning methods usually rely on prior maps and sensor fusion, which still have some problems for high-precision positioning in large-scale urban environments. In this context, in-depth processing and analysis of urban road images has become an important way to solve the problem of urban positioning. Features such as signs, buildings, and intersections on urban roads provide rich location information, but traditional methods are difficult to fully mine and utilize this information.
[0003] In urban environments, it is a key challenge to enable vehicles or devices to accurately understand complex road structures and traffic scenes in order to achieve more accurate positioning and navigation. Emerging technologies such as augmented reality (AR) have put forward higher requirements for high-precision positioning and direction information to effectively realize the integration of the virtual and real worlds. Therefore, deep learning positioning methods for urban road images have become crucial. Traditional city maps usually only contain geometric information, but lack the full utilization of semantic information. By introducing convolutional neural networks (CNN) into the field of image positioning and utilizing their powerful feature extraction capabilities, the processing of image semantic information has become more efficient. On the other hand, the acquisition and processing of map data has always been a key issue in urban positioning research. Open map data such as open street maps provide a large amount of urban structure and road information, but how to combine this information with image data to build a neural map that contains both geometric information and semantic information has always been an urgent problem to be solved in the field of urban positioning. Through deep learning, convolutional neural networks and innovative processing of map data, a more efficient and accurate urban image positioning solution can be achieved to promote the development of urban intelligent navigation and related technologies. Summary of the invention
[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a map visual positioning method and system based on deep neural network matching.
[0005] In order to achieve the above object, the present invention adopts the following technical solutions:
[0006] One aspect of the present invention provides a map visual positioning method based on deep neural network matching, comprising the following steps:
[0007] Building an image positioning deep neural network and forming a data set for training and verification, and training the image positioning deep neural network; the image positioning deep neural network includes an image network, a map network and a BEV network;
[0008] Obtain urban road image data, resize and remove noise from the image data, extract semantic features of the image using the image network, and convert the extracted semantic features into BEV representation of the image using the BEV network;
[0009] Obtain city map data, process the map data using the map network, extract the semantic and geometric information of the map, and generate a neural map representation;
[0010] The BEV representation of the image is matched with the neural map representation to estimate the camera pose probability distribution and thus determine the position and orientation of the image on the map.
[0011] As a preferred technical solution, in the image positioning deep neural network, the image network and the map network adopt a U-Net structure, wherein the image network adopts a ResNet-101 encoder and the map network adopts a VGG-16 encoder; the BEV network includes 4 residual blocks.
[0012] As a preferred technical solution, the data set for training and verification is formed to train the deep neural network for image localization, specifically:
[0013] Collect image data from public datasets and websites to build datasets for training and validating deep neural networks for image localization;
[0014] Use the images and map information in the public dataset to train the image localization deep neural network; divide the dataset into training set, validation set and test set, and use the training set and validation set to train and tune the image localization deep neural network.
[0015] As a preferred technical solution, the acquisition of urban road image data, resizing of the image data, and noise removal processing are specifically as follows:
[0016] An urban road image is acquired and elements are extracted, including roads and buildings, the elements are converted into a raster image, duplicate elements are removed, holes are filled, and edges are smoothed.
[0017] As a preferred technical solution, the semantic features are promoted to the BEV space by relying on monocular depth estimation and converted into the BEV representation of the image, specifically:
[0018] After the image is semantically segmented, the image column is mapped to the polar coordinate ray and a U×V feature map X∈R is extracted from the image. U×V×N , consider D depth planes sampled at regular intervals Δ in front of the camera, with values {i·Δ|i∈{1…D}}, and the image is aligned with gravity, and each U column in X corresponds to a vertical plane in 3D space; for each polar coordinate cell (u,v), predict a probability distribution α u,d ∈[0,1] V , overwriting the pixels in the corresponding image columns:
[0019]
[0020] Consider a set S of logarithmically distributed scales, where scale is the ratio of the size of an object in the 3D world to its size in the image:
[0021]
[0022] Predict a score vector S for each pixel (u,v) u,v , whose elements correspond to the scale container σ, and polar coordinate features α are obtained for each depth container d u,d,v :
[0023]
[0024] Where [·] represents linear interpolation;
[0025] The polar coordinate features are mapped to a Cartesian grid of size L×D by linear interpolation along the lateral direction, and U polar coordinate rays are mapped to L columns of the same interval Δ, which are then processed by the BEV network to generate a feature grid, and the BEV representation T∈R of the image consisting of N-dimensional features distributed on an L×D grid aligned with the camera cone is output. L×D×N and the confidence matrix C∈[0,1] L×D .
[0026] As a preferred technical solution, the city map data is obtained and the map data is processed using a map network, specifically:
[0027] The elements in the map data are defined as polygonal areas, polylines or single points according to semantic categories; the polygonal areas include building outlines, lawn areas and parking lots, the polylines include road or sidewalk centerlines, building outlines, and the single points include trees, bus stops, and stores;
[0028] Rasterize polygonal regions, polylines, and single points into a 3-channel image with a fixed ground sampling distance Δ.
[0029] As a preferred technical solution, the extraction of semantic and geometric information of the map and the generation of a neural map representation are specifically as follows:
[0030] Each semantic category is associated with a learned N-dimensional embedding, generating a W×H×3N feature map, which is encoded through a map network to extract geometric features useful for localization and predict a unary location prior Ω∈R for each map unit. W×H , the score reflects the probability of taking an image at each location; finally it is encoded into a neural map representation F of size W×H×N that combines geometric and semantic information.
[0031] As a preferred technical solution, the BEV representation of the image is matched with the neural map representation to estimate the camera pose probability distribution, specifically:
[0032] Estimate the discrete probability distribution over the camera pose ξ: The pose space is reduced to three dimensions. The 3-DoF pose ξ = (x, y, θ), consisting of a position (x, y) and a heading angle θ, is discretized into K rotations sampled at regular intervals for each map position, generating a W×H×K probability volume P as a combination of the image-map match M and the position prior Ω:
[0033] P = softmax(M + Ω);
[0034] Where M and Ω represent the image-dependent and image-independent unnormalized logarithmic scores, respectively;
[0035] Image-map matching: When matching the neural map representation F with the BEV representation T of the image, a volume fraction M is generated, each element of which is calculated by associating the neural map representation F with the BEV representation T of the image transformed by the corresponding pose:
[0036]
[0037] Among them: ξ(p) converts the 2D point p in BEV to the map coordinate system, and the confidence C is used to mask the correlation and ignore the occluded area;
[0038] According to the probability distribution, the position and orientation with the highest probability are selected to determine the precise position and orientation of the image on the map, and a single pose is estimated by maximum likelihood estimation:
[0039] ξ * = argmax ξ P(ξ|I,map,ξ prior );
[0040] Finally, training and optimization are performed through the back propagation algorithm. The training data includes matching pairs of images and maps and corresponding camera pose labels. The optimization goal is to minimize the error between the predicted position and the actual position.
[0041] Another aspect of the present invention further provides a map visual positioning system based on deep neural network matching, which is applied to the above-mentioned map visual positioning method based on deep neural network matching, including an image positioning deep neural network construction and training module, an image semantic feature extraction and BEV representation inference module, a map neural network encoding module, and a camera posture estimation and positioning module;
[0042] The image positioning deep neural network construction and training module is used to build an image positioning deep neural network and form a data set for training and verification, and to train the image positioning deep neural network; the image positioning deep neural network includes an image network, a map network and a BEV network;
[0043] The image semantic feature extraction and BEV representation inference module is used to obtain urban road image data, resize the image data, remove noise, extract the semantic features of the image using the image network, and convert the extracted semantic features into the BEV representation of the image using the BEV network;
[0044] The map neural network encoding module is used to obtain city map data, process the map data using the map network, extract the semantic and geometric information of the map, and generate a neural map representation;
[0045] The camera pose estimation and positioning module is used to match the BEV representation of the image with the neural map representation to obtain the probability distribution of the camera pose, determine the exact position and direction of the image on the map based on the probability distribution, and train and optimize through the back propagation algorithm to minimize the error between the predicted position and the actual position.
[0046] Another aspect of the present invention provides a storage medium storing a program, which, when executed by a processor, implements the above-mentioned map visual positioning method based on deep neural network matching.
[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0048] (1) The present invention has achieved remarkable results in image positioning and map matching, achieving high-precision positioning at the sub-meter level on 2D plane maps, far exceeding traditional methods. The use of open data sources to obtain urban road data ensures global versatility and scalability. Compared with traditional 3D point cloud-based methods, the adopted 2D plane map is more compact and efficient, significantly reducing storage and computing costs, and is suitable for embedded devices. This technical solution is widely applicable to multiple fields such as autonomous driving, intelligent transportation systems, and augmented reality devices, providing more efficient and cost-effective solutions for these fields, which has important driving significance. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 is a flow chart of a map visual positioning method based on deep neural network matching according to an embodiment of the present invention;
[0050] Figure 2 It is a structural schematic diagram of a map vision positioning system based on deep neural network matching in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.
[0052] Example
[0053] like Figure 1 As shown, this embodiment provides a map visual positioning method based on deep neural network matching, comprising the following steps:
[0054] S1: Build an image localization deep neural network and form a data set for training and verification to train the image localization deep neural network; the image localization deep neural network includes an image network, a map network and a BEV network.
[0055] S11: In the image localization deep neural network, the image network and the map network adopt a U-Net structure, wherein the image network adopts a ResNet-101 encoder and the map network adopts a VGG-16 encoder; the BEV network includes 4 residual blocks.
[0056] S12: Collect a large amount of image data from public datasets and websites to build a dataset of more than 100,000 images for training and validating deep neural networks for image localization.
[0057] S13: Use the images and map information in the public dataset to train the image localization deep neural network. Divide the dataset into training set, validation set, and test set, and use the training set and validation set to train and tune the image localization deep neural network.
[0058] S2: Obtain urban road image data, resize and remove noise from the image data, extract semantic features of the image using the image network, and convert the extracted features into the Bird's-Eye View (BEV) representation of the image using the BEV network. Specifically, the semantic features are promoted to the BEV space by monocular reasoning, and a BEV representation T∈R distributed on an L×D grid aligned with the camera cone is inferred. L×D×N , consisting of N-dimensional features. Each feature on the grid is assigned a confidence, resulting in a confidence matrix C∈[0,1] L×D ; Confidence is used to help the system determine the importance or reliability of features at each location when processing image and map data to help the system match and locate image and map data.
[0059] S21: Obtain urban road images and perform preprocessing, use OpenStreetMap (OSM) as the map data source, extract elements such as roads and buildings, and convert them into raster images, perform preprocessing operations such as removing duplicate elements, filling holes, and smoothing edges to improve the quality and accuracy of the map.
[0060] S22: Relying on monocular depth estimation, the semantic features are promoted to the BEV space and converted into the BEV representation of the image, specifically:
[0061] After the image is semantically segmented, the image column is mapped to the polar coordinate ray and a U×V feature map X∈R is extracted from the image. U×V×N , consider D depth planes sampled at regular intervals Δ in front of the camera, with values {i·Δ|i∈{1…D}}, and the image is aligned with gravity, and each U column in X corresponds to a vertical plane in 3D space. For each polar coordinate cell (u,v), predict a probability distribution α u,d ∈[0,1] V , overwriting the pixels in the corresponding image columns:
[0062]
[0063] Consider a set S of logarithmically distributed scales, where scale is the ratio of the size of an object in the 3D world to its size in the image:
[0064]
[0065] Predict a score vector S for each pixel (u,v) u,v , whose elements correspond to the scale container σ, and polar coordinate features α are obtained for each depth container d u,d,v :
[0066]
[0067] Where [·] represents linear interpolation;
[0068] S23: Map polar coordinate features to a Cartesian grid of size L×D by linear interpolation along the lateral direction, map U polar coordinate rays to L columns of the same spacing Δ, and then process them through the BEV network to generate a feature grid, and output the BEV representation T∈R of the image consisting of N-dimensional features distributed on an L×D grid aligned with the camera cone L×D×N and the confidence matrix C∈[0,1] L×D .
[0069] S3: Obtain city map data, process the map data using the map network, extract the semantic and geometric information of the map, and generate a neural map representation.
[0070] S31: Map data elements are defined as polygonal areas, polylines, or single points according to their semantic categories. Examples of areas include building outlines, lawn areas, parking lots, lines include road or sidewalk centerlines, building outlines, and points include trees, bus stops, stores, etc. Rasterize areas, lines, and points into a 3-channel image with a fixed ground sampling distance Δ;
[0071] S32: Associate each category with a learned N-dimensional embedding to generate a W×H×3N feature map, encode it through the map network, extract geometric features useful for positioning, and predict a unary location prior Ω∈R for each map unit W×H , the score reflects the probability of taking an image at each location, and is finally encoded into a neural map representation F of size W×H×N that combines geometric and semantic information.
[0072] S4: Through the neural network model, the BEV representation of the image is matched with the neural map of the map to estimate the camera pose probability distribution, thereby determining the position and orientation of the image on the map.
[0073] S41: Estimate a discrete probability distribution over the camera pose ξ, which is multimodal in uncertain scenes, making the pose estimate easy to fuse with other sensors. The pose space is reduced to three dimensions, and the 3-DoF pose ξ = (x, y, θ), consisting of a position (x, y) and a heading angle θ, is discretized into K rotations sampled at regular intervals for each map position. A probability volume P of W×H×K is generated as a combination of the image-map match term M and the position prior Ω:
[0074] P = softmax(M + Ω);
[0075] Where M and Ω represent the image-dependent and image-independent unnormalized logarithmic scores, respectively;
[0076] S42: Image-map matching: When matching the neural map representation F with the BEV representation T of the image, a volume fraction M is generated, each element of which is calculated by associating the neural map representation F with the BEV representation T of the image transformed by the corresponding pose:
[0077]
[0078] Among them: ξ(p) transforms the 2D point p in BEV to the map coordinate system, and the confidence C is used for mask correlation, ignoring some parts of the BEV space, such as occluded areas;
[0079] S43: Select the most likely position and orientation based on the probability distribution, determine the exact position and orientation of the image on the map, and estimate a single pose by maximum likelihood estimation:
[0080] ξ * = argmax ξ P(ξ|I,map,ξ prior );
[0081] S44: Finally, training and optimization are performed through the back propagation algorithm. The training data includes matching pairs of images and maps and corresponding camera pose labels. The optimization goal is to minimize the error between the predicted position and the actual position.
[0082] like Figure 2 As shown, in another embodiment of the present application, a map visual positioning system based on deep neural network matching is provided, the system includes an image positioning deep neural network construction and training module 10, an image semantic feature extraction and BEV representation inference module 20, a map neural network encoding module 30 and a camera posture estimation and positioning module 40;
[0083] The image positioning deep neural network construction and training module 10 is used to build an image positioning deep neural network and form a data set for training and verification, and to train the image positioning deep neural network; the image positioning deep neural network includes an image network, a map network and a BEV network.
[0084] The image semantic feature extraction and BEV representation inference module 20 is used to obtain urban road image data, resize the image data, remove noise, extract the semantic features of the image using the image network, convert the extracted semantic features into the BEV representation of the image using the BEV network, map the pixels in the image to points in a 3D space, and project these points onto a plane to form a BEV image; process the BEV image to infer the 3D structure of the scene.
[0085] The map neural network encoding module 30 is used to obtain city map data, process the map data using a map network, extract the semantic and geometric information of the map, and generate a neural map representation; each pixel represents an area on the map and is embedded with semantic and geometric information.
[0086] The camera attitude estimation and positioning module 30 is used to match the BEV representation of the image with the neural map of the map to obtain the probability distribution of the camera attitude, select the most likely position and orientation based on the probability distribution, determine the precise position and orientation of the image on the map, and train and optimize through the back propagation algorithm to minimize the error between the predicted position and the actual position.
[0087] Furthermore, in the image semantic feature extraction and BEV representation inference module, a convolutional neural network (CNN) is used to extract rich semantic features from the input image. Through layers of convolution and pooling, the module generates a high-dimensional vector containing the key semantic information of the input image. Then, the semantic features are mapped into a form with more geometric structure and semantic information using the Bird's-Eye View (BEV) representation. This process is completed by a rotational convolutional neural network (RCNN), which maps the pixels in the image to points in 3D space and projects them on a plane to form a BEV image. In the application of RCNN, the key features of the image are extracted by convolution and pooling the BEV image. RCNN is rotationally invariant and can effectively process images at different rotation angles, thereby inferring the 3D structure of the scene. By connecting the fully connected layers, the module finally outputs the precise positioning and orientation information of the image on the 2D semantic map.
[0088] Furthermore, in the map neural network encoding module, the two-dimensional map is mapped to an image that matches its size through pixelation processing. The map image is processed using a convolutional neural network (CNN) to extract key semantic and geometric information. The map information processed by CNN is encoded into a neural map, where each pixel represents an area on the map and carries embedded semantic and geometric information, effectively converting the map information into a form that can be understood by deep learning.
[0089] Furthermore, in the camera attitude estimation and positioning module, the steps of estimating attitude and positioning are:
[0090] (1) Matching the BEV representation with the neural map. By calculating the similarity and matching relationship between the image BEV representation and the neural map, the probability distribution of the camera pose is obtained using computer vision and pattern recognition technology.
[0091] (2) Position and orientation selection: By analyzing the probability distribution, the possible position and orientation of the image are determined, thus achieving high-precision positioning and orientation of the image;
[0092] (3) Back propagation training and optimization: Back propagation algorithm is used for training and optimization to minimize the error between the predicted position and the true position, and the model parameters are continuously adjusted so that the model can estimate the camera's posture and position more accurately.
[0093] It should be noted here that the system provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure is divided into different functional modules to complete all or part of the functions described above. The system is a map visual positioning method based on deep neural network matching applied to the above embodiment.
[0094] In another embodiment of the present application, a storage medium is further provided, storing a program, and when the program is executed by a processor, a map visual positioning method based on deep neural network matching of the above embodiment is implemented, specifically:
[0095] S1. Build an image positioning deep neural network and form a data set for training and verification to train the image positioning deep neural network; the image positioning deep neural network includes an image network, a map network and a BEV network;
[0096] S2, obtaining urban road image data, resizing and removing noise from the image data, extracting semantic features of the image using the image network, and converting the extracted semantic features into BEV representation of the image using the BEV network;
[0097] S3, obtain city map data, use the map network to process the map data, extract the semantic and geometric information of the map, and generate a neural map representation;
[0098] S4. Match the BEV representation of the image with the neural map representation, estimate the camera pose probability distribution, and determine the position and orientation of the image on the map.
[0099] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0100] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A map visual positioning method based on deep neural network matching, characterized in that: The steps include: Building an image positioning deep neural network and forming a data set for training and verification, and training the image positioning deep neural network; the image positioning deep neural network includes an image network, a map network and a BEV network; Obtain urban road image data, resize and remove noise from the image data, extract semantic features of the image using the image network, and convert the extracted semantic features into BEV representation of the image using the BEV network; Obtain city map data, process the map data using the map network, extract the semantic and geometric information of the map, and generate a neural map representation; The BEV representation of the image is matched with the neural map representation, the camera pose probability distribution is estimated, and the position and orientation of the image on the map are determined.
2. According to the method of claim 1, the map visual positioning method based on deep neural network matching is characterized in that: In the image localization deep neural network, the image network and the map network adopt a U-Net structure, wherein the image network adopts a ResNet-101 encoder and the map network adopts a VGG-16 encoder; the BEV network includes 4 residual blocks.
3. According to the map visual positioning method based on deep neural network matching according to claim 1, it is characterized in that: The dataset used for training and verification is constructed to train the deep neural network for image localization, specifically: Collect image data from public datasets and websites to build datasets for training and validating deep neural networks for image localization; Use the images and map information in the public dataset to train the image localization deep neural network; divide the dataset into training set, validation set and test set, and use the training set and validation set to train and tune the image localization deep neural network.
4. According to the map visual positioning method based on deep neural network matching according to claim 1, it is characterized in that: The obtaining of urban road image data, and resizing and removing noise from the image data are specifically as follows: An urban road image is acquired and elements are extracted, including roads and buildings, the elements are converted into a raster image, duplicate elements are removed, holes are filled, and edges are smoothed.
5. According to the map visual positioning method based on deep neural network matching according to claim 1, it is characterized in that: Relying on monocular depth estimation, the semantic features are promoted to the BEV space and converted into the BEV representation of the image, specifically: After the image is semantically segmented, the image column is mapped to the polar coordinate ray and a U×V feature map X∈R is extracted from the image. U×V×N , consider D depth planes sampled at regular intervals Δ in front of the camera, with values {i·Δ|i∈{1…D}}, and the image is aligned with gravity, and each U column in X corresponds to a vertical plane in 3D space; for each polar coordinate cell (u,v), predict a probability distribution α u,d ∈[0,1] V , overwriting the pixels in the corresponding image columns: Consider a set S of logarithmically distributed scales, where scale is the ratio of the size of an object in the 3D world to its size in the image: Predict a score vector S for each pixel (u,v) u,v , whose elements correspond to the scale container σ, and polar coordinate features α are obtained for each depth container d u,d,v : Where [·] represents linear interpolation; The polar coordinate features are mapped to a Cartesian grid of size L×D by linear interpolation along the lateral direction, and U polar coordinate rays are mapped to L columns of the same interval Δ, which are then processed by the BEV network to generate a feature grid, and the BEV representation T∈R of the image consisting of N-dimensional features distributed on an L×D grid aligned with the camera cone is output. L×D×N and the confidence matrix C∈[0,1] L×D .
6. The map visual positioning method based on deep neural network matching according to claim 1 is characterized in that: The city map data is obtained and the map data is processed using the map network, specifically: The elements in the map data are defined as polygonal areas, polylines or single points according to semantic categories; the polygonal areas include building outlines, lawn areas and parking lots, the polylines include road or sidewalk centerlines, building outlines, and the single points include trees, bus stops, and stores; Rasterize polygonal regions, polylines, and single points into a 3-channel image with a fixed ground sampling distance Δ.
7. A map visual positioning method based on deep neural network matching according to claim 6, characterized in that: The extraction of semantic and geometric information of the map and generation of a neural map representation is specifically as follows: Each semantic category is associated with a learned N-dimensional embedding, generating a W×H×3N feature map, which is encoded through a map network to extract geometric features useful for localization and predict a unary location prior Ω∈R for each map unit. W×H , the score reflects the probability of taking an image at each location; finally it is encoded into a neural map representation F of size W×H×N that combines geometric and semantic information.
8. The map visual positioning method based on deep neural network matching according to claim 1 is characterized in that: The BEV representation of the image is matched with the neural map representation to estimate the camera pose probability distribution, specifically: Estimate the discrete probability distribution over the camera pose ξ: The pose space is reduced to three dimensions. The 3-DoF pose ξ = (x, y, θ), consisting of a position (x, y) and a heading angle θ, is discretized into K rotations sampled at regular intervals for each map position, generating a W×H×K probability volume P as a combination of the image-map match M and the position prior Ω: P = softmax(M + Ω); Where M and Ω represent the image-dependent and image-independent unnormalized logarithmic scores, respectively; Image-map matching: When matching the neural map representation F with the BEV representation T of the image, a volume fraction M is generated, each element of which is calculated by associating the neural map representation F with the BEV representation T of the image transformed by the corresponding pose: Among them: ξ(p) converts the 2D point p in BEV to the map coordinate system, and the confidence C is used to mask the correlation and ignore the occluded area; According to the probability distribution, the position and orientation with the highest probability are selected to determine the precise position and orientation of the image on the map, and a single pose is estimated by maximum likelihood estimation: x * =argmax ξ P(ξ|I,map,ξ prior ); Finally, training and optimization are performed through the back propagation algorithm. The training data includes matching pairs of images and maps and corresponding camera pose labels. The optimization goal is to minimize the error between the predicted position and the actual position.
9. A map visual positioning system based on deep neural network matching, characterized in that: A map visual positioning method based on deep neural network matching applied to any one of claims 1-8, comprising an image positioning deep neural network construction and training module, an image semantic feature extraction and BEV representation inference module, a map neural network encoding module, and a camera posture estimation and positioning module; The image positioning deep neural network construction and training module is used to build an image positioning deep neural network and form a data set for training and verification, and to train the image positioning deep neural network; the image positioning deep neural network includes an image network, a map network and a BEV network; The image semantic feature extraction and BEV representation inference module is used to obtain urban road image data, resize the image data, remove noise, extract the semantic features of the image using the image network, and convert the extracted semantic features into the BEV representation of the image using the BEV network; The map neural network encoding module is used to obtain city map data, process the map data using the map network, extract the semantic and geometric information of the map, and generate a neural map representation; The camera pose estimation and positioning module is used to match the BEV representation of the image with the neural map representation to obtain the probability distribution of the camera pose, determine the exact position and direction of the image on the map based on the probability distribution, and train and optimize through the back propagation algorithm to minimize the error between the predicted position and the actual position.
10. A storage medium storing a program, characterized in that: When the program is executed by a processor, a map visual positioning method based on deep neural network matching as described in any one of claims 1 to 8 is implemented.
