A visual positioning method, device and storage medium

Abstract

The present application provides a visual positioning method, device and storage medium, which can match with the database of a remote cloud server according to a plane image, so as to achieve the purpose of positioning. The visual positioning method includes the following steps: obtaining the image to be positioned and the camera parameters; extracting the first global descriptor of the image to be positioned; comparing the first global descriptor with the pre-stored second global descriptor, and retrieving multiple first sub-images from a plurality of pre-set sub-images; determining M panoramas associated with the multiple first sub-images according to the multiple first sub-images; splitting the M panoramas into N2 second sub-images with the same horizontal field of view as the image to be positioned according to the camera parameters; performing image matching on the N2 second sub-images and the image to be positioned to obtain a common feature point set; solving PnP according to the common feature point set to obtain the final posture of the image to be positioned.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a visual positioning method, device and storage medium. Background Art

[0002] In MR / AR / VR technology, in order to more comprehensively collect the surrounding environment and perform visual three-dimensional mapping, using a panoramic camera is a good method. Panoramic images are suitable for AR / VR technology and have advantages such as convenience and low cost. Based on the three-dimensional map built by the panoramic camera, the environment image can be obtained through the end-side device and matched with the database of the remote cloud server to achieve positioning. Although the existing visual positioning method can obtain the environment image through the end-side device and match it with the database of the remote cloud server to achieve the purpose of positioning, the environment image is limited to panoramic images, and ordinary plane images cannot be matched with the database of the remote cloud server. Summary of the invention

[0003] One advantage of the present application is that it provides a visual positioning method, which can match a plane image with a database of a remote cloud server to achieve the purpose of positioning.

[0004] Based on this, in order to achieve at least one of the above advantages or other advantages and purposes of the present application, the present application provides a visual positioning method, comprising the following steps:

[0005] Get the image to be positioned and the camera parameters;

[0006] Extracting a first global descriptor of the image to be located;

[0007] Compare the first global descriptor with a pre-stored second global descriptor, retrieve multiple first sub-images from a plurality of pre-set sub-images, the similarity between the second global descriptors of the multiple first sub-images and the first global descriptor of the image to be located meets a preset condition, the second global descriptor is a global descriptor extracted from each sub-image, the sub-image is a plan view obtained by splitting each pre-stored panoramic image based on a preset number N1, and the multiple panoramic images are jointly constructed to obtain a three-dimensional visual map, and the three-dimensional visual map stores the pose information of each panoramic image and feature points on the panoramic image containing spatial 3D information;

[0008] Determine, according to the plurality of first sub-images, M panoramic images associated with the plurality of first sub-images;

[0009] According to the camera parameters, the M panoramic images are split into N2 second sub-images with the same horizontal field of view as the image to be located, the second sub-image is a plane image, and the representation information of the feature point of the second sub-image includes the spatial 3D information of the feature point in the three-dimensional visual map;

[0010] Performing image matching on the N2 second sub-images and the image to be positioned to obtain a common feature point set, wherein the feature point set includes a plurality of common feature points, and the common feature points are feature points existing in both the N2 second sub-images and the image to be positioned;

[0011] Solve PnP according to the common feature point set to obtain the final pose of the image to be positioned.

[0012] According to an embodiment of the present application, the camera parameters include resolution and a camera intrinsic parameter matrix.

[0013] According to one embodiment of the present application, the flat field angle of the sub-image=360° / N1.

[0014] According to an embodiment of the present application, the process of extracting the first global descriptor and the process of extracting the second global descriptor both use a retrieval algorithm to extract the global descriptors.

[0015] According to an embodiment of the present application, the process of extracting the global descriptor using the image retrieval algorithm is: extracting local features of the image through a pre-trained convolutional neural network, and then aggregating the local features into the global descriptor using a VLAD layer.

[0016] According to one embodiment of the present application, the horizontal field angle of the image to be positioned = 360° / N2.

[0017] According to an embodiment of the present application, the process of performing image matching on the N2 second sub-images and the image to be positioned to obtain a common feature point set, wherein the feature point set includes a plurality of common feature points, and the common feature points are feature points existing in both the N2 second sub-images and the image to be positioned is specifically as follows:

[0018] Performing feature extraction on the N2 second sub-images and the image to be positioned, and obtaining feature points of the N2 second sub-images and feature points of the image to be positioned respectively;

[0019] Based on the feature points of the N2 second sub-images and the feature points of the image to be located, a detector-based local feature matching algorithm is used to perform feature matching to obtain the common feature point set.

[0020] According to an embodiment of the present application, the process of performing image matching on the N2 second sub-images and the image to be positioned to obtain a common feature point set, wherein the feature point set includes a plurality of common feature points, and the common feature points are feature points existing in both the N2 second sub-images and the image to be positioned is specifically as follows:

[0021] Based on the N2 second sub-images and the image to be positioned, a detector-free feature matching algorithm is used to perform feature matching to obtain the common feature point set.

[0022] According to another aspect of the present application, the present application further provides a device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned visual positioning method when executing the computer program.

[0023] The present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned visual positioning method are implemented.

[0024] Beneficial effects of the present application: The present application provides a visual positioning method, which splits each panoramic image in a pre-stored three-dimensional visual map into a preset number of multiple sub-images, and the sub-images are plane images, which lays a comparable foundation for the subsequent process of searching for panoramic images that have similarities or associations with the image to be positioned (which is a plane image), and solves the problem that the plane image cannot be compared with the panoramic image in the prior art. Afterwards, the first global descriptor of the image to be positioned is compared with the second global descriptors of the multiple sub-images, and based on the similarity between the first global descriptor and the second global descriptor, multiple first sub-images with a certain degree of similarity to the image to be positioned are screened out. This screening step excludes sub-images that are greatly different from the image to be positioned, reducing the computational workload in the subsequent further refined comparison process. Afterwards, the M panoramas associated with the selected multiple first sub-images are split to obtain multiple second sub-images, which are plane images, and the splitting basis is based on the camera parameters and the horizontal field of view of the image to be positioned, which further lays a comparative foundation for the process of screening out the common feature points of the image to be positioned from the multiple second sub-images, and improves the accuracy of the comparison. Afterwards, the N2 second sub-images are image matched with the image to be positioned to obtain a common feature point set including multiple common feature points, which are feature points existing in both the N2 second sub-images and the image to be positioned; PnP is solved according to the feature point set to obtain the final posture of the image to be positioned. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 A flow chart of a visual positioning method provided by one embodiment of the present application and a visual positioning method provided by another embodiment;

[0027] Figure 2 A schematic diagram of the structure of a visual positioning device provided in other embodiments of the present application;

[0028] Figure 3 A schematic diagram of the structure of a device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0030] Considering that the existing visual positioning method can obtain the environment image through the terminal device and match it with the database of the remote cloud server to achieve the purpose of positioning, its environment image is limited to panoramic images, and ordinary plane images cannot be matched with the database of the remote cloud server. In order to solve this problem, the present application provides a visual positioning method, which can match the database of the remote cloud server based on the plane image, so as to achieve the purpose of positioning.

[0031] For details, please refer to the attached Figure 1 An embodiment of the present application provides a visual positioning method, which may include the following steps:

[0032] Step S1, obtaining the image to be located and camera parameters.

[0033] The image to be located may be an image captured in real time by a camera of the electronic device, and the camera parameters are parameters of the camera of the electronic device.

[0034] Optionally, the camera parameters include resolution and a camera intrinsic parameter matrix.

[0035] In particular, the resolution is: W×H, and the camera intrinsic parameter matrix is:

[0036]

[0037] Among them, W represents the width of the image, and H represents the height of the image. In image processing and computer vision, fx is an important parameter, which represents the value of the focal length parameter f in the image width direction, calculated by the horizontal field of view angle. This parameter is of great significance for understanding and analyzing the geometric characteristics of the image. At the same time, fy is also the focal length parameter f, but it is a value calculated based on the longitudinal field of view angle, reflecting the focal length characteristics in the image height direction. In addition, cx and cy represent half of the image width and height, respectively, that is, cx is equal to half of W, and cy is equal to half of H. These parameters play a key role in locating the geometric center of the image, which facilitates the accurate processing of image data in subsequent operations such as image transformation, correction, and three-dimensional reconstruction. In short, these parameters together provide important basic information for the analysis of perspective, depth, and geometric relationships in image processing.

[0038] Step S2: extracting a first global descriptor of the image to be located.

[0039] In particular, in step S2, the global descriptor of the image to be located is extracted by using an image retrieval algorithm. In other embodiments, the global descriptor may be extracted by other algorithms or methods, such as content-based image retrieval. In this embodiment, the image retrieval algorithm is taken as an example to specifically illustrate the steps of extracting the first global descriptor, as follows:

[0040] Step S21: extracting local features of the image to be located through a pre-trained convolutional neural network (CNN).

[0041] The pre-trained convolutional neural network in this example can adopt the convolutional neural network commonly used in the field of image feature recognition. After being trained with a large amount of image data, the convolutional neural network can automatically recognize and extract local features in the image to be located, such as edges, textures, and shapes. These local features describe the important patterns and related feature information of the image to be located.

[0042] Each local feature is usually obtained through a feature map output by some intermediate layers of the network. Each local feature extracted from the image to be located represents the visual information of a specific area in the image to be located.

[0043] Step S22: Aggregate the local features to generate a global descriptor. The global descriptor can more comprehensively characterize the features of the image to be located.

[0044] The aggregation process can use the VLAD layer ("Vector of Locally Aggregated Descriptors") to aggregate local features. The main function of the VLAD layer is to integrate multiple local features into a global feature vector. Specifically: VLAD generates a set of integrated features by calculating the distance between each local feature and certain representative visual words (i.e., prototype features). These features capture the visual content of the entire image to be located in the form of a vector. In this way, the VLAD layer can aggregate information scattered in different local areas into a unified descriptor that can better represent the characteristics of the entire image to be located. This global descriptor plays a key role in subsequent tasks, such as image retrieval, classification, and recognition, because it provides a compressed and meaningful image representation, allowing computers to more easily understand and process the content of the image.

[0045] Step S3: compare the first global descriptor with a pre-stored second global descriptor, and retrieve a plurality of first sub-images from a plurality of pre-set sub-images.

[0046] The similarity between the second global descriptors of the multiple first sub-images and the first global descriptor of the image to be located meets a preset condition. For example, the similarity between the second global descriptors of the multiple first sub-images and the first global descriptor of the image to be located reaches 80%, or 90%, etc. The preset condition can be a range value pre-set by the system based on an empirical value.

[0047] The second global descriptor is a global descriptor extracted from each sub-image, and the sub-image is a plan view obtained by splitting each pre-stored panoramic image based on a preset number N1. Multiple panoramas are jointly constructed to obtain a three-dimensional visual map, and the three-dimensional visual map stores the posture information of each panorama and the feature points on the panorama containing spatial 3D information.

[0048] It is worth noting that the feature points refer to specific pixel positions that have obvious differences in the image and can be effectively distinguished from the surrounding pixels. These feature points usually show significant visual features in the image, such as strong color changes, texture differences, corners, edges or other high-contrast areas. The uniqueness of feature points gives them strong recognition capabilities in complex visual environments. For example, feature points often have significant contrast, can reflect local structural features, and scale invariance. The same feature point will appear repeatedly in different sub-images, and feature points are used for comparison. In terms of comparison results, it can resist the influence of image noise and interference on different sub-images to a certain extent, and can still maintain the recognizability of feature points under undesirable conditions, thereby improving the robustness of the visual positioning system.

[0049] Specifically, the feature point has a significant contrast, which means that the feature point is usually located in an area with a significant light-dark contrast in the image, and the brightness value of these areas is significantly different from the values ​​of adjacent pixels. For example, at a high-contrast edge or corner, the brightness of the pixel changes sharply, so the high-contrast edge or corner area can be identified as a feature point.

[0050] Feature points reflect local structural features, which means that feature points can characterize the shape and layout of local structures. For example, at the corners, junctions, or intersections of an object, there may be multiple feature points forming a unique geometric shape, which makes these feature points not only have local contrast significance, but also have strong shape information.

[0051] Scale invariance: Regardless of the image’s scaling, rotation, or perspective, these feature points can still maintain relatively consistent performance, so they can still be correctly identified under different conditions.

[0052] Repeatability means that high-quality feature points have good repeatability, that is, the same physical features can still be accurately identified as the same feature points in different images (such as images taken from different perspectives or at different times). This ability is particularly important for object recognition and image matching.

[0053] In particular, the sub-image is a plan view obtained by splitting each pre-stored panoramic image based on a preset number N1, that is, each pre-stored panoramic image is evenly split into N1 sub-images with the same horizontal field of view, so the number of the pre-set sub-images is N1×the number of all pre-stored panoramic images.

[0054] The sub-image is a plan view obtained through a specific splitting process. This process is based on a preset number N1 to split each pre-stored panoramic image. The panorama, as an image that can show a 360-degree environmental view, contains rich visual information. However, because the panorama cannot be compared with a flat image, it is necessary to split the panorama, that is, to split each panorama into N1 sub-images with the same horizontal field of view on average. This means that if there is a panorama, it will be cut into N1 parts, each of which maintains the same horizontal field of view, so as to ensure that the sub-images after splitting are consistent and comparable in visual content. The splitting process is performed on each pre-stored panorama, so the number of sub-images finally obtained will be N1 multiplied by the number of all pre-stored panoramas. In this way, we obtain a set of several sub-images, which is not only huge in number, but also each sub-image retains a part of the important information in the original panorama, providing a basis for comparison for subsequent image matching.

[0055] In particular, the horizontal field of view of the sub-image can be obtained by simple mathematical calculation, specifically: the horizontal field of view of each panoramic image is evenly divided into N1 sub-images, so the horizontal field of view of each sub-image will be equal to 360° divided by N1, that is, the field of view covered by each sub-image is a uniform division of the complete viewing angle of the panoramic image. If the 360° field of view is regarded as a complete annular field of view, after average division, each sub-image will represent a part of this annular field of view, and its angle is 360° / N1. This division method ensures that each sub-image is relatively uniform in visual information, so that each sub-image can provide a consistent viewing angle during subsequent processing. For example, if N1 is preset to 4, the horizontal field of view of each sub-image is 90°, so that the detailed information in each panoramic image can be captured more comprehensively.

[0056] In particular, the video field of view of the sub-image should be greater than or equal to 360° / N1, that is, the contents of two adjacent sub-images evenly divided from the same panoramic image may overlap.

[0057] It is worth noting that the process of extracting the second global descriptor also uses the above-mentioned image retrieval algorithm to extract the global descriptor for the image, that is, a pre-trained convolutional neural network is used to extract the local features of each sub-image, and then the VLAD layer is used to aggregate the local features into the second global descriptor.

[0058] In particular, the three-dimensional visual map constructed by the multiple panoramas is intended to provide a comprehensive three-dimensional representation of a specific environment or scene. This process first requires the effective integration of the acquired panoramas to form a unified three-dimensional visual model. In this model, the pose information of each panorama is stored, and the pose information includes the position and orientation of each panorama in three-dimensional space, which is very important for subsequent view reconstruction and spatial understanding. In addition, this three-dimensional visual map also contains feature points in the panorama, and the representation information of the feature points contains the spatial 3D information of the feature points. These feature points are extracted from the panorama by computer vision technology, and usually represent key structures and significant features in the scene, such as corners, edges or surface textures. These feature points not only help to describe the environment visually, but can also be used for subsequent tasks such as point cloud generation, object recognition and path planning.

[0059] Step S4: Determine, according to the multiple first sub-images, M panoramic images associated with the multiple first sub-images.

[0060] In particular, because the sub-images are obtained by splitting the panorama, the M panorama images corresponding to the multiple first sub-images can be reversely determined based on the multiple first sub-images. Specifically, since the generation process of each first sub-image is obtained by splitting based on the panorama, there is a direct correspondence, so the information extracted from these sub-images can be used to determine its original panorama. By analyzing the multiple first sub-images, the perspectives and features represented by each first sub-image can be identified. Then this information is compared with the pre-stored panorama to find a matching panorama. The field of view and visual features covered by each first sub-image can be used as an important basis for judging the panorama from which it comes. After determining the M panorama images associated with the multiple first sub-images, the splitting operation can be performed to ensure that these panoramas can regenerate the same multiple sub-images as before. In this process, we are actually verifying and reconstructing the relationship between these sub-images and the panorama. Through this forward and reverse splitting and matching process, the structure and layout of the panorama can be better analyzed. That is, because the sub-images are obtained by splitting the panorama, M panorama images can be determined based on the multiple first sub-images, and the multiple first sub-images are obtained by splitting the M panorama images. In addition, because a panorama is split into N1 sub-images, the value of M is greater than or equal to the number of the multiple first sub-images.

[0061] Step S5: split the M panoramic images into N2 second sub-images with the same horizontal field of view as the image to be located according to the camera parameters, wherein the second sub-image is a plane image, and the representation information of the feature point of the second sub-image includes the spatial 3D information of the feature point in the three-dimensional visual map;

[0062] In particular, according to the explanation of the relationship between the horizontal field of view angle of the sub-image and the preset number N1, it can be obtained that the horizontal field of view angle of the image to be positioned and N2 also have a similar relationship, that is, the horizontal field of view angle of the image to be positioned = 360° / N2. And the horizontal field of view angle can be calculated based on the camera parameters including the resolution and the camera intrinsic parameter matrix, and the specific formula is:

[0063]

[0064] Where W is the width of the image, that is, the horizontal resolution of the camera, and fx is the value of the focal length parameter f calculated on the image width as a function of the horizontal field of view, that is, the focal length in the camera's intrinsic parameter matrix. The logic of this formula is to calculate the field of view through the triangular relationship from the camera's optical center to the edge of the imaging area.

[0065] Step S6, performing image matching on the N2 second sub-images and the image to be positioned to obtain a common feature point set, wherein the feature point set includes a plurality of common feature points, and the common feature points are feature points existing in both the N2 second sub-images and the image to be positioned;

[0066] In particular, the image matching process is implemented based on the computer vision technology of the deep learning model. There are two different ways to implement the image matching process based on the computer vision technology of the deep learning model. One is to use the local feature matching based on the detector (such as SuperPoint) to extract features from all images, and then perform feature matching through the matching algorithm of superglue. Specifically: first use feature detectors such as SuperPoint to extract local feature points from each image. These feature points are usually located in the image with high texture information or saliency, and they will become the basis for subsequent matching. And the feature detectors such as SuperPoint can not only detect key points but also generate corresponding feature descriptors through convolutional neural networks. After extracting the features of all images, the SuperGlue matching algorithm is applied for feature matching. SuperGlue uses the learned matching strategy to match the extracted feature points with each other. SuperGlue will consider the similarity between each feature descriptor and also introduce contextual information, thereby significantly improving the accuracy of matching. The SuperGlue algorithm enhances the matching efficiency through graph neural networks and can handle more complex matching relationships.

[0067] Another method is to use detector-free feature matching (such as LoFTR) to directly output feature matching relationships. Specifically: Unlike the detector-based local feature matching, the detector-free feature matching does not rely on a specific key point detection algorithm in the feature extraction process, but uses an end-to-end deep learning framework to directly analyze the entire structure of the image. It generates a global feature representation by learning the similarities between image pairs. The detector-free feature matching can directly output feature matching relationships, eliminating the step of extracting feature points separately in traditional methods. The matching process of the detector-free feature matching is more efficient because it can match across the entire image instead of relying on local features. This mechanism is particularly suitable for matching large-scale or dynamic scenes, and can effectively handle challenges such as illumination changes, perspective changes, and overlapping areas.

[0068] It is worth noting that the computer vision technology based on the deep learning model used in this embodiment of the present application is to use detector-based local feature matching. In step S6, the process in which the common feature points are feature points existing in both the N2 second sub-images and the image to be positioned includes:

[0069] Step S61, performing feature extraction on the N2 second sub-images and the image to be positioned, and obtaining feature points of the N2 second sub-images and feature points of the image to be positioned respectively;

[0070] Step S62: Based on the feature points of the N2 second sub-images and the feature points of the image to be located, a detector-based local feature matching algorithm is used to perform feature matching to obtain the common feature point set.

[0071] It is worth noting that the common feature points included in the common feature point set are feature points possessed by both the second sub-image and the image to be positioned, and the representation information of the feature points of the second sub-image includes the spatial 3D information of the feature points in the three-dimensional visual map. Therefore, the spatial 3D information of the feature points of the image to be positioned in the three-dimensional visual map can be obtained based on the two-dimensional feature points possessed by both the second sub-image and the image to be positioned.

[0072] The common feature points included in the common feature point set refer to feature points that exist in both the second sub-image and the image to be positioned. The common feature points are of great significance. Through the common feature points, the connection between the second sub-image and the image to be positioned can be established, and a basis for subsequent positioning is provided.

[0073] In the characterization information of the common feature points, the feature points of the second sub-image not only include the two-dimensional coordinates of the feature points on the image plane, but also include the spatial three-dimensional information of the feature points in the three-dimensional visual map. These three-dimensional information usually include the position of the feature points in three-dimensional space (X, Y, Z coordinates) and other depth information. The acquisition of the spatial three-dimensional information generally comes from the construction process of the three-dimensional visual map, which is usually achieved using methods such as multi-view image registration and structured light or stereo vision technology. By analyzing the relationship between the common feature points between the second sub-image and the image to be located, we can use the three-dimensional spatial information of the known feature points to derive the three-dimensional position of the feature points corresponding to the known feature points in the image to be located.

[0074] Step S7: solving PnP according to the common feature point set to obtain the final pose of the image to be positioned.

[0075] It is worth noting that the final posture of the image to be positioned refers to the position and direction of the image to be positioned in the three-dimensional space.

[0076] It is worth noting that the PnP (Perspective-n-Point) refers to obtaining the final pose of the image to be positioned (i.e., the position and direction of the camera when shooting the image to be positioned) based on known three-dimensional points and their projections in the two-dimensional image.

[0077] In particular, the step S7 comprises the following steps:

[0078] Initialize RANSAC: set the parameters of RANSAC, including the maximum number of iterations (how to randomly select sample points multiple times) and the allowed error threshold (the standard for judging inliers and outliers);

[0079] Randomly select samples: In each iteration, randomly select the minimum number of 3D points and corresponding 2D points. For the PnP problem, usually 4 points need to be selected because these 4 points can uniquely determine the camera's position (rotation and translation);

[0080] Solve PnP: Use the selected 4 points and call the PnP algorithm (such as EPnP, APnP, etc.) to calculate the camera's external parameters R (rotation matrix, used to describe the rotation relationship between the camera coordinate system and the world coordinate system or the three-dimensional space coordinate system) and t (translation vector, used to represent the translation of the camera in three-dimensional space relative to the world coordinate system);

[0081] Calculate the inliers: For most of the points that are removed, calculate their reprojection error. Reproject by converting the 3D point to the 2D image plane, and then calculate the difference between the real 2D point and the reprojected point. Then, based on the preset error threshold (tolerance), determine whether the point is an inlier or an outlier. If the error is less than the threshold, it is considered an inlier.

[0082] Update model: In each iteration, record the number of inliers. If the number of inliers in the current model exceeds the historical best number of inliers, update the best model parameters, that is, update the current R and t;

[0083] Iterate until convergence: Repeat the random sample selection, PnP calculation of the internal points and model update until the maximum number of iterations is reached or the number of internal points in a certain iteration meets a given condition;

[0084] Output: Output the set of 3D points and 2D points corresponding to the internal points, as well as the corresponding camera pose.

[0085] Beneficial effects: The present application provides a visual positioning method, which splits each panoramic image in a pre-stored three-dimensional visual map into a preset number of multiple sub-images, and the sub-images are plane images, which lays a comparable foundation for the subsequent process of searching for panoramic images that have similarities or associations with the image to be positioned (which is a plane image), and solves the problem that the plane image cannot be compared with the panoramic image in the prior art. Afterwards, the first global descriptor of the image to be positioned is compared with the second global descriptors of the multiple sub-images, and based on the similarity between the first global descriptor and the second global descriptor, multiple first sub-images with a certain degree of similarity to the image to be positioned are screened out. This screening step excludes sub-images that are greatly different from the image to be positioned, reducing the computational workload in the subsequent further refined comparison process. Afterwards, the M panoramas associated with the selected multiple first sub-images are split to obtain multiple second sub-images, which are plane images, and the splitting basis is based on the camera parameters and the horizontal field of view of the image to be positioned, which further lays a comparative foundation for the process of screening out the common feature points of the image to be positioned from the multiple second sub-images, and improves the accuracy of the comparison. Afterwards, the N2 second sub-images are image matched with the image to be positioned to obtain a common feature point set including multiple common feature points, which are feature points existing in both the N2 second sub-images and the image to be positioned; PnP is solved according to the feature point set to obtain the final posture of the image to be positioned.

[0086] Another embodiment of the present application also provides a visual positioning method, and the difference between this embodiment of the present application and the above-mentioned embodiment of the present application is that the specific implementation method in step S6 is different. In this embodiment, the process of matching the N2 second sub-images with the image to be positioned to obtain a common feature point set uses a computer vision technology based on a deep learning model, which directly outputs feature matching relationships using feature matching without detectors (such as LoFTR).

[0087] Specifically: Unlike the detector-based local feature matching, the feature matching method without a detector does not rely on a specific key point detection algorithm in the feature extraction process, but uses an end-to-end deep learning framework to directly analyze the entire structure of the image. The deep learning framework generates a global feature representation by learning the similarities between image pairs. The feature matching without a detector can directly output feature matching relationships, eliminating the step of extracting feature points separately in traditional methods. The matching process of the feature matching without a detector can match across the entire image instead of relying on local features, so it is more efficient. Feature matching without a detector is particularly suitable for matching large-scale scenes or dynamic scenes, and can also be applied to matching scenes where there are changes in lighting, changes in perspective, and overlapping areas between multiple sub-images split out of a three-dimensional map.

[0088] like Figure 2 As shown, other embodiments of the present application also disclose a visual positioning device, which includes: an input acquisition module for acquiring an image to be positioned and camera parameters. The module can obtain the image to be positioned by photographing the camera of the electronic device, and obtain the camera parameters from the memory of the electronic device.

[0089] The global descriptor extraction module is used to extract the first global descriptor of the image to be located. Specifically, the global descriptor can be extracted by referring to the image retrieval algorithm mentioned in the above method embodiment.

[0090] The storage module is used to store the three-dimensional visual map, multiple panoramas, multiple sub-images split into each panorama, the second global descriptor and other information.

[0091] The retrieval module is used to compare the first global descriptor with the pre-stored second global descriptor, and retrieve multiple first sub-images whose similarity with the image to be located meets a preset condition from the pre-set multiple first sub-images. Specifically, reference can be made to the part of screening the sub-images mentioned in the above method embodiment in which the similarity between the second global descriptors of the multiple first sub-images and the first global descriptor of the image to be located meets the preset condition.

[0092] The determination module is used to determine M panoramic images associated with the multiple first sub-images based on the multiple first sub-images. For details, please refer to the part of the M panoramic images associated with the multiple first sub-images mentioned in the above method embodiment.

[0093] The splitting module is used to split the M panoramic images into N2 second sub-images having the same horizontal field of view angle as the image to be located according to the camera parameters.

[0094] The matching module is used to perform image matching on the N2 second sub-images and the image to be located to obtain a set of common feature points.

[0095] The pose determination module is used to solve PnP according to the feature point set to obtain the final pose of the image to be positioned.

[0096] It is worth noting that the input acquisition module, the global descriptor extraction module, the storage module, the retrieval module, the determination module, the splitting module, the matching module and the posture determination module cooperate together to realize the visual positioning in the above embodiment.

[0097] like Figure 3 As shown, in another embodiment of the present application, a device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned visual positioning method when executing the computer program. The device can be an electronic device with a camera function, such as a mobile phone, PAD, laptop computer, camera, AR glasses, VR glasses, etc., or an electronic device without a camera function, such as a desktop computer, server, etc.

[0098] In particular, the processor is one or more central processing units (CPUs) or graphics processing units (GPUs) for performing computing tasks. These processors can be single-core or multi-core and can be integrated on one or more chips. In high-performance computing scenarios, GPUs are particularly suitable for parallel computing tasks that process large amounts of image data.

[0099] In particular, the types of memory include volatile memory (such as RAM) and non-volatile memory (such as hard disk drives, solid-state drives, flash memory, etc.). The memory is used to store program code, intermediate calculation results, and final output data. Among them, volatile memory is mainly used to temporarily store data being processed, while non-volatile memory is used for persistent storage.

[0100] In particular, the computer program is stored in a memory and is configured to be executed by one or more processors. The computer program includes instructions required to implement the above-mentioned image processing method, including but not limited to functional modules such as image acquisition, preprocessing, feature extraction, model application, result generation, post-processing and result output.

[0101] Another embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned visual positioning method are implemented.

[0102] Specifically, the storage medium may include, but is not limited to, a hard disk, an optical disk, a USB flash drive, a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a magnetic tape, a floppy disk, and an optical storage device. These storage media may be implemented by any appropriate method or technology, as long as they can store program instructions and these instructions can be run on the device. The program includes a series of instructions, and when these instructions are executed, the device can complete the various steps of the image processing method disclosed in the above embodiment.

[0103] It is worth noting that the program on the storage medium can be an independent software package or a module integrated into a larger system. It can be deployed on various types of computing devices, including but not limited to personal computers (PCs), servers, mobile devices (such as smartphones and tablets), dedicated hardware devices, etc.

[0104] In particular, the computer-readable storage medium can be widely used in multiple fields, such as digital entertainment, virtual reality (VR) and augmented reality (AR).

[0105] Virtual Reality (VR) is an advanced computer-generated technology that creates a simulated environment that allows users to immerse themselves in a completely virtual three-dimensional space through specific devices and interact with objects and scenes in the virtual world. Users are immersed in a completely virtual three-dimensional space through specific devices (such as head-mounted displays, handheld controllers, etc.) and can interact with objects and scenes in the virtual world. VR systems usually include head-mounted displays (HMDs) that can display stereoscopic images and track the user's head movements to achieve 3D effects. They also use a variety of sensors, such as accelerometers, gyroscopes, and external cameras, to track the user's body movements in real time to enhance immersion. VR systems also include handheld controllers, gloves, and other input devices that allow users to interact in a virtual environment, such as grabbing, pushing, and pulling objects. VR also enhances the sense of immersion in a VR environment by providing users with spatial stereo audio, making the sound localizable in space. Virtual reality (VR) is widely used in many fields, including games, training and simulation, virtual tourism, and psychotherapy. Specifically: In the gaming industry, VR technology provides players with an unprecedented immersive experience. Players no longer just watch the game screen, but really "enter" the game and interact directly with the game characters and environment. In addition, VR technology has also been applied to entertainment venues such as film production and theme parks, bringing new audio-visual enjoyment to the audience; in the field of education and training, VR technology can not only be used to teach complex concepts such as medical anatomy and chemical reactions, but also to simulate experimental operations, allowing students to practice in a safe environment, and VR technology can also simulate real work scenes to help employees master the necessary skills; in the field of virtual tourism, with the development of VR technology, people can visit places of interest around the world without leaving home. Virtual tourism not only saves time and money, but also allows people with physical disabilities to enjoy the fun of traveling. In addition, VR technology can also be used for the digital reconstruction of historical sites, allowing people to travel through time and space and feel the charm of ancient civilizations; in the field of psychotherapy, by creating specific scenarios, VR can help patients with psychological diseases such as phobias and anxiety disorders gradually overcome their fears and improve their mental health. In addition, VR technology is also used in rehabilitation therapy to help injured or post-operative patients with recovery training.

[0106] Augmented Reality (AR) is the process of superimposing virtual information (such as images, audio, and other data) onto the real world, thereby enhancing the user's perception of the real environment. Unlike VR, AR does not create a completely virtual environment, but rather combines computer-generated content with the real world through technical means. AR can display augmented content through devices such as smartphones, tablets, and AR glasses. Their cameras are responsible for capturing the real environment and using methods such as image recognition and SLAM (simultaneous localization and mapping) to achieve real-time analysis of the user's environment in order to correctly superimpose virtual information. The AR system also generates corresponding virtual content based on the user's real-time data, which can interact with objects in the real environment. Augmented reality (AR) is widely used in games and entertainment, education, retail, and maintenance manufacturing. Specifically: In the field of games and entertainment, AR technology brings players an unprecedented immersive experience. Through AR games, users can discover virtual creatures or objects in their living environment, interact with them, and even cooperate or compete with other players. In some games, it can support players to capture elves or occupy territories in the real world, greatly enriching the gameplay of traditional games. In addition, AR technology is also widely used in entertainment activities such as concerts, exhibitions, theme parks, etc., providing audiences with richer visual and auditory enjoyment; in the field of education, AR technology has brought innovation to teaching methods. By superimposing virtual models and information on real scenes, students can more intuitively understand abstract concepts such as molecular structure and celestial body trajectory. AR technology can also be used to create virtual laboratories, allowing students to conduct experiments in a safe environment, reducing experimental risks. In addition, AR technology is also widely used in language learning, historical and cultural teaching, etc., which can help students better absorb knowledge through interactive methods; in the retail industry, AR technology provides consumers with a new shopping experience. Through AR applications, customers can preview the effect of placing goods in their homes before purchasing. For example, some furniture manufacturers have launched applications that allow users to see how furniture will look in their homes before purchasing furniture. In addition, AR technology can also be used in scenes such as trying on clothes and makeup, making online shopping more intuitive and convenient. Retailers can also use AR technology to carry out interactive advertising activities to attract customers to shop; in the industrial field, especially in maintenance and manufacturing, the application of AR technology has demonstrated its powerful functions. Factory technicians can use AR glasses to obtain equipment operating instructions, fault diagnosis information, and maintenance steps, thereby improving work efficiency and accuracy. For example, some aircraft manufacturers have used AR technology to assist in aircraft assembly, reducing errors and speeding up production progress. In remote collaboration scenarios, experts can guide on-site workers to solve problems in real time, greatly shortening troubleshooting time.

[0107] It should be noted that the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or equipment.

[0108] In the present application, unless otherwise clearly specified and limited, a first feature being “above” or “below” a second feature may mean that the first feature is directly in contact with the second feature, or the first feature and the second feature are indirectly in contact through an intermediate medium. Moreover, a first feature being “above”, “above” or “above” a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being “below”, “below” or “below” a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0109] Unless otherwise defined, all technical and scientific terms used in the specification of this application have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in the specification of this application includes any and all combinations of one or more related listed items.

[0110] The technical features of the above embodiments can be combined without changing the basic principles of the present application. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0111] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the scope of patent protection of the present application shall be subject to the attached claims.

Claims

1. A visual positioning method, characterized in that: The following steps are involved: Get the image to be positioned and the camera parameters; Extracting a first global descriptor of the image to be located; Compare the first global descriptor with a pre-stored second global descriptor, retrieve multiple first sub-images from a plurality of pre-set sub-images, the similarity between the second global descriptors of the multiple first sub-images and the first global descriptor of the image to be located meets a preset condition, the second global descriptor is a global descriptor extracted from each sub-image, the sub-image is a plan view obtained by splitting each pre-stored panoramic image based on a preset number N1, the horizontal field of view of the sub-image = 360° / N1, and the multiple panoramic images are jointly constructed to obtain a three-dimensional visual map, and the three-dimensional visual map stores the pose information of each panoramic image and feature points on the panoramic image containing spatial 3D information; Determining, according to the plurality of first sub-images, M panoramic images associated with the plurality of first sub-images; According to the camera parameters, the M panoramic images are split into N2 second sub-images with the same horizontal field of view as the image to be located, the second sub-image is a plane image, and the representation information of the feature point of the second sub-image includes the spatial 3D information of the feature point in the three-dimensional visual map; Performing image matching on the N2 second sub-images and the image to be positioned to obtain a common feature point set, wherein the feature point set includes a plurality of common feature points, and the common feature points are feature points existing in both the N2 second sub-images and the image to be positioned; Solve PnP according to the common feature point set to obtain the final pose of the image to be positioned.

2. The visual positioning method according to claim 1, characterized in that: The camera parameters include resolution and camera intrinsic parameter matrix.

3. The visual positioning method according to claim 1, characterized in that: The process of extracting the first global descriptor and the process of extracting the second global descriptor both use a retrieval algorithm to extract the global descriptor.

4. The visual positioning method according to claim 3, characterized in that: The process of extracting global descriptors using the image retrieval algorithm is as follows: extracting local features of the image through a pre-trained convolutional neural network, and then aggregating the local features into global descriptors using a VLAD layer.

5. The visual positioning method according to claim 1, characterized in that: The horizontal field of view of the image to be positioned = 360° / N2.

6. The visual positioning method according to claim 1, characterized in that: The process of performing image matching on the N2 second sub-images and the image to be positioned to obtain a common feature point set, wherein the feature point set includes a plurality of common feature points, and the common feature points are feature points existing in both the N2 second sub-images and the image to be positioned is specifically as follows: Performing feature extraction on the N2 second sub-images and the image to be positioned, and obtaining feature points of the N2 second sub-images and feature points of the image to be positioned respectively; Based on the feature points of the N2 second sub-images and the feature points of the image to be located, a detector-based local feature matching algorithm is used to perform feature matching to obtain the common feature point set.

7. The visual positioning method according to claim 1, characterized in that: The process of performing image matching on the N2 second sub-images and the image to be positioned to obtain a common feature point set, wherein the feature point set includes a plurality of common feature points, and the common feature points are feature points existing in both the N2 second sub-images and the image to be positioned is specifically as follows: Based on the N2 second sub-images and the image to be located, a detector-free feature matching algorithm is used to perform feature matching to obtain the common feature point set.

8. A device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the visual positioning method according to any one of claims 1 to 7 are implemented.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the visual positioning method according to any one of claims 1 to 7 are implemented.

Images