A method and device for positioning scene images

Abstract

The present invention discloses a method and device for positioning scene images, which matches the global features of the image to be positioned with the global feature library of the scene image set, and obtains the category of the scene image that has been inspected according to the similarity of the location information and global features of the scene image in the scene image set; then, a sparse model of the category of the scene image that has been inspected is reconstructed according to the key points of the first image, and the sparse model is matched with the key points of the second image of the image to be positioned to obtain the position and posture of the image to be positioned. According to the embodiment of the present invention, the candidate scene images are clustered in Euclidean space according to the location information, and the category of the scene image that has been inspected is obtained based on the similarity of the global features of the candidate scene images, and then the key points of the second image are matched, so as to reduce the influence of the erroneous image on the final positioning result and improve the efficiency and accuracy of positioning.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision positioning, and in particular to a scene image positioning method and device. Background Art

[0002] Visual localization is a core problem in the field of computer vision. This technology is widely used in urban scenes, such as self-driving cars, autonomous robots, augmented reality and virtual reality systems. Images taken at different times may have large differences in lighting and appearance, which brings great challenges to visual localization.

[0003] Existing visual positioning methods have at least the following problems: the accuracy of coarse retrieval results based on global image features is low, and too many erroneous retrieval results are involved in fine positioning, which will have an adverse effect on the robustness and accuracy of the final positioning. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and device for positioning scene images, which takes into account the Euclidean space distance and the global feature distance, verifies and optimizes the results in the global feature position recognition stage and the image key point precise positioning stage, and can effectively improve the efficiency and accuracy of visual positioning in large scenes.

[0005] To solve the above technical problems, an embodiment of the present invention provides a method for positioning a scene image, comprising:

[0006] Acquire an image to be located and a scene image set; the scene image set includes an existing scene image and its global feature library and a first image key point;

[0007] Extracting the global features of the image to be located, and selecting a number of scene images from the scene image set as candidate scene images according to the global feature library;

[0008] According to the position information of the candidate scene images, performing Euclidean space clustering on the candidate scene images to divide the candidate scene images into at least one valid image category;

[0009] Calculating the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity;

[0010] Extracting a second image key point of the image to be located;

[0011] A sparse model of the image category of the inspected scene is reconstructed according to the first image key points, and the sparse model is matched with the second image key points of the image to be located to obtain the position and posture of the image to be located.

[0012] As an improvement of the above scheme, when the number of the valid image categories is greater than one, calculating the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, includes:

[0013] Calculating the average score of the global feature similarity between each valid image category and the image to be located, and obtaining the confidence ranking of the valid image category;

[0014] Perform mean value verification, maximum value verification and range verification on each valid image category, and use the verified valid image category as the image category of the scene that has passed the inspection;

[0015] According to the confidence ranking, the categories of the passed-inspection scene images are sorted to obtain the categories of the passed-inspection scene images and their serial numbers.

[0016] As an improvement of the above scheme, when the number of the valid image categories is one, the calculating of the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, include:

[0017] The valid image category is subjected to mean value verification, maximum value verification and range verification. When the valid image category passes the verification, it is regarded as a passed-inspection scene image category and its sequence number is set to be the first.

[0018] As an improvement of the above solution, the extracting of the global features of the image to be located and selecting a number of scene images from the scene image set according to the global feature library as candidate scene images include:

[0019] Extracting global features of the image to be located;

[0020] Calculating a similarity score between the global feature and each global feature in the global feature library;

[0021] Selecting a number of scene images with the highest similarity scores from the scene image set as candidate scene images. As an improvement to the above solution, performing Euclidean space clustering on the candidate scene images according to the location information of the candidate scene images to divide the candidate scene images into at least one valid image category includes:

[0022] According to the position information of the candidate scene images, performing Euclidean space clustering on the candidate scene images to divide the candidate scene images into several image categories;

[0023] An image category whose number of images is greater than a preset number threshold is taken as a valid image category to obtain at least one valid image category.

[0024] As an improvement of the above solution, reconstructing a sparse model of the image category of the inspected scene according to the first image key points, matching the sparse model with the second image key points of the image to be located, and obtaining the pose of the image to be located includes:

[0025] According to the first image key points, a sparse model of the inspected scene image category is constructed to obtain a matching relationship between the second image key points of the image to be located and the key points of the scene image in the sparse model;

[0026] The PnP algorithm is used to solve the key point matching relationship to obtain the position and posture of the image to be located.

[0027] As an improvement of the above solution, the sparse model of the inspected scene image category is constructed according to the first image key points, and the matching relationship between the second image key points of the image to be located and the key points of the scene image in the sparse model is obtained, including:

[0028] The first inspected scene image category is used as the current inspected scene image category;

[0029] When the current inspected scene image category is updated, performing local scene reconstruction on the current inspected scene image category according to the first image key points to obtain a sparse model of the current inspected scene image category;

[0030] Performing 2D-2D matching between the image to be located and the sparse model, and geometrically verifying the matching result by using the RANSAC algorithm to obtain the number of matching points with each scene image in the sparse model;

[0031] If the number of matching points is less than a preset threshold, the current inspected scene image category is selected and updated according to the sequence number of the inspected scene image category; otherwise, the current inspected scene image category is used as the inspected scene image category to be matched;

[0032] According to the correspondence between the second image key point of the inspected scene image category to be matched and the sparse model, a 2D-3D key point matching relationship between the image to be located and the scene sparse point cloud is retrieved.

[0033] The embodiment of the present invention further provides a scene image positioning device, comprising:

[0034] An image acquisition module, used to acquire an image to be located and a scene image set; the scene image set includes an existing scene image and its global feature library and a first image key point;

[0035] A candidate scene image selection module is used to extract the global features of the image to be located, and select a number of scene images from the scene image set as candidate scene images according to the global feature library;

[0036] An effective image category classification module, used to perform Euclidean space clustering on the candidate scene images according to the position information of the candidate scene images, and classify the candidate scene images into at least one effective image category;

[0037] A passed-inspection scene image category selection module is used to calculate the global feature similarity between the valid image category and the image to be located, and obtain the passed-inspection scene image category and its sequence number according to the global feature similarity;

[0038] A second image key point extraction module, used to extract the second image key point of the image to be located;

[0039] The image positioning module is used to reconstruct a sparse model of the image category of the inspected scene according to the first image key points, and match the sparse model with the second image key points of the image to be positioned to obtain the position and posture of the image to be positioned.

[0040] As an improvement of the above scheme, when the number of the valid image categories is greater than one, the calculating of the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, include:

[0041] Calculating the average score of the global feature similarity between each valid image category and the image to be located, and obtaining the confidence ranking of the valid image category;

[0042] Perform mean value verification, maximum value verification and range verification on each valid image category until a valid image category that passes the verification is obtained, and the valid image category that passes the verification is used as the image category of the scene that has passed the inspection;

[0043] According to the confidence ranking, the categories of the passed-inspection scene images are sorted to obtain the categories of the passed-inspection scene images and their serial numbers.

[0044] As an improvement of the above scheme, when the number of the valid image categories is one, the calculating of the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, include:

[0045] The valid image category is subjected to mean value verification, maximum value verification and range verification. When the valid image category passes the verification, it is regarded as a passed-inspection scene image category and its sequence number is set to be the first.

[0046] Compared with the prior art, the present invention discloses a scene image positioning method and device, which obtains an image to be positioned and a scene image set; the scene image set includes an existing scene image and its global feature library and a first image key point; extracts the global feature of the image to be positioned, and selects a number of scene images from the scene image set according to the global feature library as candidate scene images; performs Euclidean space clustering on the candidate scene images according to their position information, and divides the candidate scene images into at least one valid image category; calculates the global feature similarity between the valid image category and the image to be positioned, and obtains the inspected scene image category and its serial number according to the global feature similarity; extracts the second image key point of the image to be positioned; reconstructs a sparse model of the inspected scene image category according to the first image key point, matches the sparse model with the second image key point of the image to be positioned, and obtains the posture of the image to be positioned. By adopting the embodiment of the present invention, the candidate scene images are clustered in Euclidean space according to the position information, and the inspected scene image category is obtained based on the global feature similarity of the candidate scene images, and then the image key points are matched, thereby reducing the impact of erroneous images on the final positioning result and improving the efficiency and accuracy of positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the steps of a method for positioning a scene image provided by an embodiment of the present invention;

[0048] Figure 2 It is a structural schematic diagram of a scene image positioning device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In the description of the specification and claims, it is to be understood that the terms first, second, etc. in the specification and claims are only used for the purpose of describing the same technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated, nor necessarily describing the order or time sequence. The terms are interchangeable where appropriate. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0051] See also Figure 1, is a schematic diagram of the steps of a scene image positioning method provided by an embodiment of the present invention. In this embodiment, the scene image positioning method is specifically performed through steps S1 to S6:

[0052] S1. Acquire an image to be located and a scene image set; the scene image set includes an existing scene image and its global feature library and a first image key point.

[0053] S2. Extracting global features of the image to be located, and selecting a number of scene images from the scene image set according to the global feature library as candidate scene images.

[0054] As an example, in an embodiment of the present invention, the global features of the image to be located are extracted by a graph convolutional neural network. It can be understood that the global features are global descriptions of each image, which can effectively distinguish different images.

[0055] S3. Perform Euclidean space clustering on the candidate scene images according to the position information of the candidate scene images, and divide the candidate scene images into at least one valid image category.

[0056] In the prior art, the global features of the image to be located are directly compared with the global feature library, and the scene images with consistent global descriptions are selected as the rough search results. However, the accuracy of the rough search results based on the global features of the image is low. It is understandable that the spatial distance between images belonging to the same scene is generally closer when they are taken. The embodiment of the present invention utilizes the location information of the candidate scene images and considers the Euclidean space distance of the candidate scene images, which further improves the accuracy of positioning.

[0057] S4. Calculate the global feature similarity between the valid image category and the image to be located, and obtain the inspected scene image category and its sequence number according to the global feature similarity.

[0058] S5. Extract the second image key point of the image to be located.

[0059] As an example, in an embodiment of the present invention, a second image key point of the image to be located is extracted through a convolutional network, and the second image key point includes feature point position information and its corresponding descriptor information.

[0060] S6. Reconstruct a sparse model of the image category of the inspected scene based on the first image key points, and match the sparse model with the second image key points of the image to be located to obtain the position and posture of the image to be located.

[0061] It should be noted that in the field of visual positioning, the computer obtains features or descriptors consistent with the image to be positioned by querying the scene image set, and selects scene images with the same description relationship from the scene image set, and uses the positioning of the scene image to represent the posture of the image to be positioned. The scene image sets corresponding to different large scenes are also different. For example, the scene image set in the visual positioning system of an autonomous driving car pays more attention to the road and vehicle conditions, and when extracting the features of the image to be positioned, more attention is also paid to the road and vehicle conditions. In other words, the scene described by the scene image set should be the same as the large scene of the image to be positioned, and the specific field of the scene does not affect the beneficial effects achieved by the present invention.

[0062] It can be understood that global feature extraction is performed on each scene image in the scene image set to obtain a global feature library; image key point extraction is performed on each scene image in the scene image set to obtain a first image key point. The global feature extraction method and the first image key point extraction method of the scene image set are consistent with the extraction method of the image to be positioned. After the image key point extraction is performed in advance to obtain the first image key point, these pre-processed features can be directly used when the inspected scene image is partially reconstructed later, without repeatedly performing feature extraction, thereby improving the efficiency of image positioning.

[0063] As a preferred implementation, when the number of the valid image categories is greater than one, calculating the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, includes:

[0064] Calculating the average score of the global feature similarity between each valid image category and the image to be located, and obtaining the confidence ranking of the valid image category;

[0065] Perform mean value verification, maximum value verification and range verification on each valid image category, and use the verified valid image category as the image category of the scene that has passed the inspection;

[0066] According to the confidence ranking, the categories of the passed-inspection scene images are sorted to obtain the categories of the passed-inspection scene images and their serial numbers.

[0067] For example, the valid image categories are M valid (M valid >1), calculate each valid image category C i (i=1,...,M valid ) and the average score of the global feature similarity of the image to be located Among them, N i Valid image category C iThe number of scene images included, s i is the global feature similarity between the scene image and the image to be located, and the average score is used as the confidence score to obtain the confidence ranking of the valid image category. Each valid image category is subjected to mean verification, maximum value verification and range verification, and the valid image category that passes the verification is used as the inspected scene image category. Then, according to the confidence ranking, starting from the inspected scene image category with the highest confidence score, they are numbered one by one. In the embodiment of the present invention, the inspected scene image category with the highest confidence score is numbered first, the inspected scene image category with the second highest confidence score is numbered second, and so on.

[0068] As an example, the threshold for mean verification is T mean , the threshold value for maximum value verification is T max , the threshold value for range verification is T range , when the mean is satisfied Greater than T mean , the maximum value Greater than T max , the value range Less than T range When , the valid image category is considered to have passed the verification; where N i is the number of scene images contained in the valid image category, s i It is the global feature similarity between the scene image and the image to be located. It should be noted that the threshold of the mean verification, the threshold of the maximum verification and the threshold of the range verification are all pre-set thresholds, and their values ​​are related to different large scenes and scene image sets. The specific values ​​do not affect the beneficial effects achieved by the present invention.

[0069] In the above scheme, the higher the confidence score of the valid image category, the greater the possibility that there is a common view area with the image to be located. By sorting the confidence scores between categories and verifying the confidence scores within categories for multiple valid image categories, it is possible to accurately and efficiently query the inspected scene image category that is closest to the image to be located in terms of similarity and location information, thereby reducing the impact of erroneous images on the final positioning results.

[0070] As a preferred implementation, when the number of the valid image categories is one, calculating the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, includes:

[0071] The valid image category is subjected to mean value verification, maximum value verification and range verification. When the valid image category passes the verification, it is regarded as a passed-inspection scene image category and its sequence number is set to be the first.

[0072] As an example, the threshold for mean verification is T mean , the threshold value for maximum value verification is T max , the threshold value for range verification is T range , when the mean is satisfied Greater than T mean , the maximum value Greater than T max , the value range Less than T range When , the valid image category is considered to have passed the verification; where N i is the number of scene images contained in the valid image category, s i It is the global feature similarity between the scene image and the image to be located. It should be noted that the threshold of the mean verification, the threshold of the maximum verification and the threshold of the range verification are all pre-set thresholds, and their values ​​are related to different large scenes and scene image sets. The specific values ​​do not affect the beneficial effects achieved by the present invention.

[0073] In the above scheme, by performing intra-class verification on the valid image categories, the inspected scene image category that is closest to the image to be located in terms of similarity and location information can be accurately and efficiently queried, thereby reducing the impact of erroneous images on the final positioning results.

[0074] Preferably, the extracting the global features of the image to be located and selecting a number of scene images from the scene image set as candidate scene images according to the global feature library includes:

[0075] Extracting global features of the image to be located;

[0076] Calculating a similarity score between the global feature and each global feature in the global feature library;

[0077] A number of scene images with the highest similarity scores are selected from the scene image set as candidate scene images.

[0078] Preferably, performing Euclidean space clustering on the candidate scene images according to the position information of the candidate scene images to classify the candidate scene images into at least one valid image category includes:

[0079] According to the position information of the candidate scene images, performing Euclidean space clustering on the candidate scene images to divide the candidate scene images into several image categories;

[0080] An image category whose number of images is greater than a preset number threshold is taken as a valid image category to obtain at least one valid image category.

[0081] In the above scheme, by performing Euclidean space clustering on the candidate scene images without limiting the number of categories, several image categories can be obtained, and the spatial distance between each image category is relatively short; then by limiting the number of images in the image category, effective image categories are obtained, which can perform image matching and positioning more accurately.

[0082] As a preferred implementation manner, reconstructing a sparse model of the image category of the inspected scene according to the first image key points, matching the sparse model with the second image key points of the image to be located, and obtaining the pose of the image to be located includes:

[0083] According to the first image key points, a sparse model of the inspected scene image category is constructed to obtain a matching relationship between the second image key points of the image to be located and the key points of the scene image in the sparse model;

[0084] The PnP algorithm is used to solve the key point matching relationship to obtain the position and posture of the image to be located.

[0085] As an example, in the embodiment of the present invention, the PnP algorithm is used to solve the matching relationship to obtain the positioning and posture determination of the image to be positioned, specifically: i =(x i ,y i ,z i ) represents the i-th (i=1,2,…,m) object point, p i =(u i ,v i ) indicates P i The pixel coordinates projected on the scene image are

[0086] Where f is the variable to be determined, and m is the number of 2D-3D matching points in the matching relationship. When m = 6, the linear equations are solved directly. When m > 6, the least squares solution under the constraint of |F| = 1 is obtained through singular value decomposition (SVD). K represents the camera intrinsic parameter, R and t are the camera extrinsic parameters, which are the posture and position of the image to be located respectively.

[0087] Further, preferably, the step of constructing a sparse model of the inspected scene image category based on the first image key points, and obtaining a matching relationship between the second image key points of the image to be located and the key points of the scene image in the sparse model, includes:

[0088] The first inspected scene image category is used as the current inspected scene image category;

[0089] When the current inspected scene image category is updated, performing local scene reconstruction on the current inspected scene image category according to the first image key points to obtain a sparse model of the current inspected scene image category;

[0090] Performing 2D-2D second image key point matching between the image to be located and the sparse model, and geometrically verifying the matching result through the RANSAC algorithm to obtain the number of matching points with each scene image in the sparse model;

[0091] If the number of matching points is less than a preset threshold, the current inspected scene image category is selected and updated according to the sequence number of the inspected scene image category; otherwise, the current inspected scene image category is used as the inspected scene image category to be matched;

[0092] According to the correspondence between the second image key point of the inspected scene image category to be matched and the sparse model, a 2D-3D feature point matching relationship between the image to be located and the scene sparse point cloud is retrieved.

[0093] As an example, the local scene sparse model is obtained by performing online SfM (Structure from Motion) on the inspected scene image.

[0094] It should be noted that, in the embodiment of the present invention, the categories of scene images that have been inspected are pre-screened from the scene image set, and the number of the scene images that have been inspected is much smaller than the number of images in the scene image set. Therefore, the speed of the online SfM process is also relatively fast. According to the location information of the scene images that have been inspected, a local scene with absolute geographic coordinates and orientation can be constructed. In the prior art, it is necessary to rely on the three-dimensional model of the current scene as a positioning base, but the production of sparse model base maps of large scenes usually requires a lot of manpower and material resources, and in some cases it is even impossible to obtain. The embodiment of the present invention uses online SfM to construct a local scene sparse model of the inspected scene image, which can achieve efficient absolute positioning and posture determination of the query image without a sparse three-dimensional model of the scene, thereby reducing the time cost of positioning and improving positioning efficiency.

[0095] It can be understood that when the number of matching points between the image to be positioned and the sparse model is too small, it indicates that there is a high probability that the image to be positioned and the current inspected scene image category do not belong to the same scene. Therefore, the inspected scene image category with the second highest matching score with the image to be positioned can be selected to reconstruct the local scene, so as to obtain a more accurate positioning result.

[0096] By adopting the scene image positioning method provided in the embodiment of the present invention, the candidate scene images are clustered in Euclidean space according to the position information, and the clustering results are sorted by confidence priority between classes and internally verified within classes based on the global feature similarity of the candidate scene images, and then the key points of the second image are matched, thereby reducing the impact of erroneous images on the final positioning results and improving the efficiency and accuracy of positioning.

[0097] See also Figure 2 , is a schematic diagram of the structure of a scene image positioning device provided by an embodiment of the present invention. The scene image positioning device comprises an image acquisition module 11, a candidate scene image selection module 12, a valid image category classification module 13, a passed scene image category selection module 14, a second image key point extraction module 15 and an image positioning module 16, wherein:

[0098] The image acquisition module 11 is used to acquire the image to be located and the scene image set; the scene image set includes the existing scene image and its global feature library and the first image key point;

[0099] A candidate scene image selection module 12 is used to extract the global features of the image to be located, and select a number of scene images from the scene image set as candidate scene images according to the global feature library;

[0100] The effective image category classification module 13 is used to perform Euclidean space clustering on the candidate scene images according to the position information of the candidate scene images, and classify the candidate scene images into at least one effective image category;

[0101] In the prior art, the global features of the image to be located are directly compared with the global feature library, and the scene images with consistent global descriptions are selected as the rough search results. However, the accuracy of the rough search results based on the global features of the image is low. It is understandable that the spatial distance between images belonging to the same scene is generally closer when they are taken. The embodiment of the present invention utilizes the location information of the candidate scene images and considers the Euclidean space distance of the candidate scene images, which further improves the accuracy of positioning.

[0102] The inspection scene image category selection module 14 is used to calculate the global feature similarity between the effective image category and the image to be located, and obtain the inspection scene image category and its sequence number according to the global feature similarity;

[0103] A second image key point extraction module 15, used to extract the second image key point of the image to be located;

[0104] The image positioning module 16 is used to reconstruct a sparse model of the image category of the inspected scene according to the first image key points, and match the sparse model with the second image key points of the image to be positioned to obtain the position and posture of the image to be positioned.

[0105] It can be understood that global feature extraction is performed on each scene image in the scene image set to obtain a global feature library; image key point extraction is performed on each scene image in the scene image set to obtain a first image key point. The global feature extraction method and the first image key point extraction method of the scene image set are consistent with the extraction method of the image to be positioned. After the image key point extraction is performed in advance to obtain the first image key point, these pre-processed features can be directly used when the inspected scene image is partially reconstructed later, without repeatedly performing feature extraction, thereby improving the efficiency of image positioning.

[0106] As a preferred implementation, when the number of the valid image categories is greater than one, calculating the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, includes:

[0107] Calculating the average score of the global feature similarity between each valid image category and the image to be located, and obtaining the confidence ranking of the valid image category;

[0108] Perform mean value verification, maximum value verification and range verification on each valid image category until a valid image category that passes the verification is obtained, and the valid image category that passes the verification is used as the image category of the scene that has passed the inspection;

[0109] According to the confidence ranking, the categories of the passed-inspection scene images are sorted to obtain the categories of the passed-inspection scene images and their serial numbers.

[0110] As a preferred implementation, when the number of the valid image categories is one, calculating the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, includes:

[0111] The valid image category is subjected to mean value verification, maximum value verification and range verification. When the valid image category passes the verification, it is regarded as a passed-inspection scene image category and its sequence number is set to be the first.

[0112] As a preferred implementation manner, reconstructing a sparse model of the image category of the inspected scene according to the first image key points, matching the sparse model with the second image key points of the image to be located, and obtaining the pose of the image to be located includes:

[0113] According to the first image key points, a local scene sparse model of the inspected scene image category is constructed to obtain a key point matching relationship between the image to be located and the scene image in the sparse model;

[0114] The PnP algorithm is used to solve the key point matching relationship to obtain the position and posture of the image to be located.

[0115] Further, preferably, the step of constructing a local scene sparse model of the inspected scene image category according to the first image key points, and obtaining a key point matching relationship between the image to be located and the scene image in the sparse model includes:

[0116] The first inspected scene image category is used as the current inspected scene image category;

[0117] When the current inspected scene image category is updated, performing local scene reconstruction on the current inspected scene image category according to the first image key points to obtain a sparse model of the current inspected scene image category;

[0118] Performing 2D-2D second image key point matching between the image to be located and the sparse model, and geometrically verifying the matching result through the RANSAC algorithm to obtain the number of matching points with each scene image in the sparse model;

[0119] If the number of matching points is less than a preset threshold, the current inspected scene image category is selected and updated according to the sequence number of the inspected scene image category; otherwise, the current inspected scene image category is used as the inspected scene image category to be matched;

[0120] According to the correspondence between the second image key point of the inspected scene image category to be matched and the sparse model, a 2D-3D feature point matching relationship between the image to be located and the scene sparse point cloud is retrieved.

[0121] As an example, the local scene sparse model is obtained by performing online SfM (Structure from Motion) on the inspected scene image.

[0122] It should be noted that, in the embodiment of the present invention, the categories of scene images that have been inspected are pre-screened from the scene image set, and the number of the scene images that have been inspected is much smaller than the number of images in the scene image set. Therefore, the speed of the online SfM process is also relatively fast. According to the location information of the scene images that have been inspected, a local scene with absolute geographic coordinates and orientation can be constructed. In the prior art, it is necessary to rely on the three-dimensional model of the current scene as a positioning base, but the production of sparse model base maps of large scenes usually requires a lot of manpower and material resources, and in some cases it is even impossible to obtain. The embodiment of the present invention uses online SfM to construct a local scene sparse model of the inspected scene image, which can achieve efficient absolute positioning and posture determination of the query image without a sparse three-dimensional model of the scene, thereby reducing the time cost of positioning and improving positioning efficiency.

[0123] The scene image positioning device provided by the embodiment of the present invention performs Euclidean space clustering on the candidate scene images according to the position information, and performs confidence priority sorting between classes and internal verification within classes on the clustering results based on the global feature similarity of the candidate scene images, and then matches the key points of the second image, thereby reducing the impact of erroneous images on the final positioning results and improving the efficiency and accuracy of positioning.

[0124] An embodiment of the present invention also provides a scene image positioning device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, a scene image positioning method as described above is implemented. The working principles and beneficial effects of the two correspond one to one, and thus will not be described in detail.

[0125] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0126] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for positioning a scene image, characterized in that: include Acquire an image to be located and a scene image set; the scene image set includes an existing scene image and its global feature library and a first image key point; Extracting the global features of the image to be located, and selecting a number of scene images from the scene image set as candidate scene images according to the global feature library; According to the position information of the candidate scene images, performing Euclidean space clustering on the candidate scene images to divide the candidate scene images into at least one valid image category; Calculating the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity; Extracting a second image key point of the image to be located; Reconstructing a sparse model of the image category of the inspected scene according to the first image key points, matching the sparse model with the second image key points of the image to be located, and obtaining the position and posture of the image to be located; in, The reconstructing a sparse model of the image category of the inspected scene according to the first image key points, matching the sparse model with the second image key points of the image to be located, and obtaining the position and posture of the image to be located, includes: According to the first image key points, a sparse model of the inspected scene image category is constructed to obtain a matching relationship between the second image key points of the image to be located and the key points of the scene image in the sparse model; Using the PnP algorithm to solve the key point matching relationship, and obtain the position and posture of the image to be located; The step of constructing a sparse model of the inspected scene image category according to the first image key points, and obtaining a matching relationship between the second image key points of the image to be located and the key points of the scene image in the sparse model, includes: The first inspected scene image category is used as the current inspected scene image category; When the current inspected scene image category is updated, performing local scene reconstruction on the current inspected scene image category according to the first image key points to obtain a sparse model of the current inspected scene image category; Performing 2D-2D matching between the image to be located and the sparse model, and geometrically verifying the matching result by using the RANSAC algorithm to obtain the number of matching points with each scene image in the sparse model; If the number of matching points is less than a preset threshold, the current inspected scene image category is selected and updated according to the sequence number of the inspected scene image category; otherwise, the current inspected scene image category is used as the inspected scene image category to be matched; According to the correspondence between the second image key point of the inspected scene image category to be matched and the sparse model, a 2D-3D key point matching relationship between the image to be located and the scene sparse point cloud is retrieved.

2. A method for locating a scene image as claimed in claim 1, characterized in that: When the number of the valid image categories is greater than one, calculating the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, includes: Calculating the average score of the global feature similarity between each valid image category and the image to be located, and obtaining the confidence ranking of the valid image category; Perform mean value verification, maximum value verification and range verification on each valid image category, and use the verified valid image category as the image category of the scene that has passed the inspection; According to the confidence ranking, the categories of the passed-inspection scene images are sorted to obtain the categories of the passed-inspection scene images and their serial numbers.

3. A method for locating a scene image as claimed in claim 1, characterized in that: When the number of the valid image categories is one, calculating the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, includes: The valid image category is subjected to mean value verification, maximum value verification and range verification. When the valid image category passes the verification, it is regarded as a passed-inspection scene image category and its sequence number is set to be the first.

4. A method for locating a scene image as claimed in claim 1, characterized in that: The step of extracting the global features of the image to be located and selecting a plurality of scene images from the scene image set as candidate scene images according to the global feature library includes: Extracting global features of the image to be located; Calculating a similarity score between the global feature and each global feature in the global feature library; A number of scene images with the highest similarity scores are selected from the scene image set as candidate scene images.

5. A scene image positioning method as claimed in claim 1, characterized in that: The step of performing Euclidean space clustering on the candidate scene images according to the position information of the candidate scene images to classify the candidate scene images into at least one valid image category includes: According to the position information of the candidate scene images, performing Euclidean space clustering on the candidate scene images to divide the candidate scene images into several image categories; An image category whose number of images is greater than a preset number threshold is taken as a valid image category to obtain at least one valid image category.

6. A scene image positioning device, characterized in that: include: An image acquisition module, used to acquire an image to be located and a scene image set; the scene image set includes an existing scene image and its global feature library and a first image key point; A candidate scene image selection module is used to extract the global features of the image to be located, and select a number of scene images from the scene image set as candidate scene images according to the global feature library; An effective image category classification module, used to perform Euclidean space clustering on the candidate scene images according to the position information of the candidate scene images, and classify the candidate scene images into at least one effective image category; A passed-inspection scene image category selection module is used to calculate the global feature similarity between the valid image category and the image to be located, and obtain the passed-inspection scene image category and its sequence number according to the global feature similarity; A second image key point extraction module, used to extract the second image key point of the image to be located; An image positioning module, configured to reconstruct a sparse model of the image category of the inspection scene according to the first image key points, and match the sparse model with the second image key points of the image to be positioned to obtain the position and posture of the image to be positioned; in, The reconstructing a sparse model of the image category of the inspected scene according to the first image key points, matching the sparse model with the second image key points of the image to be located, and obtaining the position and posture of the image to be located, includes: According to the first image key points, a sparse model of the inspected scene image category is constructed to obtain a matching relationship between the second image key points of the image to be located and the key points of the scene image in the sparse model; Using the PnP algorithm to solve the key point matching relationship, and obtain the position and posture of the image to be located; The step of constructing a sparse model of the inspected scene image category according to the first image key points, and obtaining a matching relationship between the second image key points of the image to be located and the key points of the scene image in the sparse model, includes: The first inspected scene image category is used as the current inspected scene image category; When the current inspected scene image category is updated, performing local scene reconstruction on the current inspected scene image category according to the first image key points to obtain a sparse model of the current inspected scene image category; Performing 2D-2D matching between the image to be located and the sparse model, and geometrically verifying the matching result by using the RANSAC algorithm to obtain the number of matching points with each scene image in the sparse model; If the number of matching points is less than a preset threshold, the current inspected scene image category is selected and updated according to the sequence number of the inspected scene image category; otherwise, the current inspected scene image category is used as the inspected scene image category to be matched; According to the correspondence between the second image key point of the inspected scene image category to be matched and the sparse model, a 2D-3D key point matching relationship between the image to be located and the scene sparse point cloud is retrieved.

7. A scene image positioning device as claimed in claim 6, characterized in that: When the number of the valid image categories is greater than one, calculating the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, includes: Calculating the average score of the global feature similarity between each valid image category and the image to be located, and obtaining the confidence ranking of the valid image category; Perform mean value verification, maximum value verification and range verification on each valid image category until a valid image category that passes the verification is obtained, and the valid image category that passes the verification is used as the image category of the scene that has passed the inspection; According to the confidence ranking, the categories of the passed-inspection scene images are sorted to obtain the categories of the passed-inspection scene images and their serial numbers.

8. A scene image positioning device as claimed in claim 6, characterized in that: When the number of the valid image categories is one, calculating the global feature similarity between the valid image category and the image to be located, and obtaining the inspected scene image category and its sequence number according to the global feature similarity, includes: The valid image category is subjected to mean value verification, maximum value verification and range verification. When the valid image category passes the verification, it is regarded as a passed-inspection scene image category and its sequence number is set to be the first.

Images