Image-based positioning method and aircraft
Through the combination of visual big model and descriptor generation algorithm, the problem of weak generalization ability and strong data dependence of image positioning algorithm in complex environments is solved, and high-performance and robust image positioning is achieved, suitable for GPS-free The aircraft positioning in the case of signal.
Patent Information
- Application Number
- CN202410961378.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-12
AI Technical Summary
The existing image positioning algorithms have problems such as weak generalization ability and strong data dependence when dealing with complex and varied environments, resulting in insufficient positioning accuracy and robustness in the absence of GPS signals.
Using the image positioning method based on the visual big model, feature extraction of input images is performed through the visual big model, and features are converted into image descriptors using the descriptor generation algorithm, and matched with the offline image descriptor database, combining curve fitting and outgrouping Point removal algorithm improves positioning accuracy and robustness.
High performance, high robustness and low complexity image positioning in extremely complex environments is achieved, reducing dependence on data, avoiding location failure when GPS signals are unavailable, and improving the safety of the aircraft.
Smart Images

Figure CN118941760A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to an image-based positioning method and an aircraft. Background Art
[0002] In drone applications, sometimes the GPS signal is unavailable, which makes it impossible to rely on GPS for positioning. The following are some common situations where the GPS signal is unavailable:
[0003] 1) Urban canyons: In cities with tall buildings, buildings may block or reflect GPS signals, causing unstable or unavailable signals.
[0004] 2) Underground or indoor environments: In underground parking lots, tunnels or indoors, GPS signals usually cannot penetrate buildings and the ground.
[0005] 3) Densely forested areas: In dense forests, leaves and tree trunks can interfere with and absorb GPS signals.
[0006] 4) Bad weather: Extreme weather conditions, such as storms or dense fog, may affect the propagation of GPS signals.
[0007] 5) Jamming and blocking: GPS signals may be jammed by electronic jammers or intentionally blocked during military operations.
[0008] In these cases, image-based positioning technology can play an important role. Image-based positioning technology uses computer vision and machine learning algorithms to determine the position and posture of the drone by analyzing geographic images taken by the drone camera. Traditional image positioning algorithms mainly rely on classic methods of computer vision, such as feature point extraction, matching, and geometric transformation. These traditional algorithms have some problems and shortcomings when dealing with image positioning tasks:
[0009] 1) Insufficient robustness of feature point extraction and matching:
[0010] - Traditional algorithms usually rely on the extraction and matching of feature points (such as SIFT, SURF, ORB, etc.). However, in the case of illumination changes, perspective changes, scale changes, and image blur, the extraction and matching of these feature points are not robust enough, which can easily lead to positioning failure.
[0011] 2) High computational complexity:
[0012] -The feature point extraction and matching process that traditional algorithms rely on usually has high computational complexity, especially when processing high-resolution images or a large number of images, which consumes a lot of computing resources and has poor real-time performance.
[0013] 3) Poor adaptability to environmental changes:
[0014] - Traditional algorithms have poor adaptability to environmental changes (such as seasonal changes, weather changes, dynamic object occlusion, etc.). Environmental changes can cause significant changes in image features, thus affecting positioning accuracy.
[0015] 4) Depends on precise initial conditions:
[0016] - Traditional algorithms usually require more accurate initial conditions (such as initial pose estimation), otherwise they are prone to fall into local optimal solutions or fail to converge.
[0017] 5) Lack of high-level semantic information:
[0018] - Traditional algorithms mainly rely on low-level image features (such as edges, corners, etc.) and lack the use of high-level semantic information (such as object categories, scene understanding, etc.), which limits their performance in complex scenes.
[0019] 6) Poor robustness to noise and occlusion:
[0020] -In practical applications, images are often affected by noise and partial occlusion. Traditional algorithms are less robust to these interferences, which can easily lead to feature point matching errors, thus affecting the positioning results.
[0021] 7) Difficult to handle dynamic scenes:
[0022] - Traditional algorithms perform poorly when dealing with dynamic scenes (such as moving objects such as pedestrians and vehicles). These problems and shortcomings limit the application of traditional image positioning algorithms in complex and ever-changing real-world environments.
[0023] With the development of deep learning, image localization algorithms based on deep learning (such as convolutional neural networks, deep feature extraction and matching, etc.) have gradually overcome the shortcomings of these traditional algorithms and have shown higher robustness, accuracy and adaptability. However, the existing image localization algorithms based on deep learning still have some problems and limitations. The following are the two most critical points:
[0024] 1) Strong data dependence:
[0025] -Existing deep learning models usually require a large amount of labeled data for training. If the amount of data is insufficient or the data quality is not high, the performance of the model will be significantly affected.
[0026] 2) Limited generalization ability:
[0027] -Existing deep learning models have poor generalization capabilities when there is a difference between the training data distribution and the test data distribution. Summary of the invention
[0028] In view of the above analysis, the embodiments of the present invention aim to provide an image-based positioning method and an aircraft to solve the inherent limitations of existing image positioning algorithms, such as weak generalization ability and strong data dependence.
[0029] In a first aspect, an embodiment of the present invention provides an image-based positioning method, comprising the following steps:
[0030] Use the big vision model to extract features from the input image;
[0031] Converting the features extracted from the input image into an image descriptor according to a descriptor generation algorithm; and
[0032] Matching the image descriptor with an offline image descriptor in an offline image descriptor database to obtain position information corresponding to the input image;
[0033] The offline image descriptor database includes offline image descriptors of offline images and their corresponding position information.
[0034] Based on the further improvement of the above positioning method, the offline image descriptor database is obtained by the following method:
[0035] Get the location information corresponding to the offline image;
[0036] Extracting features from the offline image using a large visual model; and
[0037] The features extracted from the offline image are converted into offline image descriptors according to a descriptor generation algorithm.
[0038] Based on the further improvement of the above positioning method, the positioning method further includes the following steps:
[0039] Acquire location information corresponding to the input images at multiple historical moments within a preset time window that are adjacent to the input image at the current moment;
[0040] Performing curve fitting on the position information corresponding to the input image at the current moment and the position information corresponding to the input images at multiple historical moments; and
[0041] According to the curve fitting result, the position information corresponding to the input image at the current moment is adjusted.
[0042] Based on the further improvement of the above positioning method, before performing curve fitting on the position information corresponding to the input image at the current moment and the position information corresponding to the input images at multiple historical moments, the positioning method further includes the following steps:
[0043] An outlier removal algorithm is used to filter the position information corresponding to the input image at the current moment and the position information corresponding to the input images at multiple historical moments.
[0044] Based on the further improvement of the above positioning method, matching the image descriptor with the offline image descriptor in the offline image descriptor database includes:
[0045] calculating a Euclidean distance between the image descriptor and the offline image descriptor;
[0046] An offline image descriptor matching the image descriptor is determined according to the size of the Euclidean distance.
[0047] Based on the further improvement of the above positioning method, the position information corresponding to the input image at the historical moment is obtained by the following method:
[0048] Acquire location information corresponding to the input images at multiple moments within a preset time window that are adjacent to the input image at the historical moment;
[0049] Performing curve fitting on the position information corresponding to the input image at the historical moment and the position information corresponding to the input images at the multiple moments; and
[0050] According to the curve fitting result, the position information corresponding to the input image at the historical moment is adjusted.
[0051] Based on the further improvement of the above positioning method, the visual large model is any one of the following:
[0052] DINO-v1 model, DINO-v2 model, SAM large model.
[0053] Based on the further improvement of the above positioning method, the descriptor generation algorithm is a local feature aggregation descriptor algorithm, and the image descriptor or offline image descriptor is a 49152-dimensional vector after L2 norm normalization.
[0054] Based on the further improvement of the above positioning method, the outlier removal algorithm is a random sampling consensus algorithm.
[0055] In a second aspect, an embodiment of the present invention provides an aircraft, including:
[0056] at least one processor; and
[0057] A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image-based positioning method described in any one of the first aspects of the present invention.
[0058] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0059] 1. The solution of the present invention proposes an image positioning algorithm based on the visual big model technology, which makes full use of the visual big model to improve generalization and directly applies the visual big model without fine-tuning the visual big model, thus reducing data dependence and enabling the visual big model technology to be applied in the field of graphic positioning.
[0060] 2. The solution of the present invention can provide positioning guidance for aircraft in extremely complex situations or in the absence of GPS positioning information, thereby avoiding catastrophic accidents such as aircraft crash and loss of life.
[0061] 3. The solution of the present invention provides a high-performance, high-robustness and low-complexity image-based positioning method that can be applied to aircraft.
[0062] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;
[0064] Figure 1 A schematic flow chart of an image-based positioning method according to an embodiment of the present invention is shown.
[0065] Figure 2 A block diagram of an aircraft provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0066] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.
[0067] Figure 1 FIG. 2 shows a flow chart of an image-based positioning method according to an embodiment of the present invention. Figure 1As shown, the image-based positioning method includes the following steps:
[0068] S101: Use the visual big model to extract features from the input image.
[0069] Visual big models are visual AI models with a large number of parameters. These models are usually trained on large datasets and are able to learn complex image patterns and relationships. The number of parameters of visual big models is usually in the millions or billions. For example, the DALL-E 2 model has 13.7 billion parameters, and the Imagen model (Google's diffusion-based model) has 66 billion parameters. Generally speaking, the more parameters there are, the stronger the learning ability of the visual big model.
[0070] The visual large model in this embodiment can be a DINO-v2 model. The DINO-v2 model is a computer vision self-supervised model released by Meta, which is trained by self-supervised learning and thus has good generalization capabilities. The visual large model in this embodiment can also be a SAM (Segment Anything) model. The SAM model is a visual large model released by Meta, which is pre-trained on a wide range of data sets using tasks that can achieve powerful generalization. This embodiment can also use other visual large models, such as the DINO-v1 model, the DALL-E 2 model, and the like.
[0071] In this embodiment, the use of the visual large model to extract features from the input image can be to extract the last layer of the network before the specific task head of the visual large model, that is, the output of the last layer of the visual large model. Taking the DINO-v2 model as an example, the two-dimensional feature map of the convolutional neural layer of the last layer of the DINO-v2 model output after the ReLU function activation layer can be obtained. The number of feature maps is 512, and each feature is obtained in a data-driven manner, which physically represents the generalizable features of the image semantic information.
[0072] S102: Converting the features extracted from the input image into image descriptors according to a descriptor generation algorithm.
[0073] Descriptors are mathematical tools used to extract feature information from images. These features can help computers understand and distinguish different image contents. Descriptors usually have certain invariances, such as scale invariance, rotation invariance, and illumination invariance, so that even after the image undergoes these changes, the descriptor can still be effectively matched and recognized. Descriptor generation algorithms can convert features extracted from images into vectors in distance space.
[0074] The descriptor generation algorithm in this embodiment can be a vector of locally aggregated descriptors (VLAD) algorithm. The VLAD algorithm trains a small codebook through a clustering method, finds the nearest codebook cluster center for each feature in the input image, and then accumulates the difference between all features and the cluster center to obtain a k*d matrix (where k is the number of cluster centers and d is the feature dimension), and then transforms the matrix into a (k*d)-dimensional vector and normalizes it. The resulting vector is the image descriptor in this embodiment.
[0075] In some embodiments, the (k*d)-dimensional vector may be a 49152-dimensional vector, and the image descriptor in this embodiment may be a vector obtained by performing L2 norm normalization on the 49152-dimensional vector.
[0076] S103: Matching the image descriptor with an offline image descriptor in an offline image descriptor database to obtain position information corresponding to the input image; wherein the offline image descriptor database includes offline image descriptors of offline images and their corresponding position information.
[0077] In this embodiment, the image descriptor obtained according to step S102 can be matched with the offline image descriptor in the offline image descriptor database, and the offline image descriptor matching the image descriptor obtained in step S102 can be determined according to the similarity. For example, the offline image descriptor with the highest similarity to the image descriptor obtained in step S102 can be selected from the offline image descriptor database. In this embodiment, each offline image descriptor in the offline image descriptor database carries corresponding location information. The location information here can be longitude and latitude information. After selecting the offline image descriptor matching the image descriptor obtained in step S102, the location information corresponding to the matched offline image descriptor can be copied to the input image and used as the location information of the input image.
[0078] In some embodiments, the offline image descriptor database may be obtained by:
[0079] Step 10: Get the location information corresponding to the offline image.
[0080] In the offline operation stage, GPS can be used to obtain the location information of the offline image, and the location information here can be longitude and latitude information.
[0081] Step 20: Use the visual big model to extract features from the offline image.
[0082] The visual large model in this embodiment can be a DINO-v2 model. The DINO-v2 model is a computer vision self-supervised model released by Meta, which is trained by self-supervised learning and thus has good generalization capabilities. The visual large model in this embodiment can also be a SAM (Segment Anything) model. The SAM model is a visual large model released by Meta, which is pre-trained on a wide range of data sets using tasks that can achieve powerful generalization. This embodiment can also use other visual large models, such as the DINO-v1 model, the DALL-E 2 model, and the like.
[0083] In this embodiment, the use of the visual large model to extract features from offline images can be to extract the features output by the last layer of the network before the specific task head of the visual large model, that is, the output of the last layer of the visual large model. Taking the DINO-v2 model as an example, the two-dimensional feature map of the convolutional neural layer of the last layer of the DINO-v2 model after the ReLU function activation layer can be obtained. The number of feature maps is 512, and each feature is obtained in a data-driven manner, which physically represents the generalizable features of the image semantic information.
[0084] Step 30: Convert the features extracted from the offline image into offline image descriptors according to a descriptor generation algorithm.
[0085] The descriptor generation algorithm in this embodiment can be a Vector of Locally Aggregated Descriptors (VLAD) algorithm. The VLAD algorithm trains a small codebook through a clustering method, finds the nearest codebook cluster center for each feature in the offline image, and then accumulates the difference between all features and the cluster center to obtain a k*d matrix (where k is the number of cluster centers and d is the feature dimension), and then transforms the matrix into a (k*d)-dimensional vector and normalizes it. The resulting vector is the offline image descriptor in this embodiment.
[0086] In some embodiments, the (k*d)-dimensional vector may be a 49152-dimensional vector, and the offline image descriptor in this embodiment may be a vector obtained by performing L2 norm normalization on the 49152-dimensional vector.
[0087] Since the location information of the offline image has been acquired in step 10 , after the offline image descriptor is obtained in step 30 , the location information of the offline image descriptor is the location information of the offline image.
[0088] In some embodiments, Figure 1 The image-based positioning method shown further includes, after S103:
[0089] Step S104: Acquire location information corresponding to the input images at multiple historical moments within a preset time window that are adjacent to the input image at the current moment.
[0090] In this embodiment, the sliding window strategy can be used to filter out the position information of adjacent windows. Specifically, assuming that the length of the sliding window is L (L is an integer), and the current moment of the input image is T, the position information of the input image at the moment T-1, T-2, T-3, ..., T-(L-1) can be obtained. It should be noted that in this embodiment, the next image can be captured by hardware or software triggering at fixed intervals or by other means. Taking the capture of the next image at fixed intervals t (for example, 5 seconds) as an example, the input images at the moment T-1, T-2, T-3, ..., T-(L-1) refer to the images captured at the time when the difference from the current moment T is t, 2t, 3t, ..., (L-1)t respectively. The position information of the input images at the moment T-1, T-2, T-3, ..., T-(L-1) can be obtained by the method shown in step S101-step S103.
[0091] In other embodiments, the position information of the input images at time T-1, T-2, T-3, ..., T-(L-1) can also be obtained by the method shown in step S101 to step S106. In this embodiment, the position information corresponding to the input images at the historical moments is obtained by the following method:
[0092] Step 111: Acquire location information corresponding to the input images at multiple moments within a preset time window that are adjacent to the input image at the historical moment.
[0093] Step 222: Perform curve fitting on the position information corresponding to the input image at the historical moment and the position information corresponding to the input images at the multiple moments.
[0094] Step 333: According to the curve fitting result, adjust the position information corresponding to the input image at the historical moment.
[0095] The above steps 111 to 333 are described below by taking the position information of the input image at the historical moment T-1 as an example.
[0096] First, the position information of the input image at time T-2, T-3, ..., TL is obtained. Then, the method shown in step S105 (described in detail below) is used to perform curve fitting on the position information of the input image at time T-1, T-2, T-3, ..., TL. Finally, the method shown in step S106 (described in detail below) is used to adjust the position information of the input image at the historical moment T-1.
[0097] Step S105: performing curve fitting on the position information corresponding to the input image at the current moment and the position information corresponding to the input images at multiple historical moments.
[0098] The curve fitting in this embodiment may be polynomial curve fitting. Polynomial curve fitting refers to fitting the original data by a polynomial curve under the premise that the known data are correlated. In this embodiment, the position information of the input image at the current moment and the position information of the input images at multiple past moments may be combined into a position information list, and then the data in the position information list may be fitted using a polynomial fitting method to obtain fitting curve parameters.
[0099] Step S106: adjusting the position information corresponding to the input image at the current moment according to the curve fitting result.
[0100] The curve fitting result in this embodiment includes the fitting curve parameters. In this embodiment, the position information of the input image at the current moment can be replaced by the position information at the current moment determined according to the fitting curve parameters. For example, if the fitting curve parameters are: y = x 2 +1, the current position information obtained according to step S101-step S103 is (x 0 ,y 0 ), fitting curve: y = x 2 +1 at the current moment has the value (x 1 ,y 1 ), where x 0 ≠x 1 ,y 0 ≠y 1 ; The position information of the input image at the current moment can be obtained by (x 0 ,y 0 ) to (x 1 ,y 1 ).
[0101] In some embodiments, before performing curve fitting on the position information corresponding to the input image at the current moment and the position information corresponding to the input images at multiple historical moments, the positioning method further includes the following steps:
[0102] An outlier removal algorithm is used to filter the position information corresponding to the input image at the current moment and the position information corresponding to the input images at multiple historical moments.
[0103] Outlier removal algorithms are often used in the data preprocessing stage to identify and exclude outliers that significantly deviate from the majority of data points. Outlier removal algorithms are crucial to ensure the accuracy of subsequent analysis, as outliers may have a misleading effect on the data model.
[0104] In some embodiments, the outlier removal algorithm can be a random sample consensus (RANSAC) algorithm. The RANSAC algorithm is an iterative algorithm for correctly estimating mathematical model parameters from a set of data containing outliers. "Outliers" refer to noise in the data, such as mismatches in matching and outliers in the estimated curve.
[0105] In some embodiments, step S103 includes:
[0106] Step 100: Calculate the Euclidean distance between the image descriptor and the offline image descriptor.
[0107] In this embodiment, a metric algorithm may be used to calculate the distance between the real-time image descriptor and the offline image descriptor in the offline image descriptor database. For example, the Euclidean distance between the real-time image descriptor and the offline image descriptor may be calculated.
[0108] Step 200: Determine an offline image descriptor that matches the image descriptor according to the size of the Euclidean distance.
[0109] In this embodiment, an offline image descriptor having the smallest Euclidean distance with the real-time image descriptor may be selected from the offline image descriptor database as the offline image descriptor that matches the real-time image descriptor.
[0110] An embodiment of the present invention further provides an aircraft, Figure 2 A block diagram of an aircraft provided by an embodiment of the present invention. Figure 2 The aircraft includes a processor 201 and a memory 202 storing executable instructions of the processor 201. The processor 201 is connected to the memory via a communication bus 203. The processor 201 can read executable instructions from the memory 202 to implement any of the image-based positioning methods described above.
[0111] Compared with the prior art, the embodiments of the present invention can achieve at least one of the following beneficial effects:
[0112] 1. The solution of the present invention proposes an image positioning algorithm based on the visual large model technology. On the premise of making full use of the visual large model to improve generalization, the visual large model technology can be applied in the field of graphic positioning.
[0113] 2. The solution of the present invention can provide positioning guidance for aircraft in extremely complex situations or in the absence of GPS positioning information, thereby avoiding catastrophic accidents such as aircraft crash and loss of life.
[0114] 3. The solution of the present invention provides a high-performance, high-robustness and low-complexity image-based positioning method that can be applied to aircraft.
[0115] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.
Claims
1. An image-based positioning method, characterized in that: The following steps are involved: Use the big vision model to extract features from the input image; Converting the features extracted from the input image into an image descriptor according to a descriptor generation algorithm; as well as Matching the image descriptor with an offline image descriptor in an offline image descriptor database to obtain position information corresponding to the input image; The offline image descriptor database includes offline image descriptors of offline images and their corresponding position information.
2. The positioning method according to claim 1, characterized in that: The offline image descriptor database is obtained by the following method: Get the location information corresponding to the offline image; Extracting features from the offline image using a large visual model; and The features extracted from the offline image are converted into offline image descriptors according to a descriptor generation algorithm.
3. The positioning method according to claim 1 or 2, characterized in that: The positioning method further comprises the following steps: Acquire location information corresponding to the input images at multiple historical moments within a preset time window that are adjacent to the input image at the current moment; Performing curve fitting on the position information corresponding to the input image at the current moment and the position information corresponding to the input images at multiple historical moments; and According to the curve fitting result, the position information corresponding to the input image at the current moment is adjusted.
4. The positioning method according to claim 3, characterized in that: Before performing curve fitting on the position information corresponding to the input image at the current moment and the position information corresponding to the input images at multiple historical moments, the positioning method further includes the following steps: An outlier removal algorithm is used to filter the position information corresponding to the input image at the current moment and the position information corresponding to the input images at multiple historical moments.
5. The positioning method according to claim 1 or 2, characterized in that: Matching the image descriptor with an offline image descriptor in an offline image descriptor database comprises: calculating a Euclidean distance between the image descriptor and the offline image descriptor; An offline image descriptor matching the image descriptor is determined according to the size of the Euclidean distance.
6. The positioning method according to claim 3, characterized in that: The position information corresponding to the input image at the historical moment is obtained by the following method: Acquire location information corresponding to the input images at multiple moments within a preset time window that are adjacent to the input image at the historical moment; Performing curve fitting on the position information corresponding to the input image at the historical moment and the position information corresponding to the input images at the multiple moments; and According to the curve fitting result, the position information corresponding to the input image at the historical moment is adjusted.
7. The positioning method according to claim 1 or 2, characterized in that: The visual macro model is any one of the following: DINO-v1 model, DINO-v2 model, SAM large model.
8. The positioning method according to claim 1 or 2, characterized in that: The descriptor generation algorithm is a local feature aggregation descriptor algorithm, and the image descriptor or offline image descriptor is a 49152-dimensional vector after L2 norm normalization processing.
9. The positioning method according to claim 4 is characterized in that the outlier removal algorithm is a random sampling consensus algorithm.
10. An aircraft, comprising: at least one processor; as well as A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the positioning method described in any one of claims 1 to 9.
