Street view and satellite image matching method and equipment

Abstract

The invention provides a street view and satellite image matching method and equipment. The method comprises the following steps: acquiring a street view image, a satellite image and an unmanned aerial vehicle video library; extracting a first visual feature of the satellite image and a second visual feature of the unmanned aerial vehicle video key frame; calculating a first similarity score between the satellite image and the unmanned aerial vehicle video key frame by using the first visual feature and the second visual feature; performing visual feature comparison and semantic description association on the streetscape image and the unmanned aerial vehicle video key frame, and calculating a second similarity score between the streetscape image and the unmanned aerial vehicle video; and based on the first similarity score and the second similarity score, bridging calculation is carried out by using the unmanned aerial vehicle video as transfer to obtain an indirect matching score of the streetscape image and the satellite image, and a cross-view matching result is obtained in combination with the direct visual feature matching score. Based on the method, the invention also provides a street view and satellite image matching device. According to the method, the accuracy and practicability of cross-view image retrieval are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and geographic spatial intelligence, and particularly relates to a street view and satellite image matching method and device. BACKGROUND

[0002] The cross-view matching of street view images and satellite images is a key technology in the fields of automatic driving, smart city, precise navigation and geographic information system. The essence is to determine the corresponding geographic location of a street view image in a satellite image library covering a wide area.

[0003] However, due to the fundamental difference between the two image modalities, the existing technology faces severe challenges: first, street view images are ground-level views that mainly capture side-view information such as building facades and street facilities; satellite images are high-altitude vertical overhead views that mainly present top-view information such as building roofs and road layouts. Both lack common visual observation content. Second, traditional image matching methods based on local features (such as SIFT, ORB) or global features (such as CNN deep features) rely on the existence of similar textures, shapes or structures between images. Due to the above-mentioned view difference, street view and satellite images have little comparability at the pixel level and shallow feature level, resulting in the failure of direct matching methods based on visual features. Third, supervised deep learning-based methods require a large amount of accurately labeled "street view-satellite" image pairs for training, which has a high data acquisition cost, and the model is heavily dependent on the training data distribution, with limited generalization ability in unknown scenarios or complex urban environments. Therefore, the existing technology cannot effectively and robustly solve the cross-view matching problem between street view and satellite images. SUMMARY

[0004] To solve the above technical problems, the present application proposes a street view and satellite image matching method and device, which effectively utilizes the transitional characteristics of the unmanned aerial vehicle view and the high-level semantic understanding ability of the LLM, significantly improving the robustness and accuracy of the matching.

[0005] To achieve the above purpose, the present application adopts the following technical solutions: The present application proposes a street view and satellite image matching method, comprising the following steps: Obtaining a street view image, a satellite image and a UAV video library; Extracting a first visual feature of the satellite image and a second visual feature of a key frame of the UAV video; calculating a first similarity score between the satellite image and the key frame of the UAV video using the first visual feature and the second visual feature; By comparing the visual features of the street view image and the key frame of the UAV video and associating them with semantic descriptions, a second similarity score between the street view image and the UAV video is calculated; Based on the first similarity score and the second similarity score, the drone video is used as a relay for bridging calculation to obtain the indirect matching score between the street view image and the satellite image. Combined with the direct visual feature matching score, the cross-view matching result is obtained.

[0006] The present invention also proposes a street view and satellite image matching device, comprising at least one processor and a memory, wherein the memory stores a computer program, and the computer program, when executed by the at least one processor, implements the street view and satellite image matching method.

[0007] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects: This invention proposes a method and device for matching street view and satellite images. The method includes the following steps: acquiring street view images, satellite images, and a drone video library; extracting first visual features from the satellite images and second visual features from keyframes of the drone videos; calculating a first similarity score between the satellite images and keyframes of the drone videos using the first and second visual features; calculating a second similarity score between the street view images and drone videos by comparing visual features and associating them with semantic descriptions; based on the first and second similarity scores, using drone videos as a relay for bridging calculations to obtain an indirect matching score between the street view images and satellite images, and combining this with the direct visual feature matching score to obtain a cross-view matching result. Based on this method for matching street view and satellite images, a matching device for street view and satellite images is also proposed. This invention introduces drone videos as a relay modality and utilizes a large language model to extract high-level semantic information, constructing a robust matching framework that does not require paired supervised data and can effectively overcome viewpoint differences, significantly improving the accuracy and practicality of cross-view image retrieval.

[0008] This invention employs unsupervised geometric matching and a pre-trained model, eliminating the need for training with paired street view-satellite labeled data, thus significantly reducing data acquisition costs and enhancing the practicality and scalability of the method. Attached Figure Description

[0009] Figure 1 This is a flowchart of a street view and satellite image matching method proposed in Embodiment 1 of the present invention; Figure 2 This is an architecture diagram of a street view and satellite image matching method proposed in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of cross-view feature matching proposed in Embodiment 1 of the present invention; Figure 4This is a schematic diagram of a street view and satellite image matching device proposed in Embodiment 1 of the present invention. Detailed Implementation

[0010] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0011] Example 1 Embodiment 1 of this invention proposes a method for matching street view and satellite imagery, which is used to solve the problem of cross-view matching between street view and satellite imagery that is difficult to solve effectively and robustly in the prior art.

[0012] Figure 1 This is a flowchart of a street view and satellite image matching method proposed in Embodiment 1 of the present invention; Figure 2 This is an architecture diagram of a street view and satellite image matching method proposed in Embodiment 1 of the present invention.

[0013] In step S100, street view images, satellite images, and drone video libraries are acquired; The street view images to be matched in this application may be from open-source street view service platforms on the Internet, or image databases collected by vehicles or mobile devices. Satellite imagery can be sourced from orthophoto libraries provided by commercial or open-source satellite imagery providers; A drone video library can be built by using drones equipped with cameras to capture images of target areas along preset routes and at preset altitudes, creating a video or keyframe sequence library with geographic reference information.

[0014] The methods for obtaining data in this application are not limited to those listed in Example 1, and those skilled in the art can make reasonable choices based on the actual situation.

[0015] In step S200, the first visual features of the satellite image are extracted, and the second visual features of the keyframes of the UAV video are extracted; the first visual features and the second visual features are used to calculate the first similarity score between the satellite image and the keyframes of the UAV video. Extracting the first visual features of satellite images and the second visual features of keyframes from UAV videos involves: using a feature point extraction algorithm to extract the first visual features of satellite images, which include key points of the satellite images and their descriptors; for example, using the SuperPoint algorithm with configuration parameters set to a maximum of 1024 key points and a key point threshold of 0.005.

[0016] A feature point extraction algorithm is used to extract the second visual features of key frames in UAV video. The second visual features include key points and their descriptors of key frames in UAV video. The UAV video is temporally sampled (e.g., one frame per second) to select key frames.

[0017] This application utilizes first and second visual features to calculate a first similarity score between satellite images and keyframes of UAV videos. This first similarity score quantifies the geometric and visual feature consistency between the satellite images and the keyframes of the UAV videos. The specific process includes: A feature point matching algorithm is used to match the first set of visual features with the second set of visual features to obtain an initial matching set; for example, the LightGlue algorithm is used to calculate satellite image features. With drone frame features Matching sets between .

[0018] Geometric verification is performed on the initial matching set to estimate the homography matrix and select the interior point set; the RANSAC (Random Sample Consensus) algorithm is then applied to estimate the homography matrix. And filter inliers. The reprojection error threshold is set to 3.0 pixels. The homography transformation formula is: ; in, For UAV frame coordinates; Coordinates corresponding to the satellite image Based on the inlier set, the inlier rate, match confidence, normalized match count, and spatial distribution score are calculated. The inlier rate is used to measure geometric consistency, the match confidence is the average confidence score of inlier matches, the normalized match count is used to prevent over-reliance on the number of matches, and the spatial distribution score is calculated by dividing the image into a grid and calculating the spatial coverage of feature points to avoid mismatches caused by feature clustering.

[0019] The first similarity score is obtained by weighting and summing the inlier rate, matching confidence, normalized number of matches, and spatial distribution score.

[0020] ; in, This represents the first similarity score; Represents the interior point ratio; Represents the confidence level of the match; Represents the number of normalized matches; Represents the spatial distribution score; Represents the weighting coefficient of the inside point ratio; This represents the matching confidence weight coefficient; Represents the weighting coefficient for the normalized matching quantity; The weighting coefficients represent the spatial distribution score.

[0021] In this application , , and Used to adjust the interior point ratio Matching confidence Normalized matching count Spatial distribution score The proportion of contribution in the first similarity score. Possible values ​​are , , , .

[0022] The scope of protection of this invention is not limited to the data listed in Example 1, and those skilled in the art can make reasonable adjustments according to the actual situation.

[0023] In step S300, a second similarity score is calculated between the street view image and the drone video by comparing visual features and associating them with semantic descriptions through keyframes of the street view image and the drone video. Specifically, this includes: Extract the third visual features from the street view image and the fourth visual features from the keyframes of the drone video; calculate the visual similarity using the third and fourth visual features; In implementation, a first pre-trained visual encoder is used to extract the third visual features of the street view image to be matched; a second pre-trained visual encoder is used to extract the fourth visual features of the key frames of the drone video. Calculate the cosine similarity between the third visual feature and the fourth visual feature, and use it as the visual similarity.

[0024] In this application, a first pre-trained visual encoder (e.g., an image encoder in a GeoCLIP model pre-trained based on contrastive learning) is used to process the input street scene image to be matched and extract its high-level visual feature representation, denoted as the third visual feature.

[0025] A second pre-trained visual encoder (such as the image encoder in the RemoteCLIP model, which is also based on contrastive learning and applicable to aerial or oblique imagery) is used to process the input UAV video keyframes and extract their high-level visual feature representations, denoted as the fourth visual features.

[0026] The formula for calculating visual similarity is: ; in, Indicates visual similarity; Represents third visual characteristics; Represents the fourth visual characteristic; Based on the structured questionnaire, a visual-language big data model is used to generate text descriptions of the street view images and drone videos respectively. After encoding the text descriptions into semantic feature vectors, the semantic similarity between the two is calculated. The structured questionnaire in this application includes six dimensions: built environment / spatial context, color and aesthetics, materials and style, surrounding objects / landmarks, facade features (windows, doors), and roof features.

[0027] Examples of structured questionnaire responses include: Can you see the whole building? No Building Color Reddish-brown Building Materials Brick The shape of the building Rectangular Building Facade Characteristics Multiple rectangular windows, no visible doors or balconies. Building Roof Characteristics A flat roof with a greenish hue, featuring a central tower. Surrounding Path Sidewalk adjacent to the building Additional Information: Ornate architectural details on the facade, in decorative stonework, and the presence of two people walking on the sidewalk.

[0028] The scope of protection of this invention is not limited to the examples of structured questionnaire responses listed in Example 1. Those skilled in the art can set different questions and answers according to specific circumstances.

[0029] Street view images are input into the Visual-Language Large Model, and a first text description corresponding to the image content is generated based on a structured questionnaire.

[0030] One or more keyframes from drone video are input into the same or similar large-scale visual-language model, and a second text description corresponding to the video content is generated based on the same structured questionnaire. For videos, responses from multiple frames can be aggregated to form a more comprehensive description.

[0031] Using a pre-trained text encoder (such as the Sentence-BERT model based on the Transformer architecture), the first text description is encoded into a fixed-dimensional first semantic feature vector. Using the same pre-trained text encoder, the second text description is encoded into a fixed-dimensional second semantic feature vector.

[0032] Calculate the first semantic feature vector Second semantic feature vector The cosine similarity between them is used as a semantic similarity. ; in, Represents the first semantic feature vector Second semantic feature vector Cosine similarity between This encoding process transforms unstructured natural language text into points in a high-dimensional vector space, where semantically similar texts correspond to vectors that are closer together in space.

[0033] The visual similarity and semantic similarity are weighted and summed to calculate a second similarity score between the street view image and the drone video keyframes.

[0034] ; in, Represents visual similarity Weighting coefficients; Represents semantic similarity The weighting coefficients. In this application ; .

[0035] The scope of protection of this invention is not limited to the values ​​listed in Example 1, and those skilled in the art can make reasonable selections based on the actual situation.

[0036] In step S400, based on the first similarity score and the second similarity score, the drone video is used as an intermediary for bridging calculation to obtain the indirect matching score between the street view image and the satellite image, and combined with the direct visual feature matching score to obtain the cross-view matching result. Figure 3 This is a schematic diagram of cross-view feature matching proposed in Embodiment 1 of the present invention; The specific process includes: selecting the top K drone videos most similar to the street view images based on the second similarity score to form a first candidate video set; selecting the top K drone videos most similar to the satellite images based on the first similarity score to form a second candidate video set; in this application, K=20. Based on the drone videos commonly included in the first and second candidate video sets, and combining their corresponding first and second similarity scores, a joint association score between the street view image and the satellite image is calculated as an indirect matching score.

[0037] By utilizing the third visual features of street view images and the fourth visual features of keyframes from drone videos, the similarity between the third visual features and the fourth visual features is calculated as a direct matching score. The indirect matching score and the direct matching score are weighted and combined to obtain the final matching score. The specific formula is as follows: ; in, The weighting coefficients representing the indirect matching score; The weighting coefficients representing the direct match scores. , .

[0038] The scope of protection of this invention is not limited to the values ​​listed in Example 1, and those skilled in the art can make reasonable selections based on the actual situation.

[0039] The street view and satellite image matching method proposed in Embodiment 1 of this invention introduces drone video as a relay modality and uses a large language model to extract high-level semantic information, thereby constructing a robust matching framework that can effectively overcome perspective differences without pairwise supervision data, significantly improving the accuracy and practicality of cross-view image retrieval.

[0040] Example 2 Embodiment 2 of the present invention also proposes a street view and satellite image matching device. Figure 4 This is a schematic diagram of a street view and satellite image matching device proposed in Embodiment 1 of the present invention, comprising: Memory, used to store computer programs; When a processor executes the computer program, the method steps are as follows: In step S100, street view images, satellite images, and drone video libraries are acquired; In step S200, the first visual features of the satellite image are extracted, and the second visual features of the keyframes of the UAV video are extracted; the first visual features and the second visual features are used to calculate the first similarity score between the satellite image and the keyframes of the UAV video. In step S300, a second similarity score between the street view image and the drone video is calculated by comparing the visual features of the street view image and the drone video keyframes and associating them with semantic descriptions.

[0041] In step S400, based on the first similarity score and the second similarity score, the drone video is used as an intermediary for bridging calculation to obtain the indirect matching score between the street view image and the satellite image, and combined with the direct visual feature matching score to obtain the cross-view matching result.

[0042] It should be noted that the present invention also provides a street view and satellite image matching device, including: a communication interface capable of interacting with other devices such as network devices; and a processor connected to the communication interface to enable information interaction with other devices, used to execute a street view and satellite image matching method provided by one or more of the above technical solutions when running a computer program, wherein the computer program is stored in a memory. Of course, in practical applications, the various components in the electronic device are coupled together through a bus system. It is understood that the bus system is used to realize the connection and communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus. The memory in the embodiments of this application is used to store various types of data to support the operation of the electronic device. Examples of this data include any computer program used to operate on the electronic device. It is understood that the memory can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory, flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache.By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memory. The methods disclosed in the embodiments of this application can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The processor can be a general-purpose processor, a DSP (Digital Signal Processing, i.e., a chip capable of implementing digital signal processing technology), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, which is located in memory. The processor reads the program from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method. When the processor executes the program, it implements the corresponding processes in the various methods of the embodiments of this application; for simplicity, these will not be elaborated further here.

[0043] For a description of the relevant parts of the street view and satellite image matching device provided in Embodiment 2 of this application, please refer to the detailed description of the corresponding parts of the street view and satellite image matching method provided in Embodiment 1 of this application, which will not be repeated here.

[0044] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0045] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method of matching a street scene to a satellite image, characterized by, The method comprises the following steps: obtaining a street view image, a satellite image and a UAV video library; extracting first visual features of the satellite image and second visual features of a UAV video key frame; calculating a first similarity score between the satellite image and the UAV video key frame by using the first visual features and the second visual features; calculating a second similarity score between the street view image and the UAV video by comparing the visual features and associating the semantic descriptions of the street view image and the UAV video key frame; based on the first similarity score and the second similarity score, using the UAV video as a relay to calculate a bridging score, obtaining an indirect matching score between the street view image and the satellite image, and combining the direct visual feature matching score to obtain a cross-view matching result.

2. The method of claim 1, wherein, The first visual features of the satellite image and the second visual features of the UAV video key frame are extracted, specifically: the first visual features of the satellite image are extracted by using a feature point extraction algorithm, and the first visual features include key points and their descriptors of the satellite image; the second visual features of the UAV video key frame are extracted by using a feature point extraction algorithm, and the second visual features include key points and their descriptors of the UAV video key frame.

3. The method of claim 2, wherein, The first similarity score between the satellite image and the UAV video key frame is calculated by using the first visual features and the second visual features, specifically: the first group of visual features and the second group of visual features are matched by using a feature point matching algorithm to obtain an initial matching set; the initial matching set is subjected to geometric verification, a homography matrix is estimated, and an inlier set is screened out; based on the inlier set, an inlier rate, a matching confidence, a normalized matching number and a spatial distribution score are calculated; the inlier rate, the matching confidence, the normalized matching number and the spatial distribution score are weighted and summed to obtain the first similarity score.

4. The method of claim 1, wherein, The second similarity score between the street view image and the UAV video is calculated by comparing the visual features and associating the semantic descriptions of the street view image and the UAV video key frame, specifically: third visual features of the street view image and fourth visual features of the UAV video key frame are extracted; visual similarity is calculated by using the third visual features and the fourth visual features; based on a structured questionnaire, a visual-linguistic large model is used to generate text descriptions of the street view image and the UAV video respectively, and after the text descriptions are encoded into semantic feature vectors, a semantic similarity between the two is calculated; the visual similarity and the semantic similarity are weighted and summed to calculate the second similarity score between the street view image and the UAV video key frame.

5. The method of claim 4 wherein, The third visual features of the street view image and the fourth visual features of the UAV video key frame are extracted; the visual similarity is calculated by using the third visual features and the fourth visual features, specifically: a first pre-trained visual encoder is used to extract the third visual features of the street view image to be matched; a second pre-trained visual encoder is used to extract the fourth visual features of the UAV video key frame; the cosine similarity between the third visual features and the fourth visual features is calculated as the visual similarity.

6. The method of claim 4, wherein, Based on the structured questionnaire, the visual-linguistic large model is used to generate text descriptions of the street view image and the unmanned aerial vehicle video respectively, and the text descriptions are encoded into semantic feature vectors, and the semantic similarity between the two is calculated, specifically: The street view image and the key frame of the unmanned aerial vehicle video are input into the visual-linguistic large model, and the corresponding first text description and second text description are generated based on the pre-defined structured questionnaire. The first text description and the second text description are encoded into the first semantic feature vector and the second semantic feature vector by using the pre-trained text encoder. The cosine similarity between the first semantic feature vector and the second semantic feature vector is calculated as the semantic similarity.

7. The method of claim 6, wherein, The structured questionnaire includes descriptive questions in six dimensions of architectural environment, color, material, surrounding landmarks, facade features and roof features.

8. The method of claim 1, wherein, Based on the first similarity score and the second similarity score, the unmanned aerial vehicle video is used as a relay to bridge calculation, and the indirect matching score of the street view image and the satellite image is obtained, specifically: According to the second similarity score, the first candidate video set is constructed by selecting the top K unmanned aerial vehicle videos most similar to the street view image. According to the first similarity score, the second candidate video set is constructed by selecting the top K unmanned aerial vehicle videos most similar to the satellite image. Based on the unmanned aerial vehicle videos commonly contained in the first candidate video set and the second candidate video set, combined with the corresponding first similarity score and the second similarity score, the joint correlation score of the street view image and the satellite image is calculated as the indirect matching score.

9. The method of claim 8, wherein, Combined with the direct visual feature matching score, the cross-view matching result is obtained, specifically: The third visual feature of the street view image and the fourth visual feature of the key frame of the unmanned aerial vehicle video are used to calculate the similarity between the third visual feature and the fourth visual feature as the direct matching score. The indirect matching score and the direct matching score are weighted and fused to obtain the final matching score.

10. A street scene and satellite image matching device comprising at least one processor and a memory, said memory storing a computer program, characterized in that, The computer program is executed by the at least one processor to realize the matching method of the street view image and the satellite image according to any one of claims 1 to 9.

Images