Feature extraction method and system based on difficult sample mining and multi-granularity division

Abstract

The application discloses a feature extraction method and system based on difficult sample mining and multi-granularity division, which is used for a cross-view geographical image retrieval task. The method first preprocesses cross-view street view images and satellite images, and uses a generative model to generate cross-view images, reducing the visual difference between different view images. Then a two-stage difficult sample mining model is constructed, including a sampling strategy based on geographical location and visual similarity, mining difficult negative samples in different ranges, and enhancing the inter-class discrimination ability. Then a multi-granularity feature division module is introduced, the image features are extracted through a ResNet50 backbone network, and the features are divided and fused according to different granularities to obtain rich and robust view-invariant feature representation. Finally, the satellite image to be retrieved is input into the trained feature extraction model, the features are extracted, and similarity matching is performed with the street view image library to obtain the cross-view retrieval result.

Description

Technical Field

[0001] This invention belongs to the field of cross-view image feature extraction and retrieval technology in computer vision, specifically involving a feature extraction method and system based on hard sample mining and multi-granularity partitioning. Background Technology

[0002] In recent years, with the rapid development of the internet and digital technologies, the number of images and videos people take and share in their daily lives has exploded. Image data can depict detailed information about a region's scene, and massive amounts of multimedia data contain rich geographic location information, demonstrating enormous application potential in many fields. For example, law enforcement agencies analyze crime scene images to determine the specific location of a crime, providing crucial clues for case solving; emergency management departments use images of emergency scenes to quickly determine the location of an incident, assisting in the formulation of efficient emergency strategies. However, not all images and videos carry clear geographic markers. Although some electronic devices (such as smartphones and digital cameras) can automatically record the coordinates of the shooting location using their built-in GPS, a large number of images still lack geographic coordinate information. Therefore, accurately determining the geographic location of an image has significant practical implications.

[0003] Image geolocation involves comparing and matching images lacking geotags with geotagged images in a database to infer the location of a target image, thus completing the localization task. Current image geolocation methods primarily rely on single-viewpoints, such as ground views (street scenes) or aerial views (drones, satellites), which are not conducive to extracting complete features of ground objects.

[0004] With the widespread use of mobile social media, street view capture vehicles, drones, and other tools, image data from different perspectives is becoming increasingly abundant. The task of geolocation using images from different perspectives is gradually being studied, providing an important solution to the problem of incomplete target features in a single perspective. In modern warfare, using satellite imagery to locate and analyze images or videos captured by drones and mobile phones to achieve precise strikes against targets is becoming an important auxiliary attack method in military operations. Therefore, research on cross-perspective geolocation based on image data from different perspectives has enormous potential application value.

[0005] Current image feature extraction methods based on triplet loss metric learning strategies extract viewpoint-invariant features. However, due to the small differences between positive and negative sample features, these methods suffer from insufficient feature discriminativeness and limited image representation information, resulting in poor model robustness and difficulty in effectively extracting inter-class features. Summary of the Invention

[0006] The purpose of this invention is to address the problems of insufficient inter-class feature extraction, single viewpoint-invariant features, and redundant feature information in images. By combining metric learning strategies, it provides a feature extraction method and system based on hard sample mining and multi-granularity partitioning. This method is used to capture positive and negative sample feature information of different ranges, enhance inter-class feature learning capabilities, and integrate contextual information to enhance viewpoint-invariant features, thereby improving the accuracy of image matching and localization.

[0007] According to one aspect of the present invention, a feature extraction method based on hard sample mining and multi-granularity partitioning is provided, comprising:

[0008] Obtain the satellite view image to be retrieved;

[0009] The satellite view image to be retrieved is input into the trained image feature extraction model, which outputs satellite image features; wherein, the training of the image feature extraction model includes:

[0010] Based on geolocation label and dynamic feature similarity learning, a two-stage hard sample mining model is constructed, and a two-branch Siamese network model based on multi-granularity image partitioning is constructed by combining image features of different granularities and aggregate pooling to capture local features and global information.

[0011] Generate cross-view images and input them into a two-stage hard sample mining model for hard sample mining. Then, input the mined hard sample images into a dual-branch Siamese network model based on multi-granularity image partitioning to obtain feature vectors of different granularities.

[0012] Construct a loss function to train the model and output the trained image feature extraction model.

[0013] As a further technical solution, the method also includes: searching and matching the output satellite image features with all street view image features to obtain the street view image closest to the target satellite view image.

[0014] As a further technical solution, the generation of the cross-view image includes: adjusting the resolution and size of the cross-view image pair while performing image flipping; and generating the cross-view image using a preset generation model.

[0015] As a further technical solution, the generated cross-view images are input into a two-stage hard sample mining model for hard sample mining, including: when the model performance is lower than expected, hard sample mining is performed based on geographical location; when the model performance reaches the expected level, hard sample mining is performed based on visual similarity.

[0016] As a further technical solution, the mined hard sample images are input into a two-branch Siamese network model based on multi-granularity image partitioning, including:

[0017] The mined hard sample images are input into the ResNet50 deformable network for feature extraction.

[0018] The extracted feature vectors are divided into multiple granularities to obtain feature vectors of different granularities.

[0019] According to one aspect of the present invention, a feature extraction system based on hard sample mining and multi-granularity partitioning is provided, comprising:

[0020] The input module is used to acquire satellite-view images to be retrieved;

[0021] The feature extraction module is used to input the satellite view image to be retrieved into the trained image feature extraction model and output satellite image features; wherein, the training of the image feature extraction model includes:

[0022] Based on geolocation label and dynamic feature similarity learning, a two-stage hard sample mining model is constructed, and a two-branch Siamese network model based on multi-granularity image partitioning is constructed by combining image features of different granularities and aggregate pooling to capture local features and global information.

[0023] Generate cross-view images and input them into a two-stage hard sample mining model for hard sample mining. Then, input the mined hard sample images into a dual-branch Siamese network model based on multi-granularity image partitioning to obtain feature vectors of different granularities.

[0024] Construct a loss function to train the model and output the trained image feature extraction model.

[0025] As a further technical solution, the image feature extraction model includes: a hard sample mining module, a multi-granularity partitioning module, and a loss function module; the hard sample mining module is used to mine hard samples based on geographical location or visual similarity; the multi-granularity partitioning module is used to extract features and perform multi-granularity partitioning on the mined hard samples; and the loss function module is used to jointly optimize the hard sample mining loss and the image feature extraction loss.

[0026] As a further technical solution, the system also includes a retrieval and matching module, which is used to retrieve and match the output satellite image features with all street view image features to obtain the street view image closest to the target satellite view image.

[0027] According to one aspect of the present invention, an electronic device is provided, comprising: at least one processor, at least one memory, and a communication interface; wherein the processor, memory, and communication interface communicate with each other; the memory stores program instructions that are executed by the processor, and the processor invokes the program instructions to perform the method described thereon.

[0028] According to one aspect of the present invention, a non-transitory computer read storage medium is provided, the non-transitory computer read storage medium storing computer instructions that cause the computer to perform the method described herein.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] 1. By introducing a hard sample mining strategy, negative samples that have the greatest effect on improving the model's discrimination ability are dynamically selected for training during the training process, which enhances the model's ability to distinguish similar negative samples and improves the discriminativeness and generalization of feature representation.

[0031] 2. A dual-branch Siamese network model based on multi-granularity image partitioning is proposed. By designing feature partitioning schemes of different granularities, coarse-grained, medium-grained, and fine-grained features are fused to obtain both global scene information and local detailed features. This multi-granularity feature fusion method can capture richer and more robust view-invariant feature representations.

[0032] 3. By jointly optimizing the hard sample mining loss and image feature extraction loss, the model's ability to distinguish hard samples and its ability to learn robust view-invariant features are balanced, enabling the network to better adapt to real-world, complex, and ever-changing cross-view image matching scenarios. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of a feature extraction method based on difficult sample mining and multi-granularity partitioning provided in an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram illustrating the workflow of the image feature extraction model based on difficult sample mining and multi-granularity partitioning provided in an embodiment of the present invention.

[0036] Figure 3 The network framework diagram of the image feature extraction model based on hard sample mining and multi-granularity partitioning provided in the embodiments of the present invention is shown.

[0037] Figure 4 This is a visualization of the image feature extraction model based on hard sample mining and multi-granularity partitioning provided in this embodiment of the invention, used for retrieval on the CVACT test set.

[0038] Figure 5 This is a schematic diagram of a feature extraction system based on hard sample mining and multi-granularity partitioning, provided as an embodiment of the present invention.

[0039] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0040] This invention provides a feature extraction method based on hard sample mining and multi-granularity segmentation. The method first preprocesses cross-view street view images and satellite images, and then uses a generative model to generate cross-view images, reducing visual differences between images from different perspectives. Next, a two-stage hard sample mining model is constructed, including sampling strategies based on geographic location and visual similarity to mine hard negative samples at different ranges, enhancing inter-class discrimination capabilities. Then, a multi-granularity feature segmentation module is introduced, extracting image features through a ResNet50 backbone network and performing feature segmentation and fusion at different granularities to obtain rich and robust view-invariant feature representations. Finally, the satellite image to be retrieved is input into the trained feature extraction model, extracting features and performing similarity matching with a street view image database to obtain cross-view retrieval results.

[0041] This invention addresses the problems of insufficient feature extraction between image classes, single feature extraction with unchanged viewpoint, and redundant feature information. By enhancing the distance between features of real samples and difficult samples, a feature extraction method is constructed to achieve accurate and efficient cross-viewpoint image localization.

[0042] It should be noted that:

[0043] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0044] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0047] This invention provides a feature extraction method based on hard sample mining and multi-granularity partitioning, such as... Figure 1 As shown, the process includes: acquiring the satellite view image to be retrieved; inputting the satellite view image to be retrieved into the trained image feature extraction model, and outputting satellite image features.

[0048] After obtaining the extracted image features, the method provided in this embodiment of the invention further includes: searching and matching the output satellite image features with all street view image features to obtain the street view image closest to the target satellite view image.

[0049] The training of the image feature extraction model provided in this embodiment of the invention includes:

[0050] Based on geolocation label and dynamic feature similarity learning, a two-stage hard sample mining model is constructed, and a two-branch Siamese network model based on multi-granularity image partitioning is constructed by combining image features of different granularities and aggregate pooling to capture local features and global information.

[0051] Generate cross-view images and input them into a two-stage hard sample mining model for hard sample mining. Then, input the mined hard sample images into a dual-branch Siamese network model based on multi-granularity image partitioning to obtain feature vectors of different granularities.

[0052] Construct a loss function to train the model and output the trained image feature extraction model.

[0053] Figure 2 This is a flowchart illustrating the workflow of an image feature extraction model based on hard sample mining and multi-granularity partitioning. In this embodiment of the invention, image feature extraction based on hard sample mining and multi-granularity partitioning includes the following steps:

[0054] Step 1: Select the public cross-view geographic image training dataset CVACT.

[0055] This invention details the operation and process on the CVACT dataset. CVACT is a large-scale cross-view image pairing dataset containing pairs of ground panoramic images and corresponding aerial images collected from different locations. The ground images in CVACT were taken using panoramic cameras on the roofs of Google Street View vehicles, achieving a resolution of 1664×832; the aerial images are from telephoto lenses mounted on aircraft, with a resolution of 256×256, but include four scaling levels (4X, 8X, 16X, 32X), which is beneficial for studying the impact of scale changes on the matching algorithm. This example uses a scaling level of 8X for the experiment. CVACT contains 128,334 image pairs, of which 35,532 pairs are used as the training set and 92,802 pairs are used as the test set. The dataset provides rich metadata such as GPS location and shooting angle for each image, and includes image pairs taken during the day and at night.

[0056] Step 2: Perform image preprocessing and cross-view image visual generation.

[0057] Specifically, it includes:

[0058] Step 2.1: Adjust the size of the cross-view image pair to 512×128, and then randomly flip the image. This helps the generation model learn view-invariant features and increases the generalization ability of the image feature extraction model based on hard sample mining and multi-granularity partitioning.

[0059] Step 2.2: Input satellite view image, use DDMP-GAN generation model to generate cross-view images, convert satellite view image into street view image, reduce the visual features between satellite image and street view image;

[0060] Step 3: Train an image feature extraction model based on hard sample mining and multi-granularity partitioning.

[0061] In this embodiment of the invention, the network framework based on the image feature extraction model of hard sample mining and multi-granularity partitioning is as follows: Figure 3 As shown.

[0062] Step 3.1: Mining difficult samples based on geographical location.

[0063] In the early stages, it is difficult to directly select difficult samples because the model has not yet fully adapted to the domain of the training data. To overcome this drawback, embodiments of the present invention employ a simple sampling strategy based on geographic location to initialize negative samples before training begins.

[0064] Specifically, the input consists of cross-view image pairs and their geographic coordinates, and hard sample mining is performed based on geographic location. Location information is represented in UTM coordinates and Euclidean distance is used. Determine the nearest neighbors. Among them, and This represents an n-dimensional point (n=2 in this embodiment of the invention). and These two points respectively represent the points at the th... Coordinates on the dimensional plane. Geographic location is only used in the early stages of training, and is gradually replaced by sampling strategies based on visual similarity as model performance improves.

[0065] Step 3.2: Mining difficult samples based on visual similarity.

[0066] In the later stages, considering the existence of images that are geographically distant but visually similar, this embodiment of the invention employs cosine similarity. This serves as a metric for calculating the visual distance between all samples. This represents the feature representation function extracted by the model, which maps the input samples to the feature space. For sampling subsequent batches, the visually closest values ​​are selected for each query image. K samples are used as a candidate set. These K candidate samples are ranked according to their similarity to the query image, and the top-ranked samples are selected. The most similar sample is selected, and then randomly selected from the remaining candidate samples. These samples, together, constitute the final difficult negative samples. This random selection process helps to increase the diversity of the selected negative samples.

[0067] Therefore, the negative sample candidate set is .in, Indicates the first In each cycle, query the image Difficult sample set, Table in the first Image query in each cycle All sample sets.

[0068] Step 3.3: Extract image features using a ResNet50 deformable network. The cross-view image pairs after hard sample mining are input into a ResNet50 deformable Siamese network, where features are extracted through multiple convolutional networks. The extracted feature vectors are then input into a multi-granularity feature partitioning module for further processing.

[0069] Step 3.4: Perform multi-granularity feature partitioning. Satellite image features and street view image features extracted by the backbone network are uniformly segmented vertically, and then partitioned into columns using a 1-partition, 4-partition, and 6-partition method. A fully connected layer is then applied to obtain feature vectors of different granularities. By fusing features of different granularities and utilizing global and local visual cues, redundancy and information loss in feature representations are avoided.

[0070] Step 3.5: Optimize the total loss function using gradient descent. ,in The balancing factor controls the relative importance of the two loss functions. By jointly optimizing these two loss functions, the network can simultaneously improve its ability to distinguish difficult samples and learn robust view-invariant feature representations.

[0071] Step 3.6: Repeat steps 3.1 to 3.5 to process the CVACT training dataset; stop training when the total loss function value no longer decreases, indicating that the image feature extraction model based on hard sample mining and multi-granularity partitioning has been trained. Save this model as the final detection image feature extraction model based on hard sample mining and multi-granularity partitioning.

[0072] Step 4: Cross-view image retrieval test.

[0073] Step 4.1: Input the satellite view image to be retrieved into the trained image feature extraction model, and extract satellite image features through hard sample mining and multi-granularity partitioning network.

[0074] Step 4.2: After the model extracts features, it searches and matches the features to be retrieved with all street view image features to obtain the street view image closest to the target satellite view image.

[0075] The test results on the CVACT test set are shown in Figure 1 and... Figure 4 As shown, recall is used to measure the effectiveness of a retrieval or identification algorithm. Recall This is used to evaluate the ability of a retrieval algorithm to find all matching target images in the entire dataset, given a query image. Specifically, for a given query image, the query is considered successful if the algorithm retrieves a result containing matching target images. (TruePositive) represents the number of target images that the algorithm correctly retrieves that match the query image; (False Negative) represents the number of target images that the algorithm failed to retrieve that match the query image. The value of Recall ranges from [0, 1], and the closer it is to 1, the more relevant images the retrieval algorithm can find.

[0076] Table 1 Comparison of different feature extraction methods on the CVACT dataset

[0077]

[0078] Table 1 records the recall rates of different image feature extraction methods on the CVACT dataset. Specifically, compared to local pattern-based feature extraction methods such as CVFT, RKNet, and LPNADDIN, our method improves the R@1 metric by 30.14%, 50.66%, and 11.2%, respectively. This is attributed to the organic combination of global and local features through a multi-granularity partitioning strategy, enhancing the richness and discriminative power of image feature representation. Compared to global pattern-based feature extraction methods such as TransGeo, our method improves the R@1 metric by 6.24%, further demonstrating the superiority of multi-granularity feature fusion.

[0079] Furthermore, although methods such as SAIG-D and SAFA utilize depth and semantic information, the method of this invention still improves the R@1 metric by 1.98% and 10.16% respectively without using additional information, highlighting the effectiveness of the method. Similar to Toker et al., this invention also utilizes cross-view image generation technology, but by introducing a hard sample mining strategy, it improves the R@1 metric by 7.91%, indicating that focusing on the weak links in the learning process can effectively improve the quality of feature learning. Although Sample4Geo employs a hard sample mining strategy and achieves a slightly higher R@5 metric than the method of this invention, it falls short in the R@1 metric. This is attributed to the fine characterization of local image details by the multi-granularity partitioning strategy. Compared to SAFA and DSM, which utilize polar coordinate transformation to preprocess cross-view images, this method improves the R@1 metric by 8.7% and 10.16% respectively, further validating the advantages of multi-granularity feature extraction in mitigating viewpoint differences.

[0080] In image retrieval tasks, retrieval efficiency is a crucial indicator for evaluating the practicality of a method. The method proposed in this invention demonstrates superior image retrieval efficiency. To comprehensively evaluate retrieval efficiency, this invention analyzes the image retrieval process in two stages: feature extraction and feature retrieval. In the feature extraction stage, the method of this invention extracted features from 31,980 street view images in a total time of 185.41 seconds, with an average feature extraction time of 5.8 milliseconds per image; for 31,980 generated images, the total feature extraction time was 175.65 seconds, with an average feature extraction time of 5.5 milliseconds per image. This shows that the method of this invention can complete large-scale image feature extraction in a short time, demonstrating highly efficient feature representation capabilities. In the image retrieval stage, the method retrieved 31,980 images in a total time of 144.46 seconds, with an average retrieval time of only 4.5 milliseconds per image. Therefore, the method of this invention can quickly find results related to the query image from a large number of images, meeting the needs of rapid retrieval.

[0081] The results from both the feature extraction and image retrieval stages demonstrate that the image feature extraction method proposed in this invention, based on hard sample mining and multi-granularity partitioning, has advantages in image retrieval efficiency. Both feature extraction and image retrieval can be completed in milliseconds, comparable to the time complexity of existing mainstream methods. This is due to the efficient mining of discriminative features through the hard sample mining strategy and the effective fusion of global and local features through multi-granularity partitioning. The efficient feature representation and fast retrieval speed make this method highly usable and practical in real-world applications.

[0082] The implementation of the various embodiments of the present invention is based on programmed processing by a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a feature extraction system based on hard sample mining and multi-granularity partitioning. This system is used to execute a feature extraction method based on hard sample mining and multi-granularity partitioning from the above method embodiments.

[0083] See Figure 5The system includes: an input module for acquiring satellite view images to be retrieved; and a feature extraction module for inputting the satellite view images to be retrieved into a trained image feature extraction model and outputting satellite image features. The training of the image feature extraction model includes: constructing a two-stage hard sample mining model based on geographic location labels and dynamic feature similarity learning; and constructing a dual-branch Siamese network model based on multi-granularity image partitioning by combining image features of different granularities with pooling to capture local features and global information; generating cross-view images and inputting the generated cross-view images into the two-stage hard sample mining model for hard sample mining; inputting the mined hard sample images into the dual-branch Siamese network model based on multi-granularity image partitioning to obtain feature vectors of different granularities; and constructing a loss function to jointly optimize the hard sample mining loss and the image feature extraction loss; and outputting the trained image feature extraction model.

[0084] The feature extraction system based on hard sample mining and multi-granularity partitioning provided in this invention addresses problems such as insufficient inter-class feature extraction, single feature extraction with invariant viewpoint, and redundant feature information. Figure 5 Several modules in the system capture feature information of positive and negative samples from different ranges to enhance inter-class feature learning capabilities. At the same time, they integrate contextual information to enhance viewpoint invariant features, thereby improving the accuracy of image matching and localization.

[0085] It should be noted that the system embodiments provided by this invention, in addition to implementing the methods in the above method embodiments, are also used to implement the methods in other method embodiments provided by this invention. The difference lies only in setting corresponding functional modules, and their principles are basically the same as those of the above system embodiments provided by this invention. As long as those skilled in the art, based on the above system embodiments and referring to the specific technical solutions in other method embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and improve the equipment in the above system embodiments while ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments. For example:

[0086] Based on the above system embodiments, as a preferred embodiment, the feature extraction system based on hard sample mining and multi-granularity partitioning provided in this embodiment of the invention further includes: a hard sample mining module, a multi-granularity partitioning module, and a loss function module; the hard sample mining module is used to mine hard samples based on geographical location or visual similarity; the multi-granularity partitioning module is used to extract features and perform multi-granularity partitioning on the mined hard samples; the loss function module is used to jointly optimize the hard sample mining loss and the image feature extraction loss.

[0087] Furthermore, the hard sample mining module mainly includes location-based sampling and visual similarity-based sampling. The location-based sampling strategy utilizes prior knowledge that spatial proximity increases visual similarity, providing the model with negative samples of varying ranges. The core idea of ​​this strategy is to use the GPS metadata of the image to select the other street view images with the closest geographical location as potential hard negative samples for each query street view image. Visual similarity-based sampling utilizes street view images whose image embedding features are very similar to the query street view image, capturing a large number of negative samples for the model. The core idea of ​​this strategy is to dynamically evaluate the visual similarity between the query image and candidate matching images, i.e., their proximity in the feature embedding space, and then select the images that are most similar to the query but not actually match as hard negative samples.

[0088] The multi-granularity partitioning module of the network mainly consists of a backbone network module and a multi-granularity feature partitioning module. The backbone network module uses ResNet50 as the feature extractor, making full use of its excellent performance in feature extraction. This invention adjusts the network structure of ResNet50. In the feature extraction stage, the fully connected layers in the ResNet50 network are removed, and the total downsampling factor of the network is 32, making the size of the output feature map equivalent to 1 / 32 of the original image. The adjusted ResNet50 network structure mainly consists of one convolutional module (Conv1) and four convolutional module groups with different stages (Conv2_x, Conv3_x, Conv4_x, and Conv5_x). In the initial Conv1, a convolutional layer with a kernel size of 7×7 and a stride of 2 is used for initial feature extraction with a 2x downsampling. Subsequently, the network sequentially passes through the four stages of convolutional modules, each module using a different number of residual blocks to downsample and transform the features. In the first stage, a max-pooling layer with a kernel size of 3×3, a stride of 2, and padding of 1 is used to further reduce the size of the feature map while maintaining the number of feature channels. Then, three residual blocks are used for further feature extraction. In the second, third, and fourth stages, a residual block with a stride of 2 is used for downsampling first, followed by a residual block with a stride of 1 for further convolution. The multi-granularity feature partitioning module designs three different granularity feature partitioning schemes: 1X, 4X, and 6X. Starting from coarse-grained (1X), medium-grained (4X), and fine-grained (6X) granularities, features of different granularities are fused. The feature map extracted by the backbone network is vertically uniformly divided into column partitions according to a 1-partition, 4-partition, and 6-partition method, followed by a fully connected layer to obtain feature vectors of different granularities. During the partitioning process, the original image is not divided at the pixel level; instead, the extracted feature map is segmented. This approach not only improves efficiency but also achieves a larger receptive field and partial alignment.

[0089] Image feature extraction networks based on hard sample mining and multi-granularity partitioning improve the model's discriminative and generalization abilities through hard sample mining loss and image feature extraction loss. The Hardest-in-Batch Triplet (HiBT) loss is a triplet-based loss function designed to enhance the model's ability to discriminate against difficult negative samples. This loss function focuses on the distance between the query image and the positive sample reference image, as well as the distance between the query image and the most difficult negative sample reference image. Given that hard sample mining includes both geolocation-based sampling and visual similarity-based sampling, the HiBT loss is:

[0090] ,in Indicates a query for an image. This represents a positive sample reference image. The most challenging negative sample reference image. It is a predefined interval parameter. It is a weighting parameter that balances geographical distance and visual similarity, and its value ranges from [0, 1]. Indicates query image and reference image Geographical distance between them Indicates query image and reference image The visual similarity distance between the images. Image feature extraction loss (symmetric InfoNCE loss) utilizes all available hard samples by employing contrastive learning for a query image. Sample a positive sample from the reference image set. Table and negative samples Then, the query image is calculated. The similarity with all reference images in the coding space is calculated by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. To achieve this. Among them, This represents the encoded generated image. This represents a set of encoded street view images. Representation and query image Matched positive sample reference image, It's a temperature parameter. Represents a set of reference images The size of the query image is considered. The InfoNCE loss is lower when the query image and the positive sample reference image are similar; the InfoNCE loss is higher when the query image and the negative sample reference image are dissimilar.

[0091] Based on the above system embodiments, as a preferred embodiment, the feature extraction system based on hard sample mining and multi-granularity partitioning provided in this embodiment of the invention further includes: a retrieval and matching module, used to retrieve and match the output satellite image features with all street view image features to obtain the street view image closest to the target satellite view image.

[0092] Based on the above system embodiments, as a preferred embodiment, the feature extraction system based on hard sample mining and multi-granularity partitioning provided in this embodiment of the invention further includes: a cross-view image generation module, used to adjust the resolution and size of cross-view image pairs, perform image flipping, and generate cross-view images using a preset generation model.

[0093] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, embodiments of the present invention provide an electronic device, such as... Figure 6 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein the at least one processor, the communication interface, and the at least one memory communicate with each other via the communication bus. The at least one processor invokes logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0094] Furthermore, when the logical instructions in at least one of the aforementioned memories are implemented as software functional units and sold or used as independent products, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, is embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks—various media for storing program code.

[0095] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place, or distributed across multiple network units. The purpose of this embodiment is achieved by selecting some or all of the modules according to actual needs. Those skilled in the art will understand and implement this without any inventive effort.

[0096] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0097] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0098] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0099] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0100] In summary, the above examples are as follows:

[0101] The proposed image feature extraction method based on hard sample mining and multi-granularity partitioning achieves excellent performance in cross-view image retrieval tasks. By mining hard samples, the model can focus on challenging samples, improving its ability to discriminate complex scenes. The multi-granularity feature partitioning strategy extracts local and global features from images at different scales, enhancing the richness and robustness of feature representation. Experimental results demonstrate that the proposed method significantly outperforms traditional methods and other deep learning-based methods in terms of retrieval accuracy and recall, exhibiting superior performance advantages.

[0102] The method of this invention also has broad prospects for practical applications. In the field of autonomous driving, retrieving corresponding street view images from satellite imagery can provide more comprehensive and accurate environmental information for autonomous driving systems, assisting vehicles in positioning, navigation, and decision-making. In the fields of virtual reality and augmented reality, using the method of this invention to achieve accurate matching of satellite images to street view images can construct realistic virtual scenes, providing an immersive experience. Furthermore, cross-view image retrieval technology also plays an important role in fields such as urban planning, disaster assessment, and military reconnaissance, providing reliable image evidence for relevant decision-making.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A feature extraction method based on hard sample mining and multi-granularity partitioning, characterized in that, include: Obtain the satellite view image to be retrieved; The satellite view image to be retrieved is input into the trained image feature extraction model, which outputs satellite image features; wherein, the training of the image feature extraction model includes: Based on geolocation label and dynamic feature similarity learning, a two-stage hard sample mining model is constructed, and a two-branch Siamese network model based on multi-granularity image partitioning is constructed by combining image features of different granularities and aggregate pooling to capture local features and global information. A cross-viewpoint image is generated and input into a two-stage hard sample mining model for hard sample mining. The mined hard sample images are then input into a dual-branch Siamese network model based on multi-granularity image partitioning to obtain feature vectors of different granularities. The input of the generated cross-viewpoint image into the two-stage hard sample mining model for hard sample mining includes: when the model performance is lower than expected, hard sample mining is performed based on geographical location; when the model performance meets expectations, hard sample mining is performed based on visual similarity. Construct a loss function to train the model and output the trained image feature extraction model.

2. The feature extraction method based on hard sample mining and multi-granularity partitioning according to claim 1, characterized in that, The method further includes: searching and matching the output satellite image features with all street view image features to obtain the street view image closest to the target satellite view image.

3. The feature extraction method based on hard sample mining and multi-granularity partitioning according to claim 1, characterized in that, The generation of the cross-view image includes: adjusting the resolution and size of the cross-view image pair while flipping the image; and generating the cross-view image using a preset generation model.

4. The feature extraction method based on hard sample mining and multi-granularity partitioning according to claim 1, characterized in that, The mined hard sample images are input into a two-branch Siamese network model based on multi-granularity image partitioning, including: The mined hard sample images are input into the ResNet50 deformable network for feature extraction. The extracted feature vectors are divided into multiple granularities to obtain feature vectors of different granularities.

5. A feature extraction system based on hard sample mining and multi-granularity partitioning, characterized in that, include: The input module is used to acquire satellite-view images to be retrieved; The feature extraction module is used to input the satellite view image to be retrieved into the trained image feature extraction model and output satellite image features; wherein, the training of the image feature extraction model includes: Based on geolocation label and dynamic feature similarity learning, a two-stage hard sample mining model is constructed, and a two-branch Siamese network model based on multi-granularity image partitioning is constructed by combining image features of different granularities and aggregate pooling to capture local features and global information. A cross-viewpoint image is generated and input into a two-stage hard sample mining model for hard sample mining. The mined hard sample images are then input into a dual-branch Siamese network model based on multi-granularity image partitioning to obtain feature vectors of different granularities. The input of the generated cross-viewpoint image into the two-stage hard sample mining model for hard sample mining includes: when the model performance is lower than expected, hard sample mining is performed based on geographical location; when the model performance meets expectations, hard sample mining is performed based on visual similarity. Construct a loss function to train the model and output the trained image feature extraction model.

6. The feature extraction system based on hard sample mining and multi-granularity partitioning according to claim 5, characterized in that, The image feature extraction model includes a hard sample mining module, a multi-granularity partitioning module, and a loss function module. The hard sample mining module is used to mine hard samples based on geographical location or visual similarity. The multi-granularity partitioning module is used to extract features and perform multi-granularity partitioning on the mined hard samples. The loss function module is used to jointly optimize the hard sample mining loss and the image feature extraction loss.

7. The feature extraction system based on hard sample mining and multi-granularity partitioning according to claim 5, characterized in that, The system also includes a retrieval and matching module, which is used to retrieve and match the output satellite image features with all street view image features to obtain the street view image closest to the target satellite view image.

8. An electronic device, characterized in that, include: The system includes at least one processor, at least one memory, and a communication interface; wherein the processor, memory, and communication interface communicate with each other. The memory stores program instructions that are executed by the processor, which invokes the program instructions to perform the method described in any one of claims 1 to 4.

9. A non-transitory computer read storage medium, characterized in that, The non-transitory computer read storage medium stores computer instructions, which cause the computer to perform the method described in any one of claims 1 to 4.

Images