Tri-branch ternary quantization cross-view image retrieval method and device

Abstract

The application discloses a three-branch ternary quantization cross-view image retrieval method and device, which comprises the following steps: inputting the spatial patch token features output by a basic visual coding network into a local feature branch to extract key local features of an image; inputting the spatial patch features into a structure feature branch, which is used for modeling the spatial relationship and layout structure between different regions of the image to obtain spatial structure features; inputting the overall semantic features, the key local features and the spatial structure features into a feature fusion module, which uniformly maps and fuses the three types of features to obtain a unified fused image feature representation; when the retrieval mode is ternary quantization retrieval, inputting the obtained unified fused image feature representation into a ternary quantization module to generate ternary codes; calculating the matching scores between the ternary codes in a query image and the ternary codes of gallery images; and performing descending order sorting on the gallery images according to the similarity between the unified fused image features or the ternary code matching scores to output retrieval results.

Description

Technical Field

[0001] This invention relates to the field of cross-view geolocation and image retrieval, and in particular to a three-branch ternary quantization cross-view image retrieval method and apparatus, which can be used for geographic target retrieval and similar target matching between UAV view images and satellite view images. Background Technology

[0002] With the development of UAV remote sensing, satellite remote sensing, smart cities, and geographic information systems, the scale of geographic image data collected by different platforms is constantly expanding. UAV-view images typically feature clear target details, rich local textures, and flexible shooting angles, while satellite-view images typically feature wide coverage, stable top-down structure, and complete spatial layout. Since the two types of images differ significantly in imaging height, shooting angle, scale range, target morphology, background distribution, and spatial structure, establishing a stable target correspondence between UAV and satellite images is a crucial technical challenge in cross-view geographic target retrieval.

[0003] Existing cross-view image retrieval methods typically use single depth features or global features to represent image content and perform target matching through feature similarity calculation. However, in scenarios where there are significant differences between UAV and satellite perspectives, relying solely on a single global feature can easily overlook local salient regions and spatial structure information, leading to mismatches between similar buildings, roads, or scenes. Furthermore, traditional methods have limited adaptability to changes in local regions, target occlusion, rotation, and scale, making it difficult to fully describe the multi-layered visual features of cross-view geographic targets.

[0004] Furthermore, in large-scale image library retrieval scenarios, continuous deep features typically have high dimensionality, leading to issues such as large feature storage space requirements, high similarity calculation costs, and limited retrieval response speed. Especially in applications requiring rapid return of Top-K search results, using only high-dimensional continuous features for matching makes it difficult to balance retrieval accuracy and efficiency.

[0005] Therefore, it is necessary to propose a cross-view image retrieval method that can comprehensively utilize global semantic information, local salient information, and spatial structure information, and support compact feature representation, so as to improve the matching stability and retrieval efficiency between UAV images and satellite images. Summary of the Invention

[0006] This invention provides a three-branch ternary quantization cross-view image retrieval method and apparatus. It extracts the overall semantic features, key local features, and spatial structural features of an image through global feature branches, local feature branches, and structural feature branches, respectively. These three types of features are then uniformly mapped and fused to obtain a unified fused image feature representation for cross-view retrieval. This unified fused image feature representation is a continuous feature vector that can be directly used for fused feature retrieval. Furthermore, this invention introduces a ternary quantization retrieval mechanism, mapping this unified fused image feature representation to a ternary code composed of negative, zero, and positive values. This reduces feature storage and matching computation overhead, enabling efficient bidirectional retrieval between UAV-view images and satellite-view images. See the description below for details.

[0007] A first aspect is a three-branch, ternary quantization, cross-view image retrieval method, the method comprising:

[0008] The global labeled features output by the basic visual coding network are input into the global feature branch to extract the overall semantic features of the image;

[0009] The spatial patch token features output by the basic visual coding network are input into the local feature branch to extract key local features of the image;

[0010] The spatial patch features are input into the structural feature branch, which is used to model the spatial relationships and layout structure between different regions of the image to obtain spatial structural features.

[0011] The overall semantic features, key local features, and spatial structure features are input into the feature fusion module. The feature fusion module performs unified mapping and fusion of the three types of features to obtain a unified fused image feature representation.

[0012] When the retrieval mode is ternary quantization retrieval, the obtained unified fused image feature representation is input into the ternary quantization module to generate ternary codes; the matching score between the ternary codes in the query image and the ternary codes of the images in the library is calculated.

[0013] The images in the image library are sorted in descending order based on the similarity between unified fused image features or the three-valued encoding matching score, and the search results are output.

[0014] The encoding matching score is represented as follows:

[0015]

[0016] in, Indicates the encoding dimension. and These represent the query image and the gallery image at the [number]th [position]. Three-value encoding in each dimension.

[0017] Specifically, the overall semantic features, key local features, and spatial structure features are input into the feature fusion module. The feature fusion module performs unified mapping and fusion of the three types of features to obtain a unified fused image feature representation as follows:

[0018] ;

[0019] ;

[0020] ;

[0021] in, , , Let represent the mapping functions for the global feature branch, local feature branch, and structural feature branch, respectively. This indicates a normalization operation; For global semantic features, It is a locally significant feature. It is a spatial structural feature;

[0022] Subsequently, the final unified fused image feature representation is obtained through the fusion function:

[0023] ;

[0024] in, Employing splicing mapping, weighted summation, attention fusion, or residual fusion methods.

[0025] The ternary quantization module first generates intermediate features through linear projection:

[0026] ;

[0027] in, Represents the projection matrix. Indicates the bias term. Represents continuous features after mapping;

[0028] Convert continuous features into three-value codes based on a threshold:

[0029] ;

[0030] in, Represents a three-valued quantization function. Indicates the encoding dimension. This represents the ternary encoded vector corresponding to the input image, and In the ternary encoding, -1, 0, and +1 represent the negative, neutral, and positive responses on the corresponding feature dimensions, respectively.

[0031] The basic coding features output by the basic visual coding network are:

[0032] For any input image The basic encoded features output by the basic visual coding network are represented as follows: ,in, Represents the basic visual coding network. Represents global labeling features. The feature matrix representing the patchtoken representation space;

[0033] The spatial patch token feature matrix is ​​represented as follows: ;in, Indicates the first The feature vector corresponding to each patch Indicates the number of patches. Indicates the feature channel dimension; when the input image size is The patch area size is and and All were When divisible, the number of patches is expressed as: .

[0034] The method further includes: introducing branch decorrelation constraints.

[0035] For any input image, the decorrelation term among the three normalized branch features is represented as: ;in, This indicates that the branch removes the relevant constraint terms. This represents the normalized global branch features. This represents the local branch features after normalization. Represents the normalized structural branch features; for those containing The training batch of samples, the branch decorrelation constraint is represented as: ,in, Indicates the number of samples in the batch. Indicates the first Remove the relevant constraint terms from the branches corresponding to each sample.

[0036] The method further includes: using normalized ternary encoding similarity:

[0037] For query images and the Image library Its three-value codes are represented as follows: and Then the normalized three-valued encoding similarity is expressed as: ,in, Indicates query image With the Image library Normalized ternary code similarity between them This represents a very small constant used to avoid the denominator being zero;

[0038] The images in the image library are sorted in descending order based on the ternary coding matching score or the normalized ternary coding similarity to obtain the Top-K search results: ,in, This represents the set of Top-K search results under the ternary quantization search mode. Indicates the number of images in the gallery. This indicates the number of search results returned. This indicates that the top results are selected based on similarity scores from highest to lowest. Operations on each result.

[0039] A second aspect is a three-branch ternary quantization cross-view image retrieval device, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the method described in any one of the first aspects.

[0040] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0041] The beneficial effects of the technical solution provided by this invention are:

[0042] 1. This invention proposes a three-branch feature extraction mechanism for UAV-view images and satellite-view images: by extracting the overall semantic information, key local information and spatial structural information of the image through global feature branch, local feature branch and structural feature branch respectively, it can alleviate the problem of insufficient representation of local salient information and spatial structural information due to relying on only a single global feature, and improve the stability of cross-view image matching;

[0043] 2. This invention constructs a fusion feature retrieval mechanism: by fusing global, local and structural features into a unified image representation, it can simultaneously utilize the overall scene of the image, local salient targets and spatial layout information, thereby enhancing the feature consistency between UAV perspective images and satellite perspective images;

[0044] 3. This invention introduces a ternary quantization retrieval mechanism: by mapping the unified fusion image feature representation to a ternary code composed of negative, zero, and positive values, it can reduce feature storage and matching computation overhead while preserving the image feature discrimination capability, making it suitable for large-scale image library retrieval and lightweight retrieval scenarios;

[0045] 4. This invention supports bidirectional cross-view retrieval: This invention supports both Drone→Satellite retrieval using UAV images as query images and satellite images as library images, and Satellite→Drone retrieval using satellite images as query images and UAV images as library images, making it suitable for various cross-view geographic target retrieval tasks;

[0046] 5. This invention has strong software system integration capabilities: This invention can output query images, Top-K (the top K results obtained by sorting by matching score) search results, similarity, target number, image library path and running status information, which can be easily integrated into cross-view geolocation systems, remote sensing image retrieval platforms, UAV inspection systems and smart city management systems. Attached Figure Description

[0047] Figure 1 A flowchart for a three-branch, ternary quantization, cross-view image retrieval method;

[0048] Figure 2 A schematic diagram of the structure of a three-branch cross-view image feature extraction network;

[0049] Figure 3 A schematic diagram of the global feature branch, local feature branch, and structural feature branch;

[0050] Figure 4 A schematic diagram of a dual-mode retrieval process that integrates feature retrieval and ternary quantization retrieval;

[0051] Figure 5 This is a schematic diagram of the module structure of a cross-view image retrieval device;

[0052] Figure 6 A schematic diagram showing the results of retrieving satellite view images from UAV view images;

[0053] Figure 7 This is a schematic diagram showing the results of retrieving images from a UAV perspective from satellite-view images. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0055] Example 1

[0056] See Figures 1 to 4 This invention provides a cross-view image retrieval method for UAV-view images and satellite-view images. The method includes steps such as image data acquisition, image preprocessing, basic feature extraction, three-branch feature extraction, feature fusion, ternary quantization, retrieval ranking, and result output, as detailed below:

[0057] 101: Obtain the image set from the perspective of the UAV and the image set from the perspective of the satellite, and determine the query image set and the image library set in the cross-view search according to the preset search direction;

[0058] Among them, the UAV perspective image set is used to represent geographic target images acquired from low altitude or oblique perspective, and the satellite perspective image set is used to represent geographic target images acquired from high altitude downward perspective.

[0059] In this embodiment of the invention, the cross-view image data includes a query image set and a gallery image set. When the search direction is from a drone perspective to a satellite perspective, the drone image is used as the query image and the satellite image is used as the gallery image; when the search direction is from a satellite perspective to a drone perspective, the satellite image is used as the query image and the drone image is used as the gallery image.

[0060] 102: Perform unified preprocessing on query images and library images. Preprocessing operations include: image reading, size scaling, color format conversion, pixel normalization, and tensor dimension conversion.

[0061] In one specific implementation, the query image and the library image are used as input images, and the input images are scaled to have a uniform size, such as 518×518 or other sizes that are suitable for visual feature extraction networks; the pixel values ​​are normalized to the numerical range required by the model; and the image is converted from the storage format of height, width, and channels into the tensor format of channels, height, and width so that it can be input into the deep learning model for feature extraction.

[0062] 103: Input the preprocessed image into the basic visual coding network to obtain the basic coding features of the image;

[0063] The basic visual coding network can be a visual Transformer network, a visual Transformer network with a register token, a convolutional neural network, or other deep image coding networks. The structures of these networks are well known in the art, and will not be described in detail in this embodiment of the invention.

[0064] In one specific implementation, for any input image The basic encoded features output by the basic visual coding network can be represented as: ,in, Represents the basic visual coding network. Represents global labeling features. This represents the spatial patch token feature matrix. The spatial patch token feature matrix can be further represented as: ,in, express The feature vector corresponding to each patch Indicates the number of patches. Indicates the feature channel dimension. When the input image size is... The patch area size is and and All can be When divisible, the number of patches It can be represented as: .

[0065] As shown above, the basic visual coding network can simultaneously output global marker features related to the overall semantics of the image and spatial patch token features related to local regions of the image, providing input for subsequent global feature branches, local feature branches, and structural feature branches.

[0066] 104: Extract the overall semantic features of the image based on the basic coding features output by the basic visual coding network;

[0067] Specifically, the global label features output by the basic visual coding network are input into the global feature branch; or, the spatial patch token features output by the basic visual coding network are input into the global feature branch, and the spatial patch token features are globally aggregated in the global feature branch. The global feature branch extracts the overall semantic features of the image through methods such as global pooling, multi-scale aggregation, channel weight adjustment, or fully connected mapping.

[0068] Among them, the overall semantic features are used to characterize the overall scene semantics and target category-related information of the image, such as building distribution, road layout, regional environment, and overall geographic target features. Through these overall semantic features, the model can obtain semantic expression capabilities at the whole image level.

[0069] 105: Extract key local features from the image based on the spatial patch token features output by the basic visual coding network;

[0070] Specifically, the spatial patch token features output by the basic visual encoding network are input into the local feature branch. The local feature branch selects local regions from the spatial patch token features that have a discriminative effect on cross-view matching based on attention response, saliency score or region selection mechanism, and performs local aggregation on the selected local region features to obtain key local features.

[0071] In one specific implementation, the local feature branch can extract key local features using methods such as attention pooling, Top-K salient region pooling, or max pooling. These key local features are used to characterize local regions with stable correspondences in UAV and satellite images, such as building outlines, road intersections, green areas, water boundaries, or other local geographic structures.

[0072] 106: Input the spatial patch features into the structural feature branch. The structural feature branch is used to model the spatial relationships and layout structure between different regions of the image to obtain spatial structural features.

[0073] In one specific implementation, the structural feature branch can employ position-aware pooling, spatial pyramid pooling, or region grid partitioning to divide image features into multiple spatial regions. Features from different regions are then aggregated to obtain spatial structural features that reflect spatial distribution relationships. These spatial structural features enhance the model's adaptability to changes in regional layout across different viewpoints.

[0074] 107: Input the overall semantic features, key local features and spatial structure features into the feature fusion module. The feature fusion module performs unified mapping and fusion of the three types of features to obtain a unified fused image feature representation.

[0075] In one specific implementation, the feature fusion module can first perform dimensional unification and normalization on the three types of branch features, and then generate unified fused image features through feature-level attention, channel-level attention, concatenation mapping, or residual mapping. The unified fused image features simultaneously contain: overall semantics, local details, and spatial structure information, which can be used for subsequent cross-view image retrieval.

[0076] In a preferred implementation, the feature fusion module first maps and normalizes the overall semantic features, key local features, and spatial structural features respectively:

[0077]

[0078]

[0079] ,

[0080] in, , and These represent the mapping functions corresponding to the global feature branch, local feature branch, and structural feature branch, respectively. This indicates a normalization operation. Indicates overall semantic features. Indicates key local features, This represents the spatial structure features. Subsequently, a unified fused image feature representation is generated using an element-wise summation method:

[0081]

[0082] in, This represents a unified fusion of image features. This fusion method does not require an additional learnable fusion layer and can simultaneously integrate overall semantic information, key local information, and spatial structural information within a unified feature space.

[0083] 108: When the retrieval mode is unified fusion image feature representation retrieval, the unified fusion image feature representations corresponding to the query image and the image library image are obtained respectively, and the similarity between the unified fusion image feature representation of the query image and the unified fusion image feature representation of the image library image is calculated.

[0084] In one specific implementation, cosine similarity can be used, and its expression is:

[0085]

[0086] in, This represents a unified fused image feature representation of the query image. A unified fused image feature representation representing images from a gallery. This represents the similarity score between the query image and images in the image library.

[0087] 109: When the retrieval mode is ternary quantization retrieval, the unified fusion image feature representation obtained in step 107 is input into the ternary quantization module to generate ternary encoding;

[0088] The ternary code takes values ​​of (-1), (0), and (+1), which represent negative, neutral, and positive responses in the feature dimension, respectively.

[0089] In one specific implementation, the unified fusion of image features is represented as a continuous feature vector. The ternary quantization module can convert this vector into a ternary code consisting of negative, zero, and positive values ​​through linear mapping, normalization, thresholding, and pass-through estimation. For the i-th feature dimension, the quantization process can be represented as:

[0090]

[0091] in, The unified fused image feature representation, after linear mapping, represents the image at the th... Real-valued feature response values ​​in each dimension Indicates the first The quantization thresholds corresponding to each dimension This indicates the generated ternary code.

[0092] In a preferred implementation, the ternary quantization module first performs linear mapping and normalization on the unified fused image feature representation: ,in, Represents the quantization projection matrix. Indicates the quantization bias term. This indicates normalization processing. This represents the continuous features after mapping. Furthermore, a dimension-wise scaling parameter and a temperature parameter are introduced to scale the continuous features: ,in, Represents the dimension-wise scaling parameter. This represents element-wise multiplication. Indicates temperature parameter, This represents the scaled continuous response characteristics. For the... Each dimension, the three-value code can be based on a threshold. generate:

[0093] ;

[0094] in, express In the Response values ​​in each dimension Indicates the first The quantization thresholds corresponding to each dimension Indicates the first The ternary encoding is obtained in each dimension. This yields the ternary encoded vector corresponding to the input image: .in, This indicates the dimension of the ternary encoding.

[0095] 110: After generating the ternary codes of the query image and the library image, calculate the matching score between the ternary code in the query image and the ternary code of the library image. The matching score can be calculated based on the number of identical dimensions in the codes, the inner product of the codes, or the code distance.

[0096] In one specific implementation, the ternary encoding of the query image is: The ternary encoding of the images in the image library is Then the encoding matching score can be expressed as:

[0097]

[0098] in, Indicates the encoding dimension. and These represent the query image and the gallery image at the [number]th [position]. Three-value encoding in each dimension.

[0099] 111: Sort the images in the image library in descending order based on the similarity or three-valued encoding matching score between the unified fusion image features, select the top K images as the Top-K search results, and output the search results.

[0100] The search results include: query image, result image, sort number, similarity or match score, target number, image file name in the library, and image path in the library. The search results can be visualized through the software interface or saved as a structured file for subsequent analysis.

[0101] In summary, the embodiments of the present invention, through steps 101-111 above, extract the overall semantic information, key local information, and spatial structural information of the image through global feature branches, local feature branches, and structural feature branches respectively. This can alleviate the problem of insufficient adaptability of single feature representation to cross-view differences and improve the stability of cross-view image matching.

[0102] Example 2

[0103] The following is combined with Figures 2 to 4 The three-branch feature extraction mechanism, feature fusion mechanism, and three-value quantization retrieval mechanism in Example 1 will be further explained.

[0104] 201: Basic Feature Extraction;

[0105] In this embodiment of the invention, the input image, after standardization preprocessing, is fed into a basic visual coding network. The basic visual coding network outputs global features and spatial features. The spatial features consist of multiple patch tokens or local region features, capable of representing visual information at different spatial locations within the image.

[0106] In one specific implementation, the input image size is... ,in Indicates the height of the input image. This represents the width of the input image. If the input image is divided into dimensions... By dividing the patch region, we can obtain... Each patch region is processed by a basic visual encoding network to obtain a spatial patch token feature matrix. ,in, Indicates the number of patches. Indicates the first The feature vector corresponding to each patch.

[0107] In one specific implementation, the basic visual coding network can be represented as: For the input image The basic visual encoding network outputs globally labeled features. Spatial patch token feature matrix , can be represented as: The global label features satisfy: The spatial patch token feature matrix is ​​represented as follows: ,in, Indicates by A spatial feature matrix composed of patch tokens, Indicates the first The feature vector corresponding to each patch Indicates the feature channel dimension. If the input image size is... The patch area size is and and All can be Divisible by the number of patches It can be represented as: The above global marker features Used for inputting the global feature branch, spatial patch token feature matrix. Used for inputting local feature branches and structural feature branches.

[0108] 202: Global feature branch;

[0109] The global feature branch is used to obtain an overall representation of the image, thereby extracting its overall semantic features. This global feature branch can map the global labeled features output by the basic visual coding network, or it can perform global pooling on spatial patch features.

[0110] In one specific implementation, the global feature branch generates global features through multi-scale generalized average pooling or global average pooling:

[0111]

[0112] in, This represents a global aggregate function. Represents global semantic features.

[0113] Furthermore, in a preferred implementation, the global feature branch can simultaneously utilize global labeled features. Spatial patch token feature matrix To enhance the global modeling capability for regions with different response intensities, multi-scale generalized average pooling is first applied to the patchtoken features. For the ... Features of each patch token First, perform nonnegation: ,in, This is a very small constant used to avoid numerical instability. For pooling parameters... The result of generalized average pooling can be expressed as: .in, Indicates parameters The corresponding global pooling features. Selecting multiple different... After setting the value, multi-scale global response features can be obtained: ,in, This indicates a feature concatenation operation. This indicates the number of pooling scales. Subsequently, the multi-scale global response features are mapped to the same dimensions as the global labeled features: .in, This represents the global response mapping function. To adaptively fuse global labeled features and multi-scale global response features, channel-gated weights are introduced: ,in, Indicates feature splicing, Represents the gate weight matrix. Indicates the bias term. This represents the Sigmoid function. This represents the channel gating weights. The final overall semantic features are obtained as follows: in, This represents element-wise multiplication. Through this method, the global feature branch can simultaneously utilize image-level global labeled features and the multi-scale global response of the patchtoken, thereby enhancing the overall semantic expressive power.

[0114] 203: Local feature branches;

[0115] The local feature branch is used to enhance the model's ability to represent key regions. This branch first generates saliency scores based on spatial patch features, and then selects local regions with high responses for aggregation to obtain key local features.

[0116] In one specific implementation, the local feature branch calculates the saliency weight of each patch and selects the top K salient regions for feature aggregation:

[0117]

[0118] in, This indicates a significant region selection operation. This represents a local feature aggregation function. Indicates local salient features.

[0119] Furthermore, in a preferred implementation, the local feature branches include an attention pooling path, a Top-K salient region pooling path, and a max pooling path.

[0120] First, the attention pooling path is used to adaptively select local regions based on the content of the input image. For the first... For each patch token, the attention score can be represented as: ,in, Represents the attention weight vector. This represents the attention bias term. The corresponding normalized attention weights are: Attention pooling features are represented as follows: .

[0121] Second, the Top-K salient region pooling path is used to select local regions with high responses. The response strength of a patch token can be expressed as: Select based on response strength Given a set of patch tokens, we obtain an index set: The corresponding Top-K salient region pooling features are: .

[0122] Third, the max pooling path is used to obtain the strongest local response on each channel: ,in, The operation is performed along the patch token dimension.

[0123] Subsequently, the outputs of the three local paths are weighted and fused. Let the fusion weights of the three paths be respectively... , and but: .in, Indicates the first The fusion parameters corresponding to each local path.

[0124] The final key local features are represented as follows: .

[0125] In this way, the local feature branch can simultaneously utilize adaptive attention response, high-response local regions, and maximum response features, thereby enhancing the model's ability to express key local regions.

[0126] This local feature branch can reduce the interference of irrelevant background regions on the retrieval results, making the model pay more attention to key regions with stable correspondences in cross-view images.

[0127] 204: Structural feature branches;

[0128] The structural feature branch is used to extract spatial layout information of the image. Based on the spatial location of the patch, this branch divides the spatial patch token features into multiple spatial regions and performs feature aggregation on each region to model the spatial relationships and layout structure between different regions of the image.

[0129] In one specific implementation, the structural feature branch employs spatial pyramid pooling to divide the feature map into... , , The spatial structure features are obtained by aggregating and stitching together features from regions of multiple scales:

[0130]

[0131] in, This represents the spatial pyramid pooling operation. It represents the characteristics of spatial structure.

[0132] Furthermore, in a preferred implementation, the structural feature branch first converts the spatial patch token feature matrix... Rearrange them according to their two-dimensional spatial positions to form feature maps: ,in, and These represent the number of patch tokens in the height and width directions, respectively. This represents the feature channel dimension. The structural feature branches include location-aware pooling paths and spatial pyramid pooling paths.

[0133] First, position-aware pooling paths are used to model the contribution of different spatial locations to structural features. Let the first... The location code corresponding to each spatial location is: The position response score can then be expressed as: ,in, Represents the position response weight vector. Indicates the bias term. This represents the concatenation of patch token features and positional codes. The corresponding positional weights are: Position-aware pooling features are represented as follows: .

[0134] Second, spatial pyramid pooling paths are used to extract multi-scale spatial layout information. This is applied to two-dimensional spatial feature maps. Perform multi-scale region pooling: ,in, , and These represent adaptive average pooling operations at different scales. The concatenated spatial pyramid features are mapped to a unified dimension: ,in, Represents the spatial pyramid mapping matrix. This indicates the bias term.

[0135] Finally, the location-aware pooling features and the spatial pyramid pooling features are fused: . ,in, and These represent the fusion weights of the two structural paths, This represents the fusion parameters for the corresponding path. Using the above method, structural feature branches can simultaneously model location-sensitive spatial responses and multi-scale spatial layout information.

[0136] This spatial structure feature can describe the spatial distribution relationship between areas such as buildings, roads, water bodies, and green spaces in an image, which is beneficial to improving the structural consistency of cross-view image matching.

[0137] 205: Three-branch feature fusion;

[0138] The overall semantic features, key local features, and spatial structure features are input into the feature fusion module. These three types of features are first mapped to a unified dimension, and then normalized.

[0139] ,

[0140] ,

[0141] ,

[0142] in, , , Let represent the mapping functions for the global feature branch, local feature branch, and structural feature branch, respectively. This indicates a normalization operation.

[0143] Subsequently, the final unified fused image feature representation is obtained through the fusion function:

[0144]

[0145] in, Methods such as splicing mapping, weighted summation, attention fusion, or residual fusion can be used.

[0146] In a preferred implementation, to reduce the parameter complexity of the fusion module, the fusion function... Implemented using a parameterless element-wise summation method: ,in, This represents a unified and fused image feature representation. This represents the overall semantic features after normalization. This represents the key local features after normalization. This represents the normalized spatial structure features. This approach does not introduce an additional learnable fusion layer and can directly integrate three types of complementary features.

[0147] During model training, to reduce redundant correlations among the three types of branch features, branch decorrelation constraints can be introduced. For any input image, the decorrelation term among its three normalized branch features can be expressed as: ,in, This represents the branch decorrelation constraint. This constraint reduces the excessive correlation between overall semantic features, key local features, and spatial structural features in the projection space, enabling the three types of features to provide more complementary visual information during fusion. For features containing... The training batch of samples, the branch decorrelation constraint can be expressed as:

[0148] ,

[0149] in, Indicates the first Remove the relevant constraint terms from the branches corresponding to each sample.

[0150] 206: Feature-integrated retrieval process;

[0151] In the fusion feature retrieval mode, the system directly uses As a feature for image retrieval. For query images and stock images Unified fused image feature representations are obtained respectively. and Calculate the similarity between the two and sort them according to the similarity.

[0152] For query images and the Image library Their unified fused image feature representations are respectively and The continuous feature similarity between the two can be expressed as:

[0153] ,

[0154] in, Indicates query image Image library The continuous feature similarity score between them. For those containing The image library collection is sorted in descending order based on continuous feature similarity scores, and the top images are selected. One result:

[0155]

[0156] in, Indicates query image The corresponding Top-K search result set, This indicates that the top scorers will be selected from highest to lowest. Operations on each result.

[0157] This mode can make full use of the continuous feature representation after the fusion of the three branches, and is suitable for scenarios with high requirements for retrieval accuracy and feature expression integrity.

[0158] 207: Tri-value quantization retrieval process;

[0159] In the ternary quantization retrieval mode, the system will uniformly fuse image feature representations. Input to the ternary quantization module. The ternary quantization module first generates intermediate features through linear projection:

[0160]

[0161] in, Represents the projection matrix. Indicates the bias term. This represents the continuous features after mapping.

[0162] Subsequently, the continuous features are converted into ternary codes based on the threshold:

[0163]

[0164] in, Represents a three-valued quantization function. Indicates the encoding dimension. This represents the ternary encoded vector corresponding to the input image, and In the ternary encoding, -1, 0, and +1 represent the negative, neutral, and positive responses on the corresponding feature dimensions, respectively.

[0165] The ternary quantization module includes not only linear projection but also normalization, dimension-wise scaling, and thresholding. Its processing can be represented as follows: , in, Represents the quantization projection matrix. Indicates the quantization bias term. This indicates normalization processing. Represents the dimension-wise scaling parameter. Indicates temperature parameter, This represents element-wise multiplication. This represents the scaled continuous response characteristics. For the... Each feature dimension, three-valued quantization function It can be represented as:

[0166] ,

[0167] in, express In the Continuous response values ​​in each dimension Indicates the first The quantization thresholds corresponding to each dimension Indicates the first A 3-valued encoding in 3 dimensions. This yields the 3-valued encoded vector: ,in, Indicates the encoding dimension.

[0168] During the model training phase, since the ternary quantization function is a piecewise function, a pass-through estimation method can be used to approximate the gradient propagation:

[0169]

[0170] in, This represents the gradient truncation threshold. Using the above method, the ternary quantization module can generate discrete ternary codes during forward computation and maintain approximate gradient propagation during training.

[0171] In ternary quantization retrieval mode, for query images and the Image library Their three-value codes are respectively and The three-valued encoding matching score between the two can be expressed as:

[0172] ,

[0173] in, Indicates query image In the Three-value encoding in each dimension Indicates the first The image in the gallery is in the first... Three-value encoding in each dimension This represents the encoding dimension. To reduce the impact of differences in the number of non-zero encoding values ​​between different images on the matching results, normalized three-valued encoding similarity can also be used:

[0174] ,

[0175] in, Represents the similarity of normalized three-valued codes. This represents a very small constant used to avoid a denominator of zero. Subsequently, the images in the image library are sorted in descending order based on the ternary coding matching score or the normalized ternary coding similarity to obtain the Top-K search results. ,in, This represents the set of Top-K search results under the ternary quantization search mode.

[0176] This mode completes the retrieval matching through compact encoding, which can reduce the space occupied by feature storage and improve the matching efficiency when searching large-scale image libraries.

[0177] 208: Visual output of search results.

[0178] After the system completes the search and sorting, it returns the Top-K search results to the user interface. The interface displays the following information: query image, search direction, search method, target number, similarity or match score, result image, library file name, and running status.

[0179] This visualization output method allows users to quickly determine whether the search results correspond to the query image, and can be used for software display and application integration of cross-view geographic target retrieval systems.

[0180] In summary, the embodiments of the present invention, through steps 201-208, further illustrate the processing relationships between the basic visual coding network, the global feature branch, the local feature branch, the structural feature branch, the feature fusion module, and the ternary quantization module. By fusing overall semantic features, key local features, and spatial structural features into a unified fused image feature representation, and further mapping this unified fused image feature representation into a ternary code composed of negative, zero, and positive values, the embodiments of the present invention can reduce feature storage and matching computation overhead while preserving image feature discrimination capabilities, making it suitable for large-scale image library retrieval and lightweight retrieval scenarios.

[0181] Example 3

[0182] See Figure 5 This invention provides a cross-view image retrieval device based on three-branch feature fusion and ternary quantization. The device includes: an image acquisition module, an image preprocessing module, a three-branch feature extraction module, a feature fusion module, a ternary quantization module, a retrieval and ranking module, and a result display module.

[0183] I. Image Acquisition Module

[0184] The image acquisition module is used to acquire query images and gallery images. This module automatically determines the query image set and gallery image set based on the user's selected search direction.

[0185] When a user selects a drone view to search for a satellite view, this module uses the drone image as the query image and the satellite image as the gallery image; when a user selects a satellite view to search for a drone view, this module uses the satellite image as the query image and the drone image as the gallery image.

[0186] II. Image Preprocessing Module

[0187] The image preprocessing module is used to perform unified format processing on query images and library images, including: image reading, size adjustment, normalization and tensor quantization.

[0188] This module ensures that images from different sources, sizes, or formats can be uniformly input into the subsequent feature extraction network.

[0189] III. Three-branch feature extraction module

[0190] The three-branch feature extraction module includes: a global feature branch, a local feature branch, and a structural feature branch.

[0191] The global feature branch is used to extract the overall semantic features of the image; the local feature branch is used to extract the features of key local regions of the image; and the structural feature branch is used to extract the spatial layout and regional structural features of the image. Together, these three branches constitute a multi-level feature representation of the image across different viewpoints.

[0192] In one specific implementation, the three-branch feature extraction module further includes a global semantic extraction unit, a local salient region extraction unit, and a spatial structure modeling unit.

[0193] The global semantic extraction unit receives global labeled features and spatial patch token features output by the basic visual encoding network, and generates overall semantic features through global pooling, multi-scale aggregation, and channel gating. The local salient region extraction unit receives spatial patch token features and generates key local features through attention pooling, Top-K salient region selection, and max pooling. The spatial structure modeling unit rearranges the spatial patch token features into a spatial feature map based on the two-dimensional positional relationships of the spatial patch tokens, and generates spatial structure features through position-aware pooling and spatial pyramid pooling.

[0194] IV. Feature Fusion Module

[0195] The feature fusion module is used to fuse global semantic features, local salient features and spatial structural features to generate fused image features of a unified dimension.

[0196] In one specific implementation, the feature fusion module further includes a branch mapping unit, a normalization unit, and a fusion unit. The branch mapping unit is used to map the overall semantic features, key local features, and spatial structural features to a unified dimension; the normalization unit is used to normalize the three types of features after mapping; and the fusion unit is used to perform element-wise summation or other fusion processing on the three types of features after normalization to obtain a unified fused image feature representation.

[0197] V. Three-value quantization module

[0198] The ternary quantization module is used to convert continuously fused image features into ternary codes. The ternary codes are generated by... , and composition.

[0199] In one specific implementation, the ternary quantization module includes a linear projection unit, a normalization unit, a scaling unit, a threshold determination unit, and a ternary encoding output unit. The linear projection unit maps the unified fused image feature representation to continuous response features; the normalization unit and the scaling unit adjust the distribution of the continuous response features; the threshold determination unit converts the continuous response features into -1, 0, and +1 according to a dimension-wise threshold; and the ternary encoding output unit outputs the ternary encoded vector corresponding to the input image.

[0200] This module can convert continuous features into discrete codes through linear projection, normalization, threshold judgment, and quantization functions, thereby reducing feature storage and retrieval computation costs.

[0201] VI. Search and Sorting Module

[0202] The search and sorting module is used to perform corresponding matching calculations based on the search mode selected by the user.

[0203] In the fusion feature retrieval mode, this module calculates the similarity between the fusion features of the query image and the fusion features of the images in the library; in the ternary quantization retrieval mode, this module calculates the encoding matching score between the ternary code of the query image and the ternary code of the images in the library. Subsequently, the retrieval ranking module generates Top-K retrieval results according to the scores from high to low.

[0204] VII. Results Display Module

[0205] The results display module is used to show the query image, the retrieved image, the sort number, the similarity or matching score, the target number, the image path in the library, and the running status.

[0206] This module can be accessed through browser pages, desktop application interfaces, or other visual interactive methods, making it easy for users to view search results and system operation status.

[0207] Example 4

[0208] A cross-view image retrieval device based on three-branch feature fusion and ternary quantization includes a processor and a memory. The memory stores program instructions, and the processor calls the program instructions stored in the memory to cause the device to execute the following method steps in Embodiment 1:

[0209] The global labeled features output by the basic visual coding network are input into the global feature branch to extract the overall semantic features of the image;

[0210] The spatial patch token features output by the basic visual coding network are input into the local feature branch to extract key local features of the image;

[0211] The spatial patch features are input into the structural feature branch, which is used to model the spatial relationships and layout structure between different regions of the image to obtain spatial structural features.

[0212] The overall semantic features, key local features, and spatial structure features are input into the feature fusion module. The feature fusion module performs unified mapping and fusion of the three types of features to obtain a unified fused image feature representation.

[0213] When the retrieval mode is ternary quantization retrieval, the obtained unified fused image feature representation is input into the ternary quantization module to generate ternary codes; the matching score between the ternary codes in the query image and the ternary codes of the images in the library is calculated.

[0214] The images in the image library are sorted in descending order based on the similarity between unified fused image features or the three-valued encoding matching score, and the search results are output.

[0215] The encoding matching score is represented as follows:

[0216]

[0217] in, Indicates the encoding dimension. and These represent the query image and the gallery image at the [number]th [position]. Three-value encoding in each dimension.

[0218] Specifically, the overall semantic features, key local features, and spatial structure features are input into the feature fusion module. The feature fusion module performs unified mapping and fusion of the three types of features to obtain a unified fused image feature representation as follows:

[0219]

[0220]

[0221]

[0222] in, , , Let represent the mapping functions for the global feature branch, local feature branch, and structural feature branch, respectively. This indicates a normalization operation; For global semantic features, It is a locally significant feature. It is a spatial structural feature;

[0223] Subsequently, the final unified fused image feature representation is obtained through the fusion function:

[0224]

[0225] in, Employing splicing mapping, weighted summation, attention fusion, or residual fusion methods.

[0226] The ternary quantization module first generates intermediate features through linear projection:

[0227]

[0228] in, Represents the projection matrix. Indicates the bias term. Represents continuous features after mapping;

[0229] Convert continuous features into three-value codes based on a threshold:

[0230]

[0231] in, Represents a three-valued quantization function. Indicates the encoding dimension. This represents the ternary encoded vector corresponding to the input image, and In the ternary encoding, -1, 0, and +1 represent the negative, neutral, and positive responses on the corresponding feature dimensions, respectively.

[0232] The basic encoded features output by the basic visual coding network are:

[0233] For any input image The basic encoded features output by the basic visual coding network are represented as follows: ,in, Represents the basic visual coding network. Represents global labeling features. The feature matrix representing the patchtoken representation space;

[0234] The spatial patch token feature matrix is ​​represented as follows: ;in, Indicates the first The feature vector corresponding to each patch Indicates the number of patches. Indicates the feature channel dimension; when the input image size is The patch area size is and and All were When divisible, the number of patches is expressed as: .

[0235] The method further includes: introducing branch decorrelation constraints.

[0236] For any input image, the decorrelation term among the three normalized branch features is represented as: ;in, This indicates that the branch removes the relevant constraint terms. This represents the normalized global branch features. This represents the local branch features after normalization. Represents the normalized structural branch features; for those containing The training batch of samples, the branch decorrelation constraint is represented as: ,in, Indicates the number of samples in the batch. Indicates the first Remove the relevant constraint terms from the branches corresponding to each sample.

[0237] The method further includes: using normalized ternary encoding similarity:

[0238] For query images and the Image library Its three-value codes are represented as follows: and Then the normalized three-valued encoding similarity is expressed as: ,in, Indicates query image With the Image library Normalized ternary code similarity between them This represents a very small constant used to avoid a denominator of zero.

[0239] The images in the image library are sorted in descending order based on the ternary coding matching score or the normalized ternary coding similarity to obtain the Top-K search results: ,in, This represents the set of Top-K search results under the ternary quantization search mode. Indicates the number of images in the gallery. This indicates the number of search results returned. This indicates that the top results are selected based on similarity scores from highest to lowest. Operations on each result.

[0240] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0241] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.

[0242] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.

[0243] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0244] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0245] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0246] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0247] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0248] Example 5

[0249] This invention provides an experimental verification method for validating the retrieval performance of three-branch feature fusion and ternary quantization. This experiment demonstrates the effectiveness, stability, and retrieval efficiency of the method described in this invention in cross-view retrieval tasks involving UAV-view images and satellite-view images.

[0250] In one specific implementation, the experimental environment may include the following configurations: Hardware environment: computer, graphics processor, memory, and storage devices; Software environment: operating system, Python runtime environment, deep learning framework, image processing library, and visualization interface program; Model environment: basic visual encoding network, global feature branch, local feature branch, structural feature branch, feature fusion module, ternary quantization module, and retrieval and ranking module; Output environment: retrieval results can be displayed through the software interface or saved as a structured file, which includes information such as query image path, library image path, target number, ranking number, similarity or matching score, etc.

[0251] The experimental data includes: a set of images from the perspective of unmanned aerial vehicles (UAVs) and a set of images from the perspective of satellites. The UAV image set represents images of geographic targets acquired from low altitude or oblique perspective, while the satellite image set represents images of geographic targets acquired from a high altitude, top-down perspective.

[0252] In the experiment, each geographic target corresponded to at least one UAV-view image and one satellite-view image, and cross-view correspondences were established using target numbers. The experimental retrieval directions included:

[0253] Drone-view search for satellite-view images: using drone images as the query images and satellite images as the library images.

[0254] Satellite-view search of drone-view images means using satellite images as the query images and drone images as the image library images.

[0255] To verify the effectiveness of the method of the present invention, the following comparative experiment can be set up:

[0256] The first group uses only global features for retrieval;

[0257] The second group uses a fusion retrieval method that combines global and local features.

[0258] The third group uses a fusion retrieval method that combines global features and structural features;

[0259] The fourth group uses the global feature branch, local feature branch, and structural feature branch of this invention to perform three-branch fusion retrieval;

[0260] The fifth group was searched using the ternary quantization retrieval mode of the present invention.

[0261] The comparative experiments described above can verify the effects of local feature branching, structural feature branching, three-branch fusion mechanism, and ternary quantization mechanism on cross-view image retrieval results.

[0262] The experiment used the following evaluation indicators:

[0263] Recall@K, where the symbol @ indicates "in the first K results" or "up to the Kth position". It is used to limit consideration to the first K items in the sorted list, calculating the proportion of relevant results from these items out of all relevant results. It is used to represent the proportion of correctly matching images appearing in the first K search results;

[0264] mAP is used to evaluate the overall accuracy of search ranking results;

[0265] Top-K search results visualization is used to intuitively determine whether the search results correspond to the query images. Recall@K can be represented as: Recall@K = Number of queries with correct matches in the first K positions of the search results / Total number of query images.

[0266] Experimental Results Record

[0267] Table 1. Cross-perspective search results record table under different feature representation methods

[0268]

[0269] Based on the experimental results shown in Table 1, we can analyze the results from the aspects of different feature representation methods, three-branch fusion effect, bidirectional retrieval stability, and ternary quantization retrieval effect.

[0270] First, looking at the retrieval results using only global features, the Recall@1 for the UAV → satellite retrieval direction is 91.07%, and the mAP is 93.82%; while the Recall@1 for the satellite → UAV retrieval direction is 94.58%, and the mAP is 90.64%. This result indicates that global features can characterize the overall scene semantics of an image to a certain extent and can be used for cross-view matching between UAV and satellite images. However, single global features mainly focus on the semantic expression at the entire image level, and their ability to characterize local salient regions and spatial structural relationships is limited. Therefore, there is still room for improvement when there are significant changes in cross-view scale, viewpoint differences, and local regional variations.

[0271] Secondly, the performance of different dual-branch fusion retrieval results shows certain differences. In the UAV → satellite retrieval direction, the Recall@1 and mAP of "local features + structural features" reached 91.31% and 94.09% respectively, higher than 91.07% and 93.82% using only global features, indicating that local saliency information and spatial structure information have a certain supplementary role in UAV-to-satellite retrieval tasks. In the satellite → UAV retrieval direction, the Recall@1 of "global features + structural features" reached 94.86%, higher than 94.58% using only global features; the mAP of "local features + structural features" reached 90.96%, higher than 90.64% using only global features. This result shows that the structural feature branch can enhance the model's ability to express regional layout and spatial distribution relationships, while the local feature branch can supplement local discrimination information such as building outlines, road intersections, and water boundaries. However, some dual-branch combinations did not surpass single global features in all indicators, indicating that when introducing a certain type of auxiliary feature alone, there may be problems of insufficient feature complementarity or incomplete information expression.

[0272] Furthermore, the three-branch fusion retrieval results show that the "global features + local features + structural features" approach achieved the best results. In the UAV → satellite retrieval direction, the Recall@1 of the three-branch fusion retrieval was 93.98%, and the mAP was 94.99%, representing improvements of 2.91 and 1.17 percentage points respectively compared to using only global features. In the satellite → UAV retrieval direction, the Recall@1 was 95.44%, and the mAP was 93.32%, representing improvements of 0.86 and 2.68 percentage points respectively compared to using only global features. This result indicates that global features, local features, and structural features have good complementarity. Global features provide an overall semantic representation of the image, local features enhance the discrimination ability of key regions, and structural features describe the spatial layout relationships between different regions. After unified fusion, these three types of features can form a more complete and stable cross-view image feature representation.

[0273] Furthermore, from the bidirectional retrieval results, the three-branch fusion retrieval achieved the highest Recall@1 and mAP in both the UAV→satellite and satellite→UAV directions. Specifically, the Recall@1 in the UAV→satellite direction reached 93.98%, and the Recall@1 in the satellite→UAV direction reached 95.44%. This indicates that the method of the present invention is not only applicable to retrieval tasks using UAV images as query images and satellite images as library images, but also to reverse retrieval tasks using satellite images as query images and UAV images as library images, demonstrating good bidirectional cross-view retrieval adaptability.

[0274] Finally, based on the ternary quantization retrieval results, ternary encoding retrieval achieved 92.76% Recall@1 and 94.04% mAP in the UAV→satellite direction, and 95.29% Recall@1 and 92.79% mAP in the satellite→UAV direction. Compared with three-branch fusion retrieval, ternary quantization retrieval reduced Recall@1 by 1.22 percentage points and mAP by 0.95 percentage points in the UAV→satellite direction; and reduced Recall@1 by only 0.15 percentage points and mAP by 0.53 percentage points in the satellite→UAV direction. This result indicates that converting the unified fusion image feature representation into ternary encoding composed of -1, 0, and +1 still retains strong image discrimination capabilities. Especially in the satellite→UAV retrieval direction, the Recall@1 of ternary quantization retrieval is quite close to that of three-branch fusion retrieval, indicating that ternary encoding can effectively support cross-view image matching tasks.

[0275] In summary, the experimental results show that the three-branch fusion retrieval method outperforms single-feature retrieval and two-branch fusion retrieval in all indicators, verifying the effectiveness of joint modeling of global feature branches, local feature branches, and structural feature branches. Furthermore, the ternary quantization retrieval method, while maintaining high retrieval performance, can complete cross-view matching in discrete encoding form, verifying the feasibility of using the ternary quantization module for cross-view image retrieval.

[0276] The above experiments lead to the following conclusions:

[0277] First, the three-branch feature fusion method described in this invention can effectively improve the cross-view retrieval performance between UAV-view images and satellite-view images. Experimental results show that the three-branch fusion retrieval achieves 93.98% Recall@1 and 94.99% mAP in the UAV→satellite direction, and 95.44% Recall@1 and 93.32% mAP in the satellite→UAV direction, which are the best results under various feature representation methods in Table 1.

[0278] Second, compared to retrieval methods that only use global features, this invention, by introducing local feature branches and structural feature branches, can further supplement local discriminative information and spatial layout information. Global features mainly represent the overall semantics of the image, local features can focus on key regions with stable correspondences, and structural features can describe the spatial distribution relationships between different regions in the image. After fusing these three types of features, the problem of insufficient adaptability of single feature representation to cross-viewpoint differences can be alleviated.

[0279] Third, the results of the two-branch fusion experiment show that different types of features have a certain complementary effect, but the improvement in retrieval performance is not stable when only any two branches are used. In contrast, the three-branch fusion method can simultaneously utilize overall semantics, local details, and spatial structure information, thus exhibiting better retrieval accuracy and ranking stability in both UAV→satellite and satellite→UAV retrieval directions.

[0280] Fourth, the experimental results of ternary quantization retrieval show that converting the unified fused image feature representation into ternary encoding still maintains good retrieval performance. Ternary quantization retrieval achieved a Recall@1 of 92.76% and a mAP of 94.04% in the UAV→satellite direction; and a Recall@1 of 95.29% and a mAP of 92.79% in the satellite→UAV direction. These results demonstrate that ternary encoding can retain effective discriminative information in the unified fused image features and can be used as a discrete feature representation method in cross-view image retrieval.

[0281] Fifth, the method of this invention achieves effective retrieval results in both the UAV→satellite and satellite→UAV directions, indicating that the method has bidirectional cross-view retrieval capabilities and is applicable to application scenarios such as UAV inspection, remote sensing image retrieval, smart city management, and cross-view geographic target positioning.

[0282] In summary, this experiment verifies the effectiveness of the global feature branch, local feature branch, structural feature branch, feature fusion module, and ternary quantization module described in this invention. This invention can obtain a more stable cross-view image representation through three-branch feature fusion and can achieve discrete coding retrieval through ternary quantization, providing an effective technical solution for cross-view matching between UAV-view images and satellite-view images.

[0283] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0284] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0285] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A three-branch ternary quantization cross-view image retrieval method, characterized in that, The method includes: The global labeled features output by the basic visual coding network are input into the global feature branch to extract the overall semantic features of the image; The spatial patch token features output by the basic visual coding network are input into the local feature branch to extract key local features of the image; The spatial patch features are input into the structural feature branch, which is used to model the spatial relationships and layout structure between different regions of the image to obtain spatial structural features. The overall semantic features, key local features, and spatial structure features are input into the feature fusion module. The feature fusion module performs unified mapping and fusion of the three types of features to obtain a unified fused image feature representation. When the retrieval mode is ternary quantization retrieval, the obtained unified fused image feature representation is input into the ternary quantization module to generate ternary codes; the matching score between the ternary codes in the query image and the ternary codes of the images in the library is calculated. The images in the image library are sorted in descending order based on the similarity between unified fused image features or the three-valued encoding matching score, and the search results are output.

2. The three-branch ternary quantization cross-view image retrieval method according to claim 1, characterized in that, The encoding matching score is represented as follows: ； in, Indicates the encoding dimension. and These represent the query image and the gallery image at the [number]th [position]. Three-value encoding in each dimension.

3. The three-branch ternary quantization cross-view image retrieval method according to claim 1, characterized in that, The overall semantic features, key local features, and spatial structure features are input into the feature fusion module. The feature fusion module performs unified mapping and fusion of the three types of features to obtain a unified fused image feature representation as follows: ； ； ； in, , , Let represent the mapping functions for the global feature branch, local feature branch, and structural feature branch, respectively. This indicates a normalization operation; For global semantic features, It is a locally significant feature. It is a spatial structural feature; Subsequently, the final unified fused image feature representation is obtained through the fusion function: ； in, Employing splicing mapping, weighted summation, attention fusion, or residual fusion methods.

4. The three-branch ternary quantization cross-view image retrieval method according to claim 1, characterized in that, The ternary quantization module first generates intermediate features through linear projection: ； in, Represents the projection matrix. Indicates the bias term. Represents continuous features after mapping; Convert continuous features into three-value codes based on a threshold: ； in, Represents a three-valued quantization function. Indicates the encoding dimension. This represents the ternary encoded vector corresponding to the input image, and In the ternary encoding, -1, 0, and +1 represent the negative, neutral, and positive responses on the corresponding feature dimensions, respectively.

5. The three-branch ternary quantization cross-view image retrieval method according to claim 1, characterized in that, The basic encoded features output by the basic visual coding network are: For any input image The basic encoded features output by the basic visual coding network are represented as follows: ,in, Represents the basic visual coding network. Represents global labeling features. The feature matrix representing the patch token in the representation space; The spatial patch token feature matrix is ​​represented as follows: ;in, Indicates the first The feature vector corresponding to each patch Indicates the number of patches. Indicates the feature channel dimension; when the input image size is The patch area size is and and All were When divisible, the number of patches is expressed as: .

6. The three-branch ternary quantization cross-view image retrieval method according to claim 1, characterized in that, The method further includes: introducing branch decorrelation constraints. For any input image, the decorrelation term among the three normalized branch features is represented as: ;in, This indicates that the branch removes the relevant constraint terms. This represents the normalized global branch features. This represents the local branch features after normalization. Represents the normalized structural branch features; for those containing The training batch of samples, the branch decorrelation constraint is represented as: ,in, Indicates the number of samples in the batch. Indicates the first Remove the relevant constraint terms from the branches corresponding to each sample.

7. The three-branch ternary quantization cross-view image retrieval method according to claim 1, characterized in that, The method further includes: using normalized ternary encoding similarity: For query images and the Image library Its three-value codes are represented as follows: and Then the normalized three-valued encoding similarity is expressed as: ,in, Indicates query image With the Image library Normalized ternary code similarity between them This represents a very small constant used to avoid the denominator being zero; The images in the image library are sorted in descending order based on the ternary coding matching score or the normalized ternary coding similarity to obtain the Top-K search results: ,in, This represents the set of Top-K search results under the ternary quantization search mode. Indicates the number of images in the gallery. This indicates the number of search results returned. This indicates that the top results are selected based on similarity scores from highest to lowest. Operations on each result.

8. A three-branch ternary quantization cross-view image retrieval device, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-7.

Images