Multi-source visual scene characterization method based on query residual aggregation
By employing non-destructive complementary fusion and query residual aggregation in multi-source visual scene representation methods, the problem of unstable feature distribution in complex environments for visual scene recognition is solved, thereby improving the robustness and accuracy of recognition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2026-01-16
- Publication Date
- 2026-05-01
AI Technical Summary
Existing visual scene recognition methods struggle to simultaneously address appearance details, semantic invariance, and cross-domain robustness in complex environments. Furthermore, the unstable feature distribution after multi-backbone network fusion leads to decreased retrieval robustness and stability.
A multi-source visual scene representation method based on query residual aggregation is adopted. Through non-destructive complementary fusion and residual compensation, multi-source complementary information is injected into global features. The query residual representation is used to enhance fine-grained difference capture and anti-interference ability, and a stable global scene representation is constructed.
It improves the robustness and discriminativeness of global scene representation, and can effectively suppress the effects of viewpoint changes, lighting differences and occlusion interference under complex conditions, thereby improving the accuracy and consistency of visual scene recognition.
Smart Images
Figure CN121962827A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and computer vision, specifically to a multi-source visual scene representation method based on query residual aggregation, which can be used for retrieval tasks such as visual scene recognition. Background Technology
[0002] Visual scene recognition aims to determine whether the current visual observation corresponds to a previously visited location. It is a key component of autonomous navigation systems and can be used for loop closure detection and global relocalization. It is widely applied in scenarios such as mobile robots, drones, and augmented reality.
[0003] Early visual scene recognition methods often relied on manually extracted feature representations, such as the Bag-of-Words (BoW) model and Virtual Local Aggregate Description (VLAD), which aggregate local features into compact descriptive vectors. These methods are generally well-structured, computationally efficient, and have a certain degree of interpretability, but they are sensitive to changes in viewpoint and lighting, and their robustness in complex environments is limited.
[0004] With the development of deep learning, visual scene recognition is gradually shifting towards end-to-end learnable global descriptor frameworks. Convolutional neural network-based methods, such as NetVLAD and MixVPR, significantly improve recognition performance under significant changes in scene appearance by learning data-driven global feature representations.
[0005] In recent years, visual Transformers have demonstrated outstanding performance in various computer vision tasks, capable of modeling long-range dependencies between image regions, which is beneficial for forming more global scene representations. Building upon this, fundamental visual models, through self-supervised or multimodal pre-training on massive datasets, learn more transferable general visual representations: for example, the visual self-supervised model DINO tends to learn structural and geometrically consistent representations, while the visual-language model CLIP obtains semantically aligned and cross-domain robust embedding spaces through large-scale image-text comparison learning. The structural cues and semantic priors provided by different fundamental visual models are complementary, laying the foundation for visual scene recognition to simultaneously achieve "geometric fidelity" and "semantic invariance." Based on these general representations, methods such as AnyLoc and CricaVPR have shown strong generalization capabilities in different environments and domains.
[0006] However, despite the powerful global representation provided by the visual foundation model, existing methods generally use only a single backbone network (such as DINO) to extract features. The resulting global descriptors struggle to simultaneously maintain appearance detail, semantic invariance, and cross-domain robustness. Furthermore, when multi-backbone networks (such as visual self-supervised models and visual-language models) are introduced, the incompatible embedding spaces of features from different encoders often lead to unstable feature distributions after fusion. Especially in nearest neighbor retrieval frameworks, "strongly coupled fusion" methods such as simple concatenation, symmetrical projection, or cross-attention can easily disrupt the original feature space and amplify the differences in heterogeneous token distributions, thereby reducing retrieval robustness and stability. Additionally, existing query aggregation methods (such as BoQ) typically perform "absolute aggregation" of query responses, making them sensitive to feature scale, token imbalance, and heterogeneous distributions. This hinders the construction of stable global descriptors and makes it difficult to characterize the distribution bias and structural differences of different regions in the query prototype space. Consequently, the generated global descriptors exhibit significantly reduced discriminative power when facing highly similar scenes, failing to meet the requirements of high-precision visual scene retrieval. Summary of the Invention
[0007] The purpose of this invention is to at least partially solve the above problems by providing a multi-source visual scene representation method based on query residual aggregation, with "non-destructive complementary fusion" and "query residual aggregation" as the core: while maintaining the stable appearance of the main features as much as possible, multi-source complementary information is injected in a residual compensation manner, and a discriminative residual representation is introduced in the aggregation stage to enhance the ability to capture fine-grained differences and resist interference, thereby improving the robustness and discriminability of global scene representation under complex conditions.
[0008] This invention is achieved through the following technical solution:
[0009] In a first aspect, the present invention provides a multi-source visual scene representation method based on query residual aggregation, comprising the following steps:
[0010] The image to be processed is acquired and preprocessed to obtain a standardized image;
[0011] The standardized image is input into the first visual encoder and the second visual encoder respectively to obtain the first local feature set and the second local feature set; complementary information from the second local feature set is injected into the first local feature set through residual correction to construct multimodal fusion local features;
[0012] The multimodal fusion local features are input into the query residual aggregator for feature aggregation. Based on multiple learnable query vectors, the residual representation corresponding to each query vector is calculated.
[0013] All residual representations are concatenated, dimensionality reduced, and normalized to output a global scene representation vector.
[0014] Furthermore, the first visual encoder is a self-supervised visual encoder, and the second visual encoder is a vision-language encoder; specifically, the self-supervised visual encoder uses DINOv2, and the vision-language encoder uses CLIP.
[0015] Furthermore, the multimodal fusion local features are constructed using the following formula:
[0016]
[0017] in, This is the first local feature set. For the second local feature set, For mapping networks.
[0018] Furthermore, before calculating the multimodal fusion local features, the first local feature set and the second local feature set are L2 normalized in the channel dimension respectively; the mapping network is configured as follows: when the local features are in sequence form, it is implemented by a fully connected layer; when the local features are in feature map form, it is implemented by a 1×1 convolutional layer.
[0019] Furthermore, the input to the query residual aggregator for feature aggregation includes the following steps:
[0020] Construct N learnable query vectors ;
[0021] Application of local features in multimodal fusion Convolutional layers are used to reduce channel dimensions. Subsequently, multimodal fusion local features are represented from feature map form as a sequence and LayerNorm is applied.
[0022] query vector Considering each group of multimodal fusion local features as a cluster center, a soft partition is performed on the local feature space. and query Calculate the soft allocation weights between the two. ;
[0023] According to the soft allocation weight The residual representation is obtained by weighted aggregation of the deviations of local features relative to the query vector. .
[0024] Furthermore, the formula for calculating the soft-assignment weights is as follows:
[0025]
[0026]
[0027] Where d is the feature dimension. Represents the similarity score Normalize.
[0028] Furthermore, represent the N residuals By concatenating the vectors, we obtain the original global scene representation vector. ;
[0029] The original global scene representation vector is reduced in dimension and compressed using a linear layer.
[0030] The compressed vector is then subjected to L2 normalization to obtain the final global scene representation vector.
[0031] Furthermore, the method also includes a step of performing visual scene recognition using the global scene representation vector, specifically:
[0032] Retrieve the query image to be searched and the reference image from the database;
[0033] The global scene representation vector of the query image and the global scene representation vector of the reference image are generated using the method described above, respectively.
[0034] Calculate the similarity measure between the representation vectors of the query image and the reference image, sort the reference images according to the magnitude of the similarity measure, and determine the scene that matches the query image to complete scene recognition.
[0035] Secondly, the present invention provides a multi-source visual scene recognition system based on query residual aggregation, comprising:
[0036] The image acquisition module acquires the query image and reference image to be retrieved;
[0037] The vector calculation module is used to calculate the global scene representation vector of the query image and the global scene representation vector of the reference image according to the above method.
[0038] The scene recognition module is used to calculate the similarity measure between the representation vectors of the query image and the reference image, sort the reference images according to the magnitude of the similarity measure, and determine the scene that matches the query image to complete the scene recognition.
[0039] Thirdly, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.
[0040] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.
[0041] Optionally, the global scene representation vector output by the method of the present invention can be applied to tasks such as visual scene recognition through similarity measurement; the similarity measurement can be cosine similarity or Euclidean distance; training can use metric learning loss (such as multi-similarity loss, contrastive loss) to improve retrieval discriminability.
[0042] Compared with the prior art, the present invention has at least the following beneficial effects:
[0043] This invention employs a non-destructive complementary fusion approach when introducing a multi-backbone network. Based on a stable fundamental appearance feature space, it injects multi-source complementary information through residual compensation, avoiding feature distribution disruption and redundant stacking caused by direct splicing and fusion. This makes the fusion process more stable, and the resulting fused features are easier to train and have greater generalization ability. Compared to high-dimensional splicing fusion, the residual fusion method used in this invention has more controllable growth in parameter scale and computational complexity, making it more suitable for large-scale scene recognition and retrieval applications. Simultaneously, this invention explicitly models the "residual distribution of local features relative to the query prototype" through query residual aggregation, effectively enhancing the expressive power of fine-grained information and improving the discriminativeness of scene representation. This helps suppress non-discriminatory disturbances introduced by viewpoint changes, lighting differences, seasonal changes, occlusion interference, and dynamic objects in scene recognition tasks, thereby improving the robustness and consistency of global scene representation in complex environments and enhancing retrieval performance. Attached Figure Description
[0044] To make the technical solutions in the embodiments of the present invention or the prior art clearer, the accompanying drawings involved in the description of the embodiments or the prior art are briefly described below. Obviously, the following drawings are only schematic diagrams of some embodiments of the present invention and do not constitute a limitation of the present invention; those skilled in the art can obtain other drawings or equivalent modified schematic diagrams based on these drawings and their descriptions without creative effort.
[0045] Figure 1 This is an overall framework diagram of the multi-source visual scene representation method based on query residual aggregation in this embodiment of the invention;
[0046] Figure 2 This is a schematic diagram illustrating the scene retrieval sample results of comparing the method described in this invention with existing methods under the same query image in an embodiment of this invention. Detailed Implementation
[0047] The present invention will be further described below with reference to the embodiments.
[0048] The "global scene representation vector" described in this invention can be used for retrieval tasks such as visual scene recognition, and can be matched by cosine similarity or Euclidean distance.
[0049] Reference Figure 1 The method for calculating the global scene representation vector in this embodiment includes five stages: image acquisition and preprocessing, multi-source local feature extraction, non-destructive complementary feature fusion, global aggregation based on query residuals, and global scene representation output, corresponding to steps S10-S50.
[0050] Step S10: Image Acquisition and Preprocessing. Acquire the input image, and perform a normalization operation on any input image to obtain a standardized image. .
[0051] Step S20: Multi-source local feature extraction. The standardized image... Inputting the self-supervised visual encoder DINOv2 and the visual-language encoder CLIP, we obtain a set of token-level local features:
[0052] , ,in , d represents the number of tokens and d represents the feature dimension.
[0053] In this embodiment, both DINOv2 and CLIP use the ViT-L architecture and have a patch size of 14. For an input resolution H×W, the number of tokens M obtained is H / 14×W / 14 (excluding CLS tokens).
[0054] Optionally, the CLS token and patch tokens output by the ViT model can be separated, and only the patch tokens can be used as local features to reduce the interference of the CLS token on the aggregation of local features, or the CLS token can be retained and processed separately in the aggregator.
[0055] Optionally, when and When spatial resolutions are inconsistent, alignment can be achieved through methods such as positional encoding interpolation and feature map interpolation, ensuring that the two feature paths can correspond to each token.
[0056] Optionally, the encoder used can be completely frozen or some layers of the encoder can be fine-tuned to meet the needs of different tasks.
[0057] Step S30: Non-destructive complementary feature fusion. Using the local features output by DINOv2 as stable basic appearance feature representations, complementary semantic information is injected from CLIP through residual correction, rather than forcing the two encoders into a newly learned joint space, to obtain multimodal fused local features. The multimodal fusion local features satisfy:
[0058]
[0059] in For small multilayer perceptrons or equivalent lightweight mapping networks, the residual term It reflects the supplementary information of CLIP relative to DINOv2 encoding, and its modulation ensures that the fused features always revolve around the feature manifold of DINOv2.
[0060] This residual fusion mechanism stabilizes the feature distribution by explicitly limiting the contribution of CLIPs to the controlled correction terms. Therefore, the injection of semantic information does not overwhelm or distort the apparent structure of the DINOv2 encoding. Compared to direct multi-model fusion, this design preserves the strengths of each encoder while mitigating conflicts and scale imbalances in the embedding space. Importantly, this fusion is non-destructive: the original DINOv2 features are preserved as the backbone representation, and the residual paths are modulated only where complementary information is advantageous.
[0061] The multimodal fusion local features A coherent and easily retrieved local feature representation is provided for subsequent global aggregation modules.
[0062] In this embodiment, when... and Before generating residuals, L2 normalization is first performed on each channel to ensure they are at the same scale; the local features used are in the form of feature maps. Specifically by Convolutional layers are used to implement this; if the local features are in sequence form, This can be achieved using a fully connected layer.
[0063] Step S40: Global Aggregation Based on Query Residuals. To construct a compact global scene representation for subsequent retrieval tasks, and to ensure it can withstand feature heterogeneity and distribution shift caused by multimodal fusion, this invention proposes a Query Residual Aggregation (VLAQ) method, which encodes local features relative to a learnable query prototype, rather than aggregating absolute responses.
[0064] The core idea of VLAQ is to introduce residual aggregation into a query-based pooling framework. While existing query-based methods aggregate features directly based on their similarity to the query center, this absolute aggregation is sensitive to issues such as feature scaling, token imbalance, and heterogeneous token distributions, which are particularly prominent in retrieval-based visual scene recognition tasks. In contrast, residual aggregation can determine how local features deviate from their relevant query centers, resulting in a more stable and discriminative global representation.
[0065] Step S40 can be further divided into the following sub-steps:
[0066] Step S41: Query Vector Construction and Initialization. Construct N learnable query vectors. It uses random initialization and learns during training.
[0067] Alternatively, cluster centers can be selected for initialization (e.g., K-means on the training data) to accelerate convergence;
[0068] Step S42: Input feature dimensionality reduction and normalization. Applying this to local features from multimodal fusion. Convolutional layers are used to reduce channel dimensions, thereby reducing the computational complexity and memory usage of subsequent modules. Subsequently, multimodal fusion local features are represented as a sequence from feature maps and LayerNorm is applied to improve the numerical stability of attention computation.
[0069] Step S43: Query-Local Feature Attention Interaction. Treating the query vector as the cluster center, a soft partitioning is performed on the multimodal fusion local feature space. For each group... and query Calculate the dot product similarity:
[0070]
[0071] and through Obtain the soft-assigned weights:
[0072]
[0073] In this embodiment, this step is specifically implemented through a multi-head attention mechanism.
[0074] Optionally, soft-assigned weights of local features relative to the query center can be obtained through other normalization methods such as optimal transmission;
[0075] Step S44: Residual Aggregation. Unlike directly aggregating query responses, the VLAQ proposed in this invention performs residual aggregation on the query center:
[0076]
[0077] This concept encodes the local biases around each query prototype, similar to residual encoding in VLAD, effectively transforming query-based aggregation from a soft counting mechanism into a discriminative residual representation.
[0078] Step S50: Output the global scene representation. This involves the residual representations corresponding to N learnable queries. The final visual scene representation vector is obtained by splicing, dimensionality reduction, and normalization.
[0079] Step S50 can be further divided into the following sub-steps:
[0080] Step S51: Aggregate feature concatenation. Residual representations corresponding to N learnable queries. By concatenating the vectors, we obtain the original global scene representation vector: ;
[0081] Step S52: Dimensionality Reduction and Compression Mapping. To obtain a more compact global description and reduce storage and retrieval overhead, this embodiment uses a linear layer to modify the original global scene representation vector. Perform dimensionality reduction and compression;
[0082] Step S53: Normalization. To improve the discriminative power and numerical stability in the retrieval scenario, this embodiment performs L2 normalization on the compressed global scene representation vector to obtain a global scene representation vector with unit norm.
[0083] In this embodiment, the global scene representation vector is applied to a scene recognition task. Specifically, it includes:
[0084] Obtain a database consisting of a set of reference images. And the query image to be retrieved Normalization preprocessing is performed separately to obtain reference normalized images and query normalized images;
[0085] The reference normalized image and the query normalized image are processed according to the above method to obtain the global scene representation vector of the reference image and the global scene representation vector of the query image.
[0086] By calculating the Euclidean distance between the global scene representation vectors corresponding to the query image and the reference image, the similarity between the two images is measured and matched. The smaller the distance, the higher the consistency between the two scenes, which is then used to generate search ranking results.
[0087] This embodiment employs the multi-source visual scene representation method based on query residual aggregation proposed in this invention. The model was trained on the GSV-Cities dataset and evaluated on multiple visual scene recognition benchmark datasets. The results are shown in Tables 1 and 2. As an evaluation metric, if, for a given query image, at least one of the first K reference images returned by the retrieval matches the ground truth (ground value) of that query image, then the query image is considered to be in the corresponding ground truth. Search successful under the specified criteria.
[0088] Table 1. Experimental results of different baseline methods on multiple visual scene recognition benchmark datasets.
[0089]
[0090] Table 1 presents the experimental results of different baseline methods on multiple visual scene recognition benchmark datasets. These benchmark datasets cover typical urban scenes as well as challenging environmental and perspective changes. Overall, the method described in this invention achieves superior results compared to existing methods on most datasets and metrics, demonstrating its strong discriminative ability, generalization ability, and applicability under complex scene conditions.
[0091] Table 2. Experimental results of different baseline methods on a robust visual scene recognition dataset.
[0092]
[0093] Table 2 presents the experimental results of different baseline methods on a robust visual scene recognition dataset. This type of dataset includes more significant viewpoint variations and temporal changes in the scene, placing higher demands on the long-term stability and cross-domain generalization ability of scene representation methods. In the aforementioned robustness tests, the method proposed in this invention also achieved the best overall results, especially in... The advantages are more obvious in terms of performance indicators. Therefore, it can be seen that even with significant domain shifts, the method proposed in this invention can maintain high matching reliability and generalization ability under different time spans, different acquisition devices, and different environmental conditions, and has a better robustness improvement effect compared to existing baseline methods.
[0094] Figure 2This paper presents examples of scene retrieval results using the multi-source visual scene representation method based on query residual aggregation described in this invention, compared with existing typical methods, under the same query image conditions. Each row in the figure uses the same query image as input and shows the retrieval results returned by each method. As can be seen from the figure, when there are complex situations such as changes in viewpoint, differences in illumination, or local occlusion, existing methods are easily affected by local salient regions or semantic noise, resulting in inconsistencies between the retrieval results and the scene corresponding to the query image. In contrast, the method of this invention can still return a reference image consistent with the ground truth of the query image under the same conditions, achieving correct retrieval. This intuitively demonstrates the robustness and discriminative ability of this invention in complex scene retrieval tasks.
[0095] The above description is merely a preferred embodiment of the present invention, used to illustrate the technical solution of the present invention, and does not constitute a limitation on the scope of protection of the present invention. It should be understood that any equivalent substitutions, minor adjustments, or improvements made by those skilled in the art without departing from the basic concept and principles of the present invention should fall within the scope of protection of the present invention.
Claims
1. A multi-source visual scene representation method based on query residual aggregation, characterized in that, Includes the following steps: The image to be processed is acquired and preprocessed to obtain a standardized image; The standardized image is input into the first visual encoder and the second visual encoder respectively to obtain the first local feature set and the second local feature set; complementary information from the second local feature set is injected into the first local feature set through residual correction to construct multimodal fusion local features; The multimodal fusion local features are input into the query residual aggregator for feature aggregation. Based on multiple learnable query vectors, the residual representation corresponding to each query vector is calculated. All residual representations are concatenated, dimensionality reduced, and normalized to output a global scene representation vector.
2. The multi-source visual scene representation method based on query residual aggregation according to claim 1, characterized in that, The first visual encoder is a self-supervised visual encoder, and the second visual encoder is a visual-language encoder.
3. The multi-source visual scene representation method based on query residual aggregation according to claim 2, characterized in that, The multimodal fusion local features are constructed using the following formula: in, This is the first local feature set. For the second local feature set, For mapping networks.
4. The multi-source visual scene representation method based on query residual aggregation according to claim 3, characterized in that, Before calculating the multimodal fusion local features, the first local feature set and the second local feature set are L2 normalized in the channel dimension respectively; the mapping network is configured as follows: when the local features are in sequence form, it is implemented by a fully connected layer; when the local features are in feature map form, it is implemented by a 1×1 convolutional layer.
5. The multi-source visual scene representation method based on query residual aggregation according to claim 4, characterized in that, The input to the query residual aggregator for feature aggregation includes the following steps: Construct N learnable query vectors ; Application of local features in multimodal fusion Convolutional layers are used to reduce channel dimensions. Subsequently, multimodal fusion local features are represented from feature map form as a sequence and LayerNorm is applied. query vector Considering each group of multimodal fusion local features as a cluster center, a soft partition is performed on the local feature space. and query Calculate the soft allocation weights between the two. ; According to the soft allocation weight The residual representation is obtained by weighted aggregation of the deviations of local features relative to the query vector. .
6. The multi-source visual scene representation method based on query residual aggregation according to claim 5, characterized in that, The formula for calculating soft-assignment weights is: Where d is the feature dimension. Represents the similarity score Normalize.
7. The multi-source visual scene representation method based on query residual aggregation according to any one of claims 1-6, characterized in that, The method further includes a step of visual scene recognition using the global scene representation vector, specifically: Retrieve the query image to be searched and the reference image from the database; The global scene representation vector of the query image and the global scene representation vector of the reference image are generated respectively using the method described in any one of claims 1-6; Calculate the similarity measure between the representation vectors of the query image and the reference image, sort the reference images according to the magnitude of the similarity measure, and determine the scene that matches the query image to complete scene recognition.
8. A multi-source visual scene recognition system based on query residual aggregation that implements the method as described in claim 7, characterized in that, include: The image acquisition module acquires the query image and reference image to be retrieved; The vector calculation module is used to calculate the global scene representation vector of the query image and the global scene representation vector of the reference image; The scene recognition module is used to calculate the similarity measure between the representation vectors of the query image and the reference image, sort the reference images according to the magnitude of the similarity measure, and determine the scene that matches the query image to complete the scene recognition.
9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method as described in claim 7.
10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the method as described in claim 7.