A remote sensing image visual positioning method and related equipment

Abstract

The present invention belongs to the field of image processing and discloses a remote sensing image visual positioning method and related equipment. The method extracts text features and visual features from remote sensing images and closely combines the two based on a cross-attention approach, effectively improving the alignment of image features and text information. A pre-built progressive diffusion module using an inverse U-net architecture is then used to refine and denoise the target visual features. This not only further optimizes the image features but also improves the accuracy and robustness of coordinate positioning, thereby significantly improving the performance of remote sensing image visual positioning and enhancing positioning accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a remote sensing image visual positioning method and related equipment. Background Art

[0002] Remote sensing imagery based visual localization (RSVG) is an emerging research field that focuses on locating reference objects in remote sensing images using natural language descriptions. In recent years, with the significant improvement in the resolution of remote sensing images, the task of remote sensing image visual localization has attracted increasing attention.

[0003] When dealing with visual positioning tasks based on remote sensing images, common methods tend to use text and visual backbones to extract features. ResNet is usually used as the visual backbone to extract multi-scale visual features, and BERT is used as the text backbone to extract text features. The multi-scale Transformer encoder is then used to achieve feature interaction and semantic alignment. Finally, the image and text features after interaction are input into the Transformer decoder to obtain the target bounding box information. It can be seen that most current positioning methods are based on multi-scale Transformer encoders and decoders to achieve target bounding box positioning. Although the fusion of multi-scale features is taken into account, most of them use the image backbone network to extract multi-scale image features and then interact with the text features. They do not take into account that in the feature extraction process of the image backbone, the extracted image features are not closely aligned with the text information, and the image features corresponding to the text may be ignored or suppressed, making it difficult to obtain effective image features.

[0004] However, in the existing remote sensing image visual positioning method, during the feature extraction process of the image backbone, the extracted image features are not closely aligned with the text information, and the image features corresponding to the text are easily ignored or suppressed, making it difficult to obtain effective image features, which in turn leads to poor positioning accuracy. Summary of the Invention

[0005] The purpose of the present invention is to provide a remote sensing image visual positioning method and related equipment to solve the technical problem that the existing remote sensing image visual positioning method is not closely aligned with the text information, the image features corresponding to the text are easily ignored or suppressed, and it is difficult to obtain effective image features, which leads to poor positioning result accuracy.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a remote sensing image visual positioning method, comprising:

[0008] Extract text features and visual features from remote sensing images;

[0009] Combine text features with visual features based on cross-attention to obtain target visual features;

[0010] Based on the pre-built progressive diffusion module and the target visual features, the obtained noisy query is refined and denoised to obtain the coordinate positioning result of the reference object;

[0011] The progressive diffusion module adopts a reverse U-net architecture.

[0012] Furthermore, the process of extracting text features and visual features from the remote sensing image and combining the text features with the visual features based on the cross-attention method is implemented by a pre-built feature extraction module; wherein the feature extraction module includes a text backbone, a visual backbone, and a feature encoder;

[0013] The text backbone is used to extract text features from remote sensing images; the visual backbone is used to extract visual features from remote sensing images; and the feature encoder is used to combine text features with visual features in a cross-attention manner.

[0014] Furthermore, the specific steps of combining text features with visual features using a cross-attention approach include:

[0015] Projecting text features to match the channel dimensions of the outputs of the last multiple stages of the visual backbone;

[0016] Multi-head attention is used to realize the interaction between text features and visual features to obtain text-guided visual features. The specific process is expressed as follows;

[0017]

[0018] Where, represents the kth stage of ResNet, and They are the text features and visual features of the input cross attention layer, It is the visual feature guided by the text;

[0019] DETR encoder is used to encode visual features and project text features into hidden dimensions. In , the text features are aligned with the visual features to obtain the target visual features. The specific process is expressed as:

[0020]

[0021] Where, is the target visual feature, is the length of the visual feature sequence, and d is the hidden dimension of the processed visual features and text features.

[0022] Furthermore, the progressive diffusion module specifically includes a reverse U-Net architecture and a query-feature interaction mechanism;

[0023] The Inverse U-Net architecture mirrors the standard U-Net by starting from upsampling blocks and gradually moving to downsampling blocks. The diffusion paradigm introduced by the Inverse U-Net is to gradually refine the noisy query in each Inverse U-Net block. After processing through each Inverse U-Net block, the query refinement block uses a query-feature interaction mechanism to enhance the interaction between the query, image, and text features. Finally, the noisy query is guided towards the target direction.

[0024] Furthermore, the construction and training process of the progressive diffusion module is as follows:

[0025] The noise query is initialized using the true value coordinates, and the signal scaling function is used to scale the initialized noise query. The specific formula is as follows:

[0026]

[0027] Where, is the scaled transformed noise query, is the scale factor; is the signal scaling function; is the initialized noise query;

[0028] Noise is gradually added to the converted noise query according to the diffusion pattern, and the Markov chain of the diffusion forward process is defined. The specific formula of the forward noise process is as follows:

[0029]

[0030] Where, represents the noise variance strategy in DDPM; the diffusion forward process is passed to Add noise, initialize the query Convert to Potential noise query ;

[0031] Input the latent noise query into the reverse U-net, so that the latent noise query interacts with the visual and textual features in each reverse U-net block for refinement and denoising;

[0032] After the interaction is completed, the refined and denoised query is projected into a single channel to obtain the predicted coordinates;

[0033] The predicted coordinates are adjusted using the heavy signal function to flatten the height and width dimensions of the predicted coordinates to obtain the coordinate positioning results.

[0034] Furthermore, the potential noise query is input into the reverse U-net for refinement and denoising. The specific process is as follows:

[0035] The latent noisy queries with temporal embeddings are corrected using the inverse U-net residual block, and the sinusoidal position embeddings are used to convert the latent noisy queries into continuous features.

[0036] The multilayer perceptron maps continuous features to a multidimensional space and projects the embedded time to a multidimensional space to obtain the corrected potential noise query. The specific formula is as follows:

[0037]

[0038] Where, represents the corrected potential noise query; represents the output of the i-th level reverse U-Net block; Represents continuous features mapped to multi-dimensional space;

[0039] The modified latent noise query uses four multi-head attention layers to capture the location of the target visual features, and then interacts with the visual and textual features in each reverse U-Net block to achieve refinement and denoising.

[0040] The specific formula is as follows:

[0041]

[0042] Where, represents the potential noise query after preprocessing; represents the group regularization layer;

[0043] Add position embedding to both text features and target visual features, specifically:

[0044]

[0045] Where, Represents text features; Represents the target visual features; represents the function for obtaining learnable position embeddings; Indicates the text features of the added position embedding; represents the target visual feature with added position embedding;

[0046] The preprocessed latent noise query is processed through the cross attention layer Refine and use The convolutional layer transforms the preprocessed latent noise query Transform to the initial scale to obtain the refined and denoised query.

[0047] Furthermore, the output of each reverse U-Net block is processed by channel average pooling, and a multi-layer perceptron is used to predict auxiliary bounding boxes;

[0048] The smooth L1 loss and generalized cross-linking loss between the auxiliary bounding box and the true value box are calculated as the diffusion loss of the four-dimensional bounding box coordinates to output the correct bounding box coordinates.

[0049] In a second aspect, the present invention provides a remote sensing image visual positioning system, comprising:

[0050] A feature extraction unit, used to extract text features and visual features from remote sensing images;

[0051] A feature combination unit is used to combine text features with visual features based on a cross-attention approach to obtain target visual features;

[0052] A coordinate positioning unit is used to refine and denoise the acquired noise query based on the pre-built progressive diffusion module and the target visual features to obtain the coordinate positioning result of the reference object;

[0053] The progressive diffusion module adopts a reverse U-net architecture.

[0054] In a third aspect, the present invention provides a device comprising:

[0055] memory for storing computer programs;

[0056] A processor is used to implement the steps of the above-mentioned remote sensing image visual positioning method when executing the computer program.

[0057] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the steps of the above-mentioned remote sensing image visual positioning method.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] The present invention provides a remote sensing image visual positioning method. The method extracts text features and visual features from remote sensing images and closely combines the two based on a cross-attention approach, making full use of text information, thereby achieving more accurate object positioning and effectively improving the alignment of image features and text information. Then, a pre-built progressive diffusion module using an inverse U-net architecture is used to refine and denoise the target visual features to obtain a coordinate positioning result of a reference object. The method adopts a progressive inference training pipeline and provides a simpler and more effective method for generating a reference object frame in remote sensing visual positioning by continuously refining and denoising the target query. The method not only further optimizes the image features, but also improves the accuracy and robustness of coordinate positioning, thereby significantly improving the performance of remote sensing image visual positioning and improving positioning accuracy.

[0060] At the same time, this method provides a new progressive language-guided diffusion model based on Inverse U-Net; by taking the coordinates of the actual target box as the input of the Inverse U-Net and adding noise to these boxes as part of the diffusion process, the model is trained to reverse this process and use textual features and visual features for denoising at the same time; therefore, the model is able to gradually and accurately obtain the coordinates of the target box.

[0061] In addition, this method introduces a feature extraction module that continuously uses text information to guide visual features throughout the feature extraction process, ensuring a more robust and context-consistent interaction between visual and text features. At the same time, in order to enable the noise query to fully interact with visual and text features during the reverse U-Net denoising process, the model's language understanding ability is enhanced, thereby obtaining more accurate target boxes.

[0062] Preferably, in the present invention, the output of each reverse U-Net block adopts channel average pooling processing, and a multi-layer perceptron is used to predict the auxiliary bounding box; the above-mentioned diffusion-based training loss design can gradually refine the query and more effectively output the correct bounding box coordinates, further improving the query accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 An overall schematic diagram of a remote sensing image visual positioning method provided by an embodiment of the present invention;

[0064] Figure 2 An overall structural diagram of a progressive diffusion module provided by an embodiment of the present invention;

[0065] Figure 3 A schematic diagram of the structure of each reverse U-Net block in the progressive diffusion module provided by an embodiment of the present invention;

[0066] Figure 4A schematic diagram of the query-optimized cross-attention structure in the diffuse reverse U-Net block provided in an embodiment of the present invention;

[0067] Figure 5 A flowchart of a remote sensing image visual positioning method provided by the present invention;

[0068] Figure 6 This is a structural diagram of a remote sensing image visual positioning system provided by the present invention. DETAILED DESCRIPTION

[0069] Example 1

[0070] As described in the background, existing remote sensing image visual localization methods lack sufficient text guidance, resulting in a semantic gap between the two. Furthermore, the extracted image features are not closely aligned with the text information, and image features corresponding to the text may be ignored or suppressed during the feature extraction process of the image backbone. Furthermore, existing methods do not pay enough attention to generating bounding boxes. Using the MLP layer to directly regress the target bounding box may result in insufficient understanding of semantic information, leading to incorrect labeling or missed detections. Furthermore, progressive bounding box regression models often have complex structures, resulting in higher complexity. Furthermore, research has introduced diffusion models in natural scene visual localization tasks, but in remote sensing image-based visual localization tasks, how to achieve interaction between bounding box queries with image and text features based on the diffusion paradigm remains unexplored. Therefore, it is crucial to fully interact between image and text information. Designing progressive reasoning models that focus more on generating bounding boxes is crucial for correct localization. Designing diffusion-based remote sensing image visual localization models is crucial for the development of the field of remote sensing image visual localization.

[0071] To solve the above problems, the present invention provides a remote sensing image visual localization method. This method can utilize a cross-attention layer to guide the visual backbone through text features, and add query refinement cross-attention to the reverse U-Net framework to fully utilize text information for object localization. This method also provides a diffusion-based remote sensing image visual localization model, which realizes the interaction between bounding box queries with image and text features on the basis of the diffusion paradigm, and introduces reverse U-Net as a decoder to simplify the model structure. This method also solves the problem of insufficient attention to bounding box generation through a new diffusion-based method.

[0072] This embodiment provides a remote sensing image visual positioning method, including the following steps:

[0073] Extract text features and visual features from remote sensing images;

[0074] Combine text features with visual features based on cross-attention to obtain target visual features;

[0075] Based on the pre-built progressive diffusion module and the target visual features, the obtained noisy query is refined and denoised to obtain the coordinate positioning result of the reference object;

[0076] The progressive diffusion module adopts a reverse U-net architecture.

[0077] Specifically, the overall principle framework of this method is as follows Figure 1 As shown in the figure, it specifically includes a feature extraction module, a progressive diffusion module, and a diffusion-based training loss design. The specific principles are as follows:

[0078] The feature extraction module is used to extract and effectively integrate visual features and text features in remote sensing images. The text features are extracted using pre-trained text backbones. ,in Indicates the maximum text sequence length. Then, the text information is integrated into each stage of the visual backbone through cross-attention, and the text features are directly introduced into the visual backbone; the above method is used to enhance the feature extraction process by combining relevant text clues. Then, the pre-trained feature encoder processes the visual features , the visual features guided by text Convert to relevant target features ; Secondly, the reverse U-Net architecture is used to design the progressive diffusion module. Generate Noise Query , and use the signal scaling function Then, we calculate the time embedding and position embedding and add them to the features respectively. and input query. Then, cross attention is used to integrate the visual features and text features Combine and refine noise queries Finally, a heavy signal scaling function is applied to the denoised query Generate output coordinates.

[0079] This embodiment provides a specific implementation process of the remote sensing image visual positioning method, including a feature extraction module and a progressive diffusion module for implementing the above method steps, wherein:

[0080] First, feature extraction module

[0081] like Figure 1As shown, this embodiment uses pre-trained visual backbone and text backbone to extract features from line input. For the input remote sensing image, this embodiment uses ResNet50 to extract visual features from the last stage output with a spatial stride of 32. ,in , , The output shape of the visual features in the final stage. For the input language expression, BERT is used as the text backbone. This example obtains the output from the last BERT layer and obtains the text features. ,in Indicates the maximum text sequence length, is the text feature dimension. In this embodiment, , .

[0082] Next, we incorporate cross-attention directly into the visual backbone; this allows the visual features to be continuously guided by the textual information throughout the feature extraction process, ensuring a more robust and context-consistent interaction between image and text features. The specific steps are as follows: First, we project the textual features to match the final The channel dimension of the output of multiple stages of the stage; then, multi-head attention is applied to realize the interaction between visual and text features, and its process can be expressed as

[0083]

[0084] Where, represents the kth stage of ResNet, and are the text and visual features input to the cross-attention layer, is the visual feature of text guidance. This embodiment uses .

[0085] Fully utilize text information in the visual backbone and extract visual features from the output of the last stage After that, the pre-trained encoder is introduced to further refine the visual features; in this embodiment, the encoder preferably adopts the DETR encoder. At the same time, the text features are projected to the hidden dimension To align it with the visual features for further processing, this process can be expressed as

[0086]

[0087] in, , is the length of the visual feature sequence, is the latent dimension of the processed visual and textual features.

[0088] Second, progressive diffusion module

[0089] Different from the conventional method of generating coordinates by manually designing a multi-scale cross-attention decoder, this embodiment proposes a progressive diffusion module using a reverse U-Net architecture. Figure 2 As shown in Figure 3, progressive diffusion can gradually refine randomly initialized noisy queries (also called noise queries), integrate visual and text features, and finally obtain accurate target coordinates.

[0090] The progressive diffusion module consists of two key components: an inverted U-Net architecture and a query-feature interaction mechanism. Using an inverted version of the U-Net as the diffusion architecture, it mirrors the standard U-Net by starting with upsampling blocks and progressively moving toward downsampling blocks. The inverted U-Net introduces a diffusion paradigm that gradually refines the noisy query at each block. After processing the query through each inverted U-Net block, a query refinement block is used to enhance the interaction between the query, image, and text features, guiding the query toward the target. This entire process iteratively improves the query, gradually guiding it toward the precise target location.

[0091] During training, noise queries First, use the true value coordinates Initialization. After that, use a signal method To expand the input query, the process can be expressed as

[0092]

[0093] in, is the transformed query, is the scaling factor; in this embodiment, set .

[0094] Then, according to the diffusion pattern, the Markov chain of the diffusion forward process is defined by gradually adding noise to the initial query. The forward noise process is defined as

[0095]

[0096] in, represents the noise variance strategy in DDPM. The forward diffusion process is carried out by Add noise, initialize the query Convert to Potential noise query .

[0097] Afterwards, the potential noise query Put it into the reverse U-Net for further processing. As with diffusion in image generation, convolution blocks are used to transform Convert to ,in, represents the output of the i-th level inverse U-Net block. Figure 3 The details of each diffusion block are explained. First, the noisy query with time embedding is corrected using the U-Net residual block; for a given time step , similar to DETR, the discrete value is converted into a continuous feature using sinusoidal position embedding. Then, this continuous feature is mapped to a dimension of space, get . After that, embed The time projection to , and then added to the input query, where Indicates the channel size of the i-th level reverse U-Net block. The specific expression is as follows:

[0098]

[0099] Then, four multi-head attention layers are designed to help the noisy query capture the visual feature locations related to the text target. Figure 4 As shown, Preprocessing and visual features and text features Interaction, this embodiment uses Convolutional layer, Transform and reshape into , where d is the hidden dimension of the cross-attention transformer.

[0100]

[0101] in, Represents a group regularization layer.

[0102] At the same time, position embeddings are added to both textual features and visual features.

[0103]

[0104] In the formula represents a function for obtaining learnable position embeddings.

[0105] Afterwards, through the cross attention layer Refine and use The convolutional layer will Transform to the initial scale. This process can be expressed as

[0106]

[0107] in, Represents a query ,key ,value The multi-head attention layer between the LayerNorm layer;

[0108]

[0109] yes The dimension of .

[0110] Finally, after interacting with the visual and textual features in each reverse U-Net block, the query Project to a single channel. In order to get the predicted coordinates , using the heavy signal method Make the final adjustments and flatten the height and width dimensions.

[0111] In general, the adjustment process can be expressed as

[0112]

[0113] in, , is the predicted denoised coordinate Constrained to be in the range [0,1].

[0114] In this embodiment, in the specific application stage of the progressive diffusion module corresponding to the remote sensing image visual positioning, following the diffusion paradigm, first randomly initialize a noisy query , then use right to zoom in or out.

[0115] Afterwards, similar to the training phase, the scaled query Input into the reverse U-Net, which gradually refines and denoises the query, and finally obtains the final output .

[0116] In this embodiment, smooth L1 loss and generalized intersection over Union (GIoU) ​​loss are used as diffusion losses for the four-dimensional bounding box coordinates; following the diffusion paradigm, during the training process, by minimizing the training loss , training model from predict .

[0117]

[0118] in, is the true value coordinate, Output denoised queries for the model, , is the balance coefficient, let , ; is the auxiliary training loss. Specifically, in each query refinement block (reverse U-Net block) of the reverse U-Net, channel average pooling is applied to the output and a multi-layer perceptron (MLP) is used to predict auxiliary bounding boxes. Then, the smooth-L1 and GIoU losses are calculated between the auxiliary bounding boxes and the true value boxes, enabling the model to gradually refine the query and output the correct bounding box coordinates more efficiently. This process can be expressed as:

[0119]

[0120] in, Refine the auxiliary bounding box of the patch for each query, where , .

[0121] It can be seen that the key technical points of the remote sensing image visual positioning method provided in this embodiment are as follows:

[0122] First, a deep text guidance mechanism is proposed. This mechanism uses a cross-attention layer to guide the visual backbone with text features, and incorporates query refinement cross-attention into the reverse U-Net framework to fully utilize text information, thereby achieving more accurate object localization.

[0123] Secondly, we propose a novel diffusion-based approach to address the lack of attention paid to bounding box generation. Specifically, our approach uses a progressive inference training pipeline to provide a simpler and more efficient method for generating reference object boxes for remote sensing visual localization by continuously refining and denoising the object query.

[0124] Finally, a reverse U-Net was designed as a decoder, simplifying the model structure. By using query coordinates as the input to the reverse U-Net, the problem of a large input size causing a sharp increase in model size when generating images from the U-Net is effectively avoided. Furthermore, in the cross-attention mechanism of the query refinement block, visual and textual features are used as keys and values, enabling the query to fully interact with both image and text features. This not only simplifies the decoder structure but also unifies the decoder methods, avoiding the complexity of purely transformer-based decoders, reducing the difficulty of model design, and facilitating further research.

[0125] Example 2

[0126] like Figure 5 As shown, this embodiment provides a remote sensing image visual positioning method, including the following steps:

[0127] Extract text features and visual features from remote sensing images;

[0128] Combine text features with visual features based on cross-attention to obtain target visual features;

[0129] Based on the pre-built progressive diffusion module and the target visual features, the obtained noisy query is refined and denoised to obtain the coordinate positioning result of the reference object;

[0130] The progressive diffusion module adopts a reverse U-net architecture.

[0131] The process of extracting text features and visual features from remote sensing images and combining text features with visual features based on a cross-attention approach is implemented by a pre-built feature extraction module; wherein the feature extraction module includes a text backbone, a visual backbone, and a feature encoder;

[0132] The text backbone is used to extract text features from remote sensing images; the visual backbone is used to extract visual features from remote sensing images; and the feature encoder is used to combine text features with visual features in a cross-attention manner.

[0133] Specifically, the specific steps of combining text features with visual features using a cross-attention approach include:

[0134] Projecting text features to match the channel dimensions of the outputs of the last multiple stages of the visual backbone;

[0135] Multi-head attention is used to realize the interaction between text features and visual features to obtain text-guided visual features. The specific process is expressed as follows;

[0136]

[0137] Where, represents the kth stage of ResNet, and They are the text features and visual features of the input cross attention layer, It is the visual feature guided by the text;

[0138] DETR encoder is used to encode visual features and project text features into hidden dimensions. In , the text features are aligned with the visual features to obtain the target visual features. The specific process is expressed as:

[0139]

[0140] Where, is the target visual feature, is the length of the visual feature sequence, d is the hidden dimension of the processed visual features and text features; the purpose of projecting the text features into the hidden dimension d is to facilitate the dimension alignment of the diffusion module for cross-attention.

[0141] In addition, the progressive diffusion module specifically includes a reverse U-Net architecture and a query-feature interaction mechanism;

[0142] The Inverse U-Net architecture mirrors the standard U-Net by starting from upsampling blocks and gradually moving to downsampling blocks. The diffusion paradigm introduced by the Inverse U-Net is to gradually refine the noisy query in each Inverse U-Net block. After processing through each Inverse U-Net block, the query refinement block uses a query-feature interaction mechanism to enhance the interaction between the query, image, and text features. Finally, the noisy query is guided towards the target direction.

[0143] Here, the construction and training process of the progressive diffusion module is as follows:

[0144] The noise query is initialized using the true value coordinates, and the signal scaling function is used to scale the initialized noise query. The specific formula is as follows:

[0145]

[0146] Where, is the scaled transformed noise query, is the scale factor; is the signal scaling function; is the initialized noise query;

[0147] Noise is gradually added to the converted noise query according to the diffusion pattern, and the Markov chain of the diffusion forward process is defined. The specific formula of the forward noise process is as follows:

[0148]

[0149] Where, represents the noise variance strategy in DDPM; the diffusion forward process is passed to Add noise, initialize the query Convert to Potential noise query ;

[0150] Input the latent noise query into the reverse U-net, so that the latent noise query interacts with the visual and textual features in each reverse U-net block for refinement and denoising;

[0151] After the interaction is completed, the refined and denoised query is projected into a single channel to obtain the predicted coordinates;

[0152] The predicted coordinates are adjusted using the heavy signal function to flatten the height and width dimensions of the predicted coordinates to obtain the coordinate positioning results.

[0153] The specific process of inputting the potential noise query into the reverse U-net for refinement and denoising is as follows:

[0154] The latent noisy queries with temporal embeddings are corrected using the inverse U-net residual block, and the sinusoidal position embeddings are used to convert the latent noisy queries into continuous features.

[0155] The multilayer perceptron maps continuous features to a multidimensional space and projects the embedded time to a multidimensional space to obtain the corrected potential noise query. The specific formula is as follows:

[0156]

[0157] Where, represents the corrected potential noise query; represents the output of the i-th level reverse U-Net block; Represents continuous features mapped to multi-dimensional space;

[0158] The modified latent noise query uses four multi-head attention layers to capture the location of the target visual features, and then interacts with the visual and textual features in each reverse U-Net block to achieve refinement and denoising.

[0159] The specific formula is as follows:

[0160]

[0161] Where, represents the potential noise query after preprocessing; represents the group regularization layer;

[0162] Add position embedding to both text features and target visual features, specifically:

[0163]

[0164] Where, Represents text features; Represents the target visual features; represents the function for obtaining learnable position embeddings; Indicates the text features of the added position embedding; represents the target visual feature with added position embedding;

[0165] The preprocessed latent noise query is processed through the cross attention layer Refine and use The convolutional layer transforms the preprocessed latent noise query Transform to the initial scale to obtain the refined and denoised query.

[0166] like Figure 6 As shown, this embodiment also provides a remote sensing image visual positioning system, including: a feature extraction unit, used to extract text features and visual features in remote sensing images; a feature combination unit, used to combine text features with visual features based on a cross-attention method to obtain target visual features; a coordinate positioning unit, used to refine and denoise the acquired noise query based on a pre-built progressive diffusion module combined with the target visual features to obtain the coordinate positioning result of the reference object; wherein the progressive diffusion module adopts an inverse U-net architecture.

[0167] The present invention also provides a device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the remote sensing image visual positioning method when executing the computer program.

[0168] When the processor executes the computer program, the steps of visual positioning of the remote sensing image are implemented, for example: extracting text features and visual features from the remote sensing image; combining the text features with the visual features based on a cross-attention method to obtain target visual features; refining and denoising the obtained noise query based on a pre-built progressive diffusion module combined with the target visual features to obtain the coordinate positioning result of the reference object; the progressive diffusion module adopts an inverse U-net architecture.

[0169] Alternatively, when the processor executes the computer program, the functions of each module in the above-mentioned system are realized, for example: a feature extraction unit, used to extract text features and visual features in remote sensing images; a feature combination unit, used to combine text features with visual features based on a cross-attention method to obtain target visual features; a coordinate positioning unit, used to refine and denoise the acquired noise query based on a pre-built progressive diffusion module combined with the target visual features to obtain the coordinate positioning result of the reference object; wherein the progressive diffusion module adopts an inverse U-net architecture.

[0170] Exemplarily, the computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete preset functions, and the instruction segments are used to describe the execution process of the computer program in the remote sensing image visual positioning device. For example, the computer program can be divided into a feature extraction unit, a feature combination unit and a coordinate positioning unit; the specific functions of each module are as follows: a feature extraction unit, which is used to extract text features and visual features in remote sensing images; a feature combination unit, which is used to combine text features with visual features based on a cross-attention method to obtain target visual features; a coordinate positioning unit, which is used to refine and denoise the acquired noise query based on a pre-built progressive diffusion module combined with target visual features to obtain the coordinate positioning result of the reference object; wherein, the progressive diffusion module adopts an inverse U-net architecture.

[0171] The remote sensing image visual positioning device can be a computing device such as a desktop computer, laptop, PDA, or cloud server. The remote sensing image visual positioning device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the above are examples of remote sensing image visual positioning devices and do not constitute a limitation on remote sensing image visual positioning devices. The remote sensing image visual positioning device may include more components than those described above, or a combination of certain components, or different components. For example, the remote sensing image visual positioning device may also include input and output devices, network access devices, buses, etc.

[0172] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The processor serves as the control center of the remote sensing image visual positioning system, connecting various components of the remote sensing image visual positioning system using various interfaces and circuits.

[0173] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the remote sensing image visual positioning device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory.

[0174] The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as sound playback or image playback); the data storage area may store data generated based on the use of the mobile phone (such as audio data and a phone book). Furthermore, the memory may include high-speed random access memory (RAM) and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0175] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the remote sensing image visual positioning method are implemented.

[0176] If the integrated modules / units of the remote sensing image visual positioning system are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0177] Based on this understanding, the present invention implements all or part of the processes in the above-mentioned remote sensing image visual positioning method by means of a computer program that instructs related hardware. The computer program may be stored in a computer-readable storage medium, and when executed by a processor, the computer program may implement the steps of the above-mentioned remote sensing image visual positioning method. The computer program includes computer program code, which may be in source code form, object code form, executable file, or a pre-set intermediate form.

[0178] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0179] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable storage media do not include electrical carrier signals and telecommunication signals.

[0180] In summary, the present invention provides a remote sensing image visual positioning method and related equipment, namely, a remote sensing image visual positioning method based on diffusion model language-guided step-by-step query optimization, which can be widely used in various scenarios such as military target detection and intelligence identification, disaster monitoring and rescue, agriculture, and urban planning. The method's ability to provide geographic spatial information using text descriptions can make remote sensing image visual positioning more user-friendly, making it easier for non-professionals to access and understand information from remote sensing images. The feature extraction module and progressive diffusion module proposed in the present invention enhance the performance of the model while ensuring a relatively simple structural design, and provide a diffusion-based remote sensing image visual positioning paradigm. This can be applied to subsequent remote sensing image visual positioning research and has an inspiring effect on similar tasks such as remote sensing target detection and remote sensing semantic segmentation.

[0181] The above embodiment is only one of the implementation methods that can realize the technical solution of the present invention. The scope of protection claimed by the present invention is not limited only to this embodiment, but also includes changes, replacements and other implementation methods that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention.

Claims

1. A remote sensing image visual positioning method, characterized in that: include: Extract text features and visual features from remote sensing images; Combine text features with visual features based on cross-attention to obtain target visual features; Based on the pre-built progressive diffusion module and the target visual features, the obtained noisy query is refined and denoised to obtain the coordinate positioning result of the reference object; Wherein, the progressive diffusion module adopts a reverse U-net architecture; The construction and training process of the progressive diffusion module is as follows: The noise query is initialized using the true value coordinates, and the signal scaling function is used to scale the initialized noise query. The specific formula is as follows: Where, is the noise query after scaling transformation, α scale is the scaling factor; F signal (·) is the signal scaling function; q0 is the initialized noise query; Noise is gradually added to the converted noise query according to the diffusion pattern, and the Markov chain of the diffusion forward process is defined. The specific formula of the forward noise process is as follows: Where, β s represents the noise variance strategy in DDPM; the diffusion forward process is passed to Add noise, initialize the query Convert to a latent noise query t∈{0,1,...,T} Input the latent noise query into the reverse U-net, so that the latent noise query interacts with the visual and textual features in each reverse U-net block for refinement and denoising; After the interaction is completed, the refined and denoised query is projected into a single channel to obtain the predicted coordinates; The predicted coordinates are adjusted using the heavy signal function to flatten the height and width dimensions of the predicted coordinates to obtain the coordinate positioning results.

2. The remote sensing image visual positioning method according to claim 1, characterized in that: The process of extracting text features and visual features from the remote sensing image and combining the text features with the visual features based on the cross-attention method is implemented by a pre-built feature extraction module; wherein the feature extraction module includes a text backbone, a visual backbone, and a feature encoder; The text backbone is used to extract text features from remote sensing images; the visual backbone is used to extract visual features from remote sensing images; and the feature encoder is used to combine text features with visual features in a cross-attention manner.

3. The remote sensing image visual positioning method according to claim 2, characterized in that: The specific steps of combining text features with visual features using cross-attention include: Projecting text features to match the channel dimensions of the outputs of the last multiple stages of the visual backbone; Multi-head attention is used to realize the interaction between text features and visual features to obtain text-guided visual features. The specific process is expressed as follows; Where k represents the kth stage of ResNet, and They are the text features and visual features of the input cross attention layer, It is the visual feature guided by the text; The DETR encoder is used to encode the visual features, and the text features are projected into the hidden dimension d to align the text features with the visual features to obtain the target visual features. The specific process is expressed as: x l =Proj(x l ) Where, is the target visual feature, s v =h×w is the length of the visual feature sequence, and d is the hidden dimension of the processed visual features and text features.

4. The remote sensing image visual positioning method according to claim 1, characterized in that: The progressive diffusion module specifically includes a reverse U-Net architecture and a query-feature interaction mechanism; The Inverse U-Net architecture mirrors the standard U-Net by starting from upsampling blocks and gradually moving to downsampling blocks. The diffusion paradigm introduced by the Inverse U-Net is to gradually refine the noisy query in each Inverse U-Net block. After processing through each Inverse U-Net block, the query refinement block uses a query-feature interaction mechanism to enhance the interaction between the query, image, and text features. Finally, the noisy query is guided towards the target direction.

5. The remote sensing image visual positioning method according to claim 1, characterized in that: The specific process of inputting the potential noise query into the reverse U-net for refinement and denoising is as follows: The latent noisy queries with temporal embeddings are corrected using the inverse U-net residual block, and the sinusoidal position embeddings are used to convert the latent noisy queries into continuous features. The multilayer perceptron maps continuous features to a multidimensional space and projects the embedded time to a multidimensional space to obtain the corrected potential noise query. The specific formula is as follows: Where, represents the corrected potential noise query; q i represents the output of the i-th level reverse U-Net block; t emb Represents continuous features mapped to multi-dimensional space; The modified latent noise query uses four multi-head attention layers to capture the location of the target visual features, and then interacts with the visual and textual features in each reverse U-Net block to achieve refinement and denoising. The specific formula is as follows: Where, represents the potential noise query after preprocessing; GN(·) represents the group regularization layer; Add position embedding to both text features and target visual features, specifically: Where x l Represents text features; Represents the target visual features; pos(·) represents the function to obtain learnable position embeddings; Indicates the text features of the added position embedding; represents the target visual feature with added position embedding; The preprocessed latent noise query is processed through the cross attention layer The preprocessed latent noise query is refined using a 1×1 convolutional layer. Transform to the initial scale to obtain the refined and denoised query.

6. The remote sensing image visual positioning method according to claim 4, characterized in that: The output of each reverse U-Net block is processed by channel average pooling, and a multi-layer perceptron is used to predict auxiliary bounding boxes; The smooth L1 loss and generalized cross-linking loss between the auxiliary bounding box and the true value box are calculated as the diffusion loss of the four-dimensional bounding box coordinates to output the correct bounding box coordinates.

7. A remote sensing image visual positioning system, used to implement the steps of the remote sensing image visual positioning method according to any one of claims 1 to 6, characterized in that: include: A feature extraction unit, used to extract text features and visual features from remote sensing images; A feature combination unit is used to combine text features with visual features based on a cross-attention approach to obtain target visual features; A coordinate positioning unit is used to refine and denoise the acquired noise query based on the pre-built progressive diffusion module and the target visual features to obtain the coordinate positioning result of the reference object; The progressive diffusion module adopts a reverse U-net architecture.

8. A remote sensing image visual positioning device, characterized in that: include: memory for storing computer programs; A processor is used to implement the steps of the remote sensing image visual positioning method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the steps of the remote sensing image visual positioning method according to any one of claims 1 to 6.

Images