Remote sensing image visual positioning method and related equipment

Abstract

The invention belongs to the field of image processing, and discloses a remote sensing image visual positioning method and related device.According to the method, text features and visual features in a remote sensing image are extracted and closely combined based on a cross attention mode, and the alignment degree of the image features and text information is effectively improved; then, a pre-constructed progressive diffusion module adopting a reverse U-net architecture is used for carrying out refining and denoising processing on the visual features of the target, so that the image features are further optimized, the accuracy and robustness of coordinate positioning are improved, and the performance of remote sensing image visual positioning is remarkably improved; and the positioning accuracy is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a remote sensing image visual positioning method and related equipment. Background Art

[0002] Remote sensing imagery based visual localization (RSVG) is an emerging research field that focuses on locating reference objects in remote sensing images using natural language descriptions; in recent years, with the significant improvement in the resolution of remote sensing images, the remote sensing image visual localization task has attracted more and more attention.

[0003] When dealing with visual localization tasks based on remote sensing images, common methods tend to use text and visual backbones to extract features. ResNet is usually used as the visual backbone to extract multi-scale visual features, and BERT is used as the text backbone to extract text features. Then, a multi-scale Transformer encoder is used to achieve feature interaction and semantic alignment. Finally, the image and text features after interaction are input into the Transformer decoder to obtain the target bounding box information. It can be seen that most of the current localization methods are based on multi-scale Transformer encoders and decoders to achieve the positioning of the target bounding box. Although the fusion of multi-scale features is taken into account, most of them use the image backbone network to extract multi-scale image features and then interact with the text features. They do not take into account that in the feature extraction process of the image backbone, the extracted image features are not closely aligned with the text information, and the image features corresponding to the text may be ignored or suppressed, making it difficult to obtain effective image features. However, in the existing remote sensing image visual positioning method, during the feature extraction process of the image backbone, the extracted image features are not closely aligned with the text information, and the image features corresponding to the text are easily ignored or suppressed, making it difficult to obtain effective image features, which in turn leads to poor positioning result accuracy. Summary of the invention

[0004] The purpose of the present invention is to provide a remote sensing image visual positioning method and related equipment to solve the technical problem that the existing remote sensing image visual positioning method has the problem that the extracted image features are not closely aligned with the text information, the image features corresponding to the text are easily ignored or suppressed, and it is difficult to obtain effective image features, which leads to poor positioning result accuracy.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides a remote sensing image visual positioning method, comprising: Extract textual and visual features from remote sensing images; Combine text features with visual features based on cross-attention to obtain target visual features; Based on the pre-built progressive diffusion module and the target visual features, the acquired noise query is refined and denoised to obtain the coordinate positioning result of the reference object; Wherein, the progressive diffusion module adopts a reverse U-net architecture.

[0006] Furthermore, the process of extracting text features and visual features from the remote sensing image and combining the text features with the visual features based on the cross-attention method is implemented by a pre-built feature extraction module; wherein the feature extraction module includes a text backbone, a visual backbone and a feature encoder; The text backbone is used to extract text features in remote sensing images; the visual backbone is used to extract visual features in remote sensing images; and the feature encoder is used to combine text features with visual features in a cross-attention manner.

[0007] Furthermore, the specific steps of combining text features with visual features in a cross-attention manner include: Projecting text features to match the channel dimensions of the outputs of the last multiple stages of the visual backbone; Multi-head attention is used to realize the interaction between text features and visual features, and the text-guided visual features are obtained. The specific process is expressed as follows;

[0008] In the formula, represents the kth stage of ResNet, and They are the text features and visual features of the input cross-attention layer, It is the visual feature guided by the text; The DETR encoder is used to encode visual features and project text features into hidden dimensions. In , the text features are aligned with the visual features to obtain the target visual features. The specific process is expressed as:

[0009] In the formula, is the target visual feature, is the length of the visual feature sequence, and d is the hidden dimension of the processed visual features and text features.

[0010] Furthermore, the progressive diffusion module specifically includes a reverse U-Net architecture and a query-feature interaction mechanism; The reverse U-Net architecture is used to mirror the standard U-Net by starting from the upsampling block and gradually moving to the downsampling block; wherein the diffusion paradigm introduced by the reverse U-Net is: gradually refining the noisy query of each reverse U-Net block; after being processed by each reverse U-Net block, using the query refinement block of the query-feature interaction mechanism to enhance the interaction between the query, image and text features; and finally guiding the noisy query to develop in the target direction.

[0011] Furthermore, the construction and training process of the progressive diffusion module is as follows: The true value coordinates are used to initialize the noise query, and the signal scaling function is used to scale the initialized noise query. The specific formula is as follows:

[0012] In the formula, is the scaled transformed noise query, is the scale factor; is the signal scaling function; is the initialized noise query; According to the diffusion mode, noise is gradually added to the converted noise query, and the Markov chain of the diffusion forward process is defined. The specific formula of the forward noise process is as follows:

[0013] In the formula, represents the noise variance strategy in DDPM; the diffusion forward process passes through Add noise, initialize the query Convert to Potential noise query ; Input the latent noise query into the reverse U-net so that the latent noise query interacts with the visual and text features in each reverse U-net block for refinement and denoising. After the interaction is completed, the refined and denoised query is projected into a single channel to obtain the predicted coordinates; The predicted coordinates are adjusted using the heavy signal function, so that the height and width dimensions of the predicted coordinates are flattened to obtain the coordinate positioning result.

[0014] Furthermore, the potential noise query is input into the reverse U-net for refinement and denoising. The specific process is as follows: The latent noisy queries with temporal embeddings are corrected using the inverse U-net residual block and converted into continuous features using sinusoidal position embeddings. The multi-layer perceptron maps continuous features to a multi-dimensional space and projects the embedded time to a multi-dimensional space to obtain a corrected potential noise query. The specific formula is as follows:

[0015] In the formula, represents the corrected potential noise query; represents the output of the i-th level reverse U-Net block; Represents continuous features mapped to multi-dimensional space; The modified latent noise query uses four multi-head attention layers to capture the location of the target visual features, and then interacts with the visual and textual features in each reverse U-Net block to achieve refinement and denoising; The specific formula is as follows:

[0016] In the formula, represents the potential noisy query after preprocessing; represents the group regularization layer; Add position embedding to both text features and target visual features, specifically:

[0017] In the formula, Represents text features; Represents the target visual features; represents the function for obtaining learnable position embeddings; Text features representing the added position embedding; Indicates the target visual feature of adding position embedding; The preprocessed latent noise query is processed by the cross attention layer Refine and use The convolutional layer transforms the preprocessed latent noise query Transform to the initial scale to obtain a refined and denoised query.

[0018] Furthermore, the output of each reverse U-Net block is processed by channel average pooling, and a multi-layer perceptron is used to predict auxiliary bounding boxes; The smooth L1 loss and generalized cross-linking loss between the auxiliary bounding box and the true value box are calculated as the diffusion loss of the four-dimensional bounding box coordinates to output the correct bounding box coordinates.

[0019] In a second aspect, the present invention provides a remote sensing image visual positioning system, comprising: A feature extraction unit, used to extract text features and visual features from remote sensing images; A feature combining unit, used for combining text features with visual features based on a cross-attention approach to obtain target visual features; A coordinate positioning unit is used to refine and denoise the acquired noise query based on a pre-built progressive diffusion module combined with target visual features to obtain a coordinate positioning result of a reference object; Wherein, the progressive diffusion module adopts a reverse U-net architecture.

[0020] In a third aspect, the present invention provides a device, comprising: Memory for storing computer programs; A processor is used to implement the steps of the above-mentioned remote sensing image visual positioning method when executing the computer program.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the steps of the above-mentioned remote sensing image visual positioning method.

[0022] Compared with the prior art, the present invention has the following beneficial effects: The invention provides a remote sensing image visual positioning method. The method extracts text features and visual features from the remote sensing image, and closely combines the two based on a cross-attention method, so as to make full use of text information, thereby achieving more accurate object positioning and effectively improving the alignment degree between image features and text information; then, a pre-built progressive diffusion module using a reverse U-net architecture is used to refine and denoise the target visual features to obtain a coordinate positioning result of a reference object; the method adopts a progressive reasoning training pipeline, and provides a simpler and more effective method for generating a reference object frame in remote sensing visual positioning by continuously refining and denoising the target query; the method not only further optimizes the image features, but also improves the accuracy and robustness of coordinate positioning, thereby significantly improving the performance of remote sensing image visual positioning; and improving the positioning accuracy.

[0023] At the same time, this method provides a new progressive language-guided diffusion model based on Inverse U-Net; by taking the coordinates of the actual target box as the input of the Inverse U-Net and adding noise to these boxes as part of the diffusion process, the model is trained to reverse this process and use both textual features and visual features for denoising; therefore, the model is able to gradually and accurately obtain the coordinates of the target box.

[0024] In addition, this method introduces a feature extraction module, which continuously guides the visual features with text information throughout the feature extraction process, ensuring a more robust and context-consistent interaction between visual and text features; at the same time, in order to make the noise query fully interact with the visual and text features in the reverse U-Net denoising process, the model's language understanding ability is enhanced, thereby obtaining a more accurate target box.

[0025] Preferably, in the present invention, the output of each reverse U-Net block adopts channel average pooling processing, and a multi-layer perceptron is used to predict the auxiliary bounding box; the above-mentioned diffusion-based training loss design is adopted to gradually refine the query and more effectively output the correct bounding box coordinates, thereby further improving the query accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 An overall schematic diagram of a remote sensing image visual positioning method provided by an embodiment of the present invention; Figure 2 An overall structural diagram of a progressive diffusion module provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of each reverse U-Net block in the progressive diffusion module provided in an embodiment of the present invention; Figure 4 A schematic diagram of a query optimized cross attention structure in a diffuse reverse U-Net block provided in an embodiment of the present invention; Figure 5 A flowchart of a remote sensing image visual positioning method provided by the present invention; Figure 6 The present invention provides a schematic diagram of the structure of a remote sensing image visual positioning system. DETAILED DESCRIPTION

[0027] Example 1 As described in the background technology, currently, the existing remote sensing image visual positioning methods lack sufficient text guidance, resulting in a semantic gap between the two. At the same time, the extracted image features are not closely aligned with the text information. In the feature extraction process of the image backbone, the image features corresponding to the text may be ignored or suppressed; secondly, the existing methods do not pay enough attention to the generation of bounding boxes. Using the MLP layer to directly regress the target bounding box may lead to insufficient understanding of the semantic information, resulting in incorrect labeling or missed detection. At the same time, the progressive bounding box regression model often has a complex structure, resulting in higher complexity; in addition, there have been studies on the introduction of diffusion models in natural scene visual positioning tasks, but in the visual positioning task based on remote sensing images, how to realize the interaction between bounding box queries with image and text features based on the introduction of the diffusion paradigm has not yet been explored; it can be seen that how to fully interact between image and text information is crucial; how to design a progressive reasoning model that pays more attention to the generation of bounding boxes is crucial for correct positioning; how to design a diffusion-based remote sensing image visual positioning model is crucial for the development of the field of remote sensing image visual positioning.

[0028] In order to solve the above problems, the present invention provides a remote sensing image visual positioning method. The method can use a cross-attention layer to guide the visual backbone through text features, and add query refinement cross-attention in the reverse U-Net framework to make full use of text information for object positioning; the method also provides a remote sensing image visual positioning model based on diffusion, which realizes the interaction between bounding box queries with image and text features on the basis of the diffusion paradigm, and introduces the reverse U-Net as a decoder to simplify the model structure; the method also solves the problem of insufficient attention to bounding box generation through a new diffusion-based method.

[0029] This embodiment provides a remote sensing image visual positioning method, including the following steps: Extract textual and visual features from remote sensing images; Combine text features with visual features based on cross-attention to obtain target visual features; Based on the pre-built progressive diffusion module and the target visual features, the acquired noise query is refined and denoised to obtain the coordinate positioning result of the reference object; Wherein, the progressive diffusion module adopts a reverse U-net architecture.

[0030] Specifically, the overall principle framework of this method is as follows Figure 1 As shown in the figure, it specifically includes feature extraction module, progressive diffusion module and diffusion-based training loss design. The specific principles are as follows: The feature extraction module is used to extract and effectively integrate visual features and text features in remote sensing images. The text features are extracted using the pre-trained text backbone. ,in Indicates the maximum text sequence length. Then, the text information is integrated into each stage of the visual backbone by cross-attention, and the text features are directly introduced into the visual backbone; the above method is used to enhance the feature extraction process by combining relevant text clues. Then, the pre-trained feature encoder processes the visual features , the visual features guided by the text Transform to relevant target features ; Secondly, the reverse U-Net architecture is used to design the progressive diffusion module. Generate Noise Query , and use the signal scaling function Then, we calculate the time embedding and position embedding and add them to the features respectively. and input query. Then, cross attention is used to integrate the visual features and text features Combine and refine noise query Finally, the heavy signal scaling function is applied to the denoised query Generate output coordinates.

[0031] This embodiment provides a specific implementation process of the remote sensing image visual positioning method, including a feature extraction module and a progressive diffusion module for implementing the above method steps, wherein: First, feature extraction module like Figure 1 As shown, this embodiment uses the pre-trained visual backbone and text backbone to extract features from the line input. For the input remote sensing image, this embodiment uses ResNet50 to extract visual features from the final stage output with a spatial step size of 32. ,in , , is the output shape of the visual features in the final stage. For the input language expression, BERT is used as the text backbone. This example obtains the output from the last BERT layer and obtains the text features ,in Indicates the maximum text sequence length, is the text feature dimension. In this embodiment, , .

[0032] Next, we incorporate cross-attention directly into the visual backbone; this allows the visual features to be continuously guided by the textual information throughout the feature extraction process, ensuring a more robust and context-consistent interaction between image and text features. The specific steps are as follows: First, we project the text features to match the final The channel dimension of the output of multiple stages of the stage; then, multi-head attention is applied to realize the interaction between visual and text features, and its process can be expressed as

[0033] In the formula, represents the kth stage of ResNet, and are the text and visual features input to the cross-attention layer, is a visual feature guided by text. This embodiment uses .

[0034] Make full use of text information in the visual backbone and extract visual features from the output of the last stage After that, the pre-trained encoder is introduced to further refine the visual features; in this embodiment, the encoder preferably adopts the DETR encoder. At the same time, the text features are projected to the hidden dimension In order to align it with the visual features for further processing, this process can be expressed as

[0035] in, , is the length of the visual feature sequence, is the latent dimension of the processed visual and textual features.

[0036] Second, progressive diffusion module Different from the conventional method of generating coordinates by manually designing a multi-scale cross-attention decoder, this embodiment proposes a progressive diffusion module using a reverse U-Net architecture. Figure 2 As shown in Figure 3, progressive diffusion can gradually refine randomly initialized noisy queries (also called noise queries), integrate visual and text features, and finally obtain accurate target coordinates.

[0037] The progressive diffusion module consists of two key components: the reverse U-Net architecture and the query-feature interaction mechanism. Using a reverse version of U-Net as the diffusion architecture, it mirrors the standard U-Net by starting from the upsampling block and gradually moving to the downsampling block; the reverse U-Net introduces a diffusion paradigm that gradually refines the noisy query for each block. After processing the query through each reverse U-Net block, a query refinement block is used to enhance the interaction between the query, image, and text features, guiding the query towards the target direction. The whole process can iteratively improve the query, gradually guiding it to the precise target location.

[0038] During training, noise queries First, use the true value coordinates Initialize. After that, use a signal method To expand the input query, the process can be expressed as

[0039] in, is the transformed query, is the scaling factor; in this embodiment, set .

[0040] Then, according to the diffusion pattern, the Markov chain of the diffusion forward process is defined by gradually adding noise to the initialization query. The forward noise process is defined as

[0041] in, represents the noise variance strategy in DDPM. The diffusion forward process is Add noise, initialize the query Convert to Potential noise query .

[0042] Afterwards, the potential noise query Put it into the reverse U-Net for further processing. As with diffusion in image generation, convolution blocks are used to transform Convert to ,in, represents the output of the i-th level reverse U-Net block. Figure 3 The details of each diffusion block are explained. First, the noisy query with temporal embedding is corrected using the U-Net residual block; for a given time step , similar to DETR, the discrete value is converted into a continuous feature using sinusoidal position embedding. Then, this continuous feature is mapped to a dimension of space, get . After that, embed The time projection to , and then added to the input query, where Represents the channel size of the i-th level reverse U-Net block. The specific expression is as follows:

[0043] Then, four multi-head attention layers are designed to help the noisy query capture the visual feature locations related to the text target. Figure 4 As shown, Preprocessing and visual features and text features Interaction, this embodiment uses Convolutional layer, Transform and reshape into , where d is the hidden dimension of the cross-attention transformer.

[0044]

[0045] in, Represents a group regularization layer.

[0046] At the same time, position embeddings are added to both textual features and visual features.

[0047]

[0048] In the formula represents a function for obtaining learnable position embeddings.

[0049] Afterwards, through the cross attention layer Refine and use The convolutional layer will Transform to the initial scale. This process can be expressed as

[0050] in, Representation query ,key ,value The multi-head attention layer between the LayerNorm layer;

[0051] yes The dimension of .

[0052] Finally, after interacting with the visual and textual features in each reverse U-Net block, the query Project to a single channel. To get the predicted coordinates , using the heavy signal method Make final adjustments and flatten the height and width dimensions.

[0053] In general, the adjustment process can be described as

[0054] in, , is the predicted denoised coordinate Constrained to be in the range [0,1].

[0055] In this embodiment, in the specific application stage of the progressive diffusion module corresponding to the remote sensing image visual positioning, following the diffusion paradigm, first randomly initialize a noisy query , then use right to zoom in or out.

[0056] Afterwards, similar to the training phase, the scaled query Input to the reverse U-Net, which gradually refines and denoises the query, and finally obtains the final output .

[0057] In this embodiment, smooth L1 loss and generalized cross-link (GIoU) ​​loss are used as diffusion losses of four-dimensional bounding box coordinates; following the diffusion paradigm, during the training process, by minimizing the training loss , the training model is from predict .

[0058]

[0059] in, is the true value coordinate, Output denoised queries for the model, , is the balance coefficient, , ; is the auxiliary training loss. Specifically, in each query refinement block (reverse U-Net block) of the reverse U-Net, channel average pooling is applied to the output, and a multi-layer perceptron (MLP) is used to predict auxiliary bounding boxes; then, the smooth-L1 and GIoU losses between the auxiliary bounding boxes and the true value boxes are calculated, enabling the model to gradually refine the query and output the correct bounding box coordinates more efficiently. The process can be expressed as:

[0060] in, Refine the auxiliary bounding box of the patch for each query, where , .

[0061] It can be seen that the key technical points of the remote sensing image visual positioning method provided in this embodiment are as follows: Firstly, a deep text-guided mechanism is proposed, which uses a cross-attention layer to guide the visual backbone through text features and incorporates query refinement cross-attention in the reverse U-Net framework to fully utilize text information, thereby achieving more accurate object localization.

[0062] Secondly, we propose a novel diffusion-based approach to address the lack of attention paid to bounding box generation. Specifically, our approach uses a progressive inference training pipeline to provide a simpler and more efficient way to generate reference object boxes for remote sensing visual localization by continuously refining and denoising the object query.

[0063] Finally, a reverse U-Net is designed as a decoder to simplify the model structure; by using the query coordinates as the input of the reverse U-Net, the problem of a sharp increase in model size due to the large input size when U-Net generates images is effectively avoided. At the same time, in the cross-attention mechanism of the query refinement block, visual and text features are used as keys and values, allowing the query to fully interact with image and text features. Not only does it simplify the decoder structure, but it also unifies the decoder method, avoiding the complexity of a pure transformer-based decoder, reducing the difficulty of model design, and facilitating further research.

[0064] Example 2 like Figure 5 As shown, this embodiment provides a remote sensing image visual positioning method, including the following steps: Extract textual and visual features from remote sensing images; Combine text features with visual features based on cross-attention to obtain target visual features; Based on the pre-built progressive diffusion module and the target visual features, the acquired noise query is refined and denoised to obtain the coordinate positioning result of the reference object; Wherein, the progressive diffusion module adopts a reverse U-net architecture.

[0065] The process of extracting text features and visual features from remote sensing images and combining text features with visual features based on a cross-attention approach is implemented by a pre-built feature extraction module; wherein the feature extraction module includes a text backbone, a visual backbone, and a feature encoder; The text backbone is used to extract text features in remote sensing images; the visual backbone is used to extract visual features in remote sensing images; and the feature encoder is used to combine text features with visual features in a cross-attention manner.

[0066] Specifically, the specific steps of combining text features with visual features using a cross-attention approach include: Projecting text features to match the channel dimensions of the outputs of the last multiple stages of the visual backbone; Multi-head attention is used to realize the interaction between text features and visual features, and the text-guided visual features are obtained. The specific process is expressed as follows;

[0067] In the formula, represents the kth stage of ResNet, and They are the text features and visual features of the input cross-attention layer, It is the visual feature guided by the text; The DETR encoder is used to encode visual features and project text features into hidden dimensions. In , the text features are aligned with the visual features to obtain the target visual features. The specific process is expressed as:

[0068] In the formula, is the target visual feature, is the length of the visual feature sequence, d is the hidden dimension of the processed visual features and text features; the purpose of projecting the text features into the hidden dimension d is to facilitate the dimensional alignment of the diffusion module for cross-attention.

[0069] In addition, the progressive diffusion module specifically includes a reverse U-Net architecture and a query-feature interaction mechanism; The reverse U-Net architecture is used to mirror the standard U-Net by starting from the upsampling block and gradually moving to the downsampling block; wherein the diffusion paradigm introduced by the reverse U-Net is: gradually refining the noisy query of each reverse U-Net block; after being processed by each reverse U-Net block, using the query refinement block of the query-feature interaction mechanism to enhance the interaction between the query, image and text features; and finally guiding the noisy query to develop in the target direction.

[0070] Here, the construction and training process of the progressive diffusion module is as follows: The true value coordinates are used to initialize the noise query, and the signal scaling function is used to scale the initialized noise query. The specific formula is as follows:

[0071] In the formula, is the scaled transformed noise query, is the scale factor; is the signal scaling function; is the initialized noise query; According to the diffusion mode, noise is gradually added to the converted noise query, and the Markov chain of the diffusion forward process is defined. The specific formula of the forward noise process is as follows:

[0072] In the formula, represents the noise variance strategy in DDPM; the diffusion forward process passes through Add noise, initialize the query Convert to Potential noise query ; Input the latent noise query into the reverse U-net so that the latent noise query interacts with the visual and text features in each reverse U-net block for refinement and denoising. After the interaction is completed, the refined and denoised query is projected into a single channel to obtain the predicted coordinates; The predicted coordinates are adjusted using the heavy signal function, so that the height and width dimensions of the predicted coordinates are flattened to obtain the coordinate positioning result.

[0073] The specific process of inputting the potential noise query into the reverse U-net for refinement and denoising is as follows: The latent noisy queries with temporal embeddings are corrected using the inverse U-net residual block and converted into continuous features using sinusoidal position embeddings. The multi-layer perceptron maps continuous features to a multi-dimensional space and projects the embedded time to a multi-dimensional space to obtain a corrected potential noise query. The specific formula is as follows:

[0074] In the formula, represents the corrected potential noise query; represents the output of the i-th level reverse U-Net block; Represents continuous features mapped to multi-dimensional space; The modified latent noise query uses four multi-head attention layers to capture the location of the target visual features, and then interacts with the visual and textual features in each reverse U-Net block to achieve refinement and denoising; The specific formula is as follows:

[0075] In the formula, represents the potential noisy query after preprocessing; represents the group regularization layer; Add position embedding to both text features and target visual features, specifically:

[0076] In the formula, Represents text features; Represents the target visual features; represents the function for obtaining learnable position embeddings; Text features representing the added position embedding; Indicates the target visual feature of adding position embedding; The preprocessed latent noise query is processed by the cross attention layer Refine and use The convolutional layer transforms the preprocessed latent noise query Transform to the initial scale to obtain a refined and denoised query.

[0077] like Figure 6 As shown, this embodiment also provides a remote sensing image visual positioning system, including: a feature extraction unit, used to extract text features and visual features in remote sensing images; a feature combination unit, used to combine text features with visual features based on a cross-attention method to obtain target visual features; a coordinate positioning unit, used to refine and denoise the acquired noise query based on a pre-built progressive diffusion module combined with the target visual features to obtain the coordinate positioning result of the reference object; wherein the progressive diffusion module adopts an inverse U-net architecture.

[0078] The present invention also provides a device, comprising: a memory for storing a computer program; and a processor for implementing the steps of the remote sensing image visual positioning method when executing the computer program.

[0079] When the processor executes the computer program, the above-mentioned steps of remote sensing image visual positioning are implemented, for example: extracting text features and visual features in the remote sensing image; combining the text features with the visual features based on a cross-attention method to obtain the target visual features; based on a pre-built progressive diffusion module combined with the target visual features, the obtained noise query is refined and denoised to obtain the coordinate positioning result of the reference object; the progressive diffusion module adopts a reverse U-net architecture.

[0080] Alternatively, when the processor executes the computer program, the functions of each module in the above system are realized, for example: a feature extraction unit, used to extract text features and visual features in remote sensing images; a feature combination unit, used to combine text features with visual features based on a cross-attention method to obtain target visual features; a coordinate positioning unit, used to refine and denoise the acquired noise query based on a pre-built progressive diffusion module combined with the target visual features to obtain the coordinate positioning result of the reference object; wherein the progressive diffusion module adopts an inverse U-net architecture.

[0081] Exemplarily, the computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete preset functions, and the instruction segments are used to describe the execution process of the computer program in the remote sensing image visual positioning device. For example, the computer program can be divided into a feature extraction unit, a feature combination unit, and a coordinate positioning unit; the specific functions of each module are as follows: a feature extraction unit, which is used to extract text features and visual features in remote sensing images; a feature combination unit, which is used to combine text features with visual features based on a cross-attention method to obtain target visual features; a coordinate positioning unit, which is used to refine and denoise the acquired noise query based on a pre-built progressive diffusion module combined with target visual features to obtain the coordinate positioning result of the reference object; wherein the progressive diffusion module adopts a reverse U-net architecture.

[0082] The remote sensing image visual positioning device can be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The remote sensing image visual positioning device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the above are examples of remote sensing image visual positioning devices and do not constitute a limitation on remote sensing image visual positioning devices. More components than the above may be included, or some components may be combined, or different components may be included. For example, the remote sensing image visual positioning device may also include input and output devices, network access devices, buses, etc.

[0083] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The processor is the control center of the remote sensing image visual positioning, and uses various interfaces and lines to connect various parts of the entire remote sensing image visual positioning device.

[0084] The memory can be used to store the computer program and / or module, and the processor implements various functions of the remote sensing image visual positioning device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.

[0085] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0086] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the remote sensing image visual positioning method are implemented.

[0087] If the integrated module / unit of the remote sensing image visual positioning system is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0088] Based on such understanding, the present invention implements all or part of the processes in the above remote sensing image visual positioning method, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above remote sensing image visual positioning method can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or preset intermediate form, etc.

[0089] The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0090] It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electrical carrier signals and telecommunication signals.

[0091] In summary, the present invention provides a remote sensing image visual positioning method and related equipment, that is, a remote sensing image visual positioning method based on diffusion model language guided step-by-step query optimization, which can be widely used in military target detection and intelligence identification, disaster monitoring and rescue, agriculture, urban planning and other scenarios; the ability of this method to provide geographic spatial information using text description can make remote sensing image visual positioning more user-friendly, making it easier for non-professionals to access and understand information from remote sensing images. The feature extraction module and progressive diffusion module proposed in the present invention enhance the performance of the model while ensuring a relatively simple structural design, and provide a diffusion-based remote sensing image visual positioning paradigm, which can be applied to subsequent remote sensing image visual positioning research work, and has an inspiring effect on remote sensing target detection, remote sensing semantic segmentation and other similar tasks.

[0092] The above embodiment is only one of the implementation methods that can realize the technical solution of the present invention. The scope of protection claimed by the present invention is not limited only to this embodiment, but also includes changes, replacements and other implementation methods that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention.

Claims

1. A remote sensing image visual positioning method, characterized in that: include: Extract textual and visual features from remote sensing images; Combine text features with visual features based on cross-attention to obtain target visual features; Based on the pre-built progressive diffusion module and the target visual features, the acquired noise query is refined and denoised to obtain the coordinate positioning result of the reference object; Wherein, the progressive diffusion module adopts a reverse U-net architecture.

2. The remote sensing image visual positioning method according to claim 1, characterized in that: The process of extracting text features and visual features from the remote sensing image and combining the text features with the visual features based on the cross-attention method is implemented by a pre-built feature extraction module; wherein the feature extraction module includes a text backbone, a visual backbone and a feature encoder; The text backbone is used to extract text features in remote sensing images; the visual backbone is used to extract visual features in remote sensing images; and the feature encoder is used to combine text features with visual features in a cross-attention manner.

3. The remote sensing image visual positioning method according to claim 2, characterized in that: The specific steps of combining text features with visual features in a cross-attention manner include: Projecting text features to match the channel dimensions of the outputs of the last multiple stages of the visual backbone; Multi-head attention is used to realize the interaction between text features and visual features, and the text-guided visual features are obtained. The specific process is expressed as follows; In the formula, represents the kth stage of ResNet, and They are the text features and visual features of the input cross-attention layer, It is the visual feature guided by the text; The DETR encoder is used to encode visual features and project text features into hidden dimensions. In , the text features are aligned with the visual features to obtain the target visual features. The specific process is expressed as: In the formula, is the target visual feature, is the length of the visual feature sequence, and d is the hidden dimension of the processed visual features and text features.

4. The remote sensing image visual positioning method according to claim 1, characterized in that: The progressive diffusion module specifically includes a reverse U-Net architecture and a query-feature interaction mechanism; The reverse U-Net architecture is used to mirror the standard U-Net by starting from the upsampling block and gradually moving to the downsampling block; wherein the diffusion paradigm introduced by the reverse U-Net is: gradually refining the noisy query of each reverse U-Net block; after being processed by each reverse U-Net block, using the query refinement block of the query-feature interaction mechanism to enhance the interaction between the query, image and text features; and finally guiding the noisy query to develop in the target direction.

5. The remote sensing image visual positioning method according to claim 1, characterized in that: The construction and training process of the progressive diffusion module is as follows: The true value coordinates are used to initialize the noise query, and the signal scaling function is used to scale the initialized noise query. The specific formula is as follows: In the formula, is the scaled transformed noise query, is the scale factor; is the signal scaling function; is the initialized noise query; According to the diffusion mode, noise is gradually added to the converted noise query, and the Markov chain of the diffusion forward process is defined. The specific formula of the forward noise process is as follows: In the formula, represents the noise variance strategy in DDPM; the diffusion forward process passes through Add noise, initialize the query Convert to Potential noise query ; Input the latent noise query into the reverse U-net so that the latent noise query interacts with the visual and text features in each reverse U-net block for refinement and denoising. After the interaction is completed, the refined and denoised query is projected into a single channel to obtain the predicted coordinates; The predicted coordinates are adjusted using the heavy signal function, so that the height and width dimensions of the predicted coordinates are flattened to obtain the coordinate positioning result.

6. The remote sensing image visual positioning method according to claim 5, characterized in that: The specific process of inputting the potential noise query into the reverse U-net for refinement and denoising is as follows: The latent noisy queries with temporal embeddings are corrected using the inverse U-net residual block and converted into continuous features using sinusoidal position embeddings. The multi-layer perceptron maps continuous features to a multi-dimensional space and projects the embedded time to a multi-dimensional space to obtain a corrected potential noise query. The specific formula is as follows: In the formula, represents the corrected potential noise query; represents the output of the i-th level reverse U-Net block; Represents continuous features mapped to multi-dimensional space; The modified latent noise query uses four multi-head attention layers to capture the location of the target visual features, and then interacts with the visual and textual features in each reverse U-Net block to achieve refinement and denoising; The specific formula is as follows: In the formula, represents the potential noisy query after preprocessing; represents the group regularization layer; Add position embedding to both text features and target visual features, specifically: In the formula, Represents text features; Represents the target visual features; represents the function for obtaining learnable position embeddings; Text features representing the added position embedding; Indicates the target visual feature of adding position embedding; The preprocessed latent noise query is processed by the cross attention layer Refine and use The convolutional layer transforms the preprocessed latent noise query Transform to the initial scale to obtain a refined and denoised query.

7. The remote sensing image visual positioning method according to claim 5, characterized in that: The output of each reverse U-Net block is processed by channel average pooling, and a multi-layer perceptron is used to predict auxiliary bounding boxes; The smooth L1 loss and generalized cross-linking loss between the auxiliary bounding box and the true value box are calculated as the diffusion loss of the four-dimensional bounding box coordinates to output the correct bounding box coordinates.

8. A remote sensing image visual positioning system, characterized in that: include: A feature extraction unit, used to extract text features and visual features from remote sensing images; A feature combining unit, used for combining text features with visual features based on a cross-attention approach to obtain target visual features; A coordinate positioning unit is used to refine and denoise the acquired noise query based on a pre-built progressive diffusion module combined with target visual features to obtain a coordinate positioning result of a reference object; Wherein, the progressive diffusion module adopts a reverse U-net architecture.

9. A device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the remote sensing image visual positioning method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the steps of the remote sensing image visual positioning method according to any one of claims 1 to 7.

Images