Remote sensing image target recognition method and system based on visual language model

Abstract

The invention discloses a remote sensing image target recognition method and system based on a visual language model, and relates to the technical field of image recognition. The method comprises the following steps: constructing an initial image detection model based on a real-time open vocabulary detector YOLO-World and a remote sensing visual language basic model RemoteCLIP; adding a double-attention fusion encoder and a multi-level text guide fusion decoder on the initial image detection model to obtain a target image detection model; performing knowledge self-adaptive adjustment and optimization on the basis of the pre-training weight of the target image detection model to obtain a final image target recognition model; and performing target identification on the remote sensing image based on the image target identification model. According to the method, a powerful visual language model is introduced, and an image feature extraction and target classification strategy is optimized, so that the accuracy and efficiency of target recognition in the remote sensing image are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a remote sensing image target recognition method and system based on a visual language model. Background Art

[0002] In recent years, the rapid development of remote sensing technology has led to its widespread application in agricultural monitoring, urban management, disaster assessment and other fields. However, traditional remote sensing image target recognition methods often face the problems of low accuracy and low computational efficiency when dealing with complex scenes, changing environments and multiple target types.

[0003] With the continuous advancement of deep learning and computer vision technology, object recognition methods based on visual language models have gradually attracted attention. By combining image information with text information, visual models can better understand and process multimodal data. This method can not only improve the accuracy of object recognition, but also enhance the generalization ability of the model, so that it can still perform well in changing environments.

[0004] However, there are still some challenges in the current remote sensing image target recognition methods based on visual language models. First, the existing models have limited processing capabilities for high-resolution images and it is difficult to effectively capture detailed information. Second, in specific application scenarios, changes in the shape, color, and texture of the target may lead to a decrease in recognition accuracy. In addition, the application of existing methods in multi-target recognition and real-time processing is not mature enough. Therefore, the development of an efficient and accurate remote sensing image target recognition method and system based on a visual language model has important theoretical significance and practical application value. Summary of the invention

[0005] In order to overcome the above problems or at least partially solve the above problems, the present invention provides a remote sensing image target recognition method and system based on a visual language model, which optimizes image feature extraction and target classification strategies by introducing a powerful visual language model, thereby improving the accuracy and efficiency of target recognition in remote sensing images.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention provides a remote sensing image target recognition method based on a visual language model, comprising the following steps:

[0008] Build an initial image detection model based on the real-time open vocabulary detector YOLO-World and the remote sensing visual language foundation model RemoteCLIP;

[0009] Add a dual attention fusion encoder and a multi-level text-guided fusion decoder to the initial image detection model to obtain the target image detection model;

[0010] Based on the pre-trained weights of the target image detection model, knowledge adaptive tuning is performed to obtain the final image target recognition model;

[0011] Perform target recognition on remote sensing images based on image target recognition model.

[0012] The present invention adopts a combination of YOLO-World and RemoteCLIP, so that the model can more effectively identify and align text and visual features in remote sensing images. By using a reparameterizable visual language path aggregation network, the deployment capability of the model on edge devices is enhanced, and real-time and efficient detection is achieved. In order to better enhance feature extraction, a dual attention fusion encoder and a multi-level text-guided fusion decoder are added to the detection model, which further increases the ability to identify small targets. At the same time, the present invention performs knowledge adaptive tuning based on the pre-trained weights of the above-mentioned model, reduces the model's dependence on training data, and improves the model recognition performance.

[0013] Based on the first aspect, further, the method for constructing an initial image detection model based on the real-time open vocabulary detector YOLO-World and the remote sensing visual language basic model RemoteCLIP includes the following steps:

[0014] The remote sensing visual language basic model RemoteCLIP is used to replace CLIP in the real-time open vocabulary detector YOLO-World, and the initial image detection model is built based on the real-time open vocabulary detector YOLO-World.

[0015] Based on the first aspect, further, the above dual attention fusion encoder incorporates related class semantics into image features to enhance feature extraction.

[0016] Based on the first aspect, further, the above-mentioned multi-level text-guided fusion decoder adopts object query, multi-level class embedding and multi-scale image feature as input.

[0017] Based on the first aspect, further, the method for performing knowledge adaptive tuning based on the pre-trained weights of the target image detection model includes the following steps:

[0018] A Gaussian saliency guided adapter and a decoder layer adapter with prospective refractive learning are added to the pre-trained weights of the target image detection model to achieve knowledge adaptive tuning.

[0019] Based on the first aspect, further, the above-mentioned image target recognition model includes a YOLO detector, a text encoder and a reparameterizable visual language path aggregation network.

[0020] Based on the first aspect, further, the above-mentioned text editor, given a text Text, uses a RemoteCLIP pre-trained text encoder to replace the original CLIP text encoder to extract corresponding text embedding; when the input text is a title, a simple noun extraction algorithm is used to extract noun phrases and input them into the text encoder.

[0021] In a second aspect, the present invention provides a remote sensing image target recognition system based on a visual language model, comprising an initial model building module, a model optimization module, an adaptive tuning module and a target recognition module, wherein:

[0022] The initial model building module is used to build an initial image detection model based on the real-time open vocabulary detector YOLO-World and the remote sensing visual language foundation model RemoteCLIP;

[0023] A model optimization module is used to add a dual attention fusion encoder and a multi-level text-guided fusion decoder to the initial image detection model to obtain a target image detection model;

[0024] The adaptive tuning module is used to perform knowledge adaptive tuning based on the pre-trained weights of the target image detection model to obtain the final image target recognition model;

[0025] The target recognition module is used to perform target recognition on remote sensing images based on the image target recognition model.

[0026] This system uses a combination of YOLO-World and RemoteCLIP through the cooperation of multiple modules such as the initial model construction module, the model optimization module, the adaptive tuning module and the target recognition module, so that the model can more effectively identify and align text and visual features in remote sensing images. By using a reparameterizable visual language path aggregation network, the deployment capability of the model on edge devices is enhanced, and real-time and efficient detection is achieved. In order to better enhance feature extraction, a dual attention fusion encoder and a multi-level text-guided fusion decoder are added to the detection model, which further increases the ability to identify small targets. At the same time, the present invention performs knowledge adaptive tuning based on the pre-trained weights of the above-mentioned model to reduce the model's dependence on training data and improve the model recognition performance.

[0027] In a third aspect, the present application provides an electronic device, comprising a memory for storing one or more programs; a processor; when the one or more programs are executed by the processor, the method as described in any one of the first aspects above is implemented.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above.

[0029] The present invention has at least the following advantages or beneficial effects:

[0030] 1. Improved open vocabulary detection capabilities: The combination of YOLO-World and RemoteCLIP enables the model to more effectively identify and align text and visual features in remote sensing images. By using a reparameterizable visual language path aggregation network, the model's deployment capabilities on edge devices are enhanced, achieving real-time and efficient detection.

[0031] 2. Strengthening visual and semantic interaction: The introduction of dual attention fusion encoder and multi-level text-guided fusion decoder enables the model to not only focus on text descriptions when extracting features, but also effectively integrate class semantic information. At the same time, it can improve the recognition ability of small objects and enhance the overall feature extraction effect.

[0032] 3. Knowledge adaptive tuning: By adding Gaussian saliency guided adapters and decoder layer adapters based on pre-trained weights, the reliance on large amounts of training data is reduced. The adapter module can guide the model to focus on more relevant salient target features, thereby achieving efficient target recognition even on relatively small data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 A flowchart of a remote sensing image target recognition method based on a visual language model according to an embodiment of the present invention;

[0035] Figure 2 Schematic diagram of the architecture of an image target recognition model in an embodiment of the present invention;

[0036] Figure 3 1 is a structural diagram of RepVL-PAN in an embodiment of the present invention;

[0037] Figure 4 It is a structural diagram of a dual attention fusion encoder in an embodiment of the present invention;

[0038] Figure 5 It is a structural diagram of a multi-level text-guided fusion decoder in an embodiment of the present invention;

[0039] Figure 6 Schematic diagram of training steps based on an adaptive adapter in an embodiment of the present invention;

[0040] Figure 7 The figure is a schematic diagram showing the position of the Gaussian guided adapter in the backbone network and the detailed structure of the adapter in an embodiment of the present invention;

[0041] Figure 8 A schematic diagram of the location of the adapter of the decoding layer in the network and the detailed structure of the adapter in an embodiment of the present invention;

[0042] Fig. 9 A structural block diagram of an electronic device provided by an embodiment of the present invention.

[0043] Description of reference numerals: 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0045] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0047] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0048] In the description of the embodiments of the present invention, "plurality" means at least 2.

[0049] Example:

[0050] like Figure 1-Figure 8 As shown, in a first aspect, an embodiment of the present invention provides a remote sensing image target recognition method based on a visual language model, comprising the following steps:

[0051] S1. Build the initial image detection model based on the real-time open vocabulary detector YOLO-World and the remote sensing visual language basic model RemoteCLIP;

[0052] Furthermore, it includes: using the remote sensing visual language basic model RemoteCLIP to replace CLIP in the real-time open vocabulary detector YOLO-World, and building an initial image detection model based on the real-time open vocabulary detector YOLO-World.

[0053] In some embodiments of the present invention, an advanced real-time open vocabulary detector YOLO-World is first used to improve the efficiency and open set detection capabilities in real-world applications. The core idea is to enhance the detection capabilities of the YOLO series detectors for open vocabulary through visual language modeling and pre-training of large-scale data sets. A reparameterizable visual language path aggregation network (RepVL-PAN) is used to connect text and image features, and a region-based text contrast loss is introduced for pre-training. The training method is summarized as follows: first, data scaling / cropping, text clarity and standardization are performed, and the data is divided into training sets, validation sets and test sets; secondly, the overall model framework is constructed, including a YOLO detector, a text encoder and RepVL-PAN, wherein the text encoder encodes the input text into text embedding, the image encoder of the YOLO detector extracts multi-scale features from the input image, and RepVL-PAN realizes cross-modal fusion by fusing image features and text embedding; then, for each image, a corresponding region-text pair is generated according to its content. The region here can be a specific object or area in the image, and the text is a description of the region; finally, in the pre-training stage, the model parameters are optimized by comparing the similarity between the image features and the corresponding text features. In order to enhance the ability of the model to be effectively deployed on edge devices, the text encoder is deleted during the inference process, and the text embedding is re-parameterized as the weights of RepVL-PAN. Unlike YOLO-World, the visual language model used in the present invention uses the remote sensing visual language basic model RemoteCLIP (ViT-B-32) to replace CLIP (ViT-B-32) in World, so as to better learn text embeddings that are well aligned with remote sensing visual features, while learning robust features with rich remote sensing visual semantics.

[0054] S2. Add a double attention fusion encoder and a multi-level text-guided fusion decoder to the initial image detection model to obtain a target image detection model; the double attention fusion encoder incorporates related class semantics into image features to enhance feature extraction; the multi-level text-guided fusion decoder uses object query, multi-level class embedding and multi-scale image features as input.

[0055] In some embodiments of the present invention, a visual language model is introduced into the detector, focusing on enhancing text description, but it ignores the importance of visual-semantic interaction. In order to better enhance feature extraction, a dual attention fusion encoder and a multi-level text-guided fusion decoder are added to the initial image detection model. The dual attention fusion encoder incorporates relevant class semantics into image features to enhance feature extraction; the multi-level text-guided fusion decoder uses object queries, multi-level class embeddings, and multi-scale image features as inputs, which facilitates the extraction of image features associated with class embeddings from multi-scale image features, and further increases the ability to identify small targets. The structures of the dual attention fusion encoder and the multi-level text-guided fusion decoder are as follows: Figure 4 and Figure 5 shown.

[0056] The image and text features extracted by the RemoteCLIP encoder provide better visual-semantic relationships. This alignment can inject the corresponding class information into each image feature through cross-modal fusion. However, the background features have no class embedding. The dual attention fusion encoder in the present invention is a cross-attention module with both softmax and sigmoid activation functions, which enables the network to focus mainly on the foreground. Given the multi-scale image feature F i ∈R H×W×C From the image encoder and class embedding T∈R N×D , using cross attention to achieve cross-modal fusion. In order to suppress the merging of class embeddings into background features, the maximum similarity between image features and class embeddings is calculated, and the sigmoid function is applied to obtain the probability of each image feature being the foreground, which is then combined with the text feature representation T j (v) Multiply.

[0057] The multi-level text-guided fusion decoder in the example of the present invention utilizes multi-scale image features to enhance class embedding from the text encoder, and obtains multi-level class embedding, which not only includes class semantic information, but also includes visual feature information of objects corresponding to different levels, aiming to improve the detection capability of small objects.

[0058] Figure 7 The position of the adapter in the backbone network and the detailed structure of the adapter guided by Gaussian filtering are shown. The backbone adapter in the example of the present invention is inserted into the module of each backbone stage.

[0059] In the context of salient object detection or other related image processing tasks, Gaussian filters are often applied to input images to blur or smooth the image, an operation that helps suppress noise and fine details while preserving the overall structure of the image and larger-scale features. By convolving the image with a Gaussian kernel, the Gaussian filter effectively reduces the influence of high-frequency components and enhances the contribution of low-frequency components such as edges and larger image areas. It helps to highlight salient objects or areas in the image.

[0060] Figure 8 In order to determine the position of the adapter in the decoding layer and the detailed structure of the adapter, the decoding layer adapter in the present invention example is inserted into the feedforward network of each decoder layer. It consists of two fully connected layers, a nonlinear activation function and a scaling factor. The unique design of the adaptor is a parallel structure with the FFN layer in the encoder block, which helps to make the adaptation process easier to optimize. In the optimization process of the deep learning model, the latter layer is optimized based on the output of the previous layer. Therefore, the latter layer may have knowledge that the previous layer cannot see, especially in the adapter tuning process, all pre-trained layer weights are frozen, and only the adapter weights can be optimized. It is possible to consider using the knowledge of the latter adapter layer to help the previous adapter layer learn during the adapter tuning process. This work extracts the hidden representation of the downward projection and forwards it to the adapter of the next layer. The adapter of the next layer fuses the hidden representation with its input and the linear projection layer. The calculated gradient of the fused hidden representation can be directly back-propagated to the previous adapter to help it learn a higher-level representation, and the latter adapter layer can learn hierarchical information from the hidden representation of the previous adapter layer. The computed gradients of the fused hidden representation can be directly back-propagated to the previous adapter to help it learn higher-level representations, and the latter adapter layer can learn hierarchical information from the hidden representation of the previous adapter layer.

[0061] The decoding layer adapter can be expressed as follows:

[0062] x=LN(Linear(Concat(n,n')))

[0063] n"=x·Downsample

[0064] x'=RELU(LN(x·Downsample)·Upssaple)

[0065]

[0066] Where n is the normal input to the adapter, n′ is the hidden representation from the previous adapter layer, and n” is forwarded for fusion in the next adapter layer.

[0067] The constructed model is trained and tuned by first fine-tuning the adapter to constrain the adapter weights to a better point in the feature modulation space, and then fine-tuning the entire model in a full-parameter manner.

[0068] S3, performing knowledge adaptive tuning based on the pre-trained weights of the target image detection model to obtain the final image target recognition model;

[0069] Furthermore, it includes: adding a Gaussian saliency guided adapter and a decoder layer adapter with prospective refraction learning based on the pre-trained weights of the target image detection model to achieve knowledge adaptive tuning.

[0070] In some embodiments of the present invention, in order to reduce the model's dependence on training data and improve the model's recognition performance, the present invention proposes to perform knowledge adaptive tuning based on the pre-trained weights of the above-mentioned target image detection model, and add two adapter modules in the encoding and decoding process, namely a Gaussian saliency guided adapter and an adapter at the decoder layer with prospective refractive learning. The Gaussian guided adapter has the ability to guide Gaussian saliency information to focus on salient target features that are more relevant to the target in the remote sensing image, and the adapter at the decoder layer learns more hierarchical hidden representations. And it can achieve target recognition on a relatively small data set. The overall architecture of the image target recognition model of the present invention is as follows: Figure 2 shown.

[0071] The training steps based on adaptive adapter are as follows Figure 6 As shown, the base model is first pre-trained on a large public target detection dataset, especially a remote sensing public dataset. Then, the adapter is inserted into the base model, the pre-trained weights of the base model are frozen, and the adapter on the target dataset is fine-tuned. Finally, the example of the present invention fine-tunes the entire model in a full-parameter manner. The specific steps are as follows:

[0072] Pre-training phase: Use a large-scale unlabeled dataset to pre-train the YOLO-World model. The goal of this phase is to enable the model to master the statistical characteristics and basic knowledge of images and texts. During this process, the weights of the backbone adapter and the decoder adapter will be frozen and will not participate in training.

[0073] Adapter fine-tuning phase: After pre-training is completed, the model is applied to a small labeled dataset for a specific task. In this phase, only the backbone adapter and the decoder adapter are fine-tuned, while the weights of other parts of the pre-trained model are kept frozen. Specifically, the weights of the adapter can be optimized by inputting specific image and text pairs (such as remote sensing images and corresponding reports) and using methods such as region-text contrast loss.

[0074] Full parameter fine-tuning stage: After the adapter fine-tuning is completed, all parameters (including the weights of the pre-trained model and the weights of the adapter) can be further fine-tuned. The purpose of this stage is to make the entire model better suited to specific tasks. In this stage, the same dataset as the adapter fine-tuning can be used, or new data can be introduced as needed. At the same time, an appropriate loss function (such as cross entropy loss, IoU loss, etc.) can be selected to optimize model performance.

[0075] Evaluation and Adjustment: After each fine-tuning phase, the model needs to be evaluated to understand its performance on a specific task. This can be achieved by calculating metrics such as accuracy, recall, and F1 value. Based on the evaluation results, the model structure, hyperparameters, or training strategies are adjusted to further improve model performance.

[0076] S4. Perform target recognition on remote sensing images based on the image target recognition model.

[0077] The present invention adopts a combination of YOLO-World and RemoteCLIP, so that the model can more effectively identify and align text and visual features in remote sensing images. By using a reparameterizable visual language path aggregation network, the deployment capability of the model on edge devices is enhanced, and real-time and efficient detection is achieved. In order to better enhance feature extraction, a dual attention fusion encoder and a multi-level text-guided fusion decoder are added to the detection model, which further increases the ability to identify small targets. At the same time, the present invention performs knowledge adaptive tuning based on the pre-trained weights of the above-mentioned model, reduces the model's dependence on training data, and improves the model recognition performance.

[0078] Based on the first aspect, further, the above-mentioned image target recognition model includes a YOLO detector, a text encoder and a reparameterizable visual language path aggregation network.

[0079] Based on the first aspect, further, the above-mentioned text editor, given a text Text, uses a RemoteCLIP pre-trained text encoder to replace the original CLIP text encoder to extract corresponding text embedding; when the input text is a title, a simple noun extraction algorithm is used to extract noun phrases and input them into the text encoder.

[0080] In some embodiments of the present invention, Figure 2 As shown, the detector used in the present invention consists of a YOLO detector, a text encoder, and a reparameterizable visual language path aggregation network:

[0081] YOLO detector, including a backbone network as an image encoder, a path aggregation network (PAN) for multi-scale feature pyramids, and a head for bounding box regression and object embedding; text encoder, given a text Text, the RemoteCLIP pre-trained text encoder is used to replace the original CLIP text encoder to extract the corresponding text embedding. The RemoteCLIP text encoder provides better remote sensing-based visual-semantic functions to connect remote sensing visual objects and text. When the input text is a title, a simple noun extraction algorithm is used to extract noun phrases and input them into the text encoder; text comparison head, in order to achieve the interaction between visual and language information and promote the performance of zero-shot object detection, this mechanism of calculating cosine similarity between text features and image box features is introduced, which promotes the deep fusion between visual and language features and enhances the model's ability to represent visual semantics. Similar to YOLOv8, a decoupled head with two 3×3 convections is used to regress bounding boxes. In addition, in order to stabilize region-text training, an affine transformation with a learnable scale factor ɑ and a shift factor β is added, and L2-Norm is used for regularization. The model is trained using an online vocabulary. During the training process, an online vocabulary is built for each embedding sample containing 4 images. The main process is as follows:

[0082] (1) Constructing mosaic samples: During the training phase, each mosaic sample is composed of four images. These four images may contain different target objects.

[0083] (2) Construct an online vocabulary T: Positive nouns: Extract all positive nouns from the four images (i.e., the target object categories that actually exist in these images). Negative nouns: Randomly extract some negative nouns (i.e., the target object categories that do not exist in these images) from the corresponding dataset to increase the diversity of the vocabulary. Combined vocabulary: The online vocabulary T includes all positive nouns and the selected negative nouns. Vocabulary restriction: In order to control the size of the vocabulary, the vocabulary of each mosaic sample contains at most M nouns, which is set to 80 by default.

[0084] (3) Pre-training: The model is pre-trained using region-text contrast loss. This loss function optimizes the model parameters by comparing the similarity between image features and corresponding text features. During the pre-training process, a multi-scale training strategy can be adopted, that is, the model is trained at different resolutions to improve the robustness and generalization ability of the model.

[0085] For offline vocabulary reasoning, in the reasoning stage, the prompt post-detection strategy of the offline vocabulary is used to further improve the reasoning efficiency and accuracy by pre-encoding and storing prompt information. The specific steps are as follows:

[0086] Define custom cues: Users first need to define a series of custom cues. These cues can be titles or category names to guide the model to focus on specific objects or features.

[0087] Encode prompts: Use a pre-trained text encoder to encode custom prompts into text embeddings. This step converts human-readable language descriptions into numerical representations that machines can understand.

[0088] Build offline vocabulary: Store the encoded text embeddings as an offline vocabulary. There is no need to re-encode the prompts every time during the inference phase, saving computing resources and time.

[0089] Reparameterized RepVL-PAN: During inference, the text embeddings in the offline vocabulary are reparameterized into weights in RepVL-PAN. The model can utilize this pre-encoded text information to guide the extraction and fusion of visual features.

[0090] Perform inference: Use the re-parameterized RepVL-PAN network to perform inference on the input image and output predicted boxes and object embeddings. Since the model has already utilized text information in the offline vocabulary, it can more accurately identify and localize objects related to the prompt.

[0091] The structure of the above reparameterizable visual language path aggregation network RepVL-PAN is as follows Figure 3 As shown in the figure, it establishes a feature pyramid {P3, P4, P5} with multi-scale image features {C3, C4, C5}. Through the Top-Down path, information is transferred from high-level feature maps to low-level feature maps, so that the model has better accuracy when detecting smaller targets. Through the Bottom-Up path, information is transferred from low-level feature maps to high-level feature maps, which helps the model better retain detail information when detecting larger targets. In addition, text-guided CSPLayer (T-CSPLayer) and image pool attention are used to integrate text embedding into multi-scale image features, which enhances the interaction between image features and text features and improves the ability of visual semantic representation. The bottom-up feature propagation method is more efficient. The construction of a multi-scale feature pyramid can improve the accuracy of the model on multi-scale detection tasks, and at the same time improve the detection accuracy without affecting the detection speed.

[0092] During the inference process, by removing the text encoder and directly using the text embedding as part of the network (i.e., the weights of the convolutional layer or linear layer), the reliance on the text encoder during inference can be reduced, thereby accelerating the entire detection process. When the model is deployed to the edge device, there is no need to load and use the text encoder separately, which simplifies the model deployment process and reduces the demand for computing resources.

[0093] In a second aspect, an embodiment of the present invention provides a remote sensing image target recognition system based on a visual language model, comprising an initial model building module, a model optimization module, an adaptive tuning module and a target recognition module, wherein:

[0094] The initial model building module is used to build an initial image detection model based on the real-time open vocabulary detector YOLO-World and the remote sensing visual language foundation model RemoteCLIP;

[0095] A model optimization module is used to add a dual attention fusion encoder and a multi-level text-guided fusion decoder to the initial image detection model to obtain a target image detection model;

[0096] The adaptive tuning module is used to perform knowledge adaptive tuning based on the pre-trained weights of the target image detection model to obtain the final image target recognition model;

[0097] The target recognition module is used to perform target recognition on remote sensing images based on the image target recognition model.

[0098] This system uses a combination of YOLO and RemoteCLIP through the cooperation of multiple modules such as the initial model construction module, the model optimization module, the adaptive tuning module and the target recognition module, so that the model can more effectively identify and align text and visual features in remote sensing images. By using a reparameterizable visual language path aggregation network, the deployment capability of the model on edge devices is enhanced, and real-time and efficient detection is achieved. In order to better enhance feature extraction, a dual attention fusion encoder and a multi-level text-guided fusion decoder are added to the detection model, which further increases the ability to identify small targets. At the same time, the present invention performs knowledge adaptive tuning based on the pre-trained weights of the above-mentioned model to reduce the model's dependence on training data and improve the model recognition performance.

[0099] like Fig. 9 As shown, in a third aspect, an embodiment of the present application provides an electronic device, which includes a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, any method in the first aspect described above is implemented.

[0100] The memory 101, the processor 102 and the communication interface 103 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0101] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0102] The processor 102 may be an integrated circuit chip with signal processing capability. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0103] In the embodiments provided in the present application, it should be understood that the disclosed method and system can also be implemented in other ways. The method and system embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the method and system, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0104] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0105] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by the processor 102, a method as in any one of the first aspects described above is implemented. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0106] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0107] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A remote sensing image target recognition method based on a visual language model, characterized in that: The following steps are involved: Build an initial image detection model based on the real-time open vocabulary detector YOLO-World and the remote sensing visual language foundation model RemoteCLIP; Add a dual attention fusion encoder and a multi-level text-guided fusion decoder to the initial image detection model to obtain the target image detection model; Based on the pre-trained weights of the target image detection model, knowledge adaptive tuning is performed to obtain the final image target recognition model; Perform target recognition on remote sensing images based on image target recognition model.

2. The method for remote sensing image target recognition based on a visual language model according to claim 1, characterized in that: The method for constructing an initial image detection model based on the real-time open vocabulary detector YOLO-World and the remote sensing visual language basic model RemoteCLIP comprises the following steps: The remote sensing visual language basic model RemoteCLIP is used to replace CLIP in the real-time open vocabulary detector YOLO-World, and the initial image detection model is built based on the real-time open vocabulary detector YOLO-World.

3. The method for remote sensing image target recognition based on a visual language model according to claim 1, characterized in that: The dual attention fusion encoder incorporates related class semantics into image features and enhances feature extraction.

4. The method for remote sensing image target recognition based on a visual language model according to claim 1, characterized in that: The multi-level text-guided fusion decoder uses object queries, multi-level class embeddings and multi-scale image features as input.

5. The method for remote sensing image target recognition based on a visual language model according to claim 1, characterized in that: The method for performing knowledge adaptive tuning based on the pre-trained weights of the target image detection model comprises the following steps: A Gaussian saliency guided adapter and a decoder layer adapter with prospective refractive learning are added to the pre-trained weights of the target image detection model to achieve knowledge adaptive tuning.

6. The method for remote sensing image target recognition based on a visual language model according to claim 1, characterized in that: The image object recognition model includes a YOLO detector, a text encoder, and a reparameterizable vision-language path aggregation network.

7. The method for remote sensing image target recognition based on a visual language model according to claim 6, characterized in that: The text editor, given a text Text, uses a RemoteCLIP pre-trained text encoder to replace the original CLIP text encoder to extract the corresponding text embedding; when the input text is a title, a simple noun extraction algorithm is used to extract noun phrases and input them into the text encoder.

8. A remote sensing image target recognition system based on a visual language model, characterized in that: It includes an initial model building module, a model optimization module, an adaptive tuning module, and a target recognition module, among which: The initial model building module is used to build an initial image detection model based on the real-time open vocabulary detector YOLO-World and the remote sensing visual language foundation model RemoteCLIP; A model optimization module is used to add a dual attention fusion encoder and a multi-level text-guided fusion decoder to the initial image detection model to obtain a target image detection model; The adaptive tuning module is used to perform knowledge adaptive tuning based on the pre-trained weights of the target image detection model to obtain the final image target recognition model; The target recognition module is used to perform target recognition on remote sensing images based on the image target recognition model.

9. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images