Remote sensing target detection method, device, medium and equipment based on segmentation large model

Abstract

The present application discloses a remote sensing target detection method, device, storage medium and electronic device based on a segmentation large model. The method includes: acquiring a remote sensing image; inputting the remote sensing image into an image encoder to perform image embedding operations and position embedding operations to obtain an image feature vector; inputting the image feature vector into an adapter to obtain a channel-space feature vector; adding the image feature vector and the channel-space feature vector and inputting them into the image encoder to obtain a one-time image embedding feature; inputting the one-time image embedding feature into a lightweight mask decoder to generate a segmentation mask; calculating the target position based on the segmentation mask, and generating a minimum circumscribed horizontal frame based on the target position; obtaining the position prediction result and classification prediction result corresponding to the remote sensing image based on the coordinates of the minimum circumscribed horizontal frame. The present application can solve the problem of low adaptability of remote sensing image target detection caused by using a segmentation large model.

Description

Technical Field

[0001] The present application relates to the field of target detection technology, and in particular to a remote sensing target detection method, device, storage medium and electronic equipment based on a segmentation large model. Background Art

[0002] With the rapid development of remote sensing technology, the quality of remote sensing images has been greatly improved. As a key component of remote sensing analysis and processing, the development of remote sensing image target detection technology is crucial to the development of military technology. However, due to the large differences in target scale and shape and uneven distribution of foreground and background in remote sensing images, how to accurately identify targets in remote sensing images in complex environments has become a difficult problem in the field of remote sensing image analysis.

[0003] Inspired by the segmentation model, in order to improve the accuracy of target detection in remote sensing images, the segmentation capability of the segmentation model can be used to segment the targets in the remote sensing images, and then determine the specific coordinate information and category of the target in the image based on the segmentation results.

[0004] But at present, it is still very difficult to use large models for pre-training or full fine-tuning. Summary of the invention

[0005] The embodiments of the present application provide a remote sensing target detection method, device, storage medium and electronic device based on a large segmentation model. The large segmentation model is fine-tuned through an adapter. A small number of parameters are adjusted without changing the structure of the large segmentation model, thereby achieving an effect equivalent to full fine-tuning, and can solve the problem of low adaptability of remote sensing image target detection.

[0006] The present application embodiment provides a remote sensing target detection method based on a segmentation large model, including:

[0007] Acquisition of remote sensing images;

[0008] Inputting the remote sensing image into an image encoder to perform image embedding operations and position embedding operations to obtain an image feature vector;

[0009] Inputting the image feature vector into an adapter to obtain a channel-space feature vector;

[0010] Adding the image feature vector and the channel-space feature vector and inputting the resultant vector into the image encoder to obtain a one-time image embedding feature;

[0011] Inputting the one-shot image embedding feature into a lightweight mask decoder to generate a segmentation mask;

[0012] Calculating a target position based on the segmentation mask, and generating a minimum bounding horizontal box based on the target position;

[0013] The position prediction result and the classification prediction result corresponding to the remote sensing image are obtained based on the coordinates of the minimum circumscribed horizontal frame.

[0014] Furthermore, in the above-mentioned remote sensing target detection method based on the segmentation large model, the step of inputting the remote sensing image into an image encoder to perform image embedding operation and position embedding operation to obtain an image feature vector includes:

[0015] Performing an image scaling operation on the remote sensing image to obtain an image of a preset size;

[0016] Inputting the preset size image into a convolution layer to obtain a picture sequence block of a preset size, and obtaining a discretized vector;

[0017] Calculating the absolute position information of the picture sequence block in the remote sensing image;

[0018] The absolute position information is added to the picture sequence block to obtain an image feature vector.

[0019] Further, in the above-mentioned remote sensing target detection method based on the segmentation large model, the adapter comprises a channel-spatial attention network;

[0020] The step of inputting the image feature vector into an adapter to obtain a channel-space feature vector comprises:

[0021] The image feature vector is input into the channel-space attention network, and the channel-space feature is obtained by using a sequential channel attention mechanism and a spatial attention mechanism.

[0022] Furthermore, in the above-mentioned remote sensing target detection method based on the segmentation large model, the image feature vector is input into the channel-space attention network, and a sequential channel attention mechanism and a spatial attention mechanism are used to obtain channel-space features, including:

[0023] The three-dimensional information of the image feature vector is retained by three-dimensional permutation, and the ;

[0024] Calculate the channel characteristics by the first formula , the first formula is:

[0025]

[0026] in, Represents the channel attention mechanism;

[0027] Calculate the channel-space characteristics by the second formula , the second formula is:

[0028]

[0029] in, Represents the spatial attention mechanism.

[0030] Further, in the above-mentioned remote sensing target detection method based on segmentation large model, the image encoder includes a body encoder and a neck network;

[0031] The step of adding the image feature vector to the channel-space feature vector and inputting the resultant vector to the image encoder to obtain a one-time image embedding feature comprises:

[0032] Adding the image feature vector to the channel-space feature vector to obtain a fused feature vector;

[0033] Inputting the fused feature vector into the main encoder for calculation;

[0034] Inputting the fused feature vector into the next channel-space attention network to obtain the next channel-space feature;

[0035] Repeat the above steps until all the main encoders are calculated to obtain the final channel-space features;

[0036] The final channel-space features are input into the neck network to obtain a one-time image embedding feature.

[0037] The embodiment of the present application also provides a remote sensing target detection device based on a segmentation large model, comprising:

[0038] An acquisition module, used for acquiring remote sensing images;

[0039] An image encoder is used to perform image embedding operations and position embedding operations on the remote sensing image to obtain an image feature vector;

[0040] An adapter, used for processing the image feature vector to obtain a channel-space feature vector;

[0041] The image encoder is also used to add the image feature vector and the channel-space feature vector and then process them to obtain a one-time image embedding feature;

[0042] a lightweight mask decoder for generating a segmentation mask based on the one-shot image embedding feature;

[0043] A calculation and generation module, used for calculating a target position based on the segmentation mask, and generating a minimum circumscribed horizontal frame based on the target position;

[0044] The prediction module is used to obtain the position prediction result and the classification prediction result corresponding to the remote sensing image based on the coordinates of the minimum circumscribed horizontal frame.

[0045] Further, in the above-mentioned remote sensing target detection device based on the segmentation large model, the image encoder includes a sequence block embedding submodule and a position code embedding submodule;

[0046] The sequence block embedding submodule is used to process the image of a preset size to obtain a picture sequence block of a preset size and obtain a discretized vector;

[0047] The position coding embedding submodule is used to calculate the absolute position information of the picture sequence block in the remote sensing image, and add the absolute position information to the picture sequence block to obtain an image feature vector.

[0048] Further, in the above-mentioned remote sensing target detection device based on the segmentation large model, the image encoder further includes a body encoder and a neck network;

[0049] The main encoder includes a plurality of Vision Transformer Encoder structures;

[0050] The neck network is used to obtain a one-time image embedding feature based on the final channel-space feature.

[0051] An embodiment of the present application also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor to execute any of the above-mentioned remote sensing target detection methods based on the segmentation large model.

[0052] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for the steps in any of the above-mentioned remote sensing target detection methods based on a segmentation large model.

[0053] The present application provides a remote sensing target detection method, device, storage medium and electronic device based on a large segmentation model. The present application processes the image feature vector output by the encoder through an adapter to obtain a channel-space feature vector, and then adds the image feature vector to the channel-space feature vector and inputs it into the image encoder to obtain a one-time image embedding feature. The large segmentation model is fine-tuned through the adapter, and a small number of parameters are adjusted without changing the structure of the large segmentation model, so as to achieve an effect equivalent to full fine-tuning, which can solve the problem of low adaptability of remote sensing image target detection. The prediction results of remote sensing images based on the one-time image embedding features can improve the accuracy of remote sensing target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0055] Figure 1 A flowchart of a remote sensing target detection method based on a large segmentation model provided in an embodiment of the present application.

[0056] Figure 2 A schematic diagram of the structure of the large segmentation model provided in an embodiment of the present application.

[0057] Figure 3 A flow chart for generating channel-space features provided in an embodiment of the present application.

[0058] Figure 4 A flowchart for generating a one-time image embedding feature provided in an embodiment of the present application.

[0059] Figure 5 A schematic diagram of the structure of a remote sensing target detection device based on a large segmentation model provided in an embodiment of the present application.

[0060] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0061] Figure 7 Another schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0063] Inspired by the large segmentation model, in order to improve the accuracy of target detection in remote sensing images, the segmentation ability of the large segmentation model can be used to segment the targets in the remote sensing images, and then determine the specific coordinate information and category of the target in the image based on the segmentation results. However, it is still very difficult to use large models for pre-training or full fine-tuning. Therefore, based on the large segmentation model, it is necessary to use efficient parameter fine-tuning technology, and adapter fine-tuning technology is one of them. Adapter fine-tuning technology optimizes specific tasks by introducing adapter modules. As a set of lightweight parameters, the adapter module is added to the middle layer of the model to protect the parameters of the original pre-trained model. Without changing the overall model structure, the parameters of the adapter module are adjusted to maximize the effect of adapting to the target detection task.

[0064] It can be seen that there are still many issues that need to be studied in the field of remote sensing target detection using large segmentation models, such as how to more effectively use adapter modules to improve the adaptability of large segmentation models in downstream tasks and how to utilize the high resolution of remote sensing images.

[0065] To solve the above problems, the embodiments of the present application provide a remote sensing target detection method, device, storage medium and electronic device based on a segmentation large model. The remote sensing target detection device based on a segmentation large model provided in the embodiments of the present application can be integrated in an electronic device, which can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro processing box, or other devices.

[0066] The following first explains and illustrates the technical terms of the present invention:

[0067] Vision Transformer: A model that can directly apply the self-attention transformer network (Transformer) to image classification. The Vision Transformer Network mainly consists of three modules, including Linear Projection, Transformer Encoder, and MLP Head.

[0068] Segment Anything Model: It is a state-of-the-art image segmentation model that introduces the hint model from the field of natural language processing into the field of computer vision. It can achieve accurate image segmentation through three hint interactions: points, boxes, and texts, which has significantly improved the efficiency of image segmentation. The Segment Anything Model consists of three key components: an image encoder, which is used to generate one-time image embeddings; a hint encoder, which is used to generate hint embeddings, with hint types of points, boxes, and texts; and a lightweight mask decoder, which is used to combine the hint and the embedding of the image encoder to generate a mask result.

[0069] Multilayer Perceptron (MLP): It is the most common type of neural network model. Each layer of neurons is arranged in layers, and each neuron is only connected to the neurons in the previous layer. It receives the output of the previous layer and passes it to the next layer. There is no feedback between the layers. The multilayer perceptron can be considered as a function, which realizes a complex mapping from input space to output space through multiple compounding of simple nonlinear functions.

[0070] See also Figure 1 , Figure 1A flowchart of a remote sensing target detection method based on a segmentation large model provided in an embodiment of the present application, wherein the remote sensing target detection method based on a segmentation large model is applied to the segmentation large model, Figure 2 For a schematic diagram of the structure of the segmentation model provided in the embodiment of the present application, please refer to Figure 2 The segmentation model includes a fine-tuning module, a lightweight mask decoder and a prediction module. The fine-tuning module is used to fine-tune the segmentation accuracy of the segmentation model in the remote sensing target detection task, and includes an adapter and an image encoder; the adapter is used to fine-tune the image encoder module in the segmentation model, so that the model can adapt to the needs of the remote sensing target detection task more quickly without changing the architecture of the segmentation model; the image encoder module is used to transform the input remote sensing image features to obtain a one-time image embedding feature; the lightweight mask decoder module is used to map the one-time image embedding feature to the mask, which is updated by the decoder. The decoder uses immediate self-attention and cross-attention to obtain the segmentation mask; the prediction module is used to analyze according to the segmentation mask to obtain the classification prediction result and the horizontal box prediction result.

[0071] The remote sensing target detection method based on the segmentation large model includes the following steps:

[0072] S1, obtain remote sensing images.

[0073] S2, input the remote sensing image into the image encoder for image embedding and position embedding operations to obtain the image feature vector.

[0074] In one embodiment, the image encoder includes a sequence block embedding submodule, a position encoding embedding submodule, a body encoder and a neck network, and step S2 includes the following steps:

[0075] S21, performing an image scaling operation on the remote sensing image to obtain an image of a preset size.

[0076] Specifically, the remote sensing image P is scaled by the resize(P) function to obtain an image of a preset size of 1024×1024.

[0077] S22, inputting the preset size image into the convolution layer to obtain a picture sequence block of the preset size, and obtaining a discretized vector.

[0078] Specifically, in the sequence block cutting submodule, the preset size image is input into a 16×16 convolution layer and cut into a 16×16 image sequence block to obtain a discretized vector . Discretize the vector is a one-dimensional vector with a length of 256.

[0079] S23, calculating the absolute position information of the picture sequence block in the remote sensing image.

[0080] S24, adding the absolute position information to the image sequence block to obtain an image feature vector.

[0081] Specifically, the image sequence block is input into the position coding embedding submodule, the absolute position information of the remote sensing image is calculated, and the absolute position information is added to the image sequence block to obtain the image feature vector .

[0082] S3, input the image feature vector into the adapter to obtain the channel-space feature vector.

[0083] Specifically, the adapter includes a channel-spatial attention network, and the channel-spatial attention network specifically includes 12 channel-spatial attention sub-networks with the same structure. Step S3 may include:

[0084] S31, input the image feature vector into the channel-spatial attention network, use the sequential channel attention mechanism and spatial attention mechanism to obtain the channel-spatial features.

[0085] Further, Figure 3 A flow chart of channel-space feature generation provided in an embodiment of the present application, such as Figure 3 As shown, step S31 includes the following steps:

[0086] S311, retaining the three-dimensional information of the image feature vector by three-dimensional permutation, and obtaining .

[0087] S312, calculating channel characteristics by the first formula , the first formula is:

[0088]

[0089] in, Represents the channel attention mechanism.

[0090] Specifically, a two-layer multi-layer perceptron is used to amplify the dependency between channels and spaces, and finally a Sigmoid activation function is used to map the channel features. .

[0091] S313, calculating the channel-space feature by the second formula , the second formula is:

[0092]

[0093] in, Represents the spatial attention mechanism.

[0094] Specifically, through the spatial attention mechanism In the input channel feature , use a 7×7 convolutional layer to reduce the input features, the reduction factor is , and then use a 7×7 convolution layer to fuse spatial information, and finally use the Sigmoid activation function for mapping to obtain the channel-spatial features with channel attention and spatial attention features . Among them, the reduction factor r can be set to 4.

[0095] S4, add the image feature vector and the channel-space feature vector and input them into the image encoder to obtain a one-time image embedding feature.

[0096] In one embodiment, Figure 4 The flowchart for generating a one-time image embedding feature provided in the embodiment of the present application is as follows: Figure 4 As shown, step S4 includes the following steps:

[0097] S41, add the image feature vector and the channel-space feature vector to obtain a fused feature vector .

[0098] S42, input the fused feature vector into the main encoder for calculation.

[0099] Specifically, the main encoder is formed by repeatedly stacking 12 Vision Transformer Encoder structures. The number of Vision Transformer Encoders is the same as the number of channel-spatial attention networks.

[0100] S43, input the fused feature vector into the next channel-spatial attention network to obtain the next channel-spatial feature .

[0101] S44, repeat steps S41-S43 until all calculations of the main encoder are completed to obtain the final channel-space features.

[0102] S45, input the final channel-space feature into the neck network to obtain a one-time image embedding feature.

[0103] Specifically, the neck network consists of 2 normalization layers and 2 convolutional layers.

[0104] S5, inputs the one-shot image embedding features into a lightweight mask decoder to generate a segmentation mask.

[0105] Specifically, the lightweight mask decoder includes 2 ToWayTransformers.

[0106] S6, calculating the target position based on the segmentation mask, and generating a minimum bounding horizontal box based on the target position.

[0107] S7, obtaining a position prediction result and a classification prediction result corresponding to the remote sensing image based on the coordinates of the minimum circumscribed horizontal frame.

[0108] Specifically, the coordinates of the minimum circumscribed horizontal box are input into the prediction module, and the position prediction result and classification prediction result corresponding to the remote sensing image are obtained through a multi-layer perceptron.

[0109] The remote sensing target detection method based on the segmentation large model provided in the embodiment of the present application aims at the problem of low adaptability of remote sensing image target detection. It uses an adapter and introduces a sequential channel-spatial attention network to fine-tune the segmentation large model. On the basis of not changing the structure of the segmentation large model, a small number of parameters are adjusted to achieve the same effect as full fine-tuning. And by utilizing the segmentation capability of the segmentation large model, the targets in the remote sensing image are first segmented to obtain the mask results of each target, and then the coordinates of the target horizontal box are calculated according to the mask, so as to obtain more accurate target detection results.

[0110] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a remote sensing target detection device based on a segmented large model. The remote sensing target detection device based on a segmented large model can be implemented as an independent entity or integrated in an electronic device. The electronic device can be a terminal, a server, and the like. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a micro processing box, or other devices.

[0111] See also Figure 5 , Figure 5 The remote sensing target detection device based on the segmentation large model provided in the embodiment of the present application is specifically described, and is applied to an electronic device. The remote sensing target detection device based on the segmentation large model may include:

[0112] An acquisition module, used for acquiring remote sensing images;

[0113] An image encoder is used to perform image embedding operations and position embedding operations on remote sensing images to obtain image feature vectors;

[0114] An adapter, used for processing the image feature vector to obtain a channel-space feature vector;

[0115] The image encoder is also used to add the image feature vector and the channel-space feature vector and then process them to obtain a one-time image embedding feature;

[0116] A lightweight mask decoder for generating segmentation masks based on one-shot image embedding features;

[0117] A calculation and generation module, used for calculating the target position based on the segmentation mask, and generating a minimum bounding horizontal box based on the target position;

[0118] The prediction module is used to obtain the position prediction result and classification prediction result corresponding to the remote sensing image based on the coordinates of the minimum circumscribed horizontal frame.

[0119] Further, the image encoder includes a sequence block embedding submodule and a position coding embedding submodule;

[0120] The sequence block embedding submodule is used to process the image of preset size, obtain the image sequence block of preset size, and obtain the discretized vector;

[0121] The position encoding embedding submodule is used to calculate the absolute position information of the picture sequence block in the remote sensing image, and add the absolute position information to the picture sequence block to obtain the image feature vector.

[0122] Further, the image encoder also includes a body encoder and a neck network;

[0123] The main encoder includes multiple Vision Transformer Encoder structures;

[0124] The neck network is used to obtain a one-shot image embedding feature based on the final channel-space features.

[0125] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0126] In addition, an embodiment of the present application also provides an electronic device, which may be a computer, a tablet computer, or other device. Figure 6 As shown, the electronic device 400 includes a processor 401 and a memory 402. The processor 401 is electrically connected to the memory 402.

[0127] The processor 401 is the control center of the electronic device 400. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or loading applications stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the electronic device as a whole.

[0128] In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 runs the application programs stored in the memory 402 to implement various functions:

[0129] Acquisition of remote sensing images;

[0130] Inputting the remote sensing image into an image encoder to perform image embedding operations and position embedding operations to obtain an image feature vector;

[0131] Inputting the image feature vector into an adapter to obtain a channel-space feature vector;

[0132] Adding the image feature vector and the channel-space feature vector and inputting the resultant vector into the image encoder to obtain a one-time image embedding feature;

[0133] Inputting the one-shot image embedding feature into a lightweight mask decoder to generate a segmentation mask;

[0134] Calculating a target position based on the segmentation mask, and generating a minimum bounding horizontal box based on the target position;

[0135] The position prediction result and the classification prediction result corresponding to the remote sensing image are obtained based on the coordinates of the minimum circumscribed horizontal frame.

[0136] The electronic device can implement the steps in any embodiment of the remote sensing target detection method based on the segmentation large model provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any remote sensing target detection method based on the segmentation large model provided in the embodiments of the present invention. For details, please refer to the previous embodiments and will not be repeated here.

[0137] Figure 7 The specific structural block diagram of the electronic device provided in the embodiment of the present invention is shown, and the electronic device can be used to implement the remote sensing target detection method based on the segmentation large model provided in the above embodiment. The electronic device 500 can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a notebook computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0138] The RF circuit 510 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 510 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks may use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0139] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, that is, realizing the functions of taking pictures with the front camera, processing the captured images, and switching the display color of the display content on the display screen. The memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 520 may further include a memory remotely arranged relative to the processor 580, and these remote memories may be connected to the electronic device 500 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0140] The input unit 530 may be used to receive input digital or character information, and generate a keyboard and a mouse related to user settings and function control.

[0141] The display unit 540 may be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 540 may include a display panel 541, which may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.

[0142] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can transmit the electrical signal converted from the received audio data to the speaker 561, which is converted into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and converted into audio data, and then the audio data is output to the processor 580 for processing, and then sent to another terminal through the RF circuit 510, or the audio data is output to the memory 520 for further processing. The audio circuit 560 may also include an earplug jack to provide communication between an external headset and the electronic device 500.

[0143] The electronic device 500 can help the user receive requests, send information, etc. through the transmission module 570 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown in the figure, it can be understood that it is not a necessary component of the electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0144] The processor 580 is the control center of the electronic device 500. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory 520, it executes various functions of the electronic device 500 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 580.

[0145] The electronic device 500 also includes a power supply 590 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 590 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0146] Although not shown, the electronic device 500 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors. One or more programs include instructions for performing the following operations:

[0147] Acquisition of remote sensing images;

[0148] Inputting the remote sensing image into an image encoder to perform image embedding operations and position embedding operations to obtain an image feature vector;

[0149] Inputting the image feature vector into an adapter to obtain a channel-space feature vector;

[0150] Adding the image feature vector and the channel-space feature vector and inputting the resultant vector into the image encoder to obtain a one-time image embedding feature;

[0151] Inputting the one-shot image embedding feature into a lightweight mask decoder to generate a segmentation mask;

[0152] Calculating a target position based on the segmentation mask, and generating a minimum bounding horizontal box based on the target position;

[0153] The position prediction result and the classification prediction result corresponding to the remote sensing image are obtained based on the coordinates of the minimum circumscribed horizontal frame.

[0154] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0155] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores a plurality of instructions, and the instructions can be loaded by a processor to execute the steps of any embodiment of the remote sensing target detection method based on a segmentation large model provided in an embodiment of the present invention.

[0156] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0157] Since the instructions stored in the storage medium can execute the steps in any embodiment of the remote sensing target detection method based on the segmentation large model provided in the embodiments of the present invention, the beneficial effects that can be achieved by any remote sensing target detection method based on the segmentation large model provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0158] The above is a detailed introduction to a remote sensing target detection method, device, storage medium and electronic device based on a segmentation large model provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A remote sensing target detection method based on a segmentation large model, characterized in that: The method comprises: Acquisition of remote sensing images; Inputting the remote sensing image into an image encoder to perform image embedding operations and position embedding operations to obtain an image feature vector; Inputting the image feature vector into an adapter to obtain a channel-space feature vector; wherein the adapter includes a channel-space attention network; Inputting the image feature vector into the adapter to obtain a channel-space feature vector includes: inputting the image feature vector into the channel-space attention network, using a sequential channel attention mechanism and a space attention mechanism to obtain a channel-space feature; Adding the image feature vector and the channel-space feature vector and inputting the resultant vector into the image encoder to obtain a one-time image embedding feature; Inputting the one-shot image embedding feature into a lightweight mask decoder to generate a segmentation mask; Calculating a target position based on the segmentation mask, and generating a minimum bounding horizontal box based on the target position; The position prediction result and the classification prediction result corresponding to the remote sensing image are obtained based on the coordinates of the minimum circumscribed horizontal frame.

2. The remote sensing target detection method based on the segmentation large model according to claim 1 is characterized in that: The step of inputting the remote sensing image into an image encoder to perform image embedding operations and position embedding operations to obtain an image feature vector includes: Performing an image scaling operation on the remote sensing image to obtain an image of a preset size; Inputting the preset size image into a convolution layer to obtain a picture sequence block of a preset size, and obtaining a discretized vector; Calculating the absolute position information of the picture sequence block in the remote sensing image; The absolute position information is added to the picture sequence block to obtain an image feature vector.

3. The remote sensing target detection method based on the segmentation large model according to claim 1 is characterized in that: The inputting the image feature vector into the channel-space attention network, using a sequential channel attention mechanism and a space attention mechanism to obtain a channel-space feature, includes: The three-dimensional information of the image feature vector is retained by three-dimensional permutation, and the ; Calculate the channel characteristics by the first formula , the first formula is: in, Represents the channel attention mechanism; Calculate the channel-space characteristics by the second formula , the second formula is: in, Represents the spatial attention mechanism.

4. The remote sensing target detection method based on segmentation large model according to claim 2 is characterized in that: The image encoder includes a body encoder and a neck network; The step of adding the image feature vector to the channel-space feature vector and inputting the resultant vector to the image encoder to obtain a one-time image embedding feature comprises: Adding the image feature vector to the channel-space feature vector to obtain a fused feature vector; Inputting the fused feature vector into the main encoder for calculation; Inputting the fused feature vector into the next channel-space attention network to obtain the next channel-space feature; Repeat the above steps until all the main encoders are calculated to obtain the final channel-space features; The final channel-space features are input into the neck network to obtain a one-time image embedding feature.

5. A remote sensing target detection device based on a segmentation large model, the remote sensing target detection device based on a segmentation large model is used to implement the remote sensing target detection method based on a segmentation large model according to claim 1, characterized in that: include: An acquisition module, used for acquiring remote sensing images; An image encoder is used to perform image embedding operations and position embedding operations on the remote sensing image to obtain an image feature vector; An adapter, used for processing the image feature vector to obtain a channel-space feature vector; The image encoder is also used to add the image feature vector and the channel-space feature vector and then process them to obtain a one-time image embedding feature; a lightweight mask decoder for generating a segmentation mask based on the one-shot image embedding feature; A calculation and generation module, used for calculating a target position based on the segmentation mask, and generating a minimum circumscribed horizontal frame based on the target position; The prediction module is used to obtain the position prediction result and the classification prediction result corresponding to the remote sensing image based on the coordinates of the minimum circumscribed horizontal frame.

6. The remote sensing target detection device based on the segmentation large model according to claim 5 is characterized in that: The image encoder includes a sequence block embedding submodule and a position code embedding submodule; The sequence block embedding submodule is used to process the image of a preset size to obtain a picture sequence block of a preset size and obtain a discretized vector; The position coding embedding submodule is used to calculate the absolute position information of the picture sequence block in the remote sensing image, and add the absolute position information to the picture sequence block to obtain an image feature vector.

7. The remote sensing target detection device based on the segmentation large model according to claim 5 is characterized in that: The image encoder also includes a body encoder and a neck network; The main encoder includes a plurality of Vision Transformer Encoder structures; The neck network is used to obtain a one-time image embedding feature based on the final channel-space feature.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the remote sensing target detection method based on a segmentation large model as described in any one of claims 1 to 4.

9. An electronic device, characterized in that: It includes a processor and a memory, the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in the remote sensing target detection method based on the segmentation large model as described in any one of claims 1 to 4.

Images