End-to-end remote sensing target detection method, electronic device and storage medium based on SAM

Abstract

A SAM-based end-to-end remote sensing target detection method, electronic device, and storage medium belong to the technical field of remote sensing image target detection. In order to apply large model technology to the remote sensing field, the present invention collects RGB visible light remote sensing images and constructs a remote sensing data set; constructs a parallel dual-branch backbone network for feature extraction; constructs a feature fusion module, inputs the feature data obtained from feature extraction into the feature fusion module, outputs the fused feature data, and inputs it into a prompt generator to generate prompt information data; inputs the fused feature data and prompt information data into a frozen SAM mask decoder, calculates the maximum horizontal length and maximum vertical width of the mask, and draws a detection box with the center coordinates of the mask to detect the target; and constructs a loss function, which is composed of the classification loss of the semantic head, the regression loss of the positioning head, and the segmentation loss of the mask generated by the frozen SAM mask decoder.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image target detection, and in particular relates to an end-to-end remote sensing target detection method based on SAM, an electronic device and a storage medium. Background Art

[0002] With the development of optical remote sensing satellite technology, the resolution of remote sensing images has significantly increased, creating a need for target detection using optical remote sensing images. However, due to changes in shooting perspectives and application scenarios, remote sensing images differ from standard visual images. The introduction of deep learning into remote sensing image processing has achieved some success. With the advancement of deep learning and massive data, large models that leverage massive amounts of unlabeled data for self-supervised learning have become a new driving force for technological change. However, large models are trained on large amounts of unlabeled data, while large image models are trained on standard visual images. This processing of remote sensing images can lead to data hallucinations and incomplete interpretations. Furthermore, most models require manual prompting or fine-tuning to form large, vertical models for remote sensing. Summary of the Invention

[0003] The problem to be solved by the present invention is to apply large model technology to the field of remote sensing, and to propose an end-to-end remote sensing target detection method, electronic equipment and storage medium based on SAM.

[0004] To achieve the above object, the present invention is implemented through the following technical solutions:

[0005] An end-to-end remote sensing target detection method based on SAM includes the following steps:

[0006] S1. Collect RGB visible light remote sensing images and construct a remote sensing dataset, which is divided into a training set and a test set.

[0007] S2 constructs a parallel dual-branch backbone network for feature extraction, inputs the training set of the remote sensing data set obtained in step S1 into the parallel dual-branch backbone network for feature extraction, and outputs the feature data extracted from the feature;

[0008] S3 constructs a feature fusion module, inputs the feature data extracted from the feature obtained in step S2 into the feature fusion module, and outputs the feature fusion feature data;

[0009] S4. The feature data obtained in step S3 is input into the prompt generator to generate prompt information data;

[0010] S5. The feature data obtained by the feature fusion in step S3 and the prompt information data obtained in step S4 are input into the frozen SAM mask decoder, the maximum horizontal length and maximum vertical width of the mask are calculated, and the detection box is drawn at the center coordinates of the mask to detect the target;

[0011] S6. Construct a loss function, which includes the classification loss of the semantic head, the regression loss of the localization head, and the segmentation loss of the mask generated by the frozen SAM mask decoder. Then, use the training set obtained in step S1 to iteratively train the prompt generator, and use the test set in the remote sensing dataset obtained in step S1 to test the prompt generator.

[0012] Furthermore, in step S1, an RGB visible light remote sensing image is collected, and then the target to be detected is described by category, instance mask and edge detection box, and a remote sensing dataset is constructed, which is divided into a training set and a test set in the sensing dataset in a ratio of 4:1.

[0013] Furthermore, the specific implementation method of step S2 includes the following steps:

[0014] S2.1. Construct a parallel dual-branch backbone network for feature extraction, consisting of two parallel branches: a frozen SAM encoder and a CSPDarknet53. The frozen SAM encoder is constructed by freezing the ViT encoder parameters of the SAM encoder.

[0015] The structure of CSPDarknet53 consists of a CGM block and five FastCSP blocks. The CGM block consists of a convolutional layer, a group normalization GN layer, and a Mish activation function.

[0016] S2.2. Set the FastCSP block to consist of two branches, one branch consisting of a convolutional layer and the other branch consisting of a convolutional layer and N FastBlock blocks. The results of the two branches are input to the Concat function, and then a convolutional layer is used to smooth the output.

[0017] The FastBlock block includes a network structure that passes through a partial convolution layer PConv with a parameter K and two convolution layers, and adds the processed result to the input;

[0018] The partial convolution layer divides all data channels C into two parts according to the parameter K, which is expressed as:

[0019]

[0020] C j =CC i

[0021] Among them, C jPart is for normal convolution, C i Some of them are convolutions without processing;

[0022] S2.3. Input the training set of the remote sensing dataset obtained in step S1 into the parallel two-branch backbone network for feature extraction, and set the output features of the five FastCSP blocks of CSPdarknet53 to X i , i=0,1,2,3,4, the output features X2, X3, X4 are processed by Concat function with the output of the frozen SAM encoder, and the feature data extracted is output k=0,1,2, the expression is:

[0023] F i ′=φ DownConv (F i ) i=0,1,2...4

[0024] X j =φ FastCSPx (X j-1 ) j=1,2...4

[0025]

[0026] Among them, F i is the intermediate layer feature output by the frozen SAM encoder, φ DownConv The Convolution-ReLU block downsamples the features to the appropriate dimension, F i ′(please give the meaning of the letters), and X j The output features of each FastCSP block of CSPDarknet53 are obtained by performing the Concat operation on the corresponding features of the two branches to obtain the feature data of feature extraction.

[0027] Furthermore, the specific implementation method of step S3 includes the following steps:

[0028] S3.1. Construct a feature fusion module including the feature pyramid network FPN, lightMLP layer, and pyramid attention network PAN;

[0029] FPN has three layers from small scale to large scale. The input of each layer is the feature data extracted in step S2 and the upsampled output of a small scale layer on the FPN.

[0030] The last layer of FPN is used as the large-scale first layer of PAN and the first layer is upsampled twice to obtain a three-layer PAN structure;

[0031] S3.2. Connect the last layer of the pyramid network FPN, use convolution to smooth it, and then connect it to the lightMLP layer for processing. The smoothing expression is:

[0032]

[0033] Among them, X in is the smoothed output, Relu is the activation function, and BN is the batch normalization layer;

[0034] S3.3. Set up the lightMLP layer to consist of two parts: a depthwise convolution-based module and a channel-wise MLP-based module connected in series. The output of each module is the sum of the processed result and the input.

[0035] The module based on depth convolution is composed of four layers: group normalization layer, depth convolution layer, channel reduction and DropPath connected in sequence to obtain the output of the module based on depth convolution. The expression is:

[0036]

[0037] Among them, GN is the group normalization layer, DConv is the depth convolution layer;

[0038] The channel MLP-based module consists of four layers: group normalization layer, channel MLP layer, channel reduction and DropPath connected in sequence, and the expression is:

[0039]

[0040] Among them, CMLP() is the channel MLP, X out is the output of the lightMLP layer;

[0041] S3.4. Transform the output of lightMLP into X out The upsampling is fed back to the three layers of PAN, and the global information is supplemented to PAN. Then the three layers of PAN are fused with the upsampling features of lightMLP, the three pyramid layers of FPN and the upsampling features of PAN itself to obtain the feature fusion feature data.

[0042] Furthermore, the specific implementation method of step S4 is to use the feature fusion data obtained in step S3 as the input of the prompt generator, and use the Yolo head processing to obtain the category-related semantic head φ cls , Position-dependent positioning head φ reg and a hint header φ for hint embedding to the SAM mask decoder prompt, the semantic head is matched with the real category of the target, and the positioning head is matched with the detection box of the target, and the matching standard of the two is greedy matching IoU. The expression for generating prompt information data is:

[0043]

[0044] Among them, c k is the semantic head, r k For positioning head, p k For the prompt header.

[0045] Furthermore, the loss function L constructed in step S6 is expressed as:

[0046]

[0047] Among them, L cls Cross-EntropyLoss calculated between the predicted category and the target, L reg According to the predicted coordinate offset and the target offset between GroundTruth and PriorBox, L seg A binary cross-entropy loss between the decoded mask for SAM and the target instance mask label, where the matching criterion is determined by the IoU of the box;

[0048] The loss function is applied to the training iterations to make the hint generator produce more reliable masks and detection boxes.

[0049] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a SAM-based end-to-end remote sensing target detection method when executing the computer program.

[0050] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the end-to-end remote sensing target detection method based on SAM.

[0051] Beneficial effects of the present invention:

[0052] The present invention discloses an end-to-end remote sensing target detection method based on SAM, which realizes remote sensing target detection by utilizing the zero-shot and strong generalization performance of SAM. CSPDarknet53 is introduced into the backbone network as a parallel branch of the frozen SAM encoder backbone, which fully utilizes SAM while providing feature maps containing semantic information, and utilizes the combined structure of FPN and PAN to fuse the parallel features. At the same time, the FastCSP block is introduced into the backbone network to reduce the amount of computation, and the lightMLP layer is introduced before the neck to process the global dependency of the feature map for deep feature enhancement. This framework generates good automatic prompts by iteratively training the prompt generator to generate accurate semantic category masks to guide the SAM mask decoder to generate high-quality masks, thereby better applying the SAM large model to the remote sensing field. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Flowchart of an end-to-end remote sensing target detection method based on SAM according to the present invention;

[0054] Figure 2 It is a parallel dual-branch backbone network of the present invention;

[0055] Figure 3 It is the improved FastCSP block structure diagram;

[0056] Figure 4 It is a schematic diagram of feature fusion;

[0057] Figure 5 This is the proposed lightMLP structure diagram;

[0058] Figure 6 It is the overall framework structure diagram of the present invention. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present invention and are not intended to limit the present invention. That is, the specific embodiments described herein are only some embodiments of the present invention, not all embodiments. Generally, the components of the specific embodiments of the present invention described and illustrated in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0060] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely representative of selected specific embodiments of the present invention. All other specific embodiments obtained by those skilled in the art based on the specific embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0061] In order to further understand the content, features and effects of the present invention, the following specific embodiments are given as examples, and the attached Figure 1 -Attached Figure 6 The detailed instructions are as follows:

[0062] Example 1:

[0063] An end-to-end remote sensing target detection method based on SAM includes the following steps:

[0064] S1. Collect RGB visible light remote sensing images and construct a remote sensing dataset, which is divided into a training set and a test set.

[0065] Furthermore, in step S1, RGB visible light remote sensing images are collected, and then the target to be detected is described by category, instance mask and edge detection box, and a remote sensing dataset is constructed, which is divided into a training set and a test set in the sensing dataset in a ratio of 4:1;

[0066] S2 constructs a parallel dual-branch backbone network for feature extraction, inputs the training set of the remote sensing data set obtained in step S1 into the parallel dual-branch backbone network for feature extraction, and outputs the feature data extracted from the feature;

[0067] Furthermore, the specific implementation method of step S2 includes the following steps:

[0068] S2.1. Construct a parallel dual-branch backbone network for feature extraction, consisting of two parallel branches: a frozen SAM encoder and a CSPDarknet53. The frozen SAM encoder is constructed by freezing the ViT encoder parameters of the SAM encoder.

[0069] The structure of CSPDarknet53 consists of a CGM block and five FastCSP blocks. The CGM block consists of a convolutional layer, a group normalization (GN) layer, and a Mish activation function. The parameters of the five FastCSP blocks are 1, 2, 8, 8, and 4, respectively, and each time they are downsampled to 1 / 2 of the original scale.

[0070] Furthermore, the frozen SAM encoder and CSPDarnknet53 were selected as the two parallel branches of the two backbone networks, and the image data was fed into the two parallel branches for processing. To reduce the spatial dimension of the feature map, the CSPDarnknet53 branch used FastCSP blocks to downsample the feature map, while the intermediate layer features processed by the frozen SAM encoder were downsampled to the same scale using convolution. The features of the last three stages were concat as the three layers of the subsequent feature fusion FPN.

[0071] S2.2. Set the FastCSP block to consist of two branches, one branch consisting of a convolutional layer and the other branch consisting of a convolutional layer and N FastBlock blocks. The results of the two branches are input to the Concat function, and then a convolutional layer is used to smooth the output.

[0072] The FastBlock block includes a network structure that passes through a partial convolution layer PConv with a parameter K and two convolution layers, and adds the processed result to the input;

[0073] The partial convolution layer divides all data channels C into two parts according to the parameter K, which is expressed as:

[0074]

[0075] C j =CC i

[0076] Among them, C j Part is for normal convolution, C i Some of them are convolutions without processing;

[0077] S2.3. Input the training set of the remote sensing dataset obtained in step S1 into the parallel two-branch backbone network for feature extraction, and set the output features of the five FastCSP blocks of CSPdarknet53 to X i , i=0,1,2,3,4, the output features X2, X3, X4 are processed by Concat function with the output of the frozen SAM encoder, and the feature data extracted is output The expression is:

[0078] F i ′=φ DownConv (F i ) i=0,1,2...4

[0079] X j =φ FastCSPx (X j-1 ) j=1,2...4

[0080]

[0081] Among them, F i is the intermediate layer feature output by the frozen SAM encoder, φ DownConv The Convolution-ReLU block downsamples the features to the appropriate dimension, F i ′(please give the meaning of the letters), and X j The output features of each FastCSP block of CSPDarknet53 are obtained by performing the Concat operation on the corresponding features of the two branches to obtain the feature data of feature extraction.

[0082] S3 constructs a feature fusion module, inputs the feature data extracted from the feature obtained in step S2 into the feature fusion module, and outputs the feature fusion feature data;

[0083] Furthermore, the specific implementation method of step S3 includes the following steps:

[0084] S3.1. Construct a feature fusion module including the feature pyramid network FPN, lightMLP layer, and pyramid attention network PAN;

[0085] FPN has three layers from small scale to large scale. The input of each layer is the feature data extracted in step S2 and the upsampled output of a small scale layer on the FPN.

[0086] The last layer of FPN is used as the large-scale first layer of PAN and the first layer is upsampled twice to obtain a three-layer PAN structure;

[0087] S3.2. Connect the last layer of the pyramid network FPN, use convolution to smooth it, and then connect it to the lightMLP layer for processing. The smoothing expression is:

[0088]

[0089] Among them, X in is the smoothed output, Relu is the activation function, and BN is the batch normalization layer;

[0090] S3.3. Set up the lightMLP layer to consist of two parts: a depthwise convolution-based module and a channel-wise MLP-based module connected in series. The output of each module is the sum of the processed result and the input.

[0091] The module based on depth convolution is composed of four layers: group normalization layer, depth convolution layer, channel reduction and DropPath connected in sequence to obtain the output of the module based on depth convolution. The expression is:

[0092]

[0093] Among them, GN is the group normalization layer, DConv is the depth convolution layer;

[0094] The channel MLP-based module consists of four layers: group normalization layer, channel MLP layer, channel reduction and DropPath connected in sequence, and the expression is:

[0095]

[0096] Among them, CMLP is channel MLP, X out is the output of the lightMLP layer;

[0097] S3.4. Transform the output of lightMLP into X out The upsampling is fed back to the three layers of PAN, and the global information is supplemented to PAN. Then the three layers of PAN are fused with the upsampling features of lightMLP, the three pyramid layers of FPN and the upsampling features of PAN itself to obtain the feature fusion feature data.

[0098] S4. The feature data obtained in step S3 is input into the prompt generator to generate prompt information data;

[0099] Furthermore, the specific implementation method of step S4 is to use the feature fusion data obtained in step S3 as the input of the prompt generator, and use the Yolo head processing to obtain the category-related semantic head φ cls , Position-dependent positioning head φ reg and a hint header φ for hint embedding to the SAM mask decoder prompt , the semantic head is matched with the real category of the target, and the positioning head is matched with the detection box of the target, and the matching standard of the two is greedy matching IoU. The expression for generating prompt information data is:

[0100]

[0101] Among them, c k is the semantic head, r k For positioning head, p k For the prompt header.

[0102] S5. The feature data obtained by the feature fusion in step S3 and the prompt information data obtained in step S4 are input into the frozen SAM mask decoder, the maximum horizontal length and maximum vertical width of the mask are calculated, and the detection box is drawn at the center coordinates of the mask to detect the target;

[0103] S6. Construct a loss function, which includes the classification loss of the semantic head, the regression loss of the localization head, and the segmentation loss of the mask generated by the frozen SAM mask decoder. Then, use the training set obtained in step S1 to iteratively train the prompt generator, and use the test set in the remote sensing dataset obtained in step S1 to test the prompt generator.

[0104] Furthermore, the loss function L constructed in step S6 is expressed as:

[0105]

[0106] Among them, L cls Cross-EntropyLoss calculated between the predicted category and the target, L reg According to the predicted coordinate offset and the target offset between GroundTruth and PriorBox, L seg A binary cross-entropy loss between the decoded mask for SAM and the target instance mask label, where the matching criterion is determined by the IoU of the box;

[0107] The loss function is applied to the training iterations to make the hint generator produce more reliable masks and detection boxes.

[0108] Example 2:

[0109] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the end-to-end remote sensing target detection method based on SAM described in Example 1 are implemented.

[0110] The computer device of the present invention may include a processor and a memory, such as a single-chip microcomputer including a central processing unit. Furthermore, the processor is configured to execute a computer program stored in the memory to implement the steps of the aforementioned SAM-based end-to-end remote sensing target detection method.

[0111] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0112] The memory may mainly include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area may store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0113] Example 3:

[0114] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the end-to-end remote sensing target detection method based on SAM described in Example 1.

[0115] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by the processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. The computer-readable storage medium stores a computer program. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned SAM-based end-to-end remote sensing target detection method can be implemented.

[0116] The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0117] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0118] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and components may be substituted with equivalents without departing from the scope of the present application. In particular, as long as there are no structural conflicts, the various features of the embodiments disclosed herein may be combined with each other in any manner, and the omission of an exhaustive description of these combinations in this specification is solely for the sake of space and resource conservation. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions within the scope of the claims.

Claims

1. An end-to-end remote sensing target detection method based on SAM, characterized in that: The steps include: S1. Collect RGB visible light remote sensing images and construct a remote sensing dataset, which is divided into a training set and a test set. S2 constructs a parallel dual-branch backbone network for feature extraction, inputs the training set of the remote sensing data set obtained in step S1 into the parallel dual-branch backbone network for feature extraction, and outputs the feature data extracted from the feature; The specific implementation method of step S2 includes the following steps: S2.

1. Construct a parallel dual-branch backbone network for feature extraction, consisting of two parallel branches: a frozen SAM encoder and a CSPDarknet53. The frozen SAM encoder is constructed by freezing the ViT encoder parameters of the SAM encoder. The structure of CSPDarknet53 consists of a CGM block and five FastCSP blocks. The CGM block consists of a convolutional layer, a group normalization GN layer, and a Mish activation function. S2.

2. Set the FastCSP block to consist of two branches, one branch consisting of a convolutional layer and the other branch consisting of a convolutional layer and N FastBlock blocks. The results of the two branches are input to the Concat function, and then a convolutional layer is used to smooth the output. The FastBlock block includes a network structure that passes through a partial convolution layer PConv with a parameter K and two convolution layers, and adds the processed result to the input; The partial convolution layer divides all data channels C into two parts according to the parameter K, which is expressed as: C j =C-C i Among them, C j Part is for normal convolution, C i Some of them are convolutions without processing; S2.

3. Input the training set of the remote sensing dataset obtained in step S1 into the parallel two-branch backbone network for feature extraction, and set the output features of the five FastCSP blocks of CSPdarknet53 to X i , i=0,1,2,3,4, the output features X2, X3, X4 are processed by Concat function with the output of the frozen SAM encoder, and the feature data extracted is output k=0,1,2, the expression is: F i '=φ DownConv (F i )i=0,1,2...4 X j =φ FastCSPx (X j-1 )j=1,2...4 Among them, F i is the intermediate layer feature output by the frozen SAM encoder, φ DownConv The Convolution-ReLU block downsamples the features to the appropriate dimension, F i ' is the feature after downsampling of the SAM encoder branch, and X j The output features of each FastCSP block of CSPDarknet53 are obtained by performing the Concat operation on the corresponding features of the two branches to obtain the feature data of feature extraction. S3 constructs a feature fusion module, inputs the feature data extracted from the feature obtained in step S2 into the feature fusion module, and outputs the feature fusion feature data; The specific implementation method of step S3 includes the following steps: S3.

1. Construct a feature fusion module including the feature pyramid network FPN, lightMLP layer, and pyramid attention network PAN; FPN has three layers from small scale to large scale. The input of each layer is the feature data extracted in step S2 and the upsampled output of a small scale layer of FPN. The last layer of FPN is used as the large-scale first layer of PAN and the first layer is upsampled twice to obtain a three-layer PAN structure; S3.

2. Connect the last layer of the pyramid network FPN, use convolution to smooth it, and then connect it to the lightMLP layer for processing. The smoothing expression is: Among them, X in is the smoothed output, Relu is the activation function, and BN is the batch normalization layer; S3.

3. Set up the lightMLP layer to consist of two parts: a depthwise convolution-based module and a channel-wise MLP-based module connected in series. The output of each module is the sum of the processed result and the input. The module based on depth convolution is composed of four layers: group normalization layer, depth convolution layer, channel reduction and DropPath connected in sequence to obtain the output of the module based on depth convolution. The expression is: Among them, GN is the group normalization layer, DConv is the depth convolution layer; The channel MLP-based module consists of four layers: group normalization layer, channel MLP layer, channel reduction and DropPath connected in sequence, and the expression is: Among them, CMLP is channel MLP, X out is the output of the lightMLP layer; S3.

4. Transform the output of lightMLP into X out The upsampling is fed back to the three layers of PAN, and the global information is supplemented to PAN. Then the three layers of PAN are fused with the upsampling features of lightMLP, the three pyramid layers of FPN and the upsampling features of PAN itself to obtain the feature fusion feature data. k=0,1,2; S4. The feature data obtained in step S3 is input into the prompt generator to generate prompt information data; The specific implementation method of step S4 is to use the feature fusion data obtained in step S3 as the input of the prompt generator, and use Yolo head processing to obtain the category-related semantic head φ cls , Position-dependent positioning head φ reg and a hint header φ for hint embedding to the SAM mask decoder prompt , the semantic head is matched with the real category of the target, and the positioning head is matched with the detection box of the target, and the matching standard of the two is greedy matching IoU. The expression for generating prompt information data is: Among them, c k is the semantic head, r k For positioning head, p k For the prompt head; S5. The feature data obtained by the feature fusion in step S3 and the prompt information data obtained in step S4 are input into the frozen SAM mask decoder, the maximum horizontal length and maximum vertical width of the mask are calculated, and the detection box is drawn at the center coordinates of the mask to detect the target; S6. Construct a loss function, which includes the classification loss of the semantic head, the regression loss of the localization head, and the segmentation loss of the mask generated by the frozen SAM mask decoder. Then, use the training set obtained in step S1 to iteratively train the prompt generator, and use the test set in the remote sensing dataset obtained in step S1 to test the prompt generator.

2. The end-to-end remote sensing target detection method based on SAM according to claim 1, characterized in that: In step S1, RGB visible light remote sensing images are collected, and then the target to be detected is described by category, instance mask and edge detection box, and a remote sensing dataset is constructed. The dataset is divided into a training set and a test set in a ratio of 4:

1.

3. The end-to-end remote sensing target detection method based on SAM according to claim 2, characterized in that: The expression of the loss function L constructed in step S6 is: Among them, L cls Cross-Entropy Loss, L, calculated between the predicted category and the target reg According to the predicted coordinate offset and the target offset between Ground Truth and Prior Box, L seg A binary cross-entropy loss between the decoded mask for SAM and the target instance mask label, where the matching criterion is determined by the IoU of the box; The loss function is applied to the training iterations to make the hint generator produce more reliable masks and detection boxes.

4. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the SAM-based end-to-end remote sensing target detection method according to any one of claims 1 to 3 when executing the computer program.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the end-to-end remote sensing target detection method based on SAM according to any one of claims 1 to 3 is implemented.

Images