SAM-based end-to-end remote sensing target detection method, electronic equipment and storage medium

Abstract

The invention discloses an SAM-based end-to-end remote sensing target detection method, electronic equipment and a storage medium, and belongs to the technical field of remote sensing image target detection. In order to apply a large model technology to the field of remote sensing, RGB visible light remote sensing images are collected, and a remote sensing data set is constructed; constructing a parallel double-branch backbone network for feature extraction; constructing a feature fusion module, inputting the obtained feature data of feature extraction into the feature fusion module, outputting the feature data of feature fusion, and inputting the feature data of feature fusion into a prompt generator to generate prompt information data; inputting feature data and prompt information data after feature fusion into a mask decoder of the frozen SAM, calculating the maximum horizontal length and the maximum vertical width of a mask, and drawing a detection frame according to the center coordinate of the mask to realize target detection; and constructing a loss function, wherein the loss function comprises classification loss of a semantic head, regression loss of a positioning head and segmentation loss of a mask generated by a frozen SAM mask decoder.

Description

Technical Field

[0001] The invention belongs to the technical field of remote sensing image target detection, and in particular relates to an end-to-end remote sensing target detection method based on SAM, electronic equipment and storage medium. Background Art

[0002] With the development of optical remote sensing satellite technology, the resolution of remote sensing images has been greatly improved, and the demand for target detection through optical remote sensing images has also emerged. However, due to changes in shooting perspectives and application scenarios, there are certain differences between remote sensing images and ordinary visual images. Introducing deep learning into the field of remote sensing image processing has achieved certain results. With the development of deep learning and massive data, large models that use massive unlabeled data for self-supervised learning have gradually become a new driving force for technological change. However, large models are trained on a large amount of unlabeled data, while image large models are trained on ordinary visual images. The processing of remote sensing images will produce problems such as data hallucination and incomplete interpretation, and most models need manual prompts or fine-tuning to form vertical large models in the remote sensing field. Summary of the invention

[0003] The problem to be solved by the present invention is to apply the large model technology to the remote sensing field, and to propose an end-to-end remote sensing target detection method, electronic equipment and storage medium based on SAM.

[0004] To achieve the above object, the present invention is implemented through the following technical solutions:

[0005] An end-to-end remote sensing target detection method based on SAM comprises the following steps:

[0006] S1. Collect RGB visible light remote sensing images and construct a remote sensing dataset, which is divided into a training set and a test set;

[0007] S2. Construct a parallel dual-branch backbone network for feature extraction, input the training set in the remote sensing data set obtained in step S1 into the parallel dual-branch backbone network for feature extraction, and output feature data for feature extraction;

[0008] S3. Construct a feature fusion module, input the feature data extracted from the feature in step S2 into the feature fusion module, and output the feature fusion feature data;

[0009] S4. Input the feature data obtained by the feature fusion in step S3 into the prompt generator to generate prompt information data;

[0010] S5. The feature data of the feature fusion obtained in step S3 and the prompt information data obtained in step S4 are input into the mask decoder of the frozen SAM, the maximum horizontal length and the maximum vertical width of the mask are calculated, and the detection frame is drawn with the center coordinates of the mask to realize the detection of the target;

[0011] S6. Construct a loss function, which includes the classification loss of the semantic head, the regression loss of the positioning head, and the segmentation loss of the mask generated by the frozen SAM mask decoder, and then iteratively train the prompt generator using the training set obtained in step S1, and test the prompt generator using the test set in the remote sensing dataset obtained in step S1.

[0012] Furthermore, in step S1, an RGB visible light remote sensing image is collected, and then the target to be detected is described by category, instance mask and edge detection frame, and a remote sensing data set is constructed, which is divided into a training set and a test set in the remote sensing data set in a ratio of 4:1.

[0013] Furthermore, the specific implementation method of step S2 includes the following steps:

[0014] S2.1. Construct a parallel dual-branch backbone network for feature extraction, which consists of two parallel branches: a frozen SAM encoder and a CSPDarknet53. The frozen SAM encoder is to freeze the ViT encoder parameters of the SAM encoder.

[0015] The structure of CSPDarknet53 consists of a CGM block and five FastCSP blocks. The CGM block consists of a convolutional layer, a group normalized GN layer, and a Mish activation function.

[0016] S2.2. Set the FastCSP block to consist of two branches, one branch is composed of a convolutional layer, and the other branch is composed of a convolutional layer and N FastBlock blocks. The results of the two branches are input to the Concat function, and then a convolutional layer is used for smoothing the output;

[0017] The FastBlock block includes a network structure that passes through a partial convolution layer PConv with a parameter K and two convolution layers, and adds the processed result to the input;

[0018] The partial convolution layer divides all data channels C into two parts according to the parameter K, expressed as:

[0019]

[0020] C j =CC i

[0021] Among them, C jPart is for normal convolution, C i Some of them are convolutions without processing;

[0022] S2.3. Input the training set of the remote sensing data set obtained in step S1 into the parallel dual-branch backbone network for feature extraction, and set the output features of the five FastCSP blocks of CSPdarknet53 to X i , i = 0, 1, 2, 3, 4, the output features X2, X3, X4 are processed by the Concat function with the output of the frozen SAM encoder, and the feature data extracted by the output feature is output k=0,1,2, the expression is:

[0023] F i ′=φ DownConv (F i ) i=0,1,2...4

[0024] X j =φ FastCSPx (X j-1 ) j=1,2...4

[0025]

[0026] Among them, F i is the intermediate layer feature output by the frozen SAM encoder, φ DownConv Downsample the features to the appropriate dimension for the Convolution-ReLU block, F i ′(please give the meaning of the letters), and X j The output features of each FastCSP block of CSPDarknet53. During the Concat operation, the corresponding features of the two branches are operated to obtain the feature data of feature extraction.

[0027] Furthermore, the specific implementation method of step S3 includes the following steps:

[0028] S3.1. Construct a feature fusion module including feature pyramid network FPN, lightMLP layer and pyramid attention network PAN;

[0029] FPN has three layers from small scale to large scale, and the input of each layer is the feature data extracted in step S2 and the upsampled output of a small scale layer of FPN;

[0030] The last layer of FPN is used as the large-scale first layer of PAN and the first layer is upsampled twice to obtain a three-layer PAN structure;

[0031] S3.2. Connect to the last layer of the pyramid network FPN, use convolution for smoothing, and then connect to the lightMLP layer for processing. The expression for smoothing is:

[0032]

[0033] Among them, X in is the smoothed output, Relu is the activation function, and BN is the batch normalization layer;

[0034] S3.3. Set the lightMLP layer to consist of two parts, a module based on depthwise convolution and a module based on channel MLP, in series. The output of each module is the result of the module processing plus the input.

[0035] The module based on deep convolution is composed of four layers: group normalization layer, deep convolution layer, channel reduction and DropPath connected in sequence to obtain the output of the deep convolution module. The expression is:

[0036]

[0037] Among them, GN is the group normalization layer, and DConv is the deep convolution layer;

[0038] The channel MLP-based module consists of four layers: group normalization layer, channel MLP layer, channel reduction and DropPath connected in sequence, and the expression is:

[0039]

[0040] Among them, CMLP() is the channel MLP, X out is the output of the lightMLP layer;

[0041] S3.4. Transform the output of lightMLP into X out The upsampling is fed back to the three layers of PAN to supplement the global information to PAN. Then the three layers of PAN are fused with the upsampling features of lightMLP, the three pyramid layers of FPN and the upsampling features of PAN itself to obtain the feature fusion feature data.

[0042] Furthermore, the specific implementation method of step S4 is to use the feature data of the feature fusion obtained in step S3 as the input of the prompt generator, and use the Yolo head processing to obtain the category-related semantic head φ cls , position-dependent positioning head φ reg and a hint header φ for hint embedding to the SAM mask decoder prompt, the semantic head is matched with the real category of the target, the positioning head is matched with the detection box of the target, and the matching standard of the two is greedy matching IoU. The expression for generating prompt information data is:

[0043]

[0044] Among them, c k is the semantic head, r k For positioning head, p k For the prompt header.

[0045] Furthermore, the expression of the loss function L constructed in step S6 is:

[0046]

[0047] Among them, L cls Cross-EntropyLoss, L, calculated between the predicted category and the target reg According to the predicted coordinate offset and the target offset between GroundTruth and PriorBox, L seg A binary cross entropy loss between the decoded mask for SAM and the target instance mask label, where the matching criterion is determined by the IoU of the box;

[0048] The loss function is applied to training iterations to make the hint generator produce more reliable masks and detection boxes.

[0049] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of an end-to-end remote sensing target detection method based on SAM when executing the computer program.

[0050] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the end-to-end remote sensing target detection method based on SAM.

[0051] Beneficial effects of the present invention:

[0052] The present invention discloses an end-to-end remote sensing target detection method based on SAM, which realizes remote sensing target detection by utilizing the zero-shot and powerful generalization performance of SAM. CSPDarknet53 is introduced into the backbone network as a parallel branch of the frozen SAM encoder backbone, which fully utilizes SAM while providing feature maps containing semantic information, and utilizes the combined structure of FPN and PAN to fuse the parallel features. At the same time, the FastCSP block is introduced into the backbone network to reduce the amount of computation, and the lightMLP layer is introduced before the neck to process the global dependency of the feature map for deep feature enhancement. This framework generates good automatic prompts by iteratively training the prompt generator to generate accurate masks of semantic categories to guide the SAM mask decoder to generate high-quality masks, so that the SAM large model can be better applied to the remote sensing field. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A flow chart of an end-to-end remote sensing target detection method based on SAM according to the present invention;

[0054] Figure 2 It is a parallel dual-branch backbone network of the present invention;

[0055] Figure 3 It is the improved FastCSP block structure diagram;

[0056] Figure 4 It is a schematic diagram of feature fusion;

[0057] Figure 5 This is the proposed lightMLP structure diagram;

[0058] Figure 6 It is the overall framework structure diagram of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0060] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0061] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 -Attached Figure 6 The detailed instructions are as follows:

[0062] Embodiment 1:

[0063] An end-to-end remote sensing target detection method based on SAM comprises the following steps:

[0064] S1. Collect RGB visible light remote sensing images and construct a remote sensing dataset, which is divided into a training set and a test set;

[0065] Furthermore, in step S1, an RGB visible light remote sensing image is collected, and then the target to be detected is described by category, instance mask and edge detection frame, and a remote sensing data set is constructed, which is divided into a training set and a test set in the remote sensing data set in a ratio of 4:1;

[0066] S2. Construct a parallel dual-branch backbone network for feature extraction, input the training set in the remote sensing data set obtained in step S1 into the parallel dual-branch backbone network for feature extraction, and output feature data for feature extraction;

[0067] Furthermore, the specific implementation method of step S2 includes the following steps:

[0068] S2.1. Construct a parallel dual-branch backbone network for feature extraction, which consists of two parallel branches: a frozen SAM encoder and a CSPDarknet53. The frozen SAM encoder is to freeze the ViT encoder parameters of the SAM encoder.

[0069] The structure of CSPDarknet53 consists of a CGM block and five FastCSP blocks. The CGM block consists of a convolutional layer, a group normalization GN layer, and a Mish activation function. The parameters of the five FastCSP blocks are 1, 2, 8, 8, and 4, respectively, and each time they are downsampled to 1 / 2 of the original scale.

[0070] Furthermore, the frozen SAM encoder and CSPDarnknet53 are selected as two parallel branches of the two backbone networks, and the image data is input into the two parallel branches for processing. In order to reduce the spatial dimension of the feature map, the CSPDarnknet53 branch uses the FastCSP block to downsample the feature map, and the features of each intermediate layer processed by the frozen SAM encoder are downsampled to the same scale using convolution, and the features of the last three stages are concat as the three layers of the subsequent feature fusion FPN;

[0071] S2.2. Set the FastCSP block to consist of two branches, one branch is composed of a convolutional layer, and the other branch is composed of a convolutional layer and N FastBlock blocks. The results of the two branches are input to the Concat function, and then a convolutional layer is used for smoothing the output;

[0072] The FastBlock block includes a network structure that passes through a partial convolution layer PConv with a parameter K and two convolution layers, and adds the processed result to the input;

[0073] The partial convolution layer divides all data channels C into two parts according to the parameter K, expressed as:

[0074]

[0075] C j =CC i

[0076] Among them, C j Part is for normal convolution, C i Some of them are convolutions without processing;

[0077] S2.3. Input the training set of the remote sensing data set obtained in step S1 into the parallel dual-branch backbone network for feature extraction, and set the output features of the five FastCSP blocks of CSPdarknet53 to X i , i = 0, 1, 2, 3, 4, the output features X2, X3, X4 are processed by the Concat function with the output of the frozen SAM encoder, and the feature data extracted by the output feature is output The expression is:

[0078] F i ′=φ DownConv (F i ) i=0,1,2...4

[0079] X j =φ FastCSPx (X j-1 ) j=1,2...4

[0080]

[0081] Among them, F i is the intermediate layer feature output by the frozen SAM encoder, φ DownConv Downsample the features to the appropriate dimension for the Convolution-ReLU block, F i ′(please give the meaning of the letters), and X j The output features of each FastCSP block of CSPDarknet53. During the Concat operation, the corresponding features of the two branches are operated to obtain the feature data of feature extraction.

[0082] S3. Construct a feature fusion module, input the feature data extracted from the feature in step S2 into the feature fusion module, and output the feature fusion feature data;

[0083] Furthermore, the specific implementation method of step S3 includes the following steps:

[0084] S3.1. Construct a feature fusion module including feature pyramid network FPN, lightMLP layer and pyramid attention network PAN;

[0085] FPN has three layers from small scale to large scale, and the input of each layer is the feature data extracted in step S2 and the upsampled output of a small scale layer of FPN;

[0086] The last layer of FPN is used as the large-scale first layer of PAN and the first layer is upsampled twice to obtain a three-layer PAN structure;

[0087] S3.2. Connect to the last layer of the pyramid network FPN, use convolution for smoothing, and then connect to the lightMLP layer for processing. The expression for smoothing is:

[0088]

[0089] Among them, X in is the smoothed output, Relu is the activation function, and BN is the batch normalization layer;

[0090] S3.3. Set the lightMLP layer to consist of two parts, a module based on depthwise convolution and a module based on channel MLP, in series. The output of each module is the result of the module processing plus the input.

[0091] The module based on deep convolution is composed of four layers: group normalization layer, deep convolution layer, channel reduction and DropPath connected in sequence to obtain the output of the deep convolution module. The expression is:

[0092]

[0093] Among them, GN is the group normalization layer, and DConv is the deep convolution layer;

[0094] The channel MLP-based module consists of four layers: group normalization layer, channel MLP layer, channel reduction and DropPath connected in sequence, and the expression is:

[0095]

[0096] Among them, CMLP is the channel MLP, X out is the output of the lightMLP layer;

[0097] S3.4. Transform the output of lightMLP into X out The upsampling is fed back to the three layers of PAN to supplement the global information to PAN. Then the three layers of PAN are fused with the upsampling features of lightMLP, the three pyramid layers of FPN and the upsampling features of PAN itself to obtain the feature fusion feature data.

[0098] S4. Input the feature data obtained by the feature fusion in step S3 into the prompt generator to generate prompt information data;

[0099] Furthermore, the specific implementation method of step S4 is to use the feature data of the feature fusion obtained in step S3 as the input of the prompt generator, and use the Yolo head processing to obtain the category-related semantic head φ cls , position-dependent positioning head φ reg and a hint header φ for hint embedding to the SAM mask decoder prompt , the semantic head is matched with the real category of the target, the positioning head is matched with the detection box of the target, and the matching standard of the two is greedy matching IoU. The expression for generating prompt information data is:

[0100]

[0101] Among them, c k is the semantic head, r k For positioning head, p k For the prompt header.

[0102] S5. The feature data of the feature fusion obtained in step S3 and the prompt information data obtained in step S4 are input into the mask decoder of the frozen SAM, the maximum horizontal length and the maximum vertical width of the mask are calculated, and the detection frame is drawn with the center coordinates of the mask to realize the detection of the target;

[0103] S6. Construct a loss function, which includes the classification loss of the semantic head, the regression loss of the positioning head, and the segmentation loss of the mask generated by the frozen SAM mask decoder, and then iteratively train the prompt generator using the training set obtained in step S1, and test the prompt generator using the test set in the remote sensing dataset obtained in step S1.

[0104] Furthermore, the expression of the loss function L constructed in step S6 is:

[0105]

[0106] Among them, L cls Cross-EntropyLoss, L, calculated between the predicted category and the target reg According to the predicted coordinate offset and the target offset between GroundTruth and PriorBox, L seg A binary cross entropy loss between the decoded mask for SAM and the target instance mask label, where the matching criterion is determined by the IoU of the box;

[0107] The loss function is applied to training iterations to make the hint generator produce more reliable masks and detection boxes.

[0108] Embodiment 2:

[0109] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of an end-to-end remote sensing target detection method based on SAM described in Example 1 are implemented.

[0110] The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit, etc. Moreover, the processor is used to implement the steps of the above-mentioned end-to-end remote sensing target detection method based on SAM when executing the computer program stored in the memory.

[0111] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0112] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0113] Embodiment 3:

[0114] A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the end-to-end remote sensing target detection method based on SAM described in Example 1 is implemented.

[0115] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by a processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. A computer program is stored on the computer-readable storage medium. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned SAM-based end-to-end remote sensing target detection method can be implemented.

[0116] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0117] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0118] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. An end-to-end remote sensing target detection method based on SAM, characterized in that: The steps include: S1. Collect RGB visible light remote sensing images and construct a remote sensing dataset, which is divided into a training set and a test set; S2. Construct a parallel dual-branch backbone network for feature extraction, input the training set in the remote sensing data set obtained in step S1 into the parallel dual-branch backbone network for feature extraction, and output feature data for feature extraction; S3. Construct a feature fusion module, input the feature data extracted from the feature in step S2 into the feature fusion module, and output the feature fusion feature data; S4. Input the feature data obtained in step S3 into the prompt generator to generate prompt information data; S5. The feature data of the feature fusion obtained in step S3 and the prompt information data obtained in step S4 are input into the mask decoder of the frozen SAM, the maximum horizontal length and the maximum vertical width of the mask are calculated, and the detection frame is drawn with the center coordinates of the mask to realize the detection of the target; S6. Construct a loss function, which includes the classification loss of the semantic head, the regression loss of the positioning head, and the segmentation loss of the mask generated by the frozen SAM mask decoder, and then iteratively train the prompt generator using the training set obtained in step S1, and test the prompt generator using the test set in the remote sensing dataset obtained in step S1.

2. The end-to-end remote sensing target detection method based on SAM according to claim 1, characterized in that: In step S1, an RGB visible light remote sensing image is collected, and then the target to be detected is described by category, instance mask and edge detection frame, and a remote sensing data set is constructed, which is divided into a training set and a test set in the sense data set in a ratio of 4:

1.

3. The end-to-end remote sensing target detection method based on SAM according to claim 2, characterized in that: The specific implementation method of step S2 includes the following steps: S2.

1. Construct a parallel dual-branch backbone network for feature extraction, which consists of two parallel branches: a frozen SAM encoder and a CSPDarknet53. The frozen SAM encoder is to freeze the ViT encoder parameters of the SAM encoder. The structure of CSPDarknet53 consists of a CGM block and five FastCSP blocks. The CGM block consists of a convolutional layer, a group normalized GN layer, and a Mish activation function. S2.

2. Set the FastCSP block to consist of two branches, one branch is composed of a convolutional layer, and the other branch is composed of a convolutional layer and N FastBlock blocks. The results of the two branches are input to the Concat function, and then a convolutional layer is used for smoothing the output; The FastBlock block includes a network structure that passes through a partial convolution layer PConv with a parameter K and two convolution layers, and adds the processed result to the input; The partial convolution layer divides all data channels C into two parts according to the parameter K, expressed as: C j =C-C i Among them, C j Part is for normal convolution, C i Some of them are convolutions without processing; S2.

3. Input the training set of the remote sensing data set obtained in step S1 into the parallel dual-branch backbone network for feature extraction, and set the output features of the five FastCSP blocks of CSPdarknet53 to X i , i = 0, 1, 2, 3, 4, the output features X2, X3, X4 are processed by the Concat function with the output of the frozen SAM encoder, and the feature data extracted by the output feature is output k=0,1,2, the expression is: F i ’=φ DownConv (F i ) i=0,1,2...4 X j =φ FastCSPx (X j-1 ) j=1,2...4 Among them, F i is the intermediate layer feature output by the frozen SAM encoder, φ DownConv Downsample the features to the appropriate dimension for the Convolution-ReLU block, F i ' is the feature after downsampling of the SAM encoder branch, and X j The output features of each FastCSP block of CSPDarknet53. During the Concat operation, the corresponding features of the two branches are operated to obtain the feature data of feature extraction.

4. The end-to-end remote sensing target detection method based on SAM according to claim 3, characterized in that: The specific implementation method of step S3 includes the following steps: S3.

1. Construct a feature fusion module including feature pyramid network FPN, lightMLP layer and pyramid attention network PAN; FPN has three layers from small scale to large scale, and the input of each layer is the feature data extracted in step S2 and the upsampled output of a small scale layer of FPN; The last layer of FPN is used as the large-scale first layer of PAN and the first layer is upsampled twice to obtain a three-layer PAN structure; S3.

2. Connect to the last layer of the pyramid network FPN, use convolution for smoothing, and then connect to the lightMLP layer for processing. The expression for smoothing is: Among them, X in is the smoothed output, Relu is the activation function, and BN is the batch normalization layer; S3.

3. Set the lightMLP layer to consist of two parts, a module based on depthwise convolution and a module based on channel MLP, in series. The output of each module is the result of the module processing plus the input. The module based on deep convolution is composed of four layers: group normalization layer, deep convolution layer, channel reduction and DropPath connected in sequence to obtain the output of the deep convolution module. The expression is: Among them, GN is the group normalization layer, and DConv is the deep convolution layer; The channel MLP-based module consists of four layers: group normalization layer, channel MLP layer, channel reduction and DropPath connected in sequence, and the expression is: Among them, CMLP is the channel MLP, X out is the output of the lightMLP layer; S3.

4. Transform the output of lightMLP into out The upsampling is fed back to the three layers of PAN to supplement the global information to PAN. Then the three layers of PAN are fused with the upsampling features of lightMLP, the three pyramid layers of FPN and the upsampling features of PAN itself to obtain the feature fusion feature data. k=0,1,2.

5. The end-to-end remote sensing target detection method based on SAM according to claim 4, characterized in that: The specific implementation method of step S4 is to use the feature data of the feature fusion obtained in step S3 as the input of the prompt generator, and use the Yolo head processing to obtain the category-related semantic head φ cls , Position-dependent positioning head φ reg and a hint header φ for hint embedding to the SAM mask decoder prompt , the semantic head is matched with the real category of the target, the positioning head is matched with the detection box of the target, and the matching standard of the two is greedy matching IoU. The expression for generating prompt information data is: Among them, c k is the semantic head, r k For positioning head, p k For the prompt header.

6. The end-to-end remote sensing target detection method based on SAM according to claim 5, characterized in that: The expression of the loss function L constructed in step S6 is: Among them, L cls Cross-EntropyLoss, L, calculated between the predicted category and the target reg According to the predicted coordinate offset and the target offset between GroundTruth and PriorBox, L seg A binary cross entropy loss between the decoded mask for SAM and the target instance mask label, where the matching criterion is determined by the IoU of the box; The loss function is applied to training iterations to make the hint generator produce more reliable masks and detection boxes.

7. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a SAM-based end-to-end remote sensing target detection method as described in any one of claims 1 to 6 when executing the computer program.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the end-to-end remote sensing target detection method based on SAM described in any one of claims 1 to 6 is implemented.

Images