A SAM-based image segmentation method and system for butterfly ecology

Abstract

The application provides a SAM-based butterfly ecological image segmentation method and system, and belongs to the technical field of image segmentation, aiming to solve the problems of unsatisfactory segmentation result accuracy and high training cost of SAM in butterfly ecological image segmentation, based on the SAM model, a double-path convolution module is added, the double-path convolution module is used for further feature extraction on the feature code vectors generated by the 12 Transformer modules of the image encoder, and the double-path convolution module comprises a global path and a local path, the global path generates an image global feature vector by using the image feature vectors generated by the 3rd, 6th, 9th and 12th Transformer modules of the image encoder, and the local path generates an image local feature vector by using the image feature vectors generated by the remaining Transformer modules; the global feature and the local feature of the image are obtained, the global feature of the image and the mask feature generated by the image decoder are fused by adding a feature fusion module, the details of the mask feature are enriched, the segmentation accuracy of the model for the butterfly ecological image is improved, and only the parameters of the double-path convolution module and the feature fusion module with less training parameters are trained.

Description

Technical Field

[0001] This invention belongs to the field of image segmentation technology, specifically relating to a butterfly ecology image segmentation method and system based on SAM. Background Technology

[0002] In butterfly ecology images, the butterfly's mimicry makes it highly similar to the background. Therefore, the accuracy of automated identification of butterfly ecology images containing the background is much lower than that of butterfly ecology images with most of the background removed. Existing image segmentation networks suffer from insufficient accuracy, fragmented boundaries, and confusion between butterfly pixels and background pixels when segmenting butterfly ecology images. Therefore, a dedicated segmentation model for butterfly ecology images is needed.

[0003] Kirillov, Alexander, and others proposed the SAM model in their paper "Segment Anything." This model consists of an image encoder, an image decoder, and a cue encoder. The image is input to the image encoder, and points, bounding boxes, and text cues are input to the cue encoder. The features generated by the image encoder and cue encoder are then input to the image decoder to obtain the butterfly mask. However, SAM still lacks sufficient accuracy for butterfly ecological image segmentation and has a large number of parameters, making it difficult to train. Summary of the Invention

[0004] To address the problems existing in current technologies, this invention provides a butterfly ecological image segmentation method based on SAM (Simulation Awareness Model). It aims to solve the issues of unsatisfactory segmentation accuracy and high training costs associated with SAM in butterfly ecological image segmentation. By incorporating a dual-channel convolutional module, the features generated by the image encoder are further extracted to obtain global and local features of the image. Furthermore, a feature fusion module is added to fuse the global features of the image and the mask features generated by the image decoder, enriching the details of the mask features and improving the model's segmentation accuracy for butterfly ecological images. By freezing the parameters of the image encoder, cue encoder, and mask decoder, only the parameters of the dual-channel convolutional module and the feature fusion module, which have a smaller number of parameters, are trained.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a butterfly ecological image segmentation method based on SAM, comprising the following steps:

[0006] The resized butterfly ecological image is labeled using a masking method to obtain a binarized butterfly ecological image.

[0007] The original butterfly ecological image is input into the butterfly segmentation network. The butterfly segmentation network includes an image encoder, a cue encoder, an image decoder, a dual-channel convolutional module, and a feature fusion module. The image encoder is a pre-trained image encoder of the SAM model, composed of 12 Transformer modules connected in sequence, used to encode the butterfly ecological image. The image feature vector output by the last Transformer module is input to the image decoder. The cue encoder is a pre-trained cue encoder of the SAM model, used to encode points, text, and anchor box cues into cue vectors. The image decoder is a pre-trained image decoder of the SAM model, used to decode the butterfly ecological image feature vector generated by the image encoder and the cue vector generated by the cue encoder to generate a mask feature vector. The dual-channel convolutional module is used to further extract features from the feature encoding vectors generated by the 12 Transformer modules of the image encoder, including a global path and a local path. The global path uses the image feature vectors generated by the 3rd, 6th, 9th, and 12th Transformer modules of the image encoder to generate a global image feature vector, and the local path uses the image feature vectors generated by the remaining Transformer modules to generate a local image feature vector.

[0008] The final butterfly mask image is obtained by fusing the global feature vector, local feature vector, and mask feature vector of the image.

[0009] Furthermore, the resized butterfly ecological images are annotated using a masking method to obtain binarized butterfly ecological images, including:

[0010] Adjust the butterfly ecology image to a height (H) and width (W), where C represents the image channel. The color RGB image has 3 channels. Denote the butterfly ecology image as... The butterfly boundary is marked using a mask. The labelimg software is used to mark several points on the boundary of the butterfly. These points are then connected by line segments to form a closed image. The inside of the closed image is set to 1 and the outside is set to 0, resulting in a PNG image composed of 0 and 1. The area with a value of 1 is the butterfly area.

[0011] Furthermore, when training the butterfly segmentation network: the parameters of the image encoder, cue encoder, and image decoder are frozen, and only the parameters of the dual-channel convolutional module and the feature fusion module are trained; the loss function used to train the parameters of the dual-channel convolutional module and the feature fusion module is:

[0012]

[0013] in The binary cross-entropy loss function BCE is determined by the following formula;

[0014]

[0015] in, For specific pixels, For pixels The probability of predicting it to be a butterfly. for Corresponding to real tags, The total number of pixels in an image;

[0016] The Dice loss function is determined by the following formula;

[0017]

[0018] in, For specific pixels, For pixels The probability of predicting it to be a butterfly. for Corresponding to real tags, This represents the total number of pixels in an image.

[0019] Furthermore, the global path consists of four basic convolutional modules connected sequentially. The image feature vector generated by the third Transformer module of the image encoder is input into the first basic convolutional module of the global path. The image feature vector output by the first basic convolutional module is added to the image feature vector generated by the sixth Transformer module of the image encoder and then input into the second basic convolutional module of the global path. The image feature vector output by the second basic convolutional module is added to the image feature vector generated by the ninth Transformer module of the image encoder and then input into the third basic convolutional module of the global path. The image feature vector output by the third basic convolutional module is added to the image feature vector generated by the twelfth Transformer module of the image encoder and then input into the fourth basic convolutional module of the global path. The resulting image feature vector is the global image feature vector.

[0020] Furthermore, the local path consists of eight basic convolutional modules connected sequentially. The image feature vector generated by the first Transformer module of the image encoder is input into the first basic convolutional module of the local path. The image feature vector output by the first basic convolutional module is added to the image feature vector generated by the second Transformer module of the image encoder, and then input into the second basic convolutional module of the local path. The image feature vector output by the second basic convolutional module is added to the image feature vector generated by the fourth Transformer module of the image encoder, and then input into the third basic convolutional module of the local path. The image feature vector output by the third basic convolutional module is added to the image feature vector generated by the fifth Transformer module of the image encoder, and then input into the fourth basic convolutional module of the local path. The image feature vector output by the block is added to the image feature vector generated by the 7th Transformer module of the image encoder, and then input into the 5th basic convolution module of the local path. The image feature vector output by the 5th basic convolution module is added to the image feature vector generated by the 8th Transformer module of the image encoder, and then input into the 6th basic convolution module of the local path. The image feature vector output by the 6th basic convolution module is added to the image feature vector generated by the 10th Transformer module of the image encoder, and then input into the 7th basic convolution module of the local path. The image feature vector output by the 7th basic convolution module is added to the image feature vector generated by the 11th Transformer module of the image encoder, and then input into the 8th basic convolution module of the local path. The resulting image feature vector is the local image feature vector.

[0021] Furthermore, the basic convolutional module is used to extract features from the image feature vector generated by the image encoder Transformer module. It consists of a 3*3 convolutional layer with a stride of 1, a normalization layer, a ReLU activation function layer, a 3*3 convolutional layer with a stride of 1, a normalization layer, and a ReLU activation function layer concatenated in sequence.

[0022] Furthermore, the feature fusion module consists of a learnable 1*256-dimensional vector and a three-layer perceptron. The 1*256-dimensional vector is processed by the three-layer perceptron into a 1*32-dimensional vector. The global feature vector, local feature vector, and mask vector generated by the dual-path convolution module are added together, and then the 1*32-dimensional vector is used to perform matrix multiplication with it to obtain the final segmented butterfly mask image.

[0023] Secondly, the present invention also provides a butterfly ecological image segmentation system based on SAM, including an image preprocessing module, a feature extraction module, and a feature fusion module;

[0024] The image preprocessing module is used to annotate resized butterfly ecological images using a masking method to obtain binarized butterfly ecological images.

[0025] The feature extraction module is used to input the original butterfly ecological image into the butterfly segmentation network. The butterfly segmentation network includes an image encoder, a cue encoder, an image decoder, a dual-channel convolution module, and a feature fusion module. The image encoder is a pre-trained image encoder of the SAM model, composed of 12 Transformer modules connected in sequence, used to encode the butterfly ecological image. The image feature vector output by the last Transformer module is input into the image decoder. The cue encoder is a pre-trained cue encoder of the SAM model, used to encode points, text, and anchor box cues into cue vectors. The image decoder is a pre-trained image decoder of the SAM model, used to decode the butterfly ecological image feature vector generated by the image encoder and the cue vector generated by the cue encoder to generate a mask feature vector. The dual-channel convolution module is used to further extract features from the feature encoding vectors generated by the 12 Transformer modules of the image encoder, including a global path and a local path. The global path uses the image feature vectors generated by the 3rd, 6th, 9th, and 12th Transformer modules of the image encoder to generate a global image feature vector, and the local path uses the image feature vectors generated by the remaining Transformer modules to generate a local image feature vector.

[0026] The feature fusion module is used to fuse the global feature vector, local feature vector and mask feature vector of the image to obtain the final butterfly mask image.

[0027] Thirdly, the present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and the processor can implement the SAM-based butterfly ecological image segmentation method of the present invention when executing the executable program.

[0028] A computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, it can implement the SAM-based butterfly ecological image segmentation method described in this invention.

[0029] Compared with the prior art, the present invention has at least the following beneficial effects:

[0030] SAM requires training an image encoder, a cue encoder, and a mask decoder, requiring 1191M training parameters. This invention only requires training a dual-path convolution module and a feature fusion module, requiring approximately 14.7M training parameters, which is far less than the number of training parameters required by SAM. Training can be completed on a single GPU. Therefore, the training cost of this invention is far lower than retraining the SAM model.

[0031] By inputting the image encoding features generated by each Transformer module in the image encoder into a dual-channel convolutional module for further feature extraction, the information generated by each Transformer module is fully utilized, learning more knowledge of image details and generating global and local image features. The global image features, local image knowledge, and image mask features generated by the mask decoder are then input into a feature fusion module and fused using an additive method, enriching the details of the image mask features. Finally, a learnable vector is used to transform the image mask features into the final segmentation mask output, improving the segmentation accuracy of butterfly ecological images. Compared with the SAM model, this invention improves the IoU index of butterfly ecological image segmentation from 89.82% to 91.78%, and the MIoU index from 94.76% to 95.78%. Attached Figure Description

[0032] Figure 1 A flowchart illustrating an embodiment of the present invention.

[0033] Figure 2 The overall flowchart provided for this invention.

[0034] Figure 3 The flowchart of the dual-channel convolution module provided by this invention.

[0035] Figure 4 This is the global pathway flowchart provided by the present invention.

[0036] Figure 5 This is a partial pathway flowchart provided by the present invention.

[0037] Figure 6 This is a flowchart of the basic convolution module provided by the present invention.

[0038] Figure 7 This is a flowchart of the feature fusion module provided by the present invention.

[0039] Figure 8 This is a schematic diagram of the image segmentation results provided by the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0042] refer to Figure 1 This invention provides a butterfly ecology image segmentation method based on SAM, comprising the following steps:

[0043] S1, construct the butterfly segmentation dataset, resize the butterfly images to a height H and width W pixels, and label them as follows. C represents the three channels of the image. The labelimg software is used to mark several points on the boundary of the butterfly. These points are connected by line segments to form a closed image. The value inside the closed image is 1 and the value outside is 0, resulting in a PNG image composed of 0 and 1. The area with a value of 1 is the butterfly area. The dataset is divided into two parts in a 1:1 ratio, one part is used as the training set and the other part is used as the test set.

[0044] S2, construct a butterfly segmentation network. This network, in addition to the image encoder, cue encoder, and image decoder already present in SAM, also includes a dual-channel convolutional module and a feature fusion module. The image encoder consists of 12 Transformer modules connected sequentially. An image is input to the image encoder, and each Transformer module generates an image feature vector. The image feature vector generated by the 12th Transformer module is then used by the cue encoder to generate a cue vector. Both the image feature vector and the cue vector are input to the mask decoder, which generates a mask feature vector. The 12 image feature vectors generated by the 12 Transformer modules are then input to the dual-channel convolutional module. The dual-channel convolutional module includes a global path and a local path. The global path consists of four basic convolutional layers. The convolutional modules are connected sequentially. The image feature vector generated by the third Transformer module of the image encoder is input into the first basic convolutional module of the global path. The image feature vector output by the first basic convolutional module is added to the image feature vector generated by the sixth Transformer module of the image encoder and then input into the second basic convolutional module of the global path. The image feature vector output by the second basic convolutional module is added to the image feature vector generated by the ninth Transformer module of the image encoder and then input into the third basic convolutional module of the global path. The image feature vector output by the third basic convolutional module is added to the image feature vector generated by the twelfth Transformer module of the image encoder and then input into the fourth basic convolutional module of the global path. The resulting image feature vector is the global image feature vector.The local path consists of eight basic convolutional modules connected sequentially. The image feature vector generated by the first Transformer module of the image encoder is input into the first basic convolutional module of the local path. The image feature vector output from the first basic convolutional module is added to the image feature vector generated by the second Transformer module of the image encoder, and then input into the second basic convolutional module of the local path. The image feature vector output from the second basic convolutional module is added to the image feature vector generated by the fourth Transformer module of the image encoder, and then input into the third basic convolutional module of the local path. The image feature vector output from the third basic convolutional module is added to the image feature vector generated by the fifth Transformer module of the image encoder, and then input into the fourth basic convolutional module of the local path. The image feature vector output from the fourth basic convolutional module is added to the image feature vector generated by the seventh Transformer module of the image encoder, and then input into the fifth basic convolutional module of the local path. The image feature vector output by the basic convolutional module is added to the image feature vector generated by the 8th Transformer module of the image encoder, and then input into the 6th basic convolutional module of the local path. The image feature vector output by the 6th basic convolutional module is added to the image feature vector generated by the 10th Transformer module of the image encoder, and then input into the 7th basic convolutional module of the local path. The image feature vector output by the 7th basic convolutional module is added to the image feature vector generated by the 11th Transformer module of the image encoder, and then input into the 8th basic convolutional module of the local path. The resulting image feature vector is the local image feature vector. The global image feature vector, local feature vector, and mask feature vector are input into the feature fusion module. In the feature fusion module, the global image feature vector, local feature vector, and mask feature vector are added together. The learnable 1*256-dimensional vector is first reduced to 1*32-dimensional, and then the 1*32-dimensional vector is multiplied with the fused features to obtain the final butterfly mask image output. (Reference) Figure 3 , Figure 4 , Figure 5 and Figure 6 .

[0045] S3, construct the loss function according to the following formula. ;

[0046]

[0047] in The binary cross-entropy loss function BCE is determined by the following formula;

[0048]

[0049] in, For specific pixels, For pixels The probability of predicting it to be a butterfly. for Corresponding to real tags, The total number of pixels in an image;

[0050] The Dice loss function is determined by the following formula;

[0051]

[0052] in, For specific pixels, For pixels The probability of predicting it to be a butterfly. for Corresponding to real tags, The total number of pixels in an image;

[0053] S4. Input the training set into the butterfly detection network for training, freeze the parameters of the image encoder, cue encoder, and image decoder, and train only the parameters of the dual-channel convolutional module and the feature fusion module. The server graphics card for training is an RTX3090, the initial learning rate is 0.001, the number of training rounds is 20, the batch size for training is 4, and training continues until the loss function converges.

[0054] S5, save the converged weight file.

[0055] S6 loads the saved weight file of the butterfly ecological image network, inputs the test set into the butterfly segmentation network with the weights loaded, segments the butterfly ecological image, and outputs the final butterfly mask image.

[0056] Taking the dataset provided in the literature "Ecological Photo Dataset for Automatic Identification of Butterfly Species" as an example, this dataset contains 721 butterfly ecological images, covering 94 butterfly species, with at least one sample and a maximum of 61 samples for each species. The SAM-based butterfly ecological image segmentation model in this embodiment consists of the following steps:

[0057] (1) Constructing a butterfly segmentation dataset

[0058] Resize the butterfly image to a height (H) and width (W) of pixels, and mark it as... C represents the three channels of the image. The butterfly mask is marked according to the butterfly boundary using labelimg software to construct a butterfly segmentation dataset. The dataset is then divided into training and test sets in a 1:1 ratio.

[0059] (2) Constructing a butterfly segmentation network

[0060] Figure 2 A schematic diagram of the butterfly segmentation network is given. Figure 2 In this embodiment, the butterfly segmentation network consists of an image encoder, a cue encoder, a mask decoder, a dual-channel convolution module, and a feature fusion module. The image encoder is connected to the dual-channel convolution module and the mask decoder, and the feature fusion module is connected to the dual-channel convolution module and the mask decoder. Figure 3 A schematic diagram of the dual-channel convolution module in this embodiment is provided. Figure 3 In this context, a dual-path convolutional module consists of a global path and a local path; Figure 4 A schematic diagram of the global pathway structure in this embodiment is given. Figure 4 In this model, the global path consists of four basic convolutional modules connected sequentially. Figure 5 A schematic diagram of the local pathway in this embodiment is given. Figure 5 In this model, the local pathway is composed of eight basic convolutional modules connected sequentially; Figure 6 A schematic diagram of the basic convolutional module in this embodiment is provided. Figure 6 In this embodiment, the basic convolutional module is composed of a 3*3 convolutional layer with a stride of 1, a normalization layer, a ReLU activation function layer, a 3*3 convolutional layer with a stride of 1, a normalization layer, and a ReLU activation function layer, which are sequentially spliced ​​together. Figure 7 A schematic diagram of the feature fusion module in this embodiment is provided. Figure 7 In this embodiment, the feature fusion module consists of a 1*256 dimensional vector and a three-layer perceptron.

[0061] Because this invention employs a dual-path convolution module and a feature fusion module, it enhances the model's ability to extract features from butterfly images, solves the existing problem of difficult butterfly image feature extraction, improves the accuracy of butterfly segmentation, and freezes the parameters of the image encoder, cue encoder, and mask decoder during training, reducing the number of training parameters required by the model and thus lowering the model training cost.

[0062] (3) Loss function

[0063] loss function Construct according to the following formula;

[0064]

[0065] in The binary cross-entropy loss function BCE is determined by the following formula;

[0066]

[0067] in, For specific pixels, For pixels The probability of predicting it to be a butterfly. for Corresponding to real tags, The total number of pixels in an image;

[0068] The Dice loss function is determined by the following formula;

[0069]

[0070] in, For specific pixels, For pixels The probability of predicting it to be a butterfly. for Corresponding to real tags, The total number of pixels in an image;

[0071] (4) Training the butterfly segmentation network

[0072] The training set was fed into the butterfly detection network for training. The server used for training had an RTX 3090 graphics card, an initial learning rate of 0.001, 20 training epochs, and a batch size of 4. Training continued until the loss function converged.

[0073] (5) Save the model

[0074] During the training of the butterfly detection network, the weight file after convergence is retained.

[0075] (6) Testing the butterfly segmentation network

[0076] The test set is input into the trained butterfly segmentation network, the weight file is loaded for testing, the butterfly image is segmented, the butterfly mask is output, and the performance of the method is verified by comparing it with the real information.

[0077] Complete the butterfly segmentation method based on SAM.

[0078] The method described in this embodiment was used to conduct a computer simulation experiment on six butterflies. The experimental results are shown below. Figure 8 .

[0079] exist Figure 8 In the figures, (a)-(f) represent images of butterfly ecology in the wild environment within the test set detected by the method of the embodiments. Figure 8 As can be seen, the method of the present invention has a good effect on butterfly species segmentation in the wild environment, with smooth edges, and can obtain complete butterfly pixels.

[0080] To verify the effectiveness of the method of this invention, a comparative experiment was conducted between the butterfly segmentation method based on SAM in the embodiments and the SAM, SAM-HQ, MobileSAM, FastSAM, EfficientSAM, TinySAM, and MSA segmentation methods to compare the performance of each segmentation method. IoU (Intersection over Union) and MIoU (Mean Intersection over Union) were used as evaluation metrics. The experimental results are shown in Table 1.

[0081] Table 1. Experimental results (%) of the method in this embodiment and the comparison segmentation method

[0082]

[0083] As shown in Table 1, the IoU and MIoU evaluation indices of the detection method of the present invention are both optimal values, which are higher than those of the other 7 comparative experimental methods.

[0084] On the other hand, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the SAM-based butterfly ecological image segmentation method described in the present invention.

[0085] The present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads the computer executable program from the memory and executes it, and the processor can implement the butterfly ecological image segmentation method based on SAM described in the present invention when executing the computer executable program.

[0086] The computer device may be a laptop, a desktop computer, or a workstation.

[0087] The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or an off-the-shelf programmable gate array (FPGA).

[0088] The memory described in this invention can be an internal storage unit of a laptop, desktop computer, or workstation, such as memory or hard disk; or it can be an external storage unit, such as a portable hard disk or flash memory card.

[0089] Computer-readable storage media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. Random access memory can include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0090] This invention has been described through embodiments. Those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of this invention, modifications can be made to these features and embodiments to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, this invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of this invention.

[0091] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A SAM-based butterfly ecological image segmentation method, characterized in that, The method comprises the following steps: The butterfly boundary of the resized butterfly ecological image is labeled in a manner of using a mask to obtain a binary butterfly ecological image, which constitutes a butterfly image dataset, and the butterfly image dataset is divided into a training set and a test set; The original butterfly ecological image is input into a butterfly segmentation network; the butterfly segmentation network comprises an image encoder, a prompt encoder, an image decoder, a double-path convolution module, and a feature fusion module; the image encoder is a pre-trained image encoder of a SAM model, which is composed of 12 Transformer modules connected in sequence and is used for encoding the butterfly ecological image; the image feature vector output by the last Transformer module is input into the image decoder; the prompt encoder is a pre-trained prompt encoder of the SAM model, which is used for encoding points, texts, and anchor boxes into prompt vectors; the image decoder is a pre-trained image decoder of the SAM model, which is used for decoding the butterfly ecological image feature vector generated by the image encoder and the prompt vector generated by the prompt encoder to generate a mask feature vector; the double-path convolution module is used for further feature extraction of the feature encoding vectors generated by the 12 Transformer modules of the image encoder, and comprises a global path and a local path; the global path generates a global image feature vector using the image feature vectors generated by the 3rd, 6th, 9th, and 12th Transformer modules of the image encoder; the local path generates a local image feature vector using the image feature vectors generated by the remaining Transformer modules; the feature fusion module is composed of a learnable 1*256-dimensional vector and a three-layer perception machine; the 1*256-dimensional vector is processed by the three-layer perception machine to become a 1*32-dimensional vector; the global feature vector, the local feature vector, and the mask vector generated by the mask decoder are added, and then the 1*32-dimensional vector is used for matrix multiplication to obtain a final segmented butterfly mask image; When training the constructed butterfly segmentation network, the parameters of the image encoder, the prompt encoder, and the image decoder are frozen, and only the parameters of the double-path convolution module and the feature fusion module are trained; The image global feature vector, the local feature vector, and the mask feature vector are fused to obtain a final butterfly mask image.

2. The SAM-based butterfly ecological image segmentation method according to claim 1, characterized in that, The resized butterfly ecological image is labeled in a manner of using a mask to obtain a binary butterfly ecological image, which comprises: The butterfly ecological image is adjusted to a high-H and wide-W image, C represents an image channel, the color RGB image channel number is 3, and the butterfly ecological image is denoted as A mask is used for labeling, a labelimg software is used to mark a plurality of points on a butterfly boundary, the plurality of points are connected by line segments to form a closed image, the inside of the closed image is valued as 1, and the outside is valued as 0, and a png image composed of 0 and 1 is obtained, wherein the region with a value of 1 is a butterfly region. 3.The SAM-based butterfly ecological image segmentation method according to claim 1, characterized in that, When training the constructed butterfly segmentation network, the parameters of the double-path convolution module and the feature fusion module are trained using a loss function: wherein represents the binary cross-entropy loss function BCE, determined by the following equation; wherein, is a specific pixel point, is a pixel point predicting the probability of being a butterfly class, is corresponding to the real label, is the total number of pixel points in an image; The Dice loss function is determined by the following equation: wherein, is a specific pixel point, is a pixel point predicting the probability of being a butterfly class, is corresponding true label, is the total number of pixel points of an image.

4. The SAM-based butterfly ecological image segmentation method according to claim 1, characterized in that, The global channel is connected by four basic convolution modules in turn, the image feature vector generated by the third Transformer module of the image encoder is input into the first basic convolution module of the global channel, the image feature vector output by the first basic convolution module is added to the image feature vector generated by the sixth Transformer module of the image encoder, and then input into the second basic convolution module of the global channel, the image feature vector output by the second basic convolution module is added to the image feature vector generated by the ninth Transformer module of the image encoder, and then input into the third basic convolution module of the global channel, the image feature vector output by the third basic convolution module is added to the image feature vector generated by the twelfth Transformer module of the image encoder, and then input into the fourth basic convolution module of the global channel, and the image feature vector generated by the fourth basic convolution module is the image global feature vector.

5. The SAM-based butterfly ecological image segmentation method according to claim 1, characterized in that, The local channel is connected by eight basic convolution modules in turn, the image feature vector generated by the first Transformer module of the image encoder is input into the first basic convolution module of the local channel, the image feature vector output by the first basic convolution module is added to the image feature vector generated by the second Transformer module of the image encoder, and then input into the second basic convolution module of the local channel, the image feature vector output by the second basic convolution module is added to the image feature vector generated by the fourth Transformer module of the image encoder, and then input into the third basic convolution module of the local channel, the image feature vector output by the third basic convolution module is added to the image feature vector generated by the fifth Transformer module of the image encoder, and then input into the fourth basic convolution module of the local channel, the image feature vector output by the fourth basic convolution module is added to the image feature vector generated by the seventh Transformer module of the image encoder, and then input into the fifth basic convolution module of the local channel, the image feature vector output by the fifth basic convolution module is added to the image feature vector generated by the eighth Transformer module of the image encoder, and then input into the sixth basic convolution module of the local channel, the image feature vector output by the sixth basic convolution module is added to the image feature vector generated by the tenth Transformer module of the image encoder, and then input into the seventh basic convolution module of the local channel, the image feature vector output by the seventh basic convolution module is added to the image feature vector generated by the eleventh Transformer module of the image encoder, and then input into the eighth basic convolution module of the local channel, and the image feature vector generated by the eighth basic convolution module of the local channel is the image local feature vector.

6. The SAM-based butterfly ecological image segmentation method according to claim 4 or 5, characterized in that, The basic convolution module is used for feature extraction of the image feature vector generated by the Transformer module of the image encoder, which is sequentially spliced by one convolution layer with a size of 3*3 and a step of 1, one normalization layer, one relu activation function layer, one convolution layer with a size of 3*3 and a step of 1, one normalization layer, and one relu activation function layer.

7. A SAM-based butterfly ecological image segmentation system, characterized in that, The method comprises an image preprocessing module, a feature extraction module, and a feature fusion module. The image preprocessing module is configured to mark the butterfly boundary of the resized butterfly ecological image using a mask method to obtain a binary butterfly ecological image, and to construct a butterfly image dataset. The feature extraction module is configured to input an original butterfly ecological image into a butterfly segmentation network. The feature fusion module is configured to fuse the image global feature vector, the local feature vector, and the mask feature vector to obtain a final butterfly mask image.

8. A computer device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the SAM-based butterfly ecological image segmentation method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the SAM-based butterfly ecological image segmentation method according to any one of claims 1-6.

Images