A method for multi-part semantic segmentation and species joint identification of shrimp and crab crustaceans

Abstract

The application discloses a shrimp and crab shellfish multi-part semantic segmentation and species joint identification method, relates to the technical field of computer vision and deep learning image semantic segmentation, and comprises the following steps: S1, visual feature extraction; S2, text feature extraction; S3, obtaining target feature coding; S4, residual light weight multi-scale hollow convolution enhancement; S5, multi-level feature aggregation; S6, two-stage adaptive frame decoding; S7, segmentation mask generation; S8, shared feature extraction: the mask feature obtained in the step S6 is reused as shared pixel embedding; S9, species classification prediction; the application is oriented to single lobster species accurate multi-part segmentation, a light weight multi-scale enhancement module and a phased adaptive attention decoder are designed, cross-scale segmentation precision is improved under a very low parameter increment, a multi-task joint framework is built based on the SAM3-LC unit, and a species classification head bidirectional collaborative optimization network feature extraction capability is newly added.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning image semantic segmentation technology, specifically a method for semantic segmentation of multiple parts of crustaceans such as shrimp and crab and joint species identification. Background Technology

[0002] In the intelligent farming and processing of crustaceans such as lobsters and crabs, it is necessary to automatically distinguish the three functional areas of the claws, carapace, and tail, and assess growth specifications and meat yield by using the pixel area and outline size of each part; at the same time, it is also necessary to distinguish different shrimp and crab species and complete classification and sorting. Existing image segmentation technology has three major shortcomings: 1. Poor multi-scale feature adaptability: Lobster claws have a very small pixel ratio, while the cephalothorax and tail have large areas; the standard hollow spatial pyramid pooling void ratio is set to [6, 12, 18], resulting in an excessively large receptive field that introduces a large number of background pixels, diluting the features of small targets like claws; moreover, the standard hollow spatial pyramid pooling parameter size is approximately 0.26M, resulting in high computational overhead and making it difficult to adapt to real-time aquatic product detection equipment; directly using the SAM3 native pixel decoder without targeted multi-scale enhancement leads to low cross-scale segmentation accuracy. 2. Global attention defects of SAM3 decoder: The native 6-layer Transformer decoder uses global cross attention. In the deep decoding stage, background pixels continue to participate in feature aggregation, the effective features of small chelicerae are covered by noise, feature competition occurs between different target queries, the segmentation boundary is blurred and the inference speed is slow. 3. Single-task architecture cannot jointly identify multiple species: Existing segmentation models only complete pixel segmentation. Species identification requires building a separate CNN / ViT classification network and running it serially. The two inferences lead to accumulated errors. Training the two networks separately cannot share image morphological features. Morphological cues such as chelipeds and cephalothorax that distinguish species cannot be bidirectionally empowered for segmentation and classification tasks, resulting in low overall accuracy and efficiency. Existing mainstream segmentation models such as U-Net, DeepLabV3+, and SegFormer lack generalization ability in multi-scale and multi-species scenarios of shrimp and crabs. SAM3, as an open vocabulary-based model, has strong pre-training priors, but its original structure is not optimized for the features of aquatic crustaceans. Therefore, there is an urgent need for a method for semantic segmentation of multiple parts of shrimp and crab crustaceans and joint species identification to solve the above technical problems. Summary of the Invention

[0003] This invention provides a method for semantic segmentation of multiple body parts and joint species identification of crustaceans such as shrimp and crab, which adopts the following scheme: Option 1 (SAM3-LC Unit): For precise multi-part segmentation of a single lobster species, a lightweight multi-scale enhancement module and a staged adaptive attention decoder are designed to improve cross-scale segmentation accuracy with extremely low parameter increments. Option 2 (SAM3-MT Unit): For mixed scenarios involving multiple types of shrimp and crabs, a multi-task joint framework is built based on SAM3-LC, a new species classification head is added, and the shared backbone features are used to achieve end-to-end synchronous inference of part segmentation and species identification, eliminating serial errors and bidirectionally coordinating to optimize the network feature extraction capabilities. To achieve the above objectives, the present invention provides the following technical solution: A method for semantic segmentation of multiple body parts and joint species identification in crustaceans such as shrimp and crab includes the following steps: S1. Visual Feature Extraction: Input the input image into the visual encoder to extract multi-resolution feature maps; S2. Text feature extraction: Input the text prompts corresponding to the three parts of the chelipeds, carapace and tail into the text encoder to obtain the corresponding text feature embeddings; S3. Obtain target feature encoding: Input the multi-resolution feature map and the text feature embedding into the visual-language fusion module, and fuse them through the cross-attention mechanism between visual features and text embedding to generate target feature encoding; S4. Residual lightweight multi-scale dilated convolution enhancement: Input the target feature encoding into the lightweight dilated spatial pyramid pooling module, perform multi-scale dilated convolution enhancement, and obtain enhanced features; S5. Multi-level feature aggregation: The enhanced features are input into the FPN feature pyramid network module for multi-level fusion, and high-resolution pixel features are output. S6. Two-stage adaptive bounding box decoding: The high-resolution pixel features are input into the Transformer decoder, and coarse localization and fine-tuning iterative optimization are performed sequentially to obtain mask features; S7. Segmentation Mask Generation: The mask features are processed by Mask MLP and dot product operation is performed with the pixel embedding to output the pixel-level segmentation mask corresponding to the three parts of the chelicerae, cephalothorax and tail, thus completing semantic segmentation. S8. Shared feature extraction: The mask features obtained in step S6 are reused as shared pixel embeddings; S9. Species classification prediction: The shared pixels are embedded and compressed into a one-dimensional vector through adaptive global average pooling, and input into the species classification head to output the shrimp and crab species classification results, thus completing the joint species identification.

[0004] As a preferred embodiment of the present invention, in step S4, the lightweight void space pyramid pooling module includes a lightweight module, an ultra-lightweight channel attention module, and a residual gating module connected in series.

[0005] As a preferred embodiment of the present invention, the lightweight module performs the following operations: Set the branch hole rate set of the lightweight module to The standard empty spatial pyramid pooling global average pooling branch is removed, the number of intermediate channels is compressed, and the branch output channels are concatenated and then subjected to 1×1 convolution to restore the original number of channels. The formula for calculating the number of intermediate channels is as follows: Where C represents the input feature channel; The output format of the branch output channel is represented as follows: in, For the first The output feature map of each dilated convolution branch Values , respectively corresponding to a void ratio The dilated convolution is defined as follows: X is the input feature map, k=3 is the kernel size, d=r is the dilation rate, and BN is batch normalization. This is the ReLU activation function.

[0006] As a preferred embodiment of the present invention, the ultra-lightweight channel attention module uses global average pooling to compress features, and then removes max pooling and spatial attention branches. The channel compression ratio is fixed at 64. Two fully connected layers compress the channels to 4 dimensions and then restore them. The output channel weights are multiplied by the input features channel by channel. Its forward propagation process is expressed by the following formula: in, This is the feature map after multi-branch dilated convolution stitching and projection. This is the channel descriptor after global average pooling. and This is the weight matrix for the fully connected layer, with its bottleneck layer reduction ratio set to 64. It is the ReLU activation function. For the Sigmoid function, For dynamic channel weights, This indicates multiplication by channel.

[0007] As a preferred embodiment of the present invention, the residual gating module outputs the following formula: in, This refers to the layer index in the feature pyramid. For the first The input features of the layer For the first The output features of the layer This is the composite enhancement function of the lightweight module and the ultra-lightweight channel attention module. For the first Learnable gated scalar parameters of the layer.

[0008] In a preferred embodiment of the present invention, the multi-resolution feature map output by the visual encoder is fed into the visual-language fusion module, which outputs the highest resolution feature layer. The target feature encoding is then subjected to multi-scale dilated convolution enhancement via the lightweight dilated spatial pyramid pooling module. Specifically, the lightweight module captures multi-scale receptive field information, the ultra-lightweight channel attention module performs channel recalibration, and a residual gating module ensures the enhanced features are not corrupted during the initial training phase. These enhanced features, along with features from the other three layers, are fed into the FPN feature pyramid network module, where they are progressively upsampled and fused to generate high-resolution pixel embeddings. ; The Transformer decoder is a 6-layer decoder, which adopts a layered structure of 3 layers for coarse localization and 3 layers for fine localization.

[0009] In a preferred embodiment of the present invention, in step S6, the coarse localization stage employs global cross-attention to obtain the initial bounding box of the target; the fine-tuning iterative optimization stage dynamically generates a local attention window based on the predicted initial bounding box of the target, performs cross-attention within the local attention window, and iteratively optimizes to obtain the mask features.

[0010] As a preferred embodiment of the present invention, the local attention window generation includes the following steps: a. The coarse localization output normalized initial bounding box of the target is... The reference side length of the local attention window in the image pixel space Window relative scaling factor Input image pixel size =560; b. The reference side length The feature map window side length is obtained by cropping to the [32, 256] pixel range and dividing by the patch size of the image segmented by the visual decoder. ; c. Calculate the top, bottom, left, and right boundaries of the window with the center of the frame as the anchor. The attention bias outside the local attention window is assigned a negative infinity value, and the local attention window participates in the attention weighting calculation. The formula is as follows: in, For boundary clipping functions, restrict the input to... Within the range, and These are the set minimum and maximum pixel thresholds, respectively. , , The number of patches for segmenting the image by the visual decoder. =14; d. Establish an attention bias matrix based on the dynamically generated local attention window. , its first Each element is defined as: in, For the first The query pair for the first Attention bias at each spatial location For the first feature map The coordinates of a spatial location, For the first A local window area for each query; e. By resetting the attention weight outside the local attention window to The attention mechanism is equivalent to performing calculations only within the local attention window: in, The attention weights after applying the bias. For the first Target query vector For the first The key vector corresponding to each image location For the dimensions of query and key, Scaling factor For the first The query pair for the first Attentional bias at each spatial location.

[0011] As a preferred embodiment of the present invention, the species classification head is a single-layer fully connected classification head, which includes a global average pooling layer and a single-layer fully connected layer, with a total parameter count of 256×C+C, where C is the number of species categories, and the number of output channels of the FPN feature pyramid network module is 256.

[0012] A computer-readable storage medium storing a computer program that is executed by one or more processors to implement the method as described in this invention.

[0013] This invention provides a method for semantic segmentation of multiple body parts and joint species identification in crustaceans such as shrimp and crab. It has the following beneficial effects: 1. The lightweight hollow space pyramid pooling module of the present invention can effectively reduce the number of parameters compared with the standard hollow space pyramid pooling module. Its compact small receptive field is perfectly adapted to the pixel distribution of concentrated parts of shrimp and crab. Residual gating solves the problem of pre-trained model fine-tuning crash. 2. The two-stage adaptive box decoding of this invention: the model predicts the boundary constraints of the attention range itself, without the need for an external detector, which balances global detection capability and local boundary fineness, and reduces the amount of computation; 3. This invention has dual generalization ability: it can stably improve the accuracy of single-species lobster, multi-species shrimp and crab, and general-scenario ADE20K dataset, and can be transferred to other agricultural biological segmentation tasks; 4. Improved segmentation accuracy of this invention: mIoU of 94.38% for the SAM3-LC lobster dataset, surpassing all baseline models such as U-Net and SegNeXt; the most significant improvement is achieved for large targets such as the cephalothorax; mIoU of 82.76% for the SAM3-MT shrimp and crab hybrid dataset, with the best segmentation metrics for the tail and cephalothorax. 5. Inference efficiency optimization of this invention: During the refinement stage, local window attention reduces the computational complexity from O(HW) to O(sfeat2), resulting in a significant speed improvement for small target scenes; 6. The bidirectional collaborative gain of this invention: the species classification accuracy is 78.30% under the multi-task architecture, which is better than dedicated classification networks such as ConvNeXt-T and Swin-T; 7. The present invention has strong training stability: the residual gating module avoids the performance drop in the early stage of fine-tuning, the ablation experiment proves that each module is independent and effective, and the hierarchical hyperparameter has high robustness. 8. This invention has strong versatility: it is not only suitable for aquatic shrimp and crab, but also improves the segmentation effect of general scenes in the ADE20K dataset, and can be transferred to the field of fruit and vegetable and livestock image detection. Attached Figure Description

[0014] Figure 1 This is a flowchart of a method for semantic segmentation and species joint identification of multiple parts of crustaceans such as shrimp and crab, according to the present invention; Figure 2 This is a framework diagram of a multi-part semantic segmentation and species joint identification system for crustaceans such as shrimp and crabs according to the present invention; Figure 3 This is a frame diagram of the lightweight hollow space pyramid pooling module in this invention; Figure 4 This is a framework diagram of the lightweight module in this invention; Figure 5 This is a framework diagram of the ultra-lightweight channel attention module in this invention; Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0016] Example 1: Please see Figure 1 This invention provides a method for semantic segmentation and joint species identification of multiple parts of crustaceans such as shrimp and crab, which includes the following steps: S1. Visual Feature Extraction: Input the input image to the visual encoder to extract multi-resolution feature maps. Input image I∈ The 3×H×W array first passes through a SAM3-LC unit visual encoder, which then extracts multi-scale visual features using a Transformer layer that alternates between window attention and global attention. The visual encoder outputs four feature layers at different resolutions. , C , The spatial dimensions decrease layer by layer from low to high, while the channel dimensions increase layer by layer. It should be noted that this invention only needs to work on the highest resolution feature layer subsequently. The other three feature layers with different resolutions do not need to be processed; S2. Text Feature Extraction: Input the text prompts corresponding to the three parts of the cheliped, carapace and tail into the text encoder, and encode them to obtain the corresponding text feature embeddings. The text encoder maps the category prompts ("cheliped", "carapace" and "tail") into text feature embeddings. S3. Obtain target feature encoding: Input the multi-resolution feature map and text feature embedding into the vision-language fusion module, and fuse them through the cross-attention mechanism between visual features and text embedding to generate target feature encoding; The cross-attention mechanism acts as a "bridge" connecting information flows from different modalities (such as visual and textual). It allows the model to dynamically retrieve and focus on relevant features in another modality while processing one modality. In the fusion of visual and textual information, its core consists of three key vectors: Query (Q): Usually derived from text embedding, it represents the semantic needs or task instructions that the model needs to focus on at the moment. Q determines "what information the model is looking for". Key (K): Usually derived from visual features, it represents the semantic label or index of each region in the image. K is used to match Q to answer "What key information is in the image"; Value (V): Also derived from visual features, it contains the actual content representation of the image region. When Q and K match successfully, the model will extract the specific visual content from V. This asymmetric design allows text to serve as a conditional signal, dynamically modulating and guiding the extraction of visual features, thereby achieving cross-modal alignment of task perception. The computational process of the cross-attention mechanism can be broken down into the following four core steps: 1. Similarity Calculation: First, calculate the degree of matching between the text query (Q) and all visual keys (K), which is typically done through matrix multiplication. This is achieved by obtaining a similarity matrix that reflects the association strength between each word in the text and each region in the image; 2. Scaling: To prevent the dot product result from becoming too large and causing the Softmax function to enter the saturation region with minimal gradients, the similarity matrix needs to be divided by a scaling factor. (in (where is the dimension of the key vector), this step ensures the stability of the gradient during backpropagation; 3. Softmax normalization: The scaled similarity matrix is ​​normalized using the Softmax function, which converts it into attention weights in the form of a probability distribution. These weights sum to 1, which intuitively shows the "degree of attention" the model pays to each region of the image when generating or understanding text. 4. Weighted summation: Finally, the visual values ​​(V) are weighted and summed using the calculated attention weights. This step aggregates the scattered visual features according to the semantic requirements of the text, and outputs a visual representation that incorporates the text context. Its core mathematical formula can be expressed as: S4. Residual lightweight multi-scale dilated convolution enhancement: The target feature encoding is input into the lightweight dilated spatial pyramid pooling module, and multi-scale dilated convolution enhancement is performed to obtain enhanced features. The lightweight dilated spatial pyramid pooling module includes a lightweight module, an ultra-lightweight channel attention module, and a residual gating module connected in sequence. The core idea of ​​step S4 is as follows: The multi-resolution feature maps output by the visual encoder are fed into the vision-language fusion module, which outputs the highest resolution feature layer. The target feature encoding is then subjected to multi-scale dilated convolutional enhancement via a lightweight dilated spatial pyramid pooling module. Specifically, a lightweight module captures multi-scale receptive field information, an ultra-lightweight channel attention module performs channel recalibration, and a residual gating module ensures that the enhanced features are not corrupted during the initial training phase. The enhanced features, along with features from the other three layers, are fed into the FPN feature pyramid network module, where they are progressively upsampled and fused to generate high-resolution pixel embeddings. Specifically: The lightweight module performs the following operations: Set the branch hole rate set of the lightweight module to The standard empty spatial pyramid pooling global average pooling branch is removed, the number of intermediate channels is compressed, and the output channels of the branches are concatenated and then subjected to a 1×1 convolution to restore the original number of channels. The formula for calculating the number of intermediate channels is: Where C represents the input feature channel; The branch output channel output format is represented as follows: in, For the first The output feature map of each dilated convolution branch Values , respectively corresponding to a void ratio The dilated convolution is defined as follows: X is the input feature map, k=3 is the kernel size, d=r is the dilation rate, and BN is batch normalization. It is the ReLU activation function; The ultra-lightweight channel attention module uses global average pooling to compress features, then removes max pooling and spatial attention branches, with a fixed channel compression ratio of 64. Two fully connected layers compress the channels to 4 dimensions and then restore them. The output channel weights are multiplied by the input features channel by channel. Its forward propagation process is expressed by the following formula: in, This is the feature map after multi-branch dilated convolution stitching and projection. This is the channel descriptor after global average pooling. and This is the weight matrix for the fully connected layer, with its bottleneck layer reduction ratio set to 64. It is the ReLU activation function. For the Sigmoid function, For dynamic channel weights, This indicates multiplication by channel; The output formula of the residual gating module is: in, This refers to the layer index in the feature pyramid. For the first The input features of the layer For the first The output features of the layer This is a composite enhancement function for the lightweight module and the ultra-lightweight channel attention module. For the first Regarding the learnable gating scalar parameters of the layer, it should be noted that: since this invention only applies to the highest resolution feature layer... Therefore, in practical applications, it is necessary to take... =3, that is This is the third level index in the feature pyramid; S5, Multi-level Feature Aggregation: The enhanced features are input into the FPN feature pyramid network module for multi-level fusion, and the high-resolution pixel features are output. S6. Two-stage adaptive bounding box decoding: High-resolution pixel features are input into the Transformer decoder, which performs coarse localization and fine-tuning iterative optimization to obtain mask features. The Transformer decoder is a 6-layer decoder, employing a hierarchical structure of 3 layers for coarse localization and 3 layers for fine-tuning. The Transformer decoder queries 200 learnable targets. As input, the target representation is iteratively optimized through 6 decoding layers. This invention divides the 6 layers into two functional stages: The first three layers are the coarse localization stage, which uses global cross attention to enable the query to discover and locate the target candidate region across the entire graph. The last three layers constitute the refinement stage. Using the bounding boxes predicted in the coarse localization stage, an adaptive local window is dynamically generated for each query, compressing the cross-attention range from the entire image to the window region. This achieves refined segmentation of the target boundary. This global-then-local design enables a gradual contraction of the attention range with decoding depth. Specifically: In the coarse localization stage, global cross-attention is used to obtain the initial bounding box of the target. In the fine-tuning iterative optimization stage, based on the predicted initial bounding box of the target, a local attention window is dynamically generated. Cross-attention is performed within the local attention window, and iterative optimization is carried out to obtain mask features, wherein: The generation of a local attention window includes the following steps: a. The coarse localization output normalized initial bounding box of the target is... The reference side length of the local attention window in the image pixel space Window relative scaling factor Input image pixel size =560; b. Set the reference side length The feature map window side length is obtained by cropping to the [32, 256] pixel range and dividing by the patch size of the image segmented by the visual decoder. ; c. Calculate the upper, lower, left and right boundaries of the window with the center of the box as the anchor. Assign negative infinity to the attention bias outside the local attention window and participate in the attention weighting calculation inside the local attention window. The formula is as follows: in, For boundary clipping functions, restrict the input to... Within the range, and These are the set minimum and maximum pixel thresholds, respectively. , , The amount of patch (small chunks) that the visual decoder divides the image into. =14, the initial sampling step size of the visual encoder is 2, and in order to quickly find the corresponding feature in the window, the sampling step size is changed to 14; d. Establish an attention bias matrix based on dynamically generated local attention windows. , its first Each element is defined as: in, For the first The query pair for the first Attention bias at each spatial location For the first feature map The coordinates of a spatial location, For the first A local window area for each query; e. By resetting the attention weight outside the local attention window to The attention mechanism is equivalent to performing computation only within a local attention window: in, The attention weights after applying the bias. For the first Target query vector For the first The key vector corresponding to each image location For the dimensions of query and key, Scaling factor For the first The query pair for the first Attentional bias at each spatial location; S7. Segmentation Mask Generation: The mask features are processed by Mask MLP and dot product operation is performed with the pixel embeddings to output pixel-level segmentation masks for the chelipeds, cephalothorax, and tail, completing semantic segmentation. The final layer output is the query... After masked prediction MLP mapping, and pixel embedding Perform a dot product to generate the final segmentation mask. The model training of this invention adopts a multi-task joint loss function, including bounding box regression loss, classification loss, masking loss and semantic segmentation loss. It simultaneously optimizes all parameters of the visual encoder, the lightweight dilated spatial pyramid pooling module and the Transformer decoder in an end-to-end manner. In this embodiment, the masking MLP processing is a multilayer perceptron with 2 linear layers, a hidden layer dimension of 2048 and the ReLU activation function. The ReLU activation function has been introduced above. The multi-task joint loss function includes: bounding box loss (smoothing L1 loss + generalized intersection-union loss, weights 5.0 + 2.0), classification loss (sigmoid focus loss, weight 20.0), presence loss (presence loss, weight 20.0), masking loss (sigmoid focus loss, weight 200.0), and Dice loss (weight 10.0), as well as semantic segmentation loss (focus loss, weight 20.0 + Dice loss, weight 30.0). The overall architecture can be formalized as a multi-task learning problem, making Indicates inclusion The training dataset contains samples, where For the input image, For the first Pixel-level segmentation labels for each part Image-level species tagging ( (where the number of species is ), the model simultaneously optimizes both segmentation and classification tasks, and the overall loss function is: in The loss is for part segmentation (including mask focus loss, Dice loss, and coarse mask auxiliary loss, etc.). The loss is classified as species (cross-entropy loss). Weighting coefficients for species classification loss are used to balance the contributions of the two tasks. This makes the gradient magnitudes of segmentation and classification similar; S8. Shared Feature Extraction: Reuse the mask features obtained in step S6 as shared pixel embeddings; S9. Species Classification Prediction: Shared pixel embeddings are compressed into a one-dimensional vector by adaptive global average pooling, input to the species classification head, and output the shrimp and crab species classification results to complete joint species identification; The species classification head is a single-layer fully connected classification head, which includes a global average pooling layer and a single-layer fully connected layer, with a total of 256×C+C, where C is the number of species categories, and the FPN feature pyramid network module has 256 output channels. Example 2: Please see Figures 2-5 This embodiment provides a semantic segmentation and species joint identification system for multiple parts of crustaceans such as shrimp and crab. The method described in Embodiment 1 is based on the system provided in this embodiment. This system is based on the traditional benchmark model SAM3 and is optimized and improved. It consists of innovative SAM3-LC units and SAM3-MT units. The SAM3-LC unit is used to achieve semantic segmentation, and the SAM3-MT unit is used to achieve species identification, as described in detail below: A. Regarding the SAM3-LC unit: The SAM3-LC unit consists of a visual encoder, a text encoder, a visual-speech fusion module, a lightweight dilated spatial pyramid pooling module, an FPN feature pyramid network module, and a Transformer decoder. The outputs of the visual encoder and the text encoder are connected to the input of the visual-speech fusion module. The output of the visual-speech fusion module is connected to the input of the lightweight dilated spatial pyramid pooling module. The output of the lightweight dilated spatial pyramid pooling module is connected to the input of the FPN feature pyramid network module. The output of the FPN feature pyramid network module is connected to the output of the Transformer decoder. About visual encoders: The visual encoder uses ViT-L (Vision Transformer Large) as the visual backbone network, which contains 32 Transformer layers, 16 attention heads, and an embedding dimension of 1024. The visual encoder introduces positional information through rotational position encoding (RoPE) and adopts a strategy of alternating window attention and global attention to balance computational efficiency and receptive field. About text encoders: The text encoder converts natural language prompts into semantic embedding vectors. The text processing pipeline consists of three main components: 1) SimpleTokenizer (Tokenizer) The Byte-Pair Encoding (BPE) sub-word segmentation algorithm is adopted, and vocabulary compression is performed based on the bpe_simple_vocab_16e6.txt.gz vocabulary list; (2) VETextEncoder (text encoder) The core class of the text encoder is responsible for mapping the segmented text to a high-dimensional semantic space, and consists of the following sub-modules: Token Embedding: An embedding layer with configurable vocab_size and width; Positional Embedding: A learnable positional encoding of shape [context_length, width]; Transformer stack: composed of ResidualAttentionBlock layers stacked together (approximately 24 layers); Final Projection: An optional text projection layer used to align dimensions with the visual backbone; Key constraint: The text encoder can accept a maximum of 32 tokens (30 content tokens + start / end tokens); The text encoder has approximately 354 million (353.72M) parameters—a figure significantly higher than the estimated value of ~150M in the parameter distribution table above, reflecting the importance of the text encoder in understanding open-vocabulary concepts; Regarding the visual-language fusion module: The visual-language fusion module is responsible for fusing visual features with multimodal cue features. It receives multi-scale feature maps from the visual encoder and cue embeddings from the text / geometric encoder, and performs cross-modal feature interaction and fusion through the Transformer encoder (6 layers). In the Keras implementation, the fusion encoder can be configured with parameters such as num_layers, hidden_dim, intermediate_dim, and num_heads; Regarding the lightweight void space pyramid pooling module: The lightweight, hollow spatial pyramid pooling module is a core innovation of the SAM3-LC unit. Addressing the need for multi-scale feature extraction from lobster-like parts, this invention designs a lightweight hollow spatial pyramid pooling module. This module aims to enhance the multi-scale receptive field capability of the SAM3 pixel decoder with minimal computational overhead, while ensuring safe insertion into pre-trained models for end-to-end fine-tuning. The structure diagram is shown below. Figure 3 As shown, the lightweight void space pyramid pooling module includes a lightweight module, an ultra-lightweight channel attention module, and a residual gating module connected in series. Regarding the lightweight module: The lightweight module performs the following operations: Set the branch hole rate set of the lightweight module to The global average pooling branch of the standard hollow spatial pyramid pooling is removed, the number of intermediate channels is compressed, and the output channels of the branches are concatenated and then subjected to a 1×1 convolution to restore the original number of channels. The formula for calculating the number of intermediate channels is: Where C represents the input feature channel; The complete forward propagation of the lightweight module can be expressed as: in, This represents the output of a 1×1 convolution branch, generated through 1×1 convolution, batch normalization, and ReLU activation. Feature map of each channel This represents the output of four dilated convolution branches. Each branch generates a multi-scale feature map with the same number of channels through 3×3 dilated convolutions with different dilation rates. This represents the final output of the lightweight module. The outputs of the five branches are concatenated along the channel dimension and then compressed back to the original number of channels using a 1×1 convolutional projection layer. ,in This indicates concatenation by channel dimension; the number of channels after concatenation is... ; Regarding the ultra-lightweight attention module: To enhance the channel responses relevant to the target region in the output features of the lightweight module and suppress irrelevant noise channels, this invention introduces an ultra-lightweight attention module after each lightweight module enhancement layer. The structure of this module is as follows: Figure 5 As shown, the ultra-lightweight attention module adopts: 1. Pooling strategy selection: Only global average pooling is used, without introducing the max pooling branch. Standard CBAM adds the max pooling branch on the basis of SE to capture different statistical information. However, the gradient of max pooling is only backpropagated to the neuron with the strongest response, and the training stability is not as good as average pooling. Ultra Light Attention only retains the global average pooling branch, which reduces the amount of computation while ensuring training stability. 2. MLP structure compaction: The compression ratio of SE is further increased from 16 to 64, the dimension of the bottleneck layer is reduced from 16 to 4, and the number of parameters is reduced by about 75%. Specifically, the first layer of the fully connected network compresses 256 dimensions to 4 dimensions, and the second layer restores 4 dimensions to 256 dimensions. The significant compression of the bottleneck layer reduces the number of parameters of Ultra Light Attention to about 25% of that of SE, significantly reducing computational overhead while maintaining channel selection capability. 3. Spatial Attention Removal: Ultra Light Attention does not have a separate spatial attention branch. Standard CBAM cascades spatial attention (7×7 convolution) after channel attention, but this is redundant in the SAM3 framework. The SAM3 visual encoder has already established global spatial dependencies through a self-attention mechanism. Adding additional spatial attention not only wastes computation, but its local convolution operations may also destroy the global context already established by self-attention. Removing the spatial attention branch further reduces the computational cost and parameter count of Ultra Light Attention. Ultimately, the complete forward propagation expression of Ultra Light Attention is: Among them, formula The extrusion operation is calculated on the feature map enhanced by the lightweight module. Perform global average pooling to obtain channel descriptors. ,formula The computational stimulus operation increases the reduction ratio of the two-layer fully connected network to 64, according to the formula. Perform a scaling operation to adjust the channel weights. The final output is obtained by multiplying the enhanced features channel by channel. ; Regarding the residual gating module: The core idea of ​​the residual gating module is to treat lightweight enhancement as a correction term for the original features, and to control the correction strength asymptotically from zero using a gating scalar. Therefore, the output of the enhancement layer of the residual gating module is defined as follows: in This refers to the layer index in the feature pyramid. For the first The input features of the layer For the first The output features of the layer This is the composite enhancement function of the lightweight module and the ultra-lightweight channel attention module. For the first The learnable gating scalar parameter of the layer is initialized to 0; About the Transformer decoder: The Transformer decoder employs a two-stage Coarse-to-Fine decoding strategy based on a box-adaptive window, as detailed below: The six decoding layers are divided into two complementary stages. The coarse localization results dynamically guide the focus of the attention range in the fine stage. The first three layers are the coarse localization stage, which uses global cross-attention to perform the initial localization of the target, enabling each query to find the target candidate region in the entire map. The last three layers are the fine stage, which introduces box-adaptive local window attention. Based on the bounding boxes predicted in the coarse localization stage, an adaptive attention window of an adaptive size is dynamically generated for each query, compressing the scope of cross-attention from the entire map to the window region, and realizing the fine segmentation of the target boundary. The two stages are connected by the predicted bounding boxes of the coarse stage, which transforms the localization results of the coarse stage into spatial attention constraints for the fine stage, realizing a gradual transition from full map search to local focus. The core innovation of this design is that the decoder adopts differentiated attention strategies in shallow and deep layers, and the contraction of the attention range is driven by the intermediate prediction results of the model itself, rather than relying on external priors or fixed rules. To comprehensively evaluate the performance of the SAM3-LC unit, it was compared horizontally with current mainstream semantic segmentation models, including Mask2Former (ResNet-50 backbone), SegFormer (MIT-B0 backbone), and SegNeXt (MSCAN-B backbone) based on the Transformer architecture, and DeepLabV3+ (ResNet-50 backbone) and U-Net based on the CNN architecture. All models were trained and tested on the same lobster dataset. The specific experimental data are shown in Table 3-1. Table 1: Segmentation accuracy of different methods on the lobster dataset

[0017] From the overall metrics, the SAM3-LC unit achieved the best results in both mIoU (94.38%) and mDice (97.09%), improving upon the second-best U-Net by 0.95 and 0.50 percentage points, respectively. Analysis of fine-grained metrics for each part revealed a pattern where the performance differences between different models were closely related to the biological characteristics of different parts of the lobster. The SAM3-LC unit achieved a significant lead in the cephalothorax category, with car_IoU reaching 95.59% and car_Dice reaching 97.74%, which are 4.98 and 2.67 percentage points higher than the second-place U-Net, respectively. As the largest part of the lobster, the segmentation accuracy of the cephalothorax is highly dependent on the model's ability to depict the edge contours of large areas. The advantage of SAM3-LC stems from the fact that the effective receptive field corresponding to the adjusted void ratio combination [1, 2, 3, 4] in the lightweight void space pyramid pooling module is [3, 5, 7, 9], which effectively adapts to the medium to large scale feature requirements of the cephalothorax, allowing its edge regions to obtain richer multi-scale contextual information. In contrast, pure Transformer models such as SegNeXt (car_IoU of 90.42%) and SegFormer (car_IoU of 88.23%) performed poorly in this category, indicating that when pixel-level multi-scale enhancement modules are lacking, the Transformer architecture still has insufficient ability to finely segment the edges of large-scale single targets. The tail is the region with the middle segmentation difficulty among the three parts. SAM3-LC's tail_IoU is 94.34%, which is lower than SegNeXt's 95.71%. SegNeXt performs best in this category, indicating that its MSCAN backbone's multi-scale convolution design is outstanding in segmenting medium-scale regions with regular textures like the tail. SAM3-LC performs robustly in the tail, with a tail_Dice of 97.08%, and its overall performance is still among the top. In summary, the SAM3-LC unit achieved the best overall comparative experimental results with 94.38% mIoU and 97.09% mDice, especially establishing a significant leading advantage in the large area of ​​the cephalothorax, verifying the effectiveness of the synergistic work of residual lightweight void space pyramid pooling multi-scale enhancement and Coarse-to-Fine decoding strategy. B. Regarding the SAM3-MT unit; For mixed scenarios involving multiple types of shrimp and crabs, a multi-task joint framework is built based on the SAM3-LC unit. A species classification head is added, and the output of the Transformer decoder is connected to the input of the species classification head. The shared backbone features enable end-to-end synchronous inference of part segmentation and species identification, eliminating serial errors and bidirectionally coordinating to optimize the network feature extraction capabilities. The species classification head is designed following a minimalist principle, and its input is the high-resolution pixel embedding output from the pixel decoder. First, adaptive global average pooling is used to compress the spatial dimension into a single feature vector, eliminating the dimensionality inconsistency caused by differences in input image size. Then, this feature vector is flattened and mapped to the species category space through a fully connected layer. in For pixel embedding, Pool input feature maps of arbitrary size into fixed-size ones. , Flatten the pooled feature vectors into one-dimensional vectors. This is the weight matrix of the fully connected layer. For bias vectors, Number of species categories Given a classification log-odds vector, the number of classification head parameters for this species is only [number missing]. These few parameters are negligible compared to the hundreds of millions of parameters in the entire model, and will not have a significant impact on the model's computational efficiency or memory usage. This is because the pixel decoder outputs... It already contains high-level semantic features that have undergone multi-scale fusion and semantic enhancement, so effective species classification can be achieved without a complex classification head; Species classification employs the standard multi-class cross-entropy loss function, given a class log-odds vector. First, it is converted into a class probability distribution using the Softmax function: in For the first Logit values ​​for each category, To predict whether the input image belongs to the first... The Softmax function ensures that the sum of the predicted probabilities for all species is 1. ; Species classification loss is defined as the cross-entropy between the predicted distribution and the true distribution: in The total number of species categories, For real labels one-hot encoding, For the model to the first The predicted probability of each category is used. The loss function measures the KL divergence between the predicted distribution and the true distribution. The loss is zero when the predicted probability distribution is completely consistent with the one-hot true distribution. The cross-entropy loss has the advantages of convexity and smooth gradient, and is suitable for joint optimization with the focus loss and Dice loss in segmentation tasks. During the validation phase, the model's species classification performance was evaluated using top-1 accuracy for the input image. The predicted species category is the index corresponding to the maximum value in the log-odds vector: in For the first Logit values ​​for each category, The species classification accuracy is the species category predicted by the model. The proportion of samples to the total number of samples; The species taxonomy branch and the segmentation branch share the feature representation of the visual encoder. The gradients of both branches jointly optimize the shared parameters from different perspectives. This embodiment analyzes the working mechanism of joint optimization from the perspective of gradient propagation: make This represents the set of parameters for a shared visual encoder. The parameter representing the branch split. Let represent the parameters of species taxonomic branching. The gradient of the total loss from joint training with respect to the encoder parameters is: in, To backpropagate the gradient of the segmentation loss to the encoder, The gradient of the classification loss is backpropagated to the encoder; the joint effect of the two can be further decomposed, letting... The pixel embedding represented by the encoder output can be obtained using the chain rule: in, The gradient of the segmentation loss on the pixel embedding carries the supervision signal for part-level segmentation (forcing the encoder to pay attention to the boundary, shape and texture features of the part). The gradient of the classification loss on the pixel embedding carries the supervision signal of species-level discrimination (forcing the encoder to pay attention to the discriminative features between species). The directions of the two gradients in the pixel embedding space may not be completely consistent. Therefore, joint optimization can be regarded as finding a trade-off direction in the shared feature space that simultaneously satisfies the requirements of the two tasks. From a geometric point of view, The gradient combination in the algorithm can be understood as follows: the direction of encoder parameter update is determined by the weighted sum of the gradient vectors of the two tasks. The segmentation task provides fine-grained spatial structure information to guide the feature encoder to learn the geometric shape of the parts; the classification task provides coarse-grained category discrimination information to guide the feature encoder to enhance the response of inter-species difference features. The synergistic optimization of the two approaches has a good complementary effect. Part-level segmentation supervision forces the encoder to pay attention to the structured information of the target (such as the morphology of the claws, the outline of the carapace, and the arthropod texture of the tail). This information has potential discriminative value for distinguishing different species. For example, lobster claws are usually slender while crab claws are short and thick. The accurate modeling of claw boundaries in the segmentation task indirectly provides morphological clues for species classification. The gradient signal of species classification requires the encoder to be more sensitive to the differences between species, which helps to better distinguish the same parts of different species during the segmentation process (such as distinguishing the claws of crabs and lobsters). Without classification branches, the encoder may map the same parts of different species to similar feature spaces, leading to cross-species segmentation confusion. After introducing classification gradients, the encoder is forced to increase the distance between the same parts of different species in the feature space, thereby improving the discriminative accuracy of segmentation. This embodiment conducted a systematic comparative experiment on the Crab+Lobster joint dataset. Table 2 reports the comparison of species classification performance under different configurations: Table 2: Experimental Results of Classification Performance Comparison on the Shrimp and Crab Dataset

[0018] Table 2 reports the performance comparison of different classification models on the Crab+Lobster joint dataset. The experiment selected five mainstream general classification models as baselines, including convolutional neural networks (ResNet50, EfficientNet_B3, ConvNeXt_T) and the Vision Transformer series (ViT_B_16, Swin_T). The SAM3-MT of this invention adds a lightweight classification head on the basis of the segmentation decoder and shares encoder features for species classification. As shown in Table 2, the SAM3-MT unit achieved a Top-1 accuracy of 78.3%, significantly outperforming all the comparison models. Compared to the best-performing general model ConvNeXt_T (77.34%), SAM3-MT is 0.96 percentage points higher; compared to ResNet50 (75.53%), it is 2.77 percentage points higher. This advantage indicates that the classification head of the shared segmentation task encoder can utilize the fine-grained regional features learned from the segmentation task, thereby obtaining a stronger species discrimination ability than a pure classification model. It should be noted that this system is implemented based on the PyTorch framework, and the specific environment configuration is as follows: Hardware environment: 8 NVIDIA RTX3090 GPUs, with NVIDIA's distributed communication library backend within the node; Base model: SAM3 ViT-L pre-trained weights; Training epochs: 300 epochs; Batch size: 1 image per GPU, using gradient accumulation; Optimizer: Adaptive Moment Estimation Optimizer with decoupled weight decay (AdamW), weight decay of 0.1; Learning rate strategy: InverseSquareRoot scheduler, 20 epochs warm-up, 20 epochs cool-down; Visual backbone learning rate 2.5 × 10⁻⁶. -5 Transformer learning rate 8×10 -5 Language backbone learning rate 5×10 -6 Visual backbone attenuation coefficient 0.9; Data augmentation: random scaling (480-896), square filling, random bounding box jitter (standard deviation 0.1, maximum 20 pixels); Mixed precision: bf16 automatic mixed precision training; Validation frequency: once every 10 epochs on the test set; Number of species categories is 12, species classification loss weight is set to 1.0; The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for semantic segmentation of multiple parts of crustaceans such as shrimp and crab and joint species identification, characterized in that, Includes the following steps: S1. Visual Feature Extraction: Input the input image into the visual encoder to extract multi-resolution feature maps; S2. Text feature extraction: Input the text prompts corresponding to the three parts of the chelipeds, carapace and tail into the text encoder to obtain the corresponding text feature embeddings; S3. Obtain target feature encoding: Input the multi-resolution feature map and the text feature embedding into the visual-language fusion module, and fuse them through the cross-attention mechanism between visual features and text embedding to generate target feature encoding; S4. Residual lightweight multi-scale dilated convolution enhancement: Input the target feature encoding into the lightweight dilated spatial pyramid pooling module, perform multi-scale dilated convolution enhancement, and obtain enhanced features; S5. Multi-level feature aggregation: The enhanced features are input into the FPN feature pyramid network module for multi-level fusion, and high-resolution pixel features are output. S6. Two-stage adaptive bounding box decoding: The high-resolution pixel features are input into the Transformer decoder, and coarse localization and fine-tuning iterative optimization are performed sequentially to obtain mask features; S7. Segmentation Mask Generation: The mask features are processed by Mask MLP and dot product operation is performed with the pixel embedding to output the pixel-level segmentation mask corresponding to the three parts of the chelicerae, cephalothorax and tail, thus completing semantic segmentation. S8. Shared feature extraction: The mask features obtained in step S6 are reused as shared pixel embeddings; S9. Species classification prediction: The shared pixels are embedded and compressed into a one-dimensional vector through adaptive global average pooling, and input into the species classification head to output the shrimp and crab species classification results, thus completing the joint species identification.

2. The method for semantic segmentation and joint species identification of multiple parts of crustaceans such as shrimp and crabs according to claim 1, characterized in that, In step S4, the lightweight void space pyramid pooling module includes a lightweight module, an ultra-lightweight channel attention module, and a residual gating module connected in series.

3. The method for semantic segmentation and joint species identification of multiple parts of crustaceans such as shrimp and crabs according to claim 2, characterized in that, The lightweight module performs the following operations: Set the branch hole rate set of the lightweight module to The standard empty spatial pyramid pooling global average pooling branch is removed, the number of intermediate channels is compressed, and the branch output channels are concatenated and then subjected to 1×1 convolution to restore the original number of channels. The formula for calculating the number of intermediate channels is as follows: Where C represents the input feature channel; The output format of the branch output channel is represented as follows: in, For the first The output feature map of each dilated convolution branch Values , respectively corresponding to a void ratio The dilated convolution is defined as follows: X is the input feature map, k=3 is the kernel size, d=r is the dilation rate, and BN is batch normalization. This is the ReLU activation function.

4. The method for semantic segmentation and joint species identification of multiple parts of crustaceans such as shrimp and crabs according to claim 2, characterized in that, The ultra-lightweight channel attention module uses global average pooling to compress features, then removes max pooling and spatial attention branches, with a fixed channel compression ratio of 64. Two fully connected layers compress the channels to 4 dimensions and then restore them. The output channel weights are multiplied by the input features channel by channel. Its forward propagation process is expressed by the following formula: in, This is the feature map after multi-branch dilated convolution stitching and projection. This is the channel descriptor after global average pooling. and This is the weight matrix for the fully connected layer, with its bottleneck layer reduction ratio set to 64. It is the ReLU activation function. For the Sigmoid function, For dynamic channel weights, This indicates multiplication by channel.

5. The method for semantic segmentation and joint species identification of multiple parts of crustaceans such as shrimp and crabs according to claim 2, characterized in that, The residual gating module outputs the following formula: in, This refers to the layer index in the feature pyramid. For the first The input features of the layer For the first The output features of the layer This is the composite enhancement function of the lightweight module and the ultra-lightweight channel attention module. For the first Learnable gated scalar parameters of the layer.

6. The method for semantic segmentation and joint species identification of multiple parts of crustaceans such as shrimp and crabs according to claim 2, characterized in that, The multi-resolution feature map output by the visual encoder is fed into the vision-language fusion module, which outputs the highest resolution feature layer. The target feature encoding is then subjected to multi-scale dilated convolution enhancement via the lightweight dilated spatial pyramid pooling module. Specifically, the lightweight module captures multi-scale receptive field information, the ultra-lightweight channel attention module performs channel recalibration, and a residual gating module ensures the enhanced features are not corrupted during the initial training phase. These enhanced features, along with features from the other three layers, are fed into the FPN feature pyramid network module, where they are progressively upsampled and fused to generate high-resolution pixel embeddings. ; The Transformer decoder is a 6-layer decoder, which adopts a layered structure of 3 layers for coarse localization and 3 layers for fine localization.

7. The method for semantic segmentation and joint species identification of multiple parts of crustaceans such as shrimp and crabs according to claim 1, characterized in that, In step S6, the coarse localization stage employs global cross-attention to obtain the initial bounding box of the target; the fine-tuning iterative optimization stage dynamically generates a local attention window based on the predicted initial bounding box of the target, performs cross-attention within the local attention window, and iteratively optimizes to obtain the mask features.

8. The method for semantic segmentation and joint species identification of multiple parts of crustaceans such as shrimp and crabs according to claim 7, characterized in that, The local attention window generation includes the following steps: a. The coarse localization output normalized initial bounding box of the target is... The reference side length of the local attention window in the image pixel space Window relative scaling factor Input image pixel size =560; b. The reference side length The feature map window side length is obtained by cropping to the [32, 256] pixel range and dividing by the patch size of the image segmented by the visual decoder. ; c. Calculate the top, bottom, left, and right boundaries of the window with the center of the frame as the anchor. The attention bias outside the local attention window is assigned a negative infinity value, and the local attention window participates in the attention weighting calculation. The formula is as follows: in, For boundary clipping functions, restrict the input to... Within the range, and These are the set minimum and maximum pixel thresholds, respectively. , , The number of patches for segmenting the image by the visual decoder. =14; d. Establish an attention bias matrix based on the dynamically generated local attention window. , its first Each element is defined as: in, For the first The query pair for the first Attention bias at each spatial location For the first feature map The coordinates of a spatial location, For the first A local window area for each query; e. By resetting the attention weight outside the local attention window to The attention mechanism is equivalent to performing calculations only within the local attention window: in, The attention weights after applying the bias. For the first Target query vector For the first The key vector corresponding to each image location For the dimensions of query and key, Scaling factor For the first The query pair for the first Attentional bias at each spatial location.

9. The method for semantic segmentation and joint species identification of multiple parts of crustaceans such as shrimp and crabs according to claim 1, characterized in that, The species classification head is a single-layer fully connected classification head, which includes a global average pooling layer and a single-layer fully connected layer. The total number of parameters is 256×C+C, where C is the number of species categories. The number of output channels of the FPN feature pyramid network module is 256.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is executed by one or more processors to implement the method as described in any one of claims 1-9.

Images