Crab instance segmentation system and method based on multi-branch feature fusion

Abstract

The invention provides a crab instance segmentation system and method based on multi-branch feature fusion. The method comprises the following steps: acquiring a high-resolution crab image in a unified illumination and fixed environment by using image acquisition; performing fine polygon segmentation marking on crab shells and step feet of the crab images, converting the crab shells and the step feet into a COCO format, and constructing a high-quality data set; a multi-branch feature fusion model is constructed in model training, shallow details, a local structure and high-level semantic information are respectively extracted by introducing a three-branch feature fusion path, and feature enhancement is performed on a weak texture region and a fuzzy boundary in combination with a feature enhancement module, so that the target boundary perception capability and the small target segmentation precision are improved, and the target segmentation accuracy is improved. Training to obtain a crab identification model; and performing instance segmentation on an acquired image by using the crab identification model, and outputting a target contour, a segmentation mask and related feature information. Therefore, high efficiency and cost advantages are achieved while high-precision instance segmentation is guaranteed, and the method has good practicability and popularization value.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of high-precision instance segmentation of biological targets, in particular to a crab instance segmentation system and method based on multi-branch feature fusion. BACKGROUND

[0002] In the scenarios of aquaculture, fishery resource monitoring and ecological environment assessment, accurate detection and segmentation of biological targets such as crabs are important foundations for intelligent management. Through high-precision instance segmentation of crabs, individual contours and morphological features can be extracted without touching the samples, providing data support for breeding monitoring, resource assessment and automation. However, in practical applications, crab images often have complex backgrounds (such as water reflection, silt interference), blurred target boundaries (unclear junctions between chelae and shell), large differences in individual morphology (obvious differences in size and posture between different individuals), and serious overlapping and occlusion between individuals. These factors lead to deficiencies in accuracy, robustness and real-time performance of traditional segmentation methods.

[0003] Currently, for instance segmentation of biological targets, researchers have proposed various deep learning-based visual methods: for example, a two-stage detection and segmentation framework based on Mask R-CNN, which first detects the target and then generates a pixel-level mask to achieve segmentation. This method has certain advantages in accuracy, but its network structure is complex and the computational load is large, making it difficult to meet the real-time processing requirements in crab breeding scenarios. Another type of method represented by YOLACT, SOLOv2, YOLOv8-Seg and other single-stage segmentation networks achieves fast segmentation through an end-to-end structure, but its performance is still insufficient in dealing with small targets (such as the long chelae of crabs), weak texture areas (such as the smooth part of the crab carapace) and boundary details.

[0004] To solve the problem of uneven multi-scale feature information, some methods introduce feature fusion strategies such as FPN, PANet and ASFF, but most existing methods use simple weighting or splicing methods, which cannot fully model the heterogeneity association between multi-scale and cross-level features, leading to insufficient integration of fine-grained features and high-level semantic information, and thus limited effectiveness in segmentation of crab target boundaries and complex limb structures.

[0005] In summary, the above methods have the following defects:

[0006] 1. Insufficient preprocessing of images leads to many environmental interference factors, affecting segmentation accuracy;

[0007] 2. The feature extraction capability of existing methods is limited, making it difficult to balance global semantic information and local fine-grained features, especially in boundary blurring and small target recognition;

[0008] 3. A large amount of labeled training data set is required, which increases the difficulty of data collection and processing, and also affects the stability and generalization ability of the model. SUMMARY

[0009] In view of the above technical deficiencies, the purpose of the present application is to provide a crab instance segmentation system and method based on multi-branch feature fusion, which realizes high-precision instance segmentation of crab individuals in complex background, target boundary blur and small target scene, and can achieve a good balance between real-time performance and accuracy.

[0010] To achieve the above purpose, the present application adopts the following technical solutions:

[0011] A crab instance segmentation system based on multi-branch feature fusion comprises:

[0012] A cylindrical container for holding crabs to be detected;

[0013] An image acquisition module comprising a fixed support, a high-definition camera and an optical adjustment device; the high-definition camera is installed on the fixed support and is used for image acquisition of the target to be detected in the cylindrical container; the optical adjustment device is installed on the high-definition camera;

[0014] A data set generation module, which performs fine polygon segmentation labeling on key anatomical parts of the crab images collected by the image acquisition module through a computer, and converts the labeling results into COCO format to construct a crab data set;

[0015] A training module, which constructs a multi-branch feature fusion model, trains and adjusts the multi-branch feature fusion model using images in the data set, and obtains a crab recognition model;

[0016] A recognition module, which performs instance-level segmentation on the collected images using the multi-branch feature fusion model, generates mask segmentation results of the crabs, and outputs the contour information and category information of each target.

[0017] A crab instance segmentation method based on multi-branch feature fusion comprises the following steps:

[0018] 1) Obtain biological target images through a high-definition camera and store them in JPG format;

[0019] 2) Label the collected photos using a computer, and make a crab data set in the same format as the public COCO data set, then randomly divide the data set into a training set, a validation set and a test set;

[0020] 3) Train the multi-branch feature fusion model using the training set and the validation set, and obtain a trained crab recognition model;

[0021] 4) input the test set data into the crab recognition model for inference to generate the segmentation mask and category prediction result of the target.

[0022] 3. The crab instance segmentation method based on multi-branch feature fusion according to claim 2, wherein in step 2), the original data set is divided into a training set, a validation set and a test set according to a random sampling method, and the proportions are 80%, 10% and 10% respectively.

[0023] Preferably, in step 3), the multi-branch feature fusion model comprises feature extraction, feature fusion and a detection head; the crab images in the training set and the validation set are input into the model, multi-scale feature extraction is performed through the feature extraction layer to obtain multi-level features under different receptive fields;

[0024] The extracted features are input into the feature fusion structure, which takes high-level semantic features as the starting point, introduces middle and low-level features along the top-down path step by step, outputs through splicing the features of the previous layer and the current layer and the feature enhancement module, realizes the deep fusion of different scales and semantic levels, and performs size adaptive processing on the multi-scale feature maps, splices the feature maps of different scales in the channel dimension after adjusting them to the same spatial size, realizes feature fusion under spatial consistency, and improves the richness of semantic features and the structure perception ability;

[0025] The feature map is input into the segmentation head, each pixel point is processed through convolution operation, and is respectively sent into the classification prediction branch and the mask prediction branch to realize parallel prediction of the target category and the pixel-level segmentation mask; in the training process, a joint loss function is used for end-to-end supervision learning and parameter optimization of the model, so as to obtain the trained crab recognition model.

[0026] Preferably, the feature enhancement module is used to improve the representation ability of the model for fine-grained targets; the input features are first down-sampled to extract deep semantic features with a larger receptive field; then they enter the residual processing unit, which realizes feature enhancement through the following steps: first, 1x1 convolution is used to compress the channel dimension to reduce the computational complexity and pass through ReLU activation; then, 3x3 convolution is used to extract local spatial features to enhance the expression ability, and then ReLU activation is performed; finally, 1x1 convolution is used to restore the channel number to complete feature reconstruction; the residual unit adds the input features and the processing results through a jump connection to realize the fusion of shallow structure information and deep features, and outputs through ReLU activation; the features are then up-sampled to restore to the original spatial size, and the channel dimension is adjusted through 1x1 convolution to output the enhanced feature map; this module can significantly improve the representation ability of target boundaries and detailed features, thereby improving the instance segmentation accuracy and the robustness of the model.

[0027] Preferably, the classification prediction branch is used to output the class probability of the biological target, and the prediction result is usually optimized in the training process by using a cross-entropy loss function to improve the accuracy of class discrimination, and the calculation formula is as follows:

[0028] (1)

[0029] wherein C is the total number of classes, is the one-hot encoding of the true class, represents the probability predicted by the model that belongs to class c.

[0030] Preferably, the mask prediction branch is used to generate the instance segmentation mask of the target; in the training process, the mask prediction is optimized by using a binary cross-entropy loss function to improve the ability of the model to distinguish foreground and background pixels, and the calculation formula is as follows:

[0031] (2)

[0032] wherein represents whether the pixel (i, j) belongs to the target region, is the mask probability value predicted by the model.

[0033] Preferably, the loss function is designed to enhance the perception ability of the model to the target boundary and improve the regression accuracy of the complex structure region in the crab image; based on the Wise-IoUv3 loss function, through a dynamic adjustment mechanism and an error weighting strategy, the matching quality between the predicted box and the real box can be more effectively measured, thereby significantly improving the stability and positioning accuracy of the boundary box regression; specifically, Wise-IoUv1 introduces a distance attention mechanism based on the traditional IoU, and constructs a weighted loss function with double-layer attention regulation ability, and the calculation process is shown in formula (3) and formula (4);

[0034] (3)

[0035] (4)

[0036] wherein , is the size of the smallest enclosing box; in order to prevent from generating gradient that hinders convergence, , is separated from the calculation graph;

[0037] On this basis, the Wise-IoUv3 further proposes a dynamic non-monotonic focusing mechanism; the method realizes the intelligent regulation and control strategy of gradient gain by constructing a non-monotonic modulation function β, enhances the attention to the medium quality prediction box, and suppresses the adverse effects of extremely high or extremely low quality prediction on training, thereby improving the fitting ability and robustness of the model to the real boundary; the specific calculation is as shown in formula (4), formula (5) and formula (6);

[0038] (4)

[0039] (5)

[0040] (6)

[0041] Wherein, the outlying degree and the mapping of gradient gain , controlled by hyperparameters , .

[0042] Preferably, in step 4), the test set image is input into the trained crab recognition model, and the feature extraction module, the feature fusion module and the segmentation head are sequentially passed through by forward propagation to process the features of each pixel point; then, the output is generated in the classification prediction branch and the mask prediction branch respectively; the output includes the class prediction result of each target, the pixel-level instance segmentation mask and the target contour information; by comparing with the pre-set confidence threshold, the effective target can be screened and its class label can be determined; the inference process can realize accurate instance segmentation of crab shells and claws, and provide basis for subsequent target morphological analysis, area calculation and biological research; at the same time, the identification and segmentation performance of the model on the test set can be quantitatively verified by evaluation indexes, so as to ensure the robustness and reliability of the model; the method can improve the identification ability of complex boundaries and small targets while ensuring the segmentation accuracy.

[0043] Compared with the prior art, the beneficial effects of the present application are as follows:

[0044] 1. By constructing a multi-branch feature fusion model, the effective integration of low-level fine-grained features and high-level semantic features is realized, so as to improve the boundary perception ability of crab targets;

[0045] 2. Under the premise of ensuring real-time, the segmentation accuracy of crabs in small targets and complex backgrounds is improved, and through multi-scale feature fusion, the robustness to targets of different sizes is enhanced;

[0046] 3. Although the method takes crabs as an example, the network architecture design concept can be easily migrated to other instance segmentation tasks of agricultural products, industrial parts, biological cells and the like with irregular shapes.BRIEF DESCRIPTION OF DRAWINGS BRIEF DESCRIPTION OF DRAWINGS

[0047] Fig. 1 is a structural schematic diagram of the system of the present application;

[0048] Fig. 2 is a flow chart of the method of the present application;

[0049] Fig. 3 is a structural block diagram of the system of the present application.

[0050] wherein:

[0051] 1, high-definition camera; 2, optical adjustment device; 3, cylindrical container; 4, computer; 5, fixed support. DETAILED DESCRIPTION

[0052] The present application will be further described below in conjunction with the drawings.

[0053] As Figs. 1 to 3 shown, a crab instance segmentation system based on multi-branch feature fusion includes:

[0054] An image acquisition module: a high-definition camera 1 is installed through a fixed support 5 to acquire images of a target to be detected; an optical adjustment device 2 is arranged in front of the camera, and cooperates with a cylindrical container 3 to ensure that the shooting is completed under uniform illumination and fixed environment; the acquired images are transmitted to a computer 4 for subsequent processing.

[0055] A data set generation module: the computer 4 is used to finely polygonally segment and label key anatomical parts (such as crab shells and walking legs) of the acquired crab images through a tool Labelme, and converts the labeling results into COCO format to construct a crab data set.

[0056] A training module: a multi-branch feature fusion model is constructed, and the multi-branch feature fusion model is trained and adjusted by using images in the data set to obtain a crab recognition model.

[0057] An identification module: the multi-branch feature fusion model is used to perform instance-level segmentation on the acquired images to generate mask segmentation results of crabs, and output contour information and category information of each target.

[0058] A instance segmentation method based on multi-branch feature fusion, the specific steps are:

[0059] 1) Obtain biological target images through a high-definition camera 1 and store them in JPG format;

[0060] 2) The collected photos are labeled by computer 4, and a crab dataset in the same format as the public COCO dataset is produced, and then the dataset is randomly divided into training set, validation set and test set; the specific steps are:

[0061] 2.1) The original dataset is divided into training set, validation set and test set according to the random sampling method, and the proportions are 80%, 10% and 10% respectively.

[0062] 3) The multi-branch feature fusion model is trained using the training set and the validation set, and a trained crab recognition model is obtained. The specific steps are:

[0063] 3.1) A multi-branch feature fusion model is constructed, including feature extraction, feature fusion and detection head, and the crab images in the training set and the validation set are input into the model, and multi-scale feature extraction is performed through the feature extraction layer to obtain multi-level features under different receptive fields.

[0064] 3.2) The extracted features are input into the feature fusion structure, which takes high-level semantic features as the starting point, introduces middle and low-level features along the top-down path step by step, and outputs through the splicing of the last layer of features and the current layer of features and the feature enhancement module, realizing the deep fusion of different scales and semantic levels; the multi-scale feature maps are subjected to size adaptive processing, and the feature maps of different scales are adjusted to the same spatial size and then spliced in the channel dimension, realizing feature fusion under spatial consistency, thereby improving the richness of semantic features and the structure perception ability;

[0065] 3.2.1) The feature enhancement module is used to improve the representation ability of the model for fine-grained targets. The input features are first down-sampled to extract deep semantic features with a larger receptive field; then they enter the residual processing unit, which realizes feature enhancement through the following steps: first, use 1x1 convolution to compress the channel dimension to reduce computational complexity and pass through ReLU activation; then use 3x3 convolution to extract local spatial features to enhance the expression ability, and then pass through ReLU activation; finally, through 1x1 convolution, the channel number is restored to complete feature reconstruction. The residual unit adds the input features and the processing results through a jump connection to realize the fusion of shallow structure information and deep features, and outputs through ReLU activation. The feature is then up-sampled to restore to the original spatial size, and the channel dimension is adjusted through 1x1 convolution, outputting the enhanced feature map. This module can significantly improve the representation ability of target boundaries and detailed features, thereby improving the instance segmentation accuracy and the robustness of the model.

[0066] 3.3) The feature map input segmentation head processes each pixel point through convolution operation and sends it into the classification prediction branch and the mask prediction branch respectively to realize parallel prediction of target class and pixel-level segmentation mask. In the training process, a joint loss function is used for end-to-end supervision learning and parameter optimization of the model, so as to obtain a trained crab recognition model.

[0067] 3.3.1) Classification prediction branch: This branch is used to output the class probability of biological targets. In the training process, the cross-entropy loss function is usually used to optimize the prediction results to improve the accuracy of class discrimination. The calculation formula is as follows:

[0068] (1)

[0069] where C is the total number of classes, is the one-hot encoding of the true class, is the probability of the model predicting that the target belongs to class c.

[0070] 3.3.2) Mask prediction branch: This branch is used to generate the instance segmentation mask of the target. In the training process, the binary cross-entropy loss function is used to optimize the mask prediction, thereby improving the model's ability to distinguish foreground and background pixels. The calculation formula is as follows:

[0071] (2)

[0072] where is a binary variable indicating whether pixel (i, j) belongs to the target region, is the mask probability value predicted by the model.

[0073] 3.3.3) Loss function design: To enhance the model's perception of target boundaries and improve the regression accuracy of complex structure regions in crab images, we introduce the Wise-IoU v3 loss function. This loss function can more effectively measure the matching quality between the predicted box and the real box through dynamic adjustment mechanism and error weighting strategy, thereby significantly improving the stability and positioning accuracy of the boundary box regression. Specifically, Wise-IoU v1 introduces a distance attention mechanism based on traditional IoU and constructs a weighted loss function with double-layer attention regulation ability. The calculation process is shown in equations (3) and (4).

[0074] (3)

[0075] (4)

[0076] where , is the size of the smallest closed box. To prevent from producing gradients that hinder convergence, , is separated from the computation graph (the superscript * indicates this operation).

[0077] On this basis, the Wise-IoU v3 further proposes a dynamic non-monotonic focusing mechanism. This method realizes the intelligent regulation and control strategy of gradient gain by constructing a non-monotonic modulation function β, enhances the attention to the medium quality prediction box, and suppresses the adverse effects of extremely high or extremely low quality predictions on training, thereby improving the model's fitting ability and robustness to real boundaries. The specific calculation is shown in equations (5), (6) and (7).

[0078] (5)

[0079] (6)

[0080] (7)

[0081] where the outlying degree and the mapping of the gradient gain are controlled by hyperparameters , .

[0082] 4) Input the test set data into the crab recognition model for inference to generate the segmentation mask and class prediction results of the target. The specific steps are as follows:

[0083] 4.1) Input the test set image into the trained crab recognition model, and pass through the feature extraction module, feature fusion module and segmentation head in turn through forward propagation to process the features of each pixel point. Then, generate outputs in the classification prediction branch and the mask prediction branch. The outputs include the class prediction results of each target, pixel-level instance segmentation mask and target contour information. By comparing with the pre-set confidence threshold, valid targets can be selected and their class labels can be determined. The inference process can realize accurate instance segmentation of crab shells and claws, providing a basis for subsequent target morphological analysis, area calculation and biological research. At the same time, the recognition and segmentation performance of the model on the test set can be quantitatively verified by evaluation indicators such as average precision mAP, intersection over union IoU, etc., to ensure the robustness and reliability of the model. This method can improve the recognition ability of complex boundaries and small targets while ensuring the segmentation accuracy.

Claims

1. A crab instance segmentation system based on multi-branch feature fusion, characterized in that, The application relates to a crab recognition method and device. The application relates to a crab recognition method and device. The application relates to a crab recognition method and device. The application relates to a crab recognition method and device. The application relates to a crab recognition method and device. The application relates to a crab recognition method and device. In step 2), the original data set is divided into a training set, a verification set and a test set according to a random sampling method, and the proportions are 80%, 10% and 10% respectively.

2. A crab instance segmentation method based on multi-branch feature fusion, characterized in that, In step 3), the multi-branch feature fusion model comprises feature extraction, feature fusion and a detection head; the crab images of the training set and the verification set are input into the model, multi-scale feature extraction is carried out through a feature extraction layer, and multi-level features under different receptive fields are obtained; The extracted features are input into a feature fusion structure, the module takes high-level semantic features as a starting point, introduces middle and low-level features along a top-down path step by step, outputs through a feature enhancement module by splicing a previous layer feature and a current layer feature, realizes deep fusion of different scales and semantic levels, and carries out size adaptive processing on the multi-scale feature maps, splices the feature maps of different scales in a channel dimension after adjusting the feature maps to the same spatial size, realizes feature fusion under spatial consistency, and improves semantic feature richness and structure perception ability; The feature map is input into a segmentation head, each pixel point is subjected to feature processing through convolution operation, and is respectively sent into a classification prediction branch and a mask prediction branch, parallel prediction of target categories and pixel-level segmentation masks is realized; in the training process, a joint loss function is adopted to carry out end-to-end supervision learning and parameter optimization on the model, so that the trained crab recognition model is obtained. The feature enhancement module is used for improving the representation ability of the model to fine-grained targets; input features are first subjected to down-sampling to extract deep semantic features of a larger receptive field; ​ 3. The crab instance segmentation method based on multi-branch feature fusion according to claim 2, characterized in that, ​ 4. The crab instance segmentation method based on multi-branch feature fusion according to claim 2, characterized in that, ​ ​ ​ 5. The crab instance segmentation method based on multi-branch feature fusion according to claim 4, characterized in that, ​ The residual processing unit is then entered, and feature enhancement is achieved through the following steps: first, a 1x1 convolution is used to compress the channel dimension to reduce computational complexity and activate it through ReLU; then, a 3x3 convolution is used to extract local spatial features to enhance expression ability, and activate it through ReLU; finally, a 1x1 convolution is used to restore the channel number to complete feature reconstruction; the residual unit adds the input features and the processing results through a jump connection to realize the fusion of shallow structure information and deep features, and outputs through ReLU activation; The feature is then restored to the original spatial size through upsampling and adjusted through a 1x1 convolution to adjust the channel dimension, outputting an enhanced feature map; this module can significantly improve the representation ability of target boundary and detail features, thereby improving instance segmentation accuracy and model robustness.

6. The crab instance segmentation method based on multi-branch feature fusion according to claim 5, wherein, The classification prediction branch is used to output the class probability of the biological target, and the cross-entropy loss function is usually used to optimize the prediction results during the training process to improve the accuracy of class discrimination, and its calculation formula is as follows: （1） where C is the total number of classes, ∈ {0, 1} represents one-hot encoding of the true class, represents the probability that the model predicts belongs to class c.

7. The crab instance segmentation method based on multi-branch feature fusion according to claim 6, characterized in that, The mask prediction branch is used to generate the instance segmentation mask of the target; during the training process, the binary cross-entropy loss function is used to optimize the mask prediction to improve the model's ability to distinguish foreground and background pixels, and its calculation formula is as follows: （2） wherein, ∈ {0,1} indicates whether pixel (i,j) belongs to the target region, ∈ [0,1] is the mask probability value predicted by the model.

8. The crab instance segmentation method based on multi-branch feature fusion according to claim 7, characterized in that, The loss function design is used to enhance the model's perception of the target boundary and improve the regression accuracy of complex structure regions in crab images; Based on the Wise-IoUv3 loss function, through a dynamic adjustment mechanism and an error weighting strategy, the matching quality between the predicted box and the real box can be more effectively measured, thereby significantly improving the stability and positioning accuracy of the bounding box regression; Specifically, Wise-IoUv1 introduces a distance attention mechanism based on traditional IoU, constructing a weighted loss function with double-layer attention regulation ability, and its calculation process is shown in equations (3) and (4); (3) (4) wherein, , is the size of the smallest enclosing box; to prevent from producing gradients that impede convergence, , is separated from the computation graph; On this basis, Wise-IoUv3 further proposes a dynamic non-monotonic focusing mechanism; this method realizes intelligent regulation strategy of gradient gain by constructing a non-monotonic modulation function β, enhances attention to medium quality prediction boxes, and suppresses the adverse effects of extremely high or extremely low quality predictions on training, thereby improving the model's fitting ability and robustness to real boundaries; Its specific calculation is shown in equations (4), (5) and (6); (4) (5) (6) where the outlyingness and the gradient gain are mapped by hyperparameters , .

9. The crab instance segmentation method based on multi-branch feature fusion according to claim 2, characterized in that, In step 4), the test set image is input into the trained crab recognition model, and the features of each pixel point are processed through forward propagation through the feature extraction module, the feature fusion module and the segmentation head in turn; then, the outputs are generated in the classification prediction branch and the mask prediction branch respectively; the outputs include the class prediction results of each target, the pixel-level instance segmentation mask and the target contour information; By comparing with the pre-set confidence threshold, the effective target can be screened and its category label is determined; the inference process can realize accurate instance segmentation of crab shell and claw, and provide basis for subsequent target morphological analysis, area calculation and biological research; at the same time, the identification and segmentation performance of the model on the test set can be quantitatively verified through the evaluation index, so as to ensure the robustness and reliability of the model; the method can improve the identification ability of complex boundary and small target while ensuring the segmentation accuracy.

Images