Marine organism segmentation method for high-resolution underwater optical image

Abstract

The invention provides a marine organism segmentation method for high-resolution underwater optical images. The marine organism segmentation method comprises the steps of S1, collecting a plurality of high-resolution underwater optical images and marine organism labels and dividing the high-resolution underwater optical images and the marine organism labels into a training set and a test set; s2, extracting a high-resolution underwater optical image from the training set, inputting the high-resolution underwater optical image into a marine organism segmentation network, and performing spatial feature extraction, semantic feature extraction, feature enhancement, cross-modal cross attention fusion, embedding fusion and image segmentation to obtain marine organism segmentation images of different dimensions; s3, repeating the step S2, calculating the total network loss, and carrying out model optimization to obtain a marine organism segmentation network model; and S4, inputting the test set into the marine organism segmentation network model to obtain a marine organism segmentation image of the highest dimension corresponding to each high-resolution underwater optical image, and taking the marine organism segmentation image as a marine organism segmentation result. The method has the beneficial effects that marine organism segmentation can be carried out on the high-resolution underwater optical image, and the performance and the efficiency are considered.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of marine organism image segmentation, in particular to a marine organism segmentation method for high-resolution underwater optical images. BACKGROUND

[0002] The ocean covers 70% of the world's area and accounts for 80% of the Earth's biodiversity. Marine organisms play an indispensable role in the Earth's ecosystem and are crucial to maintaining the balance of the marine ecosystem. Marine organism segmentation technology can improve the understanding and monitoring of marine biodiversity. Accurate segmentation and automatic identification of marine organisms from underwater images can more accurately identify and classify different marine species, thereby better understanding their distribution and quantity changes, which is of great significance for the protection and management of marine biological resources.

[0003] Depending on the underwater image acquisition device, the marine organism segmentation task can be implemented based on sonar images and based on optical images. Due to the limitations of sonar images, such as low signal-to-noise ratio, insufficient resolution, and difficulty in obtaining texture details of the segmented target, the performance of marine organism segmentation technology based on sonar images is poor. In contrast, underwater optical images have the characteristics of non-invasiveness, more intuitive detection, higher image resolution, and more detailed information, which can better meet the needs of high-precision segmentation of marine organisms. With the rapid development of underwater optical imaging technology, high-resolution underwater optical images can provide unprecedented data for marine organism fine segmentation, and their extremely high spatial resolution can clearly present the microscopic texture, edge morphology, and complex structure of biological tissues (such as the pore distribution of corals or the scale features of fish), greatly improving the accuracy of marine organism morphological analysis and species identification. However, this high-information-density image data also brings three core challenges to the design of lightweight deep segmentation networks: a surge in computational load, an amplification of underwater degradation interference, and a dilemma between detail and efficiency. Therefore, there is an urgent need for a marine organism segmentation method that is oriented towards high-resolution underwater optical images and can balance performance and efficiency. SUMMARY

[0004] The technical problem to be solved by the present application is how to implement marine organism segmentation on high-resolution underwater optical images while balancing performance and efficiency. To overcome the defects of the above prior art (or related art), the present application provides a marine organism segmentation method for high-resolution underwater optical images.

[0005] The present application provides a marine organism segmentation method for high-resolution underwater optical images, comprising the following steps: Step S1, collect multiple high-resolution underwater optical images under different scenes and marine organism labels corresponding to each of the high-resolution underwater optical images, and divide them into a training set and a test set; Step S2, randomly extracting one high-resolution underwater optical image in the training set, inputting the high-resolution underwater optical image into the marine organism segmentation network constructed in advance to perform spatial feature extraction, semantic feature extraction, feature enhancement, cross-modal cross-attention fusion, embedding fusion and image segmentation to obtain a plurality of marine organism segmentation images of different dimensions; Step S3, repeating the step S2 and calculating a network total loss according to each marine organism segmentation image and the marine organism label corresponding to each marine organism segmentation image, and performing model optimization on the marine organism segmentation network according to the network total loss to obtain a marine organism segmentation network model; Step S4, inputting the test set into the marine organism segmentation network model to obtain the marine organism segmentation image of the highest dimension corresponding to each high-resolution underwater optical image in the test set as the marine organism segmentation result.

[0006] Compared with the prior art, the marine organism segmentation method for high-resolution underwater optical images has the following advantages: In the present application, high-resolution underwater image acquisition is performed through step S1, marine organism segmentation network training is performed through step S2, marine organism segmentation network optimization is performed through step S3, and marine organism segmentation network model application is performed through step S4, thereby realizing efficient and accurate segmentation of marine organisms in high-resolution underwater optical images. Through the design of a deep learning network structure and the cooperative processing of multiple modules, the calculation efficiency is improved while maintaining high accuracy.

[0007] In one possible implementation, in the step S1, each high-resolution underwater optical image and each marine organism label is preprocessed, wherein the preprocessing process of each high-resolution underwater optical image includes: For each high-resolution underwater optical image, the size of the high-resolution underwater optical image is scaled to , and then the scaled high-resolution underwater optical image is normalized, all pixel values in the R channel are normalized to have a mean of 0.485 and a variance of 0.229, all pixel values in the G channel are normalized to have a mean of 0.456 and a variance of 0.224, and all pixel values in the B channel are normalized to have a mean of 0.406 and a variance of 0.225; The preprocessing process of each marine organism label includes: For each marine organism label, the size of the marine organism label is scaled to .

[0008] Compared with the prior art, the above technical scheme can unify the input data format through scaling and normalization processing of the high-resolution underwater optical image, facilitating efficient processing of the marine organism segmentation network; only scaling is performed on the marine organism label without normalization processing, the integrity of the original information is preserved, deviation is avoided, and the accuracy of the training target is ensured.

[0009] In a possible implementation, the marine organism segmentation network in the step S2 comprises a backbone module, a feature enhancement module, a cross-modal cross-attention module, an embedding fusion module and an output processing module, the backbone module is connected with the feature enhancement module and the embedding fusion module respectively, the cross-modal cross-attention module is connected with the backbone module and the embedding fusion module respectively, the embedding fusion module is connected with the feature enhancement module and the output processing module respectively, the high-resolution underwater optical image is received by the backbone module to perform spatial feature extraction and semantic feature extraction respectively to obtain feature maps , feature maps , feature maps , feature maps , feature maps , feature maps , feature maps , feature maps , the feature maps , the feature maps , the feature maps , the feature maps are enhanced by the feature enhancement module to obtain feature maps , feature maps , feature maps , feature maps , cross-attention fusion of the feature maps and the feature maps feature maps is performed by the cross-modal cross-attention module to obtain a semantic consensus matrix , the feature maps , the feature maps , the feature maps , the feature maps , the semantic consensus matrix , the feature maps , the feature maps , the feature maps , the feature maps are processed by the embedding fusion module to obtain feature maps , feature maps , feature maps , feature maps The feature map is received through the output processing module. The feature map The feature map The feature map The semantic consensus matrix The first segmented image is obtained by performing target segmentation. Second segmented image Third segmented image Fourth segmented image and the fifth segmented image As a segmentation image of multiple marine organisms.

[0010] Compared with existing technologies, the above technical solution can achieve end-to-end optimization from feature extraction to segmentation output through the collaborative processing of the backbone module, feature enhancement module, cross-modal cross-attention module, embedding fusion module and output processing module. By extracting spatial features and semantic features through dual paths, combined with feature enhancement and cross-modal cross-attention fusion, the feature representation capability is effectively improved.

[0011] In one possible implementation, the backbone module includes a spatial feature extraction module, a semantic feature extraction module, a first convolutional block, a second convolutional block, a third convolutional block, a fourth convolutional block, a fifth convolutional block, a sixth convolutional block, a seventh convolutional block, and an eighth convolutional block. The spatial feature map module is connected to the first, second, third, and fourth convolutional blocks, respectively. The semantic feature extraction module is connected to the fifth, sixth, seventh, and eighth convolutional blocks, respectively. The spatial feature extraction module receives the high-resolution underwater optical image and performs feature extraction to obtain a feature map. Feature map Feature map Feature map The feature map is obtained by performing convolution processing through the first convolutional block, the second convolutional block, the third convolutional block, and the fourth convolutional block. The feature map The feature map and the feature map The semantic feature extraction module receives the downsampled high-resolution underwater optical image and performs feature extraction to obtain a feature map. Feature map Feature map Feature map The feature map is obtained by performing convolution processing through the 5th, 6th, 7th, and 8th convolutional blocks. The feature map The feature map and the feature map .

[0012] In one possible implementation, the spatial feature extraction module includes a ResNet-18 backbone network with four layers connected in sequence, the first layer being connected to the first convolutional block, the second layer being connected to the second convolutional block, the third layer being connected to the third convolutional block, and the fourth layer being connected to the fourth convolutional block.

[0013] In one possible implementation, the semantic feature extraction module includes a 5-layer Swin-BTransformer backbone network connected in sequence, with the 2nd layer connected to the 5th convolutional block, the 3rd layer connected to the 6th convolutional block, the 4th layer connected to the 7th convolutional block, and the 5th layer connected to the 8th convolutional block.

[0014] Compared with existing technologies, the above-mentioned technical solution can utilize the local feature extraction capability of the ResNet-18 backbone network and the global semantic modeling capability of the Swin-B Transformer backbone network to extract multi-level spatial features, preserve image details, and model features at different scales, thereby improving the understanding of complex biological structures by marine biological segmentation networks.

[0015] In one possible implementation, the feature enhancement module includes four structurally identical multi-spatial feature enhancement blocks. The input of each multi-spatial feature enhancement block is connected to the first convolutional block, the second convolutional block, the third convolutional block, and the fourth convolutional block, respectively. The output of each multi-spatial feature enhancement block is connected to the embedding fusion module.

[0016] Compared with existing technologies, the above-mentioned technical solution can expand the receptive field by using dilated convolution and global pooling operations, enhance the marine biological segmentation network's ability to perceive multi-scale context, and improve the discriminativeness of features.

[0017] In one possible implementation, the embedded fusion module includes four structurally identical multi-scale information processing blocks. The input of each multi-scale information processing block is connected to the fifth, sixth, seventh, and eighth convolutional blocks, respectively, and the output of each multi-scale information processing block is connected to the output processing module.

[0018] Compared with existing technologies, the above technical solution can effectively integrate spatial and semantic features, and combine local details and global context to achieve adaptive integration of multi-scale features.

[0019] In one possible implementation, the cross-modal cross-attention module includes two branches and a 14th convolutional block connecting the two branches, wherein the first branch receives the feature map. Perform a flattening operation to obtain a tensor The tensor is processed through the first linear layer and the second linear layer respectively. Perform a linear mapping operation to obtain the query vector. Key vector Sum value vector For the query vector and the key vector Attention feature maps are obtained by performing matrix multiplication and transformation operations. For the value vector and the attention feature map Performing matrix multiplication yields semantic attention feature maps. ; The second branch is for receiving the feature map. Perform a flattening operation to obtain a tensor The tensor is processed through the third and fourth linear layers respectively. Perform a linear mapping operation to obtain the query vector. Key vector Sum value vector For the query vector and the key vector Attention feature maps are obtained by performing matrix multiplication and transformation operations. For the value vector and the attention feature map Performing matrix multiplication yields semantic attention feature maps. ; The semantic attention feature map is processed by the 14th convolutional block. and the semantic attention feature map The semantic consensus matrix is ​​obtained by performing concatenation and convolution operations. .

[0020] Compared with existing technologies, the above technical solution can process spatial features and semantic features through two branches respectively, and realize feature interaction by using cross-attention mechanism, thereby enhancing the consistency of feature expression.

[0021] In one possible implementation, the output processing module includes multiple convolutional blocks with identical structures, and the input of each convolutional block is connected to each of the multi-scale information processing modules and the cross-modal cross-attention module.

[0022] Compared with existing technologies, the above-mentioned technical solution can improve the segmentation consistency at different scales, and is particularly suitable for fine target segmentation in high-resolution images. Attached Figure Description

[0023] Figure 1 This is a flowchart of the steps of the present invention; Figure 2 This is a schematic diagram of the framework of the marine organism segmentation network of the present invention; Figure 3 This is a framework diagram of the cross-modal cross-attention module of the present invention; Figure 4 This is a framework diagram of the multi-spatial feature enhancement block of the present invention; Figure 5 This is a framework diagram of the multi-scale information processing block of the present invention; Figure 6 This is a comparison chart of the detection performance of the method of the present invention and existing target segmentation methods on the MAS3K test set. Detailed Implementation

[0024] First, those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0026] See Figure 1 This invention discloses a method for marine organism segmentation based on high-resolution underwater optical images, comprising: Step S1: Collect multiple high-resolution underwater optical images under different scenarios and the corresponding marine biological tags for each high-resolution underwater optical image, and divide them into training set and test set; Step S2: Randomly select a high-resolution underwater optical image from the training set, and input the high-resolution underwater optical image into the pre-constructed marine organism segmentation network to perform spatial feature extraction, semantic feature extraction, feature enhancement, cross-modal cross-attention fusion, embedding fusion and image segmentation to obtain multiple marine organism segmentation images of different dimensions; Step S3: Repeat step S2 and calculate the total network loss based on each marine organism segmentation image and the corresponding marine organism label. Optimize the marine organism segmentation network model based on the total network loss to obtain the marine organism segmentation network model. Step S4: Input the test set into the marine organism segmentation network model to obtain the highest dimension marine organism segmentation image corresponding to each high-resolution underwater optical image in the test set as the marine organism segmentation result.

[0027] In this embodiment of the invention, a dataset is selected or constructed, containing several high-resolution underwater optical images and corresponding marine organism tags for each image. Each high-resolution underwater optical image contains high-resolution targets with complex morphological structures. Then, each high-resolution underwater optical image in the dataset undergoes preprocessing, including scaling and normalization operations, to achieve a preprocessed image size of [size missing]. Furthermore, each marine life label in the dataset undergoes a preprocessing step involving only scaling, resulting in a preprocessed marine life label with a size of [size missing]. Then, all preprocessed high-resolution underwater optical images and their corresponding preprocessed marine organism tags are divided into training and testing sets. The image distribution in the training set is consistent with that in the testing set. For example, in the training set, the proportion of images with target 1 is 0.1, the proportion of images with target 2 is 0.9, the proportion of images with large target 1 is 0.1, and the proportion of images with small target 1 is 0.9. The same proportion applies to the testing set. In this embodiment... , .

[0028] In this embodiment of the invention, step S1, the preprocessing of a high-resolution underwater optical image, involves: firstly, scaling the size of the high-resolution underwater optical image using existing technology. Secondly, the scaled high-resolution underwater optical image is normalized. The pixel values ​​of all pixels in the R channel are normalized to a mean of 0.485 and a variance of 0.229; the pixel values ​​of all pixels in the G channel are normalized to a mean of 0.456 and a variance of 0.224; and the pixel values ​​of all pixels in the B channel are normalized to a mean of 0.406 and a variance of 0.225. In step S1, the preprocessing of a marine organism tag involves scaling the size of the marine organism tag using existing technology. Since the input to deep neural networks usually has a fixed form, high-resolution underwater optical images need to be preprocessed. Here, the normalization process is only performed on the high-resolution underwater optical images, while the marine organism tags corresponding to the high-resolution underwater optical images are not normalized.

[0029] In this embodiment of the invention, step S2 utilizes a deep learning framework to construct a deep neural network as a marine organism segmentation network, such as... Figure 2 As shown, the marine organism segmentation network includes a backbone module, a feature enhancement module, a cross-modal cross-attention module, an embedding fusion module, and an output processing module.

[0030] In this embodiment of the invention, the backbone module mainly consists of a spatial feature extraction module for extracting spatial features from RGB images, a semantic feature extraction module for extracting semantic features from RGB images, a first convolutional block, a second convolutional block, a third convolutional block, a fourth convolutional block, a fifth convolutional block, a sixth convolutional block, a seventh convolutional block, and an eighth convolutional block. The spatial feature extraction module consists of a four-layer ResNet-18 backbone network. The four layers of the ResNet-18 backbone network are connected sequentially. The input of the first layer of the ResNet-18 backbone network serves as the input of the spatial feature extraction module, receiving an image of size [missing information]. The RGB image is used as the output of the 4th layer of the ResNet-18 backbone network as the output of the spatial domain feature extraction module. The semantic feature extraction module is a five-layer Swin-B Transformer backbone network. The five layers of the Swin-B Transformer backbone network are connected sequentially. The input of the first layer of the Swin-B Transformer backbone network serves as the input of the semantic feature extraction module, receiving the same RGB image and downsampling it to obtain a size of... The RGB image is used as the output of the semantic feature extraction module, with the output of the 5th layer of the Swin-B Transformer backbone network serving as the output of the semantic feature extraction module. The input of the first convolutional block receives the feature map output from the first layer of the ResNet-18 backbone network. The feature map output by the first convolutional block is denoted as... The input of the second convolutional block receives the feature map output from the second layer of the ResNet-18 backbone network. The feature map output by the output of the second convolutional block is denoted as... The input of the third convolutional block receives the feature map output from the third layer of the ResNet-18 backbone network. The feature map output by the third convolutional block is denoted as... The input of the fourth convolutional block receives the feature map output from the output of the fourth layer of the ResNet-18 backbone network. The feature map output by the fourth convolutional block is denoted as... The input of the 5th convolutional block receives the feature map output from the output of the 2nd layer of the Swin-B Transformer backbone network. The feature map output by the output of the 5th convolutional block is denoted as... The input of the 6th convolutional block receives the feature map output from the output of the 3rd layer of the Swin-B Transformer backbone network. The feature map output by the output of the 6th convolutional block is denoted as... The input of the 7th convolutional block receives the feature map output from the output of the 3rd layer of the Swin-B Transformer backbone network. The feature map output by the 7th convolutional block is denoted as... The input of the 8th convolutional block receives the feature map output from the output of the 4th layer of the Swin-B Transformer backbone network. The feature map output by the 8th convolutional block is denoted as... ; Among them, feature map The size is Feature map The size is Feature map The size is Feature map The size is The feature map output from the second layer of the Swin-B Transformer backbone network. The size is Feature map The size is Feature map The size is Feature map The size is Feature map The size is Feature map The size is Feature map The size is Feature map The size is Feature map The size is Feature map The size is Feature map The size is Feature map The size is ; Both the ResNet-18 backbone and the Swin-B Transformer backbone are existing structural frameworks, and their network structures have been publicly disclosed. For example, the ResNet-18 backbone is described in references such as K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016, and the Swin-B Transformer backbone is described in references such as Nian Liu, Ni Zhang, Kaiyuan Wan, Ling Shao, and Junwei Han. Visual saliency transformer. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 4722–4732, 2021.

[0031] In this embodiment of the invention, the first convolutional block consists of a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. The input of the convolutional layer serves as the input of the first convolutional block, and the output of the ReLU activation function layer serves as the output of the first convolutional block. The convolutional layer in the first convolutional block has a kernel size of 3, a stride size of 1, padding of 1, 64 input channels, and 64 output channels. The second convolutional block consists of a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. The input of the convolutional layer serves as the input of the second convolutional block, and the output of the ReLU activation function layer serves as the output of the second convolutional block. The first convolutional block has a kernel size of 3, a stride of 1, padding of 1, 128 input channels, and 64 output channels. The third convolutional block consists of a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. The input of the convolutional layer serves as the input of the third convolutional block, and the output of the ReLU activation function layer serves as the output of the third convolutional block. The kernel size of the convolutional layer in the third convolutional block is 3, the stride is 1, padding is 1, the number of input channels is 256, and the number of output channels is 64. The fourth convolutional block consists of a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. The input of the convolutional layer serves as the input of the fourth convolutional block, and the ReLU activation function layer serves as the output of the third convolutional block. The output of the liveness function layer serves as the output of the 4th convolutional block. The convolutional layer in the 4th convolutional block has a kernel size of 3, a stride of 1, padding of 1, 512 input channels, and 64 output channels. The 5th convolutional block consists of a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. The input of the convolutional layer serves as the input of the 5th convolutional block, and the output of the ReLU activation function layer serves as the output of the 5th convolutional block. The convolutional layer in the 5th convolutional block has a kernel size of 3, a stride of 1, padding of 1, 128 input channels, and 64 output channels. The 6th convolutional block consists of a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. The convolutional block consists of several layers, each with a different activation function. The input of the convolutional layer serves as the input to the 6th convolutional block, and the output of the ReLU activation function layer serves as the output of the 6th convolutional block. The convolutional layer in the 6th convolutional block has a kernel size of 3, a stride size of 1, padding of 1, 256 input channels, and 64 output channels. The 7th convolutional block consists of a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. The input of the convolutional layer serves as the input to the 7th convolutional block, and the output of the ReLU activation function layer serves as the output of the 7th convolutional block. The convolutional layer in the 7th convolutional block has a kernel size of 3, a stride size of 1, padding of 1, 512 input channels, and 64 output channels.The 8th convolutional block consists of a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. The input of the convolutional layer serves as the input of the 8th convolutional block, and the output of the ReLU activation function layer serves as the output of the 8th convolutional block. The convolutional layer in the 8th convolutional block has a kernel size of 3, a stride size of 1, padding of 1, 512 input channels, and 64 output channels.

[0032] like Figure 3 As shown, the cross-modal attention module executes in two branches, specifically: The preprocessing process for the first branch is as follows: the input receives the feature map output from the output of the fourth convolutional block. For feature maps Perform a flatten operation, flattening to a size of tensor ; for tensors Performing a linear mapping operation yields a size of query vector Size is key vector Size is value vector The first linear layer has 64 input channels and 128 output channels, with an offset of [value missing]. The second linear layer has 64 input channels and 64 output channels, with an offset of [value missing]. ; for query vector and key vector Performing matrix multiplication yields an attention feature map. Size is Attention feature map Performing a deformation operation yields an attention feature map. Size is ; value vector and attention feature map Perform matrix multiplication to obtain semantic attention feature maps. Size is ; The preprocessing process for the second branch is as follows: the input receives the feature map output from the output of the 8th convolutional block. For feature maps Perform a flatten operation, flattening to a size of tensor ; for tensors Performing a linear mapping operation yields a size of query vector Size is key vector Size is The third linear layer has 64 input channels and 128 output channels, with an offset of [value missing]. The fourth linear layer has 64 input channels and 64 output channels, with an offset of [value missing]. ; for query vector and Performing matrix multiplication yields an attention feature map. Size is Attention feature map Performing a deformation operation yields an attention feature map. Size is ;right and attention feature map Perform matrix multiplication to obtain semantic attention feature maps. Size is ; The post-processing of the first branch is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Performing the transformation operation yields a size of vector For vectors and semantic attention feature maps Performing matrix multiplication yields a result of size . semantic consensus matrix ; The post-processing for the second branch is as follows: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] Performing the deformation operation yields a size of vector For vectors and semantic attention feature maps Performing matrix multiplication yields a result of size . semantic consensus matrix ; Subsequently, the semantic consensus matrix was analyzed. Perform the transformation operation to obtain a size of Feature map For semantic consensus matrix Perform the transformation operation to obtain a size of Feature map ; feature map With feature map The size obtained by splicing along the channel direction is 128 feature map The input of the 14th convolutional block receives the feature map. The feature map output from the 14th convolutional block is used as the semantic consensus matrix output by the cross-modal attention module. Semantic consensus matrix The size is 64. In this embodiment, the flattening operation, matrix multiplication operation, and deformation operation are common operations in convolutional neural networks.

[0033] In this embodiment of the invention, the 14th convolutional block in the cross-modal attention module consists of a convolutional layer and a batch normalization layer connected in sequence. The input of the convolutional layer serves as the input of the 14th convolutional block, and the output of the batch normalization layer serves as the output of the 14th convolutional block. The convolutional layer in the 14th convolutional block has a kernel size of 3, a stride size of 1, a padding size of 1, 128 input channels, and 64 output channels.

[0034] In this embodiment of the invention, the feature enhancement module mainly consists of four structurally identical multi-spatial feature enhancement blocks; the input of the first multi-spatial feature enhancement block receives the feature map output from the output of the first convolutional block. The feature map output from the first multi-spatial feature enhancement block is denoted as... , The size is The input of the second multi-spatial feature enhancement block receives the feature map output from the output of the second convolutional block. The feature map output from the output of the second multi-spatial feature enhancement block is denoted as follows: , The scale is The input of the third multi-spatial feature enhancement block receives the feature map output from the output of the third convolutional block. The feature map output from the third multi-spatial feature enhancement block is denoted as follows: , The size is The input of the fourth multi-spatial feature enhancement block receives the feature map output from the output of the fourth convolutional block. The feature map output from the output of the fourth multi-spatial feature enhancement block is denoted as... , The size is In this embodiment, pixel multiplication is a standard operation in convolutional neural networks.

[0035] See Figure 4In this embodiment of the invention, the multi-spatial feature enhancement block includes a first dilated convolution block, a second dilated convolution block, a third dilated convolution block, a fourth dilated convolution block, a pooling convolution block, and a fifteenth convolution block. The input of the first dilated convolution block receives the feature map output from the output of the multi-spatial feature enhancement block. The input of the second dilated convolution block receives a feature map obtained by concatenating the feature map output from the multi-spatial feature enhancement block and the feature map output from the first dilated convolution block through multiple channels. The input of the third dilated convolution block receives the feature map output from the multi-spatial feature enhancement block and the feature map output from the second dilated convolution block through multiple channels. The feature map obtained by concatenating the row channels is used as the input of the 4th dilated convolutional block. The input of the 4th dilated convolutional block receives the feature map output from the multi-spatial feature enhancement block and the feature map output from the 3rd dilated convolutional block, and then performs channel concatenation on them. The input of the pooling convolutional block receives the feature map output from the multi-spatial feature enhancement block. Subsequently, pixel multiplication is performed on the feature maps output from the pooling convolutional block and the multi-spatial feature enhancement block. The input of the 15th convolutional block receives the feature maps output from the 1st, 2nd, 3rd, and 4th dilated convolutional blocks and the output of the pixel multiplication layer. The output feature map is obtained by concatenating the channels of the output feature map. The feature map output from the 15th convolutional block is used as the output feature map of the multi-spatial feature enhancement block. Specifically, the first dilated convolutional block has a kernel size of 3, a dilation rate of 1, padding of 1, 64 input channels, and 64 output channels; the second dilated convolutional block has a kernel size of 3, a dilation rate of 2, padding of 1, 128 input channels, and 64 output channels; the third dilated convolutional block has a kernel size of 3, a dilation rate of 5, padding of 1, 128 input channels, and 64 output channels. The number of channels is 64; the kernel size of the dilated convolutional layer in the 4th dilated convolutional block is 3, the dilation rate is 7, the padding is 1, the number of input channels is 128, and the number of output channels is 64; the pooling convolutional block consists of a global average pooling layer, a convolutional layer, and a sigmoid activation function layer connected in sequence. The input of the global average pooling layer serves as the input of the pooling convolutional block, and the output of the sigmoid activation function layer serves as the output of the pooling convolutional block. The kernel size of the convolutional layer in the pooling convolutional block is 1, the number of input channels is 64, and the number of output channels is 64; the kernel size of the convolutional layer in the 15th convolutional block is 1, the padding is 0, the number of input channels is 320, and the number of output channels is 64; in this embodiment, pixel multiplication is a conventional operation in convolutional neural networks.

[0036] In this embodiment of the invention, the embedding fusion module mainly consists of four structurally identical multi-scale information processing blocks; the first input of the first multi-scale information processing block receives the feature map output by the output of the first multi-spatial feature enhancement block. The second input receives the feature map output from the output of the fifth convolutional block. The third input receives the feature map output from the output of the second multi-scale information processing block. The feature map output by the first multi-scale information processing block is denoted as... The first input of the second multi-scale information processing block receives the feature map output from the output of the second multi-spatial feature enhancement block. The second input receives the feature map output from the output of the 6th convolutional block. The third input receives the feature map output from the output of the third multi-scale information processing block. The feature map output from the output of the second multi-scale information processing block is denoted as... The first input of the third multi-scale information processing block receives the feature map output from the output of the third multi-spatial feature enhancement block. The second input receives the feature map output from the output of the 7th convolutional block. The third input receives the feature map output from the output of the fourth multi-scale information processing block. The feature map output by the third multi-scale information processing block is denoted as... The first input of the fourth multi-scale information processing block receives the feature map output from the output of the fourth multi-spatial feature enhancement block. The second input receives the feature map output from the output of the 8th convolutional block. The third input receives the feature map output from the cross-modal cross-attention module. The feature map output by the fourth multi-scale information processing block is denoted as... ; The size is , The size is , The size is , The size is In this embodiment, pixel addition is a standard operation in convolutional neural networks.

[0037] See Figure 5In this embodiment of the invention, the multi-scale information processing block includes a 16th convolutional block and a 17th convolutional block. The input of the pixel multiplication layer in the multi-scale information processing block serves as the first input of the multi-scale information processing block, the input of the upsampling layer in the multi-scale information processing block serves as the second input of the multi-scale information processing block, and the input of the Sigmoid activation function layer in the multi-scale information processing block serves as the third input of the multi-scale information processing block. Pixel multiplication is performed on the feature map output from the Sigmoid activation function layer, the feature map from the first input, and the feature map output from the upsampling layer to obtain two feature maps. Channel concatenation of these two feature maps yields a feature map received by the input of the 16th convolutional block. The input of the 17th convolutional block receives the feature map output from the 16th convolutional block and the feature map received by the second input of the multi-scale information processing block, and performs pixel addition on the feature map received by the 16th convolutional block. The feature maps output from the 17 convolutional blocks are subjected to pixel addition, and the resulting feature map is used as the output feature of the multi-scale information processing block. The 16th convolutional block consists of a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected sequentially. The input of the convolutional layer serves as the input of the 16th convolutional block, and the output of the ReLU activation function layer serves as its output. The kernel size of the convolutional layer in the 16th convolutional block is 3, and the stride size is [missing value]. 1. Padding is 1, input channels are 128, and output channels are 64. The 17th convolutional block consists of a convolutional layer, a batch normalization layer, and a ReLU activation function layer connected in sequence. The input of the convolutional layer serves as the input of the 17th convolutional block, and the output of the ReLU activation function layer serves as the output of the 16th convolutional block. The convolutional layer in the 17th convolutional block has a kernel size of 3, a stride size of 1, padding of 1, 64 input channels, and 64 output channels.

[0038] In this embodiment of the invention, the output processing module mainly consists of five convolutional blocks with identical structures; the input of the 9th convolutional block receives the feature map output from the output of the 1st multi-scale information processing block. The feature map output from the 9th convolutional block is used as the first segmentation image. The input of the 10th convolutional block receives the feature map output from the output of the 2nd multi-scale information processing block. The feature map output from the 10th convolutional block is used as the second segmentation image. The input of the 11th convolutional block receives the feature map output from the output of the 3rd multi-scale information processing block. The feature map output from the 11th convolutional block is used as the third segmentation image. The input of the 12th convolutional block receives the feature map output from the output of the 4th multi-scale information processing block. The feature map output from the 12th convolutional block is used as the fourth segmentation image. The input of the 13th convolutional block receives the feature map output from the cross-modal attention module, and the feature map output from the 13th convolutional block serves as the fifth segmentation image. The first segmented image The size is Second segmented image The size is The third segmented image The size is The fourth segmented image The size is Fifth segmented image The size is .

[0039] In this embodiment of the invention, the output processing module includes a 9th convolutional block, a 10th convolutional block, an 11th convolutional block, a 12th convolutional block, and a 13th convolutional block. The 9th convolutional block consists of a convolutional layer and a sigmoid activation function layer connected sequentially. The input of the convolutional layer serves as the input of the 9th convolutional block, and the output of the sigmoid activation function layer serves as the output of the 9th convolutional block. The convolutional layer in the 9th convolutional block has a kernel size of 3, a stride of 1, padding of 1, 64 input channels, and an output... The 10th convolutional block consists of a convolutional layer and a sigmoid activation function layer connected in sequence. The input of the convolutional layer serves as the input of the 10th convolutional block, and the output of the sigmoid activation function layer serves as the output of the 10th convolutional block. The convolutional layer in the 10th convolutional block has a kernel size of 3, a stride of 1, padding of 1, 64 input channels, and 1 output channel. The 11th convolutional block consists of a convolutional layer and a sigmoid activation function layer connected in sequence. The input of the convolutional layer... The input terminal of the 11th convolutional block is used as the input terminal, and the output terminal of the sigmoid activation function layer is used as the output terminal of the 11th convolutional block. The convolutional layer in the 11th convolutional block has a kernel size of 3, a stride size of 1, padding of 1, 64 input channels, and 1 output channel. The 12th convolutional block consists of a convolutional layer and a sigmoid activation function layer connected in sequence. The input terminal of the convolutional layer is used as the input terminal of the 12th convolutional block, and the output terminal of the sigmoid activation function layer is used as the output terminal of the 12th convolutional block. The convolutional layers in the two convolutional blocks have a kernel size of 3, a stride size of 1, padding of 1, 64 input channels, and 1 output channel. The 13th convolutional block consists of a convolutional layer and a sigmoid activation function layer connected in sequence. The input of the convolutional layer serves as the input of the 13th convolutional block, and the output of the sigmoid activation function layer serves as the output of the 13th convolutional block. The convolutional layers in the 13th convolutional block have a kernel size of 3, a stride size of 1, padding of 1, 64 input channels, and 1 output channel.

[0040] In this embodiment of the invention, step S3 uses a training set to train the marine organism segmentation network. After each training round, the marine organism segmentation network outputs a first segmented image, a second segmented image, a third segmented image, and a fourth segmented image corresponding to each high-resolution underwater organism image in the training set, which are denoted as follows: , , , Then calculate the network loss, denoted as . , ;in, , = , This represents the marine organism label corresponding to each high-resolution underwater organism image in the training set. Represents the binary cross-entropy loss function. This represents the weighting coefficient, which is taken as [value] in this embodiment. The initial learning rate using the Adam optimizer is And adjust the weight decay to To update network parameters, and use a batch size of 80.

[0041] In this embodiment of the invention, a total of 80 training rounds are performed according to step S2 to obtain a marine organism segmentation network model for processing high-resolution underwater biological images.

[0042] In this embodiment of the invention, step S4 uses a marine organism segmentation network model to test each high-resolution underwater organism image in the test set. The marine organism segmentation network model outputs the first segmented image corresponding to each high-resolution underwater organism image in the test set as the marine organism segmentation result.

[0043] In this embodiment of the invention, to further verify the feasibility and effectiveness of the method, experiments are conducted on the method as follows: The existing marine animal segmentation dataset MAS3K was selected for both training and testing. Training was performed on the MAS3K training set, and testing was conducted on the MAS3K test set. Five widely used evaluation metrics were employed, including mean absolute error. Human visual perception assessment indicators Structural similarity index Weighted regional similarity metrics Average regional similarity measure index ; Specifically, mean absolute error Metrics are used to evaluate the pixel-level accuracy between predicted results and true labels; human visual perception evaluation metrics Simultaneously, pixel-level matching accuracy and image-level statistics are evaluated to assess the overall and local accuracy of the model; structural similarity metrics Used to measure the structural similarity between predicted results and true labels; a weighted metric. and average region similarity measure index By combining recall and precision, the method eliminates the influence of traditional metrics that consider each pixel equally. Table 1 below shows a quantitative comparison of the evaluation metrics between the method of this invention and existing object segmentation methods on the MAS3K test set: Table 1. Quantitative comparison of the proposed method and common target segmentation methods on the MAS3K test set. As can be seen from the data in Table 1, the method of the present invention achieved optimal results in all five common performance indicators.

[0044] In the description of this invention, the references to "one embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0045] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for marine organism segmentation based on high-resolution underwater optical images, characterized in that, Includes the following steps: Step S1: Collect multiple high-resolution underwater optical images under different scenarios and marine biological tags corresponding to each high-resolution underwater optical image, and divide them into training set and test set; Step S2: Randomly select one of the high-resolution underwater optical images from the training set, and input the high-resolution underwater optical image into the pre-constructed marine organism segmentation network to perform spatial feature extraction, semantic feature extraction, feature enhancement, cross-modal cross-attention fusion, embedding fusion and image segmentation to obtain multiple marine organism segmentation images of different dimensions; Step S3: Repeat step S2 and calculate the total network loss based on each marine organism segmentation image and the marine organism label corresponding to each marine organism segmentation image. Optimize the marine organism segmentation network based on the total network loss to obtain a marine organism segmentation network model. Step S4: Input the test set into the marine organism segmentation network model to obtain the highest dimension marine organism segmentation image corresponding to each high-resolution underwater optical image in the test set as the marine organism segmentation result.

2. The marine organism segmentation method according to claim 1, characterized in that, In step S1, each of the high-resolution underwater optical images and each of the marine biotags are preprocessed. The preprocessing of each of the high-resolution underwater optical images includes: For each of the high-resolution underwater optical images, the size of the high-resolution underwater optical image is scaled down to [value missing]. The scaled high-resolution underwater optical image was then normalized. The pixel values ​​of all pixels in the R channel were normalized to a mean of 0.485 and a variance of 0.229, the pixel values ​​of all pixels in the G channel were normalized to a mean of 0.456 and a variance of 0.224, and the pixel values ​​of all pixels in the B channel were normalized to a mean of 0.406 and a variance of 0.

225. The preprocessing procedure for each of the aforementioned marine biotags includes: For each of the marine organism tags, the size of the marine organism tag is scaled down to... .

3. The marine organism segmentation method according to claim 1, characterized in that, The marine organism segmentation network in step S2 includes a backbone module, a feature enhancement module, a cross-modal cross-attention module, an embedding fusion module, and an output processing module. The backbone module is connected to the feature enhancement module and the embedding fusion module, the cross-modal cross-attention module is connected to the backbone module and the embedding fusion module, and the embedding fusion module is connected to the feature enhancement module and the output processing module. The backbone module receives the high-resolution underwater optical image and performs spatial feature extraction and semantic feature extraction to obtain a feature map. Feature map Feature map Feature map Feature map Feature map Feature map Feature map The feature map is enhanced by the feature enhancement module. The feature map The feature map The feature map Feature enhancement is performed to obtain feature maps. Feature map Feature map Feature map The feature map is processed by the cross-modal cross-attention module. and the feature map feature map The semantic consensus matrix is ​​obtained by performing cross-attention fusion. The feature map is received through the embedding fusion module. The feature map The feature map The feature map The semantic consensus matrix The feature map The feature map The feature map The feature map Feature maps are obtained through multi-scale information processing. Feature map Feature map Feature map The feature map is received through the output processing module. The feature map The feature map The feature map The semantic consensus matrix The first segmented image is obtained by performing target segmentation. Second segmented image Third segmented image Fourth segmented image and the fifth segmented image As a segmentation image of multiple marine organisms.

4. The marine organism segmentation method according to claim 3, characterized in that, The main module includes a spatial feature extraction module, a semantic feature extraction module, and a first, second, third, fourth, fifth, sixth, seventh, and eighth convolutional block. The spatial feature map module is connected to the first, second, third, and fourth convolutional blocks, respectively. The semantic feature extraction module is connected to the fifth, sixth, seventh, and eighth convolutional blocks, respectively. The spatial feature extraction module receives the high-resolution underwater optical image and extracts features to obtain a feature map. Feature map Feature map Feature map The feature map is obtained by performing convolution processing through the first convolutional block, the second convolutional block, the third convolutional block, and the fourth convolutional block. The feature map The feature map and the feature map The semantic feature extraction module receives the downsampled high-resolution underwater optical image and performs feature extraction to obtain a feature map. Feature map Feature map Feature map The feature map is obtained by performing convolution processing through the 5th, 6th, 7th, and 8th convolutional blocks. The feature map The feature map and the feature map .

5. The marine organism segmentation method according to claim 4, characterized in that, The spatial feature extraction module includes a ResNet-18 backbone network with four layers connected in sequence. The first layer is connected to the first convolutional block, the second layer is connected to the second convolutional block, the third layer is connected to the third convolutional block, and the fourth layer is connected to the fourth convolutional block.

6. The marine organism segmentation method according to claim 4, characterized in that, The semantic feature extraction module includes a 5-layer Swin-B Transformer backbone network connected in sequence. The 2nd layer is connected to the 5th convolutional block, the 3rd layer is connected to the 6th convolutional block, the 4th layer is connected to the 7th convolutional block, and the 5th layer is connected to the 8th convolutional block.

7. The marine organism segmentation method according to claim 4, characterized in that, The feature enhancement module includes four structurally identical multi-spatial feature enhancement blocks. The input of each multi-spatial feature enhancement block is connected to the first convolutional block, the second convolutional block, the third convolutional block, and the fourth convolutional block, respectively. The output of each multi-spatial feature enhancement block is connected to the embedding fusion module.

8. The marine organism segmentation method according to claim 4, characterized in that, The embedded fusion module includes four structurally identical multi-scale information processing blocks. The input of each multi-scale information processing block is connected to the fifth, sixth, seventh, and eighth convolutional blocks, respectively, and the output of each multi-scale information processing block is connected to the output processing module.

9. The marine organism segmentation method according to claim 8, characterized in that, The cross-modal attention module includes two branches and a 14th convolutional block connecting the two branches. The first branch receives the feature map. Perform a flattening operation to obtain a tensor The tensor is processed through the first linear layer and the second linear layer respectively. Perform a linear mapping operation to obtain the query vector. Key vector Sum value vector For the query vector and the key vector Attention feature maps are obtained by performing matrix multiplication and transformation operations. For the value vector and the attention feature map Performing matrix multiplication yields semantic attention feature maps. ; The second branch is for receiving the feature map. Perform a flattening operation to obtain a tensor The tensor is processed through the third and fourth linear layers respectively. Perform a linear mapping operation to obtain the query vector. Key vector Sum value vector For the query vector and the key vector Attention feature maps are obtained by performing matrix multiplication and transformation operations. For the value vector and the attention feature map Performing matrix multiplication yields semantic attention feature maps. ; The semantic attention feature map is processed by the 14th convolutional block. and the semantic attention feature map The semantic consensus matrix is ​​obtained by performing concatenation and convolution operations. .

10. The marine organism segmentation method according to claim 8, characterized in that, The output processing module includes multiple convolutional blocks with identical structures, and the input of each convolutional block is connected to each of the multi-scale information processing modules and the cross-modal cross-attention module.

Images