Recovery method and system for fish sheltered contour

Abstract

The invention discloses a recovery method and system for a fish sheltered contour, and belongs to the technical field of computer vision, and the method comprises the steps: collecting an image of underwater fish, carrying out the manual marking and data enhancement, and generating an image of the underwater fish with sheltering, and taking the image as a data set; based on a Mask2Forme model, an attention improving module is embedded in Stage3 of a backbone network of the model, a KAN layer is used for replacing an MLP layer in an original backbone network, meanwhile, an occlusion perception attention mechanism SEAM is added to a detection head of the model, after the model is improved, training is conducted through a data set, a fish image segmentation model is constructed, and a fish image segmentation model is obtained. And the image segmentation module is used for identifying a sheltered area in the image of the underwater fish and performing image segmentation on the sheltered area according to fish types. According to the method, segmentation under the condition that the fishes are mutually shielded is more complete, and the effect on underwater fishes is better, so that a technical foundation is laid for subsequent fish school biomass calculation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and system for restoring occluded contours of fish. Background Art

[0002] At present, the aquaculture industry is developing rapidly. As aquaculture develops towards intelligence and intensiveness, the estimation of fish biomass in aquaculture is particularly important. As common biomass information of fish, fish counting, calculation of fish breeding density and size can not only help farmers estimate fish production potential and calculate reproductive capacity, but also reasonably control breeding density, determine breeding decisions and fishing time, etc., which has positive practical significance for promoting the development of intelligent fisheries. However, these biomasses often face occlusion problems when calculating, especially in dense fish schools. When using video monitoring technology to calculate fish biomass, intra-class occlusion will cause some fish in the video image to be blocked by other individuals, making it difficult to automatically or manually calculate fish biomass.

[0003] In the process of breeding fish in marine ranches, the use of computer vision technology to detect and count dense fish schools can not only help breeders estimate fish production potential and calculate fertility, but also reasonably control breeding density, determine breeding decisions and fishing time, etc., which has positive practical significance for promoting the development of intelligent fisheries. Since the number of farmed fish is relatively large and dense, there will be occlusion between fish. When fish block each other, the algorithm cannot accurately measure their number and size, and thus cannot accurately calculate their biomass. Summary of the invention

[0004] In order to solve the problem that fish biomass calculation is difficult due to intra-class occlusion in fish schools, the present invention proposes a Transformer-based technology for restoring fish occlusion contours, aiming to calculate the fish biomass by restoring the occluded fish individuals in the fish school.

[0005] In order to achieve the above technical objectives, the present application provides a method for restoring fish occlusion contours, comprising the following steps:

[0006] Collect images of underwater fish, perform manual annotation and data enhancement, and generate underwater fish images with occlusion as a dataset;

[0007] Based on the Mask2Forme model, an improved attention module was embedded in Stage 3 of the backbone network of the model, and the MLP layer in the original backbone network was replaced by the KAN layer. After the model was improved, it was trained through the dataset to construct a fish image segmentation model for identifying the occluded areas in images of underwater fish and performing image segmentation on the occluded areas according to the fish species.

[0008] Preferably, during the manual labeling process, the images of underwater fish are labeled using an ISAT labeling tool.

[0009] Preferably, in the process of data enhancement, data enhancement is performed by copy-paste based on the image of the labeled underwater fish, which is used to cut out the instance corresponding to the mask part from one image and then paste it into another image. The two images are scaled 1:1, and the instances are also scaled 1:1, so that an image of underwater fish with occlusion can be synthesized.

[0010] Preferably, in the process of constructing the fish target segmentation model, the MCBAM attention module is embedded in the downsampling part of the backbone network Swin-transformer, and trained with the data set to generate the fish target segmentation model, wherein the data set is divided into a training set and a test set in a ratio of 8:2, the model is trained with the training set, and the trained model is tested with the test set.

[0011] Preferably, in the process of obtaining the MCBAM attention module, the cascade connection of the CBAM attention model is changed to a parallel connection, the channel and spatial weights are retained respectively, the two are weighted and then added, and the channel attention and spatial attention are fused, the two parts are jointly modeled to interact directly, and the MCBAM attention module is generated.

[0012] Preferably, after the improvement process of the backbone network, the model is improved by adding an occlusion-aware attention mechanism, SEAM, in the detection head of the model.

[0013] The present invention discloses a system for restoring fish occluded contours, comprising:

[0014] A data acquisition module, used to collect images of underwater fish;

[0015] A data processing module is used to manually annotate and enhance the data of the collected underwater fish images to generate underwater fish images with occlusion as a data set;

[0016] The image segmentation module is based on the Mask2Forme model. By embedding an improved attention module in Stage 3 of the model's backbone network and replacing the MLP layer in the original backbone network with the KAN layer, the model is improved and trained through the dataset to build a fish image segmentation model for identifying occluded areas in images of underwater fish and performing image segmentation on the occluded areas according to the fish species.

[0017] Preferably, the data processing module is also used to perform data enhancement through copy-paste based on the image of the labeled underwater fish, and is used to cut out the instance corresponding to the mask part from one image, and then paste it into another image. The two images are scaled 1:1, and the instance is also scaled 1:1, so that an image of underwater fish with occlusion can be synthesized.

[0018] Preferably, the image segmentation module is also used to embed the MCBAM attention module in the downsampling part of the backbone network Swin-transformer, and train with the data set to generate a fish target segmentation model, wherein the data set is divided into a training set and a test set in a ratio of 8:2, the model is trained with the training set, and the trained model is tested with the test set.

[0019] Preferably, the image segmentation module is also used to improve the model by adding an occlusion-aware attention mechanism SEAM in the detection head of the model.

[0020] The present invention discloses the following technical effects:

[0021] The present invention can segment fish more completely when they are blocked from each other, and has better effects on underwater fish, thus laying a technical foundation for the subsequent calculation of fish biomass. The model trained by this algorithm can be deployed in the monitoring system of the marine ranch, and the biomass of the fish can be monitored in real time by connecting to the monitoring camera of the marine ranch. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is the technical roadmap described in the present invention;

[0024] Figure 2 is an example diagram of the data set annotation according to the present invention;

[0025] Figure 3 This is an example diagram of the underwater rockfish segmentation of the present invention;

[0026] Figure 4 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0027] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0028] like Figure 1-4 As shown, the present invention produces a grouper (fish) data set: underwater grouper pictures are collected, and then the labelme annotation tool is used to manually annotate the data set, and the annotated data set is subjected to the copy-paste data enhancement method, that is, the instance corresponding to the mask part is cut out from one image, and then pasted into another image. The two images are scaled in 1:1, and the instances are also scaled in 1:1 to synthesize the occluded underwater grouper pictures, which are copied and synthesized into the format of the coco data set, and finally the data set is divided into a training set and a test set in a ratio of 8:2.

[0029] Based on the Underwater oplegnathus punctatusDataset (UOPD), this paper proposes a KAN-MCBAM-Mask2Former fish occlusion contour recovery method. First, the block merging module in Stage3 of the Mask2Former backbone network is embedded with an improved MCBAM attention mechanism to enhance the model's ability to perceive the target; secondly, a new Swin Kansformer module is proposed to solve the problems of high computational complexity, large memory consumption, and poor interpretability of traditional MLP, and a SEAM module is added to the Mask_former head of the model to enhance the perception of occluded objects; specifically, the following processes are included:

[0030] The backbone network of Mask2Former used in the present invention is based on Swin Transformer, which is used to extract multi-scale features from data and capture long-range dependencies. Using the "moving window" technology, the self-attention mechanism is limited to different, non-overlapping local windows, which helps to establish mutual connections between these windows, thereby improving the overall efficiency of the system. The four-stage design of the Swin Transformer architecture includes multiple modules, including four modules: block division, block merging, linear embedding and Swin Transformer block. In the Swin Transformer model, it is mainly composed of a block merging module and a Swin Transformer module in series. The block merging module can perform downsampling operations according to the set downsampling magnification image feature map. The attention module is embedded in this layer, which can effectively extract multi-dimensional image feature information. The block division module first divides the original image into different, non-overlapping blocks, and processes each block as a separate tag. These tags combine the original pixel data from multiple channels to form their feature representation. Then the original features are converted into feature vectors of a specified dimension (C) through a linear embedding layer. Finally, these block tags will be processed by a series of TransTormer modules to help develop feature representations. The Swin Transformer has three main parts: Windowed Multi-Head Self-Attention (W-MSA), Moving Window Multi-Head Self-Attention, and Multi-Layer Perceptron (MLP). A layer normalization layer (LN) is used before each MSA module and MLP. After each module, a residual connection is established. After each stage, the architecture uses a block merging layer to reduce the number of tokens and create a new hierarchical representation. The different variants of the Swin Transformer include tiny, samll, base, and large, which contain different numbers of layers in stages 1, 2, 3, and 4. In particular, the number of layers in each stage in the tiny, samll, and base models are {2,2,6,2}, {2,2,18,2}, {2,2,18,2}, and {2,2,18,2}, respectively. In addition, the dimensions (C) of these variants are 96, 96, 128, and 129, respectively. The tiny variant of this model was selected as the base architecture for the fish occlusion restoration model.

[0031] In order to improve the serial module architecture of CBAM, the original "cascade connection" is changed to "parallel connection". The weights on the channel and space are retained separately, the two are weighted, and then added, and the channel attention and spatial attention are fused. Instead of analyzing the two separately, the two parts are jointly modeled and interact directly, which can make the obtained features more comprehensive and improve the model performance. First, the channel descriptor and spatial descriptor are obtained by global pooling, and then the correlation between the channel descriptor (global spatial feature) and the spatial descriptor (global channel feature) is modeled by MLP. Finally, it is separated into two parts again and weighted separately, so as to propose the MCBAM attention mechanism. The MCBAM attention mechanism can help the model pay more attention to the important feature information of the image and ignore the interference factors around the target, thereby improving the accuracy of the fish occlusion contour recovery model.

[0032] The present invention replaces the MLP layer in the Swin Transformer module with the KAN layer in the original ViT model, forming an improved Swin Transformer module structure consisting of a W-MSA layer, a SW-MSA layer, a layer normalization layer and a KAN network alternately. Layer normalization is applied before each block and residual connection is applied after each block. This architecture enhances the scalability and adaptability of the model, effectively solves the limitations of the traditional MLP, and improves the performance interpretability of fish occlusion contour recovery.

[0033] Based on the improved Swin Transformer model, the MCBAM attention mechanism is embedded to enhance its performance. The improved MCBAM attention mechanism is embedded in the block merging module in Stage 3 of the improved Swin Transformer. The embedding of this module can enhance the model's perception of the target, while paying attention to the local and global information of the image, and realizing the adaptive feature refinement of the image, so that the module can extract more accurate features, thereby improving the segmentation accuracy of the occluded part.

[0034] A multi-head attention network, or SEAM module, is added to the Mask_former head of the fish occlusion contour recovery model Mask2Former. This module is added for three purposes: to achieve multi-scale fish segmentation, to enhance the fish area in the image, and conversely to weaken the background area. The first part of SEAM is the depthwise separable convolution with residual connection. Depthwise separable convolution operates depthwise, that is, convolution is separated channel by channel. Although depthwise separable convolution can understand the importance of different channels and reduce the amount of parameters, it ignores the information relationship between channels. To compensate for this loss, the inputs of convolutions of different depths are then combined through pointwise (1×1) convolution. A two-layer fully connected network is then used to fuse the information of each channel so that the network can strengthen the connection between all channels. It is hoped that the model can compensate for the loss in the above occlusion case by the relationship between occluded fish and unoccluded fish learned in the previous step. The output logarithm learned by the fully connected layer is then processed by an exponential function to expand the value range from [0,1] to [1,e]. This exponential normalization provides a monotonic mapping relationship, making the result more tolerant to position errors. Finally, the output of the SEAM module is used as attention to be multiplied with the original features, so that the model can handle the fish occlusion problem more effectively.

[0035] Aiming at the problem that the intra-class occlusion in a school of fish makes it difficult to calculate the biomass of fish, this paper proposes a KM-Mask2Former fish occlusion contour recovery method to restore the occluded fish individuals, thereby calculating the biomass of fish. First, a method for synthesizing an underwater rockfish occlusion dataset was studied, and an underwater rockfish dataset (UOPD) containing 5800 images was constructed. In order to improve the segmentation accuracy and efficiency, the MCBAM attention module was embedded in the block merging block in the SwinTransformer backbone network Stage3, which enhances the model's perception of the target, takes into account the local and global information of the image, and realizes adaptive feature refinement, thereby extracting more accurate features and improving the segmentation accuracy of the occluded part. In addition, in order to solve the problems of high computational complexity, large memory consumption, and poor interpretability of traditional MLP, the present invention replaces the MLP layer in the Swin Transformer module with the KAN (Kolmogorov-Arnold Network) layer, which effectively overcomes the limitations of traditional MLP, enhances the scalability and adaptability of the model, and improves the performance interpretability of fish occlusion contour recovery. In order to solve the problems of alignment errors, local aliasing and feature loss caused by intra-class occlusion. The present invention adds an occlusion-aware attention mechanism (SEAM) to the detection head of the model. Finally, the method proposed in this paper achieves excellent segmentation effect on the UOPD dataset, significantly improves the accuracy and efficiency of fish occlusion contour recovery, and performs well in practical applications, which lays a technical foundation for subsequent operations such as fish density estimation, bait estimation and fish population estimation.

[0036] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0037] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0038] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for restoring fish occluded contours, It is characterized in that The following steps are involved: Collect images of underwater fish, perform manual annotation and data enhancement, and generate underwater fish images with occlusion as a dataset; Based on the Mask2Forme model, an improved attention module is embedded in Stage 3 of the backbone network of the model, and the MLP layer in the original backbone network is replaced by the KAN layer. After the model is improved, it is trained with the dataset to construct a fish image segmentation model for identifying occluded areas in images of underwater fish, and image segmentation of the occluded areas is performed according to the fish species.

2. A method for restoring fish occluded contours according to claim 1, Features: In the process of manual labeling, the images of underwater fish are labeled using the ISAT labeling tool.

3. A method for restoring fish occluded contours according to claim 2, Features: In the process of data enhancement, copy-paste is used to perform data enhancement based on the images of labeled underwater fish. The instance corresponding to the masked part is cut out from one image and then pasted into another image. The two images are scaled 1:1, and the instances are also scaled 1:1 to synthesize the underwater fish image with occlusion.

4. A method for restoring fish occluded contours according to claim 3, Features: In the process of constructing the fish target segmentation model, the MCBAM attention module is embedded in the downsampling part of the backbone network Swin-transformer, and the fish target segmentation model is generated by training with the data set, wherein the data set is divided into a training set and a test set in a ratio of 8:2, the model is trained with the training set, and the trained model is tested with the test set.

5. A method for restoring fish occluded contours according to claim 4, Features: In the process of obtaining the MCBAM attention module, the cascade connection of the CBAM attention model is changed to a parallel connection, the channel and spatial weights are retained respectively, the two are weighted and then added, and the channel attention and spatial attention are fused, the two parts are jointly modeled to interact directly, and the MCBAM attention module is generated.

6. A method for restoring fish occluded contours according to claim 5, Features: After the improvement process of the backbone network, the model is improved by adding the occlusion-aware attention mechanism SEAM in the detection head of the model.

7. A system for restoring fish occluded outlines, It is characterized in that include: A data acquisition module, used to collect images of underwater fish; A data processing module is used to manually annotate and enhance the data of the collected underwater fish images to generate underwater fish images with occlusion as a data set; The image segmentation module is based on the Mask2Forme model. An improved attention module is embedded in Stage 3 of the backbone network of the model, and the MLP layer in the original backbone network is replaced by the KAN layer. After the model is improved, it is trained with the data set to construct a fish image segmentation model for identifying occluded areas in images of underwater fish, and performing image segmentation on the occluded areas according to the fish species.

8. A system for restoring fish occluded contours according to claim 7, Features: The data processing module is also used to perform data enhancement through copy-paste based on the image of the marked underwater fish, and is used to cut out the instance corresponding to the mask part from one image, and then paste it into another image. The two images are scaled at a ratio of 1:1, and the instance is also scaled at a ratio of 1:1, so that an image of underwater fish with occlusion can be synthesized.

9. A system for restoring fish occluded contours according to claim 8, Features: The image segmentation module is also used to embed the MCBAM attention module in the downsampling part of the backbone network Swin-transformer, and train with the data set to generate the fish target segmentation model, wherein the data set is divided into a training set and a test set in a ratio of 8:2, the model is trained with the training set, and the trained model is tested with the test set.

10. A system for restoring fish occluded contours according to claim 9, Features: The image segmentation module is also used to improve the model by adding an occlusion-aware attention mechanism, SEAM, in the detection head of the model.

Images