Juvenile abalone shielding recovery method and system based on double-layer shielding removal network

Abstract

The invention relates to a juvenile abalone shielding recovery method and system based on a double-layer shielding removal network, and belongs to the technical field of image processing. The invention discloses a juvenile abalone occlusion recovery method based on a double-layer occlusion removal network, and the method comprises the following processing steps: 1, employing a ResNet101 backbone network and a feature pyramid network FPN as a feature integrator for an input picture, and carrying out the effective fusion and conversion of feature maps with different resolutions and semantic levels; positioning a target through a single-stage target detector; step 2, decoupling an overlapping relation of the picture information processed in the step 1 by adopting a double-layer shielding-removing structure, performing multi-scale feature fusion on a plurality of pieces of feature map information, and segmenting an instance obtained by a target detection branch; and step 3, performing feature association processing on the information processed in the step 2 so as to improve the fineness of the young abalone mask and the accuracy of target capture.

Description

Technical Field

[0001] The invention relates to a young abalone occlusion recovery method and system based on a double-layer de-occlusion network, and belongs to the technical field of image processing. Background Art

[0002] The biomass measurement of juvenile abalone is one of the key tasks in aquaculture management, which is directly related to the growth assessment of juvenile abalone, the adjustment of feeding strategies and the prediction of economic benefits. The efficiency of biomass measurement can be greatly improved by replacing manual work with image or video technology. However, in abalone farming, since juvenile abalone usually exist in groups and there are a lot of occlusions between individuals, it becomes very difficult to accurately measure the size of each juvenile abalone from images or videos, and it is difficult to accurately identify the shape characteristics of individuals, thus affecting the accuracy and reliability of grading. This occlusion not only affects the accuracy of the measurement, but may also lead to underestimation or overestimation of the group biomass, thus affecting the accuracy of grading and farming decisions. Summary of the invention

[0003] In order to solve the problem of occlusion caused by overcrowding of young abalone, which leads to decreased accuracy and incomplete outlines of young abalone, this paper proposes a young abalone occlusion recovery method based on a double-layer deocclusion network, which accurately restores the image outline of occluded young abalone and significantly improves the accuracy of biomass measurement and intelligent breeding grading. This technology can not only more accurately distinguish the outline and size of individuals, but also clearly present individual morphological characteristics, helping breeders make scientific and reasonable decisions in breeding management and breeding grading, and improving breeding efficiency and breeding success rate.

[0004] The present invention solves the above technical problems through the following technical solutions: A method for occlusion recovery of young abalone based on a double-layer deocclusion network is special in that it includes the following processing steps: Step 1: The input image uses the ResNet101 backbone network and the feature pyramid network FPN as a feature integrator to effectively fuse and transform feature maps of different resolutions and semantic levels; then, the target is located through a single-stage target detector; Step 2: Decouple the overlapping relationship of the image information processed in step 1 using a double-layer deocclusion structure, perform multi-scale feature fusion on multiple feature map information, and segment the instances obtained by the target detection branch; Step 3, the information processed in step 2 is subjected to feature association processing, that is, cosine similarity and feature association matrix are used to aggregate feature information according to the features of the occluded image, so as to improve the fineness of the young abalone mask and the accuracy of target capture.

[0005] A double-layer de-occlusion structure network is adopted to effectively decompose the occlusion relationship and improve the segmentation performance. A multi-scale feature fusion module is added to the network to solve the situation where the young abalone has similar color or texture with the background or other objects. And a feature association module is used to solve the detection and repair of occluded areas, enhancing the accuracy and reliability of occlusion processing.

[0006] Further, the step 1 specifically comprises the following steps: Step 1.1, use the ResNet101 backbone network to initially extract image feature information; Step 1.2: Pass the image feature information extracted in step 1.1 to the (feature pyramid) FPN. FPN combines low-level features with high-level features, uses a top-down path, starts from the highest-level features extracted by the backbone network, and gradually restores higher resolutions through upsampling. The upsampled features of each layer are combined with the features of the corresponding layer of the backbone network through lateral connections (usually through 1×1 convolution to match the number of channels) to enhance the expressive power and receptive field of the features, predict targets at different layers, and obtain the features of the entire input image. In step 1.3, the features of the entire input image obtained in step 1.2 are used as the target detector using a one-stage, fully convolutional FCOS to directly locate and predict the candidate box of the target region of interest (Rol); the RoI Align algorithm is used to accurately extract the feature sub-map of the target region of interest in the feature map of the entire image according to the position of the object detection box, and it is used as the input of the double-layer de-occlusion network for the final object segmentation.

[0007] Furthermore, the double-layer de-occlusion structure described in step 2 includes two de-occlusion layers; the first de-occlusion layer is used to capture the local structural information between pixels. By modeling the local structure, the first de-occlusion layer can effectively extract the contextual information around the target, thereby helping the network to better understand the environment and background of the target, thereby capturing the contour and mask information of the occluded object; the second de-occlusion layer is designed as a global context-aware module to capture the contour and mask of the occluded object.

[0008] Further, the first de-blocking layer comprises the following steps: Step 2.11, input the information extracted in step 1.3 into the first de-occlusion layer. Before processing the occlusion layer, first globally process the input information, process the input image through the superpixel segmentation algorithm, decompose the image into multiple regions, and create a node for each region; then, based on the similarity of color, position, and shape between the created nodes, construct a global adjacency matrix to represent the edge relationship between nodes; Next, according to the occlusion relationship of the nodes in the global graph, the graph is divided into two subgraphs, one is the subgraph of occluding objects, and the other is the subgraph of occluded objects; for the subgraph containing all the nodes of occluding objects and their relationships, the occlusion-aware branch is used to model the occluded objects and output the contour and mask of the occluded instance. The occlusion-aware branch is designed with a simple and effective method: a 3*3 convolution layer, a deocclusion convolution layer and an FCN (two convolution layers).

[0009] In the first deocclusion layer, multiple deocclusion convolution processes are performed on each sub-image area to obtain the feature representation of each node and output the feature map of the occluded object sub-image; the occluded object sub-image is temporarily only used as auxiliary information, and its main function is to judge the relationship with the occluding object.

[0010] The formula for deocclusion convolution is as follows: (1), in, Indicates Layer The feature vector of each node; Indicates The neighbor set of a node; Indicates The degree of each node; Indicates The weight matrix of the layer, Indicates The bias vector of the layer; represents the activation function; Step 2.12, input the occluded object features obtained in step 2.11 into the first FAM system and the MSF system at the same time; Step 2.13, the MSF system processes the information as follows: First, the feature map of the occluded object is processed by a three-dimensional convolutional layer with a size of 1×1×1. Then, the feature map is divided into four sub-feature maps in the channel dimension by adjusting the number of channels of the feature map, which are recorded as , =1, 2, 3, 4; in this way, the number of channels of the obtained sub-feature map is one quarter of the input feature map, while keeping the size of the feature map unchanged; Then, except In addition, each sub-feature map passes through a 3D convolution layer with a convolution kernel size of 3×3×3, followed by batch normalization and ReLU activation. The formula is as follows: (2), Among them, Conv3 represents the convolution operation with a convolution kernel size of 3×3×3; , =1,2,3,4, indicating feature information of 4 different scales; Next, information is exchanged between parallel features to make full use of information at different scales so that each feature can be supplemented by information at other scales. A residual connection is then used to add complementary information and enhance features at different scales. Finally, the fused features are fed into a 1×1×1 convolutional layer, which mixes features at different scales and solves the weakening of information flow caused by the segmentation layer; the formula is as follows: (3) in, represents the output feature map of the proposed multi-scale feature fusion module, and Concat represents the element-by-element concatenation operation; The information processed by the MSF module is input into the second de-occlusion layer.

[0011] Further, the second de-occlusion layer receives the image information at step 1 and the image information processed by the MSF system; The second deocclusion layer has a similar structure to the first deocclusion layer. It segments the occluded object according to the occlusion-aware features, outputs the contour and mask of the instance of the occluded object, and obtains the features of the occluded object. For each node in the input image information, the feature vectors of the occluding object sub-graph and the occluded object sub-graph are connected in sequence to form a longer feature vector; Then, the concatenated feature vector is input into a fully connected layer to output the mask and category of each node; the output mask indicates whether the node belongs to a certain instance, and the category indicates that the node belongs to a certain category.

[0012] Furthermore, the step 3 is two FAM systems, the first FAM module processes the features of the occluding object obtained through the first de-occlusion layer; and the second FAM system processes the features of the occluded object obtained through the second de-occlusion layer.

[0013] Further, the FAM module processing steps are as follows: Step 3.1, the output information of the two de-occlusion layers is used as the input of two feature association modules (FAM). The input local feature V passes through CNA, which is composed of a 1×1 convolution layer, a batch normalization layer, and a ReLU activation layer, and is used to extract and process features in the image; through the 1×1 convolution layer in the CNA, the feature dimension of the output channel is adjusted to reduce the dimension of the local feature V to 1 / 4 of the original and convert it to V1, and further capture the context information by performing a 3×3 convolution operation on V1, aiming to efficiently associate similar features and reduce the number of parameters; this process produces the key feature V2; Secondly, the key feature V2 is processed using the cosine similarity operation to obtain the relationship matrix S; the matrix S reflects the set of similarities between pixels. and The similarity F is defined by the following formula: (4) in, , Two different linear exchanges representing key features; Represents the cosine of the angle between two key features; Represents the L2 norm of the vector; represents the dot product; The relationship matrix S is a set of cosine similarities between all pixel features, expressed in matrix form as: (5) in, Indicates The transpose of the key points after linear exchange; Indicates The representation after linear exchange of key points; Representation characteristics and The dot product of is used to measure their similarity; Representation characteristics and The modulus length product is used for normalization to eliminate the impact of scale on similarity measurement; Indicates and The set of cosine similarities of feature vectors; is the number of features in key features V2.

[0014] Step 3.2, then perform Softmax normalization on the similarity matrix, and then perform associated feature mapping; after dynamically learning the relationship between pixels, normalize each row in the similarity matrix, that is, apply the Softmax algorithm, and obtain the matrix S': (6); Through the product operation between elements, the matrix It is used to model the pixel relationship between key features in V2 and then map out the associated features; these features effectively integrate the comprehensive information of the target by analyzing the similarity between pixels. The calculation formula is shown as follows: (7) in, is the matrix parameter, is the associated feature map obtained by matrix multiplication; Step 3.3, finally, the associated features are deconvolved The image details are restored and added element by element with the feature V1 to form a preliminary result feature; after the channel is expanded, this preliminary result feature is added to the original feature V through a residual connection to output a refined instance feature; the formula is as follows: (8) in, is the module output feature, Conv represents the convolution process, RC represents the residual connection process, Represents element addition operation; After obtaining the occluded area of ​​the young abalone, the outline of the young abalone is obtained by using boundary tracking. Image boundary tracking mainly starts from a single boundary starting point, determines the next boundary point based on the application of the discrimination criterion, and realizes continuous tracking of the boundary, thereby forming a complete target boundary. After obtaining the outline of the young abalone, the perimeter L of the young abalone is obtained by calculating the distances between all adjacent points on the boundary and adding them up. The specific formula is as follows: (9) in, and are the coordinates of adjacent boundary points, is the total number of boundary points; the last term is to ensure the closure of the boundary by also counting the distance between the last point and the first point.

[0015] A young abalone occlusion recovery system based on a double-layer deocclusion network is characterized in that the system comprises a backbone network, a double-layer deocclusion network and a FAM network; The backbone network includes a ResNet101 backbone network, a feature pyramid network FPN module and a single-stage target detector module; the ResNet101 backbone network module and the feature pyramid network FPN module are used as feature integrators to effectively fuse and convert feature maps of different resolutions and semantic levels; the single-stage target detector module is used to locate the target; The double-layer deocclusion network includes a first deocclusion module and a second deocclusion module; the double-layer deocclusion structure is used to decouple overlapping relationships, and multi-scale feature fusion is performed on multiple feature map information, and instances obtained by the target detection branch are segmented; The FAM network includes a first FAM module and a second FAM module; the FAM module includes a cosine similarity module and a feature association matrix module to perform feature association processing, that is, cosine similarity and feature association matrix are used to aggregate feature information according to the features of the occluded image, so as to improve the fineness of the juvenile abalone mask and the accuracy of target capture.

[0016] Furthermore, the first de-occlusion module includes a first de-occlusion layer module, a regression occlusion object information module and an MSF module; the MSF module includes a feature map division module and a parallel feature map information exchange module.

[0017] Furthermore, the second de-occlusion module includes a second de-occlusion layer module and an occluded object information regression module.

[0018] The beneficial effects of the technical features described in this application are: First, the present invention proposes a method for occlusion recovery of young abalone based on a double-layer deocclusion network. The double-layer deocclusion structure is used, in which the first deocclusion layer network is responsible for detecting the occluded object, and the second deocclusion layer network is responsible for inferring the occluded object, which effectively solves the occlusion problem in the young abalone image; and multi-scale feature fusion is used to solve the situation where the young abalone has similar color or texture with the background or other objects, and the accuracy of segmentation is improved by fusing features at different levels; finally, through the feature association module, the young abalone can be effectively associated with its surrounding environment, thereby improving the accuracy and reliability of target segmentation. Compared with traditional manual marking and manual measurement methods, the present invention effectively restores the occluded area on the basis of being different from the artificial physical method, thus laying the foundation for the future application of aquaculture by integrating the current existing technologies. The model proposed by the present invention can be applied to the real-time processing and grading detection of young abalone images, helping breeders to determine whether individuals meet breeding standards. The contour restoration image obtained by the technology of the present invention and the circumference calculation results of young abalone in a single image are shown in the figure. Figure 2 , Figure 3 shown. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This application provides a technical route for a method for restoring occlusion of young abalone based on a double-layer de-occlusion network; Figure 2 The first picture selected from the data set is used to verify the method proposed in this application, and the outline of the young abalone is obtained; Figure 3 The second picture selected from the data set is used to verify the method proposed in this application, and the outline of the young abalone is obtained; Figure 4 These are 53 juvenile abalone images involved in the embodiment, and the juvenile abalone outlines are obtained using the method proposed in this application. DETAILED DESCRIPTION

[0020] The following embodiments, in conjunction with the accompanying drawings, are only intended to illustrate the technical solutions described in the claims and are not intended to limit the protection scope of the claims.

[0021] Combination Figure 1 , a method for occlusion recovery of young abalone based on a double-layer de-occlusion network, the technical solution of the present invention includes the following contents: A 4K camera from RyzerVision USB4KHDR01 was used to capture images of densely occluded young abalone underwater. Then, the Labelme annotation tool was used to manually annotate the dataset, and the annotated dataset was subjected to the Copy-Paste data enhancement method. The object of copying and pasting is to extract the instance corresponding to the mask part from one image and then paste it randomly to another image. The main idea of ​​data enhancement uses mixed pasting and large-scale jitter. The two groups of images are divided into source images and main images, and are synthesized at a 1:1 scale to obtain densely occluded young abalone images. After data enhancement, the dataset is divided into training set, validation set, and test set at a ratio of 8:1:1.

[0022] The occlusion recovery method of young abalone based on a double-layer de-occlusion network of the present application involves the following systems: The system includes a backbone network, a double-layer de-occlusion network and a FAM network; The backbone network includes a ResNet101 backbone network, a feature pyramid network FPN module and a single-stage target detector module; the ResNet101 backbone network module and the feature pyramid network FPN module are used as feature integrators to effectively fuse and convert feature maps of different resolutions and semantic levels; the single-stage target detector module is used to locate the target; The double-layer deocclusion network includes a first deocclusion module and a second deocclusion module; the double-layer deocclusion structure is used to decouple overlapping relationships, and multi-scale feature fusion is performed on multiple feature map information, and instances obtained by the target detection branch are segmented; The first de-occlusion module includes a first de-occlusion layer module, a regression occlusion object information module and an MSF module; the MSF module includes a partition feature map module and a parallel feature map information exchange module; The first deocclusion layer module receives the information of the image features extracted by the backbone network, and the first deocclusion layer module inputs the processed image information into the regression occlusion object information module, and the image information processed by the regression occlusion object information module is input into the first FAM module and the MSF module; the MSF module inputs the processed image information into the second deocclusion layer; The second de-occlusion module includes a second de-occlusion layer module and a module for regressing the information of the occluded object; the module for regressing the information of the occluded object inputs the processed image information into the second FAM module.

[0023] The FAM network includes a first FAM module and a second FAM module; the two FAM modules have the same processing structure; The FAM module includes a cosine similarity module and a feature association matrix module to perform feature association processing, that is, cosine similarity and feature association matrix are used to aggregate feature information according to the features of the occluded image, so as to improve the fineness of the juvenile abalone mask and the accuracy of target capture.

[0024] The specific processing steps are as follows: During the network training process, the images in the training set are used as input images.

[0025] Step 1: The input image uses the ResNet101 backbone network and the feature pyramid network FPN as a feature integrator to effectively fuse and transform feature maps of different resolutions and semantic levels; then, the target is located through a single-stage target detector; The specific steps include: Step 1.1, use the ResNet101 backbone network to initially extract image feature information; Step 1.2: Pass the image feature information extracted in step 1.1 to FPN (feature pyramid). FPN combines low-level features and high-level features, uses a top-down path, starts from the highest-level features extracted by the backbone network, and gradually restores higher resolutions through upsampling. The upsampled features of each layer are combined with the features of the corresponding layer of the backbone network through lateral connections (usually through 1×1 convolution to match the number of channels) to enhance the expressive power and receptive field of the features, predict targets at different layers, and obtain the features of the entire input image. In step 1.3, the features of the entire input image obtained in step 1.2 are used as the target detector by using a one-stage, fully convolutional FCOS to directly locate and predict the candidate box of the target region of interest (Rol); the RoIAlign algorithm is used to accurately extract the feature sub-map of the target region of interest in the feature map of the entire image according to the position of the object detection box, and it is used as the input of the double-layer de-occlusion network for the final object segmentation.

[0026] Step 2: The image information processed in step 1, i.e., the feature sub-graph, is decoupled from the overlapping relationship using a double-layer deocclusion structure, and multiple feature graph information is fused at multiple scales, and the instances obtained by the target detection branch are segmented; The double-layer deocclusion structure includes two deocclusion layers; the first deocclusion layer is used to capture the local structural information between pixels. By modeling the local structure, the first deocclusion layer can effectively extract the contextual information around the target, thereby helping the network to better understand the environment and background of the target, thereby capturing the contour and mask information of the occluded object; the second deocclusion layer is designed as a global context perception module to capture the contour and mask of the occluded object; The first deocclusion layer consists of the following steps: Step 2.11, input the information (feature subgraph) extracted in step 1.3 into the first deocclusion layer. Before processing the occlusion layer, first globally process the input information, process the input image through the superpixel segmentation algorithm, decompose the image into multiple regions, and create a node for each region; then, based on the similarity of color, position, shape, etc. between the created nodes, construct a global adjacency matrix to represent the edge relationship between nodes; Then, with the help of the feature propagation capability of GCN (graph convolution), the graph is divided into two subgraphs according to the occlusion relationship of the nodes in the global graph, one is the subgraph of the occluding object, and the other is the subgraph of the occluded object; for the subgraph containing all the nodes of the occluding object and their relationship, the occlusion-aware branch is used to model the occluded object and output the contour and mask of the occluded instance. The occlusion-aware branch is designed with a simple and effective method: a 3×3 convolution layer, a deocclusion convolution layer and an FCN (two convolution layers). Since GCN has non-local properties, it can combine local features in the feature graph with global context information through global information modeling. In the feature graph input to the first deocclusion layer, each node represents a region or pixel, and the similarity and occlusion relationship between nodes are captured by constructing an adjacency matrix. With the feature propagation capability of GCN, the entire graph can be divided into two subgraphs based on the occlusion relationship between nodes: the occluding object subgraph and the occluded object subgraph.

[0027] In the first deocclusion layer, multiple deocclusion convolution processes are performed on each sub-image area to obtain the feature representation of each node and output the feature map of the occluded object sub-image; the occluded object sub-image is temporarily only used as auxiliary information, and its main function is to judge the relationship with the occluding object.

[0028] The formula for deocclusion convolution is as follows: (1) in, Indicates Layer The feature vector of each node; Indicates The neighbor set of a node; Indicates The degree of each node; Indicates The weight matrix of the layer, Indicates The bias vector of the layer; represents the activation function; Step 2.12, the feature map of the occluded object sub-image obtained in step 2.11 is simultaneously input into the MSF system and the first FAM system; Step 2.13, the MSF system processes the information as follows: First, the feature map of the occluded object sub-image is processed by a three-dimensional convolutional layer with a size of 1×1×1. Then, by adjusting the number of channels of the feature map, the feature map is divided into four sub-feature maps in the channel dimension, which are recorded as , =1, 2, 3, 4; in this way, the number of channels of the obtained sub-feature map is one quarter of the input feature map, while keeping the size of the feature map unchanged; Then, except In addition, each sub-feature map passes through a 3D convolution layer with a convolution kernel size of 3×3×3, followed by batch normalization and ReLU activation. The formula is as follows: (2) Among them, Conv3 represents the convolution operation with a convolution kernel size of 3×3×3; , =1,2,3,4, indicating feature information of 4 different scales; Next, in the parallel feature ( ) to exchange information between them, making full use of information at different scales so that each feature can be supplemented by information at other scales. The residual connection method is then used to add complementary information and enhance features at different scales. Finally, the fused features are sent to a 1×1×1 convolutional layer, which is used to mix features at different scales and solve the weakening of information flow caused by the segmentation layer; the formula is as follows: (3) in, It represents the new feature map output by the proposed multi-scale feature fusion module, and Concat represents the element-by-element concatenation operation; The information processed by the MSF module is input into the second de-occlusion layer.

[0029] Further, the second de-occlusion layer receives the image information at step 1 and the image information processed by the MSF system; The second deocclusion layer has a similar structure to the first deocclusion layer. It segments the occluded object according to the occlusion-aware features, outputs the contour and mask of the instance of the occluded object, and obtains the features of the occluded object. For each node in the input image information, the feature vectors of the occluding object sub-graph and the occluded object sub-graph are connected in sequence to form a longer feature vector; Then, the concatenated feature vector is input into a fully connected layer, and the mask and category of each node are output; the output mask indicates whether the node belongs to a certain instance, and the category indicates whether the node belongs to a certain category; The formula for deocclusion convolution is as follows: (1) in, Indicates Layer The feature vector of each node; Indicates The neighbor set of a node; Indicates The degree of each node; Indicates The weight matrix of the layer, Indicates The bias vector of the layer; Represents the activation function.

[0030] Step 3, the information processed in step 2, i.e., the feature information of the occluding object and the feature information of the occluded object, is subjected to feature association processing, i.e., cosine similarity and feature association matrix are used to aggregate the feature information according to the features of the occluding image, so as to improve the fineness of the juvenile abalone mask and the accuracy of target capture.

[0031] There are two FAM modules here. The first FAM module processes the features of occluded objects obtained after the first deocclusion layer; the second FAM system processes the features of occluded objects obtained after the second deocclusion layer.

[0032] The first FAM module processing steps are as follows: Step 3.1, the output information of the two de-occlusion layers is used as the input of two feature association modules (FAM). The input local feature V passes through CNA, which is composed of a 1×1 convolution layer, a batch normalization layer, and a ReLU activation layer, and is used to extract and process features in the image; through the 1×1 convolution layer in the CNA, the feature dimension of the output channel is adjusted to reduce the dimension of the local feature V to 1 / 4 of the original and convert it to V1, and further capture the context information by performing a 3×3 convolution operation on V1, aiming to efficiently associate similar features and reduce the number of parameters; this process produces the key feature V2; Secondly, the key feature V2 is processed using the cosine similarity operation to obtain the relationship matrix S; the matrix S reflects the set of similarities between pixels. and The similarity F is defined by the following formula: (4) in, , Two different linear exchanges representing key features; Represents the cosine of the angle between two key features; Represents the L2 norm of the vector; represents the dot product; The relationship matrix S is a set of cosine similarities between all pixel features, expressed in matrix form as: (5) in, Indicates The transpose of the key points after linear exchange; Indicates The representation after linear exchange of key points; Representation characteristics and The dot product of is used to measure their similarity; Representation characteristics and The modulus length product is used for normalization to eliminate the impact of scale on similarity measurement; Indicates and The set of cosine similarities of feature vectors; is the number of features in key features V2.

[0033] Step 3.2, then perform Softmax normalization on the similarity matrix, and then perform associated feature mapping; after dynamically learning the relationship between pixels, normalize each row in the similarity matrix, that is, apply the Softmax algorithm, and obtain the matrix S': (6); Through the product operation between elements, the matrix It is used to model the pixel relationship between key features in V2 and then map out the associated features; these features effectively integrate the comprehensive information of the target by analyzing the similarity between pixels. The calculation formula is shown as follows: (7) in, is the matrix parameter, is the associated feature map obtained by matrix multiplication; Step 3.3, finally, the image details of the associated feature V' are restored through the deconvolution step, and added element by element with the feature V1 to form a preliminary result feature; this preliminary result feature is added to the original feature V through a residual connection after channel expansion to output a refined instance feature; the formula is as follows: (8) in, is the module output feature, Conv represents the convolution process, RC represents the residual connection process, Represents element addition operation; After obtaining the occluded area of ​​the young abalone, the outline of the young abalone is obtained by using boundary tracking. Image boundary tracking mainly starts from a single boundary starting point, determines the next boundary point based on the application of the discrimination criterion, and realizes continuous tracking of the boundary, thereby forming a complete target boundary. After obtaining the outline of the young abalone, the perimeter L of the young abalone is obtained by calculating the distances between all adjacent points on the boundary and adding them up. The specific formula is as follows: (9) in, and are the coordinates of adjacent boundary points, is the total number of boundary points; the last term is to ensure the closure of the boundary by also counting the distance between the last point and the first point.

[0034] The second FAM module has a similar structure to the first FAM module, and the processing steps are as follows: In step 3.4, the output information of the occluded object obtained by the second deocclusion layer is used as the input of the second FAM module; the features of the input occluded object are used as the local features V to extract and process the features of the image through CNA. By adjusting the feature dimension of the output channel, the dimension of the local feature V is reduced to 1 / 4 of the original, and feature V1 is obtained. V1 is further convolved with 3×3 to generate the key feature V2.

[0035] Secondly, the cosine similarity operation in the first FAM module is used to calculate the similarity between the pixel features in V2 and construct the relationship matrix , which is used to reflect the similarity between all pixel features: Indicates and The cosine similarity of the eigenvectors. Perform Softmax normalization to obtain the normalized relationship matrix .

[0036] Step 3.5, using the matrix Matrix multiplication of key feature V2 completes the mapping of associated features and obtains feature Finally, the associated features are deconvolved The image details are restored, and the restored features are added element by element with V1 to form preliminary result features. After the preliminary result features are expanded through channels, they are added with the original features V through residual connection (RC) to generate the final refined instance features, which provide support for the subsequent calculation of the circumference L of the young abalone.

[0037] Combined with Figure 2-Figure 4 , 3 pictures were randomly selected from the data set to verify that the model proposed in this application can completely restore the outline of the young abalone that is blocked in each picture; Combined with Table 1, Figure 4 For example, 53 young abalones are included. Using the method proposed in this application, a total of 53 data records are extracted, each record containing the serial number of the young abalone and the corresponding circumference information, and stored in a text file.

[0038] Table 1 Figure 4 Calculation results of the circumference of 53 young abalone

[0039] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for occlusion recovery of young abalone based on a double-layer deocclusion network, characterized in that: The processing steps include: Step 1: The input image uses the ResNet101 backbone network and the feature pyramid network FPN as a feature integrator to effectively fuse and transform feature maps of different resolutions and semantic levels; then, the target is located through a single-stage target detector; Step 2: Decouple the overlapping relationship of the image information processed in step 1 using a double-layer deocclusion structure, perform multi-scale feature fusion on multiple feature map information, and segment the instances obtained by the target detection branch; Step 3, the information processed in step 2 is subjected to feature association processing, that is, cosine similarity and feature association matrix are used to aggregate feature information according to the features of the occluded image, so as to improve the fineness of the young abalone mask and the accuracy of target capture.

2. The occlusion recovery method for young abalone based on a double-layer de-occlusion network according to claim 1, characterized in that: The step 1 specifically comprises the following steps: Step 1.1, use the ResNet101 backbone network to initially extract image feature information; Step 1.2: pass the image feature information extracted in step 1.1 to FPN. FPN combines low-level features and high-level features, uses a top-down path, starts from the highest-level features extracted by the backbone network, and gradually restores higher resolutions through upsampling. The upsampled features of each layer are combined with the corresponding layer features of the backbone network through lateral connections to enhance the expressive power and receptive field of the features, predict targets at different layers, and obtain the features of the entire input image. Step 1.3, using the features of the entire input image obtained in step 1.2, a one-stage, fully convolutional FCOS is used as the target detector to directly locate and predict the candidate box of the target area of ​​interest; The RoI Align algorithm is used to accurately extract the feature sub-image of the target area of ​​interest in the feature map of the entire image according to the position of the object detection box, and it is used as the input of the two-layer de-occlusion network for the final object segmentation.

3. The occlusion recovery method for young abalone based on a double-layer de-occlusion network according to claim 1, characterized in that: The double-layer de-occlusion structure described in step 2 includes two de-occlusion layers; the first de-occlusion layer is used to capture the local structural information between pixels. By modeling the local structure, the first de-occlusion layer can effectively extract the contextual information around the target, thereby helping the network to better understand the environment and background of the target, thereby capturing the contour and mask information of the occluded object; The second deocclusion layer is designed as a global context-aware module to capture the contours and masks of occluded objects.

4. The occlusion recovery method for young abalone based on a double-layer de-occlusion network according to claim 3, characterized in that: The first de-blocking layer comprises the following steps: Step 2.11, input the information extracted in step 1.3 into the first deocclusion layer; before processing the occlusion layer, first globally process the input information, process the input image through the superpixel segmentation algorithm, decompose the image into multiple regions, and create a node for each region; then, according to the similarity of color, position, and shape between the created nodes, construct a global adjacency matrix to represent the edge relationship between the nodes; Next, according to the occlusion relationship of the nodes in the global graph, the graph is divided into two subgraphs, one is the subgraph of occluding objects, and the other is the subgraph of occluded objects. For the subgraph containing all the nodes of occluding objects and their relationships, the occlusion-aware branch is used to model the occluded objects and output the contour and mask of the occluded instance. The occlusion-aware branch is designed with a simple and effective method: it consists of a 3×3 convolution layer, a deocclusion convolution layer and an FCN. In the first deocclusion layer, multiple deocclusion convolutions are performed on each sub-image area to obtain the feature representation of each node and output the feature map of the occluded object sub-image; The occluded object sub-image is temporarily used only as auxiliary information, and its main function is to judge the relationship with the occluding object; The formula for deocclusion convolution is as follows: (1)， in, Indicates Layer The feature vector of each node; Indicates The neighbor set of a node; Indicates The degree of each node; Indicates The degree of each node; Indicates The weight matrix of the layer, Indicates The bias vector of the layer; represents the activation function; Step 2.12, input the occluded object features obtained in step 2.11 into the first FAM system and the MSF system at the same time; Step 2.13, the MSF system processes the information as follows: First, the feature map of the occluded object is processed by a three-dimensional convolutional layer with a size of 1×1×1. Then, the feature map is divided into four sub-feature maps in the channel dimension by adjusting the number of channels of the feature map, which are recorded as , =1, 2, 3, 4; in this way, the number of channels of the obtained sub-feature map is one quarter of the input feature map, while keeping the size of the feature map unchanged; Then, except In addition, each sub-feature map passes through a 3D convolution layer with a convolution kernel size of 3×3×3, followed by batch normalization and ReLU activation. The formula is as follows: (2)， Among them, Conv3 represents the convolution operation with a convolution kernel size of 3×3×3; , =1,2,3,4, indicating feature information of 4 different scales; Next, in the parallel feature The information is exchanged between the two scales, making full use of information at different scales so that each feature can be supplemented by information at other scales. The residual connection is then used to add complementary information and enhance features at different scales. Finally, the fused features are sent to a 1×1×1 convolutional layer, which is used to mix features at different scales and solve the weakening of information flow caused by the segmentation layer. The formula is as follows: (3)， in, represents the output feature map of the proposed multi-scale feature fusion module, and Concat represents the element-by-element concatenation operation; The information processed by the MSF module is input into the second de-occlusion layer.

5. The method for recovering young abalone from occlusion based on a double-layer de-occlusion network according to claim 4, characterized in that: The second de-occlusion layer receives the image information at step 1 and the image information processed by the MSF system; The second deocclusion layer has a similar structure to the first deocclusion layer. It segments the occluded objects according to the occlusion-aware features, outputs the contours and masks of the instances of the occluded objects, and obtains the features of the occluded objects. For each node in the input image information, connect the feature vectors of the occluding object sub-image and the occluded object sub-image in order to form a longer feature vector; Then, the concatenated feature vector is input into a fully connected layer to output the mask and category of each node; the output mask indicates whether the node belongs to a certain instance, and the category indicates that the node belongs to a certain category.

6. The method for recovering young abalone from occlusion based on a double-layer de-occlusion network according to claim 5, characterized in that: The step 3 is two FAM systems, the first FAM module processes the features of the occluded object obtained through the first deocclusion layer; The second FAM system processes the features of occluded objects obtained after the second deocclusion layer.

7. The method for recovering young abalone from occlusion based on a double-layer de-occlusion network according to claim 6, characterized in that: The FAM module processing steps are as follows: Step 3.1, the output information of the two de-occlusion layers is used as the input of the two feature association modules. The input local feature V passes through CNA, which is composed of a 1×1 convolution layer, a batch normalization layer, and a ReLU activation layer, and is used to extract and process features in the image; through the 1×1 convolution layer in the CNA, the feature dimension of the output channel is adjusted to reduce the dimension of the local feature V to 1 / 4 of the original and convert it to V1, and further capture the context information by performing a 3×3 convolution operation on V1, aiming to efficiently associate similar features and reduce the number of parameters; this process produces the key feature V2; Secondly, the key feature V2 is processed using the cosine similarity operation to obtain the relationship matrix S; The matrix S reflects the set of similarities between pixels. and The similarity F is defined by the following formula: (4)， in, , Two different linear exchanges representing key features; Represents the cosine of the angle between two key features; Represents the L2 norm of the vector; represents the dot product; The relationship matrix S is a set of cosine similarities between all pixel features, expressed in matrix form as: (5)， in, Indicates The transpose of the key points after linear exchange; Indicates The representation after linear exchange of key points; Representation characteristics and The dot product of is used to measure their similarity; Representation characteristics and The modulus length product is used for normalization to eliminate the impact of scale on similarity measurement; Indicates and The set of cosine similarities of feature vectors; is the number of features in key feature V2; Step 3.2, then perform Softmax normalization on the similarity matrix, and then perform associated feature mapping; after dynamically learning the relationship between pixels, normalize each row in the similarity matrix, that is, apply the Softmax algorithm, and obtain the matrix S': (6)； in, Represents the cosine of the angle between two key features; It is the abbreviation of the English word begin, which means starting from 1. The range of 1 to 49 here is the total number of pixels in the feature map, which means that the input feature map is 7×7 at the current scale. Through the product operation between elements, the matrix It is used to model the pixel relationship between key features in V2 and then map out the associated features; these features effectively integrate the comprehensive information of the target by analyzing the similarity between pixels. The calculation formula is shown as follows: (7)， in, is the matrix parameter, is the associated feature map obtained by matrix multiplication; For the key features mentioned above; Step 3.3, finally, the image details of the associated feature V' are restored through the deconvolution step, and added element by element with the feature V1 to form a preliminary result feature; this preliminary result feature is added to the original feature V through a residual connection after channel expansion to output a refined instance feature; the formula is as follows: (8)， in, is the module output feature, Conv represents the convolution process, RC represents the residual connection process, Represents element addition operation; After obtaining the occluded area of ​​the young abalone, the outline of the young abalone is obtained by using boundary tracking. Image boundary tracking mainly starts from a single boundary starting point, determines the next boundary point based on the application of the discrimination criterion, and realizes continuous tracking of the boundary, thereby forming a complete target boundary. After obtaining the outline of the young abalone, the perimeter L of the young abalone is obtained by calculating the distances between all adjacent points on the boundary and adding them up. The specific formula is as follows: (9)， in, and are the coordinates of adjacent boundary points, is the total number of boundary points; the last term is to ensure the closure of the boundary by also counting the distance between the last point and the first point.

8. A juvenile abalone occlusion recovery system based on a double-layer de-occlusion network, characterized in that: The system includes a backbone network, a double-layer de-occlusion network and a FAM network; The backbone network includes a ResNet101 backbone network, a feature pyramid network FPN module and a single-stage target detector module; the ResNet101 backbone network module and the feature pyramid network FPN module are used as feature integrators to effectively fuse and convert feature maps of different resolutions and semantic levels; the single-stage target detector module is used to locate the target; The double-layer de-occlusion network includes a first de-occlusion module and a second de-occlusion module; A double-layer deocclusion structure is used to decouple overlapping relationships, and multi-scale feature fusion is performed on multiple feature map information, and the instances obtained by the target detection branch are segmented; The FAM network includes a first FAM module and a second FAM module; the FAM module includes a cosine similarity module and a feature association matrix module to perform feature association processing, that is, cosine similarity and feature association matrix are used to aggregate feature information according to the features of the occluded image, so as to improve the fineness of the juvenile abalone mask and the accuracy of target capture.

9. The juvenile abalone occlusion recovery system based on a double-layer de-occlusion network according to claim 8, characterized in that: The first de-occlusion module includes a first de-occlusion layer module, a regression occlusion object information module and an MSF module; the MSF module includes a partition feature map module and a parallel feature map information exchange module.

10. The juvenile abalone occlusion recovery system based on a double-layer de-occlusion network according to claim 9, characterized in that: The second de-occlusion module includes a second de-occlusion layer module and a module for regressing information of an occluded object.

Images