A mathematical function image instance segmentation method based on an improved Mask-R-CNN network

Abstract

The application discloses a mathematical function image instance segmentation method based on an improved Mask-R-CNN network, and comprises the following steps: taking ConvNeXt as the basis to obtain an improved feature extraction network MA ConvNeXt and perform feature extraction on a function image; taking BiFPN as the basis to add RFA and ASPP modules to obtain an improved feature fusion network MA BiFPN; obtaining multi-scale fusion features and inputting the multi-scale fusion features to an improved RPN to generate candidate regions; outputting the same size features through RoIAlign, and then performing regression classification to generate a detection frame and a category; replacing the mask segmentation in the Mask-R-CNN with a PointHead; and inputting the fine features of the improved feature fusion network and the output features of the RoIAlign into the PointHead to generate a mask. The application can effectively realize the recognition and segmentation of slender function images.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a mathematical function image instance segmentation method based on an improved Mask-R-CNN network. Background Technology

[0002] With the rapid development of science and technology, the level of public concern for humanity is also gradually increasing. For visually impaired individuals, due to their visual impairments, they primarily rely on hearing and touch to obtain information from the outside world. For example, some textual or verbal information can be accessed through audio, while visual information can be converted into tactile images for their understanding. However, the number of reading materials available to visually impaired individuals is currently limited, and tactile images of function graphs are extremely rare. This is partly due to the high production cost of reading materials for the visually impaired, and partly because the creation of tactile images requires digitization, but the digitization efficiency of function tactile images is low. Therefore, how to more effectively recognize, segment, and convert function images into electronic form is the key to the creation of function tactile images.

[0003] Currently, deep learning, with its excellent feature learning capabilities, is being applied more and more widely in various fields. Instance segmentation is one of the important tasks in deep learning. Among them, improving the recognition ability of deep learning networks in object detection and the quality of object segmentation results has become the key to improving instance segmentation performance.

[0004] In recent years, many excellent detection and segmentation methods have been proposed for instance segmentation tasks. Traditional pattern recognition methods are effective for detecting and segmenting target function images, but limitations such as insufficient feature learning capabilities prevent them from meeting speed and accuracy requirements. Machine learning-based target recognition methods have played a pioneering role in instance segmentation, but machine learning methods are insufficient for solving highly complex problems. Therefore, deep learning networks have emerged. Many excellent instance segmentation networks proposed in deep learning, such as Mask-R-CNN, PANET, and Blend Mask, possess powerful feature learning capabilities for most entity objects.

[0005] However, existing traditional instance segmentation networks perform poorly on special samples such as function images, which are diverse, elongated, and complex. This is mainly manifested in two ways: in object recognition, they fail to find all function images and misclassify them; in object segmentation, the generated masks are sparse, with poor instance mask integrity and severely jagged mask edges, resulting in poor mask quality. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a mathematical function image instance segmentation method based on an improved Mask-R-CNN network. This method enhances the feature extraction capability of the original Mask-R-CNN network by replacing the feature extraction network, enabling the extracted features to contain local, global, and channel information. It also strengthens the multi-scale detection capability by replacing the feature fusion network in the original Mask-R-CNN network, further enhancing the acquisition of local and global feature information. Furthermore, it adjusts the aspect ratio of candidate anchor boxes in the original Mask-R-CNN's region proposal network. The mask generation strategy of Mask-R-CNN is replaced, allowing the segmentation network to obtain higher-quality masks. Through these improvements, the final modified instance segmentation network can be better applied to the detection and segmentation of function images.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution.

[0008] A mathematical function-based image instance segmentation method based on an improved Mask-R-CNN network includes the following steps:

[0009] S1. Based on ConvNeXt, an improved feature extraction network MA ConvNeXt is obtained; an Attention module is added to the improved feature extraction block MA ConvNeXt; features are extracted from the function image using the improved MA ConvNeXt:

[0010] S2. Based on BiFPN, an RFA module and an ASPP module are added to obtain the improved feature fusion network MABiFPN; including: S1, S2, S3, and S4, which represent the improved feature extraction network MA ConvNeXt with three stacks and a Dim dimension of 96, three stacks and a Dim dimension of 192, nine stacks and a Dim dimension of 384, and three stacks and a Dim dimension of 768, respectively. Features extracted by each Block; P3 represents the feature output of the feature fusion stage with the same output feature size as S3 in the top-down feature fusion path of MABiFPN; P2 represents the feature output of the feature fusion stage with the same output feature size as S2 in the top-down feature fusion path of MABiFPN; N1 represents the feature output of the feature fusion stage with the same output feature size as S1 in the bottom-up feature fusion path of MABiFPN; N2 represents the feature output of the feature fusion stage with the same output feature size as S2 in the bottom-up feature fusion path of MABiFPN; N3 represents the feature output of the feature fusion stage with the same output feature size as S3 in the bottom-up feature fusion path of MABiFPN; N4 represents the feature output of the feature fusion stage with the same output feature size as S4 in the bottom-up feature fusion path of MABiFPN.

[0011] S3. After passing through the MABiFPN feature fusion network, four multi-scale fused features N1, N2, N3, and N4 are obtained. Then, the four features N1, N2, N3, and N4 are input into the improved RPN to classify the foreground and background, and regression operations are performed on the bounding boxes to generate candidate boxes. After non-maximum suppression to remove low-confidence anchor boxes, the candidate regions are finally generated. The candidate regions are then put into RoIAlign to output features of the same size, which are then fed into the fully connected layer for feature regression classification, thereby generating the final detection boxes and categories.

[0012] S4. Replace the mask segmentation strategy in the original Mask-R-CNN with PointHead instead of fully connected layers: extract fine features from the features output by the improved feature fusion network MA BiFPN, and then input the fine features together with the features output by RoIAlign into PointHead to generate the final mask.

[0013] Further, in step S1, the process of extracting features from the function image using the improved MA ConvNeXt includes:

[0014] In the MA ConvNeXt network, after the image input, a 4×4 convolution kernel with a stride of 4 is first used to perform a convolution operation to output the initial image features. Then, layer normalization is applied to the initial features to refine them and improve their data distribution. After this processing, the features are input into the MA ConvNeXt Block and subjected to Dw Conv depthwise convolution for attention-based multi-scale feature extraction. Pw Conv weights and combines the input individual features along the depth direction to generate new features. These features are then activated by the GELU activation function to add a non-linear factor, and the processed features are input into the Attention module. After each MA ConvNeXt Block is stacked, it is downsampled and input into the next MA ConvNeXt Block stack. The final outputs are: MA ConvNeXtBlock stacked 3 times with a Dim dimension of 96, MA ConvNeXt Block stacked 3 times with a Dim dimension of 192, MA ConvNeXtBlock stacked 9 times with a Dim dimension of 384, and MA ConvNeXtBlock stacked 3 times with a Dim dimension of 768. The features extracted in the four MA ConvNeXt feature extraction stages of Block.

[0015] Furthermore, in step S1, the process of adding an Attention module to the improved feature extraction block MA ConvNeXt Block includes:

[0016] The features, after being activated by GELU and introducing nonlinear factors, are input into Asym Conv. Asym Conv with kernel sizes of 3×1 and 1×3 is used to replace the ordinary 3×3 convolution. Batch normalization is performed separately before feature fusion in the 3×1 and 1×3 Asym Conv branches. Then, the features are processed by a large kernel Dw Conv with a kernel size of 7×7 and an inflation rate of 3, and a kernel Pw Conv with a kernel size of 1×1. Finally, the output of the above steps is multiplied by the features input to the Attention module, which serves as the weight of the features input to the Attention module. DropOut and DropPath layers are added at the end of the Attention module. The Attention module can be expressed by formulas (1) and (2).

[0017]

[0018]

[0019] in Represents attention, Conv 1×1Dw Conv represents a 2D convolution with a kernel size of 1×1, and Asym Conv represents a depthwise convolution. 3×1 This represents an asymmetric convolution with a kernel size of 3×1. Representing input features, Asym Conv 1×3 This represents an asymmetric convolution with a kernel size of 1×3. Adding representative features This indicates multiplication of the two parts.

[0020] Furthermore, in step S2, the process of obtaining the improved feature fusion network MABiFPN based on BiFPN includes:

[0021] S1, S2, S3, and S4 are input into the improved feature fusion network MABiFPN, and feature multi-scale fusion is performed in three stacks. In each stack, S4 is fused with the features of S3 after passing through the RFA module to output P3. S2 is fused with the upsampled features of P3 to output P2. P2 is upsampled and fused with the features of S1 and passed through the ASPP module to output N1. N1 is max-pooled and fused with the features of P2 after passing through the ASPP module and S2 in the feature extraction network to output N2. N2 is max-pooled and fused with the features of P3 after passing through the ASPP module and S3 in the feature extraction network to output N3. N3 is max-pooled and fused with S4 after passing through the RFA and ASPP modules to output N4.

[0022] Further, in step S2, the path in the original BiFPN from feature S4 of the original feature extraction network to P3 and N4 is replaced with the RFA module, specifically including:

[0023] The RFA module uses ratio-invariant adaptive pooling on S4 to generate multi-scale features with sizes of 0.1, 0.3, and 0.5 times that of the input features. For these multi-scale features with sizes of 0.1, 0.3, and 0.5 times that of the input features, a 1×1 convolution is used to generate output features with 256 channels. Then, bilinear interpolation is used to upsample the output features after the 1×1 convolution to the same scale. Finally, an adaptive spatial fusion module adaptively combines the context features after bilinear interpolation to reduce the aliasing effect caused by interpolation. The features output from S4 by the RFA module possess multi-scale contextual information. The newly output features are combined with S3 and propagated to the P3 features for fusion. The adaptive spatial fusion module assigns weights to each feature to aggregate these context features into new features. The weighted feature fusion calculation method is as follows:

[0024]

[0025] in, Let represent the feature vector of layer l at position (i,j). This represents the learnable weights, or contributions, of the other three layers on the feature map. This represents the feature vector generated at (i,j) in layer l after scaling the features from layer n.

[0026] Furthermore, in step S2, the ASPP module is used in the improved feature fusion network MA BiFPN, specifically including:

[0027] The input features are processed through 2D convolutions with a kernel size of 1×1 and a dilation rate of 1, 2D convolutions with a kernel size of 3×3 and a dilation rate of 3, and 2D convolutions with a kernel size of 3×3 and a dilation rate of 6. After adaptive pooling, they are further processed through 2D convolutions with a kernel size of 1×1 and a dilation rate of 1, and then upsampled to obtain the corresponding scale features. Finally, these scale features are merged. After N2 max pooling, the features are fused with the P3 features from the ASPP module and the S3 features from the improved feature extraction network MA ConvNeXt to output N3. The improved MA BiFPN performs bottom-up feature fusion at layer N3 as follows:

[0028]

[0029] Among them, W i All are learnable parameters, where e is the learning rate. This represents the input to layer N3. This represents the intermediate features of N3 on the top-down path. This means scaling the output features of the previous layer N2 to the same size as the features of N3 using max pooling. This represents the final output feature of layer N3.

[0030] Furthermore, in step S3, the candidate box of the improved RPN is increased by 5%, and a penalty factor λ(G2-B1∩G2) is added to the IoU to penalize candidate boxes that do not completely cover the target region. The improved IoU calculation formula is:

[0031]

[0032] Where λ represents the penalty intensity, and G2-B1∩G2 represents the spatial difference between the predicted bounding box and the true bounding box.

[0033] Further, in step S4, the replacement of the mask segmentation strategy in the original Mask-R-CNN from the fully connected layer to PointHead includes: in PointHead, the features input from RoIAlign are first downsampled through multiple pooling operations and then coarsely predicted. Then, difficult points with poor segmentation performance are selected from the coarse segmentation features. Then, a simple multilayer perceptron network is trained on these points by combining coarse and fine features to re-predict these difficult points. The re-predicted results replace the original coarse prediction results, and finally, a function graph mask is generated.

[0034] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0035] 1. This invention replaces the feature extraction network in the original Mask-R-CNN network by using ConvNeXt as the base network and adding an Attention module. This improves the ability of the modified feature extraction network MA ConvNeXt to extract local and global information as well as features from different channels, enhances the network's feature learning ability for function images, and avoids the shortcomings of self-attention mechanism and large kernel convolution.

[0036] 2. By replacing the feature fusion network in the original Mask-R-CNN network: Based on BiFPN (Bi-directional Feature Pyramid Network), ASPP (Atrous Spatial Pyramid Pooling) and RFA (Residual Feature Augmentation) modules are added to reduce information loss in different feature layers and further enhance the detection capability of the modified network at various scales.

[0037] 3. The aspect ratio of the candidate anchor boxes in the original Mask-R-CNN's RPN (Region Proposal Network) is adjusted, and a penalty factor is added to the IoU (Intersection over Union) in the RPN layer. This allows the object detection boxes to retain a certain amount of flexible background space compared to the original Mask-R-CNN's RPN network, reducing information loss, optimizing the generation of proposed regions, and thus enhancing the network's segmentation ability.

[0038] 4. Replace the mask generation strategy in the original Mask-R-CNN network: Replace the fully connected layers with PointHead. PointHead directly predicts coarse results after multiple pooling downsampling. Then, a multilayer perceptron network is trained by combining fine features from feature fusion. During training, only difficult points where the prediction differs from the actual result are re-predicted and the original coarse result is replaced. This reduces the information loss caused by upsampling in the fully connected layers and the jagged edges of the mask, resulting in a smoother and more complete mask.

[0039] 5. After the above modifications, the improved instance segmentation network of this invention has a stronger feature learning ability, reduces information loss, and the learned features contain more local, global, and channel information. It optimizes the generation of proposed regions, changes the mask generation strategy, and ultimately can be better applied to the detection and segmentation of function images, achieving satisfactory results. It solves the technical problems in existing methods for instance segmentation of long and thin function images: inability to find the entire function image, misclassification of the function image, sparse generated masks, poor instance mask integrity, and severe jagged mask edges. Attached Figure Description

[0040] Figure 1 This is a flowchart of a mathematical function image instance segmentation method based on an improved Mask-R-CNN network according to the present invention.

[0041] Figure 2 This is a schematic diagram of the structure of the improved feature extraction network MA ConvNeXt according to an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the structure of the MA ConvNeXt Block according to an embodiment of the present invention.

[0043] Figure 4 This is a schematic diagram of the Attention module structure according to an embodiment of the present invention.

[0044] Figure 5 This is a schematic diagram of the improved feature fusion network MA BiFPN structure according to an embodiment of the present invention.

[0045] Figure 6 This is a schematic diagram of the RFA module structure according to an embodiment of the present invention.

[0046] Figure 7 This is a schematic diagram of the ASPP module structure according to an embodiment of the present invention.

[0047] Figure 8 This is a schematic diagram of the PointHead module structure according to an embodiment of the present invention. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to the accompanying drawings.

[0049] like Figure 1 As shown, the present invention provides a mathematical function image instance segmentation method based on an improved Mask-R-CNN network for recognizing and segmenting elongated function images, comprising the following steps:

[0050] S1: Based on ConvNeXt, an improved feature extraction network MA ConvNeXt is obtained; features are extracted from the function image using the improved MA ConvNeXt. For example... Figure 2 As shown in the MA ConvNeXt network structure, after the image is input, a 4×4 convolution kernel with a stride of 4 is first used to perform a convolution operation to output the initial image features. Then, the initial features are normalized to refine them and improve the distribution of the initial feature data. After the above processing, the features are input into the MA ConvNeXt Block part, where attentional multi-scale feature extraction is performed. After each MA ConvNeXt Block is stacked, it is downsampled and input into the next MA ConvNeXt Block stack. Finally, the output is the features extracted in the feature extraction stages of the four MA ConvNeXt blocks: MA ConvNeXt Block stacked 3 times with a Dim (dimension) of 96, MA ConvNeXt Block stacked 3 times with a Dim of 192, MA ConvNeXt Block stacked 9 times with a Dim of 384, and MA ConvNeXt Block stacked 3 times with a Dim of 768. The MA ConvNeXt Block structure will be explained in S1.1, and the Attention module structure in the MA ConvNeXt Block will be explained in S1.2.

[0051] S1.1: Based on the ConvNeXt Block, an improved feature extraction block, MA ConvNeXt Block, is obtained. MAConvNeXt Block is the core part of the improved feature extraction network MA ConvNeXt, responsible for extracting features from mathematical function images more deeply. The overall structure of MA ConvNeXt Block is as follows: Figure 3As shown (where K represents the kernel size, and the number after the kernel represents the output dimension), when features are input to the MAConvNeXt Block, a Dw Conv (Depth-wise Convolution) is first performed on the input features. This effectively utilizes the effective information from different layers at the same spatial location. Then, after layer normalization, the features are input to Pw Conv (Point-wise Convolution), which weights and combines the individual input features along the depth direction to generate new features. This combination of Dw Conv and Pw Conv significantly reduces computation and the number of parameters, allowing for a deeper network with the same parameters. Subsequently, the features are activated by the GELU (Gaussian Error Linear Unit) activation function to add a non-linear factor, and the processed features are then input to the Attention module.

[0052] S1.2: Add an Attention module to the improved feature extraction block MA ConvNeXt Block. For example... Figure 4As shown (where K represents the kernel size, P represents padding, and G represents the number of groups), the features, after being activated by GELU and introducing nonlinearity, are first input into Asym Conv (Asymmetric Convolution). Instead of a standard 3×3 convolution, Asym Convs with kernel sizes of 3×1 and 1×3 are used to perform convolutions on the features. This method reduces the computational cost of the model and enhances feature extraction for elongated objects such as mathematical function graphs, while also compensating for the information loss caused by the subsequent Dw Conv dilation of 3. Subsequently, batch normalization is performed separately before feature fusion in the 3×1 and 1×3 Asym Conv branches to prevent the network from uniformly normalizing features and weakening the characteristics between different convolutions. Then, the features are processed through a large kernel Dw Conv with a kernel size of 7×7 and a dilation of 3, and a Pw Conv with a kernel size of 1×1 to obtain long-range information and a stronger receptive field. Finally, the output of the above steps is multiplied by the features input to the Attention module, and this multiplication is used as the weight of the features input to the Attention module, thus achieving the function of attention. To prevent network overfitting and improve the network's generalization ability, DropOut and DropPath layers are added at the end of the Attention module. The functions of DropOut and DropPath are similar, where DropOut causes neurons to inactivate with a certain probability, while DropPath causes the main branch structure to inactivate with a certain probability.

[0053] In traditional methods, local information can be obtained through small convolutional kernels, while long-range information can only be obtained through large kernel convolutions except for self-attention mechanisms. This invention, through its Attention module, can obtain both local attention information and good long-range and channel information, thereby enhancing the network's ability to learn image features and avoiding the drawbacks of both self-attention mechanisms and large kernel convolutions. The Attention module can be expressed by formulas (1) and (2):

[0054]

[0055]

[0056] in, Represents attention, Conv 1×1 Dw Conv represents a 2D convolution with a kernel size of 1×1, and Asym Conv represents a depthwise convolution. 3×1 This represents an asymmetric convolution with a kernel size of 3×1. Representing input features, Asym Conv 1×3This represents an asymmetric convolution with a kernel size of 1×3. Adding representative features This indicates multiplication of the two parts.

[0057] S2: Based on BiFPN, an improved feature fusion network, MA BiFPN, is obtained; MA BiFPN is specifically as follows: Figure 5 As shown, S1, S2, S3, and S4 represent the improved MA ConvNeXt feature extraction network MA ConvNeXt with the following stacks: 3 stacks with a Dim (dimension) of 96, 3 stacks with a Dim of 192, 9 stacks with a Dim of 384, and 3 stacks with a Dim of 768, respectively. Features extracted by each Block; P3 represents the feature output of the feature fusion stage with the same output feature size as S3 in the top-down feature fusion path of MABiFPN; P2 represents the feature output of the feature fusion stage with the same output feature size as S2 in the top-down feature fusion path of MABiFPN; N1 represents the feature output of the feature fusion stage with the same output feature size as S1 in the bottom-up feature fusion path of MABiFPN; N2 represents the feature output of the feature fusion stage with the same output feature size as S2 in the bottom-up feature fusion path of MABiFPN; N3 represents the feature output of the feature fusion stage with the same output feature size as S3 in the bottom-up feature fusion path of MABiFPN; N4 represents the feature output of the feature fusion stage with the same output feature size as S4 in the bottom-up feature fusion path of MABiFPN. S1, S2, S3, and S4 are input into the improved feature fusion network MABiFPN, and feature multi-scale fusion is performed in three stacks. In each stack, S4 is fused with the features of S3 after passing through the RFA module to output P3. S2 is fused with the upsampled features of P3 to output P2. P2 is upsampled and fused with the features of S1 and passed through the ASPP module to output N1. N1 is max-pooled and fused with the features of P2 after passing through the ASPP module and S2 in the feature extraction network to output N2. N2 is max-pooled and fused with the features of P3 after passing through the ASPP module and S3 in the feature extraction network to output N3. N3 is max-pooled and fused with S4 after passing through the RFA and ASPP modules to output N4.

[0058] The original Mask-R-CNN uses FPN (Feature Pyramid Network) as the feature fusion network. FPN has the following defects in information fusion: (1) Information loss during the fusion of features at adjacent scales: semantic information is lost due to the decrease in the number of channels during the fusion process from high to low levels. (2) Due to its top-down or bottom-up structure, FPN focuses more on the features of adjacent layers, but high-level features cannot be directly passed to low-level layers and must pass through multiple layers, which leads to the loss of high-level semantic information. (3) Inconsistent feature information at different scales. This invention solves this problem by adding RFA and ASPP modules to BiFPN to obtain MA BiFPN. The structure of RFA module will be explained in S2.1 and the structure of ASPP module will be explained in S2.2.

[0059] S2.1: The improved feature fusion network MA BiFPN uses the RFA module. The structure of the RFA module is as follows: Figure 6 As shown, the RFA module uses ratio-invariant adaptive pooling on S4 to generate multi-scale features with sizes of 0.1, 0.3, and 0.5 times that of the input feature. Then, 1×1 convolutions are used on these multi-scale features (0.1, 0.3, and 0.5 times the input feature size) to generate output features with 256 channels. Subsequently, bilinear interpolation is used to upsample the output features after the 1×1 convolution to the same scale. Finally, an adaptive spatial fusion module adaptively combines the context features after bilinear interpolation to reduce aliasing caused by interpolation. The features output from S4 by the RFA module possess multi-scale contextual information. The newly output features are combined with S3 and propagated to the P3 features for fusion.

[0060]

[0061] The adaptive spatial fusion module assigns weights to each feature, thereby aggregating these contextual features into new features. Equation (3) shows the weighted feature fusion calculation method of the adaptive spatial fusion module, where... Let represent the feature vector of layer l at position (i,j). This represents the learnable weights (i.e., contributions) of the other three layers on the feature map. This represents the feature vector generated at (i,j) in layer l after scaling the features from layer n.

[0062] This invention replaces the path in the original BiFPN from feature S4 of the original feature extraction network to P3 and N4 with the RFA module, which avoids information loss caused by the reduction of the number of channels when the highest-level feature is passed to the feature fusion network. At the same time, it can enhance the context information that the highest-level feature information is compatible with other layers.

[0063] S2.2: The ASPP module is used in the improved feature fusion network MA BiFPN. The specific structure of the ASPP module is as follows: Figure 7 As shown, Rate represents the dilation rate. Input features are processed through 2D convolutions with a kernel size of 1×1 and a dilation rate of 1, 2D convolutions with a kernel size of 3×3 and a dilation rate of 3, and 2D convolutions with a kernel size of 3×3 and a dilation rate of 6. After adaptive pooling, they are further processed through 2D convolutions with a kernel size of 1×1 and a dilation rate of 1, and upsampled to obtain the corresponding scale features. Finally, these scale features are merged. ASPP enhances the receptive field of the fused features and also improves the ability to extract overall information, which is beneficial for improving the performance of each branch of the feature fusion network. Taking the N3 output feature as an example: N2, after max pooling, is fused with the P3 feature after ASPP and the S3 feature in the improved feature extraction network MAConvNeXt to output N3.

[0064]

[0065] Equation (4) gives the bottom-up feature fusion of the improved MA BiFPN at layer N3, where W i All are learnable parameters, where e is the learning rate. This represents the input to layer N3. This represents the intermediate features of N3 on the top-down path. This means scaling the output features of the layer above N2 to the same size as the features of N3 (max pooling). This represents the final output feature of layer N3.

[0066] S3: Based on the Region Proposal Network (RPN), an improved RPN is obtained. After passing through the MABiFPN feature fusion network, four features N1, N2, N3, and N4 are obtained. These four features are then input into the improved RPN to classify foreground and background, and regression operations are performed on the bounding boxes to generate candidate boxes. After non-maximum suppression to remove low-confidence anchor boxes, candidate regions are finally generated. These candidate regions are then fed into RoIAlign (Region of Interest Alignment) to output features of equal size, and then into a fully connected layer for feature regression classification, thereby generating the final detection boxes and categories. The improved RPN leaves some flexibility in the selected candidate boxes, providing better candidate regions for subsequent segmentation. The original RPN's bounding box regression is a coarse result, which can easily lead to information loss for segmentation tasks. The original RPN determines candidate boxes based on the IoU (Intersection over Union) between the bounding boxes and candidate boxes (intersection over union of the predicted box and the ground truth bounding box in object detection, and intersection over union of the predicted mask and the ground truth mask in segmentation). The IoU is calculated using formula (5):

[0067]

[0068] In formula (5), B1 is the candidate box and G2 is the ground truth bounding box. It can be seen from the formula that if the candidate box is too large, the IoU value will be too small, leading to its rejection. However, a candidate box that is slightly larger than the ground truth bounding box can more comprehensively encompass the segmented object, retaining more background area and reducing information loss that might result from candidate boxes being discarded due to being too close to the edge. The improved RPN of this invention increases the selection of candidate boxes by 5% and adds a penalty factor λ(G2-B1∩G2) to the IoU to penalize cases where the candidate box does not completely cover the target area. The improved IoU calculation formula is as follows (6):

[0069]

[0070] Where λ represents the penalty intensity, and G2-B1∩Gx represents the spatial difference between the predicted bounding box and the true bounding box.

[0071] S4: Replace the mask segmentation strategy in the original Mask-R-CNN with PointHead instead of fully connected layers. Extract fine-grained features from the output of the improved feature fusion network MA BiFPN, and then input these fine-grained features along with the features output by RoIAlign into PointHead. The specific structure of PointHead is as follows: Figure 8Where Kernel is the kernel size and Stride is the stride. In PointHead, the features input from RoIAlign are first downsampled through multiple pooling operations and then used for coarse prediction. Next, the most difficult segments are selected from the coarse segmentation features, and a simple multilayer perceptron network is trained on these points using both coarse and fine features to re-predict these difficult segments. The re-predicted results replace the original coarse predictions, ultimately generating a function graph mask. This method reduces the information loss caused by continuous upsampling in the original Mask-R-CNN mask generation strategy. Furthermore, because it primarily targets difficult segments with fewer pixels compared to the entire mask, PointHead not only achieves accurate segmentation and smooth segmentation edges but also significantly saves computational resources.

Claims

1. A mathematical function image instance segmentation method based on an improved Mask-R-CNN network, characterized in that, Includes the following steps: S1. Based on ConvNeXt, an improved feature extraction network MAConvNeXt is obtained; an Attention module is added to the improved feature extraction block MAConvNeXt; features are extracted from the function image using the improved MAConvNeXt: S2. Based on BiFPN, an RFA module and an ASPP module are added to obtain the improved feature fusion network MA BiFPN. This includes: S1, S2, S3, and S4, representing the features extracted by the following blocks in the improved feature extraction network MA ConvNeXt: a stacked block with 3 layers and a Dim dimension of 96, a stacked block with 3 layers and a Dim dimension of 192, a stacked block with 9 layers and a Dim dimension of 384, and a stacked block with 3 layers and a Dim dimension of 768, respectively; P3 represents the feature output of the feature fusion stage in the top-down feature fusion path of MA BiFPN, where the output feature size is the same as S3; and P2 represents the feature output of the feature fusion stage in the top-down feature fusion path of MA BiFPN, where the output feature size is the same as S2. N1 represents the feature output by the feature fusion stage in the MA BiFPN bottom-up feature fusion path with the same output feature size as S1; N2 represents the feature output by the feature fusion stage in the MA BiFPN bottom-up feature fusion path with the same output feature size as S2; N3 represents the feature output by the feature fusion stage in the MA BiFPN bottom-up feature fusion path with the same output feature size as S3; and N4 represents the feature output by the feature fusion stage in the MA BiFPN bottom-up feature fusion path with the same output feature size as S4. S3. After passing through the MA BiFPN feature fusion network, four multi-scale fused features N1, N2, N3, and N4 are obtained. Then, the four features N1, N2, N3, and N4 are input into the improved RPN to classify the foreground and background, and regression operations are performed on the bounding boxes to generate candidate boxes. After non-maximum suppression to remove low-confidence anchor boxes, the candidate regions are finally generated. The candidate regions are then put into RoIAlign to output features of the same size, which are then fed into the fully connected layer for feature regression classification, thereby generating the final detection boxes and categories. S4. Replace the mask segmentation strategy in the original Mask-R-CNN with PointHead instead of fully connected layers: extract fine features from the features output by the improved feature fusion network MA BiFPN, and then input the fine features together with the features output by RoIAlign into PointHead to generate the final mask.

2. The method for segmenting mathematical function images based on an improved Mask-R-CNN network according to claim 1, characterized in that, In step S1, the process of extracting features from the function image using the improved MA ConvNeXt includes: In the MAConvNeXt network, after image input, a 4×4 convolution kernel with a stride of 4 is first used to output initial image features. Then, layer normalization is performed on these initial features to refine them and improve their distribution. After this processing, the features are input into the MAConvNeXt Block and subjected to Dw Conv depthwise convolution for attention-based multi-scale feature extraction. Pw Conv weights and combines the individual input features along the depth direction to generate new features. These features are then activated by the GELU activation function to add non-linearity, and the processed features are input into the Attention module. After each MAConvNeXt Block is stacked, it is downsampled and input into the next MAConvNeXt Block stack. The final outputs are: a MAConvNeXt Block stacked 3 times with a Dim dimension of 96, a MAConvNeXt Block stacked 3 times with a Dim dimension of 192, a MAConvNeXt Block stacked 9 times with a Dim dimension of 384, and a MAConvNeXt Block stacked 3 times with a Dim dimension of 768. The features extracted in the four feature extraction stages of Block vNeXt.

3. The method for mathematical function image instance segmentation based on an improved Mask-R-CNN network according to claim 1, characterized in that, In step S1, the process of adding an Attention module to the improved feature extraction block MAConvNeXt Block includes: The features, after being activated by GELU and introducing nonlinear factors, are input into Asym Conv. Asym Conv with kernel sizes of 3×1 and 1×3 is used to replace the ordinary 3×3 convolution. Batch normalization is performed separately before feature fusion in the 3×1 and 1×3 Asym Conv branches. Then, the features are processed by a large kernel Dw Conv with a kernel size of 7×7 and an inflation rate of 3, and a kernel Pw Conv with a kernel size of 1×1. Finally, the output of the above steps is multiplied by the features input to the Attention module, serving as the weights of the features input to the Attention module. DropOut and DropPath layers are added at the end of the Attention module. The Attention module can be expressed by formulas (1) and (2). in Represents attention, Conv 1×1 DwConv represents a 2D convolution with a kernel size of 1×1, while Asym Conv represents a depthwise convolution. 3×1 This represents an asymmetric convolution with a kernel size of 3×1. Representing input features, Asym Conv 1×3 This represents an asymmetric convolution with a kernel size of 1×3. Adding representative features This indicates multiplication of the two parts.

4. The method for segmenting mathematical function images based on an improved Mask-R-CNN network according to claim 1, characterized in that, In step S2, the improved feature fusion network MA BiFPN based on BiFPN is obtained, and the process includes: S1, S2, S3, and S4 are input into the improved feature fusion network MA BiFPN, and feature multi-scale fusion is performed in three stacks. In each stack, S4 is fused with the features of S3 after passing through the RFA module to output P3. S2 is fused with the upsampled features of P3 to output P2. P2 is upsampled and fused with the features of S1 and passed through the ASPP module to output N1. N1 is max-pooled and fused with the features of P2 after passing through the ASPP module and S2 in the feature extraction network to output N2. N2 is max-pooled and fused with the features of P3 after passing through the ASPP module and S3 in the feature extraction network to output N3. N3 is max-pooled and fused with S4 after passing through the RFA and ASPP modules to output N4.

5. A mathematical function image instance segmentation method based on an improved Mask-R-CNN network according to claim 1 or 4, characterized in that, In step S2, the path in the original BiFPN from feature S4 of the original feature extraction network to P3 and N4 is replaced with the RFA module, specifically including: The RFA module uses ratio-invariant adaptive pooling on S4 to generate multi-scale features with sizes of 0.1, 0.3, and 0.5 times that of S4. For these multi-scale features, 1×1 convolutions are used to generate output features with 256 channels. Bilinear interpolation is then used to upsample the output features after the 1×1 convolution to the same scale. Finally, an adaptive spatial fusion module adaptively combines the context features after bilinear interpolation to reduce aliasing caused by interpolation. The features output from S4 by the RFA module possess multi-scale contextual information. The new output features are combined with S3 and propagated to the P3 features for fusion. The adaptive spatial fusion module assigns weights to each feature to aggregate these context features into new features. The weighted feature fusion calculation method is as follows: in, Let represent the feature vector of layer l at position (i,j). This represents the learnable weights, or contributions, of the other three layers on the feature map. This represents the feature vector generated at (i,j) in layer l after scaling the features from layer n.

6. A mathematical function image instance segmentation method based on an improved Mask-R-CNN network according to claim 1 or 4, characterized in that, In step S2, the ASPP module is used in the improved feature fusion network MA BiFPN, specifically including: The input features are processed through 2D convolutions with a kernel size of 1×1 and a dilation rate of 1, 2D convolutions with a kernel size of 3×3 and a dilation rate of 3, and 2D convolutions with a kernel size of 3×3 and a dilation rate of 6. After adaptive pooling, they are further processed through 2D convolutions with a kernel size of 1×1 and a dilation rate of 1, and then upsampled to obtain the corresponding scale features. Finally, these scale features are merged. After N2 max pooling, the features are fused with the P3 features from the ASPP module and the S3 features from the improved feature extraction network MAConvNeXt to output N3. The improved MA BiFPN performs bottom-up feature fusion at layer N3 as follows: Among them, W i All are learnable parameters, where e is the learning rate. This represents the input to layer N3. This represents the intermediate features of N3 on the top-down path. This means scaling the output features of the previous layer N2 to the same size as the features of N3 using max pooling. This represents the final output feature of layer N3.

7. The method for segmenting mathematical function images based on an improved Mask-R-CNN network according to claim 1, characterized in that, In step S3, the candidate bounding box of the improved RPN is increased by 5%, and a penalty factor λ(G2-B1∩G2) is added to the IoU to penalize candidate bounding boxes that do not completely cover the target region. The improved IoU calculation formula is as follows: Where λ represents the penalty intensity, and G2-B1∩G2 represents the spatial difference between the predicted bounding box and the ground truth bounding box.

8. The method for mathematical function image instance segmentation based on an improved Mask-R-CNN network according to claim 1, characterized in that, In step S4, the mask segmentation strategy in the original Mask-R-CNN is replaced by PointHead instead of fully connected layers. This includes: in PointHead, the features input from RoIAlign are first downsampled through multiple pooling operations and then coarsely predicted. Then, difficult points with poor segmentation performance are selected from the coarse segmentation features. A simple multilayer perceptron network is trained on these points by combining coarse and fine features. These difficult points are then re-predicted, and the re-predicted results replace the original coarse predictions, ultimately generating a function graph mask.

Images