Real-time instance segmentation method based on frequency domain and spatial domain information fusion

Abstract

The present invention discloses a real-time instance segmentation method based on the fusion of frequency domain and spatial domain information. First, the loss function of the model training is set; then, the stochastic gradient descent algorithm with momentum is used as the optimization method, and the non-maximum suppression algorithm is used to eliminate duplicate detection; then, a ResNet50 residual network architecture consisting of multiple stacked residual blocks is built as the backbone network; then, an improved bidirectional feature pyramid structure is used to operate and fuse the feature maps of the last three convolution modules to obtain semantic information, and a frequency domain representation is introduced to obtain more accurate global information; finally, the last layer of feature maps with completed information fusion is input into the mask branch, and the mask coefficients obtained are combined with the prototype mask feature map using an improved frequency domain representation feature fusion method to obtain the position coordinates, category information and pixel segmentation results of the target object. The present invention effectively improves the image segmentation accuracy on the basis of real-time (more than 30 frames per second) and has a reasonable structure.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation methods, and in particular to a real-time instance segmentation method for fusing frequency domain and spatial domain information. Background Art

[0002] Fully convolutional neural networks (FCNs) have dominated the field of image segmentation, especially semantic segmentation, because they can accept input images of any size, have high computing speed, and can achieve pixel-level image segmentation effects.

[0003] Extensive experimentation has been conducted with fully convolutional neural networks. Since the introduction of FCN in 2015, a number of classic models have emerged, including UNet and UNet-like structures. These models build on FCN by continuously deepening the number of network layers to improve segmentation accuracy. While these models have gradually improved image segmentation quality, real-time instance segmentation remains a challenging task due to the difficulty of extracting image features and the enormous computational complexity of digital image processing.

[0004] To achieve high-precision and fast image segmentation, we analyzed, studied, and compared a large number of existing segmentation models. We found that they generally face the following three limitations: 1) The difficulty of extracting image features and the enormous computational effort of digital image processing pose challenges to real-time instance segmentation. 2) Although deep learning has made progress, improving accuracy and achieving real-time performance remain challenges. 3) In image segmentation, it is necessary to balance the model's computational speed and accuracy, especially for real-time applications. For both one-stage and two-stage models, it is necessary to find an appropriate balance between accuracy and speed. Summary of the Invention

[0005] The present invention aims to solve the above problems in the prior art. It proposes a real-time instance segmentation network based on the fusion of frequency domain and spatial domain information to enable the model to achieve higher segmentation accuracy on a real-time basis. The technical solution of the present invention is as follows:

[0006] The real-time instance segmentation method based on the fusion of frequency domain and spatial domain information includes the following steps:

[0007] Step 1: Set up the image instance segmentation dataset and MS COCO training details, and determine the training set and test set;

[0008] Step 2: Use the ResNet50 residual network architecture consisting of multiple stacked residual blocks as the backbone network to perform preliminary feature extraction and obtain feature maps of multiple levels and sizes;

[0009] Step 3: Combined with the preliminary feature extraction basic network in step 2, the feature maps extracted by the last three convolutional modules are input into the improved bidirectional feature pyramid structure to fuse the high-level semantic information, the low-level high-resolution detail information and the same-layer supervision information, and restore the resolution of the feature map;

[0010] Step 4: Load the pre-trained network model obtained in step 3, input the last layer of feature map after information fusion into the mask branch, use the improved frequency domain representation feature fusion method to combine the obtained mask coefficients and the prototype mask feature map to complete the network construction, and obtain a real-time instance segmentation network based on the fusion of frequency domain and spatial domain information; finally, load the test image data into the network and quickly test to complete the image segmentation.

[0011] Furthermore, the instance segmentation dataset in step 1 is PASCAL VOC (Visual Object Classes Challenge) 2012, the training set is the train2017 dataset, and the test set is the val2017 and test-dev datasets.

[0012] Furthermore, the instance segmentation part in step 1 adopts the cross entropy loss function, which is expressed as:

[0013]

[0014] Where i represents each possible category, P(i) represents the probability of the i-th category of the true probability distribution, and Q(i) represents the probability of the i-th category predicted by the model;

[0015] The optimization method uses the stochastic gradient descent algorithm with momentum, which is expressed as:

[0016]

[0017] in, is the gradient of the loss function with respect to the model parameters, β is the momentum coefficient, α is the learning rate, v represents momentum, θ represents the initial parameters, and x i ,y i Represent the horizontal and vertical coordinates of the i-th vector respectively.

[0018] Furthermore, a non-maximum suppression algorithm is used during the training process, specifically including: first, the detection frame with the highest score is selected from all detection frames of the same category; then, the detection frames with lower scores are compared with it in turn, and the intersection over union (IoU) between them is calculated. If the IoU is higher than a certain threshold, it is considered that they are detecting the same target, and the detection frame with the lower score is removed; then the detection frame with the highest score in this round of comparison is regarded as a reasonable detection result and is removed from the current loop set; the above steps are repeated until all detection frames of this category are determined.

[0019] Furthermore, the ResNet50 residual network structure described in step 2 is as follows:

[0020]

[0021] Furthermore, the improved bidirectional feature pyramid described in step 3, based on the top-down feature extraction of the basic pyramid model, uses a convolution with a step size of 2 to complete the downsampling of the shallow feature map, and obtains the same size as the feature map of the penultimate convolution module of ResNet50. The two are spliced ​​to generate the feature map P4, which adds the fusion of shallow precise spatial information and deep high-level semantic information; the improved bidirectional feature pyramid structure introduces frequency domain feature fusion into the bidirectional pyramid module, replaces 3×3 convolution with frequency domain convolution, and supplements the feature map with global information and richer semantic information.

[0022] Furthermore, the improved bidirectional feature pyramid also introduces skip connections at the same level.

[0023] Furthermore, step 4 adds frequency domain representation only at the tail end of the network model, that is, at the network layer that finally generates semantic information and spatial information. First, the channels are separated by a split operation, and part of the channel c' is subjected to fast Fourier transform and inverse transform to extract global high-frequency detail information and low-frequency semantic information, which are then superimposed with the original feature map. That is, the input feature map is supplemented and adjusted with the information extracted from the high-frequency and low-frequency features respectively, and is fused with the spatial domain information in the other part of the channel c" to restore the spatial detail information and obtain more accurate semantic information.

[0024] Furthermore, the calculation process of the improved frequency domain representation feature fusion method described in step 4 is as follows: First, the input spatial domain feature map Perform two-dimensional fast Fourier transform to convert it into frequency domain feature map X∈C M×N :

[0025] Then X is divided into high frequency components X h and low-frequency component X l :

[0026]

[0027]

[0028] M, N represent the range of height and width in the frequency domain feature map, m, n represent the actual height and width in the frequency domain feature map, X mn Represents the frequency domain components, and then the high and low frequency components X h and X l Perform the inverse transform back to the spatial domain and get x h and x l :

[0029] x h =F -1 (X h ),x l =F -1 (X l )

[0030] F -1 represents the inverse Fourier transform;

[0031] Then use their corresponding convolution kernel W h and W l Convolution is performed separately, and then the high and low frequency features after convolution are obtained through the activation function σ:

[0032]

[0033]

[0034] Finally, the high-frequency and low-frequency feature maps in the spatial domain are superimposed on the original spatial domain feature map to obtain the output x out :

[0035] x out =x+x h_conv +x l_conv ,

[0036] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned real-time instance segmentation method based on the fusion of frequency domain and spatial domain information.

[0037] The advantages and beneficial effects of the present invention are as follows:

[0038] This paper optimizes the fully convolutional neural network structure using the concept of frequency domain representation, fusing global high-frequency details and low-frequency semantic information through FFT transformation and inverse transformation. At the same time, the feature pyramid structure is improved. Compared with the original structure, on the one hand, it can effectively achieve the fusion of high-level semantics, thereby enriching semantic information and improving the computational quality of the segmentation network. On the other hand, it also enhances the ability to extract low-level high-resolution spatial position information, and also guarantees the quality of the image segmentation effect to a certain extent.

[0039] Other advantages of this invention include setting relevant details and parameters before training, ensuring sufficient data training while avoiding redundancy in network training data. Furthermore, the real-time instance segmentation network based on the fusion of frequency and spatial domain information offers greater accuracy in classification and localization, higher mask quality, and promising application prospects for feature extraction.

[0040] The present invention effectively improves the image segmentation accuracy on the basis of real-time (more than 30 frames per second) and has a reasonable structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of a real-time instance segmentation method based on frequency domain and spatial domain information fusion provided by the present invention;

[0042] Figure 2 This is a detailed structural diagram of the frequency domain information fusion of the real-time instance segmentation network based on the fusion of frequency domain and spatial domain information;

[0043] Figure 3 Schematic diagram of the network structure of the frequency domain representation bidirectional feature pyramid of the real-time instance segmentation network based on the fusion of frequency domain and spatial domain information;

[0044] Figure 4 This is the overall structure diagram of the real-time instance segmentation network based on the fusion of frequency domain and spatial domain information. DETAILED DESCRIPTION

[0045] The following will describe the technical solutions in the embodiments of the present invention in detail with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention.

[0046] The technical solution of the present invention to solve the above technical problems is:

[0047] like Figure 1 As shown, the real-time instance segmentation method based on the fusion of frequency domain and spatial domain information provided in this embodiment includes the following steps:

[0048] Step 1: Set up training details for the PASCAL VOC (Visual Object Classes Challenge) 2012 and MS COCO datasets. Use the train2017 dataset for training and the val2017 and test-dev datasets for testing.

[0049] The instance segmentation part uses the cross entropy loss function, which is expressed as:

[0050]

[0051] Where i represents each possible category, P(i) represents the probability of the i-th category of the true probability distribution, and Q(i) represents the probability of the i-th category predicted by the model.

[0052] The optimization method uses the stochastic gradient descent algorithm with momentum, which is expressed as:

[0053]

[0054] in, is the gradient of the loss function with respect to the model parameters, β is the momentum coefficient, and α is the learning rate, which is used to control the step size of parameter update. v represents momentum, θ represents the initial parameters, and x i ,y i They represent the horizontal and vertical coordinates of the i-th vector respectively. The initial learning rate is 0.001, the weight learning rate decay is 0.0005, and the momentum is 0.9.

[0055] The non-maximum suppression algorithm (NMS) is used in the training process. The algorithm process is as follows:

[0056]

[0057] In the previous algorithm, M represents the detection box with the highest score among the detection boxes of the same category of targets.

[0058] The algorithm is as follows: first, select the detection frame with the highest score from all detection frames of the same category; then, compare the detection frames with lower scores with it in turn and calculate the intersection over union (IoU) between them. If the IoU is higher than a certain threshold, they are considered to be detecting the same target and the detection frame with the lower score is removed; then the detection frame with the highest score in this round of comparison is regarded as a reasonable detection result and is removed from the current loop set; repeat the above steps until all detection frames of this category are determined.

[0059] Step 2: Use the ResNet50 residual network architecture consisting of multiple stacked residual blocks as the backbone network to perform preliminary feature extraction and obtain feature maps of multiple levels and sizes.

[0060] To perform preliminary feature extraction on the input image, the widely used ResNet50 is used as the base network. The Residual Neural Network (ResNet) directly passes the previous features backward through a "short-circuit" branch, while the intermediate network layers only need to learn the residuals of the feature map. The residual module addresses the degradation problems of gradient dispersion, gradient explosion, and increased error that often occur with increasing depth in convolutional neural networks. This not only significantly reduces model training time but also produces better feature maps for further feature extraction and information fusion in subsequent networks.

[0061] The ResNet50 network structure used is as follows:

[0062]

[0063]

[0064] Step 3: Combined with the feature extraction basic network in step 2, the feature maps of the last three convolutional modules are input into the improved bidirectional feature pyramid (FPN) structure to fuse high-level semantic information, low-level high-resolution detail information layer, and same-layer supervision information, and restore the resolution of the feature map.

[0065] like Figure 2 As shown in Figure 1, in order to extract semantic information and expand the receptive field, the network depth needs to be increased, which will cause the size of the feature map to continue to decrease. In order to restore the spatial information of the feature map and fuse multi-scale information, an improved bidirectional feature pyramid structure is added to the network model.

[0066] Building on the top-down feature extraction of the basic pyramid model, the bidirectional feature pyramid structure uses convolutions with a stride of 2 to downsample shallow feature maps to the same size as the feature map of the penultimate convolutional module of ResNet50. These two are then concatenated to generate the feature map P4. This enhances the fusion of shallow, precise spatial information with deep, high-level semantic information. Furthermore, to suppress noise introduced by the fusion of information from different layers, skip connections within the same layer are introduced. The improved bidirectional feature pyramid structure incorporates frequency-domain feature fusion into the bidirectional pyramid module, replacing 3×3 convolutions with frequency-domain convolutions. This supplements the feature map with global information and richer semantic information. To address real-time requirements, the original method of running the pyramid module three times was omitted, and the module was run only once. This improves model performance at a minimal cost.

[0067] Step 4: Load the pre-trained network model obtained in step 3, input the last layer feature map with a larger resolution that contains more spatial detail information after information fusion into the mask branch, and use the improved frequency domain representation feature fusion method to combine the obtained mask coefficients with the prototype mask feature map to complete the network construction and obtain a real-time instance segmentation network based on the fusion of frequency domain and spatial domain information. Specifically:

[0068] Dual-core Fourier convolutional layers enhance the feature extraction capabilities of deep convolutional networks, but they also increase the computational complexity of the algorithm model. Therefore, frequency domain representation is only added to the network's tail end—the layer that ultimately generates semantic and spatial information—to enhance the network's feature extraction capabilities. To ensure real-time performance, frequency domain representation is only introduced in a subset of channels in the feature map. First, the channels are separated using a split operation. A subset of channels, c', undergoes a fast Fourier transform (FFT) and an inverse FFT to extract global high-frequency detail information and low-frequency semantic information. This information is then superimposed with the original feature map. This information, extracted from the high- and low-frequency features, is used to supplement and adjust the input feature map. This information is then fused with the spatial domain information from the remaining channel, c", to restore spatial detail information and obtain more accurate semantic information. This allows the feature map to be directly propagated backward, effectively completing frequency domain feature fusion and stabilizing the backpropagation of gradients during training, enhancing model convergence.

[0069] like Figure 3 As shown in the figure, the calculation process of the improved frequency domain representation fusion is as follows: First, the input spatial domain feature map Perform two-dimensional fast Fourier transform to convert it into frequency domain feature map X∈C M×N :

[0070] X=F(x)

[0071] Then X is divided into high frequency components X h and low-frequency component X l :

[0072]

[0073]

[0074] M, N represent the range of height and width in the frequency domain feature map. m, n represent the actual height and width in the frequency domain feature map. mn Represents the frequency domain components, and then the high and low frequency components X h and X l Perform the inverse transform back to the spatial domain to get x h and x l :

[0075] x h =F -1 (X h ),x l =F -1 (X l )

[0076] F -1 Represents the inverse Fourier transform.

[0077] Then use their corresponding convolution kernel W h and W l Convolution is performed separately, and then the high and low frequency features after convolution are obtained through the activation function σ and

[0078]

[0079]

[0080] j, k represent the vertical and horizontal coordinates of the convolution kernel respectively.

[0081] Finally, the high-frequency and low-frequency feature maps in the spatial domain are superimposed on the original spatial domain feature map to obtain the output x out :

[0082] x out =x+x h_conv +x l_conv ,

[0083] like Figure 4 As shown in Figure 3, we finally obtain a real-time instance segmentation network that fuses frequency and spatial domain information. Inputting the image data to be tested into this network can complete image segmentation more efficiently and accurately.

[0084]

[0085] Table 1 Performance comparison of the proposed method FRISNET with the listed advanced instance segmentation models on MS COCO test-dev

[0086] The method FRISNET of the present invention is compared with the advanced instance segmentation models listed in Table 1 on MS COCO test-dev. Methods before YOLACT performed well in segmentation (mask) and detection (box), but their inference speed was slow and far from real-time. YOLACT's inference speed is 3.8 times that of the previous fastest instance segmentation model Mask R-CNN, completing the instance segmentation task at real-time speed for the first time. Our model is similar to YOLACT in speed, but has better performance. FRISNET can achieve 34.5mAP on ResNet-50 at a speed of more than 30 frames per second, which is 1.17% higher than YOLACT++ on the ResNet-50 architecture. FRISNET's accuracy is close to that of RetinaMask on ResNet-101, but its speed is more than 5 times that of the former. The accuracy of FRISE is also close to that of several other models [42-44], but its speed is significantly faster. Subsequent methods offer a better balance between accuracy and speed, but MEInst and PolarMask are both slower than the method proposed in this paper. CeterMask is similar in speed to the method proposed in this paper, but its accuracy is 4.86% lower. CondInst is slightly more accurate than the method proposed in this paper, but its speed is less than two-thirds of that. This shows that FRISNET achieves a better balance between speed and accuracy.

[0087] In Table 3, 1 / 8, 1 / 4, and 1 / 2 represent the proportion of frequency domain transforms in feature channels when frequency domain representation is introduced. We can see that as the proportion of feature channels using frequency domain representation increases, all accuracy metrics improve, confirming the effectiveness of frequency domain representation. However, we also see that as the proportion of feature channels using frequency domain representation increases, model inference time increases. We struck a balance between accuracy and speed, and the final model only introduced frequency domain representation in 1 / 4 of the channels.

[0088] The above embodiments should be understood as merely illustrating the present invention and not as limiting the scope of protection of the present invention. After reading the contents of the present invention, technicians may make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A real-time instance segmentation method based on the fusion of frequency domain and spatial domain information, characterized in that: The following steps are involved: Step 1: Set up the image instance segmentation dataset and MS COCO training details, and determine the training set and test set; Step 2: Use the ResNet50 residual network architecture consisting of multiple stacked residual blocks as the backbone network to perform preliminary feature extraction and obtain feature maps of multiple levels and sizes; Step 3: Combined with the preliminary feature extraction basic network in step 2, the feature maps extracted by the last three convolutional modules are input into the improved bidirectional feature pyramid structure to fuse the high-level semantic information, the low-level high-resolution detail information and the same-layer supervision information, and restore the resolution of the feature map; The improved bidirectional feature pyramid, based on the top-down feature extraction of the basic pyramid model, uses convolution with a stride of 2 to complete the downsampling of the shallow feature map, obtaining the same size as the feature map of the penultimate convolution module of ResNet50, and splicing the two to generate the feature map P4, which adds the fusion of shallow precise spatial information and deep high-level semantic information. In order to suppress the noise introduced by the fusion of information at different levels, skip connections at the same level are also introduced. The improved bidirectional feature pyramid structure introduces frequency domain feature fusion into the bidirectional pyramid module, replacing 3×3 convolution with frequency domain convolution, and supplementing the feature map with global information and richer semantic information. The improved bidirectional feature pyramid is run once. Step 4: Load the pre-trained network model obtained in step 3, input the last layer of feature map after information fusion into the mask branch, and use the improved frequency domain representation feature fusion method to combine the obtained mask coefficients and the prototype mask feature map to complete the network construction, thus obtaining a real-time instance segmentation network based on the fusion of frequency domain and spatial domain information; finally, load the test image data into the network and quickly test and complete the image segmentation; Step 4 is only performed at the end of the network model, that is, at the network layer that ultimately generates semantic and spatial information. The frequency domain representation is added. First, the channels are separated by a split operation. Part of the channel c' is subjected to fast Fourier transform and inverse transform to extract global high-frequency detail information and low-frequency semantic information, which are then superimposed with the original feature map. That is, the input feature map is supplemented and adjusted using the information extracted from the high-frequency and low-frequency features, and then fused with the spatial domain information in the other part of the channel c" to restore spatial detail information and obtain more accurate semantic information. The frequency domain transform accounts for 1 / 4 of the feature channel. The calculation process of the improved frequency domain representation feature fusion method is as follows: First, the input spatial domain feature map Perform two-dimensional fast Fourier transform to convert it into frequency domain feature map X∈C M×N : Then X is divided into high frequency components X h and low-frequency component X l : M, N represent the range of height and width in the frequency domain feature map, m, n represent the actual height and width in the frequency domain feature map, X mn Represents the frequency domain components, and then the high and low frequency components X h and X l Perform the inverse transform back to the spatial domain to get x h and x l : F -1 represents the inverse Fourier transform; Then use their corresponding convolution kernel W h and W l Convolution is performed separately, and then the high and low frequency features after convolution are obtained through the activation function σ: Finally, the high-frequency and low-frequency feature maps in the spatial domain are superimposed on the original spatial domain feature map to obtain the output x out :

2. The real-time instance segmentation method based on frequency domain and spatial domain information fusion according to claim 1, characterized in that: The instance segmentation dataset in step 1 is PASCAL VOC 2012, the training set is the train2017 dataset, and the test set is the val2017 and test-dev datasets.

3. The real-time instance segmentation method based on frequency domain and spatial domain information fusion according to claim 1 or 2, characterized in that: The instance segmentation part in step 1 adopts the cross entropy loss function, which is expressed as: Where i represents each possible category, P(i) represents the probability of the i-th category of the true probability distribution, and Q(i) represents the probability of the i-th category predicted by the model; The optimization method uses the stochastic gradient descent algorithm with momentum, which is expressed as: in, is the gradient of the loss function with respect to the model parameters, β is the momentum coefficient, α is the learning rate, v represents momentum, θ represents the initial parameters, and x i ,y i Represent the horizontal and vertical coordinates of the i-th vector respectively.

4. The real-time instance segmentation method based on frequency domain and spatial domain information fusion according to claim 1 or 2, characterized in that: The non-maximum suppression algorithm is used during the training process, which specifically includes: first, the detection frame with the highest score is selected from all detection frames of the same category; then, the detection frames with lower scores are compared with it in turn, and the intersection over union (IoU) between them is calculated. If the IoU is higher than a certain threshold, it is considered that they are detecting the same target, and the detection frame with the lower score is removed; then the detection frame with the highest score in this round of comparison is regarded as a reasonable detection result and is removed from the current loop set; the above steps are repeated until all detection frames of this category are determined.

5. The real-time instance segmentation method based on frequency domain and spatial domain information fusion according to claim 1, characterized in that: The ResNet50 residual network structure described in step 2 is as follows:

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the real-time instance segmentation method based on frequency domain and spatial domain information fusion according to any one of claims 1 to 5 are implemented.

Images