Image segmentation method and system based on frequency perception anti-aliasing

Abstract

The invention discloses an anti-aliasing image segmentation method and system based on frequency perception, and belongs to the technical field of computer vision. According to the method, high-frequency detail information causing the aliasing phenomenon is adjusted and removed in feature extraction and down-sampling in a semantic segmentation model according to the sampling theorem. In the neural network, a frequency mixing module is adopted, high-frequency components with rich details are adaptively adjusted in the feature extraction process, and anti-aliasing filtering is proposed to remove the high-frequency components in the down-sampling process. Through the mode, the aliasing phenomenon in the neural network is relieved under the extremely small additional calculation amount, the object edge recognition precision of the semantic segmentation model in the feature processing process is remarkably improved, and high-precision and low-complexity semantic segmentation is achieved. The method is extremely low in extra calculation overhead, easy to implement, high in performance, high in robustness and beneficial to implementation of low-delay and high-speed semantic segmentation.

Description

Technical Field

[0001] The present invention relates to a frequency-aware anti-aliasing image segmentation method and system, and in particular to a frequency-aware anti-aliasing image segmentation method and system proposed based on a sampling theorem, and belongs to the technical field of computer vision. Background Art

[0002] Semantic segmentation is a key computer vision task that aims to associate each pixel in an image with a predefined class of interest. This requires the computer to have a deep understanding of the semantic information of the image in order to densely determine whether each pixel belongs to a specific class. This task has a wide range of applications, including identifying vehicles, pedestrians, and traffic signs in self-driving cars, as well as forests, rivers, and mountains in remote sensing images.

[0003] In order to achieve accurate semantic segmentation, images must be densely predicted, and details and high-frequency information in the image must be extracted and retained. However, modern semantic segmentation deep neural networks usually use multiple feature downsampling steps, which may lead to insufficient sampling rate, resulting in frequency aliasing and affecting the accuracy of the final segmentation results. Therefore, when designing and training semantic segmentation models, special attention should be paid to how to balance the sampling rate and retain image details to improve the performance and robustness of the model.

[0004] A deep neural network (DNN) model is a computational model consisting of multiple layers of neurons, and its design is inspired by the structure of the human nervous system. These networks process input data layer by layer to achieve complex tasks such as image recognition, speech recognition, natural language processing, etc. The hierarchical structure of a deep neural network includes an input layer, a hidden layer, and an output layer, where the hidden layer usually consists of multiple nodes, each of which is connected to the nodes in the previous layer and performs nonlinear transformations on the input through weighted and activated functions. Through a large amount of training data and iterative optimization algorithms (such as gradient descent), DNN models are able to learn complex data representations and show good performance on given tasks. In recent years, with the improvement of computing power and the development of deep learning algorithms, deep neural networks have become one of the core technologies in the field of artificial intelligence and have been widely used in various fields, including solving image segmentation tasks.

[0005] Regardless of the specific architecture, modern deep neural networks are essentially composed of a series of stacked convolutional or Transformer encoder blocks and downsampling layers. From a signal processing perspective, downsampling operations are prone to violating the Nyquist sampling theorem because the frequencies of features, especially those near boundaries, often exceed the Nyquist frequency. The resulting frequency overlap, formally known as aliasing, occurs when high frequencies above the Nyquist frequency are undersampled, resulting in signal distortion or misinterpretation.

[0006] Aliasing refers to the result in signal processing when the sampling rate is not enough to capture the high-frequency part of the signal. Specifically, when the frequency contained in the signal exceeds half of the sampling rate (i.e., the Nyquist frequency), the high-frequency part will be misinterpreted as the low-frequency part, resulting in distortion or incorrect frequency components when reconstructing the signal. This frequency overlap causes signal distortion and information loss, which affects the correct interpretation and analysis of the signal. Aliasing is a very important concept in digital signal processing, image processing, audio processing and other fields, and is crucial for the correct processing and interpretation of signals. In semantic segmentation tasks, aliasing will lead to incorrect final image segmentation results, especially in high-frequency areas such as the edges of objects. Summary of the invention

[0007] The purpose of the present invention is to creatively propose an image segmentation method and system based on frequency-aware anti-aliasing in order to solve the problem of frequency aliasing caused by downsampling in the existing semantic segmentation method based on the existing image intelligent recognition needs. The present invention aims to adjust and remove the high-frequency detail information that causes aliasing in the feature extraction and downsampling steps in the semantic segmentation model according to the sampling theorem.

[0008] Specifically, in the neural network, a frequency mixing module is used to adaptively adjust the high-frequency components with rich details during the feature extraction process, and an anti-aliasing filter is proposed to remove the high-frequency components during the downsampling process. In this way, with very little additional computation, the aliasing phenomenon in the neural network is alleviated, and the recognition accuracy of the semantic segmentation model for the edge of the object during the feature processing process is significantly improved, achieving high-precision and low-complexity semantic segmentation.

[0009] The present invention is implemented by adopting the following technical solutions.

[0010] A frequency-aware anti-aliasing image semantic segmentation method comprises the following steps:

[0011] Step 1: Build an image segmentation model based on frequency-aware anti-aliasing.

[0012] Step 2: Train a frequency-aware anti-aliasing image segmentation model using the loss function.

[0013] Step 3: Use the trained frequency-aware anti-aliasing image segmentation model to perform semantic segmentation on the image.

[0014] To achieve the purpose of the present invention, the present invention further proposes an image segmentation system based on frequency-aware anti-aliasing, such as Figure 2 As shown, it includes a frequency-aware anti-aliasing image segmentation model building module, a semantic segmentation network training module and a semantic segmentation reasoning module.

[0015] Beneficial Effects

[0016] Compared with the prior art, the present invention has the following advantages:

[0017] 1. The present invention can alleviate the frequency aliasing problem in the semantic segmentation deep neural network in principle, thereby avoiding the degradation of the final semantic segmentation result caused by aliasing and improving the accuracy of semantic segmentation.

[0018] 2. The present invention can significantly improve the ability of retaining important area detail information during the downsampling process of the semantic segmentation deep neural network.

[0019] 3. The present invention has extremely low additional computational overhead, is simple to implement, has high performance, and is highly robust, which is conducive to achieving low-latency, high-speed semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of the method of the present invention.

[0021] Figure 2 It is a schematic diagram of the structure of the frequency mixing module of the method of the present invention.

[0022] Figure 3 It is a schematic diagram of the structure of the anti-aliasing filtering module of the method of the present invention.

[0023] Figure 4 It is a flow chart of the system of the present invention. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, a frequency-aware anti-aliasing image semantic segmentation method includes the following steps:

[0026] Step 1: Build an image segmentation model using frequency-aware anti-aliasing.

[0027] Semantic segmentation is a key computer vision task that aims to associate each pixel in an image with a predefined category. For this intensive prediction task, the power of deep neural networks lies in preserving high-frequency detail information in the image, which is crucial for accurately predicting the category of each pixel. However, in the semantic segmentation process, downsampling operations are often required to reduce computational complexity and memory consumption. These downsampling steps usually include pooling or convolution operations, which can reduce the size of feature maps, thereby speeding up computation and reducing storage requirements. Although downsampling helps improve computational efficiency, it also brings a serious problem: frequency aliasing. In the downsampling process, the high-frequency part is mistakenly interpreted as the low-frequency part because the sampling rate is not enough to capture all the frequency information in the image. This aliasing phenomenon causes the loss of details in important areas and interferes with the correct extraction of feature signals. Especially in the semantic segmentation task, this can lead to misclassification and inaccurate boundary detection of the final result.

[0028] In order to solve the above problems, the present invention proposes a solution based on frequency perception, which aims to effectively alleviate the frequency aliasing phenomenon in deep neural networks. Specifically, a frequency mixing module for optimizing the frequency distribution of the feature extraction process and an anti-aliasing filtering module for optimizing the downsampling process are adopted, which can effectively alleviate the frequency aliasing problem in the deep neural network, thereby improving the accuracy and robustness of the semantic segmentation task. This method can not only improve the accuracy of image segmentation, but also has significant advantages in reducing computational and storage overhead. Frequency-aware anti-aliasing is applicable to semantic segmentation models (FCN, PSPNet, CCNet, OCNet, PCAA, UPerNet, etc.), and the method of use is to directly insert the proposed method into the corresponding position of the model.

[0029] like Figure 2 As shown in Figure 1, the frequency mixing module can be inserted into the convolution block or Transformer block in the semantic segmentation network model to predict a weight for the information of each frequency band of the input feature and perform spatial adaptive weighting f′(c, h, w), as shown in Formula 1:

[0030]

[0031] Among them, f ↓ (c,h,w),f ↑ (c,h,w) are the parts of the input feature f(c,h,w) that are lower and higher than the Nyquist frequency respectively; They are the adaptive weighted values ​​of the features below the Nyquist frequency channel by channel and space by space, which are obtained by a layer of convolution output after the sigmoid activation function. Their shapes and sizes are C×1×1 and 1×H×W, respectively. C represents the number of feature map channels, H represents the feature map width, and H represents the feature map width; (h,w) represents the two-dimensional coordinates, h represents the vertical coordinate, w represents the horizontal coordinate, and c represents the channel position; similarly, It is the adaptive weighting value for the features above the Nyquist frequency on a channel-by-channel and space-by-space basis, and its shape and size are also C×1×1 and 1×H×W respectively.

[0032] like Figure 3 As shown in the figure, the anti-aliasing filter module is inserted into the semantic segmentation network model before downsampling to remove the high-frequency components that cause aliasing, thereby preventing the aliased signal from affecting subsequent feature extraction:

[0033]

[0034] Among them, X a is the feature map after anti-aliasing filtering; is the inverse Fourier transform; Indicates frequency mask; X F is the result of Fourier transform of input feature X, which is:

[0035]

[0036] Among them, u and v are the horizontal and vertical frequency sizes respectively; H and W are the height and width of the input feature map respectively; e is the natural base, j is an imaginary number, and π represents pi; is the frequency mask, such as max(|u|,|v|)<φ ny ,but In other cases φ ny is the Nyquist frequency, which is half the downsampling rate.

[0037] Step 2: Use the semantic segmentation task loss function to train the semantic segmentation deep convolutional network.

[0038] The commonly used loss function in semantic segmentation tasks is a key training component, which performs supervised learning on deep convolutional networks. This loss function usually measures the difference between the model's prediction of the image and the actual label in a pixel-level manner, thereby guiding the update of network parameters. Among them, the cross-entropy loss function (Cross-Entropy Loss) is a common choice, also known as pixel-wise cross-entropy loss. In semantic segmentation, the classification of each pixel is regarded as an independent classification problem, so the calculation of the cross-entropy loss function is also performed for each pixel. Specifically, for each pixel, the cross-entropy loss function compares the category distribution predicted by the model with the category distribution of the actual label, and calculates the cross entropy between them to measure the difference between them. The optimization of this loss function aims to minimize the difference between the predicted and actual labels, thereby improving the performance of the model on the semantic segmentation task. By training with the cross-entropy loss function, the semantic segmentation model can gradually learn more accurate pixel-level classification decisions, thereby achieving effective segmentation of different objects and regions in the image. In addition, the cross entropy loss function also has the advantages of high computational efficiency and differentiability, making it the mainstream choice in semantic segmentation tasks.

[0039] Specifically, the pixel-wise cross-entropy loss is:

[0040]

[0041] Among them, L seg represents the cross entropy loss function used for segmentation training, N is the total number of pixels in the image, M is the number of categories, P is the prediction result, G is the true value, and m represents the category; G i,m represents the true value label of the i-th pixel category m; P i,m Represents the prediction result of the i-th pixel category m. The semantic segmentation network is trained and optimized end-to-end through the above loss function.

[0042] Step 3: Use the trained frequency-aware anti-aliasing semantic segmentation model to perform semantic segmentation on the image and obtain a segmentation result with rich details and precise edges.

[0043] In this semantic segmentation model, a frequency-aware strategy is adopted to effectively solve the frequency aliasing problem caused by the downsampling operation and improve the quality and accuracy of image segmentation.

[0044] To achieve the purpose of the present invention, the present invention further proposes an image semantic segmentation system based on frequency-aware anti-aliasing, such as Figure 3As shown, it includes a frequency-aware anti-aliasing semantic segmentation network building module 10, a semantic segmentation network training module 20, and an image semantic segmentation reasoning module 30.

[0045] Among them, the frequency-aware anti-aliasing semantic segmentation network building module 10 is used to build a semantic segmentation network model using frequency-aware anti-aliasing. This module can transform an ordinary semantic segmentation network into a semantic segmentation network model using frequency-aware anti-aliasing. Furthermore, the frequency-aware anti-aliasing semantic segmentation network building module 10 includes a frequency mixing module and an anti-aliasing filtering module. In the feature extraction process, the frequency mixing module can optimize the ratio of high and low frequencies in the features end-to-end, while retaining the high-frequency areas with rich details as much as possible, while reducing the high-frequency noise in irrelevant areas such as the background. At the same time, the anti-aliasing filtering module is used to perform anti-aliasing processing on the features to be downsampled, and the high-frequency information that may cause aliasing is removed, thereby reducing the impact of aliasing on subsequent feature extraction. Therefore, with very little additional computation, the accuracy of the semantic segmentation model can be significantly improved, and high-precision and low-complexity semantic segmentation can be achieved.

[0046] The semantic segmentation network training module 20 is used for end-to-end training of a frequency-aware anti-aliasing semantic segmentation network model.

[0047] The image semantic segmentation reasoning module 30 uses the trained model to extract image features using frequency-aware anti-aliasing and obtains the final image semantic segmentation result.

[0048] The connection relationship between the above modules is as follows:

[0049] The output end of the frequency-aware anti-aliasing semantic segmentation network building module 10 that combines frequency mixing and anti-aliasing filtering is connected to the input end of the semantic segmentation network training module 20.

[0050] The output end of the semantic segmentation network training module 20 is connected to the input end of the image semantic segmentation reasoning module 30 .

Claims

1. A frequency-aware anti-aliasing image semantic segmentation method, characterized in that: The following steps are involved: Step 1: Construct an image segmentation model based on frequency-aware anti-aliasing; Adopt frequency mixing module for optimizing frequency distribution of feature extraction process and anti-aliasing filtering module for optimizing downsampling process; Among them, the frequency mixing module can be inserted into the convolution block or Transformer block in the semantic segmentation network model to predict a weight for the information of each frequency band of the input feature and perform spatial adaptive weighting f′(c,h,w); The anti-aliasing filter module is inserted into the semantic segmentation network model. Before downsampling, it can remove the high-frequency components that cause aliasing, so as to prevent the aliased signal from affecting subsequent feature extraction: Among them, X a is the feature map after anti-aliasing filtering; is the inverse Fourier transform; Indicates frequency mask; X F It is the result after Fourier transform of input feature X; Step 2: Use the semantic segmentation task loss function to train the semantic segmentation deep convolutional network; Step 3: Use the trained frequency-aware anti-aliasing semantic segmentation model to perform semantic segmentation on the image and obtain a segmentation result with rich details and precise edges.

2. The image semantic segmentation method based on frequency-aware anti-aliasing according to claim 1, characterized in that: Among them, f ↓ (c,h,w),f ↑ (c,h,w) are the parts of the input feature f(c,h,w) that are lower and higher than the Nyquist frequency respectively; They are the adaptive weighting values ​​of the features below the Nyquist frequency channel by channel and space by space, which are obtained by a layer of convolution output after a sigmoid activation function. Their shapes and sizes are C×1×1 and 1×H×W, respectively. C represents the number of feature map channels, H represents the feature map width, and H represents the feature map width; (h,w) represents the two-dimensional coordinates, h represents the vertical coordinate, w represents the horizontal coordinate, and c represents the channel position; It is the adaptive weighting value for the features above the Nyquist frequency on a channel-by-channel and space-by-space basis, and its shape and size are also C×1×1 and 1×H×W respectively.

3. The image semantic segmentation method based on frequency-aware anti-aliasing according to claim 1, characterized in that: Among them, u and v are the horizontal and vertical frequency sizes respectively; H and W are the height and width of the input feature map respectively; e is the natural base, j is an imaginary number, and π represents pi; is the frequency mask, such as max(|u|,|v|)<φ ny ,but In other cases φ ny is the Nyquist frequency, which is half the downsampling rate.

4. The image semantic segmentation method based on frequency-aware anti-aliasing according to claim 1, characterized in that: In step 2, the pixel classification cross entropy is: Among them, L seg represents the cross entropy loss function used for segmentation training, N is the total number of pixels in the image, M is the number of categories, P is the prediction result, G is the true value, and m represents the pixel category; G i,m represents the true value label of the i-th pixel category m; P i,m Represents the prediction result of the i-th pixel category m; the semantic segmentation network is trained and optimized end-to-end through the above loss function.

5. A frequency-aware anti-aliasing image semantic segmentation system for implementing the method of claim 1, characterized in that: It includes frequency-aware anti-aliasing semantic segmentation network building module, semantic segmentation network training module, and image semantic segmentation reasoning module; Among them, the frequency-aware anti-aliasing semantic segmentation network building module is used to build a semantic segmentation network model using frequency-aware anti-aliasing; this module can transform an ordinary semantic segmentation network into a semantic segmentation network model using frequency-aware anti-aliasing; Semantic segmentation network training module, which is used for end-to-end training of frequency-aware anti-aliasing semantic segmentation network models; The image semantic segmentation inference module uses the trained model to extract image features using frequency-aware anti-aliasing and obtain the final image semantic segmentation results; The connection relationship between the above modules is as follows: The output end of the frequency-aware anti-aliasing semantic segmentation network building module combining frequency mixing and anti-aliasing filtering is connected to the input end of the semantic segmentation network training module; The output end of the semantic segmentation network training module is connected to the input end of the image semantic segmentation reasoning module.

6. The image semantic segmentation system based on frequency-aware anti-aliasing as claimed in claim 5, characterized in that: The frequency-aware anti-aliasing semantic segmentation network building module includes a frequency mixing module and an anti-aliasing filtering module. During the feature extraction process, the frequency mixing module can optimize the ratio of high and low frequencies in the features end-to-end, while retaining the high-frequency areas with rich details as much as possible and reducing the high-frequency noise in unimportant areas. At the same time, the anti-aliasing filtering module is used to perform anti-aliasing processing on the downsampled features to remove the high-frequency information that may cause aliasing, thereby reducing the impact of aliasing on subsequent feature extraction.

Images