Image semantic segmentation method and system based on frequency-aware feature fusion

Abstract

The present invention relates to an image semantic segmentation method and system based on frequency-aware feature fusion, and belongs to the field of computer vision technology. In the feature fusion process of the semantic segmentation neural network model, the present invention adaptively removes harmful high frequencies in upsampled high-level features and enhances beneficial high frequencies in low-level features, thereby improving the feature fusion effect, alleviating the problem of feature misalignment, and obtaining fusion features with rich details. The method of the present invention can be extended to the post-upsampling process of the semantic segmentation model, using the beneficial high-frequency information in the image to guide the removal of inaccurate high-frequency components after upsampling to obtain a high-resolution output result rich in details. This method greatly improves the ability of the semantic segmentation model to retain detailed information of important areas during feature fusion and post-upsampling processes while introducing extremely small amounts of additional computation and parameters, thereby achieving high-precision and low-complexity semantic segmentation.

Description

Technical Field

[0001] The present invention relates to an image semantic segmentation method and system based on frequency perception feature fusion, and belongs to the technical field of computer vision. Background Art

[0002] Semantic segmentation is a computer vision task that requires a computer to understand the semantics of image content and densely determine whether each pixel in the image belongs to a specific category of interest. Semantic segmentation can be used to accurately identify the set of pixels that constitute the target of interest, such as vehicles, pedestrians, and traffic signs in self-driving car scenarios, or forests, rivers, and mountains in remote sensing images. Because dense image predictions are required, extracting and preserving detailed content and high-frequency information is crucial for ultimate accuracy. Semantic segmentation has a wide range of applications, including medical image analysis, video surveillance, autonomous driving, robotic vision, and remote sensing image analysis.

[0003] Semantic segmentation neural network models are neural network models used for semantic segmentation tasks. They are generally based on the fully convolutional neural network architecture (FCN). Specifically, they can be implemented using convolutional neural networks and transformer neural networks. The backbone of a typical FCN is often composed of multiple feature extraction and feature downsampling stages to obtain feature information with different resolutions and different receptive fields. Among them, high-resolution features have a smaller receptive field and weaker semantic information, but contain more detailed information; low-resolution features go through more feature extraction and downsampling stages, have a larger receptive field, and contain more semantic information, but detailed information is gradually lost during the downsampling process.

[0004] Feature fusion is a common operation in semantic segmentation neural network models. It refers to the process of combining different features to form new features. Specifically, standard feature fusion upsamples the high-level, low-resolution nearest neighbors in the deep layers of the network to the same size as the low-level, high-resolution features in the shallow layers of the network before adding or concatenating them. Because high-level, low-resolution features contain rich semantic information, and low-level, high-resolution features contain rich detailed information, feature fusion produces high-resolution features with both semantics and details.

[0005] Post-upsampling is a commonly used operation in semantic segmentation neural network models. It refers to upsampling the low-resolution feature map or output map obtained by the model to obtain an output of the same size as the input image, usually performed by bilinear interpolation. Summary of the Invention

[0006] The purpose of the present invention is to start from the application requirements of existing image intelligent recognition, and to address the technical problems in the feature fusion process of existing semantic segmentation methods, such as feature misalignment caused by feature downsampling and inaccurate upsampling, and poor feature fusion caused by the gap between semantics and resolution between high-level and low-level features. This paper creatively proposes an image semantic segmentation method and system based on frequency-aware feature fusion.

[0007] The present invention can adaptively remove harmful high frequencies in upsampled high-level features and enhance beneficial high frequencies in low-level features during the feature fusion process of a semantic segmentation neural network model, thereby improving the feature fusion effect, alleviating the problem of feature misalignment, and obtaining fused features with rich details. The method of the present invention can be extended to the post-upsampling process of a semantic segmentation model, using the beneficial high-frequency information in the image to guide the removal of inaccurate high-frequency components after upsampling to obtain a high-resolution output result rich in details. This method significantly improves the ability of the semantic segmentation model to retain detailed information in important areas during feature fusion and post-upsampling processes while introducing minimal additional computational effort and parameter count, thereby achieving high-precision, low-complexity semantic segmentation.

[0008] The present invention is implemented by adopting the following technical solutions.

[0009] An image semantic segmentation method based on frequency-aware feature fusion includes the following steps:

[0010] Step 1: Build a semantic segmentation neural network model based on frequency-aware feature fusion.

[0011] Step 2: Train the semantic segmentation neural network model.

[0012] Step 3: Use the trained semantic segmentation neural network model to perform semantic segmentation on the image.

[0013] On the other hand, in order to achieve the objectives stated in the present invention, the present invention further proposes an image semantic segmentation system based on frequency-aware feature fusion, including a frequency-aware feature fusion semantic segmentation neural network model module, a semantic segmentation model training module, and a semantic segmentation reasoning module.

[0014] Beneficial effects

[0015] Compared with the prior art, the present invention has the following advantages:

[0016] 1. This invention can alleviate the problem of feature misalignment in the feature fusion process of semantic segmentation deep neural network and enhance the extraction and fusion of beneficial high-frequency detail information.

[0017] 2. The present invention can make full use of the high-frequency information in the image to optimize the upsampling process, improve the upsampling fineness, and thus improve the accuracy of semantic segmentation.

[0018] 3. The present invention has low additional computational overhead, simple implementation, high performance, and strong robustness, which is conducive to achieving low-latency and high-speed semantic segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the method of the present invention.

[0020] Figure 2 It is a schematic diagram of the structure of the frequency perception feature fusion semantic segmentation neural network model of the method of the present invention.

[0021] Figure 3 It is a schematic diagram of the frequency perception feature fusion module of the method of the present invention.

[0022] Figure 4 It is a flow chart of the system of the present invention. DETAILED DESCRIPTION

[0023] The invention will be further described below with reference to the accompanying drawings.

[0024] An image semantic segmentation method based on frequency-aware feature fusion, such as Figure 1 As shown, the following steps are included:

[0025] Step 1: Build a semantic segmentation neural network model using frequency-aware feature fusion.

[0026] Semantic segmentation is a dense prediction task, and the high-frequency details preserved in deep neural network feature maps are crucial for prediction accuracy. Feature fusion is a common operation in semantic segmentation neural network models, combining different features to form new features. Specifically, standard feature fusion upsamples the low-resolution nearest neighbors deep in the network to the same size as the high-resolution features in shallower layers, then adds or concatenates them to produce high-resolution features with semantic meaning and details. However, standard feature fusion strategies can lead to feature misalignment, disrupting high-frequency details.

[0027] This method replaces the standard feature fusion strategy with frequency-aware feature fusion, significantly alleviating the problem of feature misalignment during the feature fusion process of deep neural networks for semantic segmentation and enhancing the extraction and fusion of useful detail information. Furthermore, this method can be extended to the post-upsampling process of semantic segmentation models, leveraging the beneficial high-frequency information in the image to guide the removal of inaccurate high-frequency components after upsampling, thereby producing detailed, high-resolution output results.

[0028] Specifically, the frequency-aware feature fusion semantic segmentation neural network model is constructed by replacing the feature fusion and upsampling in the ordinary semantic segmentation deep neural network with frequency-aware feature fusion and frequency-aware upsampling. Its overall structure is as follows: Figure 2 As shown in FIG. 1 , the frequency-aware feature fusion is applicable to a semantic segmentation neural network model (optionally, the semantic segmentation neural network model can use UPerNet, SemanticFPN, U-Net, SegFormer, SegNeXt, etc.).

[0029] like Figure 3 As shown, the frequency-aware feature fusion Y l The description is as follows:

[0030]

[0031]

[0032]

[0033] in, represents low-pass filtering, Indicates high-pass filtering; X l and Y l+1 They are the features of the lth layer generated by the backbone network and the fusion features of the l+1th layer; Indicates upsampling (e.g. 2× nearest neighbor interpolation upsampling); Represents the l-th layer feature after being mapped to the same number of channels by the projection function; Represents the fusion features of the l+1th layer after upsampling; represents a projection function (e.g. a 1×1 convolution) that ensures and Have the same number of channels

[0034] For simplicity, this paper discusses feature fusion in the form of addition, which can also be applied to other forms of feature fusion, for example, the concatenation used in some semantic segmentation neural network models (U-Net, SegFormer, SegNeXt, etc.). Denote spatial low-pass and high-pass filters, respectively. A simple implementation uses classic low-pass / high-pass filters, such as Gaussian and Laplacian filters. However, due to the spatial variability of the input feature map, using only a fixed, non-learnable kernel is clearly not the best choice for removing or enhancing high-frequency components. For example, some useful edges / textures may be blurred, while some harmful high-frequency components are not fully filtered.

[0035] Therefore, the present invention generates spatially adaptive low-pass / high-pass filtering as follows:

[0036]

[0037] in, The feature vector representing the feature after fusion of the lth layer is located at coordinates i, j; i, j represent the horizontal and vertical coordinates of the input features; and They are spatially varying adaptive low-pass and high-pass kernels, respectively. For each feature coordinate i, j, a high-pass and low-pass kernel will be adaptively generated; p, q represent the adaptive kernels The horizontal and vertical coordinates of the nuclear parameters; Represents the values ​​of the adaptive low-pass and high-pass kernels generated at the feature coordinates i and j at the kernel parameter coordinates p and q; The feature vector representing the feature after fusion of the l+1th layer is located at the coordinates i+p,j+q; The feature vector representing the lth layer feature at coordinates i+p,j+q; Represents the feature vector of the l-th layer feature at coordinates i, j; Ω represents the neighborhood area that the filter depends on (usually a 3×3 area).

[0038] and Obtained by the following formula:

[0039]

[0040]

[0041]

[0042]

[0043]

[0044] Among them, Z lis the compressed feature; for adaptive low-pass filtering, the Softmax constraint is used to be all positive and unique; for the high-pass filter kernel, the Softmax constraint is first used to obtain the low-pass filter, and then the high-pass filter is obtained by inverting it, specifically using the identity kernel E minus the low-pass kernel; i, j are the horizontal and vertical coordinates of the input features, and p, q are the horizontal and vertical coordinates of the kernel parameters; Represents the features after fusion of the l+1th layer; Represents the l-th layer feature after being mapped by the projection function; represents the initial low-pass kernel parameters; represents the initial high-pass kernel parameters; Represents the initial adaptive low-pass and high-pass kernels generated for the feature coordinates i, j of the lth layer; Represents the parameters of the initial adaptive low-pass and high-pass kernels generated at the feature coordinates i, j of the lth layer at the kernel coordinates p, q; Indicates the value of the adaptive low-pass and high-pass kernels generated at the feature coordinates i, j of the lth layer at the kernel parameter coordinates p, q; E p,q Parameters representing the identity kernel at nuclear coordinates p,q.

[0045] The post-upsampling process of extending this method to the semantic segmentation model is expressed as follows:

[0046]

[0047]

[0048]

[0049] in, Represents the image details of the lth layer; Laplace filter, used to extract the input image I l The high frequency in the image, at this time the input image has been scaled to twice the size of the feature map; represents the initial kernel parameters for the l-th layer image; The model outputs the final features using twice the nearest neighbor upsampling; represents the initial adaptive kernel parameters generated for the feature coordinates i, j at the lth layer; Represents the value of the initial adaptive kernel generated at the feature coordinate i, j of the lth layer at the kernel parameter coordinate p, q; Represents the value of the adaptive low-pass kernel generated at the feature coordinates i, j of the lth layer at the kernel parameter coordinates p, q.

[0050] Utilize high-frequency information in the image to generate an adaptive low-pass filter Used to remove harmful high frequencies in the final features of the model output, resulting in accurate high-resolution output:

[0051]

[0052] in, The feature vector representing the feature after fusion of the lth layer at coordinates i, j; It represents the feature vector of the up-sampled l+1th layer fused features at coordinates i+p,j+q.

[0053] This process will double the upsampling of the final feature. Further, the above process can be repeated to achieve a higher upsampling rate.

[0054] Step 2: Train the semantic segmentation neural network model.

[0055] This method does not change the training process of the standard semantic segmentation neural network model. Instead, the semantic segmentation deep convolutional network is trained using a semantic segmentation task loss function. This loss function can use a pixel-wise cross entropy loss function, or alternatively, DICE or Lovász loss functions.

[0056] Step 3: Use the trained frequency-aware feature-fused semantic segmentation neural network model to perform semantic segmentation on the image. The final segmentation result is rich in details and precise edges.

[0057] To achieve the purpose of the present invention, the present invention further proposes an image semantic segmentation system based on frequency perception feature fusion, such as Figure 4 As shown, it includes a frequency perception feature fusion semantic segmentation neural network model building module 10, a semantic segmentation model learning module 20, and an image semantic segmentation reasoning module 30.

[0058] The frequency-aware feature fusion semantic segmentation neural network model building module 10 is used to build a semantic segmentation neural network model using frequency-aware feature fusion. This module transforms the ordinary semantic segmentation neural network model into a semantic segmentation neural network model using frequency-aware feature fusion, improving the model's ability to retain details during downsampling and its ability to utilize high-frequency information of the input image during upsampling.

[0059] The semantic segmentation model learning module 20 is used to train a semantic segmentation neural network model using frequency-aware feature fusion, enabling end-to-end model training. During this process, the frequency-aware feature fusion module learns to adaptively remove harmful high frequencies from high-level features and enhance beneficial high frequencies from low-level features, thereby improving the model's accuracy in object edge segmentation and enhancing the stability and consistency of segmentation results.

[0060] The image semantic segmentation reasoning module 30 uses the trained model to extract image features using the frequency-aware feature fusion module and obtains the final image semantic segmentation result.

[0061] The connection relationship between the above modules is as follows:

[0062] The output end of the frequency perception feature fusion semantic segmentation neural network model building module 10 is connected to the input end of the semantic high-frequency area dense sampling learning module 20.

[0063] The output end of the semantic high-frequency area dense sampling learning module 20 is connected to the input end of the image semantic segmentation reasoning module 30.

Claims

1. An image semantic segmentation method based on frequency perception feature fusion, characterized in that: The following steps are involved: Step 1: Build a semantic segmentation neural network model based on frequency-aware feature fusion; The feature fusion and upsampling in the common semantic segmentation deep neural network are replaced by frequency-aware feature fusion and frequency-aware upsampling, thereby constructing a semantic segmentation neural network model with frequency-aware feature fusion. Frequency-aware feature fusion Y l The description is as follows: in, represents low-pass filtering, Indicates high-pass filtering; X l and Y l+1 They are the features of the lth layer generated by the backbone network and the fusion features of the l+1th layer; represents upsampling; Represents the l-th layer feature after being mapped to the same number of channels by the projection function; Represents the fusion features of the l+1th layer after upsampling; represents a projection function, ensuring and Have the same number of channels; Generates spatially adaptive lowpass / highpass filtering as follows: in, The feature vector representing the feature after fusion of the lth layer is located at coordinates i, j; i, j represent the horizontal and vertical coordinates of the input features; and They are spatially varying adaptive low-pass and high-pass kernels, respectively. For each feature coordinate i, j, a high-pass and low-pass kernel will be adaptively generated; p, q represent the adaptive kernels The horizontal and vertical coordinates of the nuclear parameters; Represents the values ​​of the adaptive low-pass and high-pass kernels generated at the feature coordinates i and j at the kernel parameter coordinates p and q; The feature vector representing the feature after fusion of the l+1th layer is located at the coordinates i+p,j+q; The feature vector representing the lth layer feature at coordinates i+p,j+q; Represents the feature vector of the l-th layer feature at coordinates i, j; Ω represents the neighborhood area that the filter depends on; and Obtained by the following formula: Among them, Z l is the compressed feature; for adaptive low-pass filtering, the Softmax constraint is used to be all positive and unique; for the high-pass filter kernel, the Softmax constraint is first used to obtain the low-pass filter, and then the high-pass filter is obtained by inverting it, specifically using the identity kernel E minus the low-pass kernel; i, j are the horizontal and vertical coordinates of the input features, and p, q are the horizontal and vertical coordinates of the kernel parameters; Represents the features after fusion of the l+1th layer; Represents the l-th layer feature after being mapped by the projection function; represents the initial low-pass kernel parameters; represents the initial high-pass kernel parameters; Represents the initial adaptive low-pass and high-pass kernels generated for the feature coordinates i, j of the lth layer; Represents the parameters of the initial adaptive low-pass and high-pass kernels generated at the feature coordinates i, j of the lth layer at the kernel coordinates p, q; Indicates the value of the adaptive low-pass and high-pass kernels generated at the feature coordinates i, j of the lth layer at the kernel parameter coordinates p, q; E p,q Parameters representing the identity kernel at the nuclear coordinates p,q; The post-upsampling process of extending this method to the semantic segmentation model is expressed as follows: in, Represents the image details of the lth layer; Laplace filter, used to extract the input image I l The high frequency in the image, at this time the input image has been scaled to twice the size of the feature map; represents the initial kernel parameters for the l-th layer image; The model outputs the final features using twice the nearest neighbor upsampling; represents the initial adaptive kernel parameters generated for the feature coordinates i, j at the lth layer; Represents the value of the initial adaptive kernel generated at the feature coordinate i, j of the lth layer at the kernel parameter coordinate p, q; Represents the value of the adaptive low-pass kernel generated at the feature coordinates i, j of the lth layer at the kernel parameter coordinates p, q; Utilize high-frequency information in the image to generate an adaptive low-pass filter Used to remove harmful high frequencies in the final features of the model output, resulting in accurate high-resolution output: in, The feature vector representing the feature after fusion of the lth layer at coordinates i, j; The feature vector representing the fused features of the l+1th layer after upsampling is located at coordinates i+p,j+q; Step 2: Train the semantic segmentation neural network model; Specifically, the semantic segmentation task loss function is used to train the semantic segmentation deep convolutional network; Step 3: Use the trained frequency-aware feature fused semantic segmentation neural network model to perform semantic segmentation on the image and obtain the segmentation result.

2. The image semantic segmentation method based on frequency perception feature fusion according to claim 1, characterized in that: In step 2, the semantic segmentation task loss function includes pixel-by-pixel cross entropy loss function, DICE loss function, and Lovász loss function.

3. An image semantic segmentation system based on frequency perception feature fusion for implementing the method of claim 1, comprising a frequency perception feature fusion semantic segmentation neural network model building module (10), a semantic segmentation model learning module (20) and an image semantic segmentation reasoning module (30); in, The frequency-aware feature fusion semantic segmentation neural network model building module (10) is used to build a semantic segmentation neural network model using frequency-aware feature fusion; this module transforms the ordinary semantic segmentation neural network model into a semantic segmentation neural network model using frequency-aware feature fusion, thereby improving the model's ability to retain details during downsampling and its ability to utilize high-frequency information of the input image during upsampling. The semantic segmentation model learning module (20) is used to train a semantic segmentation neural network model using frequency-aware feature fusion, and can complete model training end-to-end. In this process, the frequency-aware feature fusion module learns to adaptively remove harmful high frequencies in high-level features and enhance beneficial high frequencies in low-level features, thereby improving the accuracy of the model in object edge segmentation and improving the stability and consistency of the segmentation results. The image semantic segmentation reasoning module (30) uses the trained model to extract image features using the frequency-aware feature fusion module and obtains the final image semantic segmentation result; The connection relationship between the above modules is as follows: The output end of the frequency perception feature fusion semantic segmentation neural network model building module (10) is connected to the input end of the semantic high frequency area dense sampling learning module (20); The output end of the semantic high-frequency region dense sampling learning module (20) is connected to the input end of the image semantic segmentation reasoning module (30).

Images