A food image segmentation method based on discrete wavelet attention network

Abstract

The application provides a food image segmentation method based on a discrete wavelet attention network, and first, in view of the problems of multiple food categories, irregular distribution, inconsistent size and fuzzy boundary information in the food image, a Food2K-ResNet trained on Food2K and obtained through transfer learning is used to extract food image features, a discrete wavelet attention network and a residual channel attention network are combined to form a double-branch network, spatial domain and frequency domain feature processing is performed, spatial features and channel features are obtained from the spatial dimension and the channel dimension respectively, and the feature dimension of the food image is highlighted and enriched; finally, a segmentation head network is constructed, different scale features processed through the two branches are subjected to convolution, feature fusion and up-sampling operations, so that the size of the finally obtained features is consistent with the size of the input image, and a high-quality and highly detailed segmentation result is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of computer vision and food computing, and particularly to a food image segmentation method based on a discrete wavelet attention network. BACKGROUND

[0002] Computer vision technology is a modern information technology that integrates recognition technology, scene reconstruction technology, image restoration technology and other advanced technology categories. At present, the development speed of computer vision technology in China is extremely fast, and in the aspect of image analysis and processing, it can classify, detect and segment the content in the image. Image semantic segmentation mainly labels the object category to which each pixel in the image belongs; compared with image classification which predicts the label of the whole image, the requirement for segmentation is usually higher. From the early traditional segmentation methods such as threshold, k-means clustering, region growing, to the current deep learning models such as FCN, PSPNet, and DeepLab series, which have achieved good results, they all use the shift-invariance and locality characteristics of convolutional networks to obtain the detailed information in the image, and perform convolution operation on the image to obtain the features of the image.

[0003] In recent years, discrete wavelet transform has been widely used in the field of vision, such as image super-resolution, image denoising processing, image semantic segmentation, etc. Discrete wavelet transform converts image processing from spatial domain to frequency domain, greatly reducing the parameter amount and even complexity of calculation, and frequency domain analysis has always been a powerful tool for image processing. Effectively using image frequency domain information can greatly improve the performance of image restoration algorithms. Some methods use the effectiveness of frequency domain information to embed frequency information into the network structure. And a wavelet residual network is proposed, which finds that neural networks can benefit from wavelet sub-band learning, and uses high-frequency information of wavelet sub-band to strengthen the texture details of the image.

[0004] Obesity is a chronic disease that is harmful to personal physical and mental health and society. According to western medical theory, the cause of obesity is the imbalance between calorie intake and consumption, which also leads to an increase in the risk of chronic metabolic diseases such as diabetes. Traditionally, nutritionists try to solve these problems by looking at images of food eaten by patients. Food image segmentation aims to distinguish various ingredients in various foods, which is crucial for food safety because estimating calories and other nutrients is important for human health and sustainable development. However, current image segmentation methods have poor segmentation performance on food image datasets due to the diversity of their appearance and the conditional differences between food ingredients and daily props, and insufficient feature extraction capability for food images. Therefore, a high-performance segmentation model can accurately identify and locate different food ingredients, which also makes necessary preparations for subsequent nutrition assessment and component estimation. SUMMARY

[0005] The technical problem solved by the present application is: for the problems of multiple food categories, irregular distribution, inconsistent size and blurred boundary information in food images, a food image segmentation method based on discrete wavelet attention network is provided to solve the problem of texture detail loss of image features in feature processing, and the frequency domain features and channel features are weighted and processed by combining the attention mechanism, so that the image semantic information is better preserved. Combining discrete wavelet transform and attention mechanism can better realize the frequency domain conversion of image texture features, strengthen high-frequency information and suppress low-frequency information, realize the fusion of spatial domain and frequency domain features, and obtain high-quality and highly detailed segmentation results.

[0006] To solve the above technical problems, one technical solution adopted by the present application is: a discrete wavelet attention network and a residual channel attention network are combined to form a double-branch network for spatial domain and frequency domain feature processing, spatial features and channel features are obtained from spatial dimensions and channel dimensions respectively to highlight and enrich the feature dimensions of food images; finally, a segmentation head network is constructed to perform convolution, feature fusion and upsampling operations on the different scale features processed by the two branches so that the size of the final obtained features is consistent with the size of the input image, which specifically includes the following steps:

[0007] (1) For the characteristics of food images and insufficient feature extraction, a Food2K-ResNet is used as a food image feature extractor through transfer learning, which is mainly trained on Food2K using the ResNet model to make the feature extraction capability more consistent with the characteristics of food images. Then a discrete wavelet attention network and a residual channel attention network are combined to form a double-branch network to realize weighted processing of image features from spatial dimensions and channel dimensions, highlight and enrich the feature representation of food images, and make the detailed features in food images more prominent, so that the segmentation result boundary is more clear;

[0008] (2) For the discrete wavelet attention branch network, it follows the U-Net network structure, processes the features extracted by Food2K-ResNet, decomposes the features into corresponding low-frequency features and high-frequency features through discrete wavelet transform, then uses the spatial attention mechanism to realize weighted strengthening processing of the high-frequency features, suppresses the low-frequency features, connects the low-frequency features and the convolution output as the down-sampling features, and adds the high-frequency features to the discrete wavelet transform up-sampling module through the jump connection;

[0009] (3) For the residual channel attention network, it mainly uses residual connection and channel attention mechanism to obtain the channel features of food images, realizes the fusion of channel features and original input features, and highlights the channel information of the original food images;

[0010] (4) Using the food image feature vectors obtained in steps 2 and 3 respectively, respectively performing convolution operation, so that the size of each feature is consistent, then performing feature fusion, forming a new food image feature with multi-dimensional and multi-domain representation, containing different semantic details of the image, finally using the constructed decoding head (segmenter) to process the fused features, so that the feature size is consistent with the input image. On this basis, use the structural similarity loss function (SSIM) to act on the discrete wavelet attention branch; use the cross-entropy loss function to act on the residual channel attention branch, construct a joint loss function, and continuously optimize the network parameters through back propagation to reduce the error loss and predict the optimal segmentation result with accurate details consistent with the input image.

[0011] In step 1, first, the traditional ResNet network as a feature extractor cannot be fully applicable to food image features in this specific field, so the Food2K-ResNet trained on the Food2K food dataset is used as the backbone network, and a double-branch network is designed; one branch is a discrete wavelet attention network, and the other branch is a residual channel attention network, which realizes the weighted processing of image features from spatial dimension and channel dimension, enriching the semantic feature representation of food images.

[0012] In step 2, for the discrete wavelet attention branch network, its design follows the U-Net network structure, which processes the features extracted by Food2K-ResNet, decomposes them into corresponding 3 low-frequency features and 1 high-frequency feature through discrete wavelet transform, and at the same time, performs convolution, normalization, pooling and activation layer processing on the input features to obtain spatial domain image features.

[0013] The low-frequency component is connected with the convolution output as the down-sampling feature, and the low-frequency feature is suppressed to retain more high-frequency detail information, and the high-frequency component is added to the discrete wavelet transform up-sampling module through the jump connection, the spatial attention mechanism acts on the high-frequency feature, realizes the weighted strengthening processing of the high-frequency feature, and at the same time, the low-frequency component is connected with the convolution output as the down-sampling feature. In step 3, the image features obtained in step 2 are fused from top to bottom, and 1x1 convolution is also used for image size alignment during the downward fusion process, and a four-layer feature pyramid is constructed, each layer of which contains the semantic information of the previous layers of features.

[0014] In step 4, the segmentation decoder is constructed, the feature fusion is performed according to the four different scale image features finally obtained in the previous three steps, a new feature vector with multi-scale and containing different semantic details of the image is formed, and finally the upsampling operation is performed in order to make the feature scale same as the size of the input image. The error between the predicted value and the true value is calculated using the joint loss function, the error is back propagated, and all parameters in the encoder are optimized, so as to generate the optimal segmentation result with food class boundary segmentation accuracy.

[0015] The beneficial effects of the present application are: in the process of food image segmentation, the present application uses the dual branches of discrete wavelet attention and residual channel attention to realize feature weighting processing of frequency domain information, better preserves image semantic features, preserves high-frequency and low-frequency information, realizes fusion of spatial domain and frequency domain features, and thus obtains a high-quality and highly detailed segmentation result. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The food image segmentation process based on the discrete wavelet attention network is described.

[0017] Figure 2 The discrete wavelet attention network model is described.

[0018] Figure 3 The structure diagram of the spatial attention mechanism, the channel attention mechanism and the discrete wavelet attention module is described. Figure 3 (a) is a discrete wavelet attention module, Figure 3 (b) is a spatial attention mechanism, Figure 3 (c) is a channel attention mechanism. DETAILED DESCRIPTION

[0019] The present application will be described below in conjunction with the drawings and specific embodiments. Among them Figure 1 The food image segmentation process based on the discrete wavelet attention network is described. Figure 2 The discrete wavelet attention network model is described. Figure 3 The structure diagram of the spatial attention mechanism, the channel attention mechanism and the discrete wavelet attention module is described.

[0020] As Figures 1-2 shown, the food image segmentation process based on the discrete wavelet attention network of the present application is as follows:

[0021] (1) The food image segmentation method based on the discrete wavelet attention network in the application firstly aims at the inconsistent problem of the size distribution of food images, uses the Food2K-ResNet trained on the Food2K food dataset as the backbone network, so that in the application to food image segmentation, more characteristics F conforming to the characteristics of food images can be extracted.

[0022] (2) A double-branch network combining discrete wavelet attention network and residual channel attention network is constructed, wherein the discrete wavelet attention branch is designed as a U-Net structure, which includes a convolution layer, a normalization layer, an activation layer, a discrete wavelet transform module and an attention module, realizing the transformation and combination between the spatial domain and the frequency domain of the characteristics. In the discrete wavelet transform, there are four filters, namely one low-pass filter f LL , three high-pass filters f LH , f HL , f HH . These filters have fixed parameters and perform stride=2 convolution operation in the transformation process. Therefore, by convolution with each filter, the image or feature map can be decomposed into x LL , x LH , x HL and x HH four subbands. We can express x LL as where denotes convolution operation, x is the input signal, and ↓2 denotes down-sampling by a factor of 2,

[0023]

[0024]

[0025] After two-dimensional Haar wavelet transform, the (i, j) value of x LL can be defined as: x LL (i, j) = x(2i-1, 2j-1) + x(2i-1, 2j) + x(2i, 2j-1) + x(2i, 2j). After obtaining the four subbands, the spatial attention mechanism of the discrete wavelet attention module is used to strengthen the high-frequency features and suppress the low-frequency features. The spatial attention mechanism aims to enhance the feature expression of the key region. In essence, it transforms the spatial information in the original picture to another space through a spatial conversion module and retains the key information, generates a weight mask for each position and weights the output, thereby enhancing the specific target region of interest while weakening the irrelevant background region. The feature map x∈R C×H×W is sent to the SA spatial attention module, as follows Figure 3The spatial attention mechanism shown in (b) obtains three feature maps Q, K and V through three convolution layers with 1x1 filters. Then the spatial attention weighting is performed to obtain M wl and M wh ,

[0026]

[0027] where M wl represents the output features of the three low frequencies after the spatial attention mechanism, M wh is the output features of the high frequency features after the attention mechanism. Attention(Q, K, V) is an attention module, and softmax is an activation function, d k represents the feature dimension.

[0028] Then, as shown in (a) of Figure 3 , M wl and the output of the original input features after the convolution module are added pixel by pixel to form M wl1 , M wl1 represents the first discrete wavelet module, M wh is directly connected to the corresponding discrete wavelet up-sampling module through a jump connection.

[0029] (3) In order to fully exert the mutual dependence between the channels of the feature map, the feature representation is combined with the channel information, and a channel attention network is introduced to obtain the mutual dependence between the channels, as shown in the channel attention mechanism shown in (c) of Figure 3 . Compared with the spatial attention mechanism, the channel operation directly performs a scale transformation operation on the original feature F to obtain three feature maps {Q, K, V} ∈ R C×N , and matrix multiplication is performed between the transpose of Q and V to obtain X'. Then the softmax function is used to calculate the attention map X.

[0030] X = softmax(X') = softmax(Q × K T )

[0031] After that, matrix multiplication is applied between X and the transpose of V, and their results are scaled to R C×H×W . Through a residual connection, the original input feature F and the output feature after the channel attention module are concatenated to obtain the final output F c .

[0032] F c = Concat(F, X·V T )

[0033] The final feature of each channel includes a weighted sum of the original feature and all channel features, models semantic dependency from channel dimension, and can maintain the relationship between channel mappings.

[0034] (4) Using the food image feature vector obtained by the above steps, respectively, convolution operation is performed to make the size of each feature consistent, then feature fusion is performed to form a new food image feature with multi-dimensional and multi-domain representation, which contains different semantic details of the image, and finally the decoding head is used to process the fused features to make the feature size consistent with the input image. On this basis, the structural similarity loss function (SSIM) is used to act on the discrete wavelet attention branch; the specific formula is as follows:

[0035]

[0036] Where x and y represent the predicted image and the real image (Ground truth) respectively, μ., σ. and σ.. represent the mean, standard deviation and covariance of the image respectively, · can represent x or y, C is a constant term to prevent the divisor from being 0. A higher SSIM indicates that the two images are more similar to each other, and for the same image, SSIM is equal to 1. The loss function of SSIM can be written as:

[0037]

[0038] Where N represents the number of images, SSIM(x i , y i ) represents the structural similarity loss of images x i and y i .

[0039] The cross-entropy loss function is used to act on the residual channel attention branch, and the joint loss function L total is constructed. Through back propagation, the network parameters are continuously optimized to reduce the error loss, and the optimal segmentation result with accurate details is predicted, which is consistent with the input image.

[0040] L total =β·L SSIM +L cross

[0041] Where β is a hyperparameter, L cross is the cross-entropy loss function, which is continuously adjusted with the continuous iteration and optimization of the network. The Adam gradient descent algorithm is applied to optimize the parameters in the image encoder until the loss function converges to the maximum number of iterations in the loop, and the final trained model parameters are obtained to generate the optimal segmentation result with accurate food class boundary segmentation.

Claims

1. A food image segmentation method based on discrete wavelet attention network, characterized in that, Includes the following steps: Step 1: Use Food2K-ResNet as a feature extractor for food images through transfer learning. This feature extractor uses the ResNet model to extract food image features on Food2K. Then, it combines the discrete wavelet attention branch network and the residual channel attention branch network to form a dual-branch network, which realizes the weighted processing of image features from the spatial dimension and the channel dimension. Step 2: For the Discrete Wavelet Attention Branch Network, its design follows the U-Net network structure. The features extracted by Food2K-ResNet are processed and decomposed into corresponding low-frequency and high-frequency features through discrete wavelet transform. Then, the spatial attention mechanism is used to implement weighted enhancement of high-frequency features and suppression of low-frequency features. The low-frequency features are connected to the convolution output to form downsampled features. The high-frequency features are added to the discrete wavelet transform upsampling module through skip connections to obtain the spatial features of the food image. Step 3: For the residual channel attention network, use residual connections and channel attention mechanisms to obtain the channel features of the food image; Step 4: Using the spatial and channel features of the food image obtained in Steps 2 and 3, convolution operations are performed to ensure that the scale of different features is consistent. Then, feature fusion is performed to form multi-dimensional, multi-domain food image features that also contain different semantic details of the image. Finally, the constructed decoding head is used to process the fused features to ensure that the feature scale is consistent with the input image. Based on this, the structural similarity loss function is applied to the discrete wavelet attention branch, and the cross-entropy loss function is applied to the residual channel attention branch to construct a joint loss function. The network parameters are optimized through backpropagation to reduce error loss and predict the optimal segmentation result with accurate details consistent with the input image.

2. The food image segmentation method based on discrete wavelet attention network according to claim 1, characterized in that: The food image processing method in step 1 is as follows: (1) First, for the food image domain, we use the Food2K-ResNet trained on the Food2K food dataset as the backbone network and design a dual-branch network. (2) One branch is a discrete wavelet attention network, and the other branch is a residual channel attention network.

3. The food image segmentation method based on discrete wavelet attention network according to claim 1, characterized in that: In step 2, a discrete wavelet attention branch network is constructed. The specific construction method is as follows: (1) For the discrete wavelet attention branch network, the features extracted by Food2K-ResNet are processed and decomposed into three low-frequency features and one high-frequency feature by discrete wavelet transform. At the same time, the extracted features are processed by convolution, normalization, pooling and activation layers to obtain the convolution output. (2) Then, the spatial attention mechanism is used to apply to the frequency domain features to achieve weighted enhancement of high-frequency features while suppressing low-frequency features, so that the features retain more high-frequency details. The low-frequency components are connected to the convolution output as downsampled features, and the high-frequency features are added to the discrete wavelet transform upsampling module through skip connections.

4. The food image segmentation method based on discrete wavelet attention network according to claim 1, characterized in that: In step 3, for the residual channel attention network, residual connections and channel attention mechanisms are used to obtain the channel features of the food image.

5. The food image segmentation method based on discrete wavelet attention network according to claim 1, characterized in that: In step 4, the specific construction method for feature fusion is as follows: (1) Using the food image features obtained in steps 2 and 3 respectively, perform 1x1 convolution operations to make the scale of each feature consistent, and then perform feature fusion to form multi-dimensional, multi-domain food image features that also contain different semantic details of the image. (2) Use the constructed decoding head to perform fusion feature processing so that the feature scale is consistent with the input image. On this basis, use the structural similarity loss function to apply to the discrete wavelet attention branch; use the cross-entropy loss function to apply to the residual channel attention branch to construct a joint loss function. (3) Apply the Adam gradient descent algorithm to optimize the parameters in the encoder until the joint loss function converges to the maximum number of iterations, generating the optimal segmentation result with accurate food category boundary segmentation.

Images