A food image semantic segmentation method based on mask classification

Abstract

The application discloses a food image semantic segmentation method based on mask classification, and relates to the technical field of food calculation; input pictures are input to a backbone network and a pixel decoder to generate four layers of different scale features; an attention query vector is pre-generated according to the first three layers of the generated scale features, and is input to an adaptive long-distance decoder to obtain a decoded query vector, which is input to a correlation perception classification enhancement module CACE for processing to obtain a category enhancement feature; the fourth layer of features output by the pixel decoder is processed by a multi-foreground mask enhancement module MFME to obtain an edge enhancement feature; and the output features processed by the CACE and the MFME are input to a post-processing module to obtain a final output prediction and calculate a total loss function of the whole model; the food image semantic segmentation method based on mask classification is used to efficiently improve segmentation precision, optimize long-distance dependence modeling and reduce computational complexity.

Description

Technical Field

[0001] This invention relates to the field of food computation technology, and in particular to a food image semantic segmentation method based on mask classification. Background Technology

[0002] Food image segmentation aims to distinguish food items in an image from the background or other food items. From the perspective of image segmentation paradigms, the techniques in the field of food image segmentation are mainly divided into pixel-wise classification schemes and mask classification schemes. Pixel-wise classification methods have inherent limitations. Small objects or complex details often lack sufficient local feature strength and are difficult to effectively capture global information or contextual relationships. Unlike pixel-wise classification, mask classification methods focus on identifying and segmenting different object instances by generating a binary mask for each instance, thereby representing its region in the image. Mask classification especially focuses on classifying the entire object or region, utilizing global features and contextual information. This method enhances the ability to handle small objects and complex details, while reducing noise interference by considering information from the entire region.

[0003] Existing semantic segmentation methods for food images mainly suffer from the following problems: First, due to the wide variety and diverse shapes of food, traditional pixel-by-pixel classification methods (such as FCN and DeepLabV3+) struggle to accurately distinguish the boundaries between different foods, leading to classification errors. Furthermore, occlusion, mixing, and irregular shapes among foods further increase the segmentation difficulty. Second, current methods are insufficient in modeling long-distance dependencies. Many food images contain multiple scattered ingredients, while CNNs primarily rely on local features, making it difficult to capture long-distance ingredient relationships. While Transformer-based methods can model global dependencies, their high computational complexity slows down inference. Additionally, existing mask classification methods (such as Mask...)... R-CNN and Mask2Former still have high error rates in food category differentiation, especially when the features between categories are similar (such as apricots and peaches, or different types of bread). The main reason is the lack of an effective category relationship modeling mechanism, which fails to make full use of the relevant information between food categories to improve classification accuracy. Finally, computational cost and efficiency issues remain important factors limiting practical applications. Traditional Transformer decoders have high computational complexity and slow inference speed. Many methods consume a lot of resources in complex food scenarios and it is difficult to balance segmentation accuracy and inference speed. Summary of the Invention

[0004] The purpose of this invention is to provide a semantic segmentation method for food images based on mask classification. This method introduces a multi-foreground mask enhancement module, an adaptive long-distance decoder, and a category-related perception classification enhancement module, which can efficiently improve segmentation accuracy, optimize long-distance dependency modeling, and reduce computational complexity.

[0005] To achieve the above objectives, this invention provides a food image semantic segmentation method based on mask classification, comprising the following steps:

[0006] S1. Input the image to the backbone network and pixel decoder to generate four layers of different scale features;

[0007] S2. Based on the scale features of the first three layers generated in S1, pre-generate the attention query vector Q and input it into the adaptive long-range decoder ALRD to obtain the decoded query vector Q'.

[0008] S3. The Q' obtained in S2 is processed through the correlation-aware classification enhancement module CACE to obtain the category-enhanced features.

[0009] S4. The fourth layer features output from the pixel decoder in S1 are then processed by the Multi-Foreground Mask Enhancement (MFME) module to obtain edge enhancement features.

[0010] S5. Input the output features processed by S3 and S4 into the post-processing module to obtain the final output prediction;

[0011] S6. Calculate the total loss function L for the entire model.

[0012] Preferably, the internal structure and specific workflow of the adaptive long-distance decoder in S2 are as follows:

[0013] S21. The adaptive long-range decoder contains several decoding blocks. Each decoding block receives three inputs: the attention mask output from the previous decoding block, the corresponding layer's image features, and the previous decoding block's output vector query. The image features and query undergo different linear transformations to obtain the attention query vector Q, attention key vector K, and attention value vector V. The adaptive long-range decoder also includes a decoding head that performs layer normalization and dimensionality transpose on the input. Then, it generates class prediction and mask embedding through different linear transformations. The mask embedding and image feature map are multiplied by tensor to generate mask prediction. Finally, the attention mask is obtained by interpolation of the mask prediction. Q, K, and V are input together with the mask into the mask attention layer, which outputs a multi-head self-attention output vector X. MSAout as follows:

[0014] X MSAout =softmax(mask+QK) T V;

[0015] Among them, K T This is the transpose of the attention key vector K.

[0016] S22, then X MSAout The sum of the residuals of Q and X is normalized through a layer to obtain the transformer output vector or converter output vector X. TFout :

[0017] X TFout =LayetNorm(X MSAout +Q);

[0018] Among them, LayerNorm is the layer normalization layer;

[0019] S23, Input X TFout The input to the State-Space Model (SSM) passes through a feedforward neural network layer, yielding three output vectors: A, x, and z, as follows:

[0020] A = X TFout W(A) T ;

[0021] x = X TFout W(x) T ;

[0022] z = X TFout W(z) T ;

[0023] Among them, W(A) T , W(x) T , W(z) T Let A, x, and z be the linear transformation matrices corresponding to A, x, and z;

[0024] S24. Pass the vector x from S23 through a one-dimensional convolutional layer to obtain x. c x c The projection is divided into three parts and then processed by an activation function to obtain vectors X, B, and C. The specific process is as follows:

[0025] x c =conv1d(x);

[0026]

[0027] Here, conv1d() is a one-dimensional convolution, and projection() is a projection layer. For activation function layers;

[0028] S24. Vectors X, B, and C are input together with vector A into the state-space model layer, and the output is vector y. Then, y is multiplied by vector z after the activation function and passed through a normalization layer to obtain y'. n y n The SSM block output S is obtained after passing through a feedforward neural network layer. out ;

[0029] y = SSM(A,B,C,X);

[0030]

[0031] S out =FFN(y n );

[0032] Among them, SSM(A,B,C,X) is the state-space model layer, groupnorm() is the group normalization layer, and FFN() is the feedforward neural network layer.

[0033] S25, S out With X TFout The residuals are summed and layer normalized to obtain the output vector X of the state-space model. SSMout X after passing through the feedforward neural network layer SSMout (State-space model output vector) and X SSMout The final output F is obtained by summing the residuals of the (state-space model output vector) and then normalizing them through layers. out And as the input to the next decoder, the input attention mask of the next layer is determined by F. out The specific calculation process, obtained through the decoding head, is as follows:

[0034]

[0035] F out =LayerNorm(FFN(X) SSMout )+X SSMout ).

[0036] Preferably, the auxiliary loss function L is calculated based on the output of the adaptive long-range decoder of S2. aux The process is as follows:

[0037]

[0038] in, To assist the mask loss function, As the auxiliary classification loss function, λ cls These are the scaling parameters set manually.

[0039] Preferably, the specific process in S3 is as follows:

[0040] S31. Input Q' into the CACE module and transform it into classification feature E through a fully connected layer. cls and mask feature E mask E cls Three features of different sizes, F1, F2, and F3, are generated through three one-dimensional convolutions, as shown below:

[0041] F1 = Conv1D(E cls );

[0042] F2=Conv1D(E cls );

[0043] F3=Conv1D(E cls );

[0044] Wherein, Conv1D() is a one-dimensional convolutional layer;

[0045] S32. After cross-product of F1 and F2 in S31, apply the Softmax function to generate the correlation map M. The specific calculation process is as follows:

[0046]

[0047] in, This is the transpose of F1;

[0048] S33. Multiply the correlation graph M of S32 by the equilibrium parameter α, and then perform a cross product with F3 to obtain the final result. The specific calculation process is as follows:

[0049]

[0050] in, This is the transpose of F3;

[0051] S34, Calculate the CACE module output. Classification loss loss function L cls :

[0052] L cls =-∑ i C gt ·log(C pre )-(1-C gt )·log(1-C pre );

[0053] Among them, C pre To predict the foreground classification one-hot vector, C gt This is a one-hot vector for classifying the true foreground.

[0054] Preferably, the specific process of S4 is as follows:

[0055] The fourth-layer scale features obtained from S41 and S1, after passing through a pooling layer, are used as the input to the MFME to enhance the edge feature E. pixel E pixel The channel attention features F are obtained by passing the data through a channel attention layer, a downsampling convolutional layer, and a position attention layer, respectively.channel downsampling convolutional features F conv Edge features F edge ;

[0056] F channel =ChannelAttention(E pixel );

[0057] F conv =Concat(E pixel );

[0058] F edge =SpatialAttention(E pixel );

[0059] Among them, ChannelAttention() is the channel attention layer function, Concat() is the feature concatenation function, and SpatialAttention() is the position attention layer function;

[0060] 42. Take F obtained from S41 channel and F conv The spliced ​​features F are obtained by splicing. concat F concat The calculation formula is as follows:

[0061] F concat =Concat(F channel ,F conv );

[0062] S43, F obtained in S41 edge F obtained from S42 concat The summation yields the final output of the MFME module. The calculation formula is as follows:

[0063]

[0064] S44, Order F edge The prediction convolutional layer produces prediction maps F for each foreground edge. edgeout F edgeout The calculation formula is as follows:

[0065] F edgeout =PreConv(F edge );

[0066] Where PredConv() is the prediction convolutional layer;

[0067] S45. Edge prediction map F obtained from S44 edgeout Solving the edge loss function L using the true edge values edThe specific calculation formula is as follows:

[0068] L ed =-∑ i GT i ·log(F edge,i )-(1-GT i )·log(1-F edge,i );

[0069] Among them, GT i Let i represent the i-th pixel in the real foreground, i = 1, 2, 3, ..., and log() denote the logarithmic function, F. edge,i This represents the i-th pixel in the predicted edge map.

[0070] Preferably, the specific process of S5 is as follows:

[0071] S51, the final output of the CACE module The classification enhancement unit is input into the post-processing module;

[0072] S52, Connect the residual obtained in S31 to the output E mask The final output of the MFME module The cross product is used to obtain the foreground mask prediction input, which is then fed into the boundary enhancement unit of the post-processing module.

[0073] S53. Perform a cross product of the outputs obtained from S51 and S52 to obtain the final prediction result.

[0074] Preferably, the classification enhancement unit in S51 performs the following steps sequentially: fusing the multilayer perceptron output, interpolation, and the sigmoid function.

[0075] The boundary enhancement unit in S52 is a process of sequentially performing layer normalization, fully connected layers, and the Softmax function.

[0076] Preferably, based on the foreground mask prediction obtained in S52, its mask loss function L is calculated. mask :

[0077] L mask =λ ce L ce +λ dice L dice ;

[0078] Among them, L ce Let L be the cross-entropy loss function. dice Let λ be the Dice loss function. ce , λ dice These are the scaling parameters set manually.

[0079] Preferably, the cross-entropy loss function L ceThe calculation formula is as follows:

[0080] L ce =-∑ i GT i ·log(F pre,i )-(1-GT i )·log(1-F pre,i );

[0081] Among them, F pre,i To predict the i-th pixel in the foreground;

[0082] Dice loss function L dice The calculation formula is as follows:

[0083]

[0084] Among them, GT j For the j-th pixel in the real foreground, F pre,j To predict the j-th pixel in the foreground, j = 1, 2, 3, ...

[0085] Preferably, according to L mask L cls L ed L aux The total loss function L of the entire model is obtained as follows:

[0086] L = L mask +λ cls L cls +λ ed L ed +L aux ;

[0087] Where, λ ed These are parameters that are dynamically adjusted during model training, and their adjustment formulas are as follows:

[0088]

[0089] Therefore, the food image semantic segmentation method based on mask classification described above has the following advantages compared with the prior art:

[0090] 1. The segmentation method of this application effectively improves the accuracy of boundary segmentation. It designs a multi-foreground mask enhancement module, introduces a boundary supervision mechanism, effectively optimizes mask generation, reduces segmentation errors caused by food occlusion and overlap, and makes the boundaries clearer.

[0091] 2. The segmentation method of this application optimizes the mask classification accuracy, designs a category correlation-aware classification enhancement mechanism, and uses category correlation relationship modeling to optimize mask classification, thereby improving the accuracy of the model in category differentiation, especially for food types with similar visual features (such as different types of fruits, bread, etc.).

[0092] 3. The segmentation method of this application achieves efficient long-distance dependency modeling capability. It introduces a state space model and designs an adaptive long-distance decoder. Its advantage is that by replacing the original self-attention layer, it solves the problem that the state space model cannot handle variable-length sequences. On the other hand, it improves the modeling capability of the model while reducing the number of model parameters, reduces the computational complexity of the original Transformer decoder, and improves the inference speed of the model while ensuring high-precision segmentation, making it more suitable for practical application scenarios.

[0093] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0094] Figure 1 This is an overall structural diagram of a food image semantic segmentation method based on mask classification according to the present invention;

[0095] Figure 2 This is a flowchart of an adaptive long-distance decoder for a food image semantic segmentation method based on mask classification according to the present invention. Detailed Implementation

[0096] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed when in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0097] Example

[0098] like Figures 1-2 As shown, the present invention provides a food image semantic segmentation method based on mask classification, comprising the following steps:

[0099] S1. Input the image to the backbone network and pixel decoder to generate four layers of different scale features;

[0100] S2. Based on the scale features of the first three layers generated in S1, pre-generate the attention query vector Q and input it into the adaptive long-range decoder ALRD to obtain the decoded query vector Q'.

[0101] The internal structure and specific workflow of the adaptive long-range decoder in S2 are as follows:

[0102] S21. The adaptive long-range decoder contains several decoding blocks. Each decoding block receives three inputs: the attention mask output from the previous decoding block, the corresponding layer's image features, and the previous decoding block's output vector query. The image features and query undergo different linear transformations to obtain the attention query vector Q, attention key vector K, and attention value vector V. The adaptive long-range decoder also includes a decoding head that performs layer normalization and dimensionality transpose on the input. Then, it generates class prediction and mask embedding through different linear transformations. The mask embedding and image feature map are multiplied by tensor to generate mask prediction. Finally, the attention mask is obtained by interpolation of the mask prediction. Q, K, and V are input together with the mask into the mask attention layer, which outputs a multi-head self-attention output vector X. MSAout as follows:

[0103] X MSAout =softmax(mask+QK) T V;

[0104] Among them, K T This is the transpose of the attention key vector K.

[0105] S22, then X MSAout The sum of the residuals of Q and X is normalized through a layer to obtain the transformer output vector or converter output vector X. TFout :

[0106] X TFout =LayerNorm(X MSAout +Q);

[0107] Among them, LayerNorm is the layer normalization layer;

[0108] S23, Input X TFout The input to the State-Space Model (SSM) passes through a feedforward neural network layer, yielding three output vectors: A, x, and z, as follows:

[0109] A = X TFout W(A) T ;

[0110] x = X TFout W(x) T ;

[0111] z = X TFout W(z) T ;

[0112] Among them, W(A) T , W(x) T , W(Z) T Let A, x, and z be the linear transformation matrices corresponding to A, x, and z;

[0113] S24. Pass the vector x from S23 through a one-dimensional convolutional layer to obtain x. c x c The projection is divided into three parts and then processed by an activation function to obtain vectors X, B, and C. The specific process is as follows:

[0114] x c =conv1d(x);

[0115]

[0116] Here, conv1d() is a one-dimensional convolution, and projection() is a projection layer. For activation function layers;

[0117] S24. Vectors X, B, and C are input together with vector A into the state-space model layer, and the output is vector y. Then, y is multiplied by vector z after the activation function and passed through a normalization layer to obtain y'. n y n The SSM block output S is obtained after passing through a feedforward neural network layer. out ;

[0118] y = SSM(A,B,C,X);

[0119]

[0120] S out =FFN(y n );

[0121] Among them, SSM(A,B,C,X) is the state-space model layer, groupnorm() is the group normalization layer, and FFM() is the feedforward neural network layer.

[0122] S25, S out With X TFout The residuals are summed and layer normalized to obtain the output vector X of the state-space model. SSMout X after passing through the feedforward neural network layer SSMout (State-space model output vector) and X SSMout The final output F is obtained by summing the residuals of the (state-space model output vector) and then normalizing them through layers. out And as the input to the next decoder, the input attention mask of the next layer is determined by F. out The specific calculation process, obtained through the decoding head, is as follows:

[0123]

[0124] F out =LayerNorm(FFN(X) SSMout )+X SSMout );

[0125] S3. Process Q′ obtained in S2 through the correlation-aware classification enhancement module CACE to obtain category-enhanced features.

[0126] S31. Input Q' into the CACE module and transform it into classification feature E through a fully connected layer. cls and mask feature E mask E cls Three features of different sizes, F1, F2, and F3, are generated through three one-dimensional convolutions, as shown below:

[0127] F1 = Conv1D(E cls );

[0128] F2=Conv1D(E cls );

[0129] F3 = Conv1D(E cls );

[0130] Wherein, Conv1D() is a one-dimensional convolutional layer;

[0131] S32. After cross-product of F1 and F2 in S31, apply the Softmax function to generate the correlation map M. The specific calculation process is as follows:

[0132]

[0133] in, This is the transpose of F1;

[0134] S33. Multiply the correlation graph M of S32 by the equilibrium parameter α, and then perform a cross product with F3 to obtain the final result. The specific calculation process is as follows:

[0135]

[0136] in, This is the transpose of F3;

[0137] S34, Calculate the CACE module output. Classification loss loss function L cls :

[0138] L cls =-∑ i Cgt ·log(C pre )-(1-C gt )·log(1-C pre );

[0139] Among them, C pre To predict the foreground classification one-hot vector, C gt Classify one-hot vectors from the real foreground;

[0140] S4. The fourth layer features output from the pixel decoder in S1 are then processed by the Multi-Foreground Mask Enhancement (MFME) module to obtain edge enhancement features.

[0141] The fourth-layer scale features obtained from S41 and S1, after passing through a pooling layer, are used as the input to the MFME to enhance the edge feature E. pixel E pixel The channel attention features F are obtained by passing the data through a channel attention layer, a downsampling convolutional layer, and a position attention layer, respectively. channel downsampling convolutional features F conv Edge features F edge ;

[0142] F channel =ChannelAttention(E pixel );

[0143] F conv =Cconcat(E pixel );

[0144] F edge =SpatialAttention(E pixel );

[0145] Among them, ChannelAttention() is the channel attention layer function, Concat() is the feature concatenation function, and SpatialAttention() is the position attention layer function;

[0146] 42. Take F obtained from S41 channel and F conv The spliced ​​features F are obtained by splicing. concat F concat The calculation formula is as follows:

[0147] F concat =Concat(F channel ,F conv );

[0148] S43, F obtained in S41 edge F obtained from S42concat The summation yields the final output of the MFME module. The calculation formula is as follows:

[0149]

[0150] S44, Order F edge The prediction convolutional layer produces prediction maps F for each foreground edge. edgeout F edgeout The calculation formula is as follows:

[0151] F edgeout =PredConv(F edge );

[0152] Where PredConv() is the prediction convolutional layer;

[0153] S45. Edge prediction map F obtained from S44 edgeout Solving the edge loss function L using the true edge values ed The specific calculation formula is as follows:

[0154] L ed =-∑ i GT i ·log(F edge,i )-(1-GT i )·log(1-F edge,i );

[0155] Among them, GT i Let i represent the i-th pixel in the real foreground, i = 1, 2, 3, ..., and log() denote the logarithmic function, F. edge,i This represents the i-th pixel in the predicted edge map;

[0156] S5. Input the output features processed by S3 and S4 into the post-processing module to obtain the final output prediction;

[0157] S51, the final output of the CACE module The input is fed to the classification enhancement unit of the post-processing module; the classification enhancement unit performs the following steps in sequence: fusing the multilayer perceptron output, interpolation, and the sigmoid function.

[0158] S52, Connect the residual obtained in S31 to the output E mask The final output of the MFME module The cross product is used to obtain the foreground mask prediction input, which is then fed into the boundary enhancement unit of the post-processing module. The boundary enhancement unit is a process of sequentially performing layer normalization, fully connected layer, and Softmax function.

[0159] S53. Perform a cross product of the outputs obtained from S51 and S52 to obtain the final prediction result.

[0160] S6. Calculate the total loss function L for the entire model;

[0161] The auxiliary loss function L is calculated based on the output of the adaptive long-range decoder of S2. aux The process is as follows:

[0162]

[0163] in, To assist the mask loss function, As the auxiliary classification loss function, λ cls Scaling parameters set manually;

[0164] Based on the foreground mask prediction obtained from S52, calculate its mask loss function L. mask :

[0165] L mask =λ ce L ce +λ dice L dice ;

[0166] Among them, L ce Let L be the cross-entropy loss function. dice Let λ be the Dice loss function. ce , λ dice These are the scaling parameters set manually.

[0167] Cross-entropy loss function L ce The calculation formula is as follows:

[0168] L ce =-∑ i GT i ·log(F pre,i )-(1-GT i )·log(1-F pre,i );

[0169] Among them, F pre,i To predict the i-th pixel in the foreground;

[0170] Dice loss function L dice The calculation formula is as follows:

[0171]

[0172] Among them, GT j For the j-th pixel in the real foreground, F pre,j To predict the j-th pixel in the foreground, j = 1, 2, 3, ...;

[0173] According to L mask L cls L ed L aux The total loss function L of the entire model is obtained as follows:

[0174] L = L mask +λ cls L cls +λ ed L ed +L aux ;

[0175] Where, λ ed These are parameters that are dynamically adjusted during model training, and their adjustment formulas are as follows:

[0176]

[0177] In practical applications, for an input image, it first passes through a backbone network and a pixel decoder to generate four layers of features at different scales. Then, the pre-generated attention query vector Q is input into an adaptive long-range decoder consisting of multiple adaptive long-range blocks concatenated together. Each block receives the output Q from the previous block and features at different scales. After Q passes through the adaptive long-range decoder, the output Q' is obtained. Then, Q' is enhanced by correlation-aware classification (CACE), while the fourth layer of features output by the pixel decoder is enhanced by multiple foreground masking (MFME). The outputs of CACE and MFME are post-processed to obtain the final output prediction.

[0178] By combining the CACE module with the MFME module, the model's boundary segmentation accuracy and class discrimination ability when processing complex food images are improved. The CACE module optimizes mask classification by modeling class correlation relationships, especially effectively distinguishing food categories with similar visual features (such as different kinds of fruits, bread, etc.). The MFME module, on the other hand, improves the sensitivity to complex food boundaries through enhanced boundary supervision mechanisms, reducing errors in the segmentation of occluded and overlapping foods. The combination of the two not only optimizes segmentation accuracy but also effectively improves classification accuracy, significantly enhancing the model's performance when dealing with diverse food images.

[0179] The Adaptive Long Distance Decoder (ALRD) offers highly efficient modeling capabilities. While the traditional Transformer decoder is slightly inferior to the State Space Model (SSM) decoder, it can accept input sequences of arbitrary length, whereas the SSM decoder can only accept fixed-length input sequences. ALRD improves upon the Transformer decoder by replacing the self-attention layer with the SSM, enabling the decoder to efficiently model long-distance dependencies in food images with variable-length input sequences. This reduces computational complexity and improves inference speed. ALRD makes segmentation models more accurate when dealing with scattered ingredients or complex scenes, addressing the shortcomings of traditional methods in modeling long-distance dependencies.

[0180] Therefore, this invention adopts a food image semantic segmentation method based on mask classification, which introduces a multi-foreground mask enhancement module, an adaptive long-distance decoder, and a category-related perception classification enhancement module to efficiently improve segmentation accuracy, optimize long-distance dependency modeling, and reduce computational complexity.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A food image semantic segmentation method based on mask classification, characterized in that: Comprising the following steps: S1, input the picture to the backbone network and the pixel decoder to generate four different scale features; S2, pre-generate an attention query vector according to the first three layer scale features generated in S1 Input an adaptive long distance decoder ALRD to obtain a decoded query vector ; S21. The adaptive long-range decoder contains several decoding blocks. Each decoding block receives three inputs: the attention mask output by the previous decoding block, etc. mask Corresponding layer image features Image Feature and the output vector of the previous decoding block query ;in Image Feature and query The attention query vector is obtained through different linear transformations. Attention key vector K Attention value vector The adaptive long-range decoder also includes a decoder head that performs layer normalization and dimensional transpose on the input. Then, it generates class predictions and mask embeddings through different linear transformations. The mask embeddings and image feature maps are multiplied by tensors to generate mask predictions. Finally, attention masks are obtained through interpolation of the mask predictions. mask , , K、 same mask The input is a masked attention layer, and the output is a multi-head self-attention output vector. as follows: ； wherein, is the transpose of the attention key vector K is the transpose of the attention key vector S22, then with the sum of both residuals is passed through a layer normalization layer to obtain transformer output vector or converter output vector : ； wherein is a layer normalization layer; S23, input to the state space model SSM, input the three-part output vector A , vector , and vector as follows: ； ； ； wherein is , , a corresponding linear transformation matrix; S24, the vector of S23 is projected into three parts and passed through an activation function to obtain a vector Through a one-dimensional convolution layer , Projection into three parts and passed through an activation function to obtain a vector X , the vector B , the vector C , the specific process is as follows: ； ； wherein, is a one-dimensional convolution, is a projection layer, is an activation function layer; S25, vector X , vector B , vector C co-vector A a state space model layer is inputted with the co-vector and the output is a vector , then is multiplied with the vector that has passed through the activation function and is normalized to obtain , , the SSM block output is obtained through the feedforward neural network layer ; ； ； ； wherein, is a state space model layer, is a group normalization layer, is a feed forward neural network layer; S26 and The residuals are summed and layer normalized to obtain the output vector of the state-space model. After passing through the feedforward neural network layer and The final output is obtained by summing the residuals and then normalizing them through layers. And it serves as the input to the next decoder, the input attention mask for the next layer. mask Depend on The specific calculation process, obtained through the decoding head, is as follows: ； ； S3, the result obtained from S2 Category-enhanced features are obtained through processing using the relevance-aware classification enhancement module CACE. ; S4, the fourth layer features decoded from the pixel decoder in S1 are processed by a multi-foreground mask enhancement module MFME to obtain edge enhanced features ; S5, input the output features processed by S3 and S4 into the post-processing module to obtain the final output prediction; S6, calculate the total loss function of the entire model .

2. The food image semantic segmentation method based on mask classification according to claim 1, characterized in that: The specific process in S3 is as follows: S31, input within the CACE module Transformed into classification features via a fully connected layer and mask features , Three different size features are generated by three one-dimensional convolutions As follows: ； ； ； wherein, is a one-dimensional convolutional layer; S32, in S31 and apply cross multiplication Softmax function to generate a correlation map The specific calculation process is as follows: ； wherein is the transpose matrix of S33, the correlation map of S32 is multiplied by the balance parameter with the balance parameter The specific calculation process is as follows:​​ ； wherein is the transpose matrix of S34, compute CACE module output classification loss loss function : ； wherein, is the predicted foreground classification one-hot vector, is the true foreground classification one-hot vector.

3. The food image semantic segmentation method based on mask classification according to claim 2, characterized in that: The specific process of S4 is as follows: S41, the fourth layer scale feature obtained in S1 is input into the edge enhancement feature after the pooling layer , and then , the channel attention feature is obtained through the channel attention layer, the down-sampling convolution layer and the position attention layer respectively , the down-sampling convolution feature , the edge feature ; ； ； ； wherein, is a channel attention layer function, is a feature concatenation function, is a position attention layer function; S42, splicing the feature obtained in S41 and to obtain a spliced feature , The calculation formula is as follows: ； S43, the final output of the MFME module is obtained by adding the output of S41 and the output of S42 S42, the output of S41 is added to the output of S42 The calculation formula is as follows:​ ； S44, let obtaining each foreground edge prediction map through a predicted convolutional layer , The calculation formula is as follows: ； wherein, is a predictive convolutional layer; S45, the edge prediction map obtained in S44 and the edge ground truth to solve the edge loss function The specific calculation formula is as follows: ； wherein, represents the i-th pixel in the real foreground, = 1, 2, 3…, represents a logarithmic function, represents the i-th pixel in the predicted edge map.​ 4. The food image semantic segmentation method based on mask classification according to claim 3, characterized in that: The specific process of S5 is as follows: S51, final output of the CACE module classification enhancement unit input to the post-processing module; S52, concatenating the residual obtained in S31 output and final output of the MFME module performing cross multiplication to obtain foreground mask prediction input to a boundary enhancement unit of the post-processing module; S53, the outputs obtained by S51 and S52 are cross-multiplied to obtain the final prediction result.

5. The food image semantic segmentation method based on mask classification according to claim 4, characterized in that: The classification enhancement unit in S51 respectively enhances the fusion multi-layer perception output, the interpolation method and the Sigmoid functions are performed in turn; The boundary enhancement unit in S52 is for sequentially performing layer normalization, a fully connected layer, and Softmax the process of the function.

6. The food image semantic segmentation method based on mask classification according to claim 1, characterized in that: The auxiliary loss function is computed from the adaptive long range decoder output of S2 The process is as follows: ； wherein, is an auxiliary mask loss function, is an auxiliary classification loss function, is a hand-set scaling parameter.

7. The food image semantic segmentation method based on mask classification according to claim 4, characterized in that: Based on the foreground mask prediction obtained in S52, a mask loss function thereof is calculated : ； wherein, is a cross-entropy loss function, is dice a loss function, , is a manually set scaling parameter.

8. The food image semantic segmentation method based on mask classification according to claim 7, characterized in that: cross-entropy loss function The formula for calculating the cross-entropy loss function is as follows: ； wherein, is the i-th pixel in the foreground of the prediction; is the i-th pixel in the foreground of the prediction; dice loss function The calculation formula is as follows: ； wherein, is the i-th pixel in the real foreground, is the i-th pixel in the predicted foreground, is the i-th pixel in the real foreground, is the i-th pixel in the predicted foreground, = 1, 2, 3....

9. The food image semantic segmentation method based on mask classification according to claim 8, characterized in that: According to , , , the total loss function for the entire model is given by: ； wherein, is a parameter dynamically adjusted in the model training, and its adjustment formula is as follows: 。

Images