Food image semantic segmentation method based on mask classification

Abstract

The invention discloses a food image semantic segmentation method based on mask classification, and relates to the technical field of food calculation. A picture is input to a backbone network and a pixel decoder to generate four layers of different scale features, attention query vectors are pre-generated according to the generated first three layers of scale features, the attention query vectors are input into an adaptive long-distance decoder to obtain decoded query vectors, and the decoded query vectors are input into a correlation perception classification enhancement module CACE to be processed to obtain category enhancement features; processing the fourth-layer features output by the pixel decoder through a multi-foreground mask enhancement module MFME to obtain edge enhancement features, inputting the output features processed by the CACE and the MFME to a post-processing module to obtain final output prediction, and calculating a total loss function of the whole model; according to the food image semantic segmentation method based on mask classification, the segmentation precision is efficiently improved, long-distance dependent modeling is optimized, and the calculation complexity is reduced.

Description

Technical Field

[0001] The present invention relates to the field of food computing technology, and in particular to a food image semantic segmentation method based on mask classification. Background Art

[0002] Food image segmentation aims to distinguish food items in an image from the background or other food items. From the perspective of image segmentation paradigm, the technologies in the field of food image segmentation are mainly divided into pixel-by-pixel classification schemes and mask classification schemes. Pixel-by-pixel classification methods have inherent limitations. Small targets or complex details often lack sufficient local feature strength, making it difficult to effectively capture global information or contextual relationships. Unlike pixel-by-pixel classification, mask classification methods focus on identifying and segmenting different object instances by generating a binary mask for each instance, thereby representing its area in the image. Mask classification particularly focuses on classifying entire objects or regions. By utilizing global features and contextual information, this method enhances the processing capabilities of small targets and complex details, while reducing noise interference by considering information from the entire region.

[0003] The existing food image semantic segmentation methods have the following main problems: First, due to the wide variety of food and different shapes, traditional pixel-by-pixel classification methods (such as FCN and DeepLabV3+) are difficult to accurately distinguish the boundaries of different foods, resulting in classification errors. At the same time, occlusion, mixing and irregular shapes between foods further increase the difficulty of segmentation; secondly, the current methods are insufficient in modeling long-distance dependency relationships. Many food images contain multiple scattered ingredients, and CNN mainly relies on local features and is difficult to capture long-distance ingredient association information. Although the Transformer-based method can model global dependencies, it has high computational complexity, resulting in a decrease in inference speed. In addition, the existing mask classification methods (such as Mask R-CNN, Mask2Former) still have a high error rate in distinguishing food categories, especially when the features between categories are similar (such as apricots and peaches, different types of bread). The main reason is the lack of an effective category relationship modeling mechanism, which fails to fully utilize the relevant information between food categories to improve classification accuracy. Finally, computational cost and efficiency issues are still important factors limiting practical applications. The traditional Transformer decoder has high computational complexity and slow inference speed. Many methods consume a lot of resources in complex food scenarios, making it difficult to strike a balance between segmentation accuracy and inference speed. Summary of the Invention

[0004] The purpose of the present invention is to provide a food image semantic segmentation method based on mask classification, which introduces a multi-foreground mask enhancement module, an adaptive long-distance decoder and a category correlation-aware classification enhancement module to efficiently improve segmentation accuracy, optimize long-distance dependency modeling and reduce computational complexity.

[0005] To achieve the above object, the present invention provides a food image semantic segmentation method based on mask classification, comprising the following steps:

[0006] S1, input the image to the backbone network and pixel decoder to generate four layers of different scale features;

[0007] S2, pre-generate the attention query vector Q based on the first three layers of scale features generated by S1 and input it into the adaptive long distance decoder ALRD to obtain the decoded query vector Q';

[0008] S3, process the Q' obtained in S2 through the correlation-aware classification enhancement module CACE to obtain category enhancement features

[0009] S4, the fourth layer features output by the pixel decoder in S1 are processed by the multi-foreground mask enhancement module MFME to obtain edge enhancement features

[0010] S5, input the output features processed by S3 and S4 into the post-processing module to obtain the final output prediction;

[0011] S6. Calculate the total loss function L of the entire model.

[0012] Preferably, the internal structure and specific workflow of the adaptive long-distance decoder in S2 are as follows:

[0013] S21. The adaptive long-distance decoder is provided with several decoding blocks. The decoding block accepts three parts of input, namely the attention mask mask output by the previous decoding block, the image feature Image Feature of the corresponding level, and the output vector query of the previous decoding block; the Image Feature and the query are subjected to different linear transformations to obtain the attention query vector Q, the attention key vector K, and the attention value vector V. The adaptive long-distance decoder is also provided with a decoding head to perform layer normalization on the input and transpose the dimension, and then generate category predictions and mask embeddings through different linear transformations. The mask embedding and the image feature map generate mask predictions through tensor multiplication. Finally, the attention mask mask is obtained by interpolation through the mask prediction. Q, K, and V are input into the mask attention layer together with the mask, and the multi-head self-attention output vector X is output. MSAout as follows:

[0014] X MSAout =softmax(mask+QK T )V;

[0015] Among them, K T is the transpose of the attention key vector K.

[0016] S22, then X MSAout The sum of the residuals of Q and Q is normalized layer to obtain the transformer output vector or converter output vector X TFout :

[0017] X TFout =LayetNorm(X MSAout +Q);

[0018] Among them, LayerNorm is the layer normalization layer;

[0019] S23, input X TFout To the state space model SSM, the input passes through the feedforward neural network layer to obtain three parts of the output vector A, vector x, and vector z as follows:

[0020] A=X TFout W(A) T ;

[0021] x=X TFout W(x) T ;

[0022] z=X TFout W(z) T ;

[0023] Among them, W(A) T , W(x) T ,W(z) T is the linear transformation matrix corresponding to A, x, and z;

[0024] S24, pass the vector x of S23 through the one-dimensional convolution layer to obtain x c , x c Projected into three parts and activated by the function to obtain vector X, vector B, and vector C. The specific process is as follows:

[0025] x c =conv1d(x);

[0026]

[0027] Among them, conv1d() is a one-dimensional convolution, projection() is a projection layer, is the activation function layer;

[0028] S24, vector X, vector B, vector C and vector A are input into the state space model layer together, and the output is vector y, which is then multiplied by the vector z after the activation function and passed through the normalization layer to obtain y n ,y n After the feedforward neural network layer, the SSM block output S is obtained out ;

[0029] y=SSM(A,B,C,X);

[0030]

[0031] S out =FFN(y n );

[0032] Among them, SSM(A,B,C,X) is the state space model layer, groupnorm() is the group normalization layer, and FFN() is the feedforward neural network layer;

[0033] S25, S out With X TFout The residuals of are added and layer normalized to obtain the state space model output vector X SSMout , X after the feedforward neural network layer SSMout (state space model output vector) and X SSMout The residuals of the state space model output vector are added and normalized through the layer to get the final output F out And as the input of the next decoder, the input attention mask of the next layer is composed of F out Obtained through the decoding header, the specific calculation process is as follows:

[0034]

[0035] F out =LayerNorm(FFN(X SSMout )+X SSMout ).

[0036] Preferably, the auxiliary loss function L is calculated based on the adaptive long distance decoder output of S2 aux The process is as follows:

[0037]

[0038] in, is the auxiliary mask loss function, is the auxiliary classification loss function, λ cls Manually set scaling parameters.

[0039] Preferably, the specific process in S3 is as follows:

[0040] S31. Input Q' in the CACE module is converted into classification features E through the fully connected layer cls and mask feature E mask , E cls Three one-dimensional convolutions generate three different size features F1, F2, and F3 as shown below:

[0041] F1=Conv1D(E cls );

[0042] F2=Conv1D(E cls );

[0043] F3=Conv1D(E cls );

[0044] Among them, Conv1D() is a one-dimensional convolution layer;

[0045] S32. After cross-producting F1 and F2 in S31, apply the Softmax function to generate the correlation graph M. The specific calculation process is as follows:

[0046]

[0047] in, is the transposed matrix of F1;

[0048] S33, multiply the result of the correlation graph M of S32 by the balance parameter α and cross-product it with F3 to finally obtain The specific calculation process is as follows:

[0049]

[0050] in, is the transposed matrix of F3;

[0051] S34. Calculate CACE module output The classification loss function L cls :

[0052] L cls =-∑ i C gt ·log(C pre )-(1-C gt )·log(1-C pre );

[0053] Among them, C pre To predict the foreground classification one-hot vector, C gt is the classified one-hot vector in the real foreground.

[0054] Preferably, the specific process of S4 is as follows:

[0055] The fourth-level scale features obtained by S41 and S1 are pooled as the input edge enhancement features E of MFME. pixel , then E pixel The channel attention feature F is obtained by passing through the channel attention layer, downsampling convolution layer and position attention layer respectively.channel , downsampled convolutional features F conv , edge feature F edge ;

[0056] F channel =ChannelAttention(E pixel );

[0057] F conv =Concat(E pixel );

[0058] F edge =SpatialAttention(E pixel );

[0059] Among them, ChannelAttention() is the channel attention layer function, Concat() is the feature splicing function, and SpatialAttention() is the position attention layer function;

[0060] 42. The F obtained in S41 channel and F conv Perform splicing to obtain the spliced ​​feature F concat , F concat The calculation formula is as follows:

[0061] F concat =Concat(F channel ,F conv );

[0062] S43, the F obtained in S41 edge F obtained with S42 concat Add up to get the final output of the MFME module The calculation formula is as follows:

[0063]

[0064] S44, let F edge After the prediction convolution layer, the foreground edge prediction map F is obtained. edgeout , F edgeout The calculation formula is as follows:

[0065] F edgeout =PreConv(F edge );

[0066] Among them, PredConv() is the prediction convolution layer;

[0067] S45, edge prediction graph F obtained according to S44 edgeout And the marginal true value solves the marginal loss function L ed, the specific calculation formula is as follows:

[0068] L ed =-∑ i GT i ·log(F edge,i )-(1-GT i )·log(1-F edge,i );

[0069] Among them, GT i represents the i-th pixel in the real foreground, i = 1, 2, 3..., log() represents the logarithmic function, F edge,i Represents the i-th pixel in the predicted edge map.

[0070] Preferably, the specific process of S5 is as follows:

[0071] S51, the final output of the CACE module Input to the classification enhancement unit of the post-processing module;

[0072] S52, connect the residual obtained in S31 to output E mask And the final output of the MFME module Perform cross multiplication to obtain the foreground mask prediction and input it into the boundary enhancement unit of the post-processing module;

[0073] S53: Perform a cross product on the outputs obtained from S51 and S52 to obtain the final prediction result.

[0074] Preferably, the classification enhancement unit in S51 is respectively integrated with the multi-layer perceptron output, the interpolation method and the Sigmoid function in sequence;

[0075] The boundary enhancement unit in S52 is a process of performing layer normalization, full connection layer and Softmax function in sequence.

[0076] Preferably, based on the foreground mask prediction obtained in S52, its mask loss function L is calculated mask :

[0077] L mask =λ ce L ce +λ dice L dice ;

[0078] Among them, L ce is the cross entropy loss function, L dice is the dice loss function, λ ce ,λ dice Manually set scaling parameters.

[0079] Preferably, the cross entropy loss function L ceThe calculation formula is as follows:

[0080] L ce =-∑ i GT i ·log(F pre,i )-(1-GT i )·log(1-F pre,i );

[0081] Among them, F pre,i To predict the i-th pixel in the foreground;

[0082] dice loss function L dice The calculation formula is as follows:

[0083]

[0084] Among them, GT j is the jth pixel in the true foreground, F pre,j To predict the jth pixel in the foreground, j = 1, 2, 3...

[0085] Preferably, according to L mask , L cls , L ed , L aux The total loss function L of the entire model is obtained as follows:

[0086] L=L mask +λ cls L cls +λ ed L ed +L aux ;

[0087] Among them, λ ed It is a parameter that is dynamically adjusted during model training. Its adjustment formula is as follows:

[0088]

[0089] Therefore, the present invention adopts the above-mentioned food image semantic segmentation method based on mask classification, which has the following beneficial effects compared with the prior art:

[0090] 1. The segmentation method of this application effectively improves the accuracy of boundary segmentation. It designs a multi-foreground mask enhancement module and introduces a boundary supervision mechanism to effectively optimize mask generation, reduce segmentation errors caused by food occlusion and overlap, and make boundaries clearer.

[0091] 2. The segmentation method of this application optimizes the accuracy of mask classification, designs a category correlation-aware classification enhancement mechanism, and uses category correlation modeling to optimize mask classification, thereby improving the model's accuracy in class differentiation, especially for food types with similar visual features (such as different types of fruit and bread).

[0092] 3. The segmentation method of this application realizes efficient long-distance dependency modeling capabilities, introduces a state-space model, and designs an adaptive long-distance decoder. Its advantage is that by replacing the original self-attention layer, on the one hand, it solves the problem that the state-space model cannot handle variable-length sequences. On the other hand, while reducing the number of model parameters, it improves the modeling ability of the model and reduces the computational complexity of the original Transformer decoder. At the same time, while ensuring high-precision segmentation, it improves the inference speed of the model, making it more suitable for actual application scenarios.

[0093] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 This is an overall structural diagram of a food image semantic segmentation method based on mask classification according to the present invention;

[0095] Figure 2 This is a flow chart of an adaptive long-distance decoder for a food image semantic segmentation method based on mask classification according to the present invention. DETAILED DESCRIPTION

[0096] In the description of the present invention, it should be noted that the terms "upper", "lower", "inside", "outside", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or are the orientations or positional relationships in which the inventive product is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they should not be understood as limiting the present invention.

[0097] Example

[0098] like Figure 1-Figure 2 As shown, a food image semantic segmentation method based on mask classification of the present invention comprises the following steps:

[0099] S1, input the image to the backbone network and pixel decoder to generate four layers of different scale features;

[0100] S2, pre-generate the attention query vector Q based on the first three layers of scale features generated by S1 and input it into the adaptive long distance decoder ALRD to obtain the decoded query vector Q';

[0101] The internal structure and specific workflow of the adaptive long-distance decoder in S2 are as follows:

[0102] S21. The adaptive long-distance decoder is provided with several decoding blocks. The decoding block accepts three parts of input, namely the attention mask mask output by the previous decoding block, the image feature Image Feature of the corresponding level, and the output vector query of the previous decoding block; the Image Feature and the query are subjected to different linear transformations to obtain the attention query vector Q, the attention key vector K, and the attention value vector V. The adaptive long-distance decoder is also provided with a decoding head to perform layer normalization on the input and transpose the dimension, and then generate category predictions and mask embeddings through different linear transformations. The mask embedding and the image feature map generate mask predictions through tensor multiplication. Finally, the attention mask mask is obtained by interpolation through the mask prediction. Q, K, and V are input into the mask attention layer together with the mask, and the multi-head self-attention output vector X is output. MSAout as follows:

[0103] X MSAout =softmax(mask+QK T )V;

[0104] Among them, K T is the transpose of the attention key vector K.

[0105] S22, then X MSAout The sum of the residuals of Q and Q is normalized layer to obtain the transformer output vector or converter output vector X TFout :

[0106] X TFout =LayerNorm(X MSAout +Q);

[0107] Among them, LayerNorm is the layer normalization layer;

[0108] S23, input X TFout To the state space model SSM, the input passes through the feedforward neural network layer to obtain three parts of the output vector A, vector x, and vector z as follows:

[0109] A=X TFout W(A) T ;

[0110] x=X TFout W(x) T ;

[0111] z=X TFout W(z) T ;

[0112] Among them, W(A) T , W(x) T ,W(Z) T is the linear transformation matrix corresponding to A, x, and z;

[0113] S24, pass the vector x of S23 through the one-dimensional convolution layer to obtain x c , x c Projected into three parts and activated by the function to obtain vector X, vector B, and vector C. The specific process is as follows:

[0114] x c =conv1d(x);

[0115]

[0116] Among them, conv1d() is a one-dimensional convolution, projection() is a projection layer, is the activation function layer;

[0117] S24, vector X, vector B, vector C and vector A are input into the state space model layer together, and the output is vector y, which is then multiplied by the vector z after the activation function and passed through the normalization layer to obtain y n ,y n After the feedforward neural network layer, the SSM block output S is obtained out ;

[0118] y=SSM(A,B,C,X);

[0119]

[0120] S out =FFN(y n );

[0121] Among them, SSM(A,B,C,X) is the state space model layer, groupnorm() is the group normalization layer, and FFM() is the feedforward neural network layer;

[0122] S25, S out With X TFout The residuals of are added and layer normalized to obtain the state space model output vector X SSMout , X after the feedforward neural network layer SSMout (state space model output vector) and X SSMout The residuals of the state space model output vector are added and normalized through the layer to get the final output F out And as the input of the next decoder, the input attention mask of the next layer is composed of F out Obtained through the decoding header, the specific calculation process is as follows:

[0123]

[0124] F out =LayerNorm(FFN(X SSMout )+X SSMout );

[0125] S3, process the Q′ obtained in S2 through the correlation-aware classification enhancement module CACE to obtain category enhancement features

[0126] S31. Input Q' in the CACE module is converted into classification features E through the fully connected layer cls and mask feature E mask , E cls Three one-dimensional convolutions generate three different size features F1, F2, and F3 as shown below:

[0127] F1=Conv1D(E cls );

[0128] F2=Conv1D(E cls );

[0129] F3=Conv1D(E cls );

[0130] Among them, Conv1D() is a one-dimensional convolution layer;

[0131] S32. After cross-producting F1 and F2 in S31, apply the Softmax function to generate the correlation graph M. The specific calculation process is as follows:

[0132]

[0133] in, is the transposed matrix of F1;

[0134] S33, multiply the result of the correlation graph M of S32 by the balance parameter α and cross-product it with F3 to finally obtain The specific calculation process is as follows:

[0135]

[0136] in, is the transposed matrix of F3;

[0137] S34. Calculate CACE module output The classification loss function L cls :

[0138] L cls =-∑ i Cgt ·log(C pre )-(1-C gt )·log(1-C pre );

[0139] Among them, C pre To predict the foreground classification one-hot vector, C gt is the one-hot vector for classification in the real foreground;

[0140] S4, the fourth layer features output by the pixel decoder in S1 are processed by the multi-foreground mask enhancement module MFME to obtain edge enhancement features

[0141] The fourth-level scale features obtained by S41 and S1 are pooled as the input edge enhancement features E of MFME. pixel , then E pixel The channel attention feature F is obtained by passing through the channel attention layer, downsampling convolution layer and position attention layer respectively. channel , downsampled convolutional features F conv , edge feature F edge ;

[0142] F channel =ChannelAttention(E pixel );

[0143] F conv =Cconcat(E pixel );

[0144] F edge =SpatialAttention(E pixel );

[0145] Among them, ChannelAttention() is the channel attention layer function, Concat() is the feature splicing function, and SpatialAttention() is the position attention layer function;

[0146] 42. The F obtained in S41 channel and F conv Perform splicing to obtain the spliced ​​feature F concat , F concat The calculation formula is as follows:

[0147] F concat =Concat(F channel ,F conv );

[0148] S43, the F obtained in S41 edge F obtained with S42concat Add up to get the final output of the MFME module The calculation formula is as follows:

[0149]

[0150] S44, let F edge After the prediction convolution layer, the foreground edge prediction map F is obtained. edgeout , F edgeout The calculation formula is as follows:

[0151] F edgeout =PredConv(F edge );

[0152] Among them, PredConv() is the prediction convolution layer;

[0153] S45, edge prediction graph F obtained according to S44 edgeout And the marginal true value solves the marginal loss function L ed , the specific calculation formula is as follows:

[0154] L ed =-∑ i GT i ·log(F edge,i )-(1-GT i )·log(1-F edge,i );

[0155] Among them, GT i represents the i-th pixel in the real foreground, i = 1, 2, 3..., log() represents the logarithmic function, F edge,i Represents the i-th pixel in the predicted edge map;

[0156] S5, input the output features processed by S3 and S4 into the post-processing module to obtain the final output prediction;

[0157] S51, the final output of the CACE module Input to the classification enhancement unit of the post-processing module; the classification enhancement unit is respectively fused with the multi-layer perceptron output, interpolation method and Sigmoid function in sequence;

[0158] S52, connect the residual obtained in S31 to output E mask And the final output of the MFME module The foreground mask prediction is obtained by cross multiplication and input into the boundary enhancement unit of the post-processing module; the boundary enhancement unit is a process of layer normalization, full connection layer and Softmax function in sequence;

[0159] S53, cross-product the outputs of S51 and S52 to get the final prediction result

[0160] S6. Calculate the total loss function L of the entire model;

[0161] Calculate the auxiliary loss function L based on the adaptive long distance decoder output of S2 aux The process is as follows:

[0162]

[0163] in, is the auxiliary mask loss function, is the auxiliary classification loss function, λ cls The scaling parameters are manually set;

[0164] Based on the foreground mask prediction obtained by S52, calculate its mask loss function L mask :

[0165] L mask =λ ce L ce +λ dice L dice ;

[0166] Among them, L ce is the cross entropy loss function, L dice is the dice loss function, λ ce ,λ dice Manually set scaling parameters.

[0167] Cross entropy loss function L ce The calculation formula is as follows:

[0168] L ce =-∑ i GT i ·log(F pre,i )-(1-GT i )·log(1-F pre,i );

[0169] Among them, F pre,i To predict the i-th pixel in the foreground;

[0170] dice loss function L dice The calculation formula is as follows:

[0171]

[0172] Among them, GT j is the jth pixel in the true foreground, F pre,j To predict the jth pixel in the foreground, j = 1, 2, 3, ...;

[0173] According to L mask , L cls , L ed , L aux The total loss function L of the entire model is obtained as follows:

[0174] L=L mask +λ cls L cls +λ ed L ed +L aux ;

[0175] Among them, λ ed It is a parameter that is dynamically adjusted during model training. Its adjustment formula is as follows:

[0176]

[0177] In the specific application process, for the input image, it first passes through the backbone network and pixel decoder to generate four layers of features at different scales. Then the pre-generated attention query vector Q will be input into the adaptive long-distance decoder composed of multiple adaptive long-distance blocks connected in series. Each block receives the Q output by the previous block and features of different scales. After passing through the adaptive long-distance decoder, Q obtains the output Q'. Then Q' is enhanced by correlation-aware classification (CACE), and the fourth layer of features output by the pixel decoder is enhanced by multiple foreground masks (MFME). The outputs of CACE and MFME are post-processed to obtain the final output prediction.

[0178] By combining the CACE module with the MFME module, the model's boundary segmentation accuracy and category differentiation ability when processing complex food images are improved. The CACE module optimizes mask classification through category correlation modeling, especially effectively distinguishing food categories with similar visual features (such as different types of fruits, bread, etc.); while the MFME module improves sensitivity to complex food boundaries through an enhanced boundary supervision mechanism, reducing errors in the segmentation of occluded and overlapping foods. The combination of the two not only optimizes segmentation accuracy, but also effectively improves classification accuracy, which can significantly improve the performance of the model when facing diverse food images.

[0179] The adaptive long-distance decoder (ALRD) has efficient modeling capabilities. The original Transformer decoder's modeling capabilities are slightly inferior to those of the state-space model (SSM) decoder, but it can accept input sequences of arbitrary length, while the state-space model (SSM) decoder can only accept fixed-length sequence inputs. ALRD improves the self-attention layer in the Transformer decoder and replaces it with the state-space model (SSM), allowing the decoder to efficiently model long-distance dependencies in food images through variable-length sequence inputs, while reducing computational complexity and improving inference speed. ALRD enables the segmentation model to be more accurate when processing dispersed ingredients or complex scenes, solving the shortcomings of traditional methods in long-distance dependency modeling.

[0180] Therefore, the present invention adopts the above-mentioned food image semantic segmentation method based on mask classification, introduces a multi-foreground mask enhancement module, an adaptive long-distance decoder and a category correlation-aware classification enhancement module, which effectively improves the segmentation accuracy, optimizes the long-distance dependency modeling and reduces the computational complexity.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A food image semantic segmentation method based on mask classification, characterized by: The following steps are involved: S1, input the image to the backbone network and pixel decoder to generate four layers of different scale features; S2, pre-generate the attention query vector Q based on the first three layers of scale features generated by S1 and input it into the adaptive long distance decoder ALRD to obtain the decoded query vector Q'; S3, process the Q' obtained in S2 through the correlation-aware classification enhancement module CACE to obtain category enhancement features S4, the fourth layer features output by the pixel decoder in S1 are processed by the multi-foreground mask enhancement module MFME to obtain edge enhancement features S5, input the output features processed by S3 and S4 into the post-processing module to obtain the final output prediction; S6. Calculate the total loss function L of the entire model.

2. The method for semantic segmentation of food images based on mask classification according to claim 1, characterized in that: The internal structure and specific workflow of the adaptive long-distance decoder in S2 are as follows: S21, the adaptive long-distance decoder is provided with several decoding blocks, each of which accepts three inputs: the attention mask output by the previous decoding block, the image feature of the corresponding level, and the output vector query of the previous decoding block; The image feature and query undergo different linear transformations to obtain the attention query vector Q, the attention key vector K, and the attention value vector V. The adaptive long-distance decoder is also equipped with a decoding head to perform layer normalization and transpose dimension on the input. Then, different linear transformations are used to generate category predictions and mask embeddings. The mask embeddings and the image feature maps are multiplied by tensors to generate mask predictions. Finally, the attention mask is interpolated through the mask prediction. Q, K, and V are input into the mask attention layer together with the mask, and the multi-head self-attention output vector X is output. MSAout as follows: X MSAout =softmax(mask+QK T )V; Among them, K T is the transpose of the attention key vector K. S22, then X MSAout The sum of the residuals of Q and Q is normalized layer to obtain the transformer output vector or converter output vector X TFout : X TFout =LayerNorm(X MSAout +Q); Among them, LayerNorm is the layer normalization layer; S23, input X TFout To the state space model SSM, the input passes through the feedforward neural network layer to obtain three parts of the output vector A, vector x, and vector z as follows: A=X TFout W(A) T ; x=X TFout W(x) T ; z=X TFout W(z) T ; Among them, W(A) T , W(x) T ,W(z) T is the linear transformation matrix corresponding to A, x, and z; S24, pass the vector x of S23 through the one-dimensional convolution layer to obtain x c , x c Projected into three parts and activated by the function to obtain vector X, vector B, and vector C. The specific process is as follows: x c =conv1d(x); Among them, conv1d() is a one-dimensional convolution, projection() is a projection layer, is the activation function layer; S24, vector X, vector B, vector C and vector A are input into the state space model layer together, and the output is vector y, which is then multiplied by the vector z after the activation function and passed through the normalization layer to obtain y n ,y n After the feedforward neural network layer, the SSM block output S is obtained out ; y=SSM(A,B,C,X); S out =FFN(y n ); Among them, SSM(A,B,C,X) is the state space model layer, groupnorm() is the group normalization layer, and FFN() is the feedforward neural network layer; S25, S out With X TFout The residuals of are added and layer normalized to obtain the state space model output vector X SSMout , X after the feedforward neural network layer SSMout (state space model output vector) and X SSMout The residuals of the state space model output vector are added and normalized through the layer to get the final output F out And as the input of the next decoder, the input attention mask of the next layer is composed of F out Obtained through the decoding header, the specific calculation process is as follows: F out =LayerNorm(FFN(X SSMout )+X SSMout )。 3. The method for semantic segmentation of food images based on mask classification according to claim 2, characterized in that: The specific process in S3 is as follows: S31. Input Q' in the CACE module is converted into classification features E through the fully connected layer cls and mask feature E mask , E cls Three one-dimensional convolutions generate three different size features F1, F2, and F3 as shown below: F1=Conv1D(E cls ); <h2 style=";text-align:left;direction:ltr">F2=Conv1D(E<h2 style=";text-align:left;direction:ltr"> cls <h2 style=";text-align:left;direction:ltr"> ); F3=Conv1D(E cls ); Among them, Conv1D() is a one-dimensional convolution layer; S32. After cross-producting F1 and F2 in S31, apply the Softmax function to generate the correlation graph M. The specific calculation process is as follows: in, is the transposed matrix of F1; S33, multiply the result of the correlation graph M of S32 by the balance parameter α and cross-product it with F3 to finally obtain The specific calculation process is as follows: in, is the transposed matrix of F3; S34. Calculate CACE module output The classification loss function L cls : L cls =-∑ i C gt ·log(C pre )-(1-C gt )·log(1-C pre ); Among them, C pre To predict the foreground classification one-hot vector, C gt is the classified one-hot vector in the real foreground.

4. The method for semantic segmentation of food images based on mask classification according to claim 3, characterized in that: The specific process of S4 is as follows: The fourth-level scale features obtained by S41 and S1 are pooled as the input edge enhancement features E of MFME. pixel , then E pixel The channel attention feature F is obtained by passing through the channel attention layer, downsampling convolution layer and position attention layer respectively. channel , downsampled convolutional features F conv , edge feature F edge ; F channel =ChannelAttention(E pixel ); F conv =Concat(E pixel ); F edge =SpatialAttention(E pixel ); Among them, ChannelAttention() is the channel attention layer function, Concat() is the feature splicing function, and SpatialAttention() is the position attention layer function; 42. The F obtained in S41 channel and F conv Perform splicing to obtain the spliced ​​feature F concat , F concat The calculation formula is as follows: F concat =Concat(F channel ,F conv ); S43, the F obtained in S41 edge F obtained with S42 concat Add up to get the final output of the MFME module The calculation formula is as follows: S44, let F edge After the prediction convolution layer, the foreground edge prediction map F is obtained. edgeout , F edgeout The calculation formula is as follows: F edgeout =PredConv(F edge ); Among them, PredConv() is the prediction convolution layer; S45, edge prediction graph F obtained according to S44 edgeout And the marginal true value solves the marginal loss function L ed , the specific calculation formula is as follows: L ed =-∑ i GT i ·log(F edge,i )-(1-GT i )·log(1-F edge,i ); Among them, GT i represents the i-th pixel in the real foreground, i=1,2,3…, log() represents the logarithmic function, F edge,i Represents the i-th pixel in the predicted edge map.

5. The method for semantic segmentation of food images based on mask classification according to claim 4, characterized in that: The specific process of S5 is as follows: S51, the final output of the CACE module Input to the classification enhancement unit of the post-processing module; S52, connect the residual obtained in S31 to output E mask And the final output of the MFME module Perform cross multiplication to obtain the foreground mask prediction and input it into the boundary enhancement unit of the post-processing module; S53: Perform a cross product on the outputs obtained from S51 and S52 to obtain the final prediction result.

6. The method for semantic segmentation of food images based on mask classification according to claim 5, characterized in that: The classification enhancement unit in S51 is to fuse the multi-layer perceptron output, interpolation method and Sigmoid function in sequence; The boundary enhancement unit in S52 is a process of performing layer normalization, full connection layer and Softmax function in sequence.

7. The method for semantic segmentation of food images based on mask classification according to claim 6, characterized in that: Calculate the auxiliary loss function L based on the adaptive long distance decoder output of S2 aux The process is as follows: in, is the auxiliary mask loss function, is the auxiliary classification loss function, λ cls Manually set scaling parameters.

8. The method for semantic segmentation of food images based on mask classification according to claim 7, characterized in that: Based on the foreground mask prediction obtained by S52, calculate its mask loss function L mask : L mask =λ ce L ce +λ dice L dice ; Among them, L ce is the cross entropy loss function, L dice is the dice loss function, λ ce ,λ dice Manually set scaling parameters.

9. The method for semantic segmentation of food images based on mask classification according to claim 8, characterized in that: Cross entropy loss function L ce The calculation formula is as follows: L ce =-∑ i GT i ·log(F pre,i )-(1-GT i )·log(1-F pre,i ); Among them, F pre,i To predict the i-th pixel in the foreground; dice loss function L dice The calculation formula is as follows: Among them, G Tj is the jth pixel in the true foreground, F pre,j To predict the jth pixel in the foreground, j = 1, 2, 3...

10. The method for semantic segmentation of food images based on mask classification according to claim 9, characterized in that: According to L mask , L cls , L ed , L aux The total loss function L of the entire model is obtained as follows: L=L mask +λ cls L cls +λ ed L ed +L aux ; Among them, λ ed It is a parameter that is dynamically adjusted during model training. Its adjustment formula is as follows:

Images