Food segmentation method and device based on synergy of neighborhood and deformation attention

By using the method of synergistic interaction between neighborhood and deformation attention in the food segmentation model, the insufficient processing of pixels and surrounding context in the food image is solved, and a more refined food segmentation effect is achieved.

CN118762037BActive Publication Date: 2024-11-08JIANGNAN UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411216338.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-11-08
Estimated Expiration
2044-09-02

AI Technical Summary

Technical Problem

The prior art cannot effectively consider the local context between pixels and surrounding pixels in food images, resulting in inaccurate segmentation of food pictures.

Method used

Using a synergistic approach based on neighborhood and deformation attention, by constructing a food segmentation model, using neighborhood attention and multi-scale deformable self-attention mechanism, extracting and fusion characteristics to capture complex boundaries and global context information between foods.

Benefits of technology

It improves the precision and accuracy of food segmentation, can handle complex shapes and occlusion more effectively, and outputs better segmentation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118762037B_ABST
    Figure CN118762037B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of image semantic segmentation technology, and in particular to a method and device for food segmentation based on the synergy of neighborhood and deformation attention, comprising: constructing a food segmentation model, the food segmentation model comprising an encoder, a multi-scale processor, a decoder and a segmenter connected in sequence; the encoder comprises a plurality of stages, each stage comprising a downsampling module, a plurality of serial structures composed of a neighborhood module and an expansion neighborhood module; the multi-scale processor comprises a dimensionally consistent module, a plurality of multi-scale deformable modules and a dimension recovery module connected in sequence; the original food image is input into the food segmentation model, and a target food segmentation image is output. The present invention uses a feature extraction method of the synergy of neighborhood attention and multi-scale deformation attention, provides a more refined food decision boundary, and realizes more effective processing of food boundaries and shapes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image semantic segmentation, and in particular to a food segmentation method and device based on the synergy of neighborhood and deformation attention. Background Art

[0002] Image semantic segmentation is a basic task in the field of computer vision. It is a technology that distinguishes and labels each pixel in an image so that pixels with the same semantics are assigned to the same category. Thanks to the development of deep learning, image semantic segmentation has been widely used in many fields, including scene understanding, autonomous driving, medical image analysis, object detection and tracking, etc.

[0003] The main goal of the Food Image Semantic Segmentation task is to label different parts of food images, such as ingredients and background, at the pixel level for further analysis. This can not only help automated catering management systems identify and classify food, but also play an important role in food safety testing and dietary health management.

[0004] The goal of the food image semantic segmentation task is to classify each pixel in the image into a predefined category, thereby determining the shape and size of each object in the image. There are currently two main methods. One is to use a fully convolutional neural network, replace the fully connected layer with a convolutional layer, and generate a feature map of the same size as the input image; this method can efficiently process various food images, but may not be fine enough in detail segmentation. The other is to use a feature pyramid-based structure through a symmetrical encoder and decoder structure, combined with a pyramid pooling module, to jump connect feature maps of different scales, which can capture the multi-scale features of food images and contextual information of different scales, and is suitable for processing food images with complex backgrounds; but this method ignores the complex boundary conditions between ingredients.

[0005] In recent years, mainstream methods use convolution-based or Transformer-based encoders for feature extraction to obtain multiple feature maps, then upsample the multiple feature maps to obtain feature maps of the same height and width, and use lateral connections for fusion operations. Finally, a segmenter is used to output the obtained fused features as the segmentation result. This method focuses on the semantic classification of objects and does not consider the diversity and complexity of ingredients. For example, ingredients often occlude each other, which makes it more difficult to determine the segmentation boundary. When ingredients are occluded, the main semantic features may be lost, making the model unable to identify the current ingredient category well. In addition, due to the complex shapes of ingredients, an ingredient may present different shapes due to the processing method, and some ingredients are irregular in shape, which increases the difficulty of the model predicting the decision boundary. Therefore, the design of the existing method is not perfect. When considering the semantic category of pixels, the local context between pixels and surrounding pixels cannot be considered, making it difficult to obtain fine decision boundaries, resulting in inaccurate food image segmentation. Summary of the invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome the problem in the prior art that the local context between pixels in the food image and the surrounding pixels cannot be considered, resulting in inaccurate food image segmentation.

[0007] In order to solve the above technical problems, the present invention provides a food segmentation method based on the synergy of neighborhood and deformation attention, comprising:

[0008] Construct a food segmentation model, including an encoder, a multi-scale processor, a decoder, and a segmenter connected in sequence;

[0009] Input the original food image into the food segmentation model and output the target food segmentation image, including:

[0010] The preprocessed original food image is divided into image blocks of fixed size and the token of each image block is obtained;

[0011] Input the features composed of tokens of each image block into an encoder to obtain multi-scale output features of each stage; the encoder includes multiple stages, each stage includes a downsampling module, a plurality of serial structures composed of neighborhood modules and dilated neighborhood modules; the neighborhood module and the dilated neighborhood module are modules that replace the self-attention in the Transformer block with neighborhood attention and void neighborhood attention respectively;

[0012] The multi-scale output features of each stage in the encoder are input into a multi-scale processor to obtain multiple multi-scale key features; the multi-scale processor includes a dimension consistency module, a plurality of multi-scale deformable modules and a dimension recovery module connected in sequence; the multi-scale deformable module is a module that replaces the self-attention in the Transformer block with a multi-scale deformable self-attention module;

[0013] Input multiple multi-scale key features into the decoder, and the decoder upsamples and fuses the multiple multi-scale key features to obtain the target fused features;

[0014] The target fusion features are input into the segmentation device for segmentation processing to obtain the target food segmentation image.

[0015] Preferably, the original food image is preprocessed, including:

[0016] Scale the original food image to a fixed size;

[0017] The food image scaled to a fixed size is converted into a tensor and normalized by mean variance to obtain the input image of the food segmentation model.

[0018] Preferably, the neighborhood module comprises a normalization layer, a neighborhood attention layer, a normalization layer and a first perception layer connected in sequence;

[0019] After the input features of the neighborhood module pass through the normalization layer and the neighborhood attention, the neighborhood attention weight score matrix is ​​output; the neighborhood attention weight score matrix and the input features of the neighborhood module are weighted and summed to obtain the neighborhood attention output features; the neighborhood attention output features pass through the normalization layer and the first perception layer in turn to obtain the first perception layer output features; the neighborhood attention output features and the first perception layer output features are added to obtain the neighborhood module output features.

[0020] Preferably, the dilated neighborhood module comprises a normalization layer, a hole neighborhood attention layer, a normalization layer, and a second perception layer connected in sequence;

[0021] After the input features of the dilated neighborhood module pass through the normalization layer and the hole neighborhood attention, the hole neighborhood attention weight score matrix is ​​output; the hole neighborhood attention weight score matrix and the input features of the dilated neighborhood module are weighted and summed to obtain the hole neighborhood attention output features; the hole neighborhood attention output features pass through the normalization layer and the second perception layer in turn to obtain the second perception layer output features; the hole neighborhood attention output features and the second perception layer output features are added to obtain the dilated neighborhood module output features.

[0022] Preferably, the multi-scale output features of each stage in the encoder are input into the multi-scale processor to obtain a plurality of multi-scale key features, including:

[0023] The multi-scale output features of each stage in the encoder are input into the dimension consistency module of the multi-scale processor, and the dimension mapping is performed on each multi-scale output feature to ensure that the number of channels of each multi-scale output feature is consistent;

[0024] The multi-scale output features with the same number of channels are spliced ​​at the channel number level, and multi-scale feature fusion is performed to obtain the first fused feature;

[0025] The first fused features are sequentially input into a plurality of multi-scale deformable modules, and the key points around each feature in the first fused features are sensed by position offset to obtain the second fused features;

[0026] The second fusion feature is decomposed into multiple multi-scale decomposition features at the channel number level, and then input into the dimension recovery module, and the channel numbers of the multiple multi-scale decomposition features are restored to be the same as the multi-scale output features of the corresponding stages, so as to obtain multiple multi-scale key features.

[0027] Preferably, the multi-scale deformable module comprises a normalization layer, a multi-scale deformable self-attention layer, a normalization layer and a third perception layer connected in sequence;

[0028] After the input features of the multi-scale deformable module pass through the normalization layer and the multi-scale deformable self-attention, a multi-scale deformable self-attention weight score matrix is ​​output; the multi-scale deformable self-attention weight score matrix and the input features of the multi-scale deformable module are weighted and summed to obtain the multi-scale deformable self-attention output features; the multi-scale deformable self-attention output features pass through the normalization layer and the third perception layer in turn to obtain the third perception layer output features; the multi-scale deformable self-attention output features and the third perception layer output features are added to obtain the multi-scale deformable module output features.

[0029] Preferably, the decoder comprises a multi-scale pooling module and an upsampling module.

[0030] Preferably, a plurality of multi-scale key features are input into a decoder, and the decoder performs upsampling and fusion on the plurality of multi-scale key features to obtain a target fusion feature, including:

[0031] The key features corresponding to the encoder stage N Input to the multi-scale pooling module of the decoder, the key features After the multi-scale pooling module performs different levels of pooling operations, the up-sampled features are obtained. ; Where N is the total number of stages of the encoder;

[0032] Upsample the features and multi-scale key features The input is sent to the upsampling module for upsampling operations in sequence, including:

[0033] Use linear interpolation to upsample features After doubling the , then Perform two convolution operations, The number of channels and Same; using the splicing operation will and Splice and then perform two convolution operations on the spliced ​​features to obtain the upsampled features ;in, ;

[0034] Multiple upsampling features are obtained through the upsampling module ; Use linear interpolation to upsample features Upsample again to get the same upsampled features Twice upsampled feature maps of the same size ; Upsample features With the twice upsampled feature map Perform splicing and fusion, and then perform two convolution operations to obtain the target fusion features .

[0035] Preferably, cross entropy loss is used as a loss function to train the food segmentation model.

[0036] The present invention also provides a food cutting device, comprising:

[0037] A model building module, for building a food segmentation model, including an encoder, a multi-scale processor, a decoder, and a segmenter connected in sequence;

[0038] The food segmentation module is used to input the original food image into the food segmentation model and output the target food segmentation image, including:

[0039] Divide the input image into image blocks of fixed size and obtain the token of each image block;

[0040] The features composed of tokens of each image block are input to an encoder, wherein the encoder includes multiple stages to obtain multi-scale output features of each stage; each stage of the encoder includes a downsampling module, a plurality of serial structures composed of neighborhood modules and dilated neighborhood modules; the neighborhood module and the dilated neighborhood module are modules that replace the self-attention in the Transformer block with neighborhood attention and void neighborhood attention respectively;

[0041] The multi-scale output features of each stage in the encoder are input into a multi-scale processor to obtain multiple multi-scale key features; the multi-scale processor includes a dimension consistency module, a plurality of multi-scale deformable modules and a dimension recovery module connected in sequence; the multi-scale deformable module is a module that replaces the self-attention in the Transformer block with a multi-scale deformable self-attention module;

[0042] Input multiple multi-scale key features into the decoder, and the decoder upsamples and fuses the multiple multi-scale key features to obtain the target fused features;

[0043] The target fusion features are input into the segmentation device for segmentation processing to obtain the target food segmentation image.

[0044] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0045] The present invention proposes a food segmentation method based on the synergy of neighborhood and deformation attention, constructs a food segmentation model, and in the encoder of the food segmentation model, the neighborhood attention mechanism is used to replace the self-attention in the Transformer block to construct a neighborhood module, and the neighborhood module is used to extract the local feature relationship of the input feature, which can effectively process the complex boundaries between foods; the hollow neighborhood attention mechanism is used to replace the self-attention in the Transformer block to construct an expansion neighborhood module, so that the model focuses on the neighborhood information around each pixel, and the relationship between the pixel and its surrounding pixels is captured. The expansion neighborhood module can better obtain the global context feature relationship. In addition, the present invention uses multi-scale deformable self-attention to construct a multi-scale deformable module, captures key points on features of multiple scales, and effectively solves the limitation that complex shapes cannot be perceived in the window calculation mechanism. The present invention uses the feature extraction method of the synergy of neighborhood attention and multi-scale deformation attention to provide a more refined food decision boundary, and to a certain extent, more effective processing of food boundaries and shapes is achieved, so that the segmenter can use fusion features to output better segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 It is a flow chart of a food segmentation method based on the synergy of neighborhood and deformation attention of the present invention;

[0048] Figure 2 It is the structural diagram of the food segmentation model. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0050] Reference Figure 1 The present invention provides a food segmentation method based on the synergy of neighborhood and deformation attention, and classifies food in a food image by constructing a food segmentation model, and the steps include:

[0051] S1. Construct a food segmentation model. The structure of the food segmentation model is referenced Figure 2 As shown, it includes an encoder, a multi-scale processor, a decoder and a segmentor connected in sequence.

[0052] S2. Preprocess the original food image and input it into the food segmentation model, including:

[0053] Original food images Scale to fixed size , where H represents the preset height and W represents the preset width;

[0054] The food image scaled to a fixed size is converted into a tensor and normalized by mean variance to obtain the input image.

[0055] Preferably, the preset height H and preset width W of this embodiment are both 768.

[0056] S3: Segment the preprocessed original food image into image blocks of fixed size and obtain tokens of each image block.

[0057] The preprocessed original food image is input into the food segmentation model. First, the Patch Embedding operation is used to segment the input image into image blocks of fixed size, and then the image blocks are flattened and embedded to obtain the tokens of each image block. Each token is a vector that represents the characteristics of a local area of ​​the image.

[0058] S4. Input the features composed of the tokens of each image block into the encoder to extract the multi-scale features of the input image and obtain the multi-scale output features of each stage.

[0059] Specifically, the encoder includes multiple stages, each stage includes a downsampling module, L serial structures consisting of a neighborhood module and an expanded neighborhood module, which are used to perform self-attention on local areas of the input image, so that the food segmentation model pays more attention to local boundary information and improves the ability to handle similar ingredients mixed together and the existence of occlusion between ingredients.

[0060] In each stage, the input features first pass through the downsampling module, and then pass through L series structures consisting of neighborhood modules and expansion neighborhood modules to obtain the output features of the current stage, which can be expressed as:

[0061] ;

[0062] in, For the The output features of each stage are represents the set of real numbers, For the The number of channels in each stage, is the number of stages, , N is the total number of encoder stages.

[0063] In this embodiment, the total number of encoder stages is 4.

[0064] Specifically, the neighborhood module replaces the self-attention in the Transformer block with a neighborhood attention mechanism (NA), including a normalization layer, a neighborhood attention, a normalization layer, and a first perception layer connected in sequence. After the input features of the neighborhood module pass through the normalization layer and the neighborhood attention, the neighborhood attention weight score matrix is ​​output; the neighborhood attention weight score matrix is ​​weighted and summed with the input features of the neighborhood module to obtain the neighborhood attention output features; the neighborhood attention output features pass through the normalization layer and the first perception layer in sequence to obtain the first perception layer output features; the neighborhood attention output features and the first perception layer output features are added to obtain the neighborhood module output features.

[0065] The neighborhood attention is a sliding window attention used to extract features in a local window and obtain local feature relationships. It performs self-attention in each window of size k×k centered on each image block in the input feature, and its input feature is recorded as , where n represents the number of tokens of the input feature image block, Indicates the dimension of the token.

[0066] Define the neighborhood attention weight of the i-th token in a window of size k×k as:

[0067] ;

[0068] in, represents the neighborhood attention weight of the i-th token, i=1,2,…,n; represents the neighborhood attention query matrix of the i-th token, Represents distance Token The recent tokens are j=1,2,…,k; Indicates the distance from the i-th token Near token The neighborhood attention key matrix, for The transpose of Indicates Tokens and The relative position offset between .

[0069] Define the neighborhood attention value matrix of the i-th token as ,in, express The neighborhood attention value matrix, for The transpose of .

[0070] The neighborhood attention query matrix, key matrix and value matrix are obtained by linearly mapping the input features of the neighborhood attention.

[0071] Neighborhood attention outputs the neighborhood attention weight score of the i-th token:

[0072] ;

[0073] in, represents the neighborhood attention weight score of the i-th token, is the scale parameter.

[0074] The neighborhood attention weight scores of all tokens constitute a neighborhood attention weight score matrix. The neighborhood attention weight score matrix is ​​weighted and summed with the input features of the neighborhood module to obtain the neighborhood attention output features.

[0075] The dilated neighborhood module replaces the self-attention in the Transformer block with a hole neighborhood attention mechanism, including a normalization layer, a hole neighborhood attention (DiNA), a normalization layer, and a second perception layer connected in sequence. After the input features of the dilated neighborhood module pass through the normalization layer and the hole neighborhood attention, a hole neighborhood attention weight score matrix is ​​output; the hole neighborhood attention weight score matrix is ​​weighted and summed with the input features of the dilated neighborhood module to obtain the hole neighborhood attention output features; the hole neighborhood attention output features pass through the normalization layer and the second perception layer in sequence to obtain the second perception layer output features; the hole neighborhood attention output features and the second perception layer output features are added to obtain the dilated neighborhood module output features.

[0076] Specifically, the hole neighborhood attention introduces a hole coefficient based on the neighborhood attention, and collides the sliding window in the neighborhood attention through the hole to obtain a larger receptive field. The hole neighborhood attention expands the local window to form a hole window, uses self-attention to extract features in the local window, and obtains the global context feature relationship by expanding the window. Neighborhood attention can be said to be a special case of the hole neighborhood attention, that is, the hole neighborhood attention with an expansion coefficient of 1.

[0077] In the void neighborhood attention, the expansion coefficient is defined as , we can get the same as The distance between tokens is Token , and satisfies .

[0078] Definition The expansion factor of a token in a window of size k×k is The empty neighborhood attention weight is:

[0079] ;

[0080] in, represents the empty neighborhood attention weight of the i-th token, i=1,2,…,n; represents the empty neighborhood attention query matrix of the i-th token, Indicates The distance between tokens is Tokens are j=1,2,…,k; express The empty neighborhood attention key matrix, for The transpose of Indicates Tokens and The relative position offset between them.

[0081] Define the expansion coefficient between the i-th token and the k-th closest token as The empty neighborhood attention value matrix is ,in, express The empty neighborhood attention value matrix, for The transpose of .

[0082] The hole neighborhood attention query matrix, key matrix and value matrix are obtained by linearly mapping the input features of the hole neighborhood attention.

[0083] Hollow Neighborhood Attention Output The hollow neighborhood attention weight score of the i-th token is:

[0084] ;

[0085] in, represents the empty neighborhood attention weight score of the i-th token, is the scale parameter.

[0086] The empty neighborhood attention weight scores of all tokens constitute the empty neighborhood attention weight score matrix. The empty neighborhood attention weight score matrix is ​​weighted and summed with the input features of the expansion neighborhood module to obtain the empty neighborhood attention output features.

[0087] S5. Multi-scale output features of each stage in the encoder All are input into the multi-scale processor to obtain multiple multi-scale key features The multi-scale processor includes a dimension consistency module, a plurality of multi-scale deformable modules and a dimension recovery module which are connected in sequence.

[0088] The multi-scale output features of each stage in the encoder are All are input into the multi-scale processor to obtain multiple multi-scale key features ,include:

[0089] The multi-scale output features of each stage in the encoder are input into the dimension consistency module of the multi-scale processor, and the dimension mapping is performed on each multi-scale output feature to ensure that the number of channels of each multi-scale output feature is consistent;

[0090] The multi-scale output features with the same number of channels are spliced ​​at the channel number level, and multi-scale feature fusion is performed to obtain the first fused feature;

[0091] The first fused features are sequentially input into a plurality of multi-scale deformable modules, and the key points around each feature in the first fused features are sensed by position offset to obtain the second fused features;

[0092] The second fusion feature is decomposed into multiple multi-scale decomposition features at the channel number level, and then input into the dimension recovery module, and the channel numbers of the multiple multi-scale decomposition features are restored to be the same as the multi-scale output features of the corresponding stages, so as to obtain multiple multi-scale key features.

[0093] The dimension consistency module uses a convolution operation on each input feature so that the number of channels of each input feature remains consistent, but the height and width are different.

[0094] The multi-scale deformable module replaces the self-attention in the Transformer block with a multi-scale deformable self-attention (MDSA) mechanism, and obtains the reference point most relevant to the current feature through position offset, thereby improving the model's ability to extract features of complex-shaped ingredients.

[0095] The multi-scale deformable module includes a normalization layer, a multi-scale deformable self-attention layer, a normalization layer, and a third perception layer connected in sequence. After the input features of the multi-scale deformable module pass through the normalization layer and the multi-scale deformable self-attention layer, a multi-scale deformable self-attention weight score matrix is ​​output; the multi-scale deformable self-attention weight score matrix is ​​weighted and summed with the input features of the multi-scale deformable module to obtain the multi-scale deformable self-attention output features; the multi-scale deformable self-attention output features pass through the normalization layer and the third perception layer in sequence to obtain the third perception layer output features; the multi-scale deformable self-attention output features and the third perception layer output features are added to obtain the multi-scale deformable module output features.

[0096] The multi-scale deformable self-attention is improved based on deformable self-attention (DSA).

[0097] Deformable self-attention only focuses on a small number of key points around the sampling point, and does not consider the size of the feature map. Deformable self-attention first initializes B key points near the sampling point, but these B key points may not have the strongest relationship with the sampling point. The purpose is to shift these B points to the point with the strongest relationship through training. Deformable self-attention is the attention of the sampling point and these B points. Output characteristics of the stage As the input feature of deformable self-attention, it is expressed as:

[0098] ;

[0099] in, represents the deformable self-attention weight score, for After feature mapping, we get the sampling points, and then After linear mapping, we get the deformable self-attention query matrix and key matrix ; Represents the coordinates of the key points of the initial sampling; represents the number of attention heads, and is a learnable weight matrix, represents the weight of the bth key point of the mth attention head, which can be regarded as ; Represents the weight of the input feature, which can be regarded as a value matrix of deformable self-attention , represents the offset of the bth key point of the mth attention head; B is the number of key points.

[0100] Multi-scale deformable self-attention is to apply deformable self-attention to multi-scale features. The formula is expressed as:

[0101] ;

[0102] in, represents the multi-scale deformable self-attention weight score, N is the total number of encoder stages, is the input feature of multi-scale deformable self-attention, Represents the normalized key point coordinates; represents the number of attention heads, and is the learnable weight matrix, and B is the number of key points; represents the mth attention head and the bth key point The weight of the features corresponding to each stage; Indicates The corresponding characteristics of each stage are Indicates that Map to In terms of the characteristics corresponding to each stage, represents the mth attention head and the bth key point The offset of the feature corresponding to each stage.

[0103] The present invention uses multi-scale deformable self-attention to process multi-scale features and dynamically adjusts the position of the features themselves, so as to better adapt to targets of different shapes. Since the shape of food is locally continuous, using attention in the neighborhood can better preserve this local property, and the extracted features can make it easier for deformable attention to capture complex boundary features.

[0104] S6. Output multiple multi-scale key features of the multi-scale processor The input is sent to the decoder, which upsamples and fuses multiple multi-scale key features to obtain the target fused features.

[0105] The decoder includes a multi-scale pooling module and an upsampling module. The input features of the decoder are multiple multi-scale key features .

[0106] First, the key features corresponding to the encoder stage N are Input to the multi-scale pooling module of the decoder, the key features The multi-scale pooling module performs different levels of pooling operations. Specifically, pooling layers of different sizes are used to capture The information of different regions in the image is obtained to obtain outputs of different levels, and then the outputs of all levels are connected together to obtain an upsampled feature containing rich contextual information. . Where N is the total number of encoder stages.

[0107] Upsample the features and multi-scale key features The input is sent to the upsampling module for upsampling operation. For example, use linear interpolation to upsample the features After doubling the , then Perform two convolution operations, The number of channels and Same; using the splicing operation will and Splice and then perform two convolution operations on the spliced ​​features to obtain the upsampled features ;in, .

[0108] After the upsampling module, multiple upsampling features are obtained . Use linear interpolation to upsample features Upsample again to get the same upsampled features Twice upsampled feature maps of the same size . The upsampled features With the twice upsampled feature map Perform splicing and fusion, and then perform two convolution operations to obtain the target fusion features .

[0109] Using multi-scale pooling in the encoder can capture the semantic information of the image at different scales, and then fuse the multi-scale feature maps to obtain the target fusion features containing multi-scale information, which can better understand the overall context in the image during segmentation, thereby improving the segmentation performance.

[0110] Since the target fusion feature It is obtained by fusing feature maps of different sizes, which will be helpful for segmentation tasks. Large-scale feature maps can provide more global context, which helps the model understand the semantic information of the entire scene; while smaller feature maps can provide more detailed local information, which helps the model capture edge information.

[0111] S7. Fusion of target features Input to the segmentation device for segmentation processing to obtain the target food segmentation image .

[0112] Since the target fusion feature output by the decoder The size should be smaller than the size required for segmentation , so additional processing is needed to obtain the final segmentation result.

[0113] Specifically, in the splitter, this embodiment will After reducing the dimension through the convolution layer, it passes through the normalization layer, ReLu activation function, and convolution layer in sequence to fuse the target features. The number of channels is mapped to the number of segmentation categories to obtain the initial segmentation features Finally, the segmentation features are interpolated by linear interpolation. The size of the target is enlarged to the target size, and the target food segmentation image features are obtained. .

[0114] Preferably, in the training stage, this embodiment uses cross entropy loss to train the food segmentation model, and the loss formula is as follows:

[0115] ;

[0116] in, is the number of categories, It is Class category label, The current pixel belongs to The probability of the class label.

[0117] It has been verified that the proposed method has achieved better performance than the baseline model in both indicators, with an average intersection-over-union ratio of 36.64%, 1% higher than the baseline model, and an average accuracy of 52.19%, 2.29% higher than the baseline model, achieving good results.

[0118] In summary, the present invention proposes a food segmentation method based on the synergy of neighborhood and deformation attention. In the encoder, the neighborhood attention mechanism is used to replace the self-attention in the Transformer block to construct a neighborhood module. The neighborhood module is used to extract the local feature relationship of the input feature, which can effectively process the complex boundaries between foods; the hollow neighborhood attention mechanism is used to replace the self-attention in the Transformer block to construct an expansion neighborhood module, so that the model focuses on the neighborhood information around each pixel. By capturing the relationship between the pixel and its surrounding pixels, the expansion neighborhood module can better obtain the global context feature relationship. The present invention uses a feature extraction method with the synergy of neighborhood and deformation attention to provide a more refined decision boundary, and to a certain extent, more effective processing of food boundaries and shapes is achieved, so that the segmenter can use fused features to output better segmentation results.

[0119] Furthermore, the present invention proposes a food segmentation method based on the synergy of neighborhood and deformable attention, which utilizes multi-scale deformable self-attention to construct a multi-scale deformable module, captures key points on features at multiple scales, and effectively solves the limitation that the window computing mechanism cannot perceive complex shapes.

[0120] Based on the above-mentioned food segmentation method based on the synergy of neighborhood and deformation attention, the present invention also provides a food segmentation device based on the synergy of neighborhood and deformation attention, comprising:

[0121] A model building module, for building a food segmentation model, including an encoder, a multi-scale processor, a decoder, and a segmenter connected in sequence;

[0122] The food segmentation module is used to input the original food image into the food segmentation model and output the target food segmentation image, including:

[0123] Divide the input image into image blocks of fixed size and obtain the token of each image block;

[0124] The features composed of tokens of each image block are input to an encoder, wherein the encoder includes multiple stages to obtain multi-scale output features of each stage; each stage of the encoder includes a downsampling module, a plurality of serial structures composed of neighborhood modules and dilated neighborhood modules; the neighborhood module and the dilated neighborhood module are modules that replace the self-attention in the Transformer block with neighborhood attention and void neighborhood attention respectively;

[0125] The multi-scale output features of each stage in the encoder are input into a multi-scale processor to obtain multiple multi-scale key features; the multi-scale processor includes a dimension consistency module, a plurality of multi-scale deformable modules and a dimension recovery module connected in sequence; the multi-scale deformable module is a module that replaces the self-attention in the Transformer block with a multi-scale deformable self-attention module;

[0126] Input multiple multi-scale key features into the decoder, and the decoder upsamples and fuses the multiple multi-scale key features to obtain the target fused features;

[0127] The target fusion features are input into the segmentation device for segmentation processing to obtain the target food segmentation image.

[0128] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0129] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0132] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A food segmentation method based on the synergy of neighborhood and deformation attention, It is characterized in that include: Construct a food segmentation model, including an encoder, a multi-scale processor, a decoder, and a segmenter connected in sequence; Input the original food image into the food segmentation model and output the target food segmentation image, including: The preprocessed original food image is divided into image blocks of fixed size and the token of each image block is obtained; Input the features composed of tokens of each image block into an encoder to obtain multi-scale output features of each stage; the encoder includes multiple stages, each stage includes a downsampling module, a plurality of serial structures composed of neighborhood modules and dilated neighborhood modules; the neighborhood module and the dilated neighborhood module are modules that replace the self-attention in the Transformer block with neighborhood attention and void neighborhood attention respectively; The multi-scale output features of each stage in the encoder are input into a multi-scale processor to obtain multiple multi-scale key features; the multi-scale processor includes a dimension consistency module, a plurality of multi-scale deformable modules and a dimension recovery module connected in sequence; the multi-scale deformable module is a module that replaces the self-attention in the Transformer block with a multi-scale deformable self-attention module; Input multiple multi-scale key features into the decoder, and the decoder upsamples and fuses the multiple multi-scale key features to obtain the target fused features; The target fusion features are input into the segmentation device for segmentation processing to obtain the target food segmentation image.

2. A food segmentation method based on the synergy of neighborhood and deformation attention according to claim 1, It is characterized in that Preprocess the original food images, including: Scale the original food image to a fixed size; The food image scaled to a fixed size is converted into a tensor and normalized by mean variance to obtain the input image of the food segmentation model.

3. The food segmentation method based on the synergy of neighborhood and deformation attention according to claim 1, It is characterized in that The neighborhood module includes a normalization layer, a neighborhood attention layer, a normalization layer and a first perception layer connected in sequence; After the input features of the neighborhood module pass through the normalization layer and neighborhood attention, the neighborhood attention weight score matrix is ​​output; the neighborhood attention weight score matrix is ​​weighted and summed with the input features of the neighborhood module to obtain the neighborhood attention output features; The neighborhood attention output features pass through the normalization layer and the first perception layer in turn to obtain the first perception layer output features; the neighborhood module output features are obtained by adding the neighborhood attention output features and the first perception layer output features.

4. The food segmentation method based on the synergy of neighborhood and deformation attention according to claim 1, It is characterized in that The dilated neighborhood module includes a normalization layer, a hole neighborhood attention layer, a normalization layer, and a second perception layer connected in sequence; After the input features of the dilated neighborhood module pass through the normalization layer and the hole neighborhood attention, the hole neighborhood attention weight score matrix is ​​output; the hole neighborhood attention weight score matrix is ​​weighted and summed with the input features of the dilated neighborhood module to obtain the hole neighborhood attention output features; The output features of the hole neighborhood attention are passed through the normalization layer and the second perception layer in turn to obtain the output features of the second perception layer; the output features of the hole neighborhood attention and the second perception layer are added to obtain the output features of the dilated neighborhood module.

5. The food segmentation method based on the synergy of neighborhood and deformation attention according to claim 1, It is characterized in that The multi-scale output features of each stage in the encoder are input into the multi-scale processor to obtain multiple multi-scale key features, including: The multi-scale output features of each stage in the encoder are input into the dimension consistency module of the multi-scale processor, and the dimension mapping is performed on each multi-scale output feature to ensure that the number of channels of each multi-scale output feature is consistent; The multi-scale output features with the same number of channels are spliced ​​at the channel number level, and multi-scale feature fusion is performed to obtain the first fused feature; The first fused features are sequentially input into a plurality of multi-scale deformable modules, and the key points around each feature in the first fused features are sensed by position offset to obtain the second fused features; The second fusion feature is decomposed into multiple multi-scale decomposition features at the channel number level, and then input into the dimension recovery module, and the channel numbers of the multiple multi-scale decomposition features are restored to be the same as the multi-scale output features of the corresponding stages, so as to obtain multiple multi-scale key features.

6. A food segmentation method based on the synergy of neighborhood and deformation attention according to claim 5, It is characterized in that The multi-scale deformable module includes a normalization layer, a multi-scale deformable self-attention layer, a normalization layer and a third perception layer connected in sequence; After the input features of the multi-scale deformable module pass through the normalization layer and the multi-scale deformable self-attention, a multi-scale deformable self-attention weight score matrix is ​​output; the multi-scale deformable self-attention weight score matrix is ​​weighted and summed with the input features of the multi-scale deformable module to obtain the multi-scale deformable self-attention output features; The multi-scale deformable self-attention output features pass through the normalization layer and the third perception layer in turn to obtain the third perception layer output features; the multi-scale deformable self-attention output features and the third perception layer output features are added together to obtain the multi-scale deformable module output features.

7. The food segmentation method based on the synergy of neighborhood and deformation attention according to claim 1, It is characterized in that The decoder includes a multi-scale pooling module and an upsampling module.

8. The food segmentation method based on the synergy of neighborhood and deformation attention according to claim 7, It is characterized in that Multiple multi-scale key features are input into the decoder, and the decoder upsamples and fuses the multiple multi-scale key features to obtain the target fusion features, including: The key features corresponding to the encoder stage N Input to the multi-scale pooling module of the decoder, the key features After the multi-scale pooling module performs different levels of pooling operations, the up-sampled features are obtained. ; Where N is the total number of stages of the encoder; Upsample the features and multi-scale key features The input is sent to the upsampling module for upsampling operations in sequence, including: Use linear interpolation to upsample features After doubling the , then Perform two convolution operations, The number of channels and Same; using the splicing operation will and Splice and then perform two convolution operations on the spliced ​​features to obtain the upsampled features ;in, ; Multiple upsampling features are obtained through the upsampling module ; Use linear interpolation to upsample features Upsample again to get the same upsampled features Twice upsampled feature maps of the same size ; Upsample features With the twice upsampled feature map Perform splicing and fusion, and then perform two convolution operations to obtain the target fusion features .

9. The food segmentation method based on the synergy of neighborhood and deformation attention according to claim 1, It is characterized in that The food segmentation model is trained using cross entropy loss as the loss function.

10. A food segmentation device based on the synergy of neighborhood and deformation attention, It is characterized in that include: A model building module, for building a food segmentation model, including an encoder, a multi-scale processor, a decoder, and a segmenter connected in sequence; The food segmentation module is used to input the original food image into the food segmentation model and output the target food segmentation image, including: Divide the input image into image blocks of fixed size and obtain the token of each image block; The features composed of tokens of each image block are input to an encoder, wherein the encoder includes multiple stages to obtain multi-scale output features of each stage; each stage of the encoder includes a downsampling module, a plurality of serial structures composed of neighborhood modules and dilated neighborhood modules; the neighborhood module and the dilated neighborhood module are modules that replace the self-attention in the Transformer block with neighborhood attention and void neighborhood attention respectively; The multi-scale output features of each stage in the encoder are input into a multi-scale processor to obtain multiple multi-scale key features; the multi-scale processor includes a dimension consistency module, a plurality of multi-scale deformable modules and a dimension recovery module connected in sequence; the multi-scale deformable module is a module that replaces the self-attention in the Transformer block with a multi-scale deformable self-attention module; Input multiple multi-scale key features into the decoder, and the decoder upsamples and fuses the multiple multi-scale key features to obtain the target fused features; The target fusion features are input into the segmentation device for segmentation processing to obtain the target food segmentation image.

Citation Information

Patent Citations

  • Remote sensing image change detection method using multi-level difference feature adaptive fusion

    CN117576567A

  • Semantic segmentation algorithm based on hole neighborhood attention

    CN118154871A

  • Image registration method based on cross neighborhood attention enhancement mechanism

    CN118447061A

  • Image target segmentation method and device

    CN118470033A