A method for food image segmentation and removal based on instance segmentation and spatial prior clustering

Abstract

This invention discloses a method for food image segmentation and removal based on instance segmentation and spatial prior clustering, comprising three steps: Step 1, inferring from the photo using the YOLO instance segmentation model, extracting the segmentation mask, and cropping it into a black background image; Step 2, extracting the foreground from the image using the GrabCut algorithm, converting it to a transparent background, and generating an RGBA format image; Step 3, performing spatial partitioning on the transparent background image based on Euclidean distance transformation, using the outer perimeter as the plate prior and the center as the food prior, performing K-means clustering in the Lab color space, automatically identifying the plate and food clusters based on the spatial enrichment ratio of each cluster at the perimeter and center, and removing plate pixels based on external reachability connectivity verification, outputting a transparent background image containing only the food. This invention eliminates the need for manual region labeling or setting color thresholds, can adaptively handle plates of different colors, shapes, and sizes, is suitable for diverse dining scenarios, and supports tasks such as food recognition and nutritional estimation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing, specifically to the field of food image analysis in the catering scenario, and relates to a method for food image segmentation and removal based on instance segmentation and spatial prior clustering. Background Technology

[0002] In applications such as digital management, nutritional analysis, and dish recognition within the catering industry, acquiring pure dish images containing only the dish itself, excluding plates and backgrounds, is of significant practical value. Existing dish image datasets typically include tableware such as plates and bowls, and the plate area overlaps with the dish in terms of color, texture, and other features, interfering with downstream tasks such as dish recognition and nutritional estimation.

[0003] Currently, methods for removing plates from food images mainly face the following difficulties: First, plates come in various shapes, including round, elliptical, boat-shaped, and other irregular shapes, making it impossible to rely on a fixed geometric model for detection; second, the edges of plates are uneven in thickness, and the width varies significantly at different locations, making it difficult to process them with a uniform erosion step size; third, some food colors are similar to the plate color (e.g., white plate with white food), and relying solely on a fixed color threshold is prone to misjudgment; fourth, there are semi-transparent pixels at the image edges caused by anti-aliasing, whose RGB values ​​are shifted due to mixing with the background, and directly participating in color sampling would seriously pollute the statistical results; fifth, there is currently a lack of a systematic and universal plate removal method that can automatically adapt to different plate colors and shapes.

[0004] Therefore, a method is needed to automatically and accurately remove plates from food images to support further analysis and application of food images. Summary of the Invention

[0005] The technical problem to be solved by this invention is to address the shortcomings of the existing technology by providing a method for food image segmentation and removal based on instance segmentation and spatial prior clustering. The method extracts single food images through YOLO instance segmentation, removes the black background to generate a transparent image through GrabCut, and then automatically identifies and removes the plate area by introducing spatial prior constraints that "the outer area of ​​the image must be the plate and the central area of ​​the image must be the food" through K-means color clustering. Finally, the method outputs a transparent background image containing only the food content, which is applicable to combinations of plates and food of various shapes and colors.

[0006] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for food image segmentation and removal based on instance segmentation and spatial prior clustering includes:

[0008] Step 1: Use the YOLO instance segmentation model to infer the image, extract the segmentation mask one by one, and crop and save it.

[0009] Step 2: Extract the foreground from the black background image output in Step 1 using the GrabCut algorithm to generate an RGBA format image with a transparent background;

[0010] Step 3: Using the K-means color clustering method based on spatial priors, identify and remove specific regions from the transparent background image output in Step 2, retaining the desired content.

[0011] To optimize the above technical solution, the specific measures also include:

[0012] Step 1 above uses the YOLO instance segmentation model to infer the image, extracting segmentation masks one by one and cropping and saving them, including:

[0013] First, a pre-trained YOLO instance segmentation model is loaded, and forward inference is performed on a photo of a meal containing one or more dishes to obtain a pixel-level binary segmentation mask for each detected target object. By performing a bitwise AND operation between the binary mask and the original image, a mask image is obtained that retains only the target region, while non-target regions are filled with black pixels. Next, contour detection is performed on the mask for each target, its minimum bounding rectangle coordinates are calculated, and the mask image is cropped within this rectangle, removing excess black border areas. Finally, each individual dish corresponds to a cropped image file, with the black area serving as the background, which is used as input for step 2.

[0014] Step 2 above uses the GrabCut algorithm to extract the foreground from the black background image output in step 1, generating an RGBA format image with a transparent background, including:

[0015] A rectangular region with 2.5% margin in both width and height is used as the initial Region of Interest (ROI) for the GrabCut algorithm. The GrabCut iterative algorithm is invoked to automatically build a Gaussian mixture model of the overall foreground of the dish and plate and the black background, optimizing the foreground / background assignment of each pixel. An alpha transparency channel is constructed based on the pixel label matrix output by GrabCut: pixels labeled as either definitive background (GC_BGD) or possible background (GC_PR_BGD) have their alpha values ​​set to 0 (completely transparent), while the alpha values ​​of the remaining pixels are set to 255 (completely opaque). The original BGR three-channel image is merged with the newly constructed alpha channel to generate an RGBA four-channel image. Finally, by scanning the distribution range of non-zero alpha values ​​in the rows and columns, the completely transparent blank areas around the edges are automatically cropped, generating a compact transparent background RGBA image, which is saved as a PNG file.

[0016] Step 3 above identifies and removes specific regions using the K-means color clustering method based on spatial priors, including:

[0017] First, the RGBA image output from step 2 is read, and its alpha channel is filtered to extract only fully opaque pixels with an alpha value of at least 250. A reliable solid region mask is then constructed to completely eliminate interference from semi-transparent mixed pixels at the image edges caused by anti-aliasing rendering. A distance transformation based on Euclidean distance (cv2.distanceTransform) is performed on the reliable solid region mask to obtain the distance value from each opaque pixel to the nearest transparent boundary. This distance value represents the "depth" of the pixel within the effective area of ​​the image; a smaller distance indicates that the pixel is closer to the outer transparent boundary (i.e., the plate), while a larger distance indicates that the pixel is more centrally located (i.e., the dish).

[0018] Based on the distance transformation results, two spatial prior regions are established: pixels with distance values ​​between 1.5 pixels and 15% of the maximum distance are defined as the outer region, and based on the scene characteristic of "the plate surrounds the dish," these pixels are determined to originate from the plate; pixels with distance values ​​greater than 50% of the maximum distance are defined as the central region, and based on the scene characteristic of "food occupying the center of the plate," these pixels are determined to originate from the dish. The median color of the pixels in the Lab color space is calculated for each region, serving as the plate color reference and the dish color reference, respectively.

[0019] The Lab color values ​​of all opaque pixels are fed into the K-means algorithm to obtain the color cluster number of each pixel, and the cluster affiliation of each pixel is recorded on a label map of the same size as the original image.

[0020] For each color cluster, calculate its periphery enrichment and center enrichment. Periphery enrichment is defined as the proportion of pixels in the periphery region of the cluster to the total number of pixels in the cluster, divided by the proportion of pixels in the periphery region to the total number of entity pixels. Center enrichment is defined similarly. If the periphery enrichment of a cluster is greater than its center enrichment, and the periphery enrichment exceeds 1.3, then the cluster is classified as a plate cluster; otherwise, it is classified as a dish cluster. An auxiliary rule is: if the periphery enrichment of a cluster is greater than 1.0, and the distance between the cluster center and the plate color reference in Lab space does not exceed 40% of the distance between the cluster center and the dish color reference, then it is also classified as a plate cluster.

[0021] All pixels covered by the plate clusters are merged into a plate candidate mask. To prevent misclassification of areas inside the dish with colors similar to those of a plate as plates, connectivity verification is performed on the plate candidate mask: a reachability graph is constructed by combining the transparent areas of the image (outer background) with the plate candidate mask. Through connected component analysis, only pixels belonging to plate candidates in connected regions directly reachable from the image boundary are retained, while isolated candidate pixels without connected paths to the outside are disconnected.

[0022] Morphological closing and opening operations are performed on the verified plate mask, and semi-transparent anti-aliased pixels adjacent to the plate area are marked as transparent. An alpha gradient feathering process is applied to the final boundary between the plate and the dish to make the edge transition natural and smooth. The alpha value of the area covered by the plate mask is set to 0, and the output is an RGBA image with a transparent background that retains only the dish.

[0023] The image segmentation and removal method of the present invention can automatically adapt to plates of different colors, shapes and sizes without manually marking plate areas or manually setting color thresholds. It utilizes the prior constraint of "the periphery must be a plate and the center must be a dish" that naturally exists in the image spatial structure, and achieves automatic judgment through K-means color clustering and spatial enrichment index. Combined with connectivity verification, it effectively prevents misjudgment and is suitable for diverse real-world dining scenarios.

[0024] The present invention has the following beneficial effects:

[0025] This invention uses the YOLO instance segmentation model (step 1) to automatically split multi-dish scenes into single-dish images, eliminating the need for manual cropping and greatly improving batch processing efficiency. It can also correctly handle situations where multiple dishes coexist in the same scene.

[0026] This invention uses the GrabCut algorithm (step 2) to convert a black background image into a transparent background RGBA image, providing a clean input for color analysis in subsequent steps, eliminating the interference of the black background on color sampling, and preserving the edge details of the image.

[0027] In the plate removal stage (step 3), this invention introduces spatial prior constraints and uses distance transformation to naturally segment the image into an outer plate area and a central food area. Spatial enrichment is used as the criterion, so that the K-means clustering results have clear semantic meaning. There is no need to manually specify the plate color or color threshold. It has good adaptability and generalization ability and is effective for various plate colors such as white, yellow, and pink.

[0028] This invention specifically addresses the interference from semi-transparent pixels at the image edges that cause anti-aliasing. By using only fully opaque pixels with an alpha value of at least 250 for color analysis, it avoids color deviation caused by semi-transparent mixed pixels and ensures the accuracy of the plate's color sampling.

[0029] This invention introduces a connectivity verification mechanism to ensure that only the plate pixels connected to the external transparent area are removed, effectively preventing the internal area of ​​the dish from being misjudged as a plate due to similar colors, thus improving the robustness of the method.

[0030] This invention employs a three-step pipeline design, with each module having independent functions and clear interfaces. It can flexibly skip existing intermediate results according to actual needs and supports both single-step invocation and end-to-end one-click processing, demonstrating excellent engineering practicality. Attached Figure Description

[0031] Figure 1 The overall flowchart of the food image processing pipeline provided by the present invention;

[0032] Figure 2 This is an operation diagram of the GrabCut algorithm for food image processing provided by the present invention;

[0033] Figure 3 This is an operational diagram of the K-means clustering algorithm for food image processing provided by the present invention; Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0035] Although the steps in this invention are arranged by numbers, steps 1 to 3 have a clear data flow dependency and must be executed sequentially. The overall process used in this invention is as follows: Figure 1 As shown.

[0036] Step 1: YOLO instance splitting

[0037] Load the YOLOv8-seg pre-trained model and perform instance segmentation inference on the input food image to obtain a segmentation mask for each detected object. Perform morphological closing (filling holes) and opening (removing minor noise) operations on the validated plate mask. Mark the semi-transparent anti-aliased pixels (alpha value between 1 and the threshold) adjacent to the plate area as the region to be removed to eliminate residual semi-transparent edges. Apply alpha gradient feathering (feather radius preferably 2 pixels) at the boundary between the plate and the dish to make the edge transition natural and smooth. Finally, set the alpha value of the area covered by the plate mask to 0 and output an RGBA image with only the transparent background of the dish. Traverse each mask, scale it to the same size as the original image using bilinear interpolation, and threshold it (>0.5) to generate a binary mask. Perform a bitwise AND operation on the original image using the binary mask to obtain a single dish mask image with a black background. Detect the mask contour and take the bounding rectangle of the largest connected region. Crop the mask image to remove the surrounding redundant black areas.

[0038] Step 2: GrabCut removes the background

[0039] Read the black background image output from step 1. Using a rectangle with its width and height inverted by 2.5% as the initial ROI for GrabCut, call cv2.grabCut to perform 10 iterations. Construct an alpha channel based on the output pixel marker matrix (mask value of 2 or 0 for background, 1 or 3 for foreground), with background pixels having alpha = 0 and foreground pixels having alpha = 255. Merge the BGR channel and alpha channel into a four-channel RGBA image, automatically crop to remove fully transparent edges, and save as a PNG file.

[0040] Step 3: K-means space prior clustering to remove disks

[0041] Read the RGBA image output from step 2.

[0042] (1) Reliable Entity Mask Construction: Read the RGBA image generated in step 2, extract its Alpha channel, and filter pixels with Alpha values ​​not lower than a preset threshold (preferably 250) to construct a reliable entity region mask. This operation is used to exclude semi-transparent mixed pixels at the image edges caused by anti-aliasing rendering, avoiding interference with subsequent color analysis. Morphological closing and opening operations are sequentially performed on the reliable entity region mask to eliminate small holes and noise.

[0043] (2) Distance Transform: Perform Euclidean distance transformation (cv2.distanceTransform) on the entity mask obtained in step (1) to obtain the Euclidean distance value from each entity pixel to the nearest transparent boundary (Alpha value is 0). This distance value represents the "depth" of the pixel in the entity region of the image. The smaller the distance, the closer the pixel is to the outer transparent boundary (i.e., the plate region), and the larger the distance, the closer the pixel is to the image center (i.e., the dish region). Construct two spatial prior regions based on the distance transformation results:

[0044] The outer region is defined as a distance value between 1.5 pixels and 15% of the maximum distance (and the maximum distance value is not less than 8 pixels). Based on the scene characteristic of "the plate surrounds the dish", it is determined that the pixels in this region must come from the plate.

[0045] Central region: defined as pixels with a distance value not less than 50% of the maximum distance. Based on the scene characteristic of "the dish occupies the center of the plate", it is determined that the pixels in this region must come from the dish.

[0046] (3) K-means color clustering: The RGB color values ​​of all reliable entity pixels are converted to the Lab color space to enhance color perception consistency. The K-means algorithm is used to cluster the Lab color values ​​(the number of clusters K is preferably 6, the initialization method is K-means++, and the maximum number of iterations is 100), to obtain the color cluster number to which each pixel belongs, and to generate a cluster label map of the same size as the original image.

[0047] (4) Clustering Determination: For each cluster k, calculate: Peripheral Enrichment = (Number of pixels in the peripheral region of cluster k / Total number of pixels in cluster k) / (Total number of pixels in the peripheral region / Total number of pixels in the entity); the same applies to the center enrichment. Determination Rule: If the peripheral enrichment > the center enrichment and the peripheral enrichment > 1.3, then the cluster is a plate cluster. Auxiliary Rule: If the peripheral enrichment > 1.0 and the Lab distance from the center of cluster k to the plate color reference is < the distance to the dish color reference × 0.4, it is also determined to be a plate cluster.

[0048] (5) Candidate Mask Generation and Connectivity Verification: All pixels identified as plate clusters are merged to form a plate candidate mask. To prevent isolated areas inside the dish with colors similar to plates from being misidentified as plates, connectivity verification is introduced: a connected graph is constructed by combining the transparent areas of the image (Alpha value of 0) with the plate candidate mask. Through connected component analysis, only candidate pixels directly connected to the image boundaries (four sides) are retained as the final plate mask, and isolated candidate areas without external connectivity are disconnected.

[0049] (6) Output: Perform morphological closing (filling holes) and opening (removing minor noise) operations on the verified plate mask. Mark the semi-transparent anti-aliased pixels (Alpha value between 1 and the threshold) adjacent to the plate area as the area to be removed to eliminate residual semi-transparent edges. Apply an Alpha gradient feathering process (feather radius preferably 2 pixels) at the boundary between the plate and the dish to make the edge transition natural and smooth. Finally, set the Alpha value of the area covered by the plate mask to 0, and output an RGBA image with only the transparent background of the dish.

Claims

1. A method for food image segmentation and removal based on instance segmentation and spatial prior clustering, characterized in that, include: Step 1: Use the YOLO instance segmentation model to infer the image, extract the segmentation mask one by one, and crop and save it. Step 2: Extract the foreground from the black background image output in Step 1 using the GrabCut algorithm to generate an RGBA format image with a transparent background; Step 3: Using the K-means color clustering method based on spatial priors, identify and remove specific regions from the transparent background image output in Step 2, retaining the desired content.

2. The method for food image segmentation and removal based on instance segmentation and spatial prior clustering according to claim 1, characterized in that, Step 1 involves inferring about the image using the YOLO instance segmentation model, extracting segmentation masks one by one, and cropping and saving them, including: Load the pre-trained YOLO instance segmentation model, perform inference on the input image, and obtain the pixel-level binary mask of each detected target; perform a bitwise AND operation between the mask and the original image to obtain a mask image containing only the corresponding dish area; calculate the minimum bounding rectangle of each target using contour detection, and crop out the corresponding local image, filling the non-target areas with black; save the cropping results of each dish as independent image files as input for subsequent steps.

3. The method for food image segmentation and removal based on instance segmentation and spatial prior clustering according to claim 1, characterized in that, Step 2 uses the GrabCut algorithm to extract the foreground from the black background image output in Step 1, generating an RGBA format image with a transparent background, including: A rectangular region with a certain margin at the edge of the image is used as the region of interest for initialization in GrabCut. This is used to start GrabCut iterations, driving the algorithm to automatically distinguish the foreground of the dishes and plates from the black background. A transparency channel is constructed based on the pixel markers output by GrabCut. The alpha value of the pixels that determine the background and the possible background is set to 0, and the rest are set to 255. The original BGR three channels and the newly created alpha channel are merged into RGBA format. The merged image is automatically cropped, and the blank rows and columns that are completely transparent around the edges are removed to generate a compact transparent background RGBA image.

4. The method for food image segmentation and removal based on instance segmentation and spatial prior clustering according to claim 1, characterized in that, Step 3 involves identifying and removing specific regions using a spatial prior-based K-means color clustering method, including: For the RGBA image output in step 2, only fully opaque pixels with an alpha value of at least 250 are extracted to construct a reliable entity region mask, thus eliminating the interference of anti-aliased semi-transparent pixels at the image edges on color analysis. An Euclidean distance transformation is performed on the entity region mask to obtain the distance value of each pixel to the nearest transparent boundary, thereby dividing the image into two spatial prior regions: a smaller distance outer plate prior region and a larger distance central dish prior region. K-means clustering is performed on all entity pixels in the Lab color space, dividing all pixels into several clusters with similar colors. For each cluster, its spatial enrichment in the outer and central regions is calculated, with the outer region being richer than the central region. The enrichment degree is defined as the proportion of pixels in the outer region of the cluster divided by the proportion of pixels in the outer region to the total number of pixels. The enrichment degree of the center is defined similarly. If the enrichment degree of the outer region of a cluster is significantly higher than that of its center, and the ratio exceeds a set threshold, it is determined to be a plate cluster; otherwise, it is determined to be a dish cluster. The connectivity of the pixels covered by the plate cluster is verified, and only pixels that can be reached from the transparent outer region of the image are retained to prevent misjudging similar color areas inside the dish as plates. Morphological cleaning is performed on the plate region, and the semi-transparent edge pixels adjacent to the plate region are made transparent. An alpha gradient feathering is applied at the boundary between the plate and the dish, and the final output is a transparent background dish image after removing the plate.

Images