Interactive image segmentation method based on mesh region guidance

Abstract

This invention relates to an interactive image segmentation method based on grid region guidance, comprising the following steps: Step S1: acquiring the image to be processed and selecting a grid size according to the segmentation target attributes; Step S2: covering the image to be processed with a grid region of the corresponding size selected in Step S1; Step S3: during interactive selection, judging click behavior and movement trajectory in real time, recording and visualizing the grid region selected by the user; Step S4: after confirming that all selections are completed, converting the recorded selection regions into Gaussian distance maps, generating positive and negative guidance, and stitching them with the original image to obtain input data; Step S5: constructing and training a segmentation network, segmenting the target according to the input data using the trained segmentation network, and returning the initial segmentation result; Step S6: based on the initial segmentation result, correcting the erroneous regions by selecting square boxes of a preset size; Step S7: modifying the interactive guidance according to the corrected selected regions, re-inputting the network segmentation, and returning the refined result.

Description

Technical Field

[0001] This invention relates to the field of image segmentation, and more specifically to an interactive image segmentation method based on grid region guidance. Background Technology

[0002] The advent of the big data era has driven the rapid development of deep learning technology, and the application and research of deep neural networks in the field of Artificial Intelligence (AI) have received unprecedented attention. The success of deep learning not only benefits from the rapid growth of computing resources but also relies on massive, high-quality training datasets; that is, high-quality labeled datasets are a fundamental guarantee of good performance. Therefore, the demand for labeling large-scale datasets has increased dramatically, and these data can be applied to fields such as semantic segmentation, instance segmentation, salient object detection, and image and video processing. However, labeling large amounts of pixel-level training datasets is an extremely difficult task, requiring a significant amount of human time and effort. Simple and efficient interactive image segmentation algorithms can greatly reduce the cost of manual labeling, making it more convenient and efficient to label training datasets and improving the efficiency of dataset generation. This has significant practical implications for achieving high-precision and intelligent operations. Interactive segmentation methods can be broadly divided into traditional methods and deep learning-based methods. Most traditional interactive image segmentation methods consider boundary segmentation or graph cutting methods. They primarily consider boundary attributes during segmentation, which leads to poor performance on weak edges. In recent years, with the development of deep learning, deep learning-based methods have developed rapidly, enabling more effective learning of various features and interactive guidance, achieving great success in interactive image segmentation tasks. Commonly used interaction methods include clicking, drawing, and bounding boxes. However, existing interaction methods still have some limitations, such as insufficient flexibility in interactive annotation, unclear response to user intent, low segmentation accuracy, and the need for significant user interaction. These interactive image segmentation methods still have considerable room for improvement when facing complex natural images, medical images, and other challenging scenarios. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide an interactive image segmentation method based on grid region guidance, which aims to solve the above-mentioned problems.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] An interactive image segmentation method based on grid region guidance includes the following steps:

[0006] Step S1: Obtain the image to be processed and select the grid size according to the segmentation target attributes;

[0007] Step S2: Cover the grid area of ​​the corresponding size selected in step S1 on the image to be processed;

[0008] Step S3: During interactive selection, the click behavior and movement trajectory are judged in real time, and the selected grid area is recorded and visualized.

[0009] Step S4: After confirming that all selections have been completed, convert the recorded selection area into a Gaussian distance map, generate positive and negative guides, and stitch them with the original image to obtain the input data;

[0010] Step S5: Construct and train the segmentation network. Use the trained segmentation network to segment the target based on the input data and return the initial segmentation result.

[0011] Step S6: Based on the initial segmentation results, the error area is corrected by selecting a square box of a preset size;

[0012] Step S7: Modify the interactive guidance based on the area selected in the correction box, re-enter the network segmentation, and return the refined results.

[0013] Furthermore, the target attributes include target size, shape, regularity, and boundary clarity.

[0014] Furthermore, in step S2, grid areas are arranged from left to right and from top to bottom on the image to be processed.

[0015] Furthermore, when selecting a grid, clicking the left mouse button again marks the end of the selection. You can also click the right mouse button to cancel the mistakenly selected grid area, and click the right mouse button again to end the cancellation operation. In addition, you can double-click the left mouse button to fill the unselected area within the closed area formed by the selected grid, or drag along the diagonal to select a rectangular area. By selecting the grid area, existing commonly used interaction methods are cleverly combined.

[0016] Furthermore, step S4 specifically involves: converting the selected area into a Gaussian distance map with a radius of 7 as positive guidance for interaction; finding the four extreme points of the selected area to obtain the circumscribed rectangle, then expanding the circumscribed rectangle by 10 pixels, and using the Gaussian distance map with a radius of 10 at its four corners as negative guidance; and then stitching the generated positive / negative guidance with the original image as input data.

[0017] Furthermore, when constructing the input data for the segmentation network, the selected region is expanded outward by 30 pixels, and the guide and original image within this range are cropped and then stitched together.

[0018] Furthermore, the segmentation network adopts a cascaded segmentation network, including two main network modules: CoarseNet and FineNet.

[0019] Furthermore, the training data for the segmentation network is obtained using a simulated sampling strategy, simulating the grid regions that the user might select as training data. The simulation takes into account the selected grid size and the chosen grid region, as detailed below:

[0020] When simulating and sampling the selected mesh size, the shorter side of the circumscribed rectangle is used as the reference. Two factors are considered: the target size s and the degree of regularity t. The sampling method for the mesh size is as follows:

[0021] size=C*w*t / s

[0022] Where C is a constant set based on experience, and w is the pixel value of the shorter side of the circumscribed rectangle;

[0023] The target size 's' is determined by the proportion of the target's bounding rectangle in the original image. Based on the proportion, it is divided into three sizes: large (s ≥ 0.6), medium (s ≥ 0.3), and small. The values ​​of 's' are then redefined as follows:

[0024]

[0025] The degree of regularity 't' is determined by the proportion of the target's actual location within its bounding box. Based on the proportion, it is divided into two cases: regular (t ≥ 0.4) and irregular. The value of 't' is then recalculated as follows:

[0026]

[0027] When simulating the selection of a grid area, all grids containing the target area are first selected. Then, based on the statistical results collected from volunteers, some grids in the boundary areas are deselected with a corresponding probability to simulate the user's actual selection.

[0028] Furthermore, step S7 specifically involves: selecting a square frame of a preset size based on the preliminary segmentation results, selecting the area with errors, and further interactively improving it.

[0029] Furthermore, the further interactive improvement specifically involves: distinguishing between positive and negative guidance based on left / right clicks; applying the selected area to the previous interactive guidance; modifying the value of the corresponding guidance area; stitching the modified guidance with the original image; inputting it back into the network for re-prediction; and obtaining the improved result. This iterative interaction continues until a satisfactory segmentation result is obtained.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] 1. The grid-region-guided interaction method of this invention can cleverly combine the advantages of commonly used interaction methods and shield them from the limitations of using them alone;

[0032] 2. This invention implements a grid-based sampling strategy, unifying different forms of sampling strategies and avoiding complex graffiti stroke sampling.

[0033] 3. This invention can significantly improve the efficiency of interactive image segmentation, achieving higher accuracy with fewer interactions while maintaining flexible interaction. It can also be well applied to datasets in different fields and has high application value. Attached Figure Description

[0034] Figure 1 This is a flowchart of the method of the present invention;

[0035] Figure 2 This is an overall framework diagram of one embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram of the structure of a cascaded segmentation network used in one embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of the segmentation result in one embodiment of the present invention. Detailed Implementation

[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0039] Please refer to Figure 1 This invention provides an interactive image segmentation method based on grid region guidance, comprising the following steps:

[0040] Step S1: Obtain the image to be processed and select the grid size according to the segmentation target attributes;

[0041] Step S2: Cover the grid area of ​​the corresponding size selected in step S1 on the image to be processed;

[0042] Step S3: During interactive selection, the click behavior and movement trajectory are judged in real time, and the selected grid area is recorded and visualized.

[0043] Step S4: After confirming that all selections have been completed, convert the recorded selection area into a Gaussian distance map, generate positive and negative guides, and stitch them with the original image to obtain the input data;

[0044] Step S5: Construct and train the segmentation network. Use the trained segmentation network to segment the target based on the input data and return the initial segmentation result.

[0045] Step S6: Based on the initial segmentation results, the error area is corrected by selecting a square box of a preset size;

[0046] Step S7: Modify the interactive guidance based on the area selected in the correction box, re-enter the network segmentation, and return the refined results.

[0047] In this embodiment, the size of the grid region used for selection can be chosen by the user based on attributes such as target size, shape, regularity, and boundary clarity. If the target is large and has a relatively regular shape, a larger grid region can be used, enabling rapid selection of the target region. If the target is small or the boundaries are not clear enough, a smaller grid can be used, which can more accurately indicate the target region and improve the accuracy of interactive segmentation.

[0048] Based on experiments and statistics, the optimal range for adjusting the grid size is 10–50 pixels, adjusted in 10-pixel increments. This is because if the grid area is too large, too many background areas will be incorrectly selected, affecting the segmentation accuracy of the boundary areas. Conversely, if it is too small, users will need to spend more time selecting the grid area, which defeats the purpose of interactive segmentation.

[0049] In this embodiment, the grid areas are arranged from left to right and from top to bottom in step S2. However, the length and width of all images may not be divisible by multiples of ten (10-50) simultaneously. Therefore, after the preceding square grid arrangement is completed, the remaining area is defined as a rectangular area of ​​the corresponding size, which can be selected normally like other grid areas.

[0050] In this embodiment, in step S3, after the selection begins, the interactive front end will continuously listen to and record the area selected by the user. Specifically, all the arranged grid objects will be continuously in a listening state. When the cursor enters the grid area, the border color of the area will be changed from gray to green, and the value of the area will be modified in a matrix of the same size as the original image to mark the area as selected.

[0051] When selecting a grid, clicking the left mouse button again marks the end of the selection. You can also click the right mouse button to cancel the mistakenly selected grid area, and click the right mouse button again to end the cancellation operation. In addition, you can double-click the left mouse button to fill the unselected area within the closed area formed by the selected grid, or drag along the diagonal to select a rectangular area. By selecting the grid area, existing common interaction methods are cleverly combined.

[0052] In this embodiment, step S4 specifically involves: the selected grid region being considered as the region where the target is located. The selected region is converted into a Gaussian distance map with a radius of 7, serving as positive guidance for interaction; the four extreme points of the selected region are found to obtain the bounding rectangle, which is then expanded by 10 pixels, and the four corner points are used as negative guidance using a Gaussian distance map with a radius of 10. The generated positive / negative guidance is then stitched together with the original image and fed into the segmentation grid for segmentation prediction.

[0053] Preferably, when constructing the input data for the network, the guide with the original image size is not directly stitched together with the original image. Instead, the selected area is expanded outward by 30 pixels, and the guide within that range and the original image are cropped and then stitched together.

[0054] In this embodiment, the segmentation network employs a cascaded structure, comprising two main network modules: CoarseNet and FineNet. The CoarseNet sub-network, together with the Pyramid Scene Parsing (PSP) module, utilizes a design similar to FPN. This combination ensures the effective fusion of features from different levels, enriching global contextual information and enabling the network to better handle objects with varying scales and semantic information. FineNet employs a multi-scale fusion structure, upsampling and concatenating information from various levels in CoarseNet to recover details lost during convolution and improve the accuracy of segmentation boundaries.

[0055] Preferably, the training data for the segmentation network is obtained using a simulated sampling strategy, simulating the grid regions that the user might select as training data. The simulation takes into account the selected grid size and the chosen grid region, as detailed below:

[0056] When simulating and sampling the selected mesh size, the shorter side of the circumscribed rectangle is used as the reference. Two factors are considered: the target size s and the degree of regularity t. The sampling method for the mesh size is as follows:

[0057] size=C*w*t / s

[0058] Where C is a constant set based on experience, and w is the pixel value of the shorter side of the circumscribed rectangle;

[0059] The target size 's' is determined by the proportion of the target's bounding rectangle in the original image. Based on the proportion, it is divided into three sizes: large (s ≥ 0.6), medium (s ≥ 0.3), and small. The values ​​of 's' are then redefined as follows:

[0060]

[0061] The degree of regularity 't' is determined by the proportion of the target's actual location within its bounding box. Based on the proportion, it is divided into two cases: regular (t ≥ 0.4) and irregular. The value of 't' is then recalculated as follows:

[0062]

[0063] When simulating the selection of a grid area, all grids containing the target area are first selected. Then, based on the statistical results collected from volunteers, some grids in the boundary areas are deselected with a corresponding probability to simulate the user's actual selection.

[0064] In this embodiment, step S7 specifically involves: the preliminary segmentation result being returned to the front end for display; based on the segmentation result, a square box of appropriate size (10-50 pixels) can be selected to identify areas with errors for further interactive improvement; positive / negative guidance is distinguished by left / right clicks; the selected area is applied to the interactive guidance from the previous step; the modified guidance is then stitched together with the original image and input into the network for re-prediction to obtain an improved result; this iterative interaction continues until a satisfactory segmentation result is obtained.

[0065] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. An interactive image segmentation method based on grid region guidance, characterized in that, Includes the following steps, Step S1: Obtain the image to be processed and select the grid size according to the segmentation target attributes; Step S2: Cover the grid area of ​​the corresponding size selected in step S1 on the image to be processed; Step S3: During interactive selection, the click behavior and movement trajectory are judged in real time, and the selected grid area is recorded and visualized. Step S4: After confirming that all selections have been completed, convert the recorded selection area into a Gaussian distance map, generate positive and negative guides, and stitch them with the original image to obtain the input data; Step S5: Construct and train the segmentation network. Use the trained segmentation network to segment the target based on the input data and return the initial segmentation result. Step S6: Based on the initial segmentation results, the error area is corrected by selecting a square box of a preset size; Step S7: Modify the interactive guidance based on the area selected in the correction box, re-enter the network segmentation, and return the refined results; The training data for the segmentation network is obtained using a simulated sampling strategy, simulating the grid region selected by the user as the training data. The simulation takes into account the selected grid size and the selected grid region, as detailed below: When simulating and sampling the selected mesh size, the shorter side of the circumscribed rectangle is used as the reference. Two factors are considered: the target size s and the degree of regularity t. The sampling method for the mesh size is as follows: size=C*w*t / s Where C is a constant set based on experience, and w is the pixel value of the shorter side of the circumscribed rectangle; The target size 's' is determined by the proportion of the target's bounding rectangle in the original image. Based on the proportion, it is divided into three sizes: large, medium, and small. 's < 0.3' indicates a small size, '0.6 > s ≥ 0.3' indicates a medium size, and 's ≥ 0.6' indicates a large size. The values ​​of 's' are then redefined as follows: The degree of regularity, 't', is determined by the proportion of the target's actual location within its bounding box. Based on this proportion, it is categorized into regular and irregular cases. t ≥ 0.4 indicates a regular case, and t < 0.4 indicates an irregular case. The following values ​​are then assigned to 't': When simulating the selection of a grid area, all grids containing the target area are first selected. Then, based on the statistical results collected from volunteers, some grids in the boundary areas are deselected with a corresponding probability to simulate the user's actual selection.

2. The interactive image segmentation method based on grid region guidance according to claim 1, characterized in that, The target attributes include target size, shape, regularity, and boundary clarity.

3. The interactive image segmentation method based on grid region guidance according to claim 1, characterized in that, In step S2, grid areas are arranged from left to right and from top to bottom on the image to be processed.

4. The interactive image segmentation method based on grid region guidance according to claim 1, characterized in that, When selecting a grid, clicking the left mouse button again marks the end of the selection. Clicking the right mouse button cancels the mistakenly selected grid area, and clicking the right mouse button again ends the cancellation operation. In addition, double-clicking the left mouse button fills the unselected area within the closed area formed by the selected grid, or drags along the diagonal to select a rectangular area. By selecting the grid area, existing common interaction methods can be combined.

5. The interactive image segmentation method based on grid region guidance according to claim 1, characterized in that, Step S4 specifically involves: converting the selected area into a Gaussian distance map with a radius of 7, which serves as the positive guide for interaction; finding the four extreme points of the selected area to obtain the circumscribed rectangle, then expanding the circumscribed rectangle by 10 pixels, and using the Gaussian distance map with a radius of 10 at its four corners as the negative guide; and then stitching the generated positive / negative guide with the original image as input data.

6. The interactive image segmentation method based on grid region guidance according to claim 5, characterized in that, When constructing the input data for the segmentation network, the selected region is expanded outward by 30 pixels. The guide and the original image within the expanded region are then cropped and stitched together.

7. The interactive image segmentation method based on grid region guidance according to claim 1, characterized in that, The segmentation network adopts a cascaded segmentation network, including two main network modules: CoarseNet and FineNet.

8. The interactive image segmentation method based on grid region guidance according to claim 1, characterized in that, Step S7 specifically involves: selecting a square frame of a preset size based on the preliminary segmentation results, selecting the areas with errors, and further interactively improving them.

9. The interactive image segmentation method based on grid region guidance according to claim 8, characterized in that, The further interactive improvements are as follows: positive / negative guidance is distinguished based on left / right clicks, the selected area is applied to the interactive guidance of the previous step, the value of the corresponding guidance area is modified, the modified guidance is stitched with the original image and then input into the network again for re-prediction, and the improved result is obtained. This iterative interaction is repeated until a satisfactory segmentation result is obtained.

Images