A small object instance segmentation method using adaptive grid

Abstract

The present invention relates to the field of image processing technology, and specifically to a small object instance segmentation method using an adaptive grid. The method comprises the following steps: formulating a grid partitioning strategy; extracting features of an image from shallow to deep layers; extracting mask features from the image; and classifying the mask features to obtain a final instance segmentation map. The method has the following beneficial effects: the small object instance segmentation method using an adaptive grid proposed by the present invention uses a coco2014 training dataset through a grid estimation algorithm to formulate a suitable grid partitioning strategy for each image, so that the network can adopt a more appropriate training strategy according to the distribution of bboxes in the image.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a small target instance segmentation method using an adaptive grid. Background Art

[0002] Image segmentation refers to dividing an image into several non-overlapping areas based on features such as color, spatial texture, and geometric shape, so that these features show consistency or similarity within the same area, but show obvious differences between different areas; object detection is to identify the content in the image and detect its location, semantic segmentation is to classify each pixel in the image by labeling it with a category, and instance segmentation is the combination of object detection and semantic segmentation.

[0003] In existing technologies, instance segmentation can be implemented in two ways: top-down and bottom-up. The top-down approach involves detection followed by segmentation. Objects in the image are first detected using object detection, and then speech segmentation is performed on the detected bounding boxes. Each segmentation is output as a distinct instance. The bottom-up approach treats each instance as a separate category. By predicting an embedding vector, pixels are then divided into separate instances using a clustering algorithm. Instance segmentation aims to detect objects in the input image and assign a category label to each pixel of the object. It can distinguish between instances with the same foreground semantic category, a key difference from semantic segmentation. Compared to semantic segmentation, instance segmentation developed later, and therefore instance segmentation models are primarily based on deep learning techniques. With the advancement of deep learning, excellent algorithms for instance segmentation have emerged, such as DeepMask and MaskRCNN. SOLOV2 has been a highly successful state-of-the-art model in the past two years.

[0004] However, instance segmentation still has some problems and difficulties, such as small object segmentation, geometric transformation, occlusion and image degradation. Summary of the Invention

[0005] The purpose of the present invention is to provide a small object instance segmentation method using an adaptive grid to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a small object instance segmentation method using an adaptive grid, the small object instance segmentation method comprising the following steps:

[0007] S1: Develop a grid partitioning strategy;

[0008] S2: Extract features from shallow to deep layers of the image;

[0009] S3: Extract mask features from the image;

[0010] S4: Classify the mask features to obtain the final instance segmentation map.

[0011] Preferably, when formulating a grid partitioning strategy, the training data set is passed through a grid estimation algorithm to formulate a suitable grid partitioning strategy.

[0012] Preferably, when extracting features of an image from shallow layers to deep layers, the image is input into a Focal Transform pre-trained model according to a specified grid strategy to perform feature extraction from shallow layers to deep layers.

[0013] Preferably, when extracting mask features from an image, the image features extracted in step S2 are input into the recursive FPN to extract mask features.

[0014] Preferably, when classifying the mask features to obtain the final instance segmentation map, the prediction network is used to classify the obtained mask features to obtain the final instance segmentation map.

[0015] Preferably, when formulating a grid partitioning strategy, the coco 2014 training dataset is used through a grid estimation algorithm to formulate a suitable grid partitioning strategy for each image, so that the network adopts a more appropriate training strategy according to the distribution of bboxes in the image.

[0016] Preferably, the image is input into a Focal Transform pre-trained model for feature extraction according to a specified grid strategy.

[0017] Preferably, a feature image from shallow to deep layers is obtained through the backbone; the image features are input into the recursive FPN to extract mask features.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] This paper proposes an adaptive grid-based small object instance segmentation method. Using the Coco 2014 training dataset and a grid estimation algorithm, a suitable grid partitioning strategy is developed for each image, enabling the network to adopt a more appropriate training strategy based on the distribution of bounding boxes within the image. Next, based on the specified grid strategy, the image is fed into a pre-trained FocalTransform model for feature extraction. A backbone algorithm is used to generate feature images from shallow to deep layers. These image features are then fed into a recursive FPN (Neck) to extract mask features. Finally, a prediction network is used to classify the mask features to produce the final instance segmentation map. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0021] In order to clearly and completely describe the objectives and technical solutions of the present invention and make the advantages more clearly understood, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, not all of them, and are only used to explain the embodiments of the present invention, not to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0022] Example 1

[0023] See also Figure 1 The present invention provides a technical solution: a small target instance segmentation method using an adaptive grid, the small target instance segmentation method comprising the following steps:

[0024] S1: Develop a grid partitioning strategy;

[0025] S2: Extract features from shallow to deep layers of the image;

[0026] S3: Extract mask features from the image;

[0027] S4: Classify the mask features to obtain the final instance segmentation map.

[0028] Example 2

[0029] Based on the first embodiment, the specific implementation is as follows:

[0030] S1: Pass the training dataset through the grid estimation module to develop a suitable grid partitioning strategy.

[0031] Here we use the COCO2014 dataset, which contains a total of 118,287 training images. First, we read the training set and input the bbox area information S contained in the image Img1, and store all the area information in the list in ascending order. Assuming that the image Img1 contains n bboxes in total, we will get an S with a length of n. list .

[0032] S list =[S bbox 1,S bbox 2,……,S bbox n] (1)

[0033] If S list If the length n is less than 8, we will use the original grid division strategy of solov2

[0034] grid = [40,36,24,16,12]

[0035] If S list The length n is greater than or equal to 8, and the grid division strategy will be formulated according to the number of bboxes.

[0036] First, set a difference geometric sequence Seq so that S list All areas in are divided into this sequence. This interpolation geometric sequence has a total of The first and last values are Min(S list ) and Max(S list ). The length of each interval is determined by the following formula:

[0037]

[0038]

[0039] Then S list Each value in is allocated according to the interval range of Seq. Since there are multiple bbox values in each interval of Seq, the values in each interval are stored in Temp list , by the mean formula Calculate the mean of the bbox in this interval, so that we can get a length of Grid strategy list Grid list For Grid list Each value in is calculated by formula (4) to obtain the final input network strategy. Each square root value is rounded up using the numpy ceil function.

[0040] Finally, take out the first four values of the grid list, calculate the average of all values after the fourth value, and then add them up to form a grid list of length 5.

[0041]

[0042] Grid=Grid list [1,2,3,4]+Avg(Grid list [4,…,n]) (5)

[0043] S2: The training dataset image Img1 is passed through the Focal-Transform pre-training model to extract 5 layers of feature maps from shallow to deep layers.

[0044] {F1,F2,F3,F4,F5}=Focal-Transform(Img1)

[0045] S3: Input the 5-layer image features extracted by S2 into the recursive FPN to extract the high-resolution features after fusion.

[0046] {P1,P2,P3,P4,P5}=FPN({F1,F2,F3,F4,F5})

[0047] S4: Input the fusion features obtained by the FPN network into the SOLOV2 instance prediction network

[0048] The SOLOV2 instance prediction network is divided into instance branch and mask branch.

[0049] S 4-1: For the mask branch, the input is the fused features {P1, P2, P3, P4} of the first four layers obtained by FPN. After passing through the mask branch, the feature map of the mask prediction will be obtained.

[0050] S4-2: For the instance branch, mask kernel prediction and semantic category prediction can be performed.

[0051] The input for both predictions is the five-layer fused feature maps {P1, P2, P3, P4, P5} obtained from FPN. The sizes of these five fused feature maps are arranged from small to large. The grid partitioning strategy set in S1 is then taken and applied to the corresponding fused feature P.

[0052] {G1,G2,G3,G4,G5}=Grid({P1,P2,P3,P4,P5})

[0053] Among them, the feature map with large size is used to predict small samples, and the corresponding number of grid divisions should be more. The feature map with small size is used to predict large samples, and the corresponding number of grid divisions should be less.

[0054] Then the five new feature maps {G1, G2, G3, G4, G5} obtained by S4-2 are respectively

[0055] Mask kernel prediction and semantic category prediction.

[0056] The final semantic category prediction obtains category information and is then combined with the true label to calculate the category loss function. The mask kernel prediction is combined with the mask prediction. That is, the mask prediction obtains the feature map, which is then convolved with the mask kernel once to obtain the result. Finally, the mask information is obtained and the loss is calculated with the true label to obtain the loss function for mask segmentation.

[0057] S5: The SOLOV2 prediction network is hyperparameter-adjusted through the category loss function and the instance mask loss function to achieve the optimal convergence effect, and finally the SOLOV2 model is obtained.

[0058] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A small object instance segmentation method using an adaptive grid, characterized by: The small object instance segmentation method comprises the following steps: S1: Develop a grid partitioning strategy; S2: Extract features from shallow to deep layers of the image; S3: Extract mask features from the image; S4: Classify the mask features to obtain the final instance segmentation map. Said S1 is: passing the training data set through the grid estimation module to formulate a suitable grid partitioning strategy; Here we use the COCO2014 dataset, which contains a total of 118,287 training images. First, we read the training set and input the bbox area information S contained in the image Img1, and store all the area information in the list in ascending order. Assuming that the image Img1 contains n bboxes in total, we will get an S with a length of n. list . S list =[S bbox 1,S bbox 2,……,S bbox n] (1) If S list The length n is less than 8, we will use the original grid division strategy of solov2 grid = [40,36,24,16,12] If S list The length n is greater than or equal to 8, and the grid division strategy will be formulated according to the number of bboxes. First, set a difference geometric sequence Seq so that S list All areas in are divided into this sequence. This interpolation geometric sequence has a total of The first and last values are Min(S list ) and Max(S list ). The length of each interval is determined by the following formula: Then S list Each value in is allocated according to the interval range of Seq. Since there are multiple bbox values in each interval of Seq, the values in each interval are stored in Temp list , by the mean formula Calculate the mean of the bbox in this interval, so that we can get a length of Grid strategy list Grid list For Grid list Each value in is calculated by formula (4) to obtain the final input network strategy. Each square root value is rounded up using the numpy ceil function. Finally, take out the first four values of the grid list, calculate the average of all values after the fourth value, and then add them up to form a grid list of length 5. Grid=Grid list [1,2,3,4]+Avg(Grid list [4, …,n]) (5)。 2. The small object instance segmentation method using an adaptive grid according to claim 1, characterized in that: When formulating a grid partitioning strategy, the training data set is passed through a grid estimation algorithm to formulate a suitable grid partitioning strategy.

3. The small object instance segmentation method using an adaptive grid according to claim 2, characterized in that: When extracting features from shallow to deep layers of an image, the image is input into the Focal Transform pre-trained model according to the specified grid strategy for feature extraction from shallow to deep layers.

4. The small object instance segmentation method using an adaptive grid according to claim 3, characterized in that: When extracting mask features from an image, the image features extracted in step S2 are input into the recursive FPN to extract mask features.

5. The small object instance segmentation method using an adaptive grid according to claim 4, characterized in that: When the mask features are classified to obtain the final instance segmentation map, the prediction network is used to classify the obtained mask features to obtain the final instance segmentation map.

6. The small object instance segmentation method using an adaptive grid according to claim 1, characterized in that: When formulating the grid partitioning strategy, we use the coco 2014 training dataset through the grid estimation algorithm to formulate a suitable grid partitioning strategy for each image, so that the network can adopt a more appropriate training strategy according to the distribution of bboxes in the image.

7. The small object instance segmentation method using an adaptive grid according to claim 6, characterized in that: According to the specified grid strategy, the image is input into the Focal Transform pre-trained model for feature extraction.

8. The small object instance segmentation method using an adaptive grid according to claim 7, characterized in that: The backbone is used to obtain feature images from shallow to deep layers; the image features are input into the recursive FPN to extract mask features.

Images