Image segmentation method, device, mobile terminal and storage medium

Abstract

The present invention discloses an image segmentation method, device, mobile terminal and storage medium, the method comprising: obtaining multiple ROI regions of an image to be segmented, and calculating first scores of the multiple ROI regions; when the first scores meet a preset condition, determining that the ROI region is a high-frequency region; and completing the segmentation of the image to be segmented after sampling the high-frequency region according to a first upsampling method. The embodiment of the present invention can improve the accuracy of image segmentation.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an image segmentation method, device, mobile terminal and storage medium. Background Art

[0002] In the prior art, the label map predicted by the image segmentation network should be basically smooth, that is, adjacent pixels generally use the same label, because the high-frequency areas of the image are often limited to the sparse boundaries between different objects. The regular grid will oversample the smooth areas (low-frequency areas) and undersample the object boundary areas (high-frequency areas). This will lead to excessive calculations in smooth areas, making the outline of the prediction results blurred. Image segmentation methods usually predict labels based on low-resolution regular grids. The final output of the model is often not fine enough in the segmentation of the area at the edge of the object.

[0003] In summary, existing image segmentation methods have the problem of low accuracy due to oversampling of smooth areas and undersampling of object boundaries. Summary of the invention

[0004] The embodiments of the present invention provide an image segmentation method, an apparatus, a mobile terminal and a storage medium, which improve the accuracy of image segmentation.

[0005] A first aspect of an embodiment of the present application provides an image segmentation method, comprising:

[0006] Acquire multiple ROI regions of the image to be segmented, and calculate first scores of the multiple ROI regions;

[0007] When the first score meets a preset condition, the ROI region is determined to be a high-frequency region;

[0008] After the high-frequency region is sampled according to the first up-sampling method, the segmentation of the image to be segmented is completed.

[0009] In a possible implementation manner of the first aspect, obtaining multiple ROI regions of the image to be segmented is specifically:

[0010] Inputting the image to be segmented into the network model so that the network model extracts image features of the image to be segmented;

[0011] Input the image features into the candidate region generation network so that the candidate region generation network obtains the candidate box according to the image features;

[0012] After performing a feature extraction operation on the image features, a first operation result is obtained, and multiple ROI regions are generated and acquired by combining the first operation result and the candidate box.

[0013] In a possible implementation manner of the first aspect, first scores of multiple ROI regions are calculated, specifically:

[0014] After calculating the classification scores of the multiple ROI regions, the classification scores are used as the first scores.

[0015] In a possible implementation manner of the first aspect, when the first score satisfies a preset condition, specifically:

[0016] When the first score belongs to the first preset interval, it is determined that the first score meets the preset condition.

[0017] In a possible implementation manner of the first aspect, the method further includes:

[0018] Calculate the bounding box regression score of the high-frequency area, and adjust the bounding boxes of multiple high-frequency areas based on the classification score.

[0019] In a possible implementation manner of the first aspect, the high-frequency area is sampled according to the first upsampling method to complete the segmentation of the image to be segmented, specifically as follows:

[0020] Calculate a first mask of the image to be segmented according to the high-frequency region;

[0021] The first mask is enlarged to a first size according to a first upsampling method to complete segmentation of the image to be segmented; wherein the first size is the size of the image to be segmented.

[0022] In a possible implementation manner of the first aspect, when the resolution of the first mask is smaller than the resolution of the image to be segmented, the method further includes:

[0023] Performing upsampling processing on the first mask according to a first upsampling method to obtain a coarse prediction result;

[0024] According to the rough prediction result, a high-frequency prediction point is obtained; wherein the classification score of the high-frequency prediction point belongs to a first preset interval;

[0025] A characterization vector of the high-frequency prediction point is obtained, a new round of prediction is performed according to the characterization vector, and the resolution of the first mask is updated according to the new prediction result until the resolution of the first mask is equal to the resolution of the image to be segmented.

[0026] A second aspect of the embodiments of the present application provides an image segmentation device including: an acquisition module, a determination module and a segmentation module;

[0027] The acquisition module is used to acquire multiple ROI regions of the image to be segmented and calculate first scores of the multiple ROI regions;

[0028] The determination module is used to determine that the ROI area is a high-frequency area when the first score meets a preset condition;

[0029] The segmentation module is used to complete the segmentation of the image to be segmented after sampling the high-frequency area according to the first up-sampling method.

[0030] A third aspect of an embodiment of the present application provides a mobile terminal, including a processor and a memory, wherein the memory stores a computer-readable program code, and when the processor executes the computer-readable program code, the steps of the above-mentioned image segmentation method are implemented.

[0031] A fourth aspect of an embodiment of the present application provides a storage medium, which stores a computer-readable program code. When the computer-readable program code is executed, the steps of the above-mentioned image segmentation method are implemented.

[0032] Compared with the prior art, the embodiments of the present invention provide an image segmentation method, device, mobile terminal and storage medium, the method comprising: obtaining multiple ROI regions of the image to be segmented, and calculating the first scores of the multiple ROI regions; when the first scores meet the preset conditions, determining that the ROI region is a high-frequency region; after sampling the high-frequency region according to the first upsampling method, completing the segmentation of the image to be segmented.

[0033] The beneficial effect is that: the embodiment of the present invention calculates the first scores of multiple ROI regions of the image to be segmented, then determines and obtains the high-frequency region from the multiple ROI regions according to the first scores, and finally samples the high-frequency region according to the first upsampling method to complete the segmentation of the image to be segmented. The embodiment of the present invention can increase the sampling frequency of the high-frequency region, avoid the problem that the object boundary of the segmentation result is too smooth and unreal, and avoid the problem that the contour of the segmentation result is blurred due to oversampling of the low-frequency region, thereby effectively improving the accuracy of image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flowchart of an image segmentation method provided by an embodiment of the present invention;

[0035] Figure 2 An embodiment of the present invention provides a schematic diagram of the structure of a network model provided by an embodiment of the present invention;

[0036] Figure 3 It is a structural schematic diagram of an image segmentation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Reference Figure 1 , is a flowchart of an image segmentation method provided by an embodiment of the present invention, including S101-S103:

[0039] S101: Acquire multiple ROI regions of the image to be segmented, and calculate first scores of the multiple ROI regions.

[0040] In this embodiment, the step of obtaining multiple ROI regions of the image to be segmented is as follows:

[0041] Inputting the image to be segmented into a network model so that the network model extracts image features of the image to be segmented;

[0042] Inputting the image features into a candidate region generation network so that the candidate region generation network obtains a candidate frame according to the image features;

[0043] After performing a feature extraction operation on the image features, a first operation result is obtained, and a plurality of ROI regions are generated and acquired by combining the first operation result and the candidate box.

[0044] In this embodiment, the first scores of the plurality of ROI regions are calculated as follows:

[0045] After calculating the classification scores of the plurality of ROI regions, the classification scores are used as the first scores.

[0046] S102: When the first score meets a preset condition, determine that the ROI region is a high-frequency region.

[0047] In this embodiment, when the first score satisfies a preset condition, specifically:

[0048] When the first score belongs to a first preset interval, it is determined that the first score meets the preset condition.

[0049] Furthermore, the high-frequency area is an area in the picture where the color changes are more obvious, such as the edge area of ​​an object; correspondingly, the low-frequency area is an area in the picture that is relatively smooth and has less color changes.

[0050] Preferably, the first preset interval is established with 0.5 as the middle value to ensure that when the first score belongs to the first preset interval, which is equivalent to the first score of the ROI area being close to 0.5, the ROI area is determined to be a high-frequency area. Among them, 0.5 is used as the middle value because, when a rough detection is performed in the high-frequency area, the classification score is generally close to 0.5; when a rough detection is performed in the low-frequency area, the classification score is generally greater than 0.5, and the classification scores close to 0.5 are relatively few. Therefore, by screening the ROI area by the classification score (i.e., the first score), the high-frequency area can be obtained, and then the high-frequency area is sampled according to the first upsampling method and then segmented, so that the sampling frequency of the high-frequency area can be increased, avoiding the problem that the object boundary of the segmentation result is too smooth and unreal, and at the same time avoiding the problem of blurred contours of the segmentation result caused by oversampling of the low-frequency area, thereby effectively improving the accuracy of image segmentation.

[0051] In a specific embodiment, it also includes:

[0052] The border regression score of the high-frequency region is calculated, and the borders of the plurality of high-frequency regions are adjusted in combination with the classification score.

[0053] S103: After sampling the high-frequency area according to the first up-sampling method, the image to be segmented is segmented.

[0054] The first upsampling method is a double bilinear interpolation upsampling method.

[0055] In this embodiment, the high-frequency region is sampled according to the first upsampling method to complete the segmentation of the image to be segmented, specifically:

[0056] Calculating a first mask of the image to be segmented according to the high-frequency area;

[0057] The first mask is enlarged to a first size according to the first upsampling method to complete the segmentation of the image to be segmented; wherein the first size is the size of the image to be segmented.

[0058] In a specific embodiment, when the resolution of the first mask is smaller than the resolution of the image to be segmented, the method further includes:

[0059] performing upsampling processing on the first mask according to the first upsampling method to obtain a coarse prediction result;

[0060] According to the rough prediction result, a high-frequency prediction point is obtained; wherein the classification score of the high-frequency prediction point belongs to the first preset interval;

[0061] A characterization vector of the high-frequency prediction point is obtained, a new round of prediction is performed according to the characterization vector, and a resolution of the first mask is updated according to a new prediction result until the resolution of the first mask is equal to the resolution of the image to be segmented.

[0062] To further illustrate the structure of the network model, please refer to Figure 2 , Figure 2 It is a structural diagram of a network model provided by an embodiment of the present invention.

[0063] Depend on Figure 2 It can be seen that the network model includes: input layer 201, backbone network (Backbone) 202, convolutional neural network (CNN) 203, candidate region generation network (RPN) 204, RoI Pooling layer 205, RoIAlign layer 206, FC layer 207, box regression layer (box regression) 208, classification layer (classification) 209, Point Align layer 210 and PointRend layer 211.

[0064] The image to be segmented is input into the network model through the input layer 201, and first passes through the backbone network 202, so that the backbone network 202 extracts image features (Feature map); the image features pass through the convolutional neural network 203, so that the convolutional neural network 203 performs a feature extraction operation to obtain a first operation result; at the same time, the image features pass through the candidate region generation network 204 to generate a candidate frame; combine the first operation result and the candidate frame to generate multiple ROI regions; the multiple ROI regions are input into the classification layer 209 to obtain the classification score of the ROI region, and the high-frequency region is selected from the multiple ROI regions according to the classification score; the high-frequency region is input into the border regression layer 208 to obtain the border regression score of the high-frequency region, and the border of the high-frequency region is adjusted according to the classification score and the border regression score to obtain a first adjustment result; the first adjustment result is input into the Point Align layer 210 to allow Point Alignment to be performed. The Align layer 210 extracts the features of the first adjustment result and calculates the first mask according to the features of the first adjustment result; then the first mask is input into the PointRend layer 211, so that the PointRend layer 211 expands the first mask to the original image size to complete the segmentation of the image to be segmented.

[0065] The feature extraction operation performed by the convolutional neural network 203 is specifically as follows:

[0066] After the features are extracted by the RoI Pooling layer 205 , the features are input into the FC layer 207 , so that the FC layer 207 predicts the category of the object in the box and performs bounding box fine-tuning, and outputs the first operation result.

[0067] Among them, the PointRend layer mainly implements an iterative upsampling process. When the output resolution is smaller than the image resolution, the following operations are performed:

[0068] 1. The features of the RoI extracted by the RoI Align layer are doubled and the calculated mask result is upsampled by bilinear interpolation to obtain the coarse prediction result (coarse prediction_i).

[0069] 2. Select N "difficult points" from the rough prediction results. "Difficult points" are points where the results are likely to be different from the surrounding points (such as the edges of objects).

[0070] 3. For each difficulty point, obtain its "representation vector". The "representation vector" consists of two parts. One is the low-level feature, which is obtained by using the coordinates of the point and performing bilinear interpolation on the low-level feature map (RoIAlign). The other is the high-level feature.

[0071] 4. Use MLP to calculate the "representation vector" to get a new prediction, update coarseprediction_i to get coarseprediction_i+1. This MLP can actually be seen as a small network composed of multiple conv1x1 that only operates on the "representation vector" of the "difficult point".

[0072] 5. Repeat the above steps until the output resolution is equal to the image resolution.

[0073] Furthermore, the training process of the network model includes:

[0074] Step 1: Initialize the feature extraction network Backbone;

[0075] Step 2: Input the image into the Backbone network for feature extraction, and extract the candidate box through the RPN network;

[0076] Step 3: Use CNN to further extract features from the feature map and combine the candidate boxes to obtain many RoI areas;

[0077] Step 4: Use RoI Pooling to extract features in each RoI area, then connect to the FC layer to predict the category of the object in the box and do bounding box fine-tuning. Select the RoI with the highest score, use the RoI Align layer to extract and calculate the mask, and then pass it through the PointRend module to get the segmentation mask.

[0078] Step 5: Calculate the loss function through the predicted value and the label value, and use the weighted loss value to update the network weight by transferring the loss value in the reverse direction.

[0079] Step6: Repeat Step2, Step3, Step4, and Step5 until the model converges.

[0080] For further explanation of the image segmentation device, please refer to Figure 3 , Figure 3 An image segmentation device provided by an embodiment of the present invention is characterized by comprising: an acquisition module 301, a determination module 302 and a segmentation module 303;

[0081] The acquisition module 301 is used to acquire multiple ROI regions of the image to be segmented, and calculate first scores of the multiple ROI regions;

[0082] The determination module 302 is used to determine that the ROI area is a high-frequency area when the first score meets a preset condition;

[0083] The segmentation module 303 is used to complete the segmentation of the image to be segmented after sampling the high-frequency area according to the first up-sampling method.

[0084] An embodiment of the present invention provides a mobile terminal, including a processor and a memory, wherein the memory stores a computer-readable program code, and when the processor executes the computer-readable program code, the steps of the above-mentioned image segmentation method are implemented.

[0085] An embodiment of the present invention provides a storage medium, wherein the storage medium stores computer-readable program codes, and when the computer-readable program codes are executed, the steps of the above-mentioned image segmentation method are implemented.

[0086] In the embodiment of the present invention, the acquisition module 301 first acquires multiple ROI regions of the image to be segmented, and calculates the first scores of the multiple ROI regions; then the determination module 302 determines that the ROI region is a high-frequency region when the first score meets the preset condition; finally, the segmentation module 303 samples the high-frequency region according to the first upsampling method to complete the segmentation of the image to be segmented.

[0087] The embodiment of the present invention calculates the first scores of multiple ROI regions of the image to be segmented, determines and obtains the high-frequency region from the multiple ROI regions by the first scores, and finally samples the high-frequency region according to the first upsampling method to complete the segmentation of the image to be segmented. The embodiment of the present invention can increase the sampling frequency of the high-frequency region, avoid the problem that the object boundary of the segmentation result is too smooth and unreal, and avoid the problem that the contour of the segmentation result is blurred due to oversampling of the low-frequency region, thereby effectively improving the accuracy of image segmentation.

[0088] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An image segmentation method, characterized in that: include: Acquire multiple ROI regions of the image to be segmented, and calculate first scores of the multiple ROI regions; When the first score meets a preset condition, determining that the ROI area is a high-frequency area; After sampling the high-frequency area according to the first up-sampling method, segmenting the image to be segmented is completed; The step of sampling the high-frequency region according to the first upsampling method to complete the segmentation of the image to be segmented is specifically as follows: Calculating a first mask of the image to be segmented according to the high-frequency area; Enlarging the first mask to a first size according to the first upsampling method to complete segmentation of the image to be segmented; wherein the first size is the size of the image to be segmented; When the resolution of the first mask is smaller than the resolution of the image to be segmented, the method further includes: performing upsampling processing on the first mask according to the first upsampling method to obtain a coarse prediction result; According to the rough prediction result, a high-frequency prediction point is obtained; wherein the classification score of the high-frequency prediction point belongs to a first preset interval; Obtaining a characterization vector of the high-frequency prediction point, performing a new round of prediction according to the characterization vector, and updating the resolution of the first mask according to the new prediction result until the resolution of the first mask is equal to the resolution of the image to be segmented; The calculating of the first scores of the plurality of ROI regions is specifically: After calculating classification scores of the plurality of ROI regions, using the classification scores as the first scores; When the first score meets a preset condition, specifically: When the first score belongs to a first preset interval, it is determined that the first score meets the preset condition.

2. An image segmentation method according to claim 1, characterized in that: The step of obtaining multiple ROI regions of the image to be segmented is specifically as follows: Inputting the image to be segmented into a network model so that the network model extracts image features of the image to be segmented; Inputting the image features into a candidate region generation network so that the candidate region generation network obtains a candidate frame according to the image features; After performing a feature extraction operation on the image features, a first operation result is obtained, and a plurality of ROI regions are generated and acquired by combining the first operation result and the candidate box.

3. An image segmentation method according to claim 2, characterized in that: Also includes: The border regression score of the high-frequency region is calculated, and the borders of the plurality of high-frequency regions are adjusted in combination with the classification score.

4. An image segmentation device, characterized in that: include: Acquisition module, determination module and segmentation module; The acquisition module is used to acquire multiple ROI regions of the image to be segmented, and calculate first scores of the multiple ROI regions; The determination module is used for determining that the ROI area is a high-frequency area when the first score meets a preset condition; The segmentation module is used to complete the segmentation of the image to be segmented after sampling the high-frequency area according to the first up-sampling method; The segmentation module samples the high-frequency area according to the first upsampling method to complete the segmentation of the image to be segmented, specifically: The segmentation module calculates a first mask of the image to be segmented according to the high-frequency area; Enlarging the first mask to a first size according to the first upsampling method to complete segmentation of the image to be segmented; wherein the first size is the size of the image to be segmented; When the resolution of the first mask is smaller than the resolution of the image to be segmented, the method further includes: performing upsampling processing on the first mask according to the first upsampling method to obtain a coarse prediction result; According to the rough prediction result, a high-frequency prediction point is obtained; wherein the classification score of the high-frequency prediction point belongs to a first preset interval; Obtaining a characterization vector of the high-frequency prediction point, performing a new round of prediction according to the characterization vector, and updating the resolution of the first mask according to the new prediction result until the resolution of the first mask is equal to the resolution of the image to be segmented; The acquisition module calculates the first scores of the plurality of ROI regions, specifically: After the acquisition module calculates the classification scores of the plurality of ROI regions, the classification scores are used as the first scores; When the first score meets a preset condition, specifically: When the first score belongs to a first preset interval, the determination module determines that the first score meets the preset condition.

5. A mobile terminal, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer-readable program code, and when the processor executes the computer-readable program code, the steps of an image segmentation method according to any one of claims 1 to 3 are implemented.

6. A storage medium, characterized in that: The storage medium stores computer-readable program codes, and when the computer-readable program codes are executed, the steps of an image segmentation method according to any one of claims 1 to 3 are implemented.

Images