Training method of instance segmentation model, instance segmentation method, device and medium

Abstract

The present application discloses a training method, an instance segmentation method, a device and a medium for an instance segmentation model. The training method introduces dynamic region of interest feature alignment (DRA) in the training of a mask recognition network. The DRA dynamically divides the candidate region into length and width according to the aspect ratio of the bounding box, and extracts features of the first candidate region based on the dynamic length and dynamic width, so as to achieve uniform sampling of the first candidate region in the horizontal and vertical directions, and avoid undersampling on the longer side of the first candidate region. At the same time, the mask data extracted based on the dynamic length and dynamic width has the characteristic of proportional scaling, can better represent the topological structure of the slender target, and effectively avoid the distortion, twisting and distortion of the slender target, so that the segmentation result of the slender target is more accurate, and the application scope of the instance segmentation model is improved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a training method for an instance segmentation model, an instance segmentation method, a device, and a medium. Background Art

[0002] Instance segmentation is a basic visual recognition task in the computer field. This task includes object detection and semantic segmentation. It locates and identifies different objects in the image and assigns a separate pixel-level mask to each object. Among them, the top-down two-stage instance segmentation method based on object detection represented by Mask RCNN has good accuracy and scalability. This type of method generally uses the detector to output candidate regions and then predicts instance masks for each candidate region.

[0003] In the process of predicting instance masks, the feature map of the feature pyramid network (FPN) is used to extract candidate region features by aligning the region of interest (RoI Align) and unify candidate regions of different sizes into a fixed size of M×M. However, the feature extraction method based on the fixed-size candidate region uses the same interval sampling strategy in the horizontal and vertical directions, which makes it difficult to extract effective candidate region features for slender targets; and this type of method uses a small mask (TinyMask, TM) of a fixed size of K×2M×2M to represent the topological structure of each instance, which easily causes the geometric features of slender targets to be distorted. It can be seen that the current instance segmentation method cannot be effectively applied to instance segmentation of slender targets. Summary of the invention

[0004] The present application provides a training method for an instance segmentation model, an instance segmentation method, a device and a medium, aiming to solve the problem that the existing instance segmentation method has a small scope of application.

[0005] In order to solve the above technical problems, in a first aspect, an embodiment of the present application provides a training method for an instance segmentation model, comprising:

[0006] Obtaining annotation data and a target feature map of the image sample, the annotation data including bounding box annotation data and mask annotation data, the bounding box annotation data including bounding box aspect ratio annotation data, and the target feature map corresponding to the first candidate region;

[0007] Based on the labeled data and the target feature map, the bounding box recognition network and the mask recognition network in the preset slender target instance segmentation model are iteratively trained until the slender target instance segmentation model reaches the preset convergence condition, and the training is stopped to obtain the final training result of the instance segmentation model;

[0008] Each time the mask recognition network is trained, the following steps are included:

[0009] Determine the dynamic length and dynamic width of the first candidate region on the target feature map based on the bounding box aspect ratio annotation data;

[0010] Using a mask recognition network, extracting features of the first candidate region according to the dynamic length and the dynamic width, and obtaining mask data of the first candidate region;

[0011] Based on a first loss function between the mask data and the mask annotation data, network parameters of the mask recognition network are updated.

[0012] In this embodiment, dynamic region of interest alignment (DRA) is introduced in the training of the mask recognition network. DRA dynamically divides the candidate region into length and width according to the aspect ratio of the bounding box, and extracts features of the first candidate region according to the dynamic length and dynamic width, so as to achieve uniform sampling of the first candidate region in the horizontal and vertical directions, and avoid undersampling of the longer side of the first candidate region. At the same time, the mask data extracted based on the dynamic length and dynamic width has the characteristic of proportional scaling, can better represent the topological structure of the slender target, and effectively avoid the distortion, twisting and distortion of the slender target, so that the segmentation result of the slender target is more accurate, and the application scope of the instance segmentation model is improved.

[0013] In one embodiment, determining the dynamic length and dynamic width of the first candidate region on the target feature map based on the bounding box aspect ratio annotation data includes:

[0014] Based on the preset hyperparameters and bounding box aspect ratio annotation data, the dynamic length and dynamic width of the first candidate region on the target feature map are determined. The preset hyperparameters are used to set the total number of pixels of the candidate region features.

[0015] In one embodiment, a mask recognition network is used to extract features of the first candidate region according to the dynamic length and the dynamic width to obtain mask data of the first candidate region, including:

[0016] Extracting features of the first candidate region according to the dynamic length and the dynamic width to obtain features of the first candidate region;

[0017] Using a preset two-layer graph convolutional neural network, the graph convolution features of the first candidate region are extracted according to the features of the first candidate region to obtain mask data of the first candidate region.

[0018] Further, using a preset two-layer graph convolutional neural network, according to the first candidate region feature, the graph convolution feature of the first candidate region is extracted to obtain mask data of the first candidate region, including:

[0019] Downsampling the first candidate region feature to obtain a downsampled first candidate region feature;

[0020] Using a two-layer graph convolutional neural network, the first candidate region features after downsampling are linearly transformed to obtain a similarity adjacency matrix;

[0021] Based on the similarity adjacency matrix, the graph convolution features of the first candidate region are extracted to obtain the mask data of the first candidate region.

[0022] In one embodiment, each time the bounding box recognition network is trained, the following steps are included:

[0023] Using a bounding box recognition network, extracting bounding box features of the first candidate area on the target feature map;

[0024] Based on the bounding box features, identify the bounding box category, bounding box coordinates, and bounding box aspect ratio of the first candidate area, and obtain bounding box data of the first candidate area;

[0025] Based on a second loss function between the bounding box data and the bounding box annotation data, network parameters of the bounding box recognition network are updated.

[0026] Optionally, the second loss function is:

[0027]

[0028] Among them, L cls is the bounding box regression loss function of the Fast RCNN network, L reg is the bounding box positioning loss function of the Fast RCNN network, L r is the smooth L1 loss function, p i is the bounding box category, is the bounding box category label data in the bounding box label data, t i are the bounding box coordinates, is the bounding box coordinate annotation data in the bounding box annotation data, r i is the bounding box aspect ratio, N is the bounding box aspect ratio annotation data, cls is the training batch, N reg is the number of candidate regions, and α and β are balance parameters.

[0029] In one embodiment, the preset convergence condition is that the number of iterations of the slender object instance segmentation model reaches a preset number or the training loss value is less than a preset value.

[0030] In one embodiment, obtaining the annotation data and the target feature map of the image sample includes:

[0031] Obtain image samples and annotation data of the image samples;

[0032] Using a preset backbone network, extracting a first image feature of the image sample;

[0033] Using a preset feature pyramid network, output a plurality of target feature maps with different resolutions according to the first image feature;

[0034] Using the preset candidate region network, predict the first candidate region of the target feature map.

[0035] In a second aspect, an embodiment of the present application provides an instance segmentation method, including:

[0036] Obtaining an instance feature map of the image to be segmented, where the instance feature map corresponds to a second candidate region;

[0037] Based on a preset instance segmentation model, instance segmentation is performed on the second candidate area on the instance feature map to obtain bounding box instance data and mask instance data of the second candidate area. The instance segmentation model is trained based on the training method of the first aspect.

[0038] In one embodiment, obtaining an instance feature map of an image to be segmented includes:

[0039] Obtain the image to be segmented;

[0040] Using a preset backbone network, extracting a second image feature of the image to be segmented;

[0041] Using a preset feature pyramid network, output a plurality of instance feature maps of different resolutions according to the second image features;

[0042] Using the preset candidate region network, predict the second candidate region of the instance feature map.

[0043] In one embodiment, based on a preset instance segmentation model, instance segmentation is performed on the second candidate region on the instance feature map to obtain bounding box instance data and mask instance data of the second candidate region, including:

[0044] Using the bounding box recognition network of the instance segmentation model, extracting bounding box features of the second candidate region on the instance feature map;

[0045] Based on the bounding box features, identify the bounding box category, bounding box coordinates, and bounding box aspect ratio of the second candidate area, and output bounding box instance data of the second candidate area;

[0046] Determine the dynamic length and dynamic width of the second candidate region based on the aspect ratio of the bounding box identified by the bounding box recognition network;

[0047] The mask recognition network of the instance segmentation model is used to extract features of the second candidate region according to the dynamic length and the dynamic width, and the mask instance data of the second candidate region is output.

[0048] Furthermore, the mask recognition network of the instance segmentation model is used to extract features of the second candidate region according to the dynamic length and the dynamic width, and the mask instance data of the second candidate region is output, including:

[0049] Extracting features of the second candidate region according to the dynamic length and the dynamic width to obtain features of the second candidate region;

[0050] Using a preset two-layer graph convolutional neural network, the graph convolution features of the second candidate region are extracted according to the features of the second candidate region, and the mask instance data of the second candidate region is output.

[0051] Optionally, using a preset two-layer graph convolutional neural network, according to the second candidate region feature, extracting the graph convolution feature of the second candidate region, and outputting the mask instance data of the second candidate region, including:

[0052] Downsampling the second candidate region feature to obtain a downsampled second candidate region feature;

[0053] Using a two-layer graph convolutional neural network, the downsampled second candidate region features are linearly transformed to obtain a similarity adjacency matrix;

[0054] Based on the similarity adjacency matrix, the graph convolution features of the second candidate region are extracted to obtain the mask instance data of the second candidate region.

[0055] In a third aspect, an embodiment of the present application provides a training device for an instance segmentation model, comprising:

[0056] A first acquisition module is used to acquire annotation data and a target feature map of an image sample, wherein the annotation data includes bounding box annotation data and mask annotation data, wherein the bounding box annotation data includes bounding box aspect ratio annotation data, and the target feature map corresponds to a first candidate region;

[0057] A training module is used to iteratively train the bounding box recognition network and the mask recognition network in the preset slender target instance segmentation model based on the labeled data and the target feature map until the slender target instance segmentation model reaches a preset convergence condition, stop the training, and obtain the final training result of the instance segmentation model;

[0058] The training modules include:

[0059] A determination submodule, used to determine the dynamic length and dynamic width of the first candidate region on the target feature map based on the bounding box aspect ratio annotation data;

[0060] An extraction submodule, used for a mask recognition network, performs feature extraction on the first candidate region according to a dynamic length and a dynamic width, and obtains mask data of the first candidate region;

[0061] The updating submodule is used to update the network parameters of the mask recognition network based on the first loss function between the mask data and the mask annotation data.

[0062] In a fourth aspect, an embodiment of the present application provides an instance segmentation device, including:

[0063] A second acquisition module is used to acquire an instance feature map of the image to be segmented, where the instance feature map corresponds to a second candidate region;

[0064] A segmentation module is used to perform instance segmentation on the second candidate area on the instance feature map based on a preset instance segmentation model to obtain bounding box instance data and mask instance data of the second candidate area. The instance segmentation model is trained based on the training method of the first aspect mentioned above.

[0065] In a fifth aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program so that the computer device executes the training method of the first aspect above, or the instance segmentation method of the second aspect above.

[0066] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the training method of the first aspect or the instance segmentation method of the second aspect.

[0067] It should be noted that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first or second aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A schematic diagram of a flow chart of a training method for an instance segmentation model provided in an embodiment of the present application;

[0069] Figure 2 A schematic diagram of a flow chart of a training process of a mask recognition network provided in an embodiment of the present application;

[0070] Figure 3 A schematic diagram of the DRA process of the graph convolutional neural network provided in an embodiment of the present application;

[0071] Figure 4A schematic diagram of an example segmentation of a slender target provided in an embodiment of the present application;

[0072] Figure 5 A schematic diagram of a flow chart of an instance segmentation method provided in an embodiment of the present application;

[0073] Figure 6 A schematic diagram of the structure of a training device for an instance segmentation model provided in an embodiment of the present application;

[0074] Figure 7 A schematic diagram of the structure of an instance segmentation device provided in an embodiment of the present application;

[0075] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] As described in the related background technology, in the process of predicting instance masks, the feature map output by the feature pyramid network (FPN) is output by the region of interest alignment (RoI Align) method to extract candidate region features, and the candidate regions of different sizes are unified into a fixed size of M×M. However, the feature extraction method based on the fixed-size candidate region uses the same interval sampling strategy in the horizontal and vertical directions, which makes it difficult to extract effective candidate region features for slender targets; and this type of method uses a small mask (TM) with a fixed size of K×2M×2M to represent the topological structure of each instance, which easily causes the geometric features of slender targets to be distorted. It can be seen that the current instance segmentation method cannot be effectively applied to the instance segmentation of slender targets.

[0078] To this end, the embodiments of the present application provide a training method, an instance segmentation method and an apparatus for an instance segmentation model. By introducing dynamic region of interest alignment (DRA) in a mask recognition network, DRA dynamically divides the candidate region into length and width according to the aspect ratio of the bounding box, and extracts features of the first candidate region based on the dynamic length and dynamic width, so that the first candidate region is uniformly sampled in the horizontal and vertical directions, and undersampling of the longer side of the first candidate region is avoided. At the same time, the mask data extracted based on the dynamic length and dynamic width has the characteristic of proportional scaling, can better represent the topological structure of the slender target, and effectively avoid distortion, distortion and distortion of the slender target, so that the segmentation result of the slender target is more accurate, and the scope of application of the instance segmentation model is improved.

[0079] Reference Figure 1 , Figure 1 The following flowchart shows a training method for an instance segmentation model provided in an embodiment of the present application. The training method for the instance segmentation model described below in the embodiment of the present application can be applied to computer devices, including but not limited to computer devices such as smart phones, tablet computers, desktop computers, supercomputers, personal digital assistants, physical servers, and cloud servers. The training method for the instance segmentation model in the embodiment of the present application includes steps S101 to S102, which are described in detail as follows:

[0080] Step S101, obtaining annotation data and a target feature map of an image sample, wherein the annotation data includes bounding box annotation data and mask annotation data, wherein the bounding box annotation data includes bounding box aspect ratio annotation data, and a first candidate region corresponds to the target feature map.

[0081] In this step, the image sample is an image sample in which a target instance (person or object) exists, and the bounding box annotation data is the annotation data corresponding to the bounding box used to annotate the target instance in the image sample, which includes but is not limited to the bounding box category (such as AABB and OBB, etc.), the bounding box coordinates (the coordinates of each end point of the line segment constituting the bounding box) and the bounding box aspect ratio (the ratio between the length and width of the bounding box). The mask annotation data is the annotation data corresponding to the mask used to annotate the target instance in the image sample, which includes but is not limited to the mask topology. The target feature map is a feature map with a resolution different from that of the image sample, which is used to predict the bounding box and mask of the target instance during model training. The first candidate region is an area on the target feature map where a target instance may exist. It should be understood that there may be multiple target instances on the target feature map, so there may be multiple first candidate regions corresponding.

[0082] Optionally, the target feature map extraction process may include: using a preset backbone network to extract a first image feature of the image sample; using a preset feature pyramid network to output a plurality of target feature maps of different resolutions based on the first image feature; and using a preset candidate region network to predict a first candidate region of the target feature map.

[0083] In this embodiment, the backbone network may be ResNet for feature extraction, wherein ResNet may also be ResNet-50, ResNet-101, ResNeXt-50 or ResNeXt-101. The feature pyramid network may be FeaturePyramid Network (FPN), which is used to output image samples into multiple target feature maps of different resolutions according to image features. The candidate region network predicts candidate regions on the target feature map output by FPN.

[0084] It should be noted that the target feature map extraction process can be performed on the computer device that implements the training method, or after being performed on other computer devices, the extracted target feature map can be transplanted to the computer device. In other words, the computer device that performs the target feature map extraction process can be the same as the computer device that implements the training method, or it can be different.

[0085] Step S102, based on the labeled data and the target feature map, iteratively train the bounding box recognition network and the mask recognition network in the preset slender target instance segmentation model until the slender target instance segmentation model reaches a preset convergence condition, stop training, and obtain an instance segmentation model.

[0086] In this step, the bounding box recognition network (RCNN Head) performs bounding box recognition on the target feature map based on the candidate regions output by the candidate region network, and the mask recognition network (Mask Head) is used to perform mask recognition of the target instance on the target feature map. The loss function of the elongated target instance segmentation model is the sum of the second loss function of the bounding box recognition network and the first loss function of the mask recognition network.

[0087] Optionally, the preset convergence condition is that the number of iterations of the slender object instance segmentation model reaches a preset number.

[0088] Optionally, the preset convergence condition is that the loss function of the slender target instance segmentation model is less than a preset value. In this embodiment, the training loss value of the model is used as the basis for determining the convergence condition, so as to adopt an early stopping training strategy, that is, if the number of iterations does not reach the preset number, but the training loss value of the model is less than the preset value, the training is stopped, which can effectively avoid overfitting of the model, thereby ensuring that the model has good generalization performance.

[0089] In the above step S102, each time the mask recognition network is trained, steps S201 to S203 are included:

[0090] Step S201, determining a dynamic length and a dynamic width of a first candidate region on the target feature map based on the bounding box aspect ratio annotation data.

[0091] In this step, the bounding box aspect ratio annotation data is the aspect ratio annotation data in the real bounding box annotation data assigned to the candidate area. The dynamic length is the area length corresponding to different first candidate areas, and the dynamic width is the area width corresponding to different first candidate areas. That is to say, for the feature extraction process of different first candidate areas, different processes use different lengths and widths. This embodiment uses the real aspect ratio corresponding to the candidate area based on DRA to annotate the bounding box aspect ratio annotation data, so that the aspect ratio of candidate areas of different shapes can be used to extract features of corresponding shapes of candidate areas, and the problem of feature distortion of candidate areas caused by fixed sizes can be effectively avoided, thereby improving the accuracy of feature extraction of candidate areas.

[0092] It should be noted that the present embodiment uses bounding box aspect ratio annotation data as the basis for dynamic feature extraction of the feature size of the candidate region in the mask recognition process, rather than using the bounding box aspect ratio output by the bounding box recognition network as the basis for dynamic feature extraction. This can avoid the situation where the bounding box aspect ratio output by the bounding box recognition network in the early stage of training is significantly different from the actual aspect ratio, which leads to a decrease in the training effect of the mask recognition process. It can be seen that the present embodiment uses the bounding box aspect ratio annotation data as the basis for dynamic segmentation, which can improve the recognition performance of the mask recognition network.

[0093] Step S202: Using the mask recognition network, extract features of the first candidate region according to the dynamic length and the dynamic width to obtain mask data of the first candidate region.

[0094] In this step, the dynamic length and dynamic width of the first candidate region are obtained according to the bounding box aspect ratio annotation data, and the first candidate region is subjected to DRA according to the dynamic length and dynamic width to extract the first candidate region features, and then the mask data of the first candidate region is output by performing a graph convolution operation on the first candidate region features. In this way, the sampling in the horizontal and vertical directions of the feature extraction process is uniform and consistent, thereby realizing the mask recognition of slender targets.

[0095] Step S203: updating network parameters of the mask recognition network based on a first loss function between the mask data and the mask annotation data.

[0096] In this step, the mask data includes a mask representation vector of the first candidate region, and soft Gdice is used to calculate the training loss generated by the mask annotation data matching the first candidate region to obtain a first loss function Mask Loss (Lm).

[0097] In one embodiment, Figure 1 Based on the illustrated embodiment, the above step S201 specifically includes: determining the dynamic length and dynamic width of the first candidate area on the target feature map based on preset hyperparameters and the bounding box aspect ratio annotation data, and the preset hyperparameters are used to set the total number of pixels of the candidate area features.

[0098] In this embodiment, according to the bounding box aspect ratio label data r corresponding to the candidate region and the preset hyperparameter M, the candidate region features are dynamically divided into width and width. Abbreviated as B, length Referred to as H, it realizes uniform sampling of candidate region features in the horizontal and vertical directions.

[0099] In one embodiment, Figure 1 Based on the illustrated embodiment, the above step S202 specifically includes:

[0100] Extracting features of the first candidate region according to the dynamic length and the dynamic width to obtain features of the first candidate region;

[0101] Using a preset two-layer graph convolutional neural network, the graph convolution features of the first candidate region are extracted according to the features of the first candidate region to obtain mask data of the first candidate region.

[0102] In this embodiment, the first candidate region feature X∈R is extracted based on DRA. B×H×c , where B represents the dynamic width of the first candidate region feature, H represents the dynamic length of the first candidate region feature, c represents the number of channels of the first candidate region feature, and each pixel point on the first candidate region feature is used as a node of GCN, and a double-layer GCN is used to obtain the mutual dependence between the feature pixels of the first candidate region.

[0103] Since slender objects have different aspect ratios, DRA that maintains the aspect ratio will extract candidate region features of different shapes, resulting in the problem that the model cannot be trained in parallel, resulting in a long training time. To address the problem of difficulty in parallel training caused by candidate region features of different shapes, this embodiment uses a two-layer graph convolutional neural network (GCN) to obtain the mutual dependence between the candidate region feature pixels, which can achieve parallel training.

[0104] Alternatively, if Figure 3 Schematic diagram of the DRA process flow of the graph convolutional neural network. Figure 3 , using a preset two-layer graph convolutional neural network, the process of extracting the graph convolution features of the first candidate region includes:

[0105] Downsampling the first candidate region feature to obtain a downsampled first candidate region feature;

[0106] Using the double-layer graph convolutional neural network, linearly transform the downsampled first candidate region features to obtain a similarity adjacency matrix;

[0107] Based on the similarity adjacency matrix, graph convolution features of the first candidate region are extracted to obtain mask data of the first candidate region.

[0108] In this embodiment, in order to reduce the amount of calculation for constructing the similarity adjacency matrix of the first candidate region, the input first candidate region feature X∈R is converted into B×H×c Mapped to feature R S ∈R B×H×c / 2 , R S =R c (X). Where R c It is a 1×1 convolution, which is used to reduce the number of channels of the input first candidate feature X to half of the original number.

[0109] Then in feature V c Two learnable linear transformations (ζ, φ) are used to construct the similarity adjacency matrix A∈R between the first candidate region features BH×BH , A=f(ξ(R S ),φ(R S ) T ). Where f is the dot product operation and T represents the matrix transpose operation.

[0110] In order to further obtain the interdependence between the features of candidate regions of different shapes, a learnable linear transformation η is used on the first candidate region feature X, and then a two-layer 3×3 convolution is used to represent the weight matrix W∈R of GCN BH ×BH×D , thereby generating the candidate region graph convolution features Gp, G p =Aη(X)W. The final output is a two-dimensional matrix of 2B×2H for representing the mask data.

[0111] In one embodiment, Figure 1 Based on the embodiment shown, in the above step S102, each time the bounding box recognition network is trained, the following steps are included:

[0112] Using the bounding box recognition network, extracting bounding box features of the first candidate area on the target feature map;

[0113] Based on the bounding box features, identifying the bounding box category, bounding box coordinates, and bounding box aspect ratio of the first candidate area, and obtaining bounding box data of the first candidate area;

[0114] Based on a second loss function between the bounding box data and the bounding box annotation data, network parameters of the bounding box recognition network are updated.

[0115] In this embodiment, compared with the traditional RCNN Head, this embodiment adds the recognition of the aspect ratio of the bounding box, so that when the model is applied, the Mask Head can use the aspect ratio of the bounding box output by the RCNN Head to dynamically align the features of interest in the candidate area, thereby realizing feature extraction of slender targets when the model is applied.

[0116] Optionally, the second loss function is:

[0117]

[0118] Among them, L cls is the bounding box regression loss function of the Fast RCNN network, L reg is the bounding box positioning loss function of the Fast RCNN network, L r is the smooth L1 loss function, p i is the bounding box category, is the bounding box category labeling data in the bounding box labeling data, t i are the bounding box coordinates, is the bounding box coordinate annotation data in the bounding box annotation data, r i is the aspect ratio of the bounding box, N is the aspect ratio label data of the bounding box, cls is the training batch, N reg is the number of candidate regions, and α and β are balance parameters.

[0119] Reference Figure 4 , Figure 4The implementation flow chart of an instance segmentation method provided by an embodiment of the present application is shown. The instance segmentation method described below in the embodiment of the present application can be applied to computer devices, including but not limited to computer devices such as smart phones, tablet computers, desktop computers, supercomputers, personal digital assistants, physical servers, and cloud servers. It is understandable that the computer device implementing the instance segmentation method can be the same as or different from the computer device implementing the above-mentioned training method. The instance segmentation method of the embodiment of the present application includes steps S401 to S402, which are described in detail as follows:

[0120] Step S401, obtaining an instance feature map of the image to be segmented, wherein the instance feature map corresponds to a second candidate region.

[0121] In this step, the instance feature map can be extracted from the image to be segmented on the computer device implementing the instance segmentation method, or it can be extracted on other computer devices and then transplanted to this computer device. The computer device used in the process of extracting the instance feature map can be the same as or different from the computer device implementing the instance segmentation method.

[0122] Optionally, the instance feature map extraction process includes: obtaining the image to be segmented; using a preset backbone network to extract a second image feature of the image to be segmented; using a preset feature pyramid network to output a plurality of instance feature maps of different resolutions according to the second image feature; and using a preset candidate region network to predict a second candidate region of the instance feature map.

[0123] It can be understood that the process of extracting the instance feature map is similar to the process of extracting the target feature map mentioned above. Please refer to the relevant description of step S101 above, which will not be repeated here.

[0124] Step S402, based on a preset instance segmentation model, performing instance segmentation on the second candidate region on the instance feature map to obtain bounding box instance data and mask instance data of the second candidate region. The instance segmentation model is based on the above Figure 1 to Figure 2 The training is obtained by the training method of the illustrated embodiment.

[0125] In this step, based on Figure 1 to Figure 2 The instance segmentation model obtained in the illustrated embodiment performs instance segmentation, which can realize instance segmentation of slender objects and improve the application scope of the instance segmentation method.

[0126] In one embodiment, Figure 4 Based on the illustrated embodiment, the above step S402 includes:

[0127] Extracting bounding box features of a second candidate region on the instance feature map using a bounding box recognition network of the instance segmentation model;

[0128] Based on the bounding box features, identifying the bounding box category, bounding box coordinates, and bounding box aspect ratio of the second candidate area, and outputting bounding box instance data of the second candidate area;

[0129] Determining a dynamic length and a dynamic width of the second candidate region based on the aspect ratio of the bounding box identified by the bounding box recognition network;

[0130] Using the mask recognition network of the instance segmentation model, feature extraction is performed on the second candidate region according to the dynamic length and the dynamic width, and mask instance data of the second candidate region is output.

[0131] In this embodiment, if Figure 5 The example segmentation diagram of the slender target is shown. In this embodiment, RCNN Head is used to extract bounding box features by Roi Align, and the bounding box category (class), bounding box coordinates (bbox) and bounding box aspect ratio (ratio) are identified; Mask Head is used to output a one-dimensional vector of size 4BH×c by Dynamic Roi Align, and according to the bounding box category of the bounding box output by RCNN Head, the one-dimensional vector of size 4BH×c is transformed into a one-dimensional vector of size 4BH×1, and then the one-dimensional vector of size 4BH×1 is reshaped into a two-dimensional matrix of 2B×2H by using the bounding box aspect ratio. Finally, RCNN Head outputs the bounding box coordinates, categories and aspect ratios of all target instances of the image to be segmented, and Mask Head outputs a two-dimensional matrix of 2B×2H for the bounding box of each target instance to represent the mask instance data of each instance.

[0132] Further, the mask recognition network using the instance segmentation model performs feature extraction on the second candidate region according to the dynamic length and the dynamic width, and outputs mask instance data of the second candidate region, including:

[0133] Extracting features of the second candidate region according to the dynamic length and the dynamic width to obtain features of the second candidate region;

[0134] Utilizing a preset two-layer graph convolutional neural network, based on the features of the second candidate region, the graph convolutional features of the second candidate region are extracted, and mask instance data of the second candidate region is output.

[0135] The recognition process of the mask instance data in this embodiment is similar to the recognition process of the mask data in the above-mentioned training method embodiment. Please refer to the above and will not be repeated here.

[0136] Optionally, the using a preset two-layer graph convolutional neural network to extract the graph convolutional features of the second candidate region according to the features of the second candidate region, and outputting the mask instance data of the second candidate region includes:

[0137] Downsampling the second candidate region feature to obtain a downsampled second candidate region feature;

[0138] Using the double-layer graph convolutional neural network, linearly transform the downsampled second candidate region features to obtain a similarity adjacency matrix;

[0139] Based on the similarity adjacency matrix, graph convolution features of the second candidate region are extracted to obtain mask instance data of the second candidate region.

[0140] The process of extracting graph convolution features in this embodiment is similar to the graph convolution feature extraction process in the above-mentioned training method embodiment. Please refer to the above and will not be repeated here.

[0141] In order to execute the training method of the instance segmentation model corresponding to the above method embodiment to achieve the corresponding functions and technical effects, see Figure 6 , Figure 6 The structural block diagram of a training device for an instance segmentation model provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the present embodiment are shown. The training device provided in an embodiment of the present application includes:

[0142] A first acquisition module 601 is used to acquire annotation data and a target feature map of an image sample, wherein the annotation data includes bounding box annotation data and mask annotation data, wherein the bounding box annotation data includes bounding box aspect ratio annotation data, and the target feature map corresponds to a first candidate region;

[0143] A training module 602 is used to iteratively train a bounding box recognition network and a mask recognition network in a preset elongated target instance segmentation model based on the labeled data and the target feature map until the elongated target instance segmentation model reaches a preset convergence condition, stop training, and obtain an instance segmentation model;

[0144] The training module 602 includes:

[0145] A first determination submodule 6021 is used to determine a dynamic length and a dynamic width of a first candidate region on the target feature map based on the bounding box aspect ratio annotation data;

[0146] A first extraction submodule 6022 is used to use the mask recognition network to perform feature extraction on the first candidate region according to the dynamic length and the dynamic width to obtain mask data of the first candidate region;

[0147] The first updating submodule 6023 is used to update the network parameters of the mask recognition network based on the first loss function between the mask data and the mask annotation data.

[0148] In one embodiment, the determination submodule 6021 includes:

[0149] A determination unit is used to determine the dynamic length and dynamic width of the first candidate area on the target feature map based on preset hyperparameters and the bounding box aspect ratio annotation data, wherein the preset hyperparameters are used to set the total number of pixels of the candidate area features.

[0150] In one embodiment, the extraction submodule 6022 includes:

[0151] A first extraction unit, configured to extract features of the first candidate region according to the dynamic length and the dynamic width, to obtain features of the first candidate region;

[0152] The second extraction unit is used to use a preset two-layer graph convolutional neural network to extract the graph convolution features of the first candidate region according to the features of the first candidate region, so as to obtain the mask data of the first candidate region.

[0153] Furthermore, the second extraction unit comprises:

[0154] A first downsampling subunit is used to downsample the first candidate region feature to obtain a downsampled first candidate region feature;

[0155] A first transformation subunit is used to use the double-layer graph convolutional neural network to perform a linear transformation on the downsampled first candidate region features to obtain a similarity adjacency matrix;

[0156] The first extraction subunit is used to extract the graph convolution features of the first candidate region based on the similarity adjacency matrix to obtain mask data of the first candidate region.

[0157] In one embodiment, the training module 602 further includes:

[0158] A second extraction submodule, configured to extract bounding box features of a first candidate area on the target feature map by using the bounding box recognition network;

[0159] A first recognition submodule is used to identify the bounding box category, bounding box coordinates and bounding box aspect ratio of the first candidate area based on the bounding box features, and obtain bounding box data of the first candidate area;

[0160] The second updating submodule is used to update the network parameters of the bounding box recognition network based on a second loss function between the bounding box data and the bounding box annotation data.

[0161] Optionally, the second loss function is:

[0162]

[0163] Among them, L cls is the bounding box regression loss function of the Fast RCNN network, L reg is the bounding box positioning loss function of the Fast RCNN network, L r is the smooth L1 loss function, p i is the bounding box category, is the bounding box category labeling data in the bounding box labeling data, t i are the bounding box coordinates, is the bounding box coordinate annotation data in the bounding box annotation data, r i is the aspect ratio of the bounding box, N is the aspect ratio label data of the bounding box, cls is the training batch, N reg is the number of candidate regions, and α and β are balance parameters.

[0164] In one embodiment, the convergence condition is that the number of iterations of the slender object instance segmentation model reaches a preset number or the training loss value is less than a preset value.

[0165] In one embodiment, the first acquisition module 601 includes:

[0166] A first acquisition submodule, used to acquire the image sample and the annotation data of the image sample;

[0167] A third extraction submodule, used to extract a first image feature of the image sample using a preset backbone network;

[0168] A first output submodule, configured to output a plurality of target feature maps of different resolutions according to the first image features by using a preset feature pyramid network;

[0169] The first prediction submodule is used to predict the first candidate region of the target feature map by using a preset candidate region network.

[0170] The above training device can implement the training method of the above method embodiment. The options in the above method embodiment are also applicable to this embodiment and will not be described in detail here. The rest of the contents of the embodiment of this application can refer to the contents of the above method embodiment and will not be described in detail in this embodiment.

[0171] In order to perform the instance segmentation method corresponding to the above method embodiment to achieve the corresponding functions and technical effects, see Figure 7 , Figure 7 The structural block diagram of an instance segmentation device provided by an embodiment of the present application is shown. For the convenience of explanation, only the parts related to the present embodiment are shown. The instance segmentation device provided by the embodiment of the present application includes:

[0172] A second acquisition module 701 is used to acquire an instance feature map of the image to be segmented, where the instance feature map corresponds to a second candidate region;

[0173] The segmentation module 702 is used to perform instance segmentation on the second candidate area on the instance feature map based on a preset instance segmentation model to obtain bounding box instance data and mask instance data of the second candidate area. The instance segmentation model is trained based on the above training method.

[0174] In one embodiment, the second acquisition module 701 includes:

[0175] A second acquisition submodule is used to acquire the image to be segmented;

[0176] A fourth extraction submodule, used to extract the second image feature of the image to be segmented by using a preset backbone network;

[0177] A second output submodule, configured to output a plurality of instance feature maps of different resolutions according to the second image features by using a preset feature pyramid network;

[0178] The second prediction submodule is used to predict the second candidate region of the instance feature map by using a preset candidate region network.

[0179] In one embodiment, the segmentation module 702 includes:

[0180] a fifth extraction submodule, configured to extract bounding box features of a second candidate region on the instance feature map by using a bounding box recognition network of the instance segmentation model;

[0181] A second recognition submodule is used to identify the bounding box category, bounding box coordinates and bounding box aspect ratio of the second candidate area based on the bounding box features, and output bounding box instance data of the second candidate area;

[0182] A second determination submodule, configured to determine a dynamic length and a dynamic width of the second candidate region based on an aspect ratio of the bounding box identified by the bounding box recognition network;

[0183] The sixth extraction submodule is used to use the mask recognition network of the instance segmentation model to perform feature extraction on the second candidate region according to the dynamic length and the dynamic width, and output mask instance data of the second candidate region.

[0184] Furthermore, the sixth extraction submodule includes:

[0185] A third extraction unit, configured to extract features of the second candidate region according to the dynamic length and the dynamic width to obtain features of the second candidate region;

[0186] The fourth extraction unit is used to use a preset two-layer graph convolutional neural network to extract the graph convolution features of the second candidate region according to the features of the second candidate region, and output the mask instance data of the second candidate region.

[0187] Optionally, the fourth extraction unit comprises:

[0188] A second downsampling subunit is used to downsample the second candidate region feature to obtain a downsampled second candidate region feature;

[0189] A second transformation subunit is used to use the double-layer graph convolutional neural network to perform a linear transformation on the downsampled second candidate region features to obtain a similarity adjacency matrix;

[0190] The second extraction subunit is used to extract the graph convolution features of the second candidate region based on the similarity adjacency matrix to obtain the mask instance data of the second candidate region.

[0191] The above-mentioned instance segmentation device can implement the instance segmentation method of the above-mentioned method embodiment. The options in the above-mentioned method embodiment are also applicable to this embodiment and will not be described in detail here. The rest of the contents of the embodiment of this application can refer to the contents of the above-mentioned method embodiment, and will not be repeated in this embodiment.

[0192] Figure 8 This is a schematic diagram of the structure of a computer device provided in one embodiment of the present application. Figure 8 As shown, the computer device 8 of this embodiment includes: at least one processor 80 ( Figure 8 Only one is shown in the figure) a processor, a memory 81, and a computer program 82 stored in the memory 81 and executable on the at least one processor 80, and when the processor 80 executes the computer program 82, the steps in any of the above method embodiments are implemented.

[0193] The computer device 8 may be a host computer, a tablet computer, a desktop computer or other computing device, and the computer device may include but is not limited to a processor 80 and a memory 81. Those skilled in the art will appreciate that Figure 8 It is only an example of the computer device 8 and does not constitute a limitation on the computer device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0194] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0195] In some embodiments, the memory 81 may be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 81 may also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 8. Further, the memory 81 may also include both an internal storage unit and an external storage device of the computer device 8. The memory 81 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0196] In addition, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0197] An embodiment of the present application provides a computer program product. When the computer program product is run on a computer device, the computer device implements the steps in the above-mentioned method embodiments when executing the computer device.

[0198] In several embodiments provided in the present application, it is understood that each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved.

[0199] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0200] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A training method for an instance segmentation model, characterized in that: include: Acquire annotation data and a target feature map of the image sample, wherein the annotation data includes bounding box annotation data and mask annotation data, wherein the bounding box annotation data includes bounding box aspect ratio annotation data, and the target feature map corresponds to a first candidate region; Based on the labeled data and the target feature map, iteratively train the bounding box recognition network and the mask recognition network in the preset elongated target instance segmentation model until the elongated target instance segmentation model reaches a preset convergence condition, stop training, and obtain a final training result of the elongated target instance segmentation model; Each time the mask recognition network is trained, the following steps are included: Determining a dynamic length and a dynamic width of a first candidate region on the target feature map based on the bounding box aspect ratio annotation data; Using the mask recognition network, extracting features of the first candidate region according to the dynamic length and the dynamic width, and obtaining mask data of the first candidate region; Based on a first loss function between the mask data and the mask annotation data, a network parameter of the mask recognition network is updated.

2. The training method of the instance segmentation model according to claim 1, characterized in that: The using the mask recognition network to extract features from the first candidate region according to the dynamic length and the dynamic width to obtain mask data of the first candidate region includes: Extracting features of the first candidate region according to the dynamic length and the dynamic width to obtain features of the first candidate region; Using a preset two-layer graph convolutional neural network, the graph convolution features of the first candidate region are extracted according to the features of the first candidate region to obtain mask data of the first candidate region.

3. The training method of the instance segmentation model according to claim 2, characterized in that: The using a preset two-layer graph convolutional neural network to extract the graph convolutional features of the first candidate region according to the features of the first candidate region to obtain mask data of the first candidate region includes: Downsampling the first candidate region feature to obtain a downsampled first candidate region feature; Using the double-layer graph convolutional neural network, linearly transform the downsampled first candidate region features to obtain a similarity adjacency matrix; Based on the similarity adjacency matrix, graph convolution features of the first candidate region are extracted to obtain mask data of the first candidate region.

4. The training method of the instance segmentation model according to claim 1, characterized in that: Each time the bounding box recognition network is trained, the following steps are included: Using the bounding box recognition network, extracting bounding box features of the first candidate area on the target feature map; Based on the bounding box features, identifying the bounding box category, bounding box coordinates, and bounding box aspect ratio of the first candidate area, and obtaining bounding box data of the first candidate area; Based on a second loss function between the bounding box data and the bounding box annotation data, network parameters of the bounding box recognition network are updated.

5. An instance segmentation method, characterized in that: include: Obtaining an instance feature map of the image to be segmented, wherein the instance feature map corresponds to a second candidate region; Based on a preset instance segmentation model, instance segmentation is performed on the second candidate area on the instance feature map to obtain bounding box instance data and mask instance data of the second candidate area, and the instance segmentation model is the final training result after training the preset slender target instance segmentation model based on the training method described in any one of claims 1 to 4 above.

6. The instance segmentation method according to claim 5, characterized in that The performing instance segmentation on the second candidate region on the instance feature map based on a preset instance segmentation model to obtain bounding box instance data and mask instance data of the second candidate region includes: Extracting bounding box features of a second candidate region on the instance feature map using a bounding box recognition network of the instance segmentation model; Based on the bounding box features, identifying the bounding box category, bounding box coordinates, and bounding box aspect ratio of the second candidate area, and outputting bounding box instance data of the second candidate area; Determining a dynamic length and a dynamic width of the second candidate region based on the aspect ratio of the bounding box identified by the bounding box recognition network; Using the mask recognition network of the instance segmentation model, feature extraction is performed on the second candidate region according to the dynamic length and the dynamic width, and mask instance data of the second candidate region is output.

7. The instance segmentation method according to claim 6, characterized in that The mask recognition network using the instance segmentation model performs feature extraction on the second candidate region according to the dynamic length and the dynamic width, and outputs mask instance data of the second candidate region, including: Extracting features of the second candidate region according to the dynamic length and the dynamic width to obtain features of the second candidate region; Utilizing a preset two-layer graph convolutional neural network, based on the features of the second candidate region, the graph convolutional features of the second candidate region are extracted, and mask instance data of the second candidate region is output.

8. The instance segmentation method according to claim 7, characterized in that: The using a preset two-layer graph convolutional neural network to extract the graph convolutional features of the second candidate region according to the features of the second candidate region, and outputting the mask instance data of the second candidate region, includes: Downsampling the second candidate region feature to obtain a downsampled second candidate region feature; Using the double-layer graph convolutional neural network, linearly transform the downsampled second candidate region features to obtain a similarity adjacency matrix; Based on the similarity adjacency matrix, graph convolution features of the second candidate region are extracted to obtain mask instance data of the second candidate region.

9. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the computer device to perform the training method according to any one of claims 1 to 4, or the instance segmentation method according to any one of claims 5 to 8.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the training method according to any one of claims 1 to 4, or the instance segmentation method according to any one of claims 5 to 8.

Images