Instance segmentation method, device, electronic device and storage medium

通过模板匹配技术改善类别响应图和实例边界响应图,解决了现有弱监督实例分割方法中欠分割和过分割的问题,提高了图像分割的准确性和一致性。

CN114549833BActive Publication Date: 2024-11-26BEIJING JIAOTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210087999.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-25
Publication Date
2024-11-26
Estimated Expiration
2042-01-25

AI Technical Summary

Technical Problem

The existing weakly supervised instance segmentation method based on image category annotation is prone to missing instance boundary parts when processing the background similar situation before processing, resulting in excessive segmentation and incomplete category response maps, resulting in undersegment.

Method used

By obtaining the category response diagram, instance center offset field and instance boundary response diagram of the image to be segmented, a prior template is established, and the category response diagram and instance boundary response diagram are improved by using template matching, reducing undersegment and oversegment, and realizing instance segmentation .

Benefits of technology

It effectively reduces the phenomenon of undersegment and oversegment, improves the accuracy of segmentation results, and makes the segmentation results more in line with human prior cognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114549833B_ABST
    Figure CN114549833B_ABST
Patent Text Reader

Abstract

The present application provides an instance segmentation method, device, electronic device and storage medium, the method comprising: obtaining a category response map, an instance center offset field and an instance boundary response map of an image to be segmented; establishing a priori template of the target according to the posture of the target in the image to be segmented; based on the category response map, the instance center offset field, the instance boundary response map and the prior template, respectively improving the category response map and the instance boundary response map through template matching to obtain corresponding template-based category response map and template-based boundary map; under the restriction of the template-based boundary map, performing region expansion based on the template category response map to obtain an instance segmentation result. This scheme can reduce under-segmentation and over-segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to an instance segmentation method, device, electronic equipment and storage medium. Background Art

[0002] The existing instance segmentation technologies in the field of image segmentation can be mainly divided into two categories, one is fully supervised instance segmentation, and the other is weakly supervised instance segmentation. The first type of fully supervised instance segmentation method, instance segmentation technology based on deep learning methods, can automatically predict the category of each pixel in the image and the instance to which it belongs. However, the main disadvantage of this method is that during the training process of the neural network, accurate image pixel-level categories and instance annotations are required, and this pixel-level annotation requires a lot of manpower and time investment, with extremely high time and economic costs, which limits the scale of available data and hinders the further development of fully supervised instance segmentation methods.

[0003] In order to reduce the complexity of data annotation, weakly supervised instance segmentation methods have emerged in recent years. Weakly supervised instance segmentation methods greatly reduce the requirements for image annotation, especially instance segmentation methods based on image category annotation, which require the lowest image data annotation cost. Instance segmentation methods based on image category annotation only require image category information, and can further obtain detailed segmentation results about the target instance through technologies such as neural networks. Taking image category annotation, which has the lowest annotation cost at present, as an example, the existing weakly supervised instance segmentation methods based on image category annotation mainly perform pixel-level instance segmentation based on perceived image information, which is easy to produce segmentation results that are obviously inconsistent with human prior cognition. Summary of the invention

[0004] The purpose of the embodiments of this specification is to provide an instance segmentation method, device, electronic device and storage medium.

[0005] To solve the above technical problems, the embodiments of the present application are implemented in the following ways:

[0006] In a first aspect, the present application provides an instance segmentation method, the method comprising:

[0007] Obtain the category response map, instance center offset field and instance boundary response map of the image to be segmented;

[0008] According to the posture of the target in the image to be segmented, establish a priori template of the target;

[0009] Based on the category response map, the instance center offset field, the instance boundary response map and the prior template, the category response map and the instance boundary response map are improved respectively through template matching to obtain the corresponding template-based category response map and template-based boundary map;

[0010] Under the restriction of the template boundary map, the region is expanded based on the template category response map to obtain the instance segmentation result.

[0011] In one embodiment, based on the category response map, the instance center offset field, the instance boundary response map and the prior template, the category response map and the instance boundary response map are improved by template matching to obtain the corresponding template-based category response map and template-based boundary map, including:

[0012] Determine the instance center position according to the instance center offset field;

[0013] Determine the candidate target location based on the instance center location;

[0014] The prior template is scaled according to a preset ratio to obtain a number of scaled templates, and the scaled templates are placed at the target candidate positions, and the template matching scores of all the scaled templates are calculated, and the scaled template corresponding to the template matching score greater than the score threshold and the maximum target matching score is selected as the matching template;

[0015] According to the matching template, the category response map and the instance boundary response map are improved respectively to obtain the corresponding template-based category response map and template-based boundary map.

[0016] In one embodiment, determining the instance center position according to the instance center offset field includes:

[0017] According to the instance center offset field, the instance center location area is estimated;

[0018] A pixel set pointing to the center position area of ​​the instance is determined as the instance area;

[0019] The center of the instance region is determined as the instance center position.

[0020] In one embodiment, determining a target candidate position according to the instance center position includes:

[0021] Select the instance center position corresponding to the instance region whose area is greater than or equal to the area threshold as the target candidate position;

[0022] and / or,

[0023] According to the instance area and the category response map, the number of pixels in the instance area whose probability of belonging to the specified category is greater than the preset probability is greater than or equal to the preset percentage of the total area of ​​the instance area, and the corresponding instance center position is the target candidate position.

[0024] In one embodiment, the template matching score includes at least one of an edge direction matching score, an offset amplitude matching score, a template area matching score, and a template boundary matching score;

[0025] The edge direction matching score is determined according to the edge contour pixel set in the scaling template, the offset field direction of each edge pixel in the scaling template relative to the scaling template center, and the instance center offset field direction;

[0026] The offset amplitude matching score is determined based on the edge contour pixel set in the scaled template and the normalized length of the instance center offset field vector;

[0027] The template area matching score is determined based on the pixel set of the template foreground area covered by the scaled template and the pixel set of the foreground area determined by the vector length of the instance center offset field;

[0028] The template boundary matching score is determined based on the set of edge contour pixels in the scaled template and the chamfer distance between the scaled template and the instance boundary response map.

[0029] In one embodiment, according to the matching template, respectively improving the category response map and the instance boundary response map to obtain the corresponding template-based category response map and template-based boundary map includes:

[0030] If the chamfer distance between the matching template and the instance boundary response map is greater than the chamfer distance threshold, the boundary of the matching template is retained to improve the instance boundary response map and obtain a template-based boundary map;

[0031] Determine the coverage area of ​​the matching template covering the category response map, and amplify the response score of the category response map in the coverage area according to a preset ratio to improve the category response map and obtain a template-based category response map.

[0032] In one embodiment, under the restriction of the template boundary map, the region is expanded based on the template category response map to obtain the instance segmentation result, including:

[0033] Determine a similarity matrix based on the instance boundary response graph;

[0034] Perform several Hadamard products on the similarity matrix and normalize the rows of the matrix to obtain a transfer matrix;

[0035] According to the boundary probability based on the template boundary map, the response score based on the template category response map is adjusted to obtain the adjusted category response map;

[0036] The transfer matrix is ​​multiplied by the adjusted category response map several times to obtain the instance segmentation result.

[0037] In a second aspect, the present application provides an instance segmentation device, the device comprising:

[0038] An acquisition module, used to acquire a category response map, an instance center offset field, and an instance boundary response map of the image to be segmented;

[0039] A template building module is used to build a priori template of the target according to the posture of the target in the image to be segmented;

[0040] A matching module is used to improve the category response map and the instance boundary response map respectively through template matching based on the category response map, the instance center offset field, the instance boundary response map and the prior template, and obtain the corresponding template-based category response map and template-based boundary map;

[0041] The expansion module is used to expand the region based on the template category response map under the restriction of the template boundary map to obtain the instance segmentation result.

[0042] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the instance segmentation method of the first aspect is implemented.

[0043] In a fourth aspect, the present application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the instance segmentation method of the first aspect.

[0044] It can be seen from the technical solution provided in the above embodiments of this specification that the solution: improves the category response map through template matching to reduce the occurrence of under-segmentation; improves the instance boundary response map through template matching to reduce the occurrence of over-segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 A flowchart of the instance segmentation method provided in this application;

[0047] Figure 2 A schematic diagram of the structure of the ResNet50 backbone network provided for this application;

[0048] Figure 3 Schematic diagram of the DP-IRNet network structure provided for this application;

[0049] Figure 4 A schematic diagram of the structure of the EDGE-IRNet network provided for this application;

[0050] Figure 5Training flow chart of the image classification network based on convolutional neural network, DP-IRNet network, and EDGE-IRNet network provided in this application;

[0051] Figure 6 A schematic diagram of the target matching process provided for this application;

[0052] Figure 7 A principle block diagram of the instance segmentation method provided in this application;

[0053] Figure 8 A comparison chart of the experimental results using the method of the present application and the results of the original method;

[0054] Fig. 9 A schematic diagram of the structure of the instance segmentation device provided in this application;

[0055] Fig.10 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0057] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0058] It will be apparent to those skilled in the art that various modifications and variations may be made to the specific embodiments of the present application description without departing from the scope or spirit of the present application. Other embodiments derived from the present application description will be apparent to those skilled in the art. The present application description and examples are exemplary only.

[0059] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0060] Unless otherwise specified, "parts" in this application are all calculated by mass.

[0061] In the related art, the target instance segmentation method based on image category annotation is mainly based on the category response map. In the image classification network based on convolutional neural network, the category response map of the target can be obtained by gradient back propagation and other technologies, that is, the salient area of ​​the corresponding category target in the image. In order to obtain the instance information in the image, the peak response map (PRM) class method guides the network to highlight the peak points in the class response map by adding a peak excitation strategy in the training process of the classification network, thereby obtaining the approximate instance target position. Then, the segmentation results from other object proposal generation methods are sorted and screened according to the peak position and the peak response map to obtain the final instance segmentation result. Subsequent researchers combined the PRM method with the fully supervised instance segmentation method, and used the segmentation results of PRM as pseudo-annotations for training the fully supervised instance segmentation network. In order to solve the problem of incomplete target area extracted by the peak response map, subsequent research uses the segmentation results generated by other object proposal generation methods as pseudo-annotations based on the peak response map extracted by PRM to enable the network to learn the ability to fill the peak response map. The filled peak response map is called the instance activation map, which is more complete in coverage area than PRM. Existing PRM-type weakly supervised instance segmentation methods not only use image category information, but also require other object proposal generation methods, which actually use additional annotation information in addition to image categories. The Inter-pixel Relation Network (IRNet) type method establishes a separate neural network to mine instance information and semantic similarity between pixels in the category response map, thereby obtaining the center offset vector of each pixel relative to the instance and the boundary map between categories. Furthermore, the random walk algorithm and conditional random field are used to improve the instance regions in the category response map.

[0062] The category-labeled instance segmentation method based on the IRNet architecture is divided into three parts. The first part is an image classification network based on a convolutional neural network, which is used to generate a category response map. In the overall process, the category response map will serve as a seed region in the semantic propagation process and provide supervision information for the IRNet network. In the second part, the IRNet network mines the category similarity between neighboring pixels from the category response map, thereby guiding the convolutional neural network to learn the ability to predict the category boundary map and the instance center offset field. The offset field indicates the location of the potential instance center in the image. In the third part, under the restriction of the category boundary map, the instance-level class response map will be regionally expanded to obtain the final instance segmentation result. The category boundary map plays a role in limiting the excessive expansion of the region in the process of regional expansion and improvement of each instance.

[0063] The above method is improved by reusing the offset field results to further obtain the instance boundary map. Specifically, in the first part, the image classification network generates a category response map. In the second part, the original IRNet is split into two separate networks, DP-IRNet and EDGE-IRNet. DP-IRNet obtains the offset field based on the category response map. The information contained in the offset field is the offset vector of each pixel position relative to its corresponding target center position. In addition to the category response map, EDGE-IRNet introduces the offset field information into the training process to obtain the instance-level boundary map. The instance-level boundary map represents the probability that each pixel position in the image belongs to the instance boundary. Finally, under the restriction of the instance boundary map, the instance-level class response map is regionally expanded to obtain the final instance segmentation result.

[0064] In summary, the existing PRM-type weakly supervised instance segmentation methods not only use image category information, but also require other object proposal methods. In fact, additional annotation information other than image category is used. The category annotation instance segmentation method based on the IRNet architecture only uses image category annotation to achieve image instance segmentation, but the current technology still has the following shortcomings:

[0065] (1) Since the segmentation boundaries between categories or instances are obtained only by relying on perceived image information, the instance boundaries are often partially missing in complex situations such as similar foreground and background, which leads to unreasonable over-segmentation problems.

[0066] (2) Since the category response map obtained based on the convolutional neural network classifier can often only focus on the local salient areas of the instance individual, such as the individual parts, the incomplete category response map is still prone to instance under-segmentation after semantic propagation expansion.

[0067] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0068] Reference Figure 1 , which shows a flow chart of an instance segmentation method applicable to the embodiment of the present application.

[0069] like Figure 1 As shown, an instance segmentation method may include:

[0070] S110, obtaining a category response map, an instance center offset field, and an instance boundary response map of the image to be segmented.

[0071] Wherein, when only image category information is supervised, the extracted category response map (or simply the class response map) can give the approximate area of ​​each category. The category response map can be obtained by using an image classification network based on a convolutional neural network, such as CAM, GradCAM, etc., which are not limited here. In this embodiment, the category response map is obtained by using an image classification network based on a convolutional neural network as an example for explanation.

[0072] Specifically, the image classification network based on the convolutional neural network can adopt the ResNet50 classification network or simply the ResNet50 network or ResNet50, and can also adopt the ResNet series, Inception series, Xception, MobileNet series and other classification networks, without limitation. In this embodiment, the ResNet50 classification network is used as an example for explanation. The backbone network structure of ResNet50 is as follows: Figure 2 As shown. During use, the step size of the last downsampling layer of the ResNet50 network is set to 1 to prevent the resolution of the category response map from being further reduced. In order to extract the category response map, the last classification layer and the maximum pooling layer of the trained classification network are removed to obtain the feature map output by the last convolution layer. The category response map is obtained by multiplying the vector composed of the corresponding classification weights of each category with the feature map and normalizing it. The value of each pixel in the category response map represents the probability that the pixel belongs to a certain category.

[0073] The information contained in the instance center offset field, referred to as the offset field, is the offset vector of each pixel position relative to its corresponding target center position. The offset field can be obtained based on the category response map using the Displacement Field Inter-pixel Relation Network (DP-IRNet) or DP-IRNet for short.

[0074] Specifically, the DP-IRNet network can also use the ResNet50 network as the backbone network. The network structure of the DP-IRNet network is as follows: Figure 3 As shown in the figure, the feature maps of the five different stages of the ResNet50 network are subjected to 1×1 convolution operations respectively, and then the features at all levels are fused through a series of convolution, upsampling, feature cascading and other operations, and finally the predicted instance center offset field is output.

[0075] When the DP-IRNet network is trained to obtain the instance center offset field, the network training data comes from the neighborhood point pairs in the category response map. Point pairs belonging to the same category in the neighborhood are considered to be from the same instance, and points belonging to different categories in the neighborhood are considered to be from different instances. The loss function of network training is defined based on the difference vector between the offset vectors of two pixels pointing to the instance center in the same instance being equal to the vector formed by the positions of the two pixels.

[0076] Among them, the instance boundary response map or instance boundary map for short can be obtained by using the edge pixel relationship network (EDGE-IRNet, Edge Inter-pixel Relation Network) or EDGE-IRNet for short, using the semantic similarity of the neighboring point pairs in the offset field and the category response map.

[0077] Specifically, the EDGE-IRNet network or EDGE-IRNet for short can also use the ResNet50 network as the backbone network. The network structure of the EDGE-IRNet network is as follows: Figure 4 As shown in the figure, the EDGE-IRNet network uses the feature maps of the five different stages of the ResNet50 network. The first-level feature map and the second-level feature map are cascaded with the up-sampled results of the third to fifth-level feature maps after 1×1 convolution, and finally a 1×1 convolution operation is performed to output the predicted instance boundary map.

[0078] When the EDGE-IRNet network is trained to obtain the instance boundary map, the network training data comes from the neighborhood point pairs in the category response map and the instance center offset field. Point pairs belonging to the same category and the same instance in the neighborhood are considered to be from the same instance, and point pairs belonging to different categories or different instances in the neighborhood are considered to be from different instances. The loss function of network training is defined based on the fact that two pixel pairs in the same instance have high semantic similarity and no instance boundary between them, and two pixel pairs from different instances have low semantic similarity and an instance boundary between them.

[0079] It can be understood that before using the image classification network based on convolutional neural network, DP-IRNet network, and EDGE-IRNet network to obtain the category response map, offset field, and instance boundary map, respectively, the image classification network based on convolutional neural network, DP-IRNet network, and EDGE-IRNet network need to be trained respectively, such as Figure 5 As shown, first train the image classification network based on convolutional neural network, obtain the class response map through the trained image classification network based on convolutional neural network, train DP-IRNet according to the class response map, obtain the offset field through the trained DP-IRNet, and train EDGE-IRNet according to the class response map and the offset field.

[0080] S120: Establish a priori template of the target according to the posture of the target in the image to be segmented.

[0081] The target in the image to be segmented can be called an object. Depending on the image to be segmented, the object can be a human body, a vehicle, etc.

[0082] For example, taking the human body in the Pascal VOC2012 dataset as an example, we constructed a total of 27 human body templates. The human body is roughly divided into three areas, namely the head and shoulder area, the torso area, and the leg area. The head and shoulder area is composed of 5 templates, consisting of 5 heads and shoulders with different tilt angles. The torso area is divided into two situations: front standing and side standing. The torsos in the two states are combined with 5 head and shoulder templates to form 10 templates for the upper body. The leg area is divided into two situations: front standing and side standing, and is combined with the corresponding 10 upper body templates to form 10 complete human body templates. In addition, in order to handle the situation where only the head is exposed in the dataset, two additional head templates with larger tilt angles are added.

[0083] It should be noted that the present application does not limit the execution order of S110 and S120. S110 may be executed first and then S120, or S120 may be executed first and then S110, or S110 and S120 may be executed simultaneously. There is no limitation on this.

[0084] S130, based on the category response map, the instance center offset field, the instance boundary response map and the prior template, improve the category response map and the instance boundary response map respectively through template matching to obtain the corresponding template-based category response map and template-based boundary map.

[0085] Specifically, step S120 introduces a priori model of the object by template matching based on the category response map, the instance center offset field and the instance boundary response map, and improves the category response map and the instance boundary map by using the prior knowledge of human cognition of the object.

[0086] Specifically, the template matching process is as follows: Figure 6 As shown, it includes: instance center positioning, instance center screening, template matching evaluation, and template information fusion, which correspond to S1301-S1304 respectively.

[0087] S1301, determining the instance center position according to the instance center offset field, may include:

[0088] According to the instance center offset field, the instance center location area is estimated;

[0089] A pixel set pointing to the center position area of ​​the instance is determined as the instance area;

[0090] The center of the instance region is determined as the instance center position.

[0091] Specifically, the instance center offset field is obtained through the DP-IRNet network. The instance center position offset vector amplitude in the instance center offset field is small. Therefore, by counting the area with a small offset vector length in the offset field, the approximate area where the instance center position of the target is located can be obtained (that is, the instance center position area is estimated). The approximate area where all the estimated instance center positions are located can be regarded as several candidate target centers. Each pixel in the image to be segmented will be divided according to the pointing position of the refined offset field, that is, the set of pixels pointing to a candidate target center will be divided into an instance area, which is also called a candidate area. These instance areas divide an image into several parts, called instance maps, each of which corresponds to a candidate target. The centers of these instance areas will be identified as instance center positions or candidate positions or target center positions.

[0092] S1301 Instance center positioning The instance center is positioned using the instance center information obtained from the offset field.

[0093] S1302: Determine a target candidate position according to the instance center position.

[0094] Specifically, this step can further filter the instance center position obtained in S1301 according to the instance area and the instance category of the instance center position.

[0095] In one embodiment, S1302 may include: selecting the instance center position corresponding to the instance region whose area is greater than or equal to the area threshold as the target candidate position, that is, excluding candidate regions with too small a size.

[0096] The area threshold can be set according to actual needs. The above area and area threshold are calculated based on the number of pixels. For example, the area threshold is set to 1000 pixels.

[0097] For example, taking the human target in the Pascal VOC2012 dataset as an example, when the instance region area where the candidate position is located is less than 1000 pixels, the corresponding instance center position is excluded, that is, the instance center position corresponding to the instance region area greater than or equal to 1000 pixels is retained as the target candidate position.

[0098] In one embodiment, S1302 may further include: based on the instance area and the category response map, selecting the number of pixels in the instance area whose probability of belonging to the specified category is greater than a preset probability and is greater than or equal to a preset percentage of the total area of ​​the instance area, and the corresponding instance center position is the target candidate position.

[0099] The category probability threshold can be set according to actual needs, for example, it can be set to 25%; the preset percentage can be set according to actual needs, for example, it can be set to 20%.

[0100] It can be understood that the category to which the target belongs can be obtained according to the category response map. By combining the instance region and the category response map, the category of similar pixels in the instance region can be determined according to the category response map.

[0101] Exemplarily, taking the human target in the Pascal VOC2012 dataset as an example, when the number of pixels in the instance area where the candidate position is located with a probability of belonging to the specified category greater than 25% does not reach 20% of the total area of ​​the instance area where the candidate position is located, the candidate position is excluded, that is, the instance center position corresponding to the number of pixels in the instance area with a probability of belonging to the specified category greater than 25% reaching 20% ​​of the total area of ​​the instance area is retained as the target candidate position.

[0102] In S1302 instance center screening, the above two methods use information such as category response diagrams to screen candidate target locations and exclude target positioning results that do not meet the priori requirements. The above two methods can be selected for screening at will, or both methods can be used for screening at the same time. When using the two methods for screening, the execution order between the two methods is not limited.

[0103] S1303, scaling the prior template according to a preset ratio to obtain several scaled templates, placing the scaled templates at target candidate positions, calculating template matching scores of all scaled templates, and selecting a scaled template whose template matching score is greater than a score threshold and whose target matching score is the largest as a matching template.

[0104] Specifically, the preset scaling ratio of the prior template is a ratio set according to actual needs, and the resolution of the prior template is magnified or reduced by multiples according to the scaling ratio to obtain a number of scaled templates. The scaled template is placed at the target candidate position and the template matching process is performed. According to the obtained template matching score, a suitable instance template and its corresponding template size are selected as the matching template to guide the template information fusion of S1304.

[0105] The score threshold can be set according to actual needs.

[0106] In this scheme, the template prior information is represented from three aspects, namely, the template boundary, the template area shape, and the template direction, which can be expressed as Based on these three prior features of the template, they are naturally combined with the boundary map, instance map, offset field and other information in the overall IRNet process to obtain an evaluation of the template matching degree.

[0107] In one embodiment, the template matching score includes at least one of an edge direction matching score, an offset amplitude matching score, a template area matching score, and a template boundary matching score.

[0108] Among them, the edge direction matching score S dp_ori Used to evaluate edge direction matching, this evaluation is used to compare the matching degree between the pointing direction of the edge pixels of the scaled template and the pointing direction of the pixels in the offset field predicted by IRNet. The edge direction matching score is determined based on the set of edge contour pixels in the scaled template, the offset field direction of each edge pixel in the scaled template relative to the center of the scaled template, and the offset field direction of the instance center. Specifically, the set of edge contour pixels in the scaled template is denoted as Ω tem_b , α TEM Indicates the offset field direction of each edge pixel in the scaling template relative to the center of the scaling template, α DP Indicates the predicted instance center offset field direction, then the edge direction matching score S dp_ori It can be obtained from formula 1. Ω tem_b , α TEM , α DP All are two-dimensional matrices. tem_b It is a binary matrix, where the edge contour pixels are 1 and the rest are 0. x,y represent the offset position relative to the center of the scaling template.

[0109]

[0110] Among them, the offset amplitude matching score S dp_mag Used to evaluate the offset magnitude match, which encourages scaling edge pixels of the template Ω tem_b The offset vector length of the offset field area covered is as large as possible, so as to encourage the size of the scaled template to cover the target area as completely as possible. The offset amplitude matching score is determined based on the edge contour pixel set in the scaled template and the normalized length of the instance center offset field vector. Specifically, γ DP represents the normalized length of the offset field vector of the predicted output, then the offset amplitude matching score S dp_mag As shown in formula 2.

[0111]

[0112] Among them, the template area matching score S dp_region Used to evaluate template region matching. This item evaluates the matching degree between the scaled template region and the area covered by the offset field by calculating the intersection-and-union ratio between the scaled template region and the predicted foreground region. The larger the intersection-and-union ratio, the closer the area covered by the scaled template is to the predicted target foreground region. The template region matching score is determined based on the pixel set of the template foreground area covered by the scaled template and the pixel set of the foreground area determined by the vector length of the instance center offset field. Specifically, Ωtem represents the pixel set of the target foreground area covered by the zoom template, Ω fg Represents the set of pixels in the foreground area determined by the vector length of the offset field of the corresponding instance position predicted by IRNet. If the offset vector length of a pixel is not less than 10% of the longest offset vector length, the pixel is considered to belong to Ω fg Then the template area matching score S dp_region The calculation method is as shown in formula 3. Ω tem It is a binary matrix, where foreground pixels are 1 and the rest are 0.

[0113]

[0114] Among them, the template boundary matching score S boundary It is used to evaluate the template boundary matching. By calculating the chamfer distance between the scaled template boundary and the instance boundary map predicted by the EDGE-IRNet network, the matching degree between the scaled template boundary and the target area boundary to be matched is evaluated. The smaller the average chamfer distance, the higher the matching degree. The template boundary matching score is determined based on the edge contour pixel set in the scaled template and the chamfer distance between the scaled template and the instance boundary response map. Specifically, represents the chamfer distance between the template and the instance boundary map, then the template boundary matching score S boundary The calculation of is shown in formula 4.

[0115]

[0116] Template matching score S match It can be any one of the edge direction matching score, offset amplitude matching score, template area matching score, template boundary matching score, or the sum of two or more of them. match When it is the sum of the above four:

[0117] S match =S dp_ori +S dp_mag +S dp_region +10×S boundary (Formula 5)

[0118] It can be understood that the template matching score can also be determined by other edge matching methods.

[0119] The scaled template corresponding to the template matching score being higher than the score threshold will proceed to the next step S1304 template information fusion process.

[0120] S1304 , improving the category response map and the instance boundary response map respectively according to the matching template to obtain a corresponding template-based category response map and template-based boundary map.

[0121] Specifically, after obtaining a matching template that meets the requirements in S1303, S1304 integrates the template prior information into the weakly supervised instance segmentation process from two aspects.

[0122] First, the matching template is used to improve the instance boundary response map obtained by EDGE-IRNet, including: if the chamfer distance between the matching template and the instance boundary response map is greater than the chamfer distance threshold, the boundary of the matching template is retained to improve the instance boundary response map and obtain a template-based boundary map.

[0123] The chamfer distance threshold can be set according to actual needs. When the chamfer distance between the matching template and the instance boundary response map is greater than the chamfer distance threshold, the boundary of the matching template is added to the instance boundary response map, thereby improving the instance boundary response map.

[0124] It is understandable that when using the matching template to improve the instance boundary response map, different strategies can be used, such as completely retaining the template edge, partially retaining the template edge, etc., which is not limited here.

[0125] Secondly, the matching template is used to improve the category response map. The category response map usually only focuses on the local significant area, determines the coverage area of ​​the category response map covered by the matching template, and amplifies the response score of the category response map in the coverage area according to a preset ratio to improve the category response map and obtain a template-based category response map.

[0126] Among them, the preset ratio can be set according to actual needs.

[0127] The aspect matching template improves the category response map and can improve the under-segmentation problem.

[0128] It is understandable that different strategies may be used to expand the scope of the category response map using the matching template, such as by region growth, increasing the response score between template coverage areas, etc., which are not limited here.

[0129] S140, under the restriction of the template boundary map, performing region expansion based on the template category response map to obtain an instance segmentation result, may include:

[0130] Determine a similarity matrix based on the instance boundary response graph;

[0131] Perform several Hadamard products on the similarity matrix and normalize the rows of the matrix to obtain a transfer matrix;

[0132] According to the boundary probability based on the template boundary map, the response score based on the template category response map is adjusted to obtain the adjusted category response map;

[0133] The transfer matrix is ​​multiplied by the adjusted category response map several times to obtain the instance segmentation result.

[0134] Among them, S140 uses a random walk algorithm to propagate the category response score or simply the response score or response fraction of the class response graph to a reasonable related area.

[0135] Specifically, through the instance boundary map obtained by EDGE-IRNet, a semantic similarity matrix between pixels can be obtained, which can be referred to as a similarity matrix. Each element in the similarity matrix represents the correlation between two pixels. The higher the element score, the higher the correlation of the pixel pair, and the higher the probability of propagating the attention score. In order to smooth the values ​​in the similarity matrix, it is subjected to several Hadamard products, and then the rows of the matrix are normalized to obtain a transfer matrix. The transfer matrix is ​​multiplied several times with the template-based category response map obtained in S1304 to obtain the result of region growth. Before multiplying with the transfer matrix, the template-based category response map is adjusted using the predicted boundary probability based on the template boundary map, specifically adjusting the response score based on the template category response map. The higher the boundary probability, the lower the response score here. Exemplarily, the adjusted response score = (1-boundary probability) × response score. The last 25% of the response score obtained in the end is identified as the background, and the other positions obtain the final instance segmentation result.

[0136] Reference Figure 7 , which shows a principle block diagram of the instance segmentation method provided by the present application. Figure 7 As shown, the overall scheme of the instance segmentation method proposed in this application includes four parts. First, a class response map is obtained using an image classification network based on a convolutional neural network. In the case of only image category information supervision, the class response map extracted from the convolution classification network can give the approximate area of ​​each category. In the second part, DP-IRNet obtains the offset field based on the category response map. The information contained in the offset field is the offset vector of each pixel position relative to its corresponding target center position. EDGE-IRNet uses the offset field information and the semantic similarity of the neighboring point pairs in the category response map to obtain the instance boundary map. In the third part, this application introduces a template matching process before the region expansion step, and adjusts and supplements the category response map and improves the instance boundary response map by explicitly introducing the prior knowledge of the object. Finally, under the restriction of the instance boundary map, the instance-level class response map is regionally expanded to obtain the final instance segmentation result.

[0137] In the embodiment of the present application, based on the boundary map and instance center offset field generated by the neural network, the prior knowledge about the object shape is explicitly introduced through the object shape template. The position and posture of the object instance are basically determined through the template matching technology, so that the prior knowledge of the object shape is used to supplement the missing instance boundaries. The more complete instance boundaries make the segmentation results consistent with the prior knowledge and avoid unreasonable over-segmentation.

[0138] In the embodiment of the present application, an object shape prior template is introduced through template matching technology, the category response score of the area within the matched object template is expanded, and the overall instance is improved to a certain extent, thereby reducing the under-segmentation phenomenon based on area expansion.

[0139] Experimental verification

[0140] Through the segmentation experiment of human targets in the Pascal VOC2012 dataset, the quality of segmentation annotations generated by the scheme of the present invention is further improved than that of the original scheme. The quantitative experimental results are shown in Table 1, with the intersection-over-union ratio greater than or equal to 50% as the correct instance segmentation.

[0141] Table 1 Comparison of instance segmentation accuracy on the Pascal VOC2012 dataset

[0142]

[0143] Some examples of experimental results are as follows Figure 8 As shown (where Figure 8 The original color is in color, and grayscale processing is performed here. The experimental results of the segmentation method of the present application are closer to the real annotation than the results of the original method, which can effectively improve the under-segmentation and over-segmentation problems of the target and improve the segmentation effect.

[0144] Reference Fig. 9 , which shows a schematic diagram of the structure of an instance segmentation device described according to an embodiment of the present application.

[0145] like Fig. 9 As shown, the instance segmentation device 900 may include:

[0146] An acquisition module 910 is used to acquire a category response map, an instance center offset field, and an instance boundary response map of the image to be segmented;

[0147] The template building module 920 is used to build a priori template of the target according to the posture of the target in the image to be segmented;

[0148] A matching module 930 is used to improve the category response map and the instance boundary response map respectively through template matching based on the category response map, the instance center offset field, the instance boundary response map and the prior template, and obtain a corresponding template-based category response map and template-based boundary map;

[0149] The expansion module 940 is used to perform region expansion based on the template category response map under the restriction of the template boundary map to obtain an instance segmentation result.

[0150] Optionally, the matching module 930 is further configured to:

[0151] Determine the instance center position according to the instance center offset field;

[0152] Determine the candidate target location based on the instance center location;

[0153] The prior template is scaled according to a preset ratio to obtain a number of scaled templates, and the scaled templates are placed at the target candidate positions, and the template matching scores of all the scaled templates are calculated, and the scaled template corresponding to the template matching score greater than the score threshold and the maximum target matching score is selected as the matching template;

[0154] According to the matching template, the category response map and the instance boundary response map are improved respectively to obtain the corresponding template-based category response map and template-based boundary map.

[0155] Optionally, the matching module 930 is further configured to:

[0156] According to the instance center offset field, the instance center location area is estimated;

[0157] A pixel set pointing to the center position area of ​​the instance is determined as the instance area;

[0158] The center of the instance region is determined as the instance center position.

[0159] Optionally, the matching module 930 is further configured to:

[0160] Select the instance center position corresponding to the instance region whose area is greater than or equal to the area threshold as the target candidate position;

[0161] and / or,

[0162] According to the instance area and the category response map, the number of pixels in the instance area whose probability of belonging to the specified category is greater than the preset probability is greater than or equal to the preset percentage of the total area of ​​the instance area, and the corresponding instance center position is the target candidate position.

[0163] Optionally, the matching module 930 is further configured to: the template matching score includes at least one of an edge direction matching score, an offset amplitude matching score, a template area matching score, and a template boundary matching score;

[0164] The edge direction matching score is determined according to the edge contour pixel set in the scaling template, the offset field direction of each edge pixel in the scaling template relative to the scaling template center, and the instance center offset field direction;

[0165] The offset amplitude matching score is determined based on the edge contour pixel set in the scaled template and the normalized length of the instance center offset field vector;

[0166] The template area matching score is determined based on the pixel set of the template foreground area covered by the scaled template and the pixel set of the foreground area determined by the vector length of the instance center offset field;

[0167] The template boundary matching score is determined based on the set of edge contour pixels in the scaled template and the chamfer distance between the scaled template and the instance boundary response map.

[0168] Optionally, the matching module 930 is further configured to:

[0169] If the chamfer distance between the matching template and the instance boundary response map is greater than the chamfer distance threshold, the boundary of the matching template is retained to improve the instance boundary response map and obtain a template-based boundary map;

[0170] Determine the coverage area of ​​the matching template covering the category response map, and amplify the response score of the category response map in the coverage area according to a preset ratio to improve the category response map and obtain a template-based category response map.

[0171] Optionally, the expansion module 940 is further used for:

[0172] Determine a similarity matrix based on the instance boundary response graph;

[0173] Perform several Hadamard products on the similarity matrix and normalize the rows of the matrix to obtain a transfer matrix;

[0174] According to the boundary probability based on the template boundary map, the response score based on the template category response map is adjusted to obtain the adjusted category response map;

[0175] The transfer matrix is ​​multiplied by the adjusted category response map several times to obtain the instance segmentation result.

[0176] The present embodiment provides an instance segmentation device that can execute the embodiment of the above method. Its implementation principle and technical effect are similar and will not be described in detail here.

[0177] Fig.10 FIG. 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Fig.10 As shown, a structural schematic diagram of an electronic device 300 suitable for implementing an embodiment of the present application is shown.

[0178] like Fig.10 As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage part 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the device 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0179] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0180] In particular, according to the embodiments of the present disclosure, the above reference Figure 1 The described process may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing the above-described instance segmentation method. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 309, and / or installed from a removable medium 311.

[0181] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of the code, and the aforementioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0182] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor. The names of these units or modules do not constitute limitations on the units or modules themselves in certain circumstances.

[0183] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop, a mobile phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0184] As another aspect, the present application further provides a storage medium, which may be the storage medium included in the aforementioned device in the above embodiment; or may be a storage medium that exists independently and is not assembled into the device. The storage medium stores one or more programs, and the aforementioned programs are used by one or more processors to execute the instance segmentation method described in the present application.

[0185] Storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0186] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0187] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. An instance segmentation method, characterized in that: The method comprises: Obtain the category response map, instance center offset field and instance boundary response map of the image to be segmented; According to the posture of the target in the image to be segmented, establishing a priori template of the target; Based on the category response map, the instance center offset field, the instance boundary response map and the prior template, the category response map and the instance boundary response map are respectively improved by template matching to obtain corresponding template-based category response map and template-based boundary map, including: Determining the instance center position according to the instance center offset field; Determine a target candidate position according to the instance center position; Scaling the prior template according to a preset ratio to obtain a plurality of scaled templates, placing the scaled templates at the target candidate positions, calculating template matching scores of all the scaled templates, and selecting a scaled template whose template matching score is greater than a score threshold and whose target matching score is the largest as a matching template; According to the matching template, respectively improving the category response map and the instance boundary response map to obtain a corresponding template-based category response map and a template-based boundary map; Under the restriction of the template-based boundary map, the template-based category response map performs region expansion to obtain an instance segmentation result, including: Determining a similarity matrix according to the instance boundary response graph; Performing several Hadamard products and matrix row normalization processes on the similarity matrix to obtain a transfer matrix; According to the boundary probability based on the template boundary map, adjusting the response score based on the template category response map to obtain an adjusted category response map; The transfer matrix is ​​multiplied by the adjusted category response map several times to obtain the instance segmentation result.

2. The method according to claim 1, characterized in that: Determining the instance center position according to the instance center offset field includes: According to the instance center offset field, an instance center position area is estimated; Determine a pixel set pointing to the central position area of ​​the instance as the instance area; The center of the instance area is determined as the instance center position.

3. The method according to claim 2, characterized in that The step of determining a target candidate position according to the instance center position includes: Selecting the instance center position corresponding to the instance region whose area is greater than or equal to the area threshold as the target candidate position; and / or, According to the instance area and the category response map, the number of pixels in the instance area whose probability of belonging to the specified category is greater than a preset probability is greater than or equal to a preset percentage of the total area of ​​the instance area, and the corresponding instance center position is the target candidate position.

4. The method according to claim 1, characterized in that: The template matching score includes at least one of an edge direction matching score, an offset amplitude matching score, a template area matching score, and a template boundary matching score; The edge direction matching score is determined according to the edge contour pixel set in the scaling template, the offset field direction of each edge pixel in the scaling template relative to the scaling template center, and the instance center offset field direction; The offset amplitude matching score is determined according to the edge contour pixel set in the scaling template and the normalized length of the instance center offset field vector; The template area matching score is determined according to a pixel set of a template foreground area covered by the scaled template and a pixel set of a foreground area determined by a vector length of the instance center offset field; The template boundary matching score is determined according to a set of edge contour pixels in the scaled template and a chamfer distance between the scaled template and the instance boundary response map.

5. The method according to claim 1, characterized in that The step of improving the category response map and the instance boundary response map respectively according to the matching template to obtain a corresponding template-based category response map and template-based boundary map comprises: If the chamfer distance between the matching template and the instance boundary response map is greater than a chamfer distance threshold, retaining the boundary of the matching template to improve the instance boundary response map to obtain the template-based boundary map; Determine the coverage area of ​​the category response map covered by the matching template, and amplify the response score of the category response map in the coverage area according to a preset ratio to improve the category response map to obtain the template-based category response map.

6. An instance segmentation device, characterized in that: The device comprises: An acquisition module, used to acquire a category response map, an instance center offset field, and an instance boundary response map of the image to be segmented; A template building module, used to build a priori template of the target according to the posture of the target in the image to be segmented; A matching module, for improving the category response map and the instance boundary response map respectively through template matching based on the category response map, the instance center offset field, the instance boundary response map and the prior template, and obtaining corresponding template-based category response map and template-based boundary map, including: determining the instance center position according to the instance center offset field; determining the target candidate position according to the instance center position; scaling the prior template according to a preset ratio to obtain a plurality of scaled templates, placing the scaled templates at the target candidate positions, calculating the template matching scores of all the scaled templates, and selecting the scaled template corresponding to the template matching score being greater than the score threshold and the target matching score being the largest as the matching template; improving the category response map and the instance boundary response map respectively according to the matching template, and obtaining corresponding template-based category response map and template-based boundary map; The expansion module is used to expand the region of the template-based category response map under the restriction of the template-based boundary map to obtain the instance segmentation result, including: determining a similarity matrix according to the instance boundary response map; performing Hadamard product and matrix row normalization processing on the similarity matrix several times to obtain a transfer matrix; adjusting the response score of the template-based category response map according to the boundary probability of the template-based boundary map to obtain an adjusted category response map; multiplying the transfer matrix and the adjusted category response map several times to obtain the instance segmentation result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the instance segmentation method according to any one of claims 1 to 5 is implemented.

8. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the instance segmentation method according to any one of claims 1 to 5 is implemented.