Image segmentation method, device, electronic device and storage medium
By combining semantic segmentation network and affinity segmentation network, using the affinity kernel to determine the affinity between pixels and neighboring pixels, the problem of low accuracy of instance segmentation in the prior art is solved, and efficient segmentation of overlapping objects and occlusions is achieved Effect.
Patent Information
- Application Number
- CN202011504508.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-17
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2040-12-17
AI Technical Summary
The existing instance segmentation method is difficult to accurately distinguish and fuse parts of the same instance when processing multiple overlapping objects and occlusions, resulting in a low segmentation accuracy.
Using a method of combining semantic segmentation networks and affinity segmentation networks, the affinity kernel is used to determine the affinity between pixels and neighboring pixels, and an instance segmentation graph that is independent of category is constructed and combined with the Semantic segmentation graph fusion to obtain instance segmentation results with categories.
Effectively distinguish overlapping instances and fuse the same instance with multiple components, improving the accuracy of instance segmentation.
Smart Images
Figure CN112614134B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image segmentation method, device, electronic equipment and storage medium. Background Art
[0002] Instance segmentation is another difficult problem after computer vision object detection methods. It requires labeling the instance to which the pixel belongs on a pixel-by-pixel basis in the input image.
[0003] In the prior art, proposal-based instance segmentation methods (maskRCNN) are often easy to distinguish single objects in an image, but for multiple overlapping objects, since the minimum enclosing rectangles of these objects overlap, the network can only output one result for the feature map within the rectangle, making it difficult to distinguish overlapping objects. For an object in an image that has more connected domains due to occlusion, or the object itself has more components, in this case, some bottom-up methods find it difficult to merge each part of the same instance.
[0004] It can be seen that for images with multiple objects overlapping, the existing instance segmentation method has a low segmentation accuracy. Summary of the invention
[0005] In view of the above problems, embodiments of the present invention are proposed to provide an image segmentation method, device, electronic device and storage medium that overcome the above problems or at least partially solve the above problems.
[0006] According to a first aspect of an embodiment of the present invention, there is provided an image segmentation method, comprising:
[0007] Perform semantic segmentation on the image to be segmented through the semantic segmentation network to obtain a semantic segmentation map;
[0008] Processing the image to be segmented by an affinity segmentation network to obtain an affinity graph, wherein the affinity graph represents the probability that the current pixel and the affinity neighbor pixel belong to the same instance, and the positional relationship between the current pixel and the affinity neighbor pixel is determined by an affinity kernel;
[0009] constructing a category-independent instance segmentation map according to the affinity map and the affinity kernel;
[0010] The instance segmentation map is fused with the semantic segmentation map to obtain an instance segmentation result with categories.
[0011] According to a second aspect of an embodiment of the present invention, there is provided an image segmentation device, comprising:
[0012] A semantic segmentation module, which is used to perform semantic segmentation on the image to be segmented through a semantic segmentation network to obtain a semantic segmentation map;
[0013] An affinity segmentation module, used for processing the image to be segmented through an instance segmentation network to obtain an affinity graph, wherein the affinity graph represents the probability that the current pixel and the affinity neighbor pixel belong to the same instance, and the positional relationship between the current pixel and the affinity neighbor pixel is determined by an affinity kernel;
[0014] A graph merging module, configured to construct a category-independent instance segmentation graph based on the affinity graph and the affinity kernel;
[0015] The fusion module is used to fuse the instance segmentation map with the semantic segmentation map to obtain an instance segmentation result with categories.
[0016] According to a third aspect of an embodiment of the present invention, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the image segmentation method as described in the first aspect is implemented.
[0017] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image segmentation method as described in the first aspect is implemented.
[0018] The image segmentation method, device, electronic device and storage medium provided by the embodiments of the present invention perform semantic segmentation on the image to be segmented through a semantic segmentation network to obtain a semantic segmentation map, process the image to be segmented through an affinity segmentation network to obtain an affinity map, construct an instance segmentation map independent of the category based on the affinity map and the affinity kernel, and fuse the instance segmentation map with the semantic segmentation map to obtain an instance segmentation result with the category. Since the affinity of the current pixel and the affinity neighbor pixel in the affinity kernel can be determined through the affinity kernel, overlapping instances can be well distinguished, and each part of the same instance with more components can be fused together, thereby improving the accuracy of instance segmentation.
[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.
[0021] Figure 1 is a flowchart of the steps of an image segmentation method provided by an embodiment of the present invention;
[0022] Figure 2 is a schematic diagram of an affinity core in an embodiment of the present invention;
[0023] Figure 3 It is a structural block diagram of an image segmentation device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0024] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.
[0025] Figure 1 is a flowchart of the steps of an image segmentation method provided by an embodiment of the present invention, such as Figure 1 As shown, the method may include:
[0026] Step 101, semantic segmentation is performed on the image to be segmented through a semantic segmentation network to obtain a semantic segmentation map.
[0027] Among them, the semantic segmentation map marks the instance category described by each pixel.
[0028] The semantic segmentation network based on convolutional neural network predicts the semantic score of the image to be segmented at the pixel level, determines the instance category to which each pixel belongs, and labels each pixel according to the instance category to which it belongs, thereby generating a semantic segmentation map. Different colors can be used to represent different instance categories in the semantic segmentation map. For example, blue can be used to represent the instance "antenna", while green can be used to represent the instance "branch". Different instances belonging to the same category cannot be distinguished in the semantic segmentation map. That is, when there are multiple instances of the same category in the image to be segmented, the semantic segmentation map can only give the categories to which the multiple instances belong, but cannot distinguish a single instance.
[0029] Step 102, the image to be segmented is processed by an affinity segmentation network to obtain an affinity graph, wherein the affinity graph represents the probability that the current pixel and the affinity neighbor pixel belong to the same instance, and the positional relationship between the current pixel and the affinity neighbor pixel is determined by an affinity kernel.
[0030] The shape of the affinity kernel is non-centrosymmetric, the number of channels of the affinity map is the same as the number of the affinity neighbor pixels, and the number of the affinity neighbor pixels is determined according to the instance segmentation data set.
[0031] The affinity map is obtained by processing the image to be segmented through the affinity segmentation network. The affinity map includes multiple channels. The number of channels in the affinity map is the same as the number of affinity neighbor pixels. Each channel in the affinity map represents the probability that the current pixel and the affinity neighbor pixel represented by the channel belong to the same instance.
[0032] The affinity kernel determines the positional relationship between the current pixel and its neighbor pixels, as well as the number of neighbor pixels. The affinity kernel can be determined by the instance segmentation dataset, that is, the positional relationship between the current pixel and its neighbor pixels, as well as the number of neighbor pixels, can be determined based on the instance segmentation dataset. The instance segmentation dataset is a dataset for training semantic segmentation networks and affinity segmentation networks, and each data sample in the dataset is labeled with individual instances.
[0033] The shape of the affinity core is non-centrosymmetric, which can reduce data redundancy. Assuming that the shape of the affinity core is a centrosymmetric shape, such as a circle, the probability of the current pixel and one of the affinity neighbor pixels belonging to the same instance will be given twice, resulting in data redundancy, while the non-centrosymmetric shape will not have this problem. The shape of the affinity core can be selected as a semicircle, which can make the interval between the affinity neighbor pixels fixed, so that relatively dense affinity neighbor pixels can be obtained, and overlapping instances can be well distinguished.
[0034] Step 103: construct a category-independent instance segmentation map based on the affinity map and the affinity kernel.
[0035] When the probability that the current pixel and the affinity neighbor pixel belong to the same instance in the affinity map is greater than or equal to the threshold, it is determined that the current pixel and the affinity neighbor pixel belong to the same instance; when the probability that the current pixel and the affinity neighbor pixel belong to the same instance in the affinity map is less than the threshold, it is determined that the current pixel and the affinity neighbor pixel do not belong to the same instance. According to the positional relationship between the current pixel and the affinity neighbor pixel determined in the affinity kernel and whether the current pixel and the affinity neighbor pixel in the affinity map belong to the same instance, the instance segmentation map is constructed to obtain an instance segmentation map that is independent of the category. The instance segmentation map that is independent of the category distinguishes individual instances, but cannot distinguish the instance category to which the instance belongs.
[0036] In one embodiment of the present invention, constructing a category-independent instance segmentation map based on the affinity map and the affinity core includes: constructing the affinity map as an undirected graph based on the affinity core; and performing graph merging on the undirected graph to obtain a category-independent instance segmentation map.
[0037] The undirected graph refers to a graph whose edges have no direction.
[0038] According to the positional relationship between the current pixel and the affinity neighbor pixels in the affinity kernel, the current pixel and the affinity neighbor pixels belonging to the same instance in the affinity graph are used as vertices of the undirected graph, and the edges between the current pixel and the affinity neighbor pixels belonging to the same instance are used as undirected edges of the undirected graph to construct an undirected graph corresponding to the affinity graph. Then, a graph merge algorithm or a graph partition algorithm is used to merge the undirected graph, and the pixels belonging to the same instance are merged into one instance to obtain an instance segmentation graph that is independent of the category.
[0039] When merging undirected graphs, we can first find the undirected edge with the highest probability, merge the two vertices of the undirected edge into a new superpixel vertex, merge the two vertices to obtain the superpixel vertex, and then update the vertex set and undirected edge set in the undirected graph. When updating the vertex set, we add the new superpixel vertex to the vertex set and remove the two vertices corresponding to the superpixel vertex. When updating the undirected edge set, we remove the undirected edges composed of the two vertices corresponding to the superpixel vertex in the undirected edge set, and connect the undirected edges composed of the two vertices to the superpixel vertex to generate a new undirected edge. The probability of the new undirected edge is determined based on the probability of the undirected edge composed of the two vertices. The above process is iteratively performed, that is, finding a new undirected edge with the highest probability until the probability of the undirected edge with the highest probability is less than the probability threshold. Finally, an updated set of vertices will be obtained, each vertex (which can be a pixel or a superpixel) represents an instance. The superpixel is restored to a pixel set, and the instance containing pixels greater than the pixel threshold is taken as the final instance. The average probability of the corresponding probabilities of all undirected edges in the final instance is calculated. The average probability represents the probability that the final instance is an instance, and whether it is an instance is determined based on the comparison between the probability and the probability threshold, thereby distinguishing each instance and obtaining an instance segmentation map that is independent of the category.
[0040] Step 104: fuse the instance segmentation map with the semantic segmentation map to obtain an instance segmentation result with categories.
[0041] The semantic segmentation map distinguishes instances of different categories, while the instance segmentation map, which is independent of the category, distinguishes individual instances. Therefore, after aligning and fusing the instance segmentation map with the semantic segmentation map, the final instance segmentation result with categories is obtained.
[0042] The image segmentation method provided in this embodiment performs semantic segmentation on the image to be segmented through a semantic segmentation network to obtain a semantic segmentation map, processes the image to be segmented through an affinity segmentation network to obtain an affinity map, constructs an instance segmentation map independent of the category based on the affinity map and the affinity kernel, and fuses the instance segmentation map with the semantic segmentation map to obtain an instance segmentation result with the category. Since the affinity kernel can determine the affinity between the current pixel and the affinity neighbor pixels in the affinity kernel, overlapping instances can be well distinguished, and each part of the same instance with more components can be fused together, thereby improving the accuracy of instance segmentation.
[0043] Based on the above technical solution, the method also includes: determining the radius of the affinity core and the interval of affinity neighbor pixels based on the instance segmentation data set; determining the number of affinity neighbor pixels according to the radius of the affinity core and the interval of affinity neighbor pixels.
[0044] When the shape of the affinity kernel is semicircular, the larger the radius of the affinity kernel, the smaller the interval between the affinity neighbor pixels, and the more affinity neighbor pixels the affinity kernel will have, which will greatly improve the instance segmentation effect, but will cause a large amount of calculation. In order to find the optimal balance between segmentation effect and operating efficiency, the optimal value of the affinity kernel radius and the optimal value of the affinity neighbor pixel interval can be determined based on the instance segmentation data set, so that the affinity kernel can well determine the pixels belonging to the same instance, and can distinguish the overlapping parts, that is, it can distinguish the parts of an instance occluded by other instances. The optimal value of the affinity kernel radius is used as the radius of the affinity kernel, and the optimal value of the affinity neighbor pixel interval is used as the interval of the affinity neighbor pixels.
[0045] In an optional embodiment, the method of determining the radius of the affinity kernel and the interval of affinity neighbor pixels based on the instance segmentation dataset includes: initializing the radius of the affinity kernel to the maximum side length of the sample image in the instance segmentation dataset, and initializing the interval of affinity neighbor pixels to 1; gradually reducing the radius of the affinity kernel, searching for the minimum radius that satisfies the requirement that any pixels of the same instance are connected by the affinity kernel based on the instance segmentation dataset, and using the minimum radius as the radius of the affinity kernel; gradually expanding the interval, searching for the maximum interval that satisfies the requirement that any pixels of the same instance are connected by the affinity kernel based on the instance segmentation dataset, and using the maximum interval as the interval of affinity neighbor pixels.
[0046] When determining the radius of the affinity kernel and the interval of the affinity neighbor pixels, a brute force search method can be used to obtain the optimal radius and interval, that is, first initialize the radius of the affinity kernel to the maximum side length of the sample image in the instance segmentation data set, and initialize the interval of the affinity neighbor pixels to 1; then, gradually reduce the radius of the affinity kernel from the maximum side length, that is, reduce the first preset value each time, for example, the first preset value can be 1, and find the minimum radius that satisfies any pixel of the same instance can be connected by the affinity kernel based on the instance segmentation data set, and use the minimum radius as the radius of the affinity kernel; then, gradually expand the interval of the affinity neighbor pixels, that is, expand the second preset value each time, for example, the second preset value can be 1, and find the maximum interval that satisfies any pixel of the same instance can be connected by the affinity kernel based on the instance segmentation data set, and use the maximum interval as the interval of the affinity neighbor pixels. In this way, a more suitable radius and interval can be found, which can further improve the accuracy of instance segmentation.
[0047] In another optional embodiment, the method of determining the radius of the affinity core and the interval of affinity neighbor pixels based on the instance segmentation dataset includes: traversing each instance in the instance segmentation dataset, respectively calculating the average values of the maximum circumscribed circle radius and the minimum inscribed circle radius of each connected domain of each instance to obtain a set of average values; determining the interval of affinity neighbor pixels based on the set of average values; for each instance in the instance segmentation dataset, determining the maximum length of any connected domain in the instance connecting other connected domains in the instance to obtain a set of maximum lengths; and determining the radius of the affinity core based on the set of maximum lengths.
[0048] The step of determining the interval of the close neighbor pixels according to the set of average values includes: determining an average value satisfying a first preset condition from the set of average values, and determining the interval of the close neighbor pixels according to the average value.
[0049] Initialize the average value set to be empty, traverse each instance in the instance segmentation data set, determine the average value of the maximum circumscribed circle radius and the minimum inscribed circle radius of each connected domain of the current instance for the current traversed instance, add the average value to the average value set, and after traversing each instance in the instance segmentation data set, obtain the average value set based on the instance segmentation data set, obtain the average value that meets the first preset condition from the average value set, and determine the interval of the affinity neighbor pixel according to the average value that meets the preset condition. The first preset condition may be the smallest average value in the average value set. In order to avoid the smallest average value being 0, each average value in the average value set may be sorted from small to large. After sorting, each average value corresponds to a sequence number, determine the average value corresponding to the first preset ratio of the largest sequence number, and use the second preset ratio of the average value as the average value that meets the first preset condition. The first preset ratio may be a smaller ratio, such as 0.2, and the second preset ratio may be less than 1, such as 0.8. The average value that meets the first preset condition may not be 0, so that a more accurate interval can be obtained. After obtaining the average value that meets the first preset condition, the square root of 2 times the average value can be used as the interval of the close neighbor pixels, so that all circles with the interval as the radius can be guaranteed to seamlessly cover all areas of the same instance.
[0050] The step of determining the radius of the affinity core according to the set of maximum length values includes: determining the maximum length value that meets a second preset condition from the set of maximum length values, and using the maximum length value as the radius of the affinity core.
[0051] Initialize the maximum length set to be empty. For each instance in the instance segmentation data set, determine the length of each connected domain in the number connecting other connected domains, and determine the maximum value of these lengths as the maximum length. Add the maximum length to the maximum length set. For each instance in the instance segmentation data set, a maximum length is obtained. Thus, the obtained maximum length set includes the same number of maximum length sets as the number of instances. Determine the maximum length that meets the second preset condition from the maximum length set, and use the maximum length as the radius of the affinity core. The second preset condition may be the largest maximum length in the maximum length set. However, in order to avoid the largest maximum length being infinite, each maximum length in the maximum length set may be sorted from small to large. After sorting, each maximum length corresponds to a sequence number. Determine the maximum length corresponding to the third preset ratio of the largest sequence number, and use the fourth preset ratio of the maximum length as the maximum length that meets the second preset condition. The third preset ratio may be a larger ratio, such as 0.8. The fourth preset ratio may be greater than 1, such as 1.2. The maximum length that meets the second preset condition obtained in this way will not be infinite, so that a more accurate affinity core radius can be obtained.
[0052] The intervals of the affinity neighbor pixels are determined by the above determined average value set, and the radius of the affinity core is determined according to the determined length maximum value set. Compared with the above brute force search method, the amount of search calculation can be reduced.
[0053] In one embodiment of the present invention, the number of the affinity neighbor pixels is determined according to the radius of the affinity core and the interval of the affinity neighbor pixels, including: taking the current pixel as the center, determining a plurality of semicircles within the radius of the affinity core with the interval of the neighbor pixels as the interval; dividing the affinity neighbor pixels on each semicircle with the interval of the neighbor pixels as the interval, to obtain the position and number of the affinity neighbor pixels.
[0054] In the affinity core, the radius starts with an initial value of 0 and gradually increases with the interval of the affinity neighbor pixels, so that multiple semicircles with the interval of the affinity neighbor pixels as the interval are obtained within the radius of the affinity core. Assuming that the interval of the affinity neighbor pixels is g, the semicircles with radii of g, 2g, 3g, ..., r are obtained within the radius r of the affinity core. The affinity neighbor pixels are divided into intervals of the affinity neighbor pixels on each semicircle, and the position and number of each affinity neighbor pixel in the affinity core are obtained. According to the position of each affinity neighbor pixel, the affinity neighbor pixel set is generated. The affinity neighbor pixels in the affinity core obtained in this way are evenly distributed in the affinity core, which can well connect different connected domains belonging to the same instance, and have a better segmentation effect in instances with more overlaps. Figure 2is a schematic diagram of the affinity core in the embodiment of the present invention, such as Figure 2 As shown, the black dot represents the center point of the affinity core, that is, the current pixel, and the white dot represents the affinity neighbor pixels, and the affinity neighbor pixels are evenly distributed in the affinity core.
[0055] Before training the semantic segmentation network and the affinity segmentation network, any sample image annotated with an instance segmentation mask in the instance segmentation dataset can be used to calculate the affinity neighbor pixels on the affinity kernel pixel by pixel according to the affinity kernel, and the pixels belonging to the same instance as the center point of the affinity kernel are marked as 1, and the pixels belonging to different instances from the center point of the affinity kernel are marked as 0, thereby forming an annotated affinity map. When training the instance segmentation network and the semantic segmentation network, the annotated affinity map is used to guide the learning of the network. At this time, the OHEM loss of the instance segmentation network can be determined, and the network parameters of the instance segmentation network can be updated based on the OHEM loss.
[0056] On the basis of the above technical solution, before the semantic segmentation of the image to be segmented is performed by the semantic segmentation network, it also includes: extracting image features of the image to be segmented by a backbone network;
[0057] The method of performing semantic segmentation on the image to be segmented by using a semantic segmentation network to obtain a semantic segmentation map includes: performing semantic segmentation on the image features by using the semantic segmentation network to obtain a semantic segmentation map;
[0058] The step of processing the image to be segmented by using an affinity segmentation network to obtain an affinity map includes: processing the image features by using the affinity segmentation network to obtain an affinity map.
[0059] Before performing semantic segmentation and affinity segmentation on the image to be segmented, the image features of the image to be segmented can be extracted through the backbone network, and then the image features are respectively input into the semantic segmentation network and the affinity segmentation network, and the image features are semantically segmented through the semantic segmentation network to obtain a semantic segmentation map, and the image features are processed through the affinity segmentation network to obtain an affinity map. The image features are first extracted through the backbone network, and then semantic segmentation and affinity segmentation are performed, which can improve the accuracy of semantic segmentation and affinity segmentation.
[0060] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0061] Figure 3 is a structural block diagram of an image segmentation device provided by an embodiment of the present invention, such as Figure 3 As shown, the image segmentation device may include:
[0062] A semantic segmentation module 301 is used to perform semantic segmentation on the image to be segmented through a semantic segmentation network to obtain a semantic segmentation map;
[0063] An affinity segmentation module 302 is used to process the image to be segmented through an affinity segmentation network to obtain an affinity graph, wherein the affinity graph represents the probability that the current pixel and the affinity neighbor pixel belong to the same instance, and the positional relationship between the current pixel and the affinity neighbor pixel is determined by an affinity kernel;
[0064] A graph merging module 303, configured to construct a category-independent instance segmentation graph based on the affinity graph and the affinity kernel;
[0065] The fusion module 304 is used to fuse the instance segmentation map with the semantic segmentation map to obtain an instance segmentation result with categories.
[0066] Optionally, the shape of the affinity kernel is non-centrosymmetric, the number of channels of the affinity map is the same as the number of affinity neighbor pixels, and the number of affinity neighbor pixels is determined according to an instance segmentation dataset.
[0067] Optionally, the affinity core is semicircular in shape.
[0068] Optionally, the device further comprises:
[0069] A radius and interval determination module, used to determine the radius of the affinity kernel and the interval of affinity neighbor pixels based on the instance segmentation data set;
[0070] The neighbor point quantity determination module is used to determine the quantity of the affinity neighbor pixels according to the radius of the affinity core and the interval between the affinity neighbor pixels.
[0071] Optionally, the radius and interval determination module includes:
[0072] An initialization unit, used to initialize the radius of the affinity kernel to the maximum side length of the sample image in the instance segmentation dataset, and to initialize the interval of the affinity neighbor pixels to 1;
[0073] A radius search unit, used to gradually reduce the radius of the affinity core, find a minimum radius that satisfies the requirement that any pixels of the same instance are connected through the affinity core based on the instance segmentation data set, and use the minimum radius as the radius of the affinity core;
[0074] The interval search unit is used to gradually expand the interval, find the maximum interval that satisfies any pixels of the same instance connected by the affinity kernel based on the instance segmentation data set, and use the maximum interval as the interval of the affinity neighbor pixels.
[0075] Optionally, the radius and interval determination module includes:
[0076] an average value set determination unit, used for traversing each instance in the instance segmentation data set, respectively calculating the average values of the maximum circumscribed circle radius and the minimum inscribed circle radius of each connected domain of each instance, and obtaining an average value set;
[0077] An interval determination unit, configured to determine the interval of the close neighbor pixels according to the set of average values;
[0078] A length set determination unit, configured to determine, for each instance in the instance segmentation data set, a maximum length of any connected domain in the instance connecting other connected domains in the instance, to obtain a maximum length set;
[0079] The radius determination unit is used to determine the radius of the affinity core according to the maximum length set.
[0080] Optionally, the interval determination unit is specifically used to:
[0081] An average value satisfying a first preset condition is determined from the average value set, and an interval between the affinity neighbor pixels is determined according to the average value.
[0082] Optionally, the radius determination unit is specifically used to:
[0083] The maximum length value that meets the second preset condition is determined from the maximum length set, and the maximum length value is used as the radius of the affinity core.
[0084] Optionally, the neighbor point quantity determination module includes:
[0085] A plurality of semicircle determination units, used to determine a plurality of semicircles with the current pixel as the center and the intervals of the neighboring pixels as the intervals within the radius of the affinity core;
[0086] The neighbor position and quantity determining unit is used to divide the close neighbor pixels on each semicircle at intervals of the neighbor pixels to obtain the positions and quantities of the close neighbor pixels.
[0087] Optionally, the graph merging module includes:
[0088] an undirected graph construction module, for constructing the affinity graph into an undirected graph according to the affinity core;
[0089] The graph merging unit is used to merge the undirected graphs to obtain an instance segmentation graph that is independent of the category.
[0090] Optionally, the device further comprises:
[0091] An image feature extraction module, used to extract image features of the image to be segmented through a backbone network;
[0092] The semantic segmentation module is specifically used for:
[0093] Performing semantic segmentation on the image features through the semantic segmentation network to obtain a semantic segmentation map;
[0094] The affinity segmentation module is specifically used for:
[0095] The image features are processed by the affinity segmentation network to obtain an affinity graph.
[0096] The image segmentation device provided by the present embodiment performs semantic segmentation on the image to be segmented through a semantic segmentation network by a semantic segmentation module to obtain a semantic segmentation map; the affinity segmentation module processes the image to be segmented through an affinity segmentation network to obtain an affinity map; the map merging module constructs an instance segmentation map independent of the category according to the affinity map and the affinity kernel; the fusion module fuses the instance segmentation map with the semantic segmentation map to obtain an instance segmentation result with a category; since the affinity between the current pixel and the affinity neighbor pixel in the affinity kernel can be determined by the affinity kernel, overlapping instances can be well distinguished, and each part of the same instance with multiple components can be fused together, thereby improving the accuracy of instance segmentation.
[0097] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0098] Furthermore, according to an embodiment of the present invention, an electronic device is provided, which may be a computer, a mobile terminal or other electronic device, and includes: a processor, a memory, and a computer program stored in the memory and executable on the processor, and the computer program implements the image segmentation method of the aforementioned embodiment when executed by the processor.
[0099] According to one embodiment of the present invention, a computer-readable storage medium is also provided, and the computer-readable storage medium includes but is not limited to a disk storage, a CD-ROM, an optical storage, etc., and a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the image segmentation method of the aforementioned embodiment is implemented.
[0100] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0101] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0102] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0105] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0106] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0107] The above is a detailed introduction to an image segmentation method, device, electronic device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for a person skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. An image segmentation method, characterized in that: include: Perform semantic segmentation on the image to be segmented through the semantic segmentation network to obtain a semantic segmentation map; Processing the image to be segmented by an affinity segmentation network to obtain an affinity graph, wherein the affinity graph represents the probability that the current pixel and the affinity neighbor pixel belong to the same instance, and the positional relationship between the current pixel and the affinity neighbor pixel is determined by an affinity kernel; constructing a category-independent instance segmentation map according to the affinity map and the affinity kernel; The instance segmentation map is merged with the semantic segmentation map to obtain an instance segmentation result with categories; Based on the instance segmentation dataset, determining the radius of the affinity kernel and the interval of affinity neighbor pixels; Determining the number of affinity neighbor pixels according to the radius of the affinity core and the interval between affinity neighbor pixels; The step of determining the radius of the affinity kernel and the interval between affinity neighbor pixels based on the instance segmentation dataset includes: Initialize the radius of the affinity kernel to the maximum side length of the sample image in the instance segmentation dataset, and initialize the interval of affinity neighbor pixels to 1; Gradually reducing the radius of the affinity core, searching for a minimum radius that satisfies the requirement that any pixels of the same instance are connected through the affinity core based on the instance segmentation data set, and using the minimum radius as the radius of the affinity core; The interval is gradually enlarged, and a maximum interval satisfying that any pixels of the same instance are connected by an affinity kernel is found based on the instance segmentation data set, and the maximum interval is used as the interval of the affinity neighbor pixels.
2. The method according to claim 1, characterized in that The shape of the affinity kernel is non-centrosymmetric, the number of channels of the affinity map is the same as the number of the affinity neighbor pixels, and the number of the affinity neighbor pixels is determined according to the instance segmentation dataset.
3. The method according to claim 2, characterized in that The affinity core is in the shape of a semicircle.
4. The method according to claim 3, characterized in that The step of determining the radius of the affinity kernel and the interval between affinity neighbor pixels based on the instance segmentation dataset includes: Traversing each instance in the instance segmentation data set, respectively calculating the average values of the maximum circumscribed circle radius and the minimum inscribed circle radius of each connected domain of each instance, and obtaining an average value set; Determining the interval of the close neighbor pixels according to the set of average values; For each instance in the instance segmentation data set, determining the maximum length of any connected domain in the instance connecting other connected domains in the instance, and obtaining a maximum length set; The radius of the affinity core is determined according to the maximum length set.
5. The method according to claim 4, characterized in that Determining the interval of the close neighbor pixels according to the set of average values includes: An average value satisfying a first preset condition is determined from the average value set, and an interval between the affinity neighbor pixels is determined according to the average value.
6. The method according to claim 4, characterized in that Determining the radius of the affinity core according to the maximum length set includes: The maximum length value that meets the second preset condition is determined from the maximum length set, and the maximum length value is used as the radius of the affinity core.
7. The method according to claim 3, characterized in that Determining the number of affinity neighbor pixels according to the radius of the affinity core and the interval between affinity neighbor pixels includes: Taking the current pixel as the center, determining a plurality of semicircles within the radius of the affinity core and spaced apart by the intervals of the neighboring pixels; On each semicircle, the close neighbor pixels are divided at intervals of the neighbor pixels to obtain the positions and numbers of the close neighbor pixels.
8. The method according to any one of claims 1 to 7, characterized in that: The step of constructing a category-independent instance segmentation map according to the affinity map and the affinity kernel comprises: constructing the affinity graph as an undirected graph according to the affinity core; Graph merging is performed on the undirected graph to obtain an instance segmentation graph that is independent of the category.
9. The method according to any one of claims 1 to 7, characterized in that: Before performing semantic segmentation on the image to be segmented by the semantic segmentation network, the method further includes: Extracting image features of the image to be segmented through a backbone network; The semantic segmentation of the image to be segmented is performed through the semantic segmentation network to obtain a semantic segmentation map, including: Performing semantic segmentation on the image features through the semantic segmentation network to obtain a semantic segmentation map; The image to be segmented is processed by the affinity segmentation network to obtain an affinity graph, including: The image features are processed by the affinity segmentation network to obtain an affinity graph.
10. An image segmentation device, characterized in that: include: A semantic segmentation module, which is used to perform semantic segmentation on the image to be segmented through a semantic segmentation network to obtain a semantic segmentation map; An affinity segmentation module, used for processing the image to be segmented through an instance segmentation network to obtain an affinity graph, wherein the affinity graph represents the probability that the current pixel and the affinity neighbor pixel belong to the same instance, and the positional relationship between the current pixel and the affinity neighbor pixel is determined by an affinity kernel; A graph merging module, configured to construct a category-independent instance segmentation graph based on the affinity graph and the affinity kernel; A fusion module, used to fuse the instance segmentation map with the semantic segmentation map to obtain an instance segmentation result with categories; A radius and interval determination module, used to determine the radius of the affinity kernel and the interval of affinity neighbor pixels based on the instance segmentation data set; A neighbor point quantity determination module, used to determine the number of affinity neighbor pixels according to the radius of the affinity core and the interval between affinity neighbor pixels; The radius and interval determination module comprises: An initialization unit, used to initialize the radius of the affinity kernel to the maximum side length of the sample image in the instance segmentation dataset, and to initialize the interval of the affinity neighbor pixels to 1; A radius search unit, used to gradually reduce the radius of the affinity core, find a minimum radius that satisfies the requirement that any pixels of the same instance are connected through the affinity core based on the instance segmentation data set, and use the minimum radius as the radius of the affinity core; The interval search unit is used to gradually expand the interval, find the maximum interval that satisfies any pixels of the same instance connected by the affinity kernel based on the instance segmentation data set, and use the maximum interval as the interval of the affinity neighbor pixels.
11. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image segmentation method according to any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image segmentation method according to any one of claims 1 to 9 is implemented.