Instance segmentation method, device and storage medium based on point supervision
By dividing the instance attention map of an instance object into multiple component attention maps, and using key point offsets and feature density updates, the problem of low segmentation accuracy caused by semantic differences between object components is solved, and higher segmentation accuracy is achieved and performance.
Patent Information
- Application Number
- CN202310379811.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-31
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-03-31
AI Technical Summary
Existing point-supervised instance segmentation methods are poor in handling semantic differences between object components, resulting in low segmentation accuracy.
By dividing the instance attention map of an instance object into multiple component attention maps, using key point offsets and feature density updates, internal feature correlations are improved for more accurate segmentation.
It significantly improves the accuracy and performance of instance segmentation, reduces the similarity between components, and improves the effectiveness of point-supervised instance segmentation method.
Smart Images

Figure CN116503595B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision, and in particular to a method, device and storage medium for instance segmentation based on point supervision. Background Art
[0002] Instance segmentation is one of the key issues in the field of image understanding and computer vision. It is a task to predict the category of a single pixel in an image and provide different instance labels for separate instances belonging to the same class. Currently, instance segmentation technology has been widely used in image processing, human-computer interaction, and assisted driving. However, Figure 1 As shown in the figure, instance segmentation based on full supervision requires detailed pixel-level annotation of each object in the image, which brings a huge burden to manual annotation, especially in the era of big data. Instance segmentation based on point supervision is an instance segmentation method with a very efficient annotation process. During the training process, it only needs to use a point and category label inside the instance object for supervision to achieve complete segmentation of the entire instance object. Instance segmentation based on point supervision can reduce the object annotation cost to 20% of full supervision, while keeping the annotation cost similar to that of weakly supervised instance segmentation, and achieve performance far exceeding the latter.
[0003] At present, the work in the field of point supervised instance segmentation mainly includes BESTIE and WISE-Net algorithms. They first estimate a pseudo label for each object, and then refine this pseudo label with a segmentation network. BESTIE uses point supervision to convert the results of weakly supervised semantic segmentation into instance segmentation information to obtain fine instance segmentation pseudo labels. WISE-NET locates the area of each object by calculating the similarity between foreground areas and realizes object segmentation.
[0004] However, in the above method, a pseudo label of an object is directly generated from a single point. This method does not take into account the potential semantic differences between the components of an object. For example, "dog tail" and "dog head" are both components of a dog, but the semantics of these components may be quite different. If only one supervisory point is used, the above method may only locate the most discriminative area of the object, or the area around the supervisory point, resulting in poor segmentation performance. Summary of the invention
[0005] In view of this, a method, device and storage medium for instance segmentation based on point supervision are proposed. The embodiment of the present application effectively improves the internal feature correlation of the required segmentation target by dividing the instance attention map corresponding to the instance object into multiple component attention maps, thereby making the segmentation accuracy more accurate.
[0006] In a first aspect, an embodiment of the present application provides an instance segmentation method based on point supervision, the method comprising:
[0007] Acquire a target image, wherein the target image includes n instance objects, where n is a positive integer;
[0008] Determine an instance attention map corresponding to each of the n instance objects;
[0009] For at least one of the instance objects, dividing the instance attention map corresponding to the instance object into a plurality of component attention maps, wherein the plurality of component attention maps indicate a maximum similarity between an image feature of the instance attention map and key point features of each semantic component of the instance object;
[0010] Determine a segmentation result of the instance object according to the multiple component attention maps corresponding to at least one of the instance objects.
[0011] In this implementation, a target image is obtained, where the target image includes n instance objects, where n is a positive integer; instance attention maps corresponding to each of the n instance objects are determined; for at least one instance object, the instance attention map corresponding to the instance object is divided into multiple component attention maps, where the multiple component attention maps indicate the maximum similarity between the image features of the instance attention map and the key point features of each semantic component of the instance object; a segmentation result of the instance object is determined based on the multiple component attention maps corresponding to the at least one instance object; since there is a non-negligible semantic deviation between the semantic components inside the instance object, it is difficult to estimate the complete area of the object based on one point, and the segmentation result of the instance object is obtained by dividing the instance attention map corresponding to the instance object into multiple component attention maps, thereby effectively improving the internal feature correlation of the required segmentation target, making the instance segmentation accuracy more accurate, and improving the instance segmentation performance based on point supervision.
[0012] In a possible implementation, determining the instance attention maps corresponding to the n instance objects respectively includes:
[0013] For at least one of the instance objects, the instance attention map corresponding to the instance object is determined based on an image block token and a preset reference token, wherein the image block token indicates an image feature of an image block of the target image, and the instance attention map indicates a similarity between the image block token and the reference token.
[0014] In this implementation, for at least one instance object, an instance attention map corresponding to the instance object is determined based on an image block token and a preset reference token, the image block token indicates the image features of the image block of the target image, and the instance attention map indicates the similarity between the image block token and the reference token; by determining the instance attention map corresponding to the instance object based on the image block token and the preset reference token, the accuracy of the determined instance attention map is guaranteed.
[0015] In another possible implementation, determining the instance attention map corresponding to the instance object according to the image block token and the preset reference token includes:
[0016] Determining instance features and background features corresponding to the instance object according to the correlation between the image block token and the preset reference token;
[0017] The instance attention map corresponding to the instance object is determined according to the instance feature, the background feature, the instance threshold and the background threshold.
[0018] In this implementation, the instance features and background features corresponding to the instance object are determined based on the correlation between the image block token and the preset reference token; the instance attention map corresponding to the instance object is determined based on the instance features, background features, instance threshold and background threshold, further ensuring the accuracy of the determined instance attention map.
[0019] In another possible implementation manner, dividing the instance attention map corresponding to the instance object into a plurality of component attention maps includes:
[0020] Randomly initialize a plurality of key points in the instance attention map corresponding to the instance object;
[0021] According to the multiple key points, the instance attention map is divided into the multiple component attention maps, and the multiple component attention maps correspond to the multiple key points one by one;
[0022] Update the position of the corresponding key point according to the feature density of each component attention map;
[0023] Re-dividing the plurality of component attention maps according to the updated plurality of key points;
[0024] The steps of updating the positions of the key points corresponding to the feature densities of the component attention maps are performed in an iterative manner, and re-dividing the multiple component attention maps according to the updated multiple key points, and obtaining the re-divided multiple component attention maps after the iteration is completed.
[0025] In this implementation, multiple key points are randomly initialized in the instance attention map corresponding to the instance object; according to the multiple key points, the instance attention map is divided into multiple component attention maps, and the multiple component attention maps correspond to the multiple key points one by one; the position of the corresponding key point is updated according to the feature density of each component attention map; the multiple component attention maps are re-divided according to the updated multiple key points; the next two steps are iterated, and after the iteration is completed, the re-divided multiple component attention maps are obtained, that is, the instance attention map is decomposed into multiple component attention maps, these component attention maps are represented by key points, and the fine-grained semantics are estimated through key point offsets, so that component cutting is more accurate.
[0026] In another possible implementation, updating the position of the corresponding key point according to the feature density of each component attention map includes:
[0027] For each of the component attention maps, determining a feature density weight corresponding to each of a plurality of image block tokens of the component attention map, wherein the feature density weight indicates a feature density of the image block token;
[0028] According to the feature density weights corresponding to each of the multiple image block tokens, the positions of the key points corresponding to the component attention map are updated.
[0029] In this implementation, for each component attention map, the feature density weights corresponding to each of the multiple image block tokens of the component attention map are determined, and the feature density weights indicate the feature density of the image block tokens; according to the feature density weights corresponding to each of the multiple image block tokens, the positions of the key points corresponding to the component attention map are updated; that is, by performing key point offsets in the feature space to find stable extreme points to represent the components of the instance object, the accuracy of component cutting is further guaranteed.
[0030] In another possible implementation, the key points are independently shifted to the mean of the feature vectors of the corresponding semantic components.
[0031] In this implementation, key points are independently offset to the mean of the feature vectors of the corresponding semantic components, so that the key points can better represent the components of the instance object, thereby making the segmentation accuracy more accurate.
[0032] In another possible implementation manner, before dividing the instance attention map corresponding to the instance object into a plurality of component attention maps, the method further includes:
[0033] Removing at least one key point in a target area of the instance attention map, wherein the target area is an area whose confidence is less than a confidence threshold; and / or,
[0034] Merge multiple key points whose similarity is higher than the similarity threshold.
[0035] In this implementation, before dividing the instance attention map corresponding to the instance object into multiple component attention maps, at least one key point in the target area of the instance attention map is removed, and the target area is an area with a confidence less than a confidence threshold; and / or, multiple key points with a similarity higher than a similarity threshold are merged; that is, noisy key points are filtered, and / or similar key points are merged, which solves the problem of key points being erroneously offset to the background area or other object area due to the existence of key point offset, and further ensures the accuracy of subsequent segmentation precision.
[0036] In a second aspect, an embodiment of the present application provides an instance segmentation device based on point supervision, the device comprising:
[0037] An acquisition unit, configured to acquire a target image, wherein the target image includes n instance objects, where n is a positive integer;
[0038] A first determining unit, configured to determine an instance attention map corresponding to each of the n instance objects;
[0039] a dividing unit, configured to divide, for at least one of the instance objects, the instance attention map corresponding to the instance object into a plurality of component attention maps, wherein the plurality of component attention maps indicate a maximum similarity between an image feature of the instance attention map and key point features of each semantic component of the instance object;
[0040] The second determining unit is used to determine the segmentation result of the instance object according to the multiple component attention maps corresponding to at least one of the instance objects.
[0041] In a possible implementation manner, the first determining unit is further configured to:
[0042] For at least one of the instance objects, the instance attention map corresponding to the instance object is determined based on an image block token and a preset reference token, wherein the image block token indicates an image feature of an image block of the target image, and the instance attention map indicates a similarity between the image block token and the reference token.
[0043] In another possible implementation manner, the first determining unit is further configured to:
[0044] Determining instance features and background features corresponding to the instance object according to the correlation between the image block token and the preset reference token;
[0045] The instance attention map corresponding to the instance object is determined according to the instance feature, the background feature, the instance threshold and the background threshold.
[0046] In another possible implementation, the dividing unit is further configured to:
[0047] Randomly initialize a plurality of key points in the instance attention map corresponding to the instance object;
[0048] According to the multiple key points, the instance attention map is divided into the multiple component attention maps, and the multiple component attention maps correspond to the multiple key points one by one;
[0049] Update the position of the corresponding key point according to the feature density of each component attention map;
[0050] Re-dividing the plurality of component attention maps according to the updated plurality of key points;
[0051] The steps of updating the positions of the key points corresponding to the feature densities of the component attention maps are performed in an iterative manner, and re-dividing the multiple component attention maps according to the updated multiple key points, and obtaining the re-divided multiple component attention maps after the iteration is completed.
[0052] In another possible implementation, the dividing unit is further configured to:
[0053] For each of the component attention maps, determining a feature density weight corresponding to each of a plurality of image block tokens of the component attention map, wherein the feature density weight indicates a feature density of the image block token;
[0054] According to the feature density weights corresponding to each of the multiple image block tokens, the positions of the key points corresponding to the component attention map are updated.
[0055] In another possible implementation, the key points are independently shifted to the mean of the feature vectors of the corresponding semantic components.
[0056] In another possible implementation, the device further includes: a key point filtering unit; the key point filtering unit is configured to:
[0057] Removing at least one key point in a target area of the instance attention map, wherein the target area is an area whose confidence is less than a confidence threshold; and / or,
[0058] Merge multiple key points whose similarity is higher than the similarity threshold.
[0059] In a third aspect, an embodiment of the present application provides an instance segmentation device based on point supervision, the device comprising:
[0060] processor;
[0061] a memory for storing processor-executable instructions;
[0062] The processor is configured to implement the method provided by the first aspect or any possible implementation manner of the first aspect when executing the instruction.
[0063] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method provided by the first aspect or any possible implementation of the first aspect.
[0064] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a computing device, the processor in the computing device executes the method provided in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present application and, together with the description, serve to explain the principles of the present application.
[0066] Figure 1 A schematic diagram showing the difference between a fully supervised instance segmentation algorithm and a point-supervised instance segmentation algorithm provided in the related art.
[0067] Figure 2 A schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present application is shown.
[0068] Figure 3 A flowchart of an instance segmentation method based on point supervision provided by an exemplary embodiment of the present application is shown.
[0069] Figure 4 A schematic diagram showing the principles of an instance segmentation method based on point supervision provided by another exemplary embodiment of the present application is shown.
[0070] Figure 5 A block diagram of an instance segmentation device based on point supervision provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION
[0071] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0072] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0073] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.
[0074] First, the application scenarios involved in this application are introduced.
[0075] The present application embodiment provides an instance segmentation method based on point supervision, and the execution subject is a computing device. Figure 2 , which shows a schematic diagram of the structure of a computing device provided by an exemplary embodiment of the present application.
[0076] The computing device may be a terminal or a server. The terminal includes a mobile terminal or a fixed terminal, such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc. The server may be a server, a server cluster consisting of several servers, or a cloud computing service center.
[0077] like Figure 2 As shown, the computing device includes a processor 10, a memory 20, and a communication interface 30. Those skilled in the art will appreciate that Figure 2 The structure shown in the figure does not constitute a limitation on the computing device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0078] The processor 10 is the control center of the computing device, and uses various interfaces and lines to connect various parts of the entire computing device. It executes various functions of the computing device and processes data by running or executing software programs and / or modules stored in the memory 20, and calling data stored in the memory 20, thereby controlling the computing device as a whole. The processor 10 can be implemented by a CPU or a graphics processing unit (GPU).
[0079] The memory 20 can be used to store software programs and modules. The processor 10 executes various functional applications and data processing by running the software programs and modules stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system 21, an acquisition unit 22, a first determination unit 23, a division unit 24, a second determination unit 25 and an application required for at least one function, etc.; the data storage area may store data created according to the use of the computing device, etc. The memory 20 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk. Accordingly, the memory 20 may further include a memory controller to provide the processor 10 with access to the memory 20 .
[0080] Among them, the processor 10 performs the following functions by running the acquisition unit 22: acquiring a target image, the target image includes n instance objects, n is a positive integer; the processor 10 performs the following functions by running the first determination unit 23: determining the instance attention maps corresponding to each of the n instance objects; the processor 10 performs the following functions by running the division unit 24: for at least one instance object, dividing the instance attention map corresponding to the instance object into multiple component attention maps, the multiple component attention maps indicating the maximum similarity between the image features of the instance attention map and the key point features of each semantic component of the instance object; the processor 10 performs the following functions by running the second determination unit 25: determining the segmentation result of the instance object according to the multiple component attention maps corresponding to the at least one instance object.
[0081] The embodiment of the present application includes two parts: (1) Token Querying, which divides the instance attention map corresponding to a single instance object into multiple component attention maps, and preliminarily cuts the single instance object into multiple components based on the attention maps; (2) Key-point Shift, which re-estimates the attention maps of each component and optimizes the component cutting results.
[0082] Below, an exemplary embodiment is used to introduce the instance segmentation method based on point supervision provided in the embodiment of the present application.
[0083] Please refer to Figure 3 , which shows a flowchart of an instance segmentation method based on point supervision provided by an exemplary embodiment of the present application. In this embodiment, the method is used for Figure 2 The method is illustrated by using the computing device shown in FIG.
[0084] Step 301: Acquire a target image, where the target image includes n instance objects, where n is a positive integer.
[0085] The computing device obtains an input target image, where the target image is an image including n instance objects, where n is a positive integer.
[0086] Step 302: determine the instance attention map corresponding to each of the n instance objects.
[0087] Optionally, in order to generate instance attention maps, a learnable reference token is introduced and a point prediction task is defined. The computing device determines instance attention maps and a background attention map corresponding to each of the n instance objects based on the reference token.
[0088] The computing device obtains the input target image and divides the target image into W×H image blocks, each of which corresponds to an image token, which indicates the image feature of the image block, where W and H are both positive integers. A component area in an instance object usually contains multiple image blocks.
[0089] Multiple image tokens are concatenated with reference tokens and then input into the backbone model to extract features. After that, the features of the reference tokens are input into two multi-layer perception models for prediction of point coordinates and categories. Point supervision uses binary matching loss to drive the prediction of point coordinates and categories. Among them, the backbone model is based on the image attention network (Vision transformer, ViT) model, and the multi-layer perception model is based on the multi-layer perceptron network (Multi-layer perception, MLP) model.
[0090] It should be noted that a point supervision can only be matched with one reference token. The reference token matched with the point supervision can perceive the instance object, that is, the reference token can be used to generate an instance attention map for each instance object, and the unmatched reference token will be regarded as the background, that is, each instance object in the target image is the foreground, and the rest of the target image except the foreground is the background. Schematically, x reference tokens are preset (x>n), where x reference tokens can be matched with n instance attention maps in the target image. Figure 1Once a match is made, the remaining reference tokens are matched with the background image.
[0091] Optionally, for at least one instance object, the computing device determines an instance attention map corresponding to the instance object based on the image block token and a preset reference token, where the image block token indicates image features of the image block of the target image, and the instance attention map indicates the similarity between the image block token and the reference token.
[0092] Optionally, the computing device determines the instance features and background features corresponding to the instance object based on the correlation between the image block token and the preset reference token; and determines the instance attention map corresponding to the instance object based on the instance features, background features, instance threshold and background threshold.
[0093] Optionally, the computing device obtains the instance region and the background region according to the correlation between the reference token and the image block token, and calculates each instance feature and background feature. The instance region and the background region are regions whose confidence is greater than the confidence threshold, the instance region is also called the foreground region, and the instance feature is also called the foreground feature. Each image block token calculates its similarity distribution with the instance feature and the background feature, respectively, and obtains the instance attention map of each instance object by jointly estimating the distribution of the two.
[0094] Schematically, in order to fully activate the object, a corresponding self-attention map is generated for each instance object based on the reference token in, Represents a single image patch token φ i,j The similarity between the matched benchmark token and the benchmark token is expressed as: Where D represents the dimension of the feature.
[0095] Considering A S is learned under the supervision of one point, so A S It suffers from semantic bias problems, which results in partial activation and background noise. Therefore, it is only used as a guide to divide the image patch tokens into high-confidence instance regions Φ + , high confidence background region Φ - , and ignored tokens. S Set the foreground threshold and background threshold on the graph, and use the following formula (1) to estimate the instance attention map
[0096]
[0097] Among them, d(a,b) represents the cosine similarity between feature vector a and feature vector b. as well as Respectively represent the instance area Φ +and background area Φ - The first term in the above formula (1) optimizes the estimation of the instance area, and the second term can reduce the background noise in the background area.
[0098] Step 303: for at least one instance object, divide the instance attention map corresponding to the instance object into multiple component attention maps, where the multiple component attention maps indicate the maximum similarity between the image features of the instance attention map and the key point features of each semantic component of the instance object.
[0099] Since the instance attention map is still affected by the non-negligible semantic deviation between the various parts of the object, in order to reduce the impact of semantic deviation, the instance attention map can be divided into multiple component attention maps, these components are represented by key points, and the fine-grained semantics are estimated by key point offsets.
[0100] Optionally, for at least one instance object, the computing device randomly initializes multiple key points in the instance attention map corresponding to the instance object; divides the instance attention map into multiple component attention maps based on the multiple key points, and the multiple component attention maps correspond one-to-one to the multiple key points; updates the positions of the corresponding key points based on the feature density of each component attention map; re-divides the multiple component attention maps based on the updated multiple key points; and iteratively executes the steps of updating the positions of the corresponding key points based on the feature density of each component attention map and re-dividing the multiple component attention maps based on the updated multiple key points, and obtaining the re-divided multiple component attention maps after the iteration is completed.
[0101] It should be noted that a component area usually contains multiple image blocks. A key point corresponds to a component, and a weight is calculated for each image block feature in the component area, and the position of the key point is updated using these weights. When calculating the weight, each key point is independent and does not affect each other.
[0102] Optionally, the computing device updates the position of the corresponding key point according to the feature density of each component attention map, including: for each component attention map, determining the feature density weights corresponding to each of the multiple image block tokens of the component attention map, the feature density weights indicating the feature density of the image block tokens; and updating the position of the key point corresponding to the component attention map according to the feature density weights corresponding to each of the multiple image block tokens.
[0103] Optionally, the shifted keypoints are stable extreme points in the instance attention map. Optionally, the keypoints are independently shifted to the mean of the feature vectors of the corresponding semantic components.
[0104] Schematically, the computing device converts a single instance attention map A IDivided into K component attention maps However, in the point-supervised setting, the exact object components are not provided. For complex irregular objects, it is very challenging to directly learn the object components. Assuming that the image patch token features in a sub-region share fine semantics and are therefore dense in the feature space, the stable extreme points of the feature vector can be used to represent each sub-region of the instance object.
[0105] To achieve this goal, K key points are randomly initialized in the feature space of the instance object, and they are gradually shifted to stable extreme points in the feature space through iteration. The key point is denoted as Q = {Q k ,k=1,2,···,K}, where It is the position subscript of the key point. k =(i,j) indicates that the kth key point is in the feature space and the image block feature φ i,j The distance is the shortest. The feature vector of the key point is recorded as For example, given an instance attention map A I , from A I K points are uniformly sampled from the high confidence region of as the initialization of key points. The features of these key points ψ k Initialize to Based on these initialized key points, the instance attention map A is transformed into I Partition into component attention graphs
[0106]
[0107] in, Represents the image block token φ i,j and the key point ψ k The cosine similarity between .
[0108] Since the key points are randomly initialized, they are not good enough to represent the components of the object. We can learn from the key point offset in the feature space to find stable extreme points to represent the components of the instance object.
[0109] First, the token φ of each image block is given by the following formula (3): i,j Calculate a feature density weight ω:
[0110]
[0111] Among them, β k is an adaptive bandwidth, as shown in the following formula (4), β k is defined as:
[0112]
[0113] Among them, β k The value of ψ k The density of the surrounding feature distribution, when ψ k When the surrounding features are more densely distributed, β k The value of is larger. α is a temperature coefficient. It can be seen that ω represents the feature density around the key point. By weighting ω(·), the distance from the key point ψ k Near feature φ i,j It will make a greater contribution in the process of migration. Based on the weights defined above, as shown in the following formula (5), the key point ψ k It can be expressed as:
[0114]
[0115] Among them, only when hour, The value of is 1, otherwise The value of is 0.
[0116] Finally, when the key point positions converge, the quantized position of the kth key point can be calculated according to the following formula (6):
[0117]
[0118] Among them, formula (5) enables the key points to be moved to high-density areas in the feature space. According to the updated key points, the component attention map can be updated by formula (2). By iteratively calculating formula (3) and formula (5) N times (N is a preset value, for example, N is set to 10 through experiments), each key point can be independently offset to the mean of the feature vector of each component. It should be noted that the embodiment of the present application does not limit the method of key point offset.
[0119] Since there is a problem of mistakenly shifting key points to background areas or other object areas during the key point shifting process, the computing device can perform key point filtering, that is, filtering noisy key points and / or merging similar key points. In other words, remove key points that are not located in the high confidence area of the instance attention map, and then merge highly similar key points.
[0120] Optionally, before the computing device divides the instance attention map corresponding to the instance object into multiple component attention maps, it also includes: removing at least one key point in a target area of the instance attention map, the target area is an area with a confidence level less than a confidence threshold; and / or merging multiple key points with a similarity higher than a similarity threshold.
[0121] The confidence threshold is set by default or is set by the user. Similarly, the similarity threshold is set by default or is set by the user. This embodiment of the present application does not limit this.
[0122] Optionally, after filtering the key points, the computing device performs an attention map re-estimation. That is, the computing device can estimate the position of each key point using the above formula (4). By using these key points as supervision, the features of the image can perceive the fine-grained semantics within the object range, and the instance attention map is re-estimated using the above formula (1).
[0123] Step 304: Determine a segmentation result of the instance object according to a plurality of component attention maps corresponding to at least one instance object.
[0124] Optionally, the computing device iteratively updates multiple component attention maps corresponding to each instance object, and finally synthesizes the segmentation results of each instance.
[0125] In summary, the embodiment of the present application provides an instance segmentation method based on point supervision of block estimation, which divides the instance attention map corresponding to the instance object, and obtains the attention map of each component through an iterative estimation method. This method can effectively estimate the fine-grained semantics of the object and systematically alleviate the problem of semantic deviation. In addition, the embodiment of the present application also proposes a target semantic center positioning method based on attention offset, which finds the semantic center of each component in an iterative manner, and represents this center point as the component key point, thereby achieving simple and effective optimization of the component attention map.
[0126] In an illustrative example, Figure 4 As shown, for each input image, the instance segmentation method based on point supervision provided in the embodiment of the present application is used to obtain the segmentation result of the instance object, including but not limited to the following steps: first, the image features are extracted through the backbone network ViT, and each single point supervision is converted into an instance attention map through token query; then, through the key point offset, the attention map in the feature space is re-estimated, and the key points corresponding to each component of the instance object are obtained. After obtaining the key points, the key points are used as supervision points to supervise the object mask predicted by the instance segmentation. It should be noted that the relevant details of each step can be referred to the relevant description in the above embodiment, which will not be repeated here.
[0127] The embodiments of the present application have been experimented on the public data sets Pascal VOC 2012, MSCOCO's val2017, and MSCOCO's test-dev data sets. The experiments show that the implementation performance of the embodiments of the present application significantly surpasses many current methods. On the Pascal VOC 2012 data set, as shown in Table 1, the embodiments of the present application surpass the current optimal method BESTIE by 7.7% (54.4% vs. 46.7%) in mAP50. On the MSCOCO data set, as shown in Table 2, when ViT-S is used as the backbone network, the embodiments of the present application surpass the current method by 1.4% (19.1% vs. 17.7%) in the average precision (AP) index. On the AP50 index, the method provided by the embodiments of the present application significantly surpasses method 13 by 4.8% (38.8% vs. 34.0%). On the more challenging AP75 index, the method provided by the embodiments of the present application still surpasses method 13 by 1% (17.4% vs. 16.4%). These results show that the embodiment of the present application still has a strong advantage on the more challenging dataset MSCOCO. When ViT-B is used as the backbone network, the method provided by the embodiment of the present application further achieves an AP value of 21.1%. Among them, F, I, and P respectively represent the supervision used in the training process as full supervision (mask supervision), image category supervision, and point supervision.
[0128] Table 1
[0129]
[0130]
[0131] Table 2
[0132]
[0133] In summary, the instance segmentation method based on point supervision provided by the embodiment of the present application effectively improves the internal feature correlation of the required segmentation target, thereby making the segmentation accuracy more accurate, and the result is greatly improved compared with the baseline (for example, 8.8%). In addition, the method provided by the embodiment of the present application makes component cutting more accurate, which can effectively improve the feature similarity of each pixel point inside the component and reduce the similarity between components.
[0134] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0135] Please refer to Figure 5, which shows a block diagram of an instance segmentation device based on point supervision provided by an exemplary embodiment of the present application. The device can be implemented as all or part of a computing device through software, hardware, or a combination of both. The device may include: an acquisition unit 510, a first determination unit 520, a division unit 530, and a second determination unit 540.
[0136] An acquisition unit 510 is used to acquire a target image, where the target image includes n instance objects, where n is a positive integer;
[0137] A first determining unit 520 is used to determine the instance attention map corresponding to each of the n instance objects;
[0138] A dividing unit 530 is used to divide, for at least one instance object, an instance attention map corresponding to the instance object into a plurality of component attention maps, where the plurality of component attention maps indicate a maximum similarity between an image feature of the instance attention map and key point features of each semantic component of the instance object;
[0139] The second determining unit 540 is used to determine the segmentation result of the instance object according to multiple component attention maps corresponding to at least one instance object.
[0140] In a possible implementation manner, the first determining unit 520 is further configured to:
[0141] For at least one instance object, an instance attention map corresponding to the instance object is determined based on the image block token and a preset reference token, the image block token indicates the image features of the image block of the target image, and the instance attention map indicates the similarity between the image block token and the reference token.
[0142] In another possible implementation, the first determining unit 520 is further configured to:
[0143] Determine the instance features and background features corresponding to the instance object according to the correlation between the image block token and the preset reference token;
[0144] According to the instance features, background features, instance threshold and background threshold, the instance attention map corresponding to the instance object is determined.
[0145] In another possible implementation, the dividing unit 530 is further configured to:
[0146] Randomly initialize multiple key points in the instance attention map corresponding to the instance object;
[0147] According to multiple key points, the instance attention map is divided into multiple component attention maps, and the multiple component attention maps correspond to the multiple key points one by one;
[0148] Update the position of the corresponding key point according to the feature density of each component attention map;
[0149] Re-divide the multiple component attention maps according to the updated multiple key points;
[0150] The steps of updating the positions of the corresponding key points according to the feature density of each component attention map in an iterative manner and re-dividing the multiple component attention maps according to the updated multiple key points are performed, and the re-divided multiple component attention maps are obtained after the iteration is completed.
[0151] In another possible implementation, the dividing unit 530 is further configured to:
[0152] For each component attention map, determining a feature density weight corresponding to each of a plurality of image block tokens of the component attention map, wherein the feature density weight indicates a feature density of the image block token;
[0153] According to the feature density weights corresponding to the multiple image block tokens, the positions of the key points corresponding to the component attention map are updated.
[0154] In another possible implementation, key points are independently shifted to the mean of the feature vectors of the corresponding semantic components.
[0155] In another possible implementation, the device further includes: a key point filtering unit; the key point filtering unit is used to:
[0156] Removing at least one key point in a target region of the instance attention map, where the target region is a region whose confidence is less than a confidence threshold; and / or,
[0157] Merge multiple key points whose similarity is higher than the similarity threshold.
[0158] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0159] An embodiment of the present application provides an instance segmentation device based on point supervision, the device comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the method performed by the computing device in the above-mentioned various embodiments when executing the instructions.
[0160] An embodiment of the present application provides a non-volatile computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the methods performed by the computing device in the above-mentioned various embodiments are implemented.
[0161] An embodiment of the present application provides a computer program product, which includes a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a computing device, a processor in the computing device executes the methods executed by the computing device in the above-mentioned embodiments.
[0162] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing.
[0163] The computer-readable program instructions or codes described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0164] The computer program instructions for performing the operation of the present application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present application.
[0165] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0166] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0167] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0168] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, system, method and computer program product according to multiple embodiments of the present application. In this regard, each square frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the square frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square frames can actually be executed substantially in parallel, and they can also be executed in reverse order sometimes, depending on the functions involved.
[0169] It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by hardware (such as a circuit or ASIC (Application Specific Integrated Circuit)) that performs the corresponding function or action, or can be implemented by a combination of hardware and software, such as firmware.
[0170] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.
[0171] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A point-supervised instance segmentation method, characterized in that: The method comprises: Acquire a target image, wherein the target image includes n instance objects, where n is a positive integer; Determine an instance attention map corresponding to each of the n instance objects; For at least one of the instance objects, dividing the instance attention map corresponding to the instance object into a plurality of component attention maps, wherein the plurality of component attention maps indicate a maximum similarity between an image feature of the instance attention map and key point features of each semantic component of the instance object; Determine a segmentation result of the instance object according to the multiple component attention maps corresponding to at least one of the instance objects; The determining the instance attention maps corresponding to the n instance objects respectively includes: For at least one of the instance objects, the instance attention map corresponding to the instance object is determined based on an image block token and a preset reference token, wherein the image block token indicates an image feature of an image block of the target image, and the instance attention map indicates a similarity between the image block token and the reference token.
2. The method according to claim 1, characterized in that: The determining, according to the image block token and the preset reference token, the instance attention map corresponding to the instance object comprises: Determining instance features and background features corresponding to the instance object according to the correlation between the image block token and the preset reference token; The instance attention map corresponding to the instance object is determined according to the instance feature, the background feature, the instance threshold and the background threshold.
3. The method according to claim 1, characterized in that The step of dividing the instance attention map corresponding to the instance object into a plurality of component attention maps comprises: Randomly initialize a plurality of key points in the instance attention map corresponding to the instance object; According to the multiple key points, the instance attention map is divided into the multiple component attention maps, and the multiple component attention maps correspond to the multiple key points one by one; Update the position of the corresponding key point according to the feature density of each component attention map; Re-dividing the plurality of component attention maps according to the updated plurality of key points; The steps of updating the positions of the key points corresponding to the feature densities of the component attention maps are performed in an iterative manner, and re-dividing the multiple component attention maps according to the updated multiple key points, and obtaining the re-divided multiple component attention maps after the iteration is completed.
4. The method according to claim 3, characterized in that The updating of the position of the corresponding key point according to the feature density of each component attention map includes: For each of the component attention maps, determining a feature density weight corresponding to each of a plurality of image block tokens of the component attention map, wherein the feature density weight indicates a feature density of the image block token; According to the feature density weights corresponding to each of the multiple image block tokens, the positions of the key points corresponding to the component attention map are updated.
5. The method according to claim 4, characterized in that The key points are independently shifted to the mean of the feature vectors of the corresponding semantic components.
6. The method according to any one of claims 3 to 5, characterized in that: Before dividing the instance attention map corresponding to the instance object into a plurality of component attention maps, the method further includes: Removing at least one key point in a target area of the instance attention map, wherein the target area is an area whose confidence is less than a confidence threshold; and / or, Merge multiple key points whose similarity is higher than the similarity threshold.
7. A point-supervised instance segmentation device, characterized in that: The device comprises: An acquisition unit, configured to acquire a target image, wherein the target image includes n instance objects, where n is a positive integer; A first determining unit, configured to determine an instance attention map corresponding to each of the n instance objects; a dividing unit, configured to divide, for at least one of the instance objects, the instance attention map corresponding to the instance object into a plurality of component attention maps, wherein the plurality of component attention maps indicate a maximum similarity between an image feature of the instance attention map and key point features of each semantic component of the instance object; A second determining unit, configured to determine a segmentation result of the instance object according to the multiple component attention maps corresponding to at least one of the instance objects; The first determining unit is further configured to: For at least one of the instance objects, the instance attention map corresponding to the instance object is determined based on an image block token and a preset reference token, wherein the image block token indicates an image feature of an image block of the target image, and the instance attention map indicates a similarity between the image block token and the reference token.
8. A point-supervised instance segmentation device, characterized in that: The device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method according to any one of claims 1 to 6 when executing the instructions.
9. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, characterized in that: When the computer readable code is executed in a computing device, a processor in the computing device executes the method according to any one of claims 1 to 6.
