Instance segmentation methods, apparatus, equipment and media
By determining the instance center point in the instance segmentation method and updating the query vector using the Transformer network structure, combined with sparse sampling and attention masking, the problem of multiple query vectors corresponding to the same instance is solved, achieving higher accuracy and faster instance segmentation.
Patent Information
- Application Number
- CN202311220049.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-20
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-09-20
AI Technical Summary
Existing instance segmentation methods based on instance queries often result in multiple query vectors corresponding to the same instance, leading to low accuracy in instance segmentation.
By obtaining the instance center points of the image to be segmented, the query vector of the instance is determined, and the query vector is updated using the Transformer network structure. By combining sparse sampling and inter-instance intersection-union ratio to form an attention mask, the possibility of querying the same instance multiple times is reduced.
It improves the accuracy and speed of instance segmentation, reduces computational load, and enhances the efficiency of the inference model.
Smart Images

Figure CN117372691B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer vision, and more specifically, to an instance segmentation method, an instance segmentation apparatus, an electronic device, and a computer-readable storage medium. Background Technology
[0002] In the field of computer vision, instance segmentation is a crucial research topic, providing vital technical support for applications in geographic information systems, medical imaging, autonomous driving, and robotics. Instance segmentation methods process images containing instances, yielding segmentation results (i.e., the outline or segmented region of each instance), classification results (i.e., the category of each instance), and localization results (i.e., the location or bounding box of each instance).
[0003] Currently, instance segmentation methods based on instance queries achieve excellent segmentation results. This method determines an initial instance query vector based on all pixels of an instance, then updates the instance query vector using a Transformer network structure, and finally determines the instance segmentation, classification, and localization results using the updated instance query vector.
[0004] However, the above methods generally suffer from situations where multiple query vectors correspond to the same instance, making it difficult to guarantee the uniqueness of instance queries and thus affecting the accuracy of instance segmentation. Therefore, there is an urgent need to provide a new instance segmentation method to improve the accuracy of instance segmentation. Summary of the Invention
[0005] One objective of this application is to provide a new technical solution for instance segmentation to improve the accuracy of instance segmentation.
[0006] According to a first aspect of this application, an instance segmentation method is provided, comprising: acquiring an image to be segmented; determining a query vector of instances in the image to be segmented based on instance center points of the image to be segmented; and determining a classification result, a localization result, and a segmentation result of instances in the image to be segmented based on the query vector of instances in the image to be segmented.
[0007] According to a second aspect of this application, an instance segmentation apparatus is provided, comprising: an acquisition module for acquiring an image to be segmented; a first determination module for determining a query vector of an instance in the image to be segmented based on an instance center point of the image to be segmented; and a second determination module for determining a classification result, a localization result, and a segmentation result of the instance in the image to be segmented based on the query vector of the instance in the image to be segmented.
[0008] According to a third aspect of this application, an electronic device is provided, including a memory and a processor, the memory being configured to store computer instructions, and the processor being configured to invoke the computer instructions from the memory to perform a method as described in any one of the first aspects of this application.
[0009] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects of this application.
[0010] The instance segmentation method provided in this application determines the instance query vector based on the instance centroid and then achieves instance segmentation. This can reduce the occurrence of multiple queries corresponding to the same instance in existing instance segmentation methods based on instance queries, and improve the accuracy of instance segmentation.
[0011] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments of this application with reference to the accompanying drawings. Attached Figure Description
[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.
[0013] Figure 1 This is a schematic diagram of an instance segmentation model provided in an embodiment of this application.
[0014] Figure 2 This is a schematic flowchart of an instance segmentation method provided in an embodiment of this application.
[0015] Figure 3 This is a schematic diagram illustrating a local maximum probability calculation process provided in an embodiment of this application.
[0016] Figure 4 This is a schematic diagram showing the location of sparse sampling points in a determined instance provided in the embodiments of this application.
[0017] Figure 5 This is an attention mask image formed based on instance intersection-union ratio provided in the embodiments of this application.
[0018] Figure 6 This is a schematic block diagram of the instance segmentation device provided in the embodiments of this application.
[0019] Figure 7 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation
[0020] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present application.
[0021] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.
[0022] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.
[0023] In all the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.
[0024] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0025] To facilitate understanding, this application first provides a brief description of the query-based instance segmentation method. The query-based instance segmentation method first determines the instance query vector, then updates the instance query vector using a cross-attention mechanism and a self-attention mechanism, and finally determines the instance segmentation result, classification result, and localization result using the updated instance query vector.
[0026] like Figure 1 As shown in the figure, this application embodiment provides an instance segmentation model. The model includes a feature extraction module 1000, an initial query selection module 1100, and a query update module 1200.
[0027] The feature extraction module 1000 can be used to extract pixel features from the image to be segmented. This module can include a backbone network and a pyramid network. The backbone network can be, for example, ResNet-50. The pyramid network can output multi-scale pixel features.
[0028] The initial query selection module 1100 can be used to select the initial query vector for an instance. The initial query vector for an instance can be selected based on the instance's estimated centroid.
[0029] The query update module 1200 can be used to update the initial query vector of an instance using cross-attention and self-attention mechanisms to obtain the updated instance query vector. The query update module 1200 may include a cross-attention module 1210 and a self-attention module 1220. The query update module 1200 may also include a location query update module 1230 or a pixel feature update module 1240.
[0030] The cross-attention module 1210 is used to update the query vector of an instance using a cross-attention mechanism. The cross-attention mechanism can be applied to the instance's pixel features, content query vector, and location query vector to obtain an updated content query vector. In some embodiments, the instance's pixel features can be the pixel features updated by the pixel feature update module 1240, or the pixel features after sparse sampling of the updated pixel features. The location query vector can be the location query vector updated by the location query update module 1230. The content query vector can be the content query vector in the initial query vector, or the content query vector updated by the cross-attention module 1210 and the self-attention module 1220.
[0031] The self-attention module 1220 is used to update the query vector of an instance using a self-attention mechanism. The updated content query vector output by the cross-attention module 1210 can be further updated using the self-attention mechanism. In some embodiments, the self-attention mechanism can be combined with an attention mask formed by the intersection-union ratio of instances to perform content query updates.
[0032] The location query update module 1230 is used to update the location query vector of the instance. The pixel feature update module 1240 is used to update the pixel features of the instance. The location query update module 1230 and the pixel feature update module 1240 can use a cross-attention mechanism for updating.
[0033] Next, we will introduce an instance segmentation method based on the above instance segmentation model provided by an embodiment of this application. For example... Figure 2 As shown, the method includes steps S100 to S120.
[0034] Step S100: Obtain the image to be segmented.
[0035] The image to be segmented may include instances of multiple categories, and each category may include multiple instances. For example, the image to be segmented may include instances of different categories such as people, cars, and balloons. The image to be segmented may include multiple people, multiple cars, multiple balloons, etc. This application does not specifically limit the specific category of the instances, nor does it specifically limit the number of instances.
[0036] The image to be segmented can be input into the feature extraction module to extract the pixel features of the image to be segmented.
[0037] In some embodiments, the first pixel features output from the second layer of the pyramid network can be extracted, or the second pixel features output from other layers of the pyramid network can be extracted.
[0038] Step S110: Determine the query vector of the instance of the image to be segmented based on the instance center point of the image to be segmented.
[0039] The center points of instances in the image to be segmented can be pre-labeled manually, or the center points of instances in the image to be segmented can be predicted by a deep learning model.
[0040] In some embodiments, for each pixel feature of the first pixel feature output by the second layer of the pyramid network, the probability that it is an instance center point can be calculated. Based on the magnitude of this probability, the location of the instance center point in the image to be segmented is predicted. Determining the instance query vector using the first pixel feature output by the second layer of the pyramid network can improve inference speed.
[0041] In other embodiments, the categories of instances in the image to be segmented can be manually labeled. Alternatively, all categories of instances can be obtained from a pre-stored database in a deep learning model. For each pixel feature of the image to be segmented, the probability that it is the center point of an instance of each category is calculated. Based on the magnitude of its probability of being the center point of an instance of each category, the category of the instance in the image to be segmented is predicted.
[0042] Taking an image to be segmented as an example, where the instances are categorized into two classes: people and balloons, and the first pixel feature map of the image to be segmented is a 4x4 matrix. Table 1 schematically represents the probability that each pixel feature is the center point of the "person" instance. Table 2 schematically represents the probability that each pixel feature is the center point of the "balloon" instance.
[0043] Table 1
[0044] 0.9 0.89 0.3 0.1 0.89 0.7 0.2 0.1 0.7 0.6 0.1 0.05 0.6 0.5 0.05 0.03
[0045] Table 2
[0046] 0.1 0.2 0.6 0.5 0.1 0.3 0.7 0.6 0.3 0.8 0.9 0.8 0.4 0.5 0.8 0.8
[0047] Taking pixel feature positions (1,1) in Table 1 and (3,3) in Table 2 as examples, the probabilities corresponding to these two positions are both very high, indicating that these two positions are highly likely to be the center points of the instances. Furthermore, the probability of position (1,1) in the probability map for the "person" category is 0.9, while the probability of position (1,1) in the probability map for the "balloon" category is 0.1, indicating that the instance corresponding to position (1,1) is most likely a "person." Similarly, the instance corresponding to position (3,3) is most likely a "balloon."
[0048] A probability threshold can be preset, and the location corresponding to the pixel feature with a probability greater than the probability threshold can be taken as the location of the possible instance center point, and the instance category corresponding to the probability greater than the probability threshold can be taken as the category of the possible instance.
[0049] Alternatively, other preset rules can be used to determine the instance center point as accurately as possible, thereby determining the location of the instance center point and the instance category as accurately as possible.
[0050] Step S120: Based on the query vector of the instance of the image to be segmented, determine the classification result, localization result and segmentation result of the instance of the image to be segmented.
[0051] In this embodiment, the query vector may include a content query vector and a location query vector. Based on the content query vector of an instance in the image to be segmented, the classification result of the instance is determined. Based on the location query vector of the instance in the image to be segmented, the localization result of the instance is determined. Based on the content query vector of the instance in the image to be segmented and the pixel features of the image to be segmented, the segmentation result of the instance is determined.
[0052] In some embodiments, the segmentation result of an instance can be determined based on the content query vector of the instance to be segmented and the pixel features of the image to be segmented output from the second or higher layers of the pyramid network. This can improve the accuracy of the classification and segmentation results.
[0053] Considering the content query vector of the instance in the image to be segmented and the pixel features of the image to be segmented from the second layer output of the pyramid network, the inference time required to determine the segmentation result of the instance is short, but the accuracy is not high enough. Conversely, considering the content query vector of the instance in the image to be segmented and the pixel features of the image to be segmented from the higher layers output of the pyramid network, the inference time required to determine the segmentation result of the instance is long, but the accuracy is high.
[0054] Preferably, the segmentation result of an instance can be determined based on the content query vector of the instance in the image to be segmented and the second pixel feature of the image to be segmented output by the third layer of the pyramid network. This can improve the accuracy of the classification and segmentation results without significantly affecting the time required for the inference process.
[0055] The instance segmentation method provided in this application determines the instance query vector based on the instance center point to achieve instance segmentation. Compared with the existing method that directly determines the first query vector of an instance based on the position of each pixel feature and the corresponding instance category, this method can reduce the occurrence of multiple queries corresponding to the same instance in the existing instance segmentation method based on instance query, thereby improving the accuracy of instance segmentation.
[0056] As shown in Table 1 above, the probability value for position (1,1) is 0.9, and the probabilities for positions (1,2) and (2,1) are 0.89. These two probability values are very close, and the instances corresponding to positions (1,1), (1,2), and (2,1) are very likely the same instance. If we use the query vectors corresponding to these three positions to query for instances, we will still encounter the situation where multiple queries return the same instance.
[0057] Therefore, in some embodiments, step S110 may include steps S200 to S230.
[0058] Step S200: Obtain all pre-stored instance categories.
[0059] A deep learning model can be pre-trained to include all instance categories. This can be done using the COCO dataset.
[0060] Step S210: For each instance category, calculate the probability that each pixel feature in the image to be segmented is the center point of an instance of that instance category, and obtain the probability map of the instance center points corresponding to that instance category.
[0061] For example, there are 15 instance categories, such as people, cars, balloons, and dogs. The probability that each pixel feature in the image to be segmented is the center point of an instance in one of these 15 categories is calculated. This yields a probability map of instance center points corresponding to the 15 categories.
[0062] It's understandable that for the probability map of instance center points corresponding to certain categories, all probabilities might be very small, or even 0. In this case, it indicates that there are no instances of that category in the image to be segmented. Alternatively, for the probability map of instance center points corresponding to certain categories, there might be multiple high probabilities. In this case, it indicates that there is a high probability that there are instances of that category in the image to be segmented. Furthermore, when multiple high-probability points are scattered, it indicates that there might be multiple instances of that category in the image to be segmented.
[0063] Step S220: Determine the local maximum probability in the probability map of each instance center point.
[0064] The calculation of local maximum probability can be achieved using the MaxPool method. In some embodiments, steps (1) to (4) may be included. Step (1): The original matrix is padded with "0"s to obtain the padded matrix one. Step (2): MaxPool is performed on the padded matrix one to obtain matrix two. Step (3): The probability value of each position in matrix two is compared with the probability value of the same position in the original matrix. If they are the same, they are marked as T; if they are different, they are marked as F, thus obtaining matrix three. Step (4): The probability in the original matrix corresponding to the marked T in matrix three is taken as the local maximum probability. As an example, the maximum value in the current window can be selected using a 3×3 sliding window to perform the MaxPool calculation.
[0065] Figure 3 Taking the probability map of the center point of the instance in Table 1 as an example, the process of calculating the local maximum probability of Table 1 is illustrated. In the process of calculating the local maximum probability, step (1) fills the boundary of the original 4×4 matrix with "0" to obtain a 6×6 matrix one. Step (2) for the 6×6 matrix one obtained in step (1), starting from position (1,1), calculate the maximum value in the current window from left to right through a 3×3 sliding window to obtain the first row of matrix two. Then the window moves down, and continues to calculate the maximum value in the current window from left to right through a 3×3 sliding window from position (2,1) of matrix one to obtain the second row of matrix two. And so on, to obtain the third and fourth rows of matrix two. Among them, matrix two is a matrix with the same number of rows and columns as the original matrix. Then execute steps (3) and (4) to obtain the following: Figure 3 The probability corresponding to the position (1,1) in the middle is the local maximum probability of the original matrix.
[0066] In Table 1, all probability values not belonging to a local maximum probability will be cleared to 0. Although the probability values of 0.89 at positions (1,2) and (2,1) are also large, they will also be cleared to 0. This prevents the query vector from being determined based on these two positions, reducing the occurrence of multiple queries corresponding to the same instance. Furthermore, positions corresponding to some significantly small probability values can be generally determined not to be the center points of instances. Removing the query vectors corresponding to these positions from the instance query vector can also improve the speed of instance segmentation.
[0067] Step S230: Determine the first query vector of the instance based on the position of the pixel feature corresponding to the local maximum probability and the corresponding instance category.
[0068] The first query vector is the initial query vector for the instance. The position of the pixel feature corresponding to the local maximum probability can be directly used as the first position query vector for the instance in the image to be segmented, and the instance category corresponding to the local maximum probability can be used as the first content query vector for the instance in the image to be segmented.
[0069] Alternatively, in the local maximum probability map corresponding to each category, the top K local maximum probabilities can be selected. The positions of the pixel features corresponding to the top K local maximum probabilities can be used as the first position query vector of the instance in the image to be segmented, and the instance categories corresponding to the top K local maximum probabilities can be used as the first content query vector of the instance in the image to be segmented. The value of K can be determined according to the actual situation, and no specific limitation is made here.
[0070] In some embodiments, step S230 may further include steps S231 to S232.
[0071] Step S231: Sort the local maximum probabilities in the probability graph of the instance centroids corresponding to all instance categories from largest to smallest.
[0072] Step S232: The positions of the pixel features corresponding to the top K local maximum probabilities after sorting and the corresponding instance categories are used as the first query vector of the instance.
[0073] In the embodiments of this application, the value of K can be determined according to the actual situation, and can be determined by the number of instances of the image to be segmented.
[0074] The embodiments of this application determine the first query vector of an instance based on the position of the pixel feature corresponding to the local maximum probability and the corresponding instance category. Compared with directly determining the first query vector of an instance based on the position of the pixel feature corresponding to the probability greater than the threshold and the corresponding instance category, this can further reduce the occurrence of multiple instance queries corresponding to the same instance.
[0075] In this embodiment of the application, the position of the pixel feature corresponding to the local maximum probability and the category of the corresponding instance can be encoded to obtain the first position query vector and the first content query vector of the instance.
[0076] In this embodiment, the first content query vector, the first position query vector, and the second pixel features of the image to be segmented can be input into the cross-attention module based on the Transformer network structure to obtain the second content query vector, and the second content query vector can be input into the self-attention module based on the Transformer network structure to obtain the third content query vector.
[0077] The first content query vector and the second pixel features of the image to be segmented can be input into the cross-attention module and the self-attention module based on the Transformer network structure to obtain the third pixel features.
[0078] The first content query vector and the first position query vector of an instance can be input into the cross-attention module and the self-attention module based on the Transformer network structure to obtain the second position query vector.
[0079] In some embodiments, the second pixel features of the image to be segmented, the first content query vector of the instance, and the first position query vector of the instance can be input into the cross-attention module and self-attention module of other network structures to obtain the third content query vector.
[0080] In these embodiments, in step S120, the third content query vector of the instance of the image to be segmented and the third pixel feature of the image to be segmented can be convolved to obtain the segmentation result of the instance. The third content query vector and the second location query vector of the instance can be decoded to obtain the classification result and the localization result of the instance.
[0081] Querying instances using a query vector determined by the location and category of the instance's center point reduces the likelihood of multiple queries returning the same instance. This method offers higher accuracy compared to existing techniques that use query vectors determined by each pixel's features.
[0082] The embodiments of this application update the first position query vector and the second pixel feature of the instance based on the Transformer network structure to obtain a more accurate second position query vector and third pixel feature, thereby improving the accuracy of instance segmentation.
[0083] Furthermore, by using the first query vector of the instance determined by the above instance center point to calculate the second content query vector, the second position query vector, the third content query vector, and the third pixel feature, the computational load of the entire inference model can be reduced, and the inference speed can be increased.
[0084] However, the applicant has found that updating instance query vectors based on the Transformer network structure generally suffers from high computational cost and slow inference speed. Therefore, in this embodiment, step S110 may further include steps S300 to S320.
[0085] Step S300: Determine the position of the sparse sampling point of the instance corresponding to the first position query vector.
[0086] In some embodiments, step S300 may include steps S400 to S410.
[0087] Step S400: Generate offset parameters based on the center point for the instance corresponding to the first position query vector.
[0088] The first position query vector can be passed through a pre-trained linear layer to generate offset parameters based on the center point. Multiple sets of offset parameters can be generated according to the actual situation.
[0089] Step S410: Determine the location of the sparse sampling points of the instance based on the offset parameter based on the center point and the location of the instance's center point.
[0090] The positions of multiple sparse sampling points of an instance can be determined based on multiple sets of offset parameters and the position of the instance's center point.
[0091] like Figure 4 As shown, the center point of the instance is O, and the corresponding sparse sampling points are located at A, B, C, and D.
[0092] Step S310: At the location of the sparse sampling points, sample the image to be segmented to obtain the pixel features after sparse sampling.
[0093] It is possible to sparsely sample the third pixel feature of the image to be segmented.
[0094] The image to be segmented can be sampled using bilinear interpolation, or other methods can be used; no specific limitation is made here.
[0095] Step S320: The sparsely sampled pixel features and the first content query vector are weighted and summed to obtain the second content query vector of the instance.
[0096] This application embodiment reduces the amount of computation and improves the speed of instance segmentation by sparsely sampling the pixel features of the image to be segmented based on the instance center point.
[0097] In some embodiments, step S110 may further include steps S400 to S440.
[0098] Step S400: Determine the inter-instance intersection-union ratio of the instance corresponding to the first position query vector.
[0099] The localization results of all instances in the image to be segmented can be obtained by decoding the first position query vector, and the crossover ratio (CRO) between instances can be calculated based on the localization results of each instance.
[0100] Step S410: Set relevant instances according to the inter-instance crossover and union ratio.
[0101] When the intersection-union ratio (IU) between two instances is greater than a preset threshold, the two instances are designated as related instances. Similarly, when the IU between two instances is less than a preset threshold, the two instances are designated as unrelated instances.
[0102] The preset threshold can be set flexibly according to the actual situation, and there is no specific limitation here. For example, the preset threshold can be 0.15.
[0103] Step S420: Based on relevant instances, form an attention mask.
[0104] like Figure 5 As shown, the position of the pixel feature corresponding to the relevant instance can be labeled as F, and the position of the pixel feature corresponding to the irrelevant instance can be labeled as T.
[0105] Step S430: Superimpose the attention mask with the mask obtained based on the second content query vector to obtain the target mask.
[0106] The second content query vector can be processed through a linear layer to generate a query vector (Q), a key vector (K), and a value vector (V). The similarity between Q and K is then calculated, and the result, divided by the variance, is used to obtain a mask. This mask is then superimposed on the attention mask to obtain a fused mask. This fused mask is used as the target mask.
[0107] Step S440: Based on the first target mask, obtain the third content query vector.
[0108] The result of the above mask superposition can be normalized and then weighted and summed with V to obtain the third content query vector.
[0109] In this embodiment of the application, the attention calculation between irrelevant instances in the self-attention module is reduced based on the intersection-union relationship between instances, thereby reducing the computational workload and improving the speed of instance segmentation. Furthermore, it can reduce noise introduced by irrelevant instances, further improving the accuracy of instance segmentation.
[0110] To further improve the accuracy and speed of instance segmentation, embodiments of this application may also include multi-level cascading of the cross-attention module based on sparse sampling of instance center points and the self-attention module based on the cross-intersection over union ratio (CUI) between instances.
[0111] The first position query vector and the first content query vector can be input into the cross-attention mechanism to obtain the second position query vector; the third content query vector and the second position query vector can be used as the first content query vector and the first position query vector respectively, and the steps to obtain the third content query vector and the second position query vector can be repeated.
[0112] In some embodiments, the cascade number of the cross-attention module based on sparse sampling of instance center points and the self-attention module based on the cross-union ratio between instances is 3.
[0113] In some embodiments, the second pixel feature and the first content query vector can be input into the cross-attention mechanism to obtain the third pixel feature; the third content query vector, the second position query vector, and the third pixel feature are respectively used as the first content query vector, the first position query vector, and the second pixel feature, and the steps of obtaining the third content query vector, the second position query vector, and the third pixel feature are repeated.
[0114] like Figure 6 As shown in the figure, this application embodiment provides an instance segmentation device. The instance segmentation device 1000 includes an acquisition module 1100, a first determination module 1200, and a second determination module 1300.
[0115] The acquisition module 1100 can be used to acquire the image to be segmented.
[0116] The first determining module 1200 can be used to determine the query vector of the instance of the image to be segmented based on the instance center point of the image to be segmented.
[0117] Optionally, the first determining module 1200 can be specifically used to determine the first query vector of an instance in the instance segmentation method described above.
[0118] Optionally, the first determining module 1200 can also be used to determine the second query vector of an instance in the instance segmentation method described above.
[0119] Optionally, the first determining module 1200 can also be used to determine the third query vector of an instance in the instance segmentation method described above.
[0120] The second determining module 1300 can be used to determine the classification result, localization result, and segmentation result of the instance of the image to be segmented based on the query vector of the instance of the image to be segmented.
[0121] Optionally, the second determining module 1300 can also be used to determine the classification result, localization result, and segmentation result of the instance of the image to be segmented by using the third content query vector, the second location query vector, and the third pixel feature determined by the multi-level cascading method in the above instance segmentation method.
[0122] like Figure 7As shown, this application provides an electronic device. The electronic device 2000 includes a memory 2100 and a processor 2200. The memory 2100 is used to store computer instructions, and the processor 2200 is used to retrieve the computer instructions from the memory 2100 to execute any instance segmentation method as described in the embodiments of this application.
[0123] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements any of the instance segmentation methods described in this application.
[0124] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.
[0125] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0126] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.
[0127] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.
[0128] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0129] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0130] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.
[0132] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.
Claims
1. An instance segmentation method, characterized in that, include: Obtain the image to be segmented; Based on the center point of the instance of the image to be segmented, a query vector of the instance of the image to be segmented is determined. The query vector includes a first content query vector, a first position query vector, a second content query vector, a second position query vector, and a third content query vector. Based on the query vector of the instance, the classification result, localization result, and segmentation result of the instance are determined; Determining the query vector of an instance of the image to be segmented based on the center point of the instance includes: For each pre-stored instance category, calculate the probability that each pixel feature in the image to be segmented is the center point of an instance of that instance category, and obtain a probability map of the instance center points corresponding to that instance category; In the probability map of each instance center point, determine the local maximum probability; Based on the position of the pixel feature corresponding to the local maximum probability and the instance category, determine the first position query vector and the first content query vector of the instance; Determine the position of the sparse sampling point of the instance corresponding to the first position query vector; At the location of the sparse sampling point, the image to be segmented is sampled to obtain the sparsely sampled pixel features; The sparsely sampled pixel features and the first content query vector are weighted and summed to obtain the second content query vector of the instance. Determine the inter-instance intersection-union ratio (IUCN) of the instance corresponding to the first position query vector; Based on the intersection-union ratio between the instances, relevant instances are set; Based on the aforementioned relevant examples, an attention mask is formed; The attention mask is superimposed with the mask obtained based on the second content query vector to obtain the target mask; Based on the target mask, a third content query vector is obtained; The first position query vector and the third content query vector are input into the cross-attention mechanism to obtain the second position query vector.
2. The instance segmentation method according to claim 1, characterized in that, The step of determining the first location query vector and the first content query vector of the instance based on the location of the pixel feature corresponding to the local maximum probability and the instance category includes: Sort the local maximum probabilities in the probability graph of the instance centroids corresponding to all instance categories from largest to smallest; The positions of the pixel features corresponding to the top K local maximum probabilities after sorting are used as the first position query vector, and the instance categories corresponding to the top K local maximum probabilities after sorting are used as the first content query vector, wherein K is determined by the number of instances of the image to be segmented.
3. The instance segmentation method according to claim 1, characterized in that, Determining the position of the sparse sampling point of the instance corresponding to the first position query vector includes: Generate offset parameters based on the center point for the instance corresponding to the first position query vector; The location of the sparse sampling point of the instance is determined based on the offset parameter based on the center point and the location of the center point of the instance.
4. The instance segmentation method according to claim 1, characterized in that, The step of determining the query vector of the instance in the image to be segmented based on the instance center point of the image to be segmented further includes: The steps of obtaining the third content query vector and the second position query vector are repeated multiple times, using the third content query vector and the second position query vector as the first content query vector and the first position query vector, respectively.
5. The instance segmentation method according to claim 1, characterized in that, Each pixel feature in the image to be segmented is a feature output from the second layer of the pyramid network.
6. An instance segmentation device, characterized in that, include: The acquisition module is used to acquire the image to be segmented; The first determining module is used to determine the query vector of the instance of the image to be segmented based on the instance center point of the image to be segmented. The query vector includes a first content query vector, a first position query vector, a second content query vector, a second position query vector, and a third content query vector. The second determining module is used to determine the classification result, localization result, and segmentation result of the instance of the image to be segmented based on the query vector of the instance of the image to be segmented. The first determining module is specifically used for: For each pre-stored instance category, calculate the probability that each pixel feature in the image to be segmented is the center point of an instance of that instance category, and obtain a probability map of the instance center points corresponding to that instance category; In the probability map of each instance center point, determine the local maximum probability; Based on the position of the pixel feature corresponding to the local maximum probability and the instance category, determine the first position query vector and the first content query vector of the instance; Determine the position of the sparse sampling point of the instance corresponding to the first position query vector; At the location of the sparse sampling point, the image to be segmented is sampled to obtain the sparsely sampled pixel features; The sparsely sampled pixel features and the first content query vector are weighted and summed to obtain the second content query vector of the instance. Determine the inter-instance intersection-union ratio (IUCN) of the instance corresponding to the first position query vector; Based on the intersection-union ratio between the instances, relevant instances are set; Based on the aforementioned relevant examples, an attention mask is formed; The attention mask is superimposed with the mask obtained based on the second content query vector to obtain the target mask; Based on the target mask, a third content query vector is obtained; The first position query vector and the third content query vector are input into the cross-attention mechanism to obtain the second position query vector.
7. An electronic device, characterized in that, It includes a memory and a processor, the memory being used to store computer instructions, and the processor being used to retrieve the computer instructions from the memory to execute the instance segmentation method as described in any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the instance segmentation method as described in any one of claims 1-5.
