An image segmentation method, an image segmentation device, an electronic device, and a storage medium
Patent Information
- Application Number
- CN202311776948.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-21
- Publication Date
- 2026-11-03
- Estimated Expiration
- 2043-12-21
AI Technical Summary
Existing technologies are prone to undersegmentation or missegmentation when performing image segmentation on a test image based on a single reference image, especially when the test image contains multiple target objects.
By calculating the feature similarity between the reference image and the test image, a feature similarity map is generated. Then, image segmentation is performed using prior positive and negative cue points to generate a target mask. The feature similarity map and the segmentation model are combined for analysis and filtering to ensure the accuracy of the segmentation results.
It achieves accurate segmentation of multi-object images, reduces undersegmentation and missegmentation, and improves the accuracy of image segmentation.
Smart Images
Figure CN117745750B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image processing technology, and in particular relates to an image segmentation method, an image segmentation apparatus, an electronic device, and a computer-readable storage medium. Background Technology
[0002] Images often contain a wealth of information. Image segmentation is a crucial step in image processing, extracting specific objects of interest from an image to facilitate subsequent targeted processing. Currently, when segmenting a test image based on a single reference image, only one object within the test image can be segmented, which can easily lead to undersegmentation or missegmentation. Summary of the Invention
[0003] This application provides an image segmentation method, an image segmentation apparatus, an electronic device, and a computer-readable storage medium, which can improve the accuracy of image segmentation and reduce the occurrence of undersegmentation or missegmentation.
[0004] In a first aspect, this application provides an image segmentation method, including:
[0005] Calculate the feature similarity between the target object in the reference image and each pixel in the test image to obtain a feature similarity map, wherein the reference image has been labeled with the mask of the target object.
[0006] Analyze the feature similarity map;
[0007] Given at least one prior positive and negative cue point pair based on the analysis results, the test image is segmented based on each prior positive and negative cue point pair, wherein the negative cue points in each prior positive and negative cue point pair are the same;
[0008] If at least one target mask is obtained based on the segmentation process, the segmentation result of the test image is obtained through at least one target mask.
[0009] Secondly, this application provides an image segmentation apparatus, comprising:
[0010] The calculation module is used to calculate the feature similarity between the target object in the reference image and each pixel in the test image to obtain a feature similarity map, wherein the reference image has been labeled with the mask of the target object.
[0011] The analysis module is used to analyze the feature similarity map;
[0012] The segmentation module is used to segment the test image based on each prior positive and negative cue point pair, provided that at least one prior positive and negative cue point pair is obtained based on the analysis results. The negative cue points in each prior positive and negative cue point pair are the same.
[0013] The results module is used to obtain the segmentation result of the test image through at least one target mask, provided that at least one target mask is obtained based on the results of the segmentation process.
[0014] Thirdly, this application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the first aspect.
[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.
[0016] Fifthly, this application provides a computer program product comprising a computer program that, when executed by one or more processors, implements the steps of the method described in the first aspect.
[0017] The advantages of this application compared to existing technologies are as follows: During image segmentation, an unlimited number of prior positive and negative cue point pairs are obtained by analyzing the feature similarity maps obtained based on the reference image and the test image. Each prior positive and negative cue point pair indicates a possible prior position of the target object in the test image, thereby obtaining an unlimited number of target masks. After mapping these target masks back to the test image, the segmentation result of the test image can be obtained. This scheme is applicable to test images containing multiple target objects as well as those not containing target objects. It ensures that multiple target objects in the test image can be segmented, and avoids forcing incorrect segmentation results on test images that do not contain target objects, thus improving the accuracy of image segmentation and helping to reduce undersegmentation or missegmentation.
[0018] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a schematic diagram illustrating the implementation flow of the image segmentation method provided in the embodiments of this application;
[0021] Figure 2 This is an example diagram illustrating the process of obtaining prior positive and negative cue point pairs provided in the embodiments of this application;
[0022] Figure 3 This is an example diagram illustrating the working process of the segmentation model provided in the embodiments of this application;
[0023] Figure 4 This is an example diagram illustrating the filtering process of the mask output by the segmentation model, provided in an embodiment of this application.
[0024] Figure 5 This is an example diagram of image segmentation of a test image provided in an embodiment of this application;
[0025] Figure 6 This is a structural block diagram of the image segmentation apparatus provided in the embodiments of this application;
[0026] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0028] The image segmentation method proposed in this application can be applied to electronic devices with image processing capabilities. In some examples, the electronic device may be a personal computer, smartphone, tablet computer, or robot, etc., and is not limited here. Please refer to... Figure 1 The implementation process of this image segmentation method is detailed below:
[0029] Step 101: Calculate the feature similarity between the target object in the reference image and each pixel in the test image to obtain a feature similarity map.
[0030] Users can pre-specify their target objects. Based on the specified target object, the electronic device first obtains a reference image containing the target object, and this reference image already contains a mask labeled with the target object's information. The reference image only needs to contain one target object. It can be understood that the target object's mask accurately indicates its position within the reference image. To ensure the accuracy of subsequent image segmentation, the mask labeled with the target object in the reference image should be as precise as possible. For example, assuming the user specifies a dog as the target object, then: the electronic device should pre-acquire an image containing dog image information, and this image should already contain a mask labeled with the dog's image information.
[0031] The test image refers to the image on which image segmentation based on the target object is currently being performed; that is, the test image is the object of this image segmentation, and the purpose of this image segmentation is to segment the target object contained in the test image. Specifically, when the electronic device is equipped with a vision sensor, the test image can be an image acquired by the electronic device through the vision sensor; or, when the electronic device has network connectivity, the test image can be an image received or captured by the electronic device through the Internet; or, the test image can be an image loaded by the electronic device through interaction with a user or other devices. This application embodiment does not limit the method of acquiring the test image. It should be noted that in this application embodiment, the reference image and the test image are not required to have the same size.
[0032] After obtaining a reference image and a test image, the electronic device can calculate the feature similarity between the target object in the reference image and each pixel in the test image based on the PerSAM (Personalization approach for Segment Anything Model) method, thereby obtaining a feature similarity map. Specifically, the encoder of a pre-trained SAM segmentation model can be used to encode the target object in the reference image. For example, the target object in the reference image can be encoded as a 256*1 feature vector. Simultaneously, the same SAM segmentation model can be used to encode the test image. For example, the test image is encoded as 64*64 pixels (i.e., 4096 pixels), with each pixel represented by a 256-dimensional feature vector. Based on the encoding results of both, the feature similarity between the encoded result of the test image (i.e., 4096 pixels, each a 256-dimensional feature vector) and the encoded result of the target object (i.e., a 256*1 feature vector) can be calculated, resulting in 4096 feature similarity values. These 4096 feature similarity values can be reconstructed into a 64*64 feature similarity map. Through interpolation and padding removal, the size of this feature similarity map can be restored to the same size as the test image, thus obtaining the final visualized feature similarity map.
[0033] It can be understood that the feature similarity map is actually a feature matrix, and the pixel value of each pixel ranges from [-1, 1]. The whiter the pixel in the feature similarity map, the higher the feature similarity between the pixel in the test image and the target object in the reference image; conversely, the darker the pixel in the feature similarity map, the lower the feature similarity between the pixel in the test image and the target object in the reference image.
[0034] Step 102: Analyze the feature similarity map.
[0035] Electronic devices can analyze feature similarity maps and determine whether prior positive and negative cue point pairs exist in the feature similarity map based on the analysis results. Since the number of target objects in the test image is uncertain, this application embodiment does not limit the number of prior positive and negative cue point pairs obtained from the analysis; that is, the number of prior positive and negative cue point pairs in the feature similarity map may be zero or more than one.
[0036] It can be understood that each prior positive-negative cue point pair is a cue marker, where the positive cue point in each prior positive-negative cue point pair indicates a more probable position of the target object in the test image, and the negative cue point in each prior positive-negative cue point pair indicates a less probable position of the target object in the test image. In the embodiments of this application, to simplify processing, the negative cue points in each prior positive-negative cue point pair can be the same; that is, the negative cue points obtained by the electronic device analysis are universal.
[0037] Step 103: If at least one prior positive and negative cue point pair is obtained based on the analysis results, the test image is segmented based on each prior positive and negative cue point pair.
[0038] As described earlier, the positive cue point in each prior positive-negative cue point pair indicates a more probable location of the target object in the test image, while the negative cue point in each pair indicates a less probable location. Based on this, given at least one prior positive-negative cue point pair obtained from the analysis results, the electronic device can use each pair as a basis to segment the test image, resulting in a corresponding mask for each pair. Since there is at least one prior positive-negative cue point pair, there is also at least one resulting mask. Each mask can serve as the final target mask for segmentation; alternatively, to avoid interference from irrelevant objects and improve segmentation accuracy, these masks can be further processed through filtering and / or merging, with the results serving as the final target mask. It should be noted that the result of post-processing may be empty; that is, after post-processing, there may be an extreme case where all masks are filtered out, resulting in no target mask being obtained (this is usually because the target object does not exist in the test image).
[0039] Step 104: If at least one target mask is obtained based on the segmentation process, the segmentation result of the test image is obtained through the at least one target mask.
[0040] It is understandable that if at least one target mask is obtained based on the segmentation process, it can be assumed that a target object does exist in the test image. After mapping these target masks back to the test image, that is, after segmenting the corresponding region of the target mask in the test image, a visualized segmentation result of the test image can be obtained, which will not be elaborated here.
[0041] In some embodiments, step 102 may specifically include:
[0042] A1. Binarize the feature similarity map to obtain a binary image.
[0043] Electronic devices can binarize feature similarity maps based on a preset binarization threshold. Specifically, pixels with values greater than or equal to the binarization threshold can be set to "255", and pixels with values less than the threshold can be set to "0", thus obtaining a binary image suitable for OpenCV processing. In some examples, when the pixel values in the feature similarity map range from [-1, 1], the binarization threshold can be set to 0.7; of course, the binarization threshold can also be set to other values depending on the actual situation, which is not limited here.
[0044] A2. Extract contours from binary images.
[0045] In a binary image, there are at most black regions composed of black pixels and white regions composed of white pixels; it can be understood that the junction of the black and white regions constitutes a contour. Based on this, an electronic device can extract the contours that may be contained in the binary image through neighborhood analysis, where the contours are generally closed curves. Since the number of target objects in the test image is uncertain, this application embodiment does not limit the number of extracted contours; that is, the electronic device may extract zero contours from the binary image, or it may extract more than one contour, which is not limited here.
[0046] In some embodiments, to reduce noise interference, erosion and dilation operations can be performed on the binary image after step A1 and before step A2, which will not be elaborated here. Accordingly, step A2 can be optimized to: extract contours in the eroded and dilated binary image.
[0047] A3. If at least one contour is extracted, determine the positive and negative prompt points in each prior positive and negative prompt point pair based on each contour and feature similarity map.
[0048] If at least one contour is extracted, the electronic device can preliminarily conclude that a target object may exist in the test image. Based on this, the electronic device can directly analyze the feature similarity map to determine the general negative cue points, i.e., the negative cue points in each prior positive-negative cue point pair. In addition, the electronic device can also map the contour back to the feature similarity map and analyze the feature similarity map based on the mapping results to determine the positive cue points in each prior positive-negative cue point pair.
[0049] Specifically, the positive cue point can be obtained through the following process: For each contour, find the target point within the contour region corresponding to that contour in the feature similarity map, and use the target point as the positive cue point in a priori positive-negative cue point pair, where the target point is the pixel with the largest pixel value in the contour region. The process is described below: By mapping each contour back to the feature similarity map (i.e., selecting a bounding box in the feature similarity map based on the contour's coordinates), the electronic device can obtain the contour region corresponding to that contour in the feature similarity map; within that contour region in the feature similarity map, traverse the pixel values of all pixels; based on the traversal results, find the pixel with the largest pixel value, which is the target point; the electronic device can use the target point as the positive cue point in a priori positive-negative cue point pair.
[0050] Specifically, negative cue points can be obtained through the following process: the pixel with the smallest pixel value in the feature similarity map is taken as the negative cue point in all prior positive-negative cue point pairs. This process is described below: traverse the pixel values of all pixels in the feature similarity map; based on the traversal results, the pixel with the smallest pixel value can be found, and this pixel is the universal negative cue point; that is, this pixel is the negative cue point in each prior positive-negative cue point pair.
[0051] In some embodiments, to further eliminate noise interference, before determining the positive and negative cue points in each prior positive and negative cue point pair based on each contour and feature similarity map, the electronic device may also perform filtering processing on the extracted contours. Specifically, this involves calculating the contour area of each contour and performing filtering processing based on the contour area of each contour and preset filtering conditions. Here, the contour area refers to the number of pixels within the contour (i.e., the number of pixels selected within the closed curve of the contour); the preset filtering conditions can be set based on area ratio, area size, a combination of area ratio and area size, or based on the extreme values of pixel values after specific processing of the feature similarity map. The specific process of contour filtering processing is described below based on different filtering conditions:
[0052] 1. Filtering conditions are set based on area ratio: For each contour, the electronic device can calculate the ratio of the contour area to the total area of the feature similarity map, where the total area refers to the total number of pixels in the feature similarity map; if the ratio is greater than or equal to a preset ratio threshold, the contour is retained; if the ratio is less than the ratio threshold, the contour is discarded.
[0053] 2. Filtering conditions are set based on area size: For each contour, if its area is greater than or equal to a preset area threshold, the contour is retained; if the area of the contour is less than the area threshold, the contour is discarded.
[0054] 3. Filtering conditions are set based on area ratio and area size: For each contour, the electronic device can calculate the ratio of the contour area to the total area of the feature similarity map; if the ratio is greater than or equal to a preset ratio threshold and the contour area is greater than or equal to a preset area threshold, the contour is retained; otherwise, the contour is removed, which will not be elaborated here.
[0055] 4. Filtering conditions are set based on the extreme pixel values after specific processing of the feature similarity map: The electronic device can perform preliminary filtering of contours using any of the filtering methods proposed in 1-3 above; subsequently, for each contour retained after preliminary filtering, the contour region corresponding to the contour in the feature similarity map is obtained by mapping the contour back to the feature similarity map (i.e., selecting a bounding box in the feature similarity map based on the contour's coordinates); within the contour region in the feature similarity map, the pixel values of all pixels are traversed to determine the extreme pixel values (maximum and / or minimum pixel values) within the contour region; if the extreme pixel value is greater than or equal to a preset extreme value threshold (e.g., the maximum pixel value is greater than or equal to a preset maximum value threshold, and / or the minimum pixel value is greater than or equal to a preset minimum value threshold), the contour is retained; if the extreme pixel value is less than the extreme value threshold (e.g., the maximum pixel value is less than the maximum value threshold, and / or the minimum pixel value is less than the minimum value threshold), the contour is discarded. Thus, the final filtering is completed.
[0056] Accordingly, the step of determining the positive and negative cue points in each prior positive and negative cue point pair based on each contour and feature similarity map can be optimized to: determining the positive and negative cue points in each prior positive and negative cue point pair based on each retained contour and feature similarity map. Let N be the number of retained contours, then the electronic device can obtain N prior positive and negative cue point pairs. That is, each retained contour corresponds to one prior positive and negative cue point pair, and there is a one-to-one correspondence between the retained contours and the prior positive and negative cue point pairs.
[0057] Please see Figure 2 , Figure 2 An example of the process for obtaining prior positive and negative cue pairs is given below. Figure 2 Explanation: Electronic devices can first be based on a reference image ( Figure 2 (Not shown in the image) Obtain the feature similarity map corresponding to the test image; then, by binarizing the feature similarity map, a binary image can be obtained; next, by extracting and filtering the contours of the binary image, three contours can be obtained; based on each contour, a priori positive and negative cue point pair can be determined; the electronic device can record the coordinates of the positive and negative cue points in each priori positive and negative cue point pair in the feature similarity map. Figure 2Green dots represent positive cue points, and red dots represent negative cue points. Specifically, Positive Prior Point 1 and Negative Prior Point 1 form a priori positive-negative cue point pair, and so on; this will not be elaborated further here. As you can see, Figure 2 In the three prior positive and negative cue point pairs shown, the negative cue point (i.e., the red point) is the same pixel point in the feature similarity map, and the positive cue point (i.e., the green point) is a different pixel point in the feature similarity map.
[0058] In some embodiments, step 103 may specifically include:
[0059] B1. Take the test image as one input to the preset segmentation model, and take each prior positive and negative cue point pair as another input to the segmentation model to obtain at least one candidate mask output by the segmentation model.
[0060] The preset segmentation model can remain the SAM segmentation model proposed earlier; however, it can also be other models, which are not limited here. Specifically, the segmentation model has two different inputs: the test image and prior positive and negative cue point pairs. It can be understood that when there are multiple prior positive and negative cue point pairs, each pair can be used as input to the segmentation model along with the test image. Through multiple runs of the segmentation model, multiple candidate masks output by the model can be obtained. The final number of candidate masks is the same as the number of prior positive and negative cue point pairs.
[0061] In other words, given N prior positive and negative cue point pairs, the electronic device can use the first prior positive and negative cue point pair and the test image as input to the segmentation model to obtain the first candidate mask output by the segmentation model; it can use the second prior positive and negative cue point pair and the test image as input to the segmentation model to obtain the second candidate mask output by the segmentation model; and so on, ultimately obtaining N candidate masks based on the N prior positive and negative cue point pairs.
[0062] Taking the SAM segmentation model as an example, please refer to [link / reference]. Figure 3 , Figure 3 An example of how this segmentation model works is given below. Figure 3 Explanation: The image encoder in this segmentation model (i.e. Figure 3 The input to the Image Encoder is the image to be segmented, and the output is the image's embedding representation. Figure 3 (not shown); the cue encoder in this segmentation model (i.e. Figure 3 The Prompt Encoder takes a priori positive-negative cue pair as input and outputs a 256-dimensional vector. Figure 3(Not shown); It should be noted that the process of obtaining the prior positive and negative cue point pairs is in Figure 3 This is just a simple example; some steps have been omitted. Please refer to the previous description for details, which will not be repeated here. The mask decoder in this segmentation model (i.e. Figure 3 The inputs to the Mask Decoder are the outputs of the image encoder and the cue encoder, respectively. The output of the Mask Decoder is a mask. After mapping the mask back to the image, that is, after segmenting the region corresponding to the mask in the image, the visual segmentation result of the image can be obtained.
[0063] B2. Based on preset constraints, filter the candidate masks.
[0064] Due to potential errors in the positive prompts (e.g., points not corresponding to the actual target object region), and / or potential errors in the segmentation model (e.g., misclassifying points not belonging to the target object as part of the mask), the candidate masks obtained by the electronic device may not be accurate. Therefore, to ensure the accuracy of the final segmentation result, the electronic device can filter the candidate masks based on preset constraints. Considering that the average feature similarity between irrelevant objects (i.e., objects other than the target object) and the actual target object will be low, the electronic device can set this constraint based on feature similarity. The filtering process can be specifically described as follows:
[0065] The electronic device obtains the mask region corresponding to each candidate mask in the feature similarity map (that is, the region in the feature similarity map with the same positional information as the candidate mask is determined as the mask region). Then, the electronic device can continue to calculate the average pixel value within each mask region. It can be understood that since the mask region is part of the feature similarity map, based on the physical meaning of the pixel values in the feature similarity map, the average pixel value within the mask region is equivalent to the average feature similarity between the mask region and the target object in the reference image. Finally, the electronic device can retain the candidate masks corresponding to mask regions with an average pixel value greater than a preset average pixel value threshold, and remove the candidate masks corresponding to mask regions with an average pixel value less than or equal to the average pixel value threshold, thereby achieving the filtering of candidate masks. As an example, the average pixel value threshold can be 0.5; of course, it can also be set to other values according to the actual situation, which is not limited here.
[0066] It is understood that, since the number of target objects in the test image is uncertain, the embodiments of this application do not limit the number of candidate masks retained after filtering; that is, the number of retained candidate masks may be zero or more than one, which is not limited here.
[0067] like Figure 4 As shown, Figure 4 An example of filtering the mask output by a segmentation model is given. The target object is a dog, but the test image only contains a toy robot and no dog image information. Assuming the segmentation model outputs a mask associated with the toy robot region based on this test image, since the toy robot has low feature similarity to the real target object (i.e., the dog), the average pixel value within the mask region corresponding to this mask in the feature similarity map will be small. This allows the mask associated with the toy robot region output by the segmentation model to be removed. Ultimately, the segmentation result for this test image is empty; that is, the test image will not be incorrectly segmented even when it does not contain the target object.
[0068] B3. If there are reserved candidate masks, determine the target mask based on the reserved candidate masks.
[0069] Since the acquisition of multiple candidate masks in step B1 is performed in parallel, there is a possibility of overlap between the different candidate masks that have already been retained. To address this issue, the electronic device can merge the retained candidate masks and determine which candidate mask the overlapping part belongs to during this process, thus achieving deduplication.
[0070] The electronic device can first obtain the quality scores of each candidate mask that has been retained. These quality scores are also the outputs of the segmentation model (e.g., the SAM segmentation model) mentioned above. That is, in addition to outputting the candidate mask, the segmentation model also outputs the quality score of the candidate mask.
[0071] When at least two retained candidate masks overlap, the electronic device can determine that the overlapping portion belongs to the first candidate mask and remove the overlapping portion from the second candidate mask. The first candidate mask is the candidate mask with the highest quality score among the at least two retained candidate masks that overlap, and the second candidate mask is any other candidate mask besides the first candidate mask among the at least two retained candidate masks that overlap. Specifically, this operation can be implemented through the following merging process:
[0072] Create a new image with the same size as the test image; according to the quality scores of the retained candidate masks in ascending order, mark the pixels in the corresponding area of the new image with different label values (that is, assign the pixel value of the pixel to the corresponding label value, and the label values corresponding to different candidate masks are different); if a pixel in the current pixel to be marked has already been marked (that is, it has been assigned a label value), since the marking order is from low to high quality scores, the quality score of the candidate mask referenced by the current marking will necessarily be higher than the quality score of the candidate mask that has been marked, so the pixel that has been marked can be remarked (that is, the pixel value of the pixel is reassigned to a new label value).
[0073] The electronic device eventually obtains a set of non-overlapping candidate masks, which can then be identified as the target masks. It can be understood that through the marking operation described in the previous paragraph, the final marked image will not contain a single pixel with two different mark values, thus achieving deduplication. This marked new image is the result of merging all target masks; that is, it contains all valid mask information obtained based on the test image.
[0074] Please see Figure 5 , Figure 5 An example of image segmentation is given when the test image contains other objects and multiple target objects. The following section... Figure 5 Explanation:
[0075] Assume the target object is a dog. The test image contains image information of three target objects, located at the top left, bottom left, and top right positions respectively; in addition, the test image also contains image information of another object, located at the bottom right position. Based on the reference image ( Figure 5 For the target object (not shown in the image), the electronic device obtains a feature similarity map corresponding to the test image. By binarizing the feature similarity map, the electronic device obtains a binary image. Through contour extraction and filtering of the binary image, the electronic device obtains three contours. Based on each contour, the electronic device determines a priori positive and negative cue point pair, thereby recording the coordinates of the positive and negative cue points in each priori pair in the feature similarity map. Through a segmentation model, the electronic device obtains a candidate mask corresponding to each priori positive and negative cue point pair, and each candidate mask is retained as the target mask (the specific process is not detailed in the image). Figure 5(As shown in the image). After merging these three target masks, a mask merging result is obtained. Mapping the mask merging result back to the test image, that is, segmenting the corresponding region of the test image using the mask merging result, yields a visualized segmentation result for the test image. It can be seen that this segmentation result includes image information of the three target objects located in the upper left, lower left, and upper right positions of the test image, but does not include image information of other objects located in the lower right position of the test image. Therefore, it can reduce the occurrence of undersegmentation when multiple target objects exist in the image, and also reduce the occurrence of missegmentation when no target objects exist in the image.
[0076] As can be seen from the above, in the image segmentation process of this application embodiment, an unlimited number of prior positive and negative cue point pairs are obtained by analyzing the feature similarity maps obtained based on the reference image and the test image. Each prior positive and negative cue point pair indicates a possible prior position of the target object in the test image, thereby obtaining an unlimited number of target masks. After mapping these target masks back to the test image, the segmentation result of the test image can be obtained. This scheme is applicable to test images containing multiple target objects and test images not containing target objects. It allows multiple target objects in the test image to be segmented, and avoids forcibly segmenting test images that do not contain target objects, resulting in incorrect segmentation results. This improves the accuracy of image segmentation and helps reduce the occurrence of undersegmentation or missegmentation.
[0077] Corresponding to the image segmentation method provided above, this application also provides an image segmentation apparatus. For example... Figure 6 As shown, the image segmentation device 6 includes:
[0078] The calculation module 601 is used to calculate the feature similarity between the target object in the reference image and each pixel of the test image to obtain a feature similarity map, wherein the reference image has been labeled with the mask of the target object.
[0079] Analysis module 602 is used to analyze the feature similarity map;
[0080] The segmentation module 603 is used to segment the test image based on each prior positive and negative cue point pair when at least one prior positive and negative cue point pair is obtained based on the analysis results, wherein the negative cue points in each prior positive and negative cue point pair are the same.
[0081] Result module 604 is used to obtain the segmentation result of the test image through at least one target mask when at least one target mask is obtained based on the result of the segmentation process.
[0082] In some embodiments, the analysis module 602 includes:
[0083] Binarization units are used to binarize feature similarity maps to obtain binary images;
[0084] An extraction unit is used to extract contours in a binary image;
[0085] The first determining unit is used to determine the positive and negative cue points in each prior positive and negative cue point pair based on each contour and feature similarity map, provided that at least one contour has been extracted.
[0086] In some embodiments, the first determining unit includes:
[0087] The first determining subunit is used to find the target point in the contour region corresponding to the contour in the feature similarity map for each contour, and to use the target point as a positive cue point in a prior positive and negative cue point pair, wherein the target point is the pixel with the largest pixel value in the contour region.
[0088] The second determining subunit is used to select the pixel with the smallest pixel value in the feature similarity map as the negative cue point in all prior positive and negative cue point pairs.
[0089] In some embodiments, the analysis module 602 further includes:
[0090] The calculation unit is used to calculate the contour area of each contour separately;
[0091] The first filtering unit is used to perform filtering based on the contour area of each contour and preset filtering conditions.
[0092] Accordingly, the first determining unit is specifically used to determine the positive and negative cue points in each prior positive and negative cue point pair based on each of the preserved contour and feature similarity maps.
[0093] In some embodiments, the segmentation module 603 includes:
[0094] The segmentation unit is used to take the test image as one input to the preset segmentation model and take each prior positive and negative cue point pair as another input to the segmentation model to obtain at least one candidate mask output by the segmentation model.
[0095] The second filtering unit is used to filter candidate masks based on preset constraints.
[0096] The second determining unit is used to determine the target mask based on the retained candidate masks if there are retained candidate masks.
[0097] In some embodiments, the second filtering unit includes:
[0098] The first acquisition subunit is used to acquire the mask region corresponding to each candidate mask in the feature similarity map;
[0099] The calculation subunit is used to calculate the average pixel value in each mask region.
[0100] The reserved sub-unit is used to retain candidate masks corresponding to mask regions where the average pixel value is greater than a preset average pixel value threshold.
[0101] In some embodiments, the second determining unit includes:
[0102] The second acquisition subunit is used to acquire the quality scores of each retained candidate mask;
[0103] The deduplication subunit is used to determine that the overlapping part belongs to the first candidate mask and remove the overlapping part from the second candidate mask when there is overlap between at least two retained candidate masks. The first candidate mask is the candidate mask with the highest quality score among the at least two retained candidate masks, and the second candidate mask is the other candidate mask besides the first candidate mask among the at least two retained candidate masks.
[0104] The third determining subunit is used to determine the retained candidate masks that do not overlap as the target mask.
[0105] As can be seen from the above, in the image segmentation process of this application embodiment, an unlimited number of prior positive and negative cue point pairs are obtained by analyzing the feature similarity maps obtained based on the reference image and the test image. Each prior positive and negative cue point pair indicates a possible prior position of the target object in the test image, thereby obtaining an unlimited number of target masks. After mapping these target masks back to the test image, the segmentation result of the test image can be obtained. This scheme is applicable to test images containing multiple target objects as well as test images that do not contain target objects. It allows multiple target objects in the test image to be segmented, and avoids forcing incorrect segmentation results on test images that do not contain target objects, thereby improving the accuracy of image segmentation and helping to reduce the occurrence of undersegmentation or missegmentation.
[0106] Corresponding to the image segmentation method provided above, this application also provides an electronic device. Please refer to... Figure 7 , Figure 7 A schematic diagram of the electronic device 7 is given, which includes: a memory 701, and one or more processors 702. Figure 7(Only one is shown) and a computer program stored in memory 701 and executable on the processor. Memory 701 stores software programs and units. Processor 702 executes various functional applications and data processing by running the software programs and units stored in memory 701 to obtain resources corresponding to the aforementioned preset events. Specifically, processor 702 performs the following steps when running the aforementioned computer program stored in memory 701:
[0107] Calculate the feature similarity between the target object in the reference image and each pixel in the test image to obtain a feature similarity map, wherein the reference image has been labeled with the mask of the target object.
[0108] Analyze the feature similarity map;
[0109] Given at least one prior positive and negative cue point pair based on the analysis results, the test image is segmented based on each prior positive and negative cue point pair, wherein the negative cue points in each prior positive and negative cue point pair are the same;
[0110] If at least one target mask is obtained based on the segmentation process, the segmentation result of the test image is obtained through at least one target mask.
[0111] Assuming the above is the first possible implementation, then in the second possible implementation provided based on the first possible implementation, the feature similarity map is analyzed, including:
[0112] Binarize the feature similarity map to obtain a binary image;
[0113] Contour extraction in binary images;
[0114] If at least one contour is extracted, the positive and negative cue points in each prior positive and negative cue point pair are determined based on each contour and feature similarity map.
[0115] In the third possible implementation provided based on the second possible implementation described above, the positive and negative cue points in each prior positive and negative cue point pair are determined based on each contour and feature similarity map, including:
[0116] For each contour, find the target point within the contour region corresponding to the contour in the feature similarity map, and use the target point as a positive cue point in a prior positive-negative cue point pair, where the target point is the pixel with the largest pixel value in the contour region;
[0117] The pixel with the smallest pixel value in the feature similarity map is taken as the negative cue point in all prior positive and negative cue point pairs.
[0118] In the fourth possible implementation provided based on the second possible implementation described above, before determining the positive and negative cue points in each prior positive and negative cue point pair based on each contour and feature similarity map, the processor 702 further performs the following steps when running the computer program stored in the memory 701:
[0119] Calculate the area of each contour separately;
[0120] Filtering is performed based on the contour area of each contour and preset filtering conditions.
[0121] Accordingly, based on the contour and feature similarity maps, the positive and negative cue points in each prior positive and negative cue point pair are determined, including:
[0122] Based on the preserved contour and feature similarity maps, the positive and negative cue points in each prior positive and negative cue point pair are determined.
[0123] In the fifth possible implementation provided based on the first possible implementation described above, the test image is segmented based on each prior positive and negative cue point pair, including:
[0124] The test image is used as one input to the preset segmentation model, and each prior positive and negative cue point pair is used as another input to the segmentation model to obtain at least one candidate mask output by the segmentation model.
[0125] Based on preset constraints, the candidate masks are filtered.
[0126] If there are reserved candidate masks, the target mask is determined based on the reserved candidate masks.
[0127] In the sixth possible implementation provided based on the fifth possible implementation described above, the candidate mask is filtered based on preset constraints, including:
[0128] Obtain the mask region corresponding to each candidate mask in the feature similarity map;
[0129] Calculate the average pixel value within each mask region;
[0130] Retain candidate masks corresponding to mask regions where the average pixel value is greater than a preset average pixel value threshold.
[0131] In a seventh possible implementation based on the fifth possible implementation described above, or based on the sixth possible implementation described above, determining the target mask based on the reserved candidate masks includes:
[0132] Obtain the quality scores of each retained candidate mask;
[0133] If at least two retained candidate masks overlap, the overlapping part is determined to belong to the first candidate mask, and the overlapping part is removed from the second candidate mask. The first candidate mask is the candidate mask with the highest quality score among the at least two retained candidate masks, and the second candidate mask is the other candidate mask besides the first candidate mask among the at least two retained candidate masks.
[0134] Each of the retained, non-overlapping candidate masks is selected as the target mask.
[0135] It should be understood that, in the embodiments of this application, the processor 702 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.
[0136] Memory 701 may include read-only memory and random access memory, and provides instructions and data to processor 702. Some or all of memory 701 may also include non-volatile random access memory. For example, memory 701 may also store device category information.
[0137] As can be seen from the above, in the image segmentation process of this application embodiment, an unlimited number of prior positive and negative cue point pairs are obtained by analyzing the feature similarity maps obtained based on the reference image and the test image. Each prior positive and negative cue point pair indicates a possible prior position of the target object in the test image, thereby obtaining an unlimited number of target masks. This scheme is applicable to test images containing multiple target objects as well as test images that do not contain target objects. It allows multiple target objects in the test image to be segmented, and avoids forcibly segmenting test images that do not contain target objects, resulting in incorrect segmentation results. This improves the accuracy of image segmentation and helps reduce the occurrence of undersegmentation or missegmentation.
[0138] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0139] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0140] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of external device software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0141] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules or units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.
[0142] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0143] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing associated hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer-readable storage device, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the contents of the aforementioned computer-readable storage media may be appropriately added to or subtracted from the contents according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media may not include electrical carrier signals and telecommunication signals.
[0144] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. An image segmentation method, characterized in that, include: Calculate the feature similarity between the target object in the reference image and each pixel in the test image to obtain a feature similarity map, wherein the reference image has been labeled with the mask of the target object; The feature similarity map is analyzed; If at least one pair of prior positive and negative cue points is obtained based on the analysis results, the test image is segmented based on each pair of prior positive and negative cue points, wherein the negative cue points in each pair of prior positive and negative cue points are the same. If at least one target mask is obtained based on the segmentation process, the segmentation result of the test image is obtained through the at least one target mask; The analysis of the feature similarity map includes: The feature similarity map is binarized to obtain a binary image; Contour extraction is performed on the binary image; When at least one contour is extracted, based on each contour and the feature similarity map, the positive and negative cue points in each of the prior positive and negative cue point pairs are determined, including: for each contour, finding the target point within the contour region corresponding to the contour in the feature similarity map, and taking the target point as a positive cue point in a prior positive and negative cue point pair, wherein the target point is the pixel with the largest pixel value in the contour region; and taking the pixel with the smallest pixel value in the feature similarity map as the negative cue point in all the prior positive and negative cue point pairs.
2. The image segmentation method as described in claim 1, characterized in that, Before determining the positive and negative cue points in each of the prior positive and negative cue point pairs based on each of the contours and the feature similarity maps, the method further includes: Calculate the area of each of the aforementioned contours; Filtering is performed based on the contour area of each contour and preset filtering conditions. Accordingly, determining the positive and negative cue points in each of the prior positive and negative cue point pairs based on each of the contours and the feature similarity maps includes: Based on the preserved contours and feature similarity maps, the positive and negative cue points in each of the prior positive and negative cue point pairs are determined.
3. The image segmentation method as described in claim 1, characterized in that, The segmentation process of the test image based on each of the prior positive and negative cue point pairs includes: The test image is used as one input to a preset segmentation model, and each of the prior positive and negative cue point pairs is used as another input to the segmentation model to obtain at least one candidate mask output by the segmentation model. The candidate masks are filtered based on preset constraints. If there are reserved candidate masks, the target mask is determined based on the reserved candidate masks.
4. The image segmentation method as described in claim 3, characterized in that, The filtering process for the candidate mask based on preset constraints includes: Each candidate mask is obtained from the mask region corresponding to the feature similarity map; Calculate the average pixel value within each of the mask regions; The candidate mask is retained for the mask region whose average pixel value is greater than a preset average pixel value threshold.
5. The image segmentation method as described in claim 3 or 4, characterized in that, Determining the target mask based on the reserved candidate masks includes: Obtain the quality scores of each of the retained candidate masks; If at least two retained candidate masks overlap, the overlapping portion is determined to belong to the first candidate mask, and the overlapping portion is removed from the second candidate mask. The first candidate mask is the candidate mask with the highest quality score among the at least two retained candidate masks, and the second candidate mask is the other candidate mask among the at least two retained candidate masks besides the first candidate mask. Each of the retained candidate masks that does not overlap is determined as the target mask.
6. An image segmentation apparatus, characterized in that, include: The calculation module is used to calculate the feature similarity between the target object in the reference image and each pixel of the test image to obtain a feature similarity map, wherein the reference image has been labeled with the mask of the target object. The analysis module is used to analyze the feature similarity map; The segmentation module is used to segment the test image based on each of the prior positive and negative cue point pairs when at least one prior positive and negative cue point pair is obtained based on the analysis results, wherein the negative cue points in each prior positive and negative cue point pair are the same. The result module is used to obtain the segmentation result of the test image through the at least one target mask when at least one target mask is obtained based on the result of the segmentation process; The analysis module includes: A binarization unit is used to perform binarization processing on the feature similarity map to obtain a binary image; Extraction unit, used for contour extraction in the binary image; The first determining unit is configured to, when at least one contour has been extracted, determine the positive and negative cue points in each of the prior positive and negative cue point pairs based on each contour and the feature similarity map. The first determining unit includes: The first determining subunit is used to find, for each contour, a target point within the contour region corresponding to the contour in the feature similarity map, and to use the target point as a positive cue point in a prior positive-negative cue point pair, wherein the target point is the pixel with the largest pixel value in the contour region; The second determining subunit is used to select the pixel with the smallest pixel value in the feature similarity map as the negative cue point in all the prior positive and negative cue point pairs.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.