A few-shot semantic segmentation method, system, device and medium

Abstract

The application discloses a few-sample semantic segmentation method, system, device and medium, and belongs to the technical field of image processing. The method is as follows: an input query image is segmented into a plurality of first candidate regions, and probability estimation is performed according to a plurality of sample images to obtain a probability estimation result; semantic mixed noise in the first candidate regions is removed according to the probability estimation result to obtain a plurality of second candidate regions; a confidence weight is determined according to the correlation between the sample images, and the probability estimation result is weighted and fused according to the confidence weight to obtain a fixed position confidence of each first candidate region; a core region is selected from the second candidate regions according to the fixed position confidence, and a first similarity between each second candidate region and the core region is calculated; and the semantic segmentation result is obtained by merging the two according to the first similarity. Therefore, by implementing the application, the problem of low image segmentation accuracy in a few-sample and complex actual scene can be solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a few-sample semantic segmentation method, system, device and medium. Background Technology

[0002] Semantic segmentation, a core task in computer vision, aims to perform pixel-level category prediction and region division in images. It is now widely used in various practical scenarios such as autonomous driving, medical imaging diagnosis, intelligent robots, and industrial inspection. Accurate semantic segmentation provides machines with deep scene visual understanding and complete target contour information, serving as a crucial foundation for building intelligent perception, analysis, and automated decision-making systems.

[0003] To reduce computational overhead and eliminate reliance on massive pixel-level annotations in practical applications, existing few-shot segmentation techniques have begun to introduce classless base models to pre-segment the query image to generate candidate regions, and then use a small number of support samples to directly match and merge these candidate regions with their features. However, this direct overall matching method suffers from severe performance bottlenecks in few-shot and complex real-world scenarios. The fundamental flaw lies in the fact that existing techniques treat the candidate regions segmented by the pre-model as perfect, semantically pure wholes, completely ignoring the microscopic semantic noise inevitably introduced when the classless base model only follows physical texture segmentation. This leads to background impurities being frequently and incorrectly introduced into the final segmentation result. Furthermore, existing techniques lack an effective mechanism for local automatic denoising and re-segmentation of candidate regions without relying on dense pixel annotations. Simultaneously, under the constraint of extremely scarce reference sample images, existing techniques rely solely on isolated local visual features to blindly assemble fragmented regions, making it highly prone to missed or incorrect selections when fragments overlap. Therefore, existing techniques exhibit low image segmentation accuracy in situations with few samples and complex real-world scenarios. Summary of the Invention

[0004] This application provides a few-sample semantic segmentation method, system, device, and medium, which can solve the problem of low image segmentation accuracy in existing technologies under conditions of few samples and complex real-world scenarios.

[0005] Some embodiments of this application provide a few-shot semantic segmentation method, including: The system acquires a query image input by the user and multiple preset sample images, segments the query image into several first candidate regions, and estimates the probability of each first candidate region relative to each of the sample images. Based on the probability estimation results, semantic noise in each of the first candidate regions is removed to obtain several second candidate regions. The confidence weight of each sample image is determined based on the correlation between the sample images, and the probability estimation results are weighted and fused using the confidence weight to obtain the location confidence corresponding to each first candidate region. Core regions are selected from the second candidate regions based on the location confidence, and a first similarity between each second candidate region and the core region is calculated based on the location confidence. Based on the first similarity, each of the second candidate regions is merged with the core region to obtain the semantic segmentation result.

[0006] Compared with the prior art, the above embodiments have the following beneficial effects: This application obtains the matching degree between each candidate region and the sample image through probability estimation, and then uses the probability estimation result to remove the background noise mixed in the candidate region, which can reduce the interference of the noise region on the subsequent core region selection; at the same time, the confidence weight of each sample image is determined by the correlation between the sample images, and the probability estimation result is weighted and fused to obtain the localization confidence, which can reduce the influence of poor quality sample images in the fusion process; on this basis, the localization confidence is used to select the core region and calculate the similarity, and the candidate regions that meet the requirements are merged with the core region, which can avoid the core region positioning deviation caused by the uneven quality of sample images, thereby improving the accuracy of semantic segmentation with few samples in complex scenes.

[0007] Further, based on the probability estimation results, semantic noise in each of the first candidate regions is removed to obtain several second candidate regions, including: A first type of activation map of the query image is generated based on the probability estimation results, and an upper limit and a lower limit of activation value are determined based on the first type of activation map; wherein, the first type of activation map includes the activation value corresponding to each pixel in the query image; Based on the upper limit of the activation value and the lower limit of the activation value, a mixed region containing the semantic noise is selected from the first candidate region, and the corresponding foreground region is segmented from the mixed region. The foreground regions and the first candidate regions that were not identified as mixed regions are combined together to form the plurality of second candidate regions.

[0008] Compared with the prior art, the above embodiments have the following beneficial effects: This application generates a class activation map of the query image through probability estimation results, and determines the upper and lower thresholds based on the activation value distribution of the class activation map, thereby filtering out semantic noise in the candidate region and segmenting the foreground region from the mixed region. It can split the candidate region with mixed foreground and background into a semantically pure region without relying on additional annotation information, and avoid mistaking background noise as part of the target object to participate in subsequent merging.

[0009] Further, the step of filtering mixed regions containing semantic noise from the first candidate region based on the upper limit of the activation value and the lower limit of the activation value, and segmenting the corresponding foreground region from the mixed region, includes: For each of the first candidate regions, calculate the variance of the activation values ​​of each pixel within the first candidate region in the first type of activation map; For each first candidate region, if there is a first pixel with an activation value greater than the upper limit of the activation value, a second pixel with an activation value less than the lower limit of the activation value, and the variance of the activation value corresponding to the first candidate region is greater than a preset variance threshold, then the first candidate region is determined to be a mixed region with semantic noise. For each of the blended regions, the corresponding foreground region is segmented from the corresponding blended region.

[0010] Compared with the prior art, the above embodiments have the following beneficial effects: For each candidate region, this application determines whether the region is a mixed region by combining the variance of its internal activation values ​​and whether there are activation points exceeding the upper and lower thresholds. The dual determination conditions can reduce misjudgment of pure regions. At the same time, activation points above the upper limit in the mixed region are used as positive prompts and activation points below the lower limit are used as negative prompts to drive foreground segmentation. The activation differences within the region can be used to guide the segmentation process, thereby accurately separating the foreground part from the mixed region.

[0011] Further, each of the sample images corresponds to a support mask; the estimation result of the probability of each first candidate region relative to each of the sample images includes: By using global flat pooling, a first feature vector corresponding to each of the sample images is extracted, and a second feature vector corresponding to each of the first candidate regions is extracted. Calculate the correlation between each first feature vector and each second feature vector, and determine the class consistency probability of each first candidate region relative to each sample image based on the correlation; Based on the category consistency probability and the sample image, a second type activation map is generated for each of the first candidate regions. By binarizing each second type activation map, a binary map corresponding to each first candidate region is obtained. The intersection-union ratio (IUU) of each binary map and each support mask is then calculated to obtain the likelihood probability of each first candidate region relative to each sample image. The category consistency probability and the likelihood probability are used as the probability estimation results of the corresponding first candidate region relative to the corresponding sample image.

[0012] Compared with the prior art, the above embodiments have the following beneficial effects: This application extracts feature vectors through global average pooling, calculates the class consistency probability between candidate regions and sample images, and generates a class activation map based on this probability, thereby activating the class... Figure 2 After value-based evaluation, the likelihood probability is obtained by calculating the intersection-union ratio (IUU) with the support mask. Using both as the probability estimation result can avoid the one-sided representation of the location confidence of the candidate region by a single index, thus providing a more reliable basis for subsequent noise removal and confidence fusion.

[0013] Further, the step of calculating the first similarity between each second candidate region and the core region based on the location confidence includes: Calculate the first cosine similarity between each second candidate region and the second feature vector corresponding to the core region; The third feature vector of the query image is extracted. Through the cross attention mechanism, the semantic context correlation between the third feature vector and the second feature vector corresponding to each second candidate region is calculated. The context feature vector corresponding to each second candidate region is generated based on the semantic context correlation. Calculate the second cosine similarity between each of the context feature vectors and the second feature vector corresponding to the core region; For each second candidate region, the second cosine similarity corresponding to the second candidate region is weighted according to the location confidence to obtain the corresponding global similarity. The first cosine similarity and the global similarity corresponding to each second candidate region are combined to obtain the first similarity corresponding to each second candidate region.

[0014] Compared with the prior art, the above embodiments have the following beneficial effects: When calculating the similarity between the second candidate region and the core region, this application simultaneously adopts the cosine similarity of local feature vectors and the global context similarity based on the cross-attention mechanism, and uses the positional confidence as a weight to weight the global similarity. This can take into account the local association of spatially adjacent regions and the contextual dependency of distant regions, so that candidate regions that have long-range semantic associations but are not spatially adjacent can still be correctly identified and merged. At the same time, the confidence weighting reduces the interference of false detection regions in the merging decision.

[0015] Further, the step of determining the confidence weight of each sample image based on the correlation between the sample images, and using the confidence weight to perform weighted fusion of the probability estimation results to obtain the location confidence corresponding to each first candidate region, includes: Calculate the second similarity between any two of the sample images; For each of the sample images, the average of the second similarity between the sample image and the remaining sample images is calculated to obtain the average correlation of each sample image; The average correlation degree is subjected to exponential normalization to obtain the confidence weight corresponding to each sample image. The probability estimation results are weighted and fused according to the confidence weights to obtain the location confidence corresponding to each first candidate region.

[0016] Compared with the prior art, the above embodiments have the following beneficial effects: This application determines the average correlation of each sample image by calculating the similarity between each pair of sample images, and uses exponential normalization to obtain the confidence weight, so that sample images that are consistent with most sample images receive higher weights, while sample images that are significantly different from most sample images receive lower weights, thereby suppressing the negative impact of low-quality sample images caused by factors such as occlusion, blurring or abnormal lighting on the location confidence calculation during fusion.

[0017] Further, the step of merging each of the second candidate regions with the core region based on the first similarity to obtain the semantic segmentation result includes: Filter the third candidate region from the second candidate region whose first similarity is greater than a preset threshold; The semantic segmentation result is obtained by merging each of the third candidate regions with the core region.

[0018] Compared with the prior art, the above embodiments have the following beneficial effects: This application filters the first similarity of the second candidate region by setting a preset threshold, and merges only the candidate regions that exceed the threshold with the core region. This can filter out noise regions with low similarity to the target object, so that the final output semantic segmentation result only contains candidate regions with high consistency with the core region, reducing the possibility of the background region being missegmented as the target object.

[0019] Another embodiment of this application also provides a few-shot semantic segmentation system, which is used to implement the steps of the few-shot semantic segmentation method of this application.

[0020] Another embodiment of this application also provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the steps of the few-sample semantic segmentation method of this application.

[0021] Another embodiment of this application also provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform steps of the few-sample semantic segmentation method of this application. Attached Figure Description

[0022] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a few-sample semantic segmentation method provided in some embodiments of this application; Figure 2 This is a schematic diagram of a query image provided in some embodiments of this application; Figure 3 This is a schematic diagram of a sample image provided in some embodiments of this application; Figure 4 This is a schematic diagram of an image of a candidate region after image segmentation provided in some embodiments of this application; Figure 5 This is a schematic diagram of a class activation graph provided in some embodiments of this application; Figure 6 This is a schematic diagram of a core area provided in some embodiments of this application; Figure 7 This is a schematic diagram of a semantic segmentation result provided in some embodiments of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0026] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0028] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0029] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0030] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0031] To reduce computational overhead and eliminate reliance on massive pixel-level annotations in practical applications, existing few-shot segmentation techniques have begun to introduce classless base models to pre-segment the query image to generate candidate regions, and then use a small number of support samples to directly match and merge these candidate regions with their features. However, this direct overall matching method suffers from severe performance bottlenecks in few-shot and complex real-world scenarios. The fundamental flaw lies in the fact that existing techniques treat the candidate regions segmented by the pre-model as perfect, semantically pure wholes, completely ignoring the microscopic semantic noise that inevitably arises when the classless base model only follows physical texture segmentation. This leads to background impurities being frequently and incorrectly introduced into the final segmentation result. Furthermore, existing techniques lack an effective mechanism for local automatic denoising and re-segmentation of candidate regions without relying on dense pixel annotations. At the same time, under the constraint of extremely scarce reference sample images, existing techniques rely solely on isolated local visual features to blindly assemble fragmented regions, failing to establish long-range global contextual relationships between discrete fragments and the core anchor points of objects under weak supervision signals. This results in fragments being easily missed or misselected when they overlap.

[0032] Please refer to Figure 1 To address the issue of low image segmentation accuracy in existing technologies when dealing with limited samples and complex real-world scenarios, this application provides a limited-sample semantic segmentation method, comprising the following steps S101 to S105: S101: Obtain the query image input by the user and multiple preset sample images, segment the query image into several first candidate regions, and estimate the probability estimation result of each first candidate region relative to each of the sample images.

[0033] Furthermore, in some embodiments of this application, each of the sample images corresponds to a support mask.

[0034] Preferably, in some embodiments of this application, reference is made to Figure 2 For a user-input query image, Figure 3 The sample image is a preset sample image. It can be seen that the sample image includes a support mask, which is used to mark the target object corresponding to the preset semantic category in the sample image. The support mask is a binary mask.

[0035] Preferably, in some embodiments of this application, segmenting the query image into several first candidate regions includes: dividing the query image... Inputting into the pre-trained semantic segmentation model SAM, its visual structure-based segmentation capability yields a combination of first candidate regions that do not contain semantic information of a specific category, which can be formally represented as: ; in, The semantic segmentation model SAM generates the first... The first candidate region For the semantic segmentation model SAM, This represents the total number of first candidate regions, where each first candidate region is represented as a connected pixel region. (See reference) Figure 4 , which are the first candidate regions obtained by the final segmentation. It can be understood that each first candidate region also corresponds to a region mask.

[0036] Furthermore, in some embodiments of this application, estimating the probability estimation result of each first candidate region relative to each of the sample images includes the following steps S1011 to S1015: S1011: Extract the first feature vector corresponding to each sample image and the second feature vector corresponding to each first candidate region through global flat pooling.

[0037] Preferably, in some embodiments of this application, the set of sample images is set as follows: Query image and sample image set The input is fed into the backbone of a pre-trained convolutional neural network, and features are extracted to obtain the query feature map. and sample feature set .

[0038] Furthermore, for any first candidate region The corresponding region mask is compared with the query feature map. Element-wise multiplication is performed to obtain the feature map of the first candidate region, and global average pooling is then applied to the feature map of the first candidate region to obtain the first feature vector of the first candidate region. ; The calculation formula is as follows: ; in, Indicates global average pooling. This indicates element-wise multiplication.

[0039] For any sample image For its sample feature map Perform global average pooling to obtain the second feature vector. ; The calculation formula is as follows: ; in, This indicates global average pooling.

[0040] S1012: Calculate the correlation between each of the first feature vectors and each of the second feature vectors, and determine the class consistency probability of each first candidate region relative to each of the sample images based on the correlation.

[0041] Preferably, in some embodiments of this application, the category consistency probability is determined by the first feature vector. With the second eigenvector The images are concatenated and then fed into a multilayer perceptron with a sigmoid activation function for binary classification to predict the query image. and sample images Whether they belong to the same category or not, the classification probability output by the network is the category consistency probability. ; The calculation formula is as follows: ; in, This represents the sigmoid function. Indicates feature splicing, It is a multilayer perceptron.

[0042] S1013: Generate a second type activation map corresponding to each of the first candidate regions based on the category consistency probability and the sample image.

[0043] Preferably, in some embodiments of this application, the second type of activation graph is generated by: utilizing class consistency probability. Backpropagation is performed as a supervisory signal to calculate the feature map of the samples. The gradient is calculated, and the LayerCAM algorithm is used to generate the second type of activation map based on the gradient. (Reference) Figure 5 The image shows the final generated class activation graph.

[0044] S1014: By binarizing each of the second type of activation maps, a binary map corresponding to each of the first candidate regions is obtained, and the intersection-union ratio (IUU) of each of the binary maps and each of the support masks is calculated to obtain the likelihood probability of each of the first candidate regions relative to each of the sample images.

[0045] Preferably, in some embodiments of this application, the formula for calculating the likelihood probability is: Support mask of the sample image corresponding to the sample feature map after binarizing the second type of activation map. Calculate the intersection-union ratio to obtain the likelihood probability. ; The calculation formula is as follows: ; in, This indicates that the LayerCAM algorithm is used to generate the class activation graph. This indicates a binarization operation. This indicates the operation for calculating intersection-union ratio.

[0046] S1015: Use the category consistency probability and the likelihood probability as the probability estimation result of the corresponding first candidate region relative to the corresponding sample image.

[0047] This application extracts feature vectors using global average pooling, calculates the class consistency probability between candidate regions and sample images, and generates a class activation map based on this probability, thereby activating the class... Figure 2 After value-based evaluation, the likelihood probability is obtained by calculating the intersection-union ratio (IUU) with the support mask. Using both as the probability estimation result can avoid the one-sided representation of the location confidence of the candidate region by a single index, thus providing a more reliable basis for subsequent noise removal and confidence fusion.

[0048] S102: Based on the probability estimation results, semantic noise in each of the first candidate regions is removed to obtain several second candidate regions.

[0049] Furthermore, in some embodiments of this application, the step of removing semantic noise from each of the first candidate regions based on the probability estimation results to obtain a plurality of second candidate regions includes the following steps S1021 to S1023: S1021: Generate a first type of activation map of the query image based on the probability estimation results, and determine the upper limit and lower limit of activation values ​​based on the first type of activation map; wherein, the first type of activation map includes the activation value corresponding to each pixel in the query image.

[0050] Preferably, in some embodiments of this application, the first type of activation graph is generated through the following process: utilizing class consistency probability. Backpropagation is performed as a supervisory signal to compute information about the query feature map. The gradient is calculated, and the LayerCAM algorithm is used to generate the original class activation map based on the gradient. The original class activation map is then upsampled to the same size as the query image and normalized. The first type of activation graph is obtained. .

[0051] Preferably, in some embodiments of this application, the upper limit of the activation value is... and the lower limit of the activation value The results are obtained by calculating using the following formulas: ; ; in, Indicates the first Quantiles and The quantile coefficients are determined in advance based on the validation set, and .

[0052] S1022: Based on the upper limit of the activation value and the lower limit of the activation value, filter the mixed regions containing the semantic noise from the first candidate regions, and segment the corresponding foreground regions from the mixed regions.

[0053] Furthermore, in some embodiments of this application, the step of filtering mixed regions containing semantic noise from the first candidate region based on the upper limit of the activation value and the lower limit of the activation value, and segmenting the corresponding foreground region from the mixed region, includes the following steps T221 to T223: T221: For each of the first candidate regions, calculate the variance of the activation values ​​of each pixel within the first candidate region in the first type of activation map.

[0054] Preferably, in some embodiments of this application, the variance of the activation value is calculated using the following formula: ; ; in, Indicates the first candidate region Internal pixels, For pixels The corresponding activation value, Indicates the first candidate region The number of internal pixels, Indicates the first candidate region The average value of the activation values ​​corresponding to each pixel within the unit.

[0055] T222: For each first candidate region, if there is a first pixel with an activation value greater than the upper limit of the activation value, a second pixel with an activation value less than the lower limit of the activation value, and the variance of the activation value corresponding to the first candidate region is greater than a preset variance threshold, then the first candidate region is determined to be a mixed region with semantic noise.

[0056] Preferably, in some embodiments of this application, for each first candidate region ,Will Internal satisfaction The pixel is determined as the first pixel, which satisfies... The pixel is determined as the second pixel. If the first candidate region The first pixel and the second pixel exist simultaneously, and the variance... Greater than the preset variance threshold Then the first candidate region is determined. This is a mixed region.

[0057] T223: For each of the blended regions, segment out the corresponding foreground region from the corresponding blended region.

[0058] Preferably, in some embodiments of this application, for candidate regions determined to be mixed regions... Set its corresponding bounding box Using the first pixel as the input image and the second pixel as the negative cues, the semantic segmentation model (SAM) is called again to segment the bounding box, obtaining the foreground mask inside the bounding box. and background mask ;Will The original first candidate region is replaced by the split foreground region. This is to complete the update of the candidate regions.

[0059] This application determines whether a region is a mixed region by considering the variance of its internal activation values ​​and whether there are activation points exceeding the upper and lower thresholds for each candidate region. This dual determination condition can reduce misjudgment of pure regions. At the same time, activation points above the upper limit in the mixed region are used as positive cues and activation points below the lower limit are used as negative cues to drive foreground segmentation. This can use the activation differences within the region to guide the segmentation process, thereby accurately separating the foreground part from the mixed region.

[0060] S1023: Combine each of the foreground regions and the first candidate regions that were not determined to be mixed regions into the plurality of second candidate regions.

[0061] This application generates a class activation map of the query image based on the probability estimation results, and determines the upper and lower thresholds based on the activation value distribution of the class activation map. This filters out semantic noise in the candidate region and segments the foreground region from the mixed region. It can split the candidate region with mixed foreground and background into a semantically pure region without relying on additional annotation information, and avoids the background noise being mistakenly included as part of the target object in the subsequent merging.

[0062] S103: Determine the confidence weight of each sample image based on the correlation between each sample image, and use the confidence weight to perform weighted fusion of the probability estimation results to obtain the location confidence corresponding to each first candidate region.

[0063] Furthermore, in some embodiments of this application, the step of determining the confidence weight of each sample image based on the correlation between the sample images, and using the confidence weight to perform weighted fusion of the probability estimation results to obtain the location confidence corresponding to each first candidate region, includes the following steps S1031 to S1034: S1031: Calculate the second similarity between any two of the sample images.

[0064] Preferably, in some embodiments of this application, the second similarity is calculated according to the following formula: For any two sample images and Calculate their cosine similarity using the following formula. : ; in, This is the cosine similarity operator.

[0065] S1032: For each of the sample images, calculate the average of the second similarity between the sample image and the remaining sample images to obtain the average correlation of each sample image.

[0066] S1033: Perform exponential normalization on each of the average correlation degrees to obtain the confidence weight corresponding to each of the sample images.

[0067] Preferably, in some embodiments of this application, the formula for calculating the confidence weight is: For any sample image ,according to The confidence weight of a sample image is calculated based on its average similarity to other sample images. ; The calculation formula is as follows: ; in, for Average similarity to other sample images; This is the exponentiation operator.

[0068] S1034: The probability estimation results are weighted and fused according to the confidence weights to obtain the location confidence corresponding to each first candidate region.

[0069] Preferably, in some embodiments of this application, the location confidence is calculated according to the following formula: All The joint probability is obtained by multiplying the likelihood probability and class consistency probability calculated from each supporting image, and then the average value is used as the second candidate region. joint probability That is, location reliability; The calculation formula is as follows: ; in, and These represent the second candidate regions. With the Likelihood probability and class consistency probability among sample images.

[0070] This application determines the average correlation of each sample image by calculating the similarity between each pair of sample images, and uses exponential normalization to obtain the confidence weight, so that sample images that are consistent with most sample images receive higher weights, while sample images that are significantly different from most sample images receive lower weights. This suppresses the negative impact of low-quality sample images caused by factors such as occlusion, blurring, or abnormal lighting on the location confidence calculation during fusion.

[0071] S104: Select a core region from the second candidate regions based on the location confidence, and calculate a first similarity between each second candidate region and the core region based on the location confidence.

[0072] Preferably, in some embodiments of the application, the second candidate region with the highest location confidence is selected as the core region. ; The calculation formula is as follows: ; in, Indicates the second candidate region The location reliability of the corresponding likelihood probability and class consistency probability. For example... Figure 6 As shown, this is the core area that was ultimately selected.

[0073] Furthermore, in some embodiments of this application, the step of calculating the first similarity between each second candidate region and the core region based on the location confidence includes the following steps S1041 to S1045: S1041: Calculate the first cosine similarity between each of the second candidate regions and the second feature vector corresponding to the core region.

[0074] Preferably, in some embodiments of this application, the formula for calculating the first cosine similarity is: For each second candidate region Cosine similarity is used to calculate its similarity with the core region. Local similarity between That is, the first cosine similarity; The calculation formula is as follows: ; in, It is the second candidate region The second eigenvector, It is the core area The second eigenvector.

[0075] S1042: Extract the third feature vector of the query image, calculate the semantic context correlation between the third feature vector and the second feature vector corresponding to each second candidate region through a cross-attention mechanism, and generate the context feature vector corresponding to each second candidate region based on the semantic context correlation.

[0076] Preferably, in some embodiments of this application, the context feature vector is calculated using the following formula: The third feature vector of the query image is calculated using a cross-attention mechanism. The second feature vector of each second candidate region Contextual feature vectors between ; The calculation formula is as follows: ; in, For feature dimension, , , The projection matrix is ​​learnable. This is the softmax activation function.

[0077] S1043: Calculate the second cosine similarity between each of the context feature vectors and the second feature vector corresponding to the core region.

[0078] S1044: For each second candidate region, the second cosine similarity corresponding to the second candidate region is weighted according to the location confidence to obtain the corresponding global similarity.

[0079] Preferably, in some embodiments of this application, the formula for calculating the global similarity is: For each second candidate region Calculate its context feature vector The second feature vector of the core region The cosine similarity between them, and the location confidence corresponding to the second candidate region. We perform weighted averages to obtain the global similarity. ; The calculation formula is as follows: in, Indicates candidate region The location reliability of the corresponding likelihood probability and class consistency probability.

[0080] S1045: Combine the first cosine similarity and the global similarity corresponding to each second candidate region to obtain the first similarity corresponding to each second candidate region.

[0081] Preferably, in some embodiments of this application, for each second candidate region Through a learnable parameter The global similarity Similarity to the first cosine Combine the results and calculate the final first similarity. ; The calculation formula is as follows: .

[0082] When calculating the similarity between the second candidate region and the core region, this application simultaneously employs the cosine similarity of local feature vectors and the global contextual similarity based on the cross-attention mechanism, and uses the location confidence as a weight to weight the global similarity. This approach can take into account both the local associations of spatially adjacent regions and the contextual dependencies of distant regions, enabling candidate regions that have long-range semantic associations but are not spatially adjacent to be correctly identified and merged. At the same time, the confidence weighting reduces the interference of false detection regions in the merging decision.

[0083] S105: Based on the first similarity, merge each of the second candidate regions with the core region to obtain the semantic segmentation result.

[0084] Furthermore, in some embodiments of this application, the step of merging each of the second candidate regions with the core region based on the first similarity to obtain a semantic segmentation result includes: Filter the third candidate region from the second candidate region whose first similarity is greater than a preset threshold; The semantic segmentation result is obtained by merging each of the third candidate regions with the core region.

[0085] Preferably, in some embodiments of this application, for each second candidate region If their first similarity Then the second candidate region is determined. With core area Belonging to the same semantic category; core area The region is merged with all second candidate regions belonging to the same semantic category to obtain the final semantic segmentation result, such as... Figure 7 As shown.

[0086] This application filters the first similarity of the second candidate region by setting a preset threshold, and merges only the candidate regions that exceed the threshold with the core region. This can filter out noisy regions with low similarity to the target object, so that the final semantic segmentation result only contains candidate regions with high consistency with the core region, reducing the possibility of the background region being missegmented as the target object.

[0087] In summary, the few-shot semantic segmentation method provided in this application has the following advantages compared with the prior art: This application obtains the matching degree between each candidate region and the sample image through probability estimation, and then uses the probability estimation result to remove background noise mixed in the candidate region, which can reduce the interference of noisy regions on the subsequent selection of core regions; at the same time, the confidence weight of each sample image is determined by the correlation between sample images, and the probability estimation results are weighted and fused to obtain the localization confidence, which can reduce the influence of poor quality sample images in the fusion process; on this basis, the localization confidence is used to select the core region and calculate the similarity, and the candidate regions that meet the requirements are merged with the core region, which can avoid the core region positioning deviation caused by the uneven quality of sample images, thereby improving the accuracy of few-shot semantic segmentation in complex scenes.

[0088] Based on the above method embodiments, one embodiment of this application provides a few-shot semantic segmentation system, which can implement the few-shot semantic segmentation method provided in any of the above method embodiments of this application.

[0089] It should be noted that the system embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Based on the above embodiments of the few-shot semantic segmentation method, another embodiment of this application provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the few-shot semantic segmentation method of any embodiment of this application.

[0091] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete this application. The one or more module units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0092] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0093] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0094] Based on the above-described method embodiments, another embodiment of this application provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the few-sample semantic segmentation method described in any of the above-described method embodiments of this application.

[0095] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

Claims

1. A few-sample semantic segmentation method, characterized in that, include: The system acquires a query image input by the user and multiple preset sample images, segments the query image into several first candidate regions, and estimates the probability of each first candidate region relative to each of the sample images. Based on the probability estimation results, semantic noise in each of the first candidate regions is removed to obtain several second candidate regions. The confidence weight of each sample image is determined based on the correlation between the sample images, and the probability estimation results are weighted and fused using the confidence weight to obtain the location confidence corresponding to each first candidate region. Core regions are selected from the second candidate regions based on the location confidence, and a first similarity between each second candidate region and the core region is calculated based on the location confidence. Based on the first similarity, each of the second candidate regions is merged with the core region to obtain the semantic segmentation result.

2. The few-sample semantic segmentation method as described in claim 1, characterized in that, Based on the probability estimation results, semantic noise in each of the first candidate regions is removed to obtain several second candidate regions, including: A first type of activation map of the query image is generated based on the probability estimation results, and an upper limit and a lower limit of activation value are determined based on the first type of activation map; wherein, the first type of activation map includes the activation value corresponding to each pixel in the query image; Based on the upper limit of the activation value and the lower limit of the activation value, a mixed region containing the semantic noise is selected from the first candidate region, and the corresponding foreground region is segmented from the mixed region. The foreground regions and the first candidate regions that were not identified as mixed regions are combined together to form the plurality of second candidate regions.

3. The few-sample semantic segmentation method as described in claim 2, characterized in that, The step of filtering mixed regions containing semantic noise from the first candidate region based on the upper limit and the lower limit of the activation value, and segmenting the corresponding foreground region from the mixed region, includes: For each of the first candidate regions, calculate the variance of the activation values ​​of each pixel within the first candidate region in the first type of activation map; For each first candidate region, if there is a first pixel with an activation value greater than the upper limit of the activation value, a second pixel with an activation value less than the lower limit of the activation value, and the variance of the activation value corresponding to the first candidate region is greater than a preset variance threshold, then the first candidate region is determined to be a mixed region with semantic noise. For each of the blended regions, the corresponding foreground region is segmented from the corresponding blended region.

4. The few-sample semantic segmentation method as described in claim 1, characterized in that, Each of the sample images corresponds to a support mask; The estimation of the probability of each first candidate region relative to each of the sample images includes: By using global flat pooling, a first feature vector corresponding to each of the sample images is extracted, and a second feature vector corresponding to each of the first candidate regions is extracted. Calculate the correlation between each first feature vector and each second feature vector, and determine the class consistency probability of each first candidate region relative to each sample image based on the correlation; Based on the category consistency probability and the sample image, a second type activation map is generated for each of the first candidate regions. By binarizing each second type activation map, a binary map corresponding to each first candidate region is obtained. The intersection-union ratio (IUU) of each binary map and each support mask is then calculated to obtain the likelihood probability of each first candidate region relative to each sample image. The category consistency probability and the likelihood probability are used as the probability estimation results of the corresponding first candidate region relative to the corresponding sample image.

5. A few-sample semantic segmentation method as described in claim 4, characterized in that, The step of calculating the first similarity between each second candidate region and the core region based on the location confidence includes: Calculate the first cosine similarity between each second candidate region and the second feature vector corresponding to the core region; The third feature vector of the query image is extracted. Through the cross attention mechanism, the semantic context correlation between the third feature vector and the second feature vector corresponding to each second candidate region is calculated. The context feature vector corresponding to each second candidate region is generated based on the semantic context correlation. Calculate the second cosine similarity between each of the context feature vectors and the second feature vector corresponding to the core region; For each second candidate region, the second cosine similarity corresponding to the second candidate region is weighted according to the location confidence to obtain the corresponding global similarity. The first cosine similarity and the global similarity corresponding to each second candidate region are combined to obtain the first similarity corresponding to each second candidate region.

6. The few-sample semantic segmentation method as described in claim 1, characterized in that, The step of determining the confidence weight of each sample image based on the correlation between the sample images, and using the confidence weight to perform weighted fusion of the probability estimation results to obtain the location confidence corresponding to each first candidate region, includes: Calculate the second similarity between any two of the sample images; For each of the sample images, the average of the second similarity between the sample image and the remaining sample images is calculated to obtain the average correlation of each sample image; The average correlation degree is subjected to exponential normalization to obtain the confidence weight corresponding to each sample image. The probability estimation results are weighted and fused according to the confidence weights to obtain the location confidence corresponding to each first candidate region.

7. The few-sample semantic segmentation method as described in claim 1, characterized in that, The step of merging each of the second candidate regions with the core region based on the first similarity to obtain the semantic segmentation result includes: Filter the third candidate region from the second candidate region whose first similarity is greater than a preset threshold; The semantic segmentation result is obtained by merging each of the third candidate regions with the core region.

8. A few-shot semantic segmentation system, characterized in that, The few-shot semantic segmentation system is used to implement a few-shot semantic segmentation method as described in any one of claims 1 to 7.

9. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements a few-sample semantic segmentation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform a few-sample semantic segmentation method as described in any one of claims 1 to 7.

Images