Segmentation method, device, medium and electronic equipment of target of interest

Abstract

The application provides a segmentation method and device of an object of interest, a medium and an electronic equipment. The segmentation method comprises: acquiring an image to be segmented and label information of the image to be segmented; inputting the image to be segmented and the label information of the image to be segmented into a pre-trained object of interest segmentation model to determine a feature image of the image to be segmented and a preliminary clustering feature; performing block clustering processing on the feature image of the image to be segmented to determine an object block contained in the image to be segmented; and segmenting the object of interest of the image to be segmented according to the feature similarity between the object block of the image to be segmented and the object blocks. Through the present disclosure, the object of interest is effectively segmented with less labeled information, the privacy of image data is ensured, and the segmentation efficiency of the object of interest is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method, apparatus, medium, and electronic device for segmenting a target of interest. Background Technology

[0002] In image capture scenarios, deep learning-based neural network algorithms struggle to focus on extracting information from the region of interest, especially for identifying targets in complex scenes. With limited training samples and label information, deep learning-based neural network algorithms tend to concentrate on areas with high pixel contrast in the image or targets with drastic changes in optical flow in video images.

[0003] Deep neural networks with multiple layers can be designed to extract deep semantic information from images and train models to identify targets of interest. However, excessively deep neural network layers can easily lead to the loss of image feature information. Furthermore, fully supervised instance segmentation algorithms require pixel-level annotation, which consumes significant time and impacts efficiency. Weakly supervised instance segmentation algorithms require concatenating the weak supervision information extraction module and the model training module to determine the instance segmentation model, resulting in an overly redundant model. Moreover, the model is prone to data leakage during training, leading to poor generalization ability. Summary of the Invention

[0004] In view of the deficiencies in the prior art, the purpose of this disclosure is to provide a method, apparatus, medium and electronic device for segmenting a target of interest.

[0005] To achieve the above objectives, according to a first aspect of this disclosure, a method for segmenting a target of interest is provided, comprising:

[0006] Obtain the image to be segmented and its label information, wherein the label information includes vertex information of the smallest rectangular region containing the target of interest on the image to be segmented;

[0007] The image to be segmented and its label information are input into the feature extraction module of a pre-trained target of interest segmentation model to determine the feature image and preliminary clustering features of the image to be segmented. The feature extraction module includes a U-shaped backbone network, which is used to extract deep semantic features of the image.

[0008] The feature images of the image to be segmented are subjected to block clustering processing to determine the target blocks contained in the image to be segmented;

[0009] The target of interest in the image to be segmented is segmented based on the feature similarity between target blocks in the image to be segmented.

[0010] Optionally, the step of performing block-based clustering processing on the feature image of the image to be segmented to determine the target blocks contained in the image to be segmented includes:

[0011] The feature information of each image channel of the feature image to be segmented is divided into mutually independent first blocks according to a preset size;

[0012] The first blocks at the same position in each image channel of the feature image of the image to be segmented are superimposed to determine the target block contained in the image to be segmented.

[0013] Optionally, the method further includes:

[0014] Based on the location of each target block, determine the feature matrix of the feature image of the image to be segmented after the block clustering process;

[0015] The similarity between target blocks in the image to be segmented is determined by multiplying the feature matrix with its transpose.

[0016] Optionally, the step of inputting the image to be segmented and its label information into a pre-trained feature extraction module to determine the preliminary clustering features of the image to be segmented includes:

[0017] The image to be segmented and its label information are input into the U-shaped backbone network to determine the deep semantic features of the image to be segmented.

[0018] Optionally, segmenting the target of interest in the image to be segmented based on the similarity between target blocks in the image to be segmented includes:

[0019] The target matrix of the target of interest is determined based on the similarity between target blocks in the image to be segmented.

[0020] Based on the target matrix of the target of interest, determine the second minimum eigenvector of the generalized feature system;

[0021] The target of interest in the image to be segmented is segmented based on the second smallest feature vector of the generalized feature system.

[0022] Optionally, the pre-trained target of interest segmentation model includes a feature extraction module and a block segmentation module;

[0023] The method for determining the segmentation model of the target of interest includes:

[0024] Obtain training image samples and their label information, wherein the label information of the training image samples includes vertex information of the smallest rectangular region where the target of interest is located on the training image sample;

[0025] The forward deep residual network and the backward deep residual network are concatenated to determine the main framework of the U-shaped backbone network. An indexed max-pooling layer is added after each level of the forward deep residual network within the main framework of the U-shaped backbone network. The output features of each level of the forward deep residual network are connected to the corresponding input features of the same level in the backward deep residual network using channel concatenation to determine the feature extraction module. The backward deep residual network is an inverse symmetric structure of the forward deep residual network. The indexed max-pooling layer is used to adjust the feature image size of the image to be segmented. The U-shaped backbone network uses the forward deep residual network to encode the image to be segmented, and the U-shaped backbone network uses the backward deep residual network to decode the image to be segmented.

[0026] The feature extraction module and the block segmentation module are connected to determine the segmentation model for the target of interest;

[0027] According to a preset loss function, the training image samples and their label information are input into the target of interest segmentation model for model training, thereby determining the pre-trained target of interest segmentation model.

[0028] Optionally, the vertex information of the smallest rectangular region containing the target of interest in the image to be segmented includes the coordinates of the top-left vertex and the bottom-right vertex of the smallest rectangular region.

[0029] According to a first aspect of this disclosure, a segmentation apparatus for a target of interest is provided, comprising:

[0030] The first acquisition module is used to acquire the image to be segmented and the label information of the image to be segmented, wherein the label information includes vertex information of the smallest rectangular region where the target of interest is located on the image to be segmented;

[0031] The first determining module is used to input the image to be segmented and the label information of the image to be segmented into the feature extraction module of the pre-trained target of interest segmentation model to determine the feature image and preliminary clustering features of the image to be segmented. The feature extraction module includes a U-shaped backbone network, which is used to extract deep semantic features of the image.

[0032] The second determining module is used to perform block-based clustering processing on the preliminary clustering features of the image to be segmented, and to determine the target blocks contained in the image to be segmented.

[0033] The segmentation module is used to segment the target of interest in the image to be segmented based on the feature similarity between target blocks in the image to be segmented.

[0034] According to a third aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the segmentation method for the target of interest provided in the first aspect of this disclosure.

[0035] According to a fourth aspect of this disclosure, an electronic device is provided, comprising:

[0036] A memory on which computer programs are stored;

[0037] A processor for executing the computer program in the memory to implement the steps of the segmentation method for the target of interest provided in the first aspect of this disclosure.

[0038] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0039] The above technical solution acquires the image to be segmented and its label information. The label information consists of the vertex information of the smallest rectangular region containing the target of interest (ROI) on the image to be segmented. This reduces the amount of label information used in segmenting ROI, thus decreasing the time spent on label annotation and improving the segmentation efficiency. The acquired image to be segmented and its label information are input into the feature extraction model of a pre-trained ROI segmentation model to determine the deep semantic features of the image to be segmented. The U-shaped backbone network prevents the loss of image feature information due to excessively deep layers in the ROI segmentation model. Using the vertex information of the smallest region where the target is located as the label information of the image to be segmented can ensure the privacy of the image to be segmented during the segmentation of the target of interest. Preliminary clustering processing is performed on the deep semantic features to determine the feature image and preliminary clustering features of the image to be segmented. The feature image of the image to be segmented is input into the block segmentation module of the pre-trained target of interest segmentation model for block clustering processing to determine the feature similarity between target blocks of the image to be segmented, thereby performing saliency analysis and segmenting the target of interest of the image to be segmented. This effectively reduces model redundancy, improves the training efficiency of the target of interest segmentation model, and improves the segmentation efficiency of the target of interest. Attached Figure Description

[0040] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0041] Figure 1 This is a flowchart illustrating a segmentation method for a target of interest according to an exemplary embodiment.

[0042] Figure 2 This is a flowchart illustrating a method for determining target blocks contained in an image to be segmented, according to another exemplary embodiment.

[0043] Figure 3 This is a flowchart illustrating a segmentation method for a target of interest according to another exemplary embodiment.

[0044] Figure 4 This is a flowchart illustrating a method for segmenting a target of interest in an image to be segmented according to an exemplary embodiment.

[0045] Figure 5 This is a flowchart illustrating a method for determining a pre-trained target-of-interest segmentation model according to an exemplary embodiment.

[0046] Figure 6 This is a flowchart illustrating a method for determining a segmentation model of a target of interest according to an exemplary embodiment.

[0047] Figure 7 This is a schematic diagram illustrating the segmentation effect of a target of interest according to an exemplary embodiment.

[0048] Figure 8 This is a block diagram illustrating a segmentation device for a target of interest according to an exemplary embodiment.

[0049] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0050] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0051] Figure 1 This is a flowchart illustrating a method for segmenting a target of interest according to an exemplary embodiment. For example... Figure 1 As shown, a segmentation method for a target of interest includes steps S11 to S14.

[0052] S11, obtain the image to be segmented and its label information.

[0053] The label information includes the vertex information of the smallest rectangular region where the target of interest is located on the image to be segmented. The vertex information can include the coordinates of the top left vertex and the bottom right vertex of the smallest rectangular region.

[0054] The image to be segmented can be a real-world scene captured in a photograph. The image of the target of interest can be placed at the center of the photograph of the image to be segmented, or at another location in the photograph of the image to be segmented.

[0055] The size of the sample in the image to be segmented can be H×W×C=224×224×3, where H represents the image height, W represents the image width, and C represents the number of image channels.

[0056] In this disclosure, the label information of the image to be segmented is [x1,y1,x2,y2], where x1 represents the x-coordinate of the top left vertex of the smallest rectangular region, y1 represents the y-coordinate of the top left vertex of the smallest rectangular region, x2 represents the x-coordinate of the bottom right vertex of the smallest rectangular region, and y2 represents the y-coordinate of the bottom right vertex of the smallest rectangular region.

[0057] S12, input the image to be segmented and its label information into the feature extraction module of the pre-trained target of interest segmentation model to determine the feature image and preliminary clustering features of the image to be segmented.

[0058] The pre-trained target of interest segmentation model includes a feature extraction module and a block segmentation module.

[0059] The feature extraction module includes a U-shaped backbone network, which is used to extract deep semantic features of the image.

[0060] In one possible embodiment, the image to be segmented and its label information are input into a U-shaped backbone network to determine the deep semantic features of the image to be segmented.

[0061] After extracting the deep semantic features of the image, the U-shaped backbone network's feature extraction module also performs preliminary clustering processing on the deep semantic features of the image to determine the preliminary clustering features of the image to be segmented, so as to prevent the loss of feature information of the image to be segmented.

[0062] S13, perform block clustering processing on the feature image of the image to be segmented to determine the target blocks contained in the image to be segmented.

[0063] The target block of the image to be segmented includes feature information from the same location in multiple different image channels of the feature image of the image to be segmented.

[0064] S14, segment the target of interest in the image to be segmented based on the feature similarity between the target blocks of the image to be segmented.

[0065] In one possible embodiment, the image to be segmented is divided into two non-overlapping sets A and B based on the feature similarity between blocks of the image to be segmented. Set A represents the foreground of the image to be segmented, and set B represents the background of the image to be segmented. The region where the target of interest is located is segmented and masked, i.e., the target of interest, thus realizing coarse-grained foreground and background segmentation and masking functions.

[0066] The above technical solution acquires the image to be segmented and its label information. The label information consists of the vertex information of the smallest rectangular region containing the target of interest (ROI) on the image to be segmented. This reduces the amount of label information used in segmenting ROI, thus decreasing the time spent on label annotation and improving the segmentation efficiency. The acquired image to be segmented and its label information are input into the feature extraction model of a pre-trained ROI segmentation model to determine the deep semantic features of the image to be segmented. The U-shaped backbone network prevents the loss of image feature information due to excessively deep layers in the ROI segmentation model. Using the vertex information of the smallest region where the target is located as the label information of the image to be segmented can ensure the privacy of the image to be segmented during the segmentation of the target of interest. Preliminary clustering processing is performed on the deep semantic features to determine the feature image and preliminary clustering features of the image to be segmented. The feature image of the image to be segmented is input into the block segmentation module of the pre-trained target of interest segmentation model for block clustering processing to determine the feature similarity between target blocks of the image to be segmented, thereby performing saliency analysis and segmenting the target of interest of the image to be segmented. This effectively reduces model redundancy, improves the training efficiency of the target of interest segmentation model, and improves the segmentation efficiency of the target of interest.

[0067] Figure 2 This is a flowchart illustrating a method for determining target blocks contained in an image to be segmented, according to another exemplary embodiment. Figure 2 As shown, in some possible embodiments, the feature images of the image to be segmented are subjected to block clustering processing to determine the target blocks contained in the image to be segmented, including S21 to S22.

[0068] S21, the feature information of each image channel of the feature image to be segmented is divided into mutually independent first blocks according to a preset size.

[0069] Following the example above, where the preset size can be k×k, the feature information of each image channel of the feature image of the image to be segmented will be divided into multiple independent first blocks of size k×k.

[0070] S22, the first blocks at the same position in each image channel of the feature image of the image to be segmented are superimposed to determine the target blocks contained in the image to be segmented.

[0071] The process of superimposing the first block at the same position in each image channel of the feature image of the image to be segmented means superimposing the feature information at the same position in different image channels of the feature image of the image to be segmented.

[0072] The size segmentation changes of the feature image of the image to be segmented include:

[0073]

[0074] Where N represents the number of training image samples, i.e. the number of batch sizes, C represents the number of image channels of the image to be segmented, H represents the height of the image to be segmented, W represents the width of the image to be segmented, scale1 represents the height of the target block, and scale2 represents the width of the target block.

[0075] The above technical solution integrates the feature information of each image channel of the feature image of the image to be segmented, and performs block-based clustering processing on the feature image of the image to be segmented from the height H direction and the width W direction, so as to facilitate subsequent comprehensive consideration of the feature similarity between each block.

[0076] Figure 3 This is a flowchart illustrating a method for segmenting a target of interest according to another exemplary embodiment. For example... Figure 3 As shown, in some possible embodiments, the method further includes S15 to S16.

[0077] S15, based on the location of each target block, determine the feature matrix of the feature image of the image to be segmented after block clustering processing.

[0078] Each target block contains some image features of the image to be segmented. The partial image features contained in each target block are superimposed sequentially from left to right and from top to bottom according to the position of each target block to determine the block-based feature matrix of the image to be segmented.

[0079] S16, multiply the feature matrix with the transpose of the feature matrix to determine the similarity between target blocks in the image to be segmented.

[0080] Through the above technical solution, the target of interest segmentation model integrates the feature images of the image to be segmented in the form of blocks, which can significantly reduce the amount of computation of model parameters.

[0081] Figure 4 This is a flowchart illustrating a method for segmenting a target of interest in an image to be segmented, according to an exemplary embodiment. Figure 4 As shown, the target of interest in the image to be segmented is segmented based on the similarity between target blocks in the image to be segmented, including S31 to S33.

[0082] S31, Based on the similarity between target blocks in the image to be segmented, the target matrix of the target of interest is obtained.

[0083] S32, based on the target matrix of the target of interest, determine the second smallest eigenvector of the generalized feature system.

[0084] In one possible embodiment, the real solution value of the normalized segmentation Ncut problem is determined based on the similarity between target blocks in the image to be segmented. The image to be segmented is divided into two independent sets A and B by the Ncut problem, where set A represents the background of the image to be segmented and set B represents the foreground of the image to be segmented. Sets A and B are disjoint.

[0085] The real solution to the Ncut problem corresponds to the second smallest eigenvector of the generalized feature system, and is solved using the following formula:

[0086]

[0087] Where E represents the target matrix of the target region of interest, D represents the diagonal matrix, x represents the image features corresponding to each target block, and y∈{1, -b} N b satisfies y T D1 = 0

[0088] The elements on the diagonal of the diagonal matrix D include d. i =∑ j E i,j .

[0089] Under relaxed constraints, define z = D 1 / 2 If y, then the above formula (2) is equivalent to:

[0090]

[0091] Where z0=D 1 / 2 1 is one of the eigenvectors corresponding to the smallest eigenvalue λ = 0.

[0092] As an example, according to the Rayleigh quotient, the second minimum eigenvector z1 is perpendicular to z0 to minimize the above formula (3).

[0093] The second smallest eigenvector z1 is determined according to the following formula:

[0094]

[0095] Its corresponding y1 is:

[0096]

[0097] S33, segment the target of interest in the image to be segmented based on the second smallest eigenvector of the generalized feature system.

[0098] Following the example above, the second smallest eigenvector of the generalized feature system (DE)y=λDy represents the real solution value of the NCut problem and indicates the feature information of the target region of interest.

[0099] By using the above technical solution, the second minimum feature vector of the generalized feature system corresponding to the NCut problem is solved based on the feature similarity between target blocks, and the region where the target of interest is located is determined, thus realizing coarse-grained masking under weak supervision information.

[0100] Figure 5 This is a flowchart illustrating a method for determining a segmentation model of a target of interest according to an exemplary embodiment.

[0101] like Figure 5 As shown, in some possible embodiments, the method for determining the pre-trained target segmentation model includes S41 to S44.

[0102] S41, obtain the training image samples and their label information.

[0103] The label information for the training image samples includes the vertex information of the smallest rectangular region containing the target of interest. Specifically, the label information for the training image samples can be the coordinates of the top-left and bottom-right vertices of the smallest rectangular region containing the target of interest.

[0104] The training image samples can be images captured from real scenes. The training image samples can include a variety of objects. The target of interest can be placed at any position on the training image sample. For example, the target of interest can be placed entirely at a preset position in the center of the training image sample, partially at a preset position in the center of the training image sample, or all targets of interest can be located at preset positions not in the center of the training image sample.

[0105] The size of the training image samples can be H×W×C=224×224×3, where H represents the image height, W represents the image width, and C represents the number of image channels.

[0106] S42, the forward deep residual network and the reverse deep residual network are concatenated to determine the main framework of the U-shaped backbone network. An indexed max pooling layer is added after each level of the forward deep residual network in the main framework of the U-shaped backbone network. The output features of each level of the forward deep residual network and the corresponding input features of the same level in the reverse deep residual network are connected by channel concatenation to determine the feature extraction module. The reverse deep residual network is an inverse symmetric structure of the forward deep residual network. The indexed max pooling layer is used to adjust the feature image size of the image to be segmented. The U-shaped backbone network uses the forward deep residual network to encode the image to be segmented and uses the reverse deep residual network to decode the image to be segmented.

[0107] The deep residual network can use a ResNet-18 network. The forward ResNet-18 network and the reverse ResNet-18 network are spliced ​​together to determine the U-shaped backbone network. The forward ResNet-18 network serves as the forward input part of the U-shaped backbone network, and the reverse ResNet-18 network, which is completely symmetrical to the forward ResNet-18 network, serves as the decoding output part of the U-shaped backbone network.

[0108] Indexed max pooling layers are used to adjust the feature image size of the image to be segmented, either by enlarging or shrinking the feature image.

[0109] S43, connect the feature extraction module and the block segmentation module to determine the segmentation model for the target of interest.

[0110] The block segmentation module is used to perform block-based clustering processing on the feature image and preliminary clustering features output by the feature extraction module, and to solve the second minimum feature vector of the generalized feature system, so as to achieve coarse-grained foreground and background segmentation and segment out the target of interest.

[0111] S44. Based on the preset loss function, the training image samples and their label information are input into the target of interest segmentation model for model training, thereby determining the pre-trained target of interest segmentation model.

[0112] In one possible embodiment, the deep features corresponding to pixels in the image have semantic information, and adjacent pixels are spatially continuous, by employing L... sim The function categorizes pixels with similar features in an image into the same cluster, with different clusters having different feature vectors.

[0113] Among them, L sim function:

[0114]

[0115] Among them, L sim X represents the clustering similarity loss. n C represents the feature image output by the feature extraction model. n This represents the clustering information determined by argmax processing of the feature image, where N represents the total number of pixels in the feature image, and n represents the nth pixel. X represents the sum of cross-entropy from the i-th cluster to the q-th cluster. n，i This represents the cluster value corresponding to the nth pixel.

[0116] In another possible embodiment, adjacent pixels have spatial continuity to prevent the generation of a large number of blocks due to complex image texture. Based on the vertex coordinates of the smallest rectangular region containing the target of interest in the image, i.e., the image's label information (weakly supervised label information), L... pse The function is used to train the target of interest (ROI) segmentation model to predict the pseudo-mask region of the ROI in the training image samples, which is close to the true mask region of the ROI in the training image samples.

[0117] Among them, L pse function:

[0118]

[0119] in,

[0120]

[0121]

[0122] Among them, L pse M represents the loss value between the predicted pseudo-mask region of the target of interest and the true mask region of the target of interest. n G represents the coarse-grained pseudo-mask region of the feature extraction model. n The text describes the labeling of the true mask region, the IoU (Intersection over Union) ratio between the predicted candidate bounding boxes and the true labeled bounding boxes, and the center point of the pseudo-mask region. gt denoted by , where 'a' represents the center point of the real mask region, 'c' represents the balance ratio, 'v' represents the diagonal distance between the closure regions of the pseudo-mask region and the real mask region, and 'w' represents the consistency coefficient of the relative aspect ratio between the pseudo-mask region and the real mask region. gt h represents the actual width of the target of interest.gt represents the true height of the target of interest, w represents the predicted width of the target of interest, and h represents the predicted height of the target of interest.

[0123] The preset loss function is:

[0124] Loss = L sim +L pse .

[0125] The training image samples and their label information are input into a pre-defined target of interest (ROI) segmentation model. The model is trained according to a pre-defined loss function to determine the pre-trained ROI segmentation model. The pre-trained ROI segmentation model can be used to perform saliency analysis on the deep semantic features extracted by the feature extraction module directly using a classifier, and effectively segment the ROI by combining a small amount of labeled information. No additional segmentation model is needed for concatenation, which reduces model parameters and model training costs while achieving instance segmentation.

[0126] By using the above technical solutions, the loss function is determined based on feature similarity and spatial continuity, avoiding the generation of too many blocks due to complex image stripes. This reduces the loss of feature information caused by excessive network layers, ensures the privacy of image data, reduces model redundancy, improves the generalization ability of the target of interest segmentation model, and prevents the target of interest segmentation model from overfitting, thereby improving the efficiency of segmenting the target of interest.

[0127] Figure 6 This is a schematic diagram illustrating a coarse-grained foreground and background segmentation process performed when an image to be segmented is input into a target of interest segmentation model, according to an exemplary embodiment.

[0128] like Figure 6 As shown, in some possible embodiments, the image to be segmented and its label information are input into a pre-trained target of interest segmentation model to segment the target of interest in the image to be segmented, including the following steps.

[0129] The image to be segmented and its label information are input into the target segmentation model. First, the feature extraction module is entered to determine the feature image and preliminary clustering features of the image to be segmented.

[0130] The image to be segmented and its label information pass through a convolutional layer and a max pooling layer in the feature extraction module, and then enter the forward deep residual network's Layer 1, Layer 2, and Layer 3 layers in sequence. After each layer, it passes through an indexed max pooling layer, and then enters the inverse deep residual network's D-Layer 3, D-Layer 2, and D-Layer 1 layers in sequence. The deep semantic features of the image to be segmented are output by the D-Layer 1 layer.

[0131] The features output from Layer 1 are connected and input into Layer D-Layer 1 through a U-shaped network, the features output from Layer 2 are connected and input into Layer D-Layer 2 through a U-shaped network, and the features output from Layer 3 are connected and input into Layer D-Layer 3 through a U-shaped network.

[0132] The features output from Layer 3 are concatenated with the upsampled features of the max-pooling layer with its index and then input into Layer D-Layer 3. The features output from Layer 2 are concatenated with the upsampled features of Layer D-Layer 3 and then input into Layer D-Layer 2. The features output from Layer 1 are concatenated with the upsampled features of Layer D-Layer 2 and then input into Layer D-Layer 1. Finally, Layer D-Layer 1 outputs the deep semantic features of the image to be segmented. These deep semantic features are then input into a classifier for preliminary feature clustering, outputting preliminary clustering features. Finally, the feature extraction module outputs the feature image.

[0133] By using the above technical solution, while retaining the residual connection method, the output features of the Layer in the forward input part can be input into the D-Layer in the decoding output part, which can prevent the loss of feature information due to too many network layers and protect the privacy of image data.

[0134] The feature image output by the feature extraction module is input into the block segmentation module. The block segmentation module performs block clustering processing on the feature image, dividing the feature information in each image channel of the feature image into a first block of size k×k. Then, the first blocks at the same position in different channels are superimposed to determine the target block.

[0135] The real solution to the NCut problem is obtained by calculating the feature similarity between target blocks, which is the second smallest feature vector of the generalized feature system. This enables coarse-grained foreground and background segmentation under weak supervision information, thus segmenting the target of interest in the image to be segmented.

[0136] Figure 7This is a schematic diagram illustrating the segmentation effect of a target of interest according to an exemplary embodiment.

[0137] like Figure 7 As shown, the first row of images is the input image to be segmented, the boxes in the second row of images are the targets of interest, the third row of images is the visualization result of solving the second minimum eigenvector of the generalized feature system, and the fourth row of images is the effect of the coarse-grained mask, that is, the segmented target of interest.

[0138] By employing the above technical solutions and performing saliency analysis by solving the second-minimum eigenvector of the generalized feature system, the redundancy of the target segmentation model can be reduced, eliminating the need for model splicing, improving model training efficiency, and preventing poor model generalization ability.

[0139] Based on the same concept, this disclosure also provides a segmentation device for a target of interest. Figure 8 This is a block diagram illustrating a segmentation device for a target of interest according to an exemplary embodiment. (Refer to...) Figure 8 The segmentation device 100 for the target of interest includes: a first acquisition module 110, a first determination module 120, a second determination module 130, and a segmentation module 140.

[0140] The first acquisition module 110 is used to acquire the image to be segmented and the label information of the image to be segmented, wherein the label information includes vertex information of the smallest rectangular region where the target of interest is located on the image to be segmented;

[0141] The first determining module 120 is used to input the image to be segmented and the label information of the image to be segmented into the feature extraction module of the pre-trained target of interest segmentation model to determine the feature image and preliminary clustering features of the image to be segmented. The feature extraction module includes a U-shaped backbone network, which is used to extract deep semantic features of the image.

[0142] The second determining module 130 is used to perform block clustering processing on the feature image of the image to be segmented to determine the target block contained in the image to be segmented.

[0143] The segmentation module 140 is used to segment the target of interest in the image to be segmented based on the feature similarity between target blocks in the image to be segmented.

[0144] The above technical solution acquires the image to be segmented and its label information. The label information consists of the vertex information of the smallest rectangular region containing the target of interest (ROI) on the image to be segmented. This reduces the amount of label information used in segmenting ROI, thus decreasing the time spent on label annotation and improving the segmentation efficiency. The acquired image to be segmented and its label information are input into the feature extraction model of a pre-trained ROI segmentation model to determine the deep semantic features of the image to be segmented. The U-shaped backbone network prevents the loss of image feature information due to excessively deep layers in the ROI segmentation model. Using the vertex information of the smallest region where the target is located as the label information of the image to be segmented can ensure the privacy of the image to be segmented during the segmentation of the target of interest. Preliminary clustering processing is performed on the deep semantic features to determine the feature image and preliminary clustering features of the image to be segmented. The feature image of the image to be segmented is input into the block segmentation module of the pre-trained target of interest segmentation model for block clustering processing to determine the feature similarity between target blocks of the image to be segmented, thereby performing saliency analysis and segmenting the target of interest of the image to be segmented. This effectively reduces model redundancy, improves the training efficiency of the target of interest segmentation model, and improves the segmentation efficiency of the target of interest.

[0145] Optionally, the second determining module 130 includes:

[0146] The segmentation submodule is used to divide the feature information of each image channel of the feature image to be segmented into mutually independent first blocks according to a preset size;

[0147] The first determining submodule is used to superimpose the first blocks at the same position of each image channel of the feature image to be segmented to determine the target block contained in the image to be segmented.

[0148] Optionally, the device 100 further includes:

[0149] The third determining module is used to determine the feature matrix of the feature image of the image to be segmented after the block clustering process, based on the position of each target block;

[0150] The fourth determining module is used to multiply the feature matrix with the transpose of the feature matrix to determine the similarity between target blocks of the image to be segmented.

[0151] Optionally, the first determining module 120 includes:

[0152] The second determining submodule is used to input the image to be segmented and its label information into the U-shaped backbone network to determine the deep semantic features of the image to be segmented.

[0153] Optionally, the segmentation module 140 includes:

[0154] The third determining submodule is used to determine the target matrix of the target of interest based on the similarity between target blocks in the image to be segmented;

[0155] The fourth determining submodule is used to determine the second minimum eigenvector of the generalized feature system based on the target matrix of the target of interest.

[0156] The segmentation submodule is used to segment the target of interest in the image to be segmented based on the second smallest feature vector of the generalized feature system.

[0157] Optionally, the pre-trained target of interest segmentation model includes a feature extraction module and a block segmentation module.

[0158] Optionally, the apparatus 100 further includes means for determining the segmentation model of the target of interest, including:

[0159] The model training sample acquisition module is used to multiply the feature matrix with the transpose of the feature matrix to determine the similarity between target blocks of the image to be segmented.

[0160] The first model determination module is used to concatenate the forward deep residual network and the reverse deep residual network to determine the main framework of the U-shaped backbone network. An indexed max-pooling layer is added after each level of the forward deep residual network within the main framework of the U-shaped backbone network. The output features of each level of the forward deep residual network are connected to the corresponding input features of the same level in the reverse deep residual network using channel concatenation. This determines the feature extraction module. The reverse deep residual network is an inverse symmetric structure of the forward deep residual network. The indexed max-pooling layer is used to adjust the feature image size of the image to be segmented. The U-shaped backbone network uses the forward deep residual network to encode the image to be segmented, and the U-shaped backbone network uses the reverse deep residual network to decode the image to be segmented.

[0161] The second model determination module is used to connect the feature extraction module and the block segmentation module to determine the segmentation model of the target of interest;

[0162] The model training module is used to input the training image samples and their label information into the target of interest segmentation model according to a preset loss function to train the model and determine the pre-trained target of interest segmentation model.

[0163] Optionally, the vertex information of the smallest rectangular region containing the target of interest in the image to be segmented includes the coordinates of the top-left vertex and the bottom-right vertex of the smallest rectangular region.

[0164] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0165] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Figure 9 As shown, the electronic device 700 may include a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output interface 704, and a communication component 705.

[0166] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the segmentation method of the target of interest in the first aspect described above. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. Input / output interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0167] In another exemplary embodiment, a non-transitory computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the segmentation method for the target of interest described in the first aspect. For example, the computer-readable storage medium may be the aforementioned memory including program instructions, which may be executed by a processor of an electronic device to complete the segmentation method for the target of interest.

[0168] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the segmentation method of the target of interest described above when executed by the programmable device.

[0169] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A method for segmenting a target of interest, characterized in that, include: Obtain the image to be segmented and its label information, wherein the label information includes vertex information of the smallest rectangular region containing the target of interest on the image to be segmented; The image to be segmented and its label information are input into the feature extraction module of the pre-trained target of interest segmentation model to determine the feature image and preliminary clustering features of the image to be segmented. The feature extraction module includes a U-shaped backbone network, which is used to extract deep semantic features of the image. The pre-trained target of interest segmentation model includes a feature extraction module and a block segmentation module. The feature images of the image to be segmented are subjected to block clustering processing to determine the target blocks contained in the image to be segmented; Based on the feature similarity between target blocks in the image to be segmented, the target of interest in the image to be segmented is segmented. The method for determining the segmentation model of the target of interest includes: Obtain training image samples and their label information, wherein the label information of the training image samples includes vertex information of the smallest rectangular region where the target of interest is located on the training image sample; The forward deep residual network and the backward deep residual network are concatenated to determine the main framework of the U-shaped backbone network. An indexed max-pooling layer is added after each level of the forward deep residual network within the main framework of the U-shaped backbone network. The output features of each level of the forward deep residual network are connected to the corresponding input features of the same level in the backward deep residual network using channel concatenation to determine the feature extraction module. The backward deep residual network is an inverse symmetric structure of the forward deep residual network. The indexed max-pooling layer is used to adjust the feature image size of the image to be segmented. The U-shaped backbone network uses the forward deep residual network to encode the image to be segmented, and the U-shaped backbone network uses the backward deep residual network to decode the image to be segmented. The feature extraction module and the block segmentation module are connected to determine the segmentation model for the target of interest; According to a preset loss function, the training image samples and their label information are input into the target of interest segmentation model for model training, thereby determining the pre-trained target of interest segmentation model; The step of performing block-based clustering processing on the feature images of the image to be segmented to determine the target blocks contained in the image to be segmented includes: The feature information of each image channel of the feature image to be segmented is divided into mutually independent first blocks according to a preset size; The first blocks at the same position in each image channel of the feature image of the image to be segmented are superimposed to determine the target block contained in the image to be segmented. The step of segmenting the target of interest in the image to be segmented based on the similarity between target blocks in the image to be segmented includes: The target matrix of the target of interest is determined based on the similarity between target blocks in the image to be segmented. Based on the target matrix of the target of interest, determine the second minimum eigenvector of the generalized feature system; The target of interest in the image to be segmented is segmented based on the second smallest feature vector of the generalized feature system.

2. The method according to claim 1, characterized in that, The method further includes: Based on the location of each target block, determine the feature matrix of the feature image of the image to be segmented after the block clustering process; The similarity between target blocks in the image to be segmented is determined by multiplying the feature matrix with its transpose.

3. The method according to claim 1, characterized in that, The step of inputting the image to be segmented and its label information into a pre-trained feature extraction module to determine the preliminary clustering features of the image to be segmented includes: The image to be segmented and its label information are input into the U-shaped backbone network to determine the deep semantic features of the image to be segmented.

4. The method according to claim 1, characterized in that, The pre-trained target of interest segmentation model includes a feature extraction module and a block segmentation module.

5. The method according to claim 1, characterized in that, The vertex information of the smallest rectangular region containing the target of interest in the image to be segmented includes the coordinates of the top-left vertex and the bottom-right vertex of the smallest rectangular region.

6. A segmentation device for a target of interest, characterized in that, include: The first acquisition module is used to acquire the image to be segmented and the label information of the image to be segmented, wherein the label information includes vertex information of the smallest rectangular region where the target of interest is located on the image to be segmented; The first determining module is used to input the image to be segmented and the label information of the image to be segmented into the feature extraction module of the pre-trained target of interest segmentation model to determine the feature image and preliminary clustering features of the image to be segmented. The feature extraction module includes a U-shaped backbone network, which is used to extract deep semantic features of the image. The pre-trained target of interest segmentation model includes a feature extraction module and a block segmentation module. The second determining module is used to perform block-based clustering processing on the preliminary clustering features of the image to be segmented, and to determine the target blocks contained in the image to be segmented. The segmentation module is used to segment the target of interest in the image to be segmented based on the feature similarity between target blocks in the image to be segmented; The apparatus further includes means for determining the segmentation model of the target of interest, including: The model training sample acquisition module is used to multiply the feature matrix with the transpose of the feature matrix to determine the similarity between target blocks of the image to be segmented. The first model determination module is used to concatenate the forward deep residual network and the reverse deep residual network to determine the main framework of the U-shaped backbone network. An indexed max-pooling layer is added after each level of the forward deep residual network within the main framework of the U-shaped backbone network. The output features of each level of the forward deep residual network are connected to the corresponding input features of the same level in the reverse deep residual network using channel concatenation. This determines the feature extraction module. The reverse deep residual network is an inverse symmetric structure of the forward deep residual network. The indexed max-pooling layer is used to adjust the feature image size of the image to be segmented. The U-shaped backbone network uses the forward deep residual network to encode the image to be segmented, and the U-shaped backbone network uses the reverse deep residual network to decode the image to be segmented. The second model determination module is used to connect the feature extraction module and the block segmentation module to determine the segmentation model of the target of interest; The model training module is used to input the training image samples and the label information of the training image samples into the target of interest segmentation model for model training according to a preset loss function, and to determine the pre-trained target of interest segmentation model; The second determining module includes: The segmentation submodule is used to divide the feature information of each image channel of the feature image to be segmented into mutually independent first blocks according to a preset size; The first determining submodule is used to superimpose the first blocks at the same position of each image channel of the feature image to be segmented to determine the target block contained in the image to be segmented. The segmentation module includes: The third determining submodule is used to determine the target matrix of the target of interest based on the similarity between target blocks in the image to be segmented; The fourth determining submodule is used to determine the second minimum eigenvector of the generalized feature system based on the target matrix of the target of interest. The segmentation submodule is used to segment the target of interest in the image to be segmented based on the second smallest feature vector of the generalized feature system.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-5.

8. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-5.

Images