A smearing image segmentation method and device, electronic equipment and medium

Abstract

The embodiment of the application discloses a smearing image segmentation method, which comprises the following steps: obtaining an image to be segmented, and processing the image to be segmented into scale feature maps of different scales; processing the scale feature maps of different scales into scale set feature maps of different scales through different set modules; aggregating the scale set feature maps of different scales through an aggregation module to obtain a mask function; obtaining final prediction mask maps of different scales based on the mask function, a prediction guide mask map and different back propagation modules; inputting the prediction guide mask map and the final prediction mask maps of different scales into a summary module respectively, and obtaining first prediction segmentation losses and second prediction segmentation losses of different scales; and obtaining a target mask map according to the prediction guide mask map and the final prediction mask maps of different scales when the sum of the first prediction segmentation losses and the second prediction segmentation losses of different scales meets a preset condition. Compared with the related art, the method can effectively compress the space occupied by the model and improve the image segmentation efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing and analysis, in particular, to a scribble image segmentation method and device, electronic equipment and medium. BACKGROUND

[0002] Nowadays, artificial intelligence technology is rising explosively and becoming more and more mature. Artificial intelligence algorithms evolve from fine-grained detection, recognition, segmentation and generation to more and more professional and high-end functions. This is a trend. Facing more and more artificial intelligence algorithms, their composition and transformation will produce new algorithms and applications.

[0003] Image segmentation is a very key artificial intelligence technology, which mainly divides an image into several specific regions with specific properties and extracts the target of interest. It is a key step from image processing to image analysis. With the development of deep learning technology in recent years, the main segmentation task is concentrated in the field of automatic segmentation. However, in some specific tasks, the automatic segmentation method cannot flexibly realize the user's demand (for example, only segmenting the target of interest), and the automatic segmentation also has the problem of inaccurate segmentation of the target region. At this time, it is very necessary to intervene and correct the segmentation result in a user-supervised manner. Interactive segmentation can achieve more fine and specific location target segmentation by providing foreground and background labels by the user in the case of insufficient automatic segmentation.

[0004] Interactive video segmentation is composed of interactive image segmentation algorithm and video mask tracking algorithm to meet people's complex and variable functional applications. Specifically, interactive video segmentation includes scribble interactive image segmentation, such as Scribble-to-mask, and click image segmentation, such as fBRS, RiTM and ClickSEG, etc. At present, most of the interactive segmentation methods are concentrated on the research of click interactive segmentation. Click interactive segmentation pursues to quickly realize the global segmentation of the target region with the least amount of interaction, so it sacrifices part of the precision.

[0005] The scribble image segmentation technology can ensure high segmentation accuracy. However, the scribble image segmentation technology in the related art still has the shortcomings of complex model architecture and large volume, and slow processing performance on CPU. There is an urgent need for an efficient image segmentation model with less model space occupation, high image segmentation efficiency and high image segmentation accuracy. SUMMARY

[0006] In view of the problems in the related art described above, the embodiments of the present application provide a scribble image segmentation method, which can effectively compress the space occupied by the model, improve the image segmentation efficiency, and ensure high accuracy of image segmentation.

[0007] In a first aspect, embodiments of this application provide a smearing-style image segmentation method, which may include:

[0008] Obtain the image to be segmented, which is processed into scale feature maps including different scales;

[0009] The scale feature maps of different scales are processed into different scale set feature maps by using different set modules;

[0010] The different scale feature maps are aggregated by the aggregation module to obtain the mask function;

[0011] Different final prediction masks are obtained by using the mask map and prediction guidance mask map of the mask function, and based on different backpropagation modules;

[0012] The predicted guidance mask and the different final predicted masks are respectively input into the aggregation module to obtain different first prediction segmentation losses and second prediction segmentation losses;

[0013] When the sum of the different first prediction segmentation losses and the different second prediction segmentation losses meets a preset condition, a target mask is obtained based on the prediction guidance mask and the different final prediction mask.

[0014] Furthermore, the process of acquiring the image to be segmented, wherein the image to be segmented is processed into scale feature maps including different scales, may include:

[0015] The image to be segmented is processed by a backbone network into a first-scale feature map, a second-scale feature map, and a third-scale feature map.

[0016] Furthermore, the process of processing the scale feature maps of different scales into different scale set feature maps through different set modules may include:

[0017] The first scale feature map is processed into a first scale set feature map through the first set module;

[0018] The second-scale feature map is processed into a second-scale set feature map through the second set module;

[0019] The third-scale feature map is processed into a third-scale set feature map through the third set module.

[0020] Furthermore, before the step of aggregating the feature maps of the different scale sets through the aggregation module to obtain the mask map of the mask function and the prediction guidance mask map, the method further includes:

[0021] The prediction guidance function is obtained by performing a first preset sampling on the mask function;

[0022] The prediction guidance mask is obtained based on the prediction guidance function.

[0023] Furthermore, the process of obtaining different final prediction masks based on the mask map and prediction guidance mask map obtained through the mask function and different backpropagation modules may include:

[0024] By performing a second preset sampling on the mask function, a first guiding function is obtained, and based on the first guiding function, a first guiding mask image is obtained;

[0025] The third-scale set feature map and the first guide mask map are input into the first backpropagation module to obtain the first predicted reverse guide mask map.

[0026] The first predicted reverse guide mask map is added to the first guide mask map to obtain the first predicted mask feature map.

[0027] Furthermore, the process of obtaining different final prediction masks based on the mask map and prediction guidance mask map obtained through the mask function and different backpropagation modules may include:

[0028] By performing a third preset sampling on the first predicted mask feature map, a second guiding function is obtained, and a second guiding mask map is obtained based on the second guiding function;

[0029] The second scale set feature map and the second guide mask map are input into the second backpropagation module to obtain the second predicted reverse guide mask map;

[0030] The second predicted reverse guide mask map is added to the second guide mask map to obtain the second predicted mask feature map.

[0031] Furthermore, the process of obtaining different final prediction masks based on the mask map and prediction guidance mask map obtained through the mask function and different backpropagation modules may include:

[0032] By performing a third preset sampling on the second predicted mask feature map, a third guiding function is obtained, and based on the third guiding function, a third guiding mask map is obtained;

[0033] The first scale set feature map and the third guide mask map are input into the third backpropagation module to obtain the third predicted reverse guide mask map;

[0034] The third predicted reverse guide mask map is added to the third guide mask map to obtain the third predicted mask feature map.

[0035] Furthermore, the process of obtaining different final prediction masks based on the mask map and prediction guidance mask map obtained through the mask function and different backpropagation modules may include:

[0036] The first final prediction mask image is obtained by performing a first preset sampling on the first prediction mask feature map;

[0037] The second final prediction mask image is obtained by performing a fourth preset sampling on the second prediction mask feature map;

[0038] The third final prediction mask image is obtained by performing a fifth preset sampling on the third prediction mask feature map.

[0039] Further, the step of inputting the predicted guidance mask and the different final predicted masks into the aggregation module respectively, and obtaining different first prediction segmentation losses and second prediction segmentation losses, may include:

[0040] The predicted guidance mask and the different final predicted masks are converted into a first mode map and a second mode map;

[0041] The first segmentation loss is obtained by calculating the loss of the first pattern graph based on the first loss function;

[0042] The second prediction segmentation loss is obtained by calculating the loss of the second pattern map based on the second loss function.

[0043] Secondly, embodiments of this application also provide a smearing image segmentation apparatus, which may include:

[0044] An acquisition unit is used to acquire an image to be segmented, wherein the image to be segmented is processed into scale feature maps including different scales;

[0045] Processing unit, used for:

[0046] The scale feature maps of different scales are processed into different scale set feature maps by using different set modules;

[0047] The different scale feature maps are aggregated by the aggregation module to obtain the mask function;

[0048] Different final prediction masks are obtained by using the mask map and prediction guidance mask map of the mask function, and based on different backpropagation modules;

[0049] The predicted guidance mask and the different final predicted masks are respectively input into the aggregation module to obtain different first prediction segmentation losses and second prediction segmentation losses;

[0050] When the sum of the different first prediction segmentation losses and the different second prediction segmentation losses meets a preset condition, a target mask is obtained based on the prediction guidance mask and the different final prediction mask.

[0051] Thirdly, embodiments of this application also provide an electronic device, comprising: a memory and a processor, the memory being used to store and support the processor in executing a program of any of the methods described in the first aspect, the processor being configured to execute the program stored in the memory.

[0052] Fourthly, embodiments of this application also provide a computer-readable medium having processor-executable non-volatile program code, wherein the program code causes the processor to perform any of the methods described in the first aspect.

[0053] The embodiments of this application bring the following beneficial effects:

[0054] In this embodiment, the image to be segmented is first acquired and processed into scale feature maps of different scales. Then, the scale feature maps of different scales are processed into different scale set feature maps through different aggregation modules. The different scale set feature maps are aggregated through an aggregation module to obtain a mask function. Different final prediction masks are obtained through the mask map of the mask function and the prediction guidance mask map, and based on different backpropagation modules. The prediction guidance mask map and the different final prediction masks are respectively input into a summarization module to obtain different first prediction segmentation losses and second prediction segmentation losses. When the sum of the different first prediction segmentation losses and the second prediction segmentation losses meets a preset condition, the target mask map is obtained according to the prediction guidance mask map and the different final prediction masks. Compared with related technical solutions, the embodiments of this application first segment the image to be segmented into scale feature maps of different scales, and then through the coordinated cooperation of different set modules, aggregation modules and different backpropagation modules, the sum of the first predicted segmentation loss and the second predicted segmentation loss can quickly meet the preset conditions, thereby effectively improving the processing speed of smearing image segmentation and ensuring the image segmentation effect. Moreover, due to the fast inference speed, the space occupied by the model is small, which effectively improves the user experience. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0056] Figure 1 One of the schematic diagrams of an embodiment of a smearing image segmentation method provided in this application;

[0057] Figure 2 A second schematic diagram of an embodiment of a smearing image segmentation method provided in this application;

[0058] Figure 3 A schematic diagram of an embodiment of a smearing image segmentation method provided in this application is shown in Figure 3.

[0059] Figure 4 A schematic diagram of an embodiment of a smearing image segmentation method provided in this application is shown below;

[0060] Figure 5 This is a schematic diagram of the structure of a smearing image segmentation device provided in an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of an embodiment of an electronic device provided in this application.

[0062] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0064] In the specification, claims, and accompanying drawings of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0065] Artificial intelligence technology is booming and maturing. AI algorithms are evolving from fine-grained detection, recognition, segmentation, and generation to increasingly professional and high-end functions. This is a trend. With more and more AI algorithms, changing their composition will generate new algorithms and applications.

[0066] Image segmentation is a crucial artificial intelligence technique, primarily involving dividing an image into several specific regions with specific attributes and extracting the target of interest. It is a key step in the transition from image processing to image analysis. With the development of deep learning technology in recent years, the main segmentation tasks have focused on fully automated segmentation. However, for certain specific tasks, automated segmentation methods cannot flexibly meet user needs (e.g., segmenting only the target of interest), and automated segmentation also suffers from inaccurate target region segmentation. In such cases, user-supervised intervention to correct the segmentation results becomes essential. Interactive segmentation can achieve more refined and location-specific target segmentation by allowing users to provide foreground and background labels when automated segmentation is insufficient.

[0067] Interactive video segmentation combines interactive image segmentation algorithms with video mask tracking algorithms to meet the complex and diverse functional applications. Specifically, interactive video segmentation includes smear-based interactive image segmentation, such as Scribble-to-mask, and click-based image segmentation, such as fBRS, RiTM, and ClickSEG. Currently, most interactive segmentation methods focus on click-based interactive segmentation, which aims to achieve rapid global segmentation of the target region with minimal interaction, thus sacrificing some accuracy.

[0068] While smearing image segmentation techniques can guarantee high segmentation accuracy, they still suffer from drawbacks such as complex and large model architectures and slow processing performance on CPUs. There is an urgent need for an efficient image segmentation model that occupies less space, has high image segmentation efficiency, and high image segmentation accuracy.

[0069] For ease of understanding, the specific processes in the embodiments of this application are described below. Please refer to [link / reference]. Figure 1 , Figure 1 This is a schematic diagram of one embodiment of a smearing image segmentation method provided in this application. The smearing image segmentation method of this invention can be applied to a server, and implemented through a software application installed on the server to improve the effect of image style. The smearing image segmentation method will be described in detail below. Figure 1 As shown, the method includes the following steps S100-S600.

[0070] S100: Obtain the image to be segmented, which is processed into scale feature maps of different scales.

[0071] Suppose there is a video clip that can be broken down into frames over time. These frames are saved in a pre-defined image library for later use. Understandably, users can highlight regions of interest (ROIs) before the video is broken down into images. For example, a geometry containing the highlighting trajectory can be constructed and expanded to form a marked area, thus identifying a region containing the highlighting trajectory as the ROI:

[0072] Construct the minimum bounding rectangle of the smearing trajectory, and expand the minimum bounding rectangle according to the length L of the smearing trajectory to form a marking area;

[0073] A geometric shape template is applied to the smearing trajectory, and a region of interest containing the smearing trajectory is generated based on the geometric shape template. If the region of interest does not contain the marker area, the region of interest is expanded to include the marker area; or the marker area is expanded to form a region of interest.

[0074] Of course, users can also paint over the areas of interest after the video is split into frames; this application does not limit this.

[0075] The image to be segmented can be obtained directly from a preset image library, or an image with a marked region or region of interest can be obtained from the preset image library, which helps to reduce the computational difficulty in the subsequent image segmentation process.

[0076] Please see Figure 2 , Figure 2 This is a second schematic diagram of an embodiment of a smearing image segmentation method provided in this application. When the image to be segmented is processed into scale feature maps including different scales, it may specifically include:

[0077] S101: The image to be segmented is processed by a backbone network into a first-scale feature map, a second-scale feature map, and a third-scale feature map.

[0078] For example, after obtaining the image to be segmented, it is possible to simultaneously obtain the single-channel mask image, the foreground smear mask image, and the background smear mask image of the image to be segmented. First, the foreground smear mask image and the background smear mask image are respectively unsqueezed and cast, and then concatenated to obtain the concatenated information of the foreground smear mask image and the background smear mask image. Then, the image to be segmented, the single-channel mask image, and the concatenated information of the foreground smear mask image and the background smear mask image are concatenated to obtain a multi-channel scale feature map. This multi-channel scale feature map can be processed by the backbone network into scale feature maps of different scales, such as a first scale feature map, a second scale feature map, and a third scale feature map.

[0079] Furthermore, the steps of S100 can be implemented using a ghost module.

[0080] The Ghost module consists of two sets of convolutional blocks: the first convolutional block (e.g., primary, hereinafter referred to as primary) and the second convolutional block (e.g., cheap, hereinafter referred to as cheap). Both the primary and cheap convolutional blocks are composed of convolutions, accelerated neural networks (BatchNorm), and ReLU activation functions of neurons. The number of input convolutional kernels is the same for both, and the sum of the number of input convolutional kernels is greater than or equal to the number of output convolutional kernels.

[0081] The input parameters for the primary and cheap convolutional combinations depend on the settings of the Ghost module, and are determined by in_channel, out_channel, kernel_size, ratio, dw_size, stride, and ReLU, respectively (for ease of understanding by those skilled in the art, common neural network abbreviations are used here). Here, in_channel is the number of input channels for the primary convolutional layer, in_channel = out_channel / / ratio is the number of output channels, kernel_size is the kernel size, stride is the stride, and padding is the spacing, where the spacing equals kernel_size / / 2. The above parameters and operations are input into the BatchNorm layer (input channel number is init_channel) and the ReLU activation layer; while the cheap convolutional layer has init_channel as input, new_channel = out_channel / / ratio * (ratio - 1) as output, kernel size is dw_size, stride is 1, padding is dw_size / / 2, and groups are init_channel. The above parameters and operations are then input into the BatchNorm layer (input is new_channel) and the ReLU activation layer.

[0082] Assuming the input is a 1×in_channel×H×W feature map, where in_channel is the number of input channels, H represents the height of the image to be segmented, and W represents the width of the image to be segmented, then the first convolution is used to obtain a 1×init_channel×H×W feature map, and the second convolution is used to obtain a 1×new_channel×H×W feature map. Concatenating these two features along the second dimension yields a 1×(init_channel+new_channel)×H×W feature map. Finally, taking the first 1×out_channel×H×W feature map, we obtain the feature map after processing by the ghost module.

[0083] In some implementations, the input parameters are assumed to be: a 1×3×384×384 RGB three-channel image as the image to be segmented, a single-channel 1×1×384×384 mask as the single-channel mask, a 1×1×384×384 foreground smearing mask as the foreground smearing mask, and a 1×1×384×384 background smearing mask as the background smearing mask. The 1×1×384×384 foreground smearing mask and the 1×1×384×384 background smearing mask are concatenated, and then the 1×3×384×384 RGB three-channel image and the single-channel 1×1×384×384 mask are... The values ​​obtained by concatenating the 4×384 mask, the foreground smear mask, and the 1×1×384×384 background smear mask are concatenated to obtain a 1×6×384×384 multi-scale feature map. The 1×6×384×384 multi-scale feature map is then input into the backbone network (e.g., MobileNet series network) to obtain three feature maps of different scales, namely the first scale feature map feat8 (1×32×48×48), the second scale feature map feat16 (1×96×24×24), and the third scale feature map feat32 (1×320×12×12).

[0084] In this embodiment, by using a ghost module instead of a regular convolution module to perform preprocessing such as dimensional transformation, layering, and cascading of the image to be segmented, the data processing efficiency and speed of the image to be segmented are greatly improved.

[0085] S200: The scale feature maps of different scales are processed into different scale set feature maps through different set modules;

[0086] After obtaining feature maps at different scales in step S100, such as the aforementioned first-scale feature map, second-scale feature map, and third-scale feature map, different processing is required on the different scale feature maps to generate different scale set feature maps. This further improves the data computation speed and effectively compresses the space occupied by the computation model.

[0087] For example, a set module may include multiple modules: a first set module, a second set module, and a third set module. The specific modules can be changed according to the actual situation, and this application does not limit them.

[0088] Step S200 specifically includes the following steps:

[0089] S201: The first scale feature map is processed into a first scale set feature map through the first set module;

[0090] S202: The second scale feature map is processed into a second scale set feature map through the second set module;

[0091] S203: The third-scale feature map is processed into a third-scale set feature map through the third set module.

[0092] Furthermore, the S200's collection module can use the LRASPP module.

[0093] The LRASPP module is a lightweight version of the ASPP module, consisting of two convolutional layers. Assuming the input is a 1×in_channel×H×W feature map, and we need to obtain a 1×out_channel×H×W feature map, the first convolutional layer of LRASPP (input kernel number is in_channel, output kernel number is out_channel, kernel size is 1, other parameters are default) passes through a batch normalization layer (input number of out_channel), and then through a ReLU layer. The input feature map is then fed into the second convolutional layer of LRASPP, first passing through a binary adaptive mean pooling layer (nn.AdaptiveAvgPool2d, output feature map size is 2×2) to obtain the 1×in_channel×H×W feature map. The _channel×2×2 feature map is passed through a convolutional layer (input kernel number is in_channel, output kernel number is out_channel, kernel size is 1, other parameters are default) to obtain a 1×out_channel×2×2 feature map. Then, it is passed through an activation (Sigmoid) layer to obtain a 1×out_channel×2×2 feature map. The 1×out_channel×2×2 feature map obtained from the second convolutional group is then interpolated and enlarged to a 1×out_channel×H×W feature map. Finally, the 1×out_channel×H×W feature map obtained from the first convolutional group is multiplied by the 1×out_channel×H×W feature map to obtain the final 1×out_channel×H×W feature map.

[0094] For example, after obtaining the first-scale feature map feat8 (1×32×48×48), the second-scale feature map feat16 (1×96×24×24), and the third-scale feature map feat32 (1×320×12×12), inputting the first-scale feature map feat8 (1×32×48×48) into the first set module can yield the first-scale set features. Figure 1 A 32×48×48 feature map is generated, where the first set of modules has 32 input channels and 32 output channels. The second-scale feature map feat16 (1×96×24×24) is input into the second set of modules to obtain the second-scale set feature map. Figure 1 The second set of modules contains a 32×24×24 feature map, where the second set of modules has 96 input channels and 32 output channels. The third-scale feature map feat32 (1×320×12×12) is input into the third set of modules to obtain the third-scale feature map. Figure 1 The feature map is 32×12×12, where the third set of modules has 320 input channels and 32 output channels.

[0095] In this embodiment, by using ensemble modules, especially different ensemble modules such as the LRASPP module to process feature maps at different scales, it is possible to effectively improve the data processing speed and greatly compress the model size while ensuring high data processing quality.

[0096] S103: The different scale feature maps are aggregated by the aggregation module to obtain the mask function.

[0097] In this embodiment, by using the aggregation module to process the feature maps of different scales obtained in step 200, the learning ability of the smearing image segmentation model of this application can be further improved.

[0098] In some implementations, please refer to Figure 3 The aggregation module aggregates feature maps from different scales to obtain a mask function. Specifically, before the step of aggregating the feature maps from different scales through the aggregation module to obtain the mask map of the mask function and the prediction guidance mask map, the following steps are also included:

[0099] S301: The prediction guidance function is obtained by performing a first preset sampling on the mask function;

[0100] S302: Obtain the prediction guidance mask based on the prediction guidance function.

[0101] Furthermore, the aggregation module can adopt the NCD module.

[0102] The NCD module, also known as the Neighbor Connectivity Decoder, primarily aggregates multi-scale features to achieve more effective learning capabilities. This module consists of eight Ghost modules (with the same number of input and output channels) and a regular convolutional layer, with three feature maps as input.

[0103] Assume the input feature maps are the first scale set feature map feat32 (1×input channel×H / 32×W / 32) obtained in step 300, the second scale set feature map feat16 (1×input channel×H / 16×W / 16), and the third scale set feature map feat8 (1×input channel×H / 8×W / 8).

[0104] The NCD module first upsamples the first scale set feature map feat32 (1×input channel×H / 32×W / 32) by a factor of 2 to obtain 1×input channel×H / 16×W / 16. This 1 / 16 feature map is then input into the first Ghost module to obtain the 1×input channel×H / 16×W / 16 feature map. Finally, it is multiplied by the input feat16 (1×input channel×H / 16×W / 16) feature map to obtain the new feat16_1 (1×input channel×H / 16×W / 16) feature map.

[0105] The newly obtained feat16_1 (1×input channel×H / 16×W / 16) feature map is upsampled by 2 times to obtain a 1×input channel×H / 8×W / 8 feature map. This is then input into the second Ghost module to obtain the first new 1×input channel×H / 8×W / 8 feature map. At the same time, the original input feat16 (1×input channel×H / 16×W / 16) feature map is also upsampled by 2 times and then input into the third Ghost module to obtain the second new 1×input channel×H / 8×W / 8 feature map.

[0106] Multiply the first new 1×input channel×H / 8×W / 8 feature map, the second new 1×input channel×H / 8×W / 8 feature map, and the original input feat8 (1×input channel×H / 8×W / 8) feature map to obtain the latest feat8_1 (1×input channel×H / 8×W / 8) feature map; upsample the original input feat32 (1×input channel×H / 32×W / 32) feature map by 2 times to obtain 1×input channel×H / 16×W / 16, and then input it into the fourth Ghost module to obtain 1×input channel×H / 16×W / 16, and then concatenate it with feat16_1 (1×input channel×H / 16×W / 16) to obtain 1×2*input channel×H / 16×W / 16 feature map, and input it into the fifth Ghost module to obtain feat16_2 (1×2*input channel×H / 16×W / 16) feature map.

[0107] The feature map of feat16_2 (1×2*input channel×H / 16×W / 16) obtained above is upsampled by 2 times to obtain a feature map of 1×2*input channel×H / 8×W / 8. This feature map is then input into the sixth Ghost module to obtain a feature map of 1×2*input channel×H / 8×W / 8. This feature map is then concatenated with the feature map of feat8_1 (1×input channel×H / 8×W / 8) to obtain a feature map of 1×3*input channel×H / 8×W / 8. This feature map is then input into the seventh Ghost module to obtain a feature map of feat8_2 (1×3*input channel×H / 8×W / 8).

[0108] The feature map obtained above, feat8_2 (1×3*input channels×H / 8×W / 8), is input into the eighth Ghost module to obtain a 1×3*input channels×H / 8×W / 8 feature map. Then, it is input into a regular convolutional layer (input kernel number is 3*input channels, output kernel number is 1, kernel size is 1, padding is 1, and other parameters are default) to obtain a 1×1×H / 8×W / 8 single-channel mask map, and returns a 1×1×H / 8×W / 8 mask map.

[0109] In this embodiment, the first scale set feature map feat32 (1×input channel×H / 32×W / 32), the second scale set feature map feat16 (1×input channel×H / 16×W / 16), and the third scale set feature map feat8 (1×input channel×H / 8×W / 8) obtained in step 300 are input into the NCD module to obtain the mask map of the mask function mask_g (1×1×48×48). After upsampling the mask function mask_g (1×1×48×48) by 8 times (first preset sampling), the prediction guidance mask map of the prediction guidance function pred_guidance (1×1×384×384) is obtained.

[0110] This implementation method can aggregate multi-scale features to obtain more effective learning capabilities, greatly improving the accuracy of the smearing image segmentation model of this application.

[0111] S400: Different final prediction masks are obtained through the mask map and prediction guidance mask map of the mask function, and based on different backpropagation modules;

[0112] The backpropagation module consists of a first backpropagation function, a second backpropagation function, and a third backpropagation function.

[0113] Step S400 can be performed by the following steps:

[0114] S4011: By performing a second preset sampling on the mask function, a first guiding function is obtained, and based on the first guiding function, a first guiding mask image is obtained;

[0115] S4012: Input the third-scale set feature map and the first guide mask map into the first backpropagation module to obtain the first predicted reverse guide mask map;

[0116] S4013: Add the first predicted reverse guide mask map to the first guide mask map to obtain the first predicted mask feature map.

[0117] Step S400 can also be performed by the following steps:

[0118] S4021: By performing a third preset sampling on the first predicted mask feature map, a second guiding function is obtained, and a second guiding mask map is obtained based on the second guiding function;

[0119] S4022: Input the second scale set feature map and the second guide mask map into the second backpropagation module to obtain the second predicted reverse guide mask map;

[0120] S4023: Add the second predicted reverse guide mask map to the second guide mask map to obtain the second predicted mask feature map.

[0121] Step S400 can also be performed by the following steps:

[0122] S4031: By performing a third preset sampling on the second predicted mask feature map, a third guiding function is obtained, and based on the third guiding function, a third guiding mask map is obtained;

[0123] S4032: Input the first scale set feature map and the third guide mask map into the third backpropagation module to obtain the third predicted reverse guide mask map;

[0124] S4033: Add the third predicted reverse guide mask map to the third guide mask map to obtain the third predicted mask feature map.

[0125] The reverse propagation module can be either the GRA module or the ReverseStage module.

[0126] The GRA module, also known as the group inversion attention module, primarily refines the coarse prediction by progressively using different feature pyramids. The module's input parameters are the input channel and the sub-channel. The input consists of two feature maps, one of which is a multi-channel feature map. Figure 1 ×Input channel×H×W, one is a single-channel guide mask Figure 1 ×1×H×W.

[0127] First, the input multi-channel features Figure 1 The input channel (H) and W are divided into groups = input channel / / sub_channel blocks (Note: only groups of 1, 2, 4, 8, 16, and 32 are supported; others are not). After dividing into groups, they are combined with another single-channel guide mask. Figure 1 The ×1×H×W features are cascaded at intervals to obtain a 1×(input channel + group)×H×W feature map;

[0128] Then, it is input into a regular convolution (the number of input kernels is the number of input channels + gruppup, the number of output kernels is the number of input channels, the kernel size is 3, and the padding is 1) to obtain a 1×input channel×H×W feature map. This is then added to the original input 1×input channel×H×W feature map to obtain a new feat_out (1×input channel×H×W feature map).

[0129] Next, the feat_out (1×input channel×H×W) feature map is fed into the second convolutional layer (the number of input kernels is the number of input channels, the number of output kernels is 1, the kernel size is 3, and the padding is 1) to obtain the final 1×1×H×W feature map. This final 1×1×H×W feature map is then added to the original input mask map to obtain the new mask_out (1×1×H×W) feature map. Finally, the feat_out (1×input channel×H×W) feature map and the mask_out (1×1×H×W) feature map are returned.

[0130] The ReverseStage module consists of three GRA modules: weak GRA (weak_gra), medium GRA (medium_gra), and strong GRA (strong_gra). The input consists of two feature maps, one of which is a multi-channel feature map. Figure 1×Input channel ×H×W and a single-channel mask map, consistent with the GRA input. First, the input single-channel 1×1×H×W mask is passed through an activation (sigmoid) layer to obtain a mask with values ​​in the (0, 1) interval. This mask is then multiplied by -1 and added by 1 to obtain a reversed 1×1×H×W mask. This reversed mask is then input into the first GRA module, namely weak_gra (the number of input channels is the input channel number, and the number of subchannels is the input channel number), to obtain two outputs: a 1×input channel×H×W feature map and a single-channel 1×1×H×W feature map. This is then input into medium_gra (the number of input channels is the input channel number, and the subchannel number is 8) to obtain a 1×input channel×H×W feature map and a 1×1×H×W mask. Finally, this 1×1×H×W mask feature map is returned.

[0131] In this embodiment, the detailed process is as follows:

[0132] By performing a second preset sampling (e.g., downsampling by 4 times) on the mask function mask_g(1×1×48×48), the first guidance function guidance_g(1×1×12×12) is obtained. Based on the first guidance function guidance_g(1×1×12×12), the mask image of the first guidance mask guidance_g(1×1×12×12) is obtained.

[0133] The third-scale set features Figure 1 The 32×12×12 feature map and the mask map of the first guidance mask map guidance_g (1×1×12×12) are input into the first backpropagation module to obtain the first predicted inverse guidance mask. Figure 1 A prediction mask of ×1×12×12;

[0134] The first predictive inversion guide mask Figure 1 The predicted mask image of ×1×12×12 is added to the mask image of the first guidance mask image guidance_g(1×1×12×12) to obtain the first predicted mask feature image mask_32×(1×1×12×12).

[0135] By performing a third preset sampling (upsampling by 2 times) on the first predicted mask feature map mask_32×(1×1×12×12), the second guidance function guidance_g(1×1×24×24) is obtained. Based on the second guidance function guidance_g(1×1×24×24), the mask map of the second guidance mask map guidance_g(1×1×24×24) is obtained.

[0136] The second scale set features Figure 1 The 32×24×24 feature map and the mask map of the second guidance mask map guidance_g (1×1×24×24) are input into the second backpropagation module to obtain the second prediction inversion guidance mask. Figure 1 A prediction mask of ×1×24×24;

[0137] The second predictive inversion guide mask Figure 1 The predicted mask image of ×1×24×24 is added to the mask image of the second guidance mask image guidance_g(1×1×24×24) to obtain the second predicted mask feature image mask_16×(1×1×24×24).

[0138] By performing a third preset sampling (upsampling by 2 times) on the second prediction mask feature map mask_16×(1×1×24×24), the third guidance function guidance_g(1×1×48×48) is obtained. Based on the third guidance function guidance_g(1×1×48×48), the guidance mask map of the third guidance mask map guidance_g(1×1×48×48) is obtained.

[0139] The first scale set features Figure 1 The 32×48×48 feature map and the third guidance mask map guidance_g (1×1×48×48) are input into the third backpropagation module to obtain the third predicted inversion guidance mask. Figure 4 A prediction mask of ×1×48×48;

[0140] The third predictive inversion guide mask Figure 4 The predicted mask image of ×1×48×48 is added to the third guidance mask image guidance_g(1×1×48×48) to obtain the third predicted mask feature image mask_8×(1×1×48×48).

[0141] Step S400 can also be performed by the following steps:

[0142] S4041: Obtain the first final prediction mask by performing a first preset sampling on the first prediction mask feature map;

[0143] S4042: The second final prediction mask is obtained by performing a fourth preset sampling on the second prediction mask feature map;

[0144] S4043: The third final prediction mask is obtained by performing a fifth preset sampling on the third prediction mask feature map.

[0145] Specifically, by performing a first preset sampling (upsampling by 8 times) on the first predicted feature map mask_g (1×1×48×48), the first final predicted mask map pred_guidance (1×1×384×384) is obtained.

[0146] By performing a fourth preset sampling (upsampling 32) on the second prediction mask feature map mask_32×(1×1×12×12), the second final prediction mask map pred_guidance(1×1×384×384) is obtained.

[0147] By performing a fifth preset sampling (upsampling by 16 times) on the third prediction mask feature map mask_16×(1×1×24×24), the third final prediction mask map pred_guidance(1×1×384×384) is obtained.

[0148] In this embodiment, by employing a backpropagation module, the coarse prediction can be gradually refined through different feature pyramids, further improving the processing efficiency of the smearing-style image segmentation method of this application.

[0149] S500: Input the predicted guidance mask and the different final predicted masks into the aggregation module respectively, and obtain different first prediction segmentation losses and second prediction segmentation losses.

[0150] Please see Figure 5 , Figure 6 This is a fourth schematic diagram of an embodiment of a smearing image segmentation method provided in this application.

[0151] After inputting the prediction guidance mask of red_guidance (1×1×384×384) obtained in step S300 and the first, second, and third final prediction masks of pred_guidance (1×1×384×384) obtained in step S400 into the aggregation module, the aggregation module can calculate the first prediction segmentation loss and the second prediction segmentation loss of the four images.

[0152] Specifically, step S500 can be implemented by the following steps:

[0153] S501: The predicted guidance mask and the different final predicted masks are converted into a first mode map and a second mode map;

[0154] S502: Calculate the loss of the first pattern map based on the first loss function to obtain the first segmentation loss;

[0155] S503: Calculate the loss of the second pattern map based on the second loss function to obtain the second prediction segmentation loss.

[0156] S600: When the sum of the different first prediction segmentation losses and the second prediction segmentation losses meets a preset condition, a target mask is obtained based on the prediction guidance mask and the different final prediction mask.

[0157] In some implementations, the summarization module can be understood as a loss calculation summarization module, which mainly summarizes the losses. When the total loss meets the preset conditions, the target mask map can be output. When the total loss does not meet the preset conditions, it is necessary to perform iterative optimization on the prediction guidance mask map of the aforementioned pred_guidance (1×1×384×384) and the first final prediction mask map, the second final prediction mask map, and the third final prediction mask map of pred_guidance (1×1×384×384) obtained in step S400, as well as the relevant network parameters.

[0158] In some implementations, the aggregation module uses the aggregate module.

[0159] The `aggregate` module is a transformation function without any parameters. Assuming the input is a 1×1×384×384 mask, it first calculates the product of its background mask (1-mask yields the value of its background mask) (torch.prod), concatenates this product with the input mask to obtain a 1×2×384×384 feature map, and then normalizes the result to the range [1e-7, 1-1e-7] to prevent value overflow. Next, calculate the torch.log values ​​of the 1×2×384×384 feature map and the 1-1×2×384×384 feature map obtained above to get the logits(1×2×384×384) feature map. Then, normalize the 1×2×384×384 feature map with softma× to get the normalized new mask(1×2×384×384) feature map. Finally, return logits(1×2×384×384) and the foreground image mask [:,1:](1×1×384×384). The former is used for segmentation loss calculation and the latter is used for matting loss calculation.

[0160] Specifically, in this embodiment, the obtained prediction guidance mask of pred_guidance (1×1×384×384) and the first, second, and third final prediction masks of pred_guidance (1×1×384×384) obtained in step S400 are converted into a segmented 1×2×384×384 image and a single-channel 1×1×384×384 mask image respectively via the aggregation module. The loss is calculated using the BCE segmentation loss function. The 1×1×384×384 image is then processed using a matting loss function, and the result is fed into a composite loss function such as F1 / Dice / Laplacian. The two losses are then added together to obtain the loss of the predicted mask. Similarly, to calculate the total loss of the four predicted masks, eight losses need to be calculated. The total loss is then added together and fed into the optimizer to optimize and update the network parameters so that the network loss converges, thus outputting the target mask.

[0161] In this embodiment, the output of a single-channel mask is converted into two outputs: one is a single-channel mask image corresponding to the loss function F1 / Dice / Laplacian for matting, and the other is a two-channel segmentation image corresponding to the loss function BCE for segmentation. By calculating the loss of different outputs using different loss functions, the accuracy of achieving the preset conditions is effectively improved, and the accuracy of the smearing-style image segmentation method is enhanced.

[0162] In the embodiments of this application, the model size is 7.82M. Under the same conditions, compared with related technologies, such as the SiNetv2 open-source model size of 103M, the space occupied is greatly reduced. Compared with other related technologies, such as (Scribble-to-Mask hereinafter referred to as S2M), the inference speed is nearly 10 times faster and the model size is reduced by 19 times while ensuring a certain effect. The onn×runtime-python inference speed of the model in the embodiments of this application is 16ms, while S2M is at least 100ms. The model size of the model in the embodiments of this application is 7.82M, while the model size of S2M is 152M. The embodiments of this application effectively speed up and compress the size by using the more efficient Ghost module to replace the ordinary convolutional group, the more efficient LRASPP to replace RFB, and the backbone network.

[0163] In summary, compared with related technical solutions, the embodiments of this application first segment the image to be segmented into scale feature maps of different scales, and then through the coordinated cooperation of different set modules, aggregation modules and different backpropagation modules, it can quickly make the sum of the first predicted segmentation loss and the second predicted segmentation loss meet the preset conditions, thereby effectively improving the processing speed of smearing image segmentation and ensuring the image segmentation effect. Moreover, due to the fast inference speed, the space occupied by the model is small, which effectively improves the user experience.

[0164] Please see Figure 6 This application also provides a smear-type image segmentation device 100, which may include:

[0165] The acquisition module 101 is used to acquire the image to be segmented, wherein the image to be segmented is processed into scale feature maps including different scales;

[0166] Processing module 102 is used for:

[0167] The scale feature maps of different scales are processed into different scale set feature maps by using different set modules;

[0168] The different scale feature maps are aggregated by the aggregation module to obtain the mask function;

[0169] Different final prediction masks are obtained by using the mask map and prediction guidance mask map of the mask function, and based on different backpropagation modules;

[0170] The predicted guidance mask and the different final predicted masks are respectively input into the aggregation module to obtain different first prediction segmentation losses and second segmentation losses;

[0171] When the sum of the first prediction segmentation loss and the second segmentation loss of different values ​​meets a preset condition, a target mask is obtained based on the prediction guidance mask and the different final prediction mask.

[0172] Optionally, as a possible implementation, in this embodiment of the application, when the acquisition module 101 is used to acquire the image to be segmented, and the image to be segmented is processed into scale feature maps including different scales, the image to be segmented is specifically used to: process the image to be segmented into a first scale feature map, a second scale feature map and a third scale feature map through a backbone network.

[0173] Optionally, as a possible implementation, in this embodiment of the application, when processing unit 102 processes the scale feature maps of different scales into different scale set feature maps through different set modules, it is specifically used for:

[0174] The first scale feature map is processed into a first scale set feature map through the first set module;

[0175] The second-scale feature map is processed into a second-scale set feature map through the second set module;

[0176] The third-scale feature map is processed into a third-scale set feature map through the third set module.

[0177] Optionally, as a possible implementation, in this embodiment of the application, before the processing unit 102 performs the step of aggregating the feature maps of the different scale sets through the aggregation module to obtain the mask map of the mask function and the prediction guidance mask map, it is further specifically used for:

[0178] The prediction guidance function is obtained by performing a first preset sampling on the mask function;

[0179] The prediction guidance mask is obtained based on the prediction guidance function.

[0180] Optionally, as a possible implementation, in this embodiment of the application, when the processing unit 102 implements the steps of obtaining different final prediction masks based on different backpropagation modules through the mask map and prediction guidance mask map obtained by the mask function, it is specifically used for:

[0181] By performing a second preset sampling on the mask function, a first guiding function is obtained, and based on the first guiding function, a first guiding mask image is obtained;

[0182] The third-scale set feature map and the first guide mask map are input into the first backpropagation module to obtain the first predicted reverse guide mask map.

[0183] The first predicted reverse guide mask map is added to the first guide mask map to obtain the first predicted mask feature map.

[0184] By performing a third preset sampling on the first predicted mask feature map, a second guiding function is obtained, and a second guiding mask map is obtained based on the second guiding function;

[0185] The second scale set feature map and the second guide mask map are input into the second backpropagation module to obtain the second predicted reverse guide mask map;

[0186] The second predicted reverse guide mask map is added to the second guide mask map to obtain the second predicted mask feature map.

[0187] as well as

[0188] By performing a third preset sampling on the second predicted mask feature map, a third guiding function is obtained, and based on the first guiding function, a third guiding mask map is obtained;

[0189] The first scale set feature map and the third guide mask map are input into the third backpropagation module to obtain the third predicted reverse guide mask map;

[0190] The third predicted reverse guide mask map is added to the third guide mask map to obtain the third predicted mask feature map.

[0191] Optionally, as a possible implementation, in this embodiment of the application, when the processing unit 102 implements the steps of obtaining different final prediction masks based on different backpropagation modules, the processing unit 102 is further configured to:

[0192] The first final prediction mask image is obtained by performing a first preset sampling on the first prediction mask feature map;

[0193] The second final prediction mask image is obtained by performing a fourth preset sampling on the second prediction mask feature map;

[0194] The third final prediction mask image is obtained by performing a fifth preset sampling on the third prediction mask feature map.

[0195] Optionally, as a possible implementation, in this embodiment of the application, when the processing unit 102 inputs the prediction guidance mask and the different final prediction masks into the aggregation module respectively, and obtains different first prediction segmentation losses and second segmentation losses, it is further specifically used for:

[0196] The predicted guidance mask and the different final predicted masks are converted into a first mode map and a second mode map;

[0197] The first segmentation loss is obtained by calculating the loss of the first pattern graph based on the first loss function;

[0198] The second segmentation loss is obtained by calculating the loss of the second pattern map based on the second loss function.

[0199] The data processing system in this application embodiment has been described above from the perspective of modular functional entities. Please refer to [link / reference]. Figure 6 The electronic device in the embodiments of this application is described below from the perspective of hardware processing:

[0200] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned smearing image segmentation device 100 and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0201] The aforementioned smear-type image segmentation device 100 can be implemented as a computer program, which can, for example... Figure 6 It runs on the computer device shown.

[0202] Please see Figure 6 Figure 200 is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 200 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0203] See ​ The computer device 200 includes a processor 202, a memory, and a network interface 205 connected via a system bus 201. The memory may include a computer-readable medium 203 containing non-volatile program code and internal memory 204.

[0204] The non-volatile computer-readable medium 203 may store an operating system 2031 and a computer program 2032. The computer program 9032 includes program instructions that, when executed, cause the processor 202 to perform a time-frequency combined smearing image segmentation method.

[0205] The processor 202 provides computing and control capabilities to support the operation of the entire computer device 200.

[0206] The internal memory 204 provides an environment for the execution of a computer program 2032 in a computer-readable medium 203 containing non-volatile program code. When the computer program 2032 is executed by the processor 202, the processor 202 can perform a smearing image segmentation method.

[0207] This network interface 205 is used for network communication with other devices. Those skilled in the art will understand that... ​ The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 200 to which the present application is applied. The specific computer device 200 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0208] The processor 202 is used to run a computer program 2032 stored in the memory to perform the following steps:

[0209] Obtain the image to be segmented, which is processed into scale feature maps including different scales;

[0210] The scale feature maps of different scales are processed into different scale set feature maps by using different set modules;

[0211] The different scale feature maps are aggregated by the aggregation module to obtain the mask function;

[0212] Different final prediction masks are obtained by using the mask map and prediction guidance mask map of the mask function, and based on different backpropagation modules;

[0213] The predicted guidance mask and the different final predicted masks are respectively input into the aggregation module to obtain different first prediction segmentation losses and second segmentation losses;

[0214] When the sum of the first prediction segmentation loss and the second segmentation loss of different values ​​meets a preset condition, a target mask is obtained based on the prediction guidance mask and the different final prediction mask.

[0215] Optionally, as a possible implementation, in this embodiment of the application, when the acquisition module 101 is used to acquire the image to be segmented, and the image to be segmented is processed into scale feature maps including different scales, the image to be segmented is specifically used to: process the image to be segmented into a first scale feature map, a second scale feature map and a third scale feature map through a backbone network.

[0216] In some implementations, when processing unit 202 processes the scale feature maps of different scales into different scale set feature maps through different set modules, it is specifically used for:

[0217] The first scale feature map is processed into a first scale set feature map through the first set module;

[0218] The second-scale feature map is processed into a second-scale set feature map through the second set module;

[0219] The third-scale feature map is processed into a third-scale set feature map through the third set module.

[0220] In some embodiments, before performing the step of aggregating the feature maps of the different scale sets through the aggregation module to obtain the mask map of the mask function and the prediction guidance mask map, the processing unit 202 is specifically used for:

[0221] The prediction guidance function is obtained by performing a first preset sampling on the mask function;

[0222] The prediction guidance mask is obtained based on the prediction guidance function.

[0223] In some implementations, when processing unit 202 implements the steps of obtaining different final prediction masks based on different backpropagation modules, it is specifically used for:

[0224] By performing a second preset sampling on the mask function, a first guiding function is obtained, and based on the first guiding function, a first guiding mask image is obtained;

[0225] The third-scale set feature map and the first guide mask map are input into the first backpropagation module to obtain the first predicted reverse guide mask map.

[0226] The first predicted reverse guide mask map is added to the first guide mask map to obtain the first predicted mask feature map.

[0227] By performing a third preset sampling on the first predicted mask feature map, a second guiding function is obtained, and a second guiding mask map is obtained based on the second guiding function;

[0228] The second scale set feature map and the second guide mask map are input into the second backpropagation module to obtain the second predicted reverse guide mask map;

[0229] The second predicted reverse guide mask map is added to the second guide mask map to obtain the second predicted mask feature map.

[0230] as well as

[0231] By performing a third preset sampling on the second predicted mask feature map, a third guiding function is obtained, and based on the first guiding function, a third guiding mask map is obtained;

[0232] The first scale set feature map and the third guide mask map are input into the third backpropagation module to obtain the third predicted reverse guide mask map;

[0233] The third predicted reverse guide mask map is added to the third guide mask map to obtain the third predicted mask feature map.

[0234] In some embodiments, when processing unit 202 implements the steps of obtaining different final prediction masks based on different backpropagation modules, it is further configured to:

[0235] The first final prediction mask image is obtained by performing a first preset sampling on the first prediction mask feature map;

[0236] The second final prediction mask image is obtained by performing a fourth preset sampling on the second prediction mask feature map;

[0237] The third final prediction mask image is obtained by performing a fifth preset sampling on the third prediction mask feature map.

[0238] In some embodiments, when processing unit 202 performs the step of inputting the prediction guidance mask and the different final prediction masks into the aggregation module respectively, and obtaining different first prediction segmentation losses and second segmentation losses, it is specifically further used for:

[0239] The predicted guidance mask and the different final predicted masks are converted into a first mode map and a second mode map;

[0240] The first segmentation loss is obtained by calculating the loss of the first pattern graph based on the first loss function;

[0241] The second segmentation loss is obtained by calculating the loss of the second pattern map based on the second loss function.

[0242] It should be understood that, in the embodiments of this application, the processor 202 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0243] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0244] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0245] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0246] The steps in the methods of this application embodiment can be adjusted, merged, or deleted according to actual needs. The units in the apparatus of this application embodiment can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0247] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0248] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A smearing-style image segmentation method, characterized in that, include: Obtain the image to be segmented, which is processed into scale feature maps including different scales; The scale feature maps of different scales are processed into different scale set feature maps by using different set modules; The different scale feature maps are aggregated by the aggregation module to obtain the mask function; Different final prediction masks are obtained by using the mask map and prediction guidance mask map of the mask function, and based on different backpropagation modules; The predicted guide mask and the different final predicted masks are respectively input into the aggregation module to obtain different first predicted segmentation loss and second predicted segmentation loss. The first predicted segmentation loss is calculated using the BCE segmentation loss function, and the second predicted segmentation loss is calculated using the matting loss function. When the sum of the different first prediction segmentation losses and the different second prediction segmentation losses satisfies the preset convergence condition, the target mask map is obtained according to the prediction guidance mask map and the different final prediction mask maps.

2. The method according to claim 1, characterized in that, The process of acquiring the image to be segmented, wherein the image to be segmented is processed into scale feature maps including different scales, includes: The image to be segmented is processed by a backbone network into a first-scale feature map, a second-scale feature map, and a third-scale feature map.

3. The method according to claim 2, characterized in that, The process of processing the scale feature maps of different scales into different scale set feature maps through different set modules includes: The first scale feature map is processed into a first scale set feature map through the first set module; The second-scale feature map is processed into a second-scale set feature map through the second set module; The third-scale feature map is processed into a third-scale set feature map through the third set module.

4. The method according to claim 3, characterized in that, Before the step of aggregating the feature maps of the different scale sets through the aggregation module to obtain the mask map of the mask function and the prediction guidance mask map, the method further includes: The prediction guidance function is obtained by performing a first preset sampling on the mask function; The prediction guidance mask is obtained based on the prediction guidance function.

5. The method according to claim 3, characterized in that, The process involves using the mask image and prediction guidance mask obtained through the mask function, and based on different backpropagation modules, to obtain different final prediction masks, including: By performing a second preset sampling on the mask function, a first guiding function is obtained, and based on the first guiding function, a first guiding mask image is obtained; The third-scale set feature map and the first guide mask map are input into the first backpropagation module to obtain the first predicted reverse guide mask map. The first predicted reverse guide mask map is added to the first guide mask map to obtain the first predicted mask feature map.

6. The method according to claim 5, characterized in that, The process of obtaining different final prediction masks based on the mask map and prediction guidance mask map obtained through the mask function, and based on different backpropagation modules, also includes: By performing a third preset sampling on the first predicted mask feature map, a second guiding function is obtained, and a second guiding mask map is obtained based on the second guiding function; The second scale set feature map and the second guide mask map are input into the second backpropagation module to obtain the second predicted reverse guide mask map; The second predicted reverse guide mask map is added to the second guide mask map to obtain the second predicted mask feature map.

7. The method according to claim 6, characterized in that, The process of obtaining different final prediction masks based on the mask map and prediction guidance mask map obtained through the mask function, and based on different backpropagation modules, also includes: By performing a third preset sampling on the second predicted mask feature map, a third guiding function is obtained, and based on the third guiding function, a third guiding mask map is obtained; The first scale set feature map and the third guide mask map are input into the third backpropagation module to obtain the third predicted reverse guide mask map; The third predicted reverse guide mask map is added to the third guide mask map to obtain the third predicted mask feature map.

8. A smear-type image segmentation device, characterized in that, include: An acquisition unit is used to acquire an image to be segmented, wherein the image to be segmented is processed into scale feature maps including different scales; Processing unit, used for: The scale feature maps of different scales are processed into different scale set feature maps by using different set modules; The different scale feature maps are aggregated by the aggregation module to obtain the mask function; Different final prediction masks are obtained by using the mask map and prediction guidance mask map of the mask function, and based on different backpropagation modules; The predicted guide mask and the different final predicted masks are respectively input into the aggregation module to obtain different first predicted segmentation loss and second predicted segmentation loss. The first predicted segmentation loss is calculated using the BCE segmentation loss function, and the second predicted segmentation loss is calculated using the matting loss function. When the sum of the different first prediction segmentation losses and the different second prediction segmentation losses satisfies the preset convergence condition, the target mask map is obtained according to the prediction guidance mask map and the different final prediction mask maps.

9. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store and support the processor in executing a program of any one of claims 1 to 7, the processor being configured to execute the program stored in the memory.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the method of any one of claims 1 to 7.

Images