An automatic image cutout method and system based on salient object detection

Abstract

The present invention belongs to the field of computer vision technology and discloses an automatic cutout method based on salient target detection and its supporting system. The implementation method includes: inputting an image into a salient target detection neural network module to obtain a foreground and background separation mask; then entering a cutout module to mask the mask with the original image to generate an image with a transparent background and a visible foreground; the user determines whether further modification is required based on the cutout result; if the cutout result is complete and accurate, the cutout result generated by the salient target detection neural network can be directly used; if the cutout is incomplete or insufficiently accurate, the interactive cutout module is entered, and the user makes local modifications based on the actual situation; finally, the modified mask is used together with the original image for calculation and cutout. This method can automatically cut out the foreground of the original image and can also more accurately edit the automatic cutout result, conveniently and efficiently assisting users in achieving precise cutout.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and more particularly to an automatic image cutout method and system based on salient object detection. Background Art

[0002] On e-commerce design platforms, users frequently perform various image editing operations, with cutout being a frequently used technique. Specifically, cutout involves segmenting a target area within an image to precisely separate the foreground and background. It's a crucial technique in image and video processing, and designers typically utilize tools like Photoshop to achieve this. However, this requires a high level of proficiency in image processing software, and cutting out complex objects (such as hair, meshes, and translucent glass) can be time-consuming and labor-intensive, resulting in low efficiency and instability.

[0003] With the advancement of image processing technology, especially deep convolutional neural networks (CNNs) in recent years, many excellent image matting algorithms have emerged. In 2017, Xu et al. (Ning Xu, Brian Price, Scott Cohen, and Thomas Huang, Deep Image Matting, CVPR 2017) proposed: First, a deep convolutional encoder-decoder network takes the image and its corresponding trimap as input to predict the image's alpha matte. Then, a small convolutional network is used to fine-tune the mask obtained in the first step, resulting in a result with sharper edges. In 2019, Cai et al. (Shaofan Cai, Xiaoshuai Zhang, Haoqiang Fan, Haibin Huang, Jiangyu Liu, Jiaming Liu, Jiaying Liu, Jue Wang, and Jian Sun, Disentangled Image Matting, ICCV 2019) proposed decomposing the image matting problem into two subtasks: trimap adaptation (a classification task) and alpha estimation (a correction task). Among them, trimap adaptation is a pixel-level classification problem that infers the global structure of the input image by identifying certain foreground, background, and semi-transparent image regions. Alpha estimation is a regression problem that calculates the opacity value of each pixel.

[0004] Although these methods have achieved significant breakthroughs in the accuracy of the final cutout, they still rely heavily on trimaps, and the accuracy of trimaps significantly impacts the final cutout results, preventing true end-to-end, high-precision automatic cutout. To address the shortcomings of salient object detection algorithms in practical applications with multiple objects and complex scenes, the present invention also provides an auxiliary cutout system that combines automatic and interactive cutouts. Summary of the Invention

[0005] In order to solve the problems existing in the existing technology, inspired by the principle of salient target detection, the present invention proposes an automatic cutout method based on salient target detection. The algorithm can automatically identify the most visually attractive objects in the image and accurately segment them, and the input and intermediate processes are completely independent of the trimap, thus achieving true end-to-end automatic cutout. At the same time, in order to solve the drawbacks of the salient target detection algorithm in practical applications under multi-target and complex scenes, and to allow users to flexibly edit and modify the cutout objects, the present invention also provides an automatic cutout system based on salient target detection.

[0006] To achieve the above objectives, the first aspect of the present invention proposes an automatic image cutout method based on salient object detection, comprising:

[0007] S1_1. First, the image is input into the Res_Swish backbone network, which consists of a "Bottleneck" module after replacing the original ReLU activation function with the more accurate Swish function. It then goes through the five encoding stages (Encode_stage1, Encode_stage2, Encode_stage3, Encode_stage4, Encode_stage5) and the five decoding stages (Decode_stage4, Decode_stage3, Decode_stage2, Decode_stage1). At the same time, each output of the Encode stage is added to the symmetrical Decode stage to effectively utilize multi-scale feature information.

[0008] S1_2, respectively, performs convolution operations on the output of the Decode_stage1 stage with a kernel size of 3*3 and padding=1. Since the Decode_stage1 stage does not perform downsampling, no upsampling operation is required to directly obtain the output tensor M1. Then, the output of the Decode_stage2, Decode_stage3, Decode_stage4, and Decode_stage5 stages are respectively convolved with a kernel size of 3*3 and padding=1, and bilinear interpolation is used to upsample them to output tensors M2, M3, M4, and M5 of the same size as the input.

[0009] S1_3. Concatenate M1, M2, M3, M4, and M5 horizontally into a 6-channel tensor M0 according to dimension 1, and then convolve M0 into a single-channel tensor using a convolution kernel of size 1*1.

[0010] S1_4. Finally, perform a sigmoid operation on the single-channel tensor obtained in step S1_3 to obtain the probability matrix M[:, :] for each pixel belonging to the foreground and background. Multiply the probability matrix M by 255 to obtain the mask map Alpha_SOD predicted by the salient object detection module.

[0011] To achieve the above objectives, the second aspect of the present invention proposes an automatic image cutout system based on salient object detection, comprising:

[0012] S2_1. Input the image to be processed into the salient object detection module. The trained neural network model automatically infers the probability that each pixel in the image belongs to the foreground. , we get the two-dimensional probability matrix M[:, :] of the entire image. Multiply the probability matrix M by 255 to get the mask map Alpha_SOD predicted by the salient object detection module.

[0013] S2_2, import the mask image Alpha_SOD obtained in step S2_1 and the original image Image into the matting module to obtain the foreground and background separation image Matting;

[0014] S2_3. The user judges the effect of the matting image obtained in step S2_2: if it meets the requirements of the completeness and accuracy of the main area matting, the user downloads the matting image for use (steps S2_4 to S2_9 do not need to be executed); otherwise, the user enters the interactive matting module;

[0015] S2_4, binarize the Alpha_SOD obtained in step S2_1 to obtain a binary image Binary_alpha1;

[0016] S2_5. Perform mask calculation on the binary image Binary_alpha1 obtained in step S2_4 and the original image Image. Use the GrabCut-based interactive cutout algorithm (blue strokes represent retention, red areas represent deletion) to locally modify the main area to obtain a more accurate binary mask image Binary_alpha2.

[0017] S2_6. Calculate the difference between Binary_alpha1 and Binary_alpha2, and generate a trimap for the edges of the difference pixel areas by erosion, dilation, etc. (Trimap is generated only for the modified areas, not for all areas).

[0018] S2_7, input the original image Image and the Trimap generated in step S2_6 into the trained trimap-dependent neural network model to generate a more accurate mask image Alpha_mat;

[0019] S2_8, import the more accurate mask image Alpha_mat obtained in step S2_7 into the matting module, and calculate to obtain the matting image;

[0020] S2_9, the user judges the effect of the matting image obtained in step S2_8: if it meets the requirements of the completeness and accuracy of the matting of the main area, the matting image is downloaded; otherwise, the interactive matting module is entered again and steps S2_4 to S2_8 are executed until the matting effect meets the user's satisfaction;

[0021] Compared to the online cutout tools offered by commercial e-commerce design platforms, this paper proposes an automatic cutout method based on salient object detection. This method first trains a salient object detection neural network using a large amount of high-precision training data. It then uses the trained neural network to infer the image to be cutout, achieving precise cutout results. This process requires no human interaction and is independent of trimaps. After manual verification and review, the proposed automatic cutout method based on salient object detection has an accuracy rate of approximately 90% for the completeness of the cutout of the main image area, significantly reducing manual intervention and improving cutout efficiency.

[0022] At the same time, in order to better solve the drawback of the salient object detection algorithm in actual application of inaccurate segmentation of the subject area in individual multi-object and complex scenes, the present invention also provides an automatic cutout system based on salient object detection, which allows users to edit and modify the subject area according to actual conditions, further improving the flexibility of the cutout product and the accuracy of the final cutout effect; BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a flow chart of the overall system framework of the present invention.

[0024] Figure 2 This is a neural network framework diagram for salient object detection based on the U-Net structure of the present invention.

[0025] Figure 3 This is the structural diagram of the Res_Swish module improved based on the "Bottleneck" module of the present invention.

[0026] Figure 4 It is a specific effect diagram of the present invention. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0028] According to the first aspect of the present invention, this embodiment discloses an automatic image cutout method based on salient object detection. Figure 2 As shown, further operation description of this embodiment is as follows:

[0029] The overall network proposed in step S1_1 is an improvement on the classic U-Net architecture of the Encode-Decode structure, in which the encoding and decoding network units in each stage are improved from the "Bottleneck" module. Specifically, the ReLU activation function of the "Bottleneck" module is replaced with the more accurate Swish function to form a new Res_Swish network (structure as shown in Figure 2). Figure 3 As shown). The Swish function is expressed as:

[0030]

[0031] Where x represents the input. The input image is uniformly resized to 480*480. In the encoding stage, max pooling with kernel_size=2 and stride=2 is used for downsampling. In the decoding stage, bilinear interpolation is used for upsampling. The feature extraction part goes through five stages of encoding, Encode_stage1, Encode_stage2, Encode_stage3, Encode_stage4, Encode_stage5, and five stages of decoding, Decode_stage4, Decode_stage3, Decode_stage2, Decode_stage1, with corresponding spatial resolutions of 480*480, 240*240, 120*120, 60*60, and 30*30 respectively; in step S1_1, each output of the Encode stage is added to the symmetrical Decode stage, mainly considering that: in the multi-level feature extraction process in the encoding stage, feature information of different scales has different information, among which the lower-level high-resolution features contain rich local information, which is helpful for the detection of significant target key points, while the high-level low-resolution features contain the global semantic information of the entire target, so the use of multi-scale feature information helps to improve the accuracy of the neural network;

[0032] In step S1_2, since the size of the decode_stage1 stage is still 480*480 without downsampling, it is only necessary to perform a convolution operation with a kernel size of 3*3 and padding=1 on the output of the decode_stage1 stage to directly obtain the output tensor M1. However, due to downsampling in other decoding stages, after performing a convolution operation with a kernel size of 3*3 and padding=1 on the output of the decode_stage2, decode_stage3, decode_stage4, and decode_stage5 stages, it is necessary to use a bilinear interpolation algorithm to upsample to 480*480. The corresponding output tensors are recorded as M2, M3, M4, and M5.

[0033] In step S1_3, M1, M2, M3, M4, and M5 are horizontally spliced ​​to generate a 6-channel tensor M0, and then M0 is convolved into a single-channel tensor using a convolution kernel of size 1*1. This step realizes the fusion of multi-scale feature information at different stages.

[0034] ;

[0035] Among them, M0 represents the fused feature map, conv represents the convolutional layer, and concat represents the horizontal splicing operation.

[0036] In step S1_4, a sigmoid operation is performed on the single-channel feature map M0 obtained in step S1_3 to obtain the probability matrix M[:, :] for each pixel belonging to the foreground and background. This probability matrix M is multiplied by 255 to obtain the mask map Alpha_SOD predicted by the salient object detection module. During training, the model parameters are optimized using binary cross entropy loss, with the loss function being: ;

[0037] Among them, h and w represent the height and width of the image respectively, t i represents the true probability that the ground truth label sample belongs to the foreground at pixel i, o i Represents the predicted probability of the neural network at pixel i belonging to the foreground.

[0038] The entire loss process uses a multi-level binary cross entropy loss function to supervise the multi-layer network, including 6 loss functions for feature maps M0, M1, M2, M3, M4, and M5. The full loss function is defined as follows:

[0039]

[0040] Among them, loss tota l represents the total loss, loss M0 to loss M5 The overall salient object detection neural network was trained for 600 epochs using the Adam optimization algorithm with an initial learning rate of 0.001 and an exponential descent learning rate decay strategy.

[0041] To validate the effectiveness of this method, we randomly selected 1,000 images uploaded by users and used the trained model for predictions. The predictions were manually reviewed, using completeness and accuracy as metrics. The statistical results showed that both completeness and accuracy of the main area cutout were around 90%, demonstrating the method's robust end-to-end cutout performance.

[0042] According to the second aspect of the present invention, this embodiment discloses an automatic image cutout system based on salient object detection. Figure 1 As shown, further operation description of this embodiment is as follows:

[0043] In step S2_1, the image to be processed is input into the salient object detection module, and the trained salient object detection neural network model automatically infers the probability that each pixel in the image belongs to the foreground. , we get the two-dimensional probability matrix M[:, :] of the entire image. Multiply the probability matrix M by 255 to get the mask map Alpha_SOD predicted by the salient object detection module.

[0044] In step S2_2, the mask image Alpha_SOD obtained in step S2_1 and the original image Image are imported into the matting module to obtain the foreground and background separation image Matting.

[0045] In step S2_3, the user judges the effect of the matting image obtained in step S2_2. If it meets the requirements, the user downloads the matting image for use (steps S2_4 to S2_9 do not need to be executed); otherwise, the user enters the interactive matting module.

[0046] In step S2_4, the Alpha_SOD obtained in step S2_1 is binarized to obtain a binary image Binary_alpha1;

[0047] In step S2_5, the binary image Binary_alpha1 obtained in step S2_4 is masked with the original image Image, and the main area is locally modified using the GrabCut-based interactive cutout algorithm (blue strokes represent retention, and red areas represent deletion) to obtain a more accurate binary mask image Binary_alpha2.

[0048] In step S2_6, the difference between Binary_alpha1 and Binary_alpha2 is calculated, and the edges of the difference pixel areas are processed by corrosion, dilation, etc. to generate trimap maps for those areas (Trimaps are only generated for the modified areas, not for all areas).

[0049] In step S2_7, the original image Image and the Trimap generated in step S2_6 are input into the trained trimap-dependent neural network model to generate a more accurate mask image Alpha_mat.

[0050] In step S2_8, the more accurate mask image Alpha_mat obtained in step S2_7 is imported into the matting module to obtain the matting image.

[0051] In step S2_9, the user judges the effect of the cutout obtained in step S2_8: if it meets the requirements of the completeness and accuracy of the cutout of the main area, the matting map is downloaded; otherwise, the interactive cutout module is entered again and steps S2_4 to S2_8 are executed until the cutout effect satisfies the user.

[0052] Compared with the existing auxiliary cutout tools on the market, the present invention first uses the salient target detection neural network to obtain accurate cutout results, reducing the uncertainty interference introduced by manual interaction and trimap, and truly realizing end-to-end automatic cutout. The present invention improves the U-Net network, integrates multi-scale feature information at different stages, and has a strong detail segmentation ability (such as Figure 4 To address the shortcomings of salient object detection algorithms in practical applications, the present invention also provides an automatic cutout system based on salient object detection. This system allows users to make secondary adjustments to the subject area based on the actual cutout results from the first step, further enhancing the flexibility of the cutout product and achieving more accurate cutout results. This significantly reduces the difficulty of cutouts and improves the image editing efficiency of e-commerce design platforms.

[0053] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations that come within the meaning and range of equivalents of the claims be embraced therein.

Claims

1. An automatic image cutout system based on salient object detection, characterized in that: The following steps are involved: S2_1. Input the image to be processed Image into the salient object detection module. The trained neural network model automatically infers the probability that each pixel in the image belongs to the foreground, and obtains the two-dimensional probability matrix M[:, :] of the entire image. Multiply the probability matrix M by 255 to obtain the mask map Alpha_SOD predicted by the salient object detection module; S2_2, import the mask image Alpha_SOD obtained in step S2_1 and the original image Image into the matting module to obtain the foreground and background separation image Matting; S2_3. The user judges the effect of the matting image obtained in step S2_2: if it meets the requirements of the completeness and accuracy of the main area matting, the matting image is downloaded and used, and steps S2_4 to S2_9 do not need to be performed; otherwise, the interactive matting module is entered; S2_4, binarize the Alpha_SOD obtained in step S2_1 to obtain a binary image Binary_alpha1; S2_5. Perform mask calculation on the binary image Binary_alpha1 obtained in step S2_4 and the original image Image. Use the interactive cutout algorithm based on GrabCut. The blue strokes represent retention and the red areas represent deletion. Perform local modification on the main area to obtain a more accurate binary mask image Binary_alpha2. S2_6. Calculate the difference between Binary_alpha1 and Binary_alpha2, and generate a trimap for the locally modified area by eroding and dilating the edges of the difference pixel areas. Only the modified area is generated with a trimap, and the trimap for the non-whole area is regenerated. S2_7, input the original image Image and the Trimap generated in step S2_6 into the trained trimap-dependent neural network model to generate a more accurate mask image Alpha_mat; S2_8, import the more accurate mask image Alpha_mat obtained in step S2_7 into the matting module, and calculate to obtain the matting image; S2_9. The user judges the effect of the cutout obtained in step S2_8: if it meets the requirements of the completeness and accuracy of the cutout of the main area, the matting map is downloaded; otherwise, the interactive cutout module is entered again and steps S2_4 to S2_8 are executed until the cutout effect satisfies the user.

2. An automatic image cutout method based on salient object detection applied to the automatic image cutout system based on salient object detection according to claim 1, characterized in that: The following steps are involved: S1_1. First, the image is input into the Res_Swish network composed of the "Bottleneck" module after the original ReLU activation function is replaced with the more accurate Swish function. After the five encoding stages Encode_stage1, Encode_stage2, Encode_stage3, Encode_stage4, Encode_stage5 and the five decoding stages Decode_stage4, Decode_stage3, Decode_stage2, Decode_stage1, each output of the Encode stage is added to the symmetrical Decode stage to effectively utilize multi-scale feature information; S1_2, respectively, perform convolution operations on the output of the Decode_stage1 stage with a convolution kernel size of 3*3 and padding=1. Since the Decode_stage1 stage does not perform downsampling, the output tensor M1 is directly obtained without upsampling operation. Then, the output of the Decode_stage2, Decode_stage3, Decode_stage4, and Decode_stage5 stages are respectively subjected to convolution operations with a convolution kernel size of 3*3 and padding=1, and bilinear interpolation algorithm is used to upsample to output tensors M2, M3, M4, and M5 of the same size as the input. S1_3, horizontally splice M1, M2, M3, M4, and M5 into a 6-channel tensor M0 according to dimension 1, and then convolve M0 into a single-channel tensor using a convolution kernel of size 1*1; S1_4. Finally, perform a sigmoid operation on the single-channel tensor obtained in step S1_3 to obtain the probability matrix M[:, :] for each pixel belonging to the foreground and background. Multiply the probability matrix M by 255 to obtain the mask map Alpha_SOD predicted by the salient object detection module.

3. The automatic image cutout method based on salient object detection according to claim 2, characterized in that: The neural network based on salient object detection is improved on the classic U-Net architecture of the Encode-Decode structure. The encoding and decoding network units in each stage are improved from the "Bottleneck" module. Specifically, the ReLU activation function of the "Bottleneck" module is replaced with the more accurate Swish function to form a new Res_Swish network. The Swish function is expressed as: ； Here, x represents the input.

4. The automatic image cutout method based on salient object detection according to claim 2, wherein: In step S1_3, M1, M2, M3, M4, and M5 are horizontally spliced ​​to generate a 6-channel tensor M0, and then M0 is convolved into a single-channel tensor using a convolution kernel of size 1*1. This step realizes the fusion of multi-scale feature information at different stages. The formula is as follows: ； Among them, M0 represents the fused feature map, conv represents the convolutional layer, and concat represents the horizontal splicing operation.

5. The automatic image cutout method based on salient object detection according to claim 2, characterized in that: During the training process, binary cross entropy loss is used to optimize the model parameters. The loss function is: ； Among them, h and w represent the height and width of the image respectively, t i represents the true probability that the ground truth label sample belongs to the foreground at pixel i, o i Represents the predicted probability of the neural network at pixel i belonging to the foreground.

6. The automatic image cutout method based on salient object detection according to claim 2, characterized in that: The entire loss process uses a multi-level binary cross entropy loss function to supervise the multi-layer network, including 6 loss functions for feature maps M0, M1, M2, M3, M4, and M5. The full loss function is defined as follows: ； Among them, losstotal represents the total loss, and lossM0 to lossM5 represent the losses in the corresponding stages.

Images