Foreground segmentation method, device, equipment and storage medium

Abstract

The present application discloses a foreground segmentation method, apparatus, device and storage medium, which relates to the field of image technology. The method comprises: inputting the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained to determine multiple alignment condition representations, wherein the network is composed of an image embedding module, an intermediate state mask alignment module and a multi-stage cascade encoding module; inputting the multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, thereby obtaining a target foreground segmentation model, wherein the denoising network includes a downsampling unit and an upsampling unit, wherein the downsampling unit includes a cascade flow and a fusion flow; inputting the foreground image to be segmented into the target foreground segmentation model to obtain a segmentation mask. By the above method, the target foreground segmentation model has excellent performance and cross-task generalization ability, ensuring that both accuracy and generalization can be guaranteed in the foreground segmentation task.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image technology, in particular to a foreground segmentation method and device, equipment and a storage medium. BACKGROUND

[0002] In order to obtain the characteristics in a single task domain, the traditional single-task foreground segmentation model often designs a complex and ingenious network structure and a customized training method, resulting in a lack of generalization ability in the field of foreground segmentation. In addition, these models tend to overfit to the data distribution of a specific task. In contrast, the current general model for solving multiple tasks of foreground segmentation starts from the perspective of obtaining the commonality between multiple tasks, extracting low-level features such as frequency, multi-scale, texture and edge; however, such a way is easy to ignore the characteristics between different tasks and cannot accurately distinguish the emphasis between different tasks. The discriminative network faces the dilemma of being unable to simultaneously consider task performance and task generalization. In addition, some methods based on diffusion models use their image generation capabilities to solve the segmentation task from a generated perspective; however, due to the complex and large network architecture of the original diffusion model and the improper adaptation of the denoising mechanism to the segmentation task, the performance of such methods is far inferior to that of previous methods. SUMMARY

[0003] The main purpose of the present application is to provide a foreground segmentation method, device, equipment and storage medium, which aims to solve the technical problem that the foreground segmentation model in the prior art cannot simultaneously guarantee accuracy and generalization when performing a foreground segmentation task.

[0004] To achieve the above-mentioned purpose, the present application provides a foreground segmentation method, which comprises:

[0005] inputting the intermediate state mask of each sample training image into an alignment condition extraction network in a to-be-trained segmentation model for alignment to determine a plurality of alignment condition representations of each sample training image, wherein the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module and a multi-stage cascaded encoding module;

[0006] inputting the plurality of alignment condition representations of each sample training image into a bidirectional cascaded fusion denoising network in the to-be-trained segmentation model for denoising to obtain a target foreground segmentation model, wherein the bidirectional cascaded fusion denoising network comprises a downsampling unit and an upsampling unit, and a cascaded flow and a fusion flow are established between a plurality of progressive fusion units in the downsampling unit;

[0007] inputting a to-be-segmented foreground image into the target foreground segmentation model to obtain a foreground segmentation mask of the to-be-segmented foreground image.

[0008] In an embodiment, the step of inputting the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment comprises:

[0009] The intermediate state mask of each sample training image and each sample training image are input into the alignment condition extraction network in the segmentation model to be trained, and each sample training image is processed by an image embedding module in the alignment condition extraction network to obtain a plurality of first processed images;

[0010] The intermediate state mask alignment module in the alignment condition extraction network aligns each first processed image and the intermediate state mask of each sample training image to obtain the to-be-encoded feature of each sample training image;

[0011] The to-be-encoded feature of each sample training image is encoded by a multi-order cascaded encoding module in the alignment condition extraction network to obtain a plurality of alignment condition representations of each sample training image.

[0012] In an embodiment, the step of aligning each first processed image and the intermediate state mask of each sample training image by the intermediate state mask alignment module in the alignment condition extraction network to obtain the to-be-encoded feature of each sample training image comprises:

[0013] The intermediate state mask alignment module in the alignment condition extraction network respectively embeds the intermediate state mask of each sample training image according to the image dimension of each first processed image to obtain the embedded feature of each sample training image;

[0014] The embedded feature of each sample training image and each first processed image are flattened to obtain a first feature sequence and a second feature sequence of each sample training image;

[0015] The first feature sequence and the second feature sequence of each sample training image are fused and aligned to obtain the to-be-encoded feature of each sample training image.

[0016] In an embodiment, the step of fusing and aligning the first feature sequence and the second feature sequence of each sample training image to obtain the to-be-encoded feature of each sample training image comprises:

[0017] A plurality of projection layers in the intermediate state mask alignment module fuse the first feature sequence and the second feature sequence of each sample training image to obtain a fused feature sequence of each sample training image;

[0018] A plurality of multi-layer perceptrons and activation layers in the intermediate state mask alignment module align the fused feature sequence of each sample training image to obtain a fused alignment feature of each sample training image;

[0019] Residual connection is performed between the fused alignment features of each sample training image and each first processing image after passing through the upper projection layer in the intermediate state mask alignment module, to obtain the to-be-encoded features of each sample training image.

[0020] In an embodiment, the step of inputting the multiple alignment condition representations of each sample training image into the bidirectional cascaded fusion denoising network in the to-be-trained segmentation model for denoising to obtain the target foreground segmentation model comprises:

[0021] The multiple alignment condition representations of each sample training image are input into the bidirectional cascaded fusion denoising network in the to-be-trained segmentation model, and the multiple alignment condition representations of each sample training image are fused in a first preset direction through a cascaded flow in the bidirectional cascaded fusion denoising network to output a cascaded feature of each sample training image.

[0022] The feature mapping of each sample training image is output by a fusion flow in the bidirectional cascaded fusion denoising network according to the cascaded feature of each sample training image and the intermediate state mask of each sample training image.

[0023] The feature mapping of each sample training image is input into an up-sampling unit in the cascaded fusion denoising network for feature restoration to obtain a target foreground segmentation network.

[0024] In an embodiment, the step of outputting the feature mapping of each sample training image by the fusion flow in the bidirectional cascaded fusion denoising network according to the cascaded feature of each sample training image and the intermediate state mask of each sample training image comprises:

[0025] The intermediate state of each sample training image is obtained by down-sampling the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascaded fusion denoising network.

[0026] The embedding representation of each sample training image is obtained by fusing the intermediate state of each sample training image and the cascaded feature of each sample training image.

[0027] The cell layer output corresponding to each sample training image is obtained by step-by-step fusion of the embedding representation of each sample training image and the time step.

[0028] The feature mapping of each sample training image is obtained according to the cell layer output corresponding to each sample training image and the embedding representation of each sample training image.

[0029] In an embodiment, before the step of inputting the multiple intermediate state masks of each sample training image into the alignment condition extraction network in the to-be-trained segmentation model for alignment to determine the multiple alignment condition representations of each sample training image, the method further comprises:

[0030] obtain each sample training image and a ground truth mask of each sample training image;

[0031] forwardly add noise to the ground truth mask of each sample training image at randomly selected time steps to obtain an intermediate state mask of each sample training image.

[0032] In addition, to achieve the above object, the present application further provides a foreground segmentation device, which comprises:

[0033] an alignment module configured to input the intermediate state mask of each sample training image into an alignment condition extraction network in a to-be-trained segmentation model to determine a plurality of alignment condition representations of each sample training image, wherein the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module and a multi-stage cascaded encoding module;

[0034] a denoising module configured to input the plurality of alignment condition representations of each sample training image into a bidirectional cascaded fusion denoising network in the to-be-trained segmentation model to obtain a target foreground segmentation model, wherein the bidirectional cascaded fusion denoising network comprises a downsampling unit and an upsampling unit, and a cascaded flow and a fusion flow are established between a plurality of progressive fusion units in the downsampling unit;

[0035] a segmentation module configured to input a to-be-segmented foreground image into the target foreground segmentation model to obtain a foreground segmentation mask of the to-be-segmented foreground image.

[0036] In addition, to achieve the above object, the present application further provides a foreground segmentation device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the foreground segmentation method as described above.

[0037] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and the computer program is executed by a processor to implement the steps of the foreground segmentation method as described above.

[0038] In addition, to achieve the above object, the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the foreground segmentation method as described above.

[0039] The application inputs the intermediate state mask of each sample training image into an alignment condition extraction network in a to-be-trained segmentation model for alignment to determine a plurality of alignment condition representations of each sample training image, the alignment condition extraction network being composed of an image embedding module, an intermediate state mask alignment module and a multi-stage cascaded encoding module; the plurality of alignment condition representations of each sample training image are input into a bidirectional cascaded fusion denoising network in the to-be-trained segmentation model for denoising to obtain a target foreground segmentation model, the bidirectional cascaded fusion denoising network comprising a downsampling unit and an upsampling unit, a plurality of progressive fusion units in the downsampling unit establishing a cascaded flow and a fusion flow therebetween; a to-be-segmented foreground image is input into the target foreground segmentation model to obtain a foreground segmentation mask of the to-be-segmented foreground image. In the foregoing manner, the alignment module based on the intermediate state in the mask diffusion process is used to improve the multi-task generalization capability, the bootstrap strategy based on the previous denoising result is used to guide the subsequent denoising process in a self-recurrent manner to gradually improve the sampling quality, and the U-shaped denoising network architecture based on the bidirectional feature cascaded fusion is used to realize accurate reception and efficient processing of the multi-stage guidance conditions, so that the target foreground segmentation model has excellent performance and cross-task generalization capability, and the dilemma of simultaneously ensuring accuracy and generalization in the foreground segmentation task is solved. BRIEF DESCRIPTION OF DRAWINGS

[0040] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief descriptions will be given below for the drawings needed to be used in the embodiments or prior art descriptions. Obviously, for those skilled in the art, other drawings can also be obtained from these drawings without any creative effort.

[0042] Figure 1 A flowchart is provided for the foreground segmentation method embodiment one of the present application;

[0043] Figure 2 A model architecture diagram is provided for the foreground segmentation method of the present application embodiment one;

[0044] Figure 3 A flowchart is provided for the foreground segmentation method embodiment two of the present application;

[0045] Figure 4 A flowchart is provided for the foreground segmentation method embodiment three of the present application;

[0046] Figure 5 A brief flowchart is provided for the foreground segmentation method of the present application embodiment three;

[0047] Figure 6 Fig. 1 is a schematic diagram of a module structure of a foreground segmentation device according to an embodiment of the present application;

[0048] Figure 7 Fig. 2 is a schematic diagram of a device structure of a hardware running environment involved in a foreground segmentation method according to an embodiment of the present application.

[0049] The object, features and advantages of the present application will be further illustrated in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are merely intended to explain the technical solutions of the present application, and are not intended to limit the present application.

[0051] In order to better understand the technical solutions of the present application, the specific embodiments will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0052] The main solution of the embodiment of the present application is: inputting the intermediate state mask of each sample training image into the alignment condition extraction network in the to-be-trained segmentation model for alignment to determine the multiple alignment condition representations of each sample training image, the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module and a multi-stage cascaded encoding module; inputting the multiple alignment condition representations of each sample training image into the bidirectional cascaded fusion denoising network in the to-be-trained segmentation model for denoising to obtain a target foreground segmentation model, the bidirectional cascaded fusion denoising network includes a downsampling unit and an upsampling unit, a cascaded flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit; inputting the to-be-segmented foreground image into the target foreground segmentation model to obtain the foreground segmentation mask of the to-be-segmented foreground image.

[0053] In order to obtain the characteristics in a single task domain, the traditional single-task foreground segmentation model often designs a complex and ingenious network structure and a customized training method, resulting in a lack of generalization ability of these models in the field of foreground segmentation. In addition, these models tend to overfit to the data distribution of a specific task. In contrast, the current general model for solving multiple tasks of foreground segmentation starts from the perspective of obtaining the commonality between multiple tasks, extracting low-level features such as frequency, multi-scale, texture and edge; however, such a way is easy to ignore the characteristics between different tasks and cannot accurately distinguish the emphasis between different tasks. Discriminative networks face the dilemma of being unable to simultaneously consider task performance and task generalization. In addition, some methods based on diffusion models rely on their image generation capabilities to solve the segmentation task from a generated perspective; however, due to the complex and large network architecture of the original diffusion model and the improper adaptation of the denoising mechanism to the segmentation task, the performance of such methods is far inferior to that of previous methods.

[0054] This application provides a solution, which improves the multi-task generalization ability through an alignment module based on the intermediate states in the mask diffusion process, and gradually improves the sampling quality by using a bootstrap strategy to autoregressively guide the subsequent denoising process based on the previous denoising results. The U-shaped denoising network architecture based on the bidirectional feature cascade fusion realizes the accurate reception and efficient processing of multi-level guidance conditions, so that the target foreground segmentation model has excellent performance and cross-task generalization ability, ensuring that the dilemma of simultaneously ensuring accuracy and generalization in the foreground segmentation task can be solved.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or a foreground segmentation device capable of performing the above functions. The following describes this embodiment and the following embodiments using the foreground segmentation device as the execution subject.

[0056] Based on this, the embodiment of the present application provides a foreground segmentation method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the foreground segmentation method of the present application.

[0057] In this embodiment, the foreground segmentation method includes steps S10 to S30:

[0058] In step S10, the intermediate state mask of each sample training image is input into the alignment condition extraction network in the segmentation model to be trained for alignment, and multiple alignment condition representations of each sample training image are determined. The alignment condition extraction network consists of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascade encoding module.

[0059] It should be noted that the segmentation model to be trained in this embodiment is an untrained foreground segmentation model. The segmentation model to be trained includes an alignment condition extraction network and a lightweight bidirectional cascade fusion denoising network. The specific architecture diagram is shown in FIG. Figure 2The alignment condition extraction network is composed of an image embedding module (i.e., an embedding layer), an intermediate state mask alignment module, and a multi-stage cascaded encoding module (i.e., an encoding layer). The lightweight bidirectional cascaded fusion denoising network includes an up-sampling unit and a down-sampling unit. The down-sampling unit includes three progressive fusion units, each of which is composed of a cascaded module, a down-sampling block, and a residual fusion module. The up-sampling unit includes three small units, each of which is composed of a residual module and an up-sampling block. In this embodiment, a progressive bidirectional feature flow, i.e., a cascaded flow and a fusion flow, is established between the three progressive fusion units. The cascaded flow fuses the condition representation of the output of the alignment condition extraction network in a top-down direction between each cascaded module. The fusion flow is composed of a down-sampling block and a residual fusion module, which fully fuse the input of the cascaded flow and the input of the down-sampling block through residual connection and the like.

[0060] It can be understood that in the training phase of the embodiment, a large number of sample training images and their corresponding true value masks are used as inputs for training through image and mask alignment and mask denoising restoration. In the bootstrap inference phase, a self-recurrent mode is adopted, and the temporary intermediate sampling result is used as the input for enhancing the next sampling result. In the training phase of the embodiment, three main processes are involved: a forward process, in which the true value mask is forwardly added with noise through random sampling time steps to obtain diffusion intermediate state masks with different noise levels; a condition extraction process, in which the original image and the intermediate state mask are aligned to extract multi-scale condition features; and a denoising process, in which a U-shaped structure is used for step-by-step denoising to improve the mask.

[0061] In a specific implementation, the intermediate state mask is obtained by forwardly adding noise to the true value mask of each sample training image at any time step, and one sample training image corresponds to one set of intermediate state masks. In this embodiment, a large number of sample training images are used to train the to-be-trained segmentation model.

[0062] It should be noted that for each sample training image, in order to extract clues obtained by accurately aligning the intermediate state mask and sufficient scene context in the sample training image, the sample training image and the intermediate state mask are provided as inputs to the alignment condition extraction network.

[0063] It can be understood that for each sample training image and its intermediate state mask at a time step: the embedding layer in the alignment condition extraction network divides and linearly projects the sample training image, and then the sample training image is normalized and then input into the intermediate state mask alignment module. The intermediate state mask alignment module aligns the mask of the foreground object in the intermediate state under different degrees of noise with the rich scene context of the image, providing aligned representations for the reverse denoising process. The above-mentioned representations have a comprehensive understanding of the foreground objects and the scene context under different diffusion states. When the forward intermediate diffusion state of the mask is reasonably aligned with the scene information of the image, it can provide a combination of specific clues and common clues of the image at different forward stages, and then these clues are used as conditional representations for the reverse denoising process to reconstruct the mask.

[0064] In a specific implementation, the multi-stage cascaded encoding module down-samples and encodes the features output by the intermediate state mask alignment module to extract features at different levels, and finally obtains the conditional representations at multiple scales corresponding to one intermediate state mask of the sample training image. In this embodiment, the conditional representations at multiple scales output based on one intermediate state mask of the sample training image are the multiple aligned conditional representations of the sample training image.

[0065] In a feasible implementation, before step S10, steps A11-A12 can be included.

[0066] Step A11, obtaining each sample training image and the ground truth mask of each sample training image.

[0067] Step A12, forward noise is added to the ground truth mask of each sample training image at a randomly selected time step to obtain the intermediate state mask of each sample training image.

[0068] It should be noted that during training, for each sample training image: the sample training image and its corresponding ground truth mask are input, and the ground truth mask is reconstructed through a forward process at a randomly selected different time step. The forward process outputs multiple intermediate state masks x t of the sample training image, which constitute an intermediate state set, and the intermediate state set includes a group of diffusion foreground object states under different noise levels.

[0069] Step S20, inputting the multiple aligned conditional representations of each sample training image into a bidirectional cascaded fusion denoising network in the to-be-trained segmentation model for denoising to obtain a target foreground segmentation model, wherein the bidirectional cascaded fusion denoising network includes a down-sampling unit and an up-sampling unit, and a cascaded flow and a fusion flow are established between multiple progressive fusion units in the down-sampling unit.

[0070] It should be noted that in the inference stage of the present embodiment, an improved bootstrap strategy is adopted to maximize the refinement capability in the autoregressive sampling process. The purpose of this strategy is to fully utilize the ability of the intermediate state learned in the training stage to align with the image, and further enhance this ability during inference through the bootstrap method. Specifically, the bootstrap strategy not only takes the sampling results of the previous step as the input of the subsequent step, but more importantly, it takes these results as temporary intermediate states to simulate the mask alignment in the training process. The purpose of this is to extract specific clues from the previous sampling that are helpful to the refinement process, and use these clues to enhance the results of the next sampling. This method allows the system to gradually accumulate and refine information in each iteration, thereby achieving more fine and accurate mask prediction in the inference stage. Since the diffusion process involves random noise at different time steps, the present embodiment must learn specific clues at different time points from the entire process from the noise-free clean mask to the Gaussian noise. However, traditional diffusion models usually use autoregressive methods in the inference stage, repeatedly refining the previous sampling results through the reverse process to recover the final result from Gaussian noise. This paradigm, while effective to some extent, often fails to fully utilize the alignment capabilities learned in the training stage.

[0071] It can be understood that in order to overcome the above limitations, the bootstrap strategy proposed in the present embodiment allows the model to constantly review and utilize previously learned knowledge during the inference process by taking the sampling results of the previous step as temporary intermediate states. This method not only improves the model's ability to capture details, but also enhances the model's adaptability to complex scenes, thereby achieving better performance in various foreground segmentation tasks. When implementing this bootstrap strategy, the model will carefully analyze the results of the previous sampling, identify key information, and use this information to guide the next denoising process. In this way, each iteration is built on the basis of the previous iteration, gradually improving the accuracy and quality of the results. In this way, the model can make more full use of the knowledge learned in the training stage, and achieve more efficient and accurate foreground segmentation.

[0072] In a specific implementation, in the model proposed in this embodiment, the alignment condition extraction network successfully captures multi-scale conditional features aligned with the image, which are further used in the lightweight bidirectional cascading fusion denoising network to realize the reverse denoising process. In order to comprehensively understand and effectively utilize these conditional representations, there is a lightweight bidirectional cascading fusion denoising network in the model of this embodiment. The core of the design of the lightweight bidirectional cascading fusion denoising network lies in the construction of a step-by-step bidirectional feature flow, which allows the system to gradually perform feature cascading and fusion in the reverse denoising process. Specifically, the three progressive fusion units work together in the network's downsampling unit, realizing the deep integration of multi-scale conditional representations through cascading modules, downsampling modules and residual fusion modules. This integration process not only enhances the system's ability to integrate local and global information, but also improves the system's capture and expression of complex foreground structures through bidirectional flow of features.

[0073] It should be noted that further, the lightweight bidirectional cascading fusion denoising network proposed in this embodiment not only comprehensively understands the feature representation, but also effectively eliminates the noise difference in a cost-effective manner. This is crucial for solving the problems encountered by traditional diffusion models in the training process. Complex diffusion models are often difficult to apply to practical problems due to their heavy computational burden and slow convergence speed. However, through innovative network structure and algorithm optimization, this embodiment significantly reduces the complexity of system training and speeds up the convergence, enabling the system to quickly and accurately perform foreground segmentation.

[0074] It can be understood that the multiple alignment condition representations of each sample training image are input into the bidirectional cascading fusion denoising network in the training segmentation model for denoising, which is the lightweight bidirectional cascading fusion denoising network. In the downsampling unit of the lightweight bidirectional cascading fusion denoising network, through the cascading flow and fusion flow, the model can integrate multi-scale information from bottom to top, while refining the feature representation from top to bottom. This bidirectional information flow ensures that sufficient information is obtained at each level from coarse to fine during the denoising process. The upsampling unit gradually restores the deep features obtained in the downsampling process to the resolution of the sample training image through the corresponding upsampling block and residual module, and finally outputs an accurate prediction mask. Through a large number of samples for training, a target foreground segmentation model that can directly output an accurate foreground segmentation prediction mask is obtained.

[0075] Step S30, inputting the foreground image to be segmented into the target foreground segmentation model to obtain the foreground segmentation mask of the foreground image to be segmented.

[0076] The embodiment determines a plurality of alignment condition representations of each sample training image by inputting the intermediate state mask of each sample training image to an alignment condition extraction network in a to-be-trained segmentation model for alignment, the alignment condition extraction network being composed of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascaded encoding module; the plurality of alignment condition representations of each sample training image are input to a bidirectional cascaded fusion denoising network in the to-be-trained segmentation model for denoising to obtain a target foreground segmentation model, the bidirectional cascaded fusion denoising network including a downsampling unit and an upsampling unit, and a cascaded flow and a fusion flow being established between a plurality of progressive fusion units in the downsampling unit; a to-be-segmented foreground image is input to the target foreground segmentation model to obtain a foreground segmentation mask of the to-be-segmented foreground image. In the foregoing manner, the alignment module based on the intermediate state in the mask diffusion process is used to improve the multi-task generalization capability, the bootstrap strategy based on the previous denoising result is used to guide the subsequent denoising process in a self-recurrent manner to gradually improve the sampling quality, and the U-shaped denoising network architecture based on the bidirectional feature cascaded fusion is used to accurately receive and efficiently process the multi-stage guidance conditions, so that the target foreground segmentation model has excellent performance and cross-task generalization capability, and the dilemma of simultaneously ensuring accuracy and generalization in the foreground segmentation task is solved.

[0077] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and the subsequent will not be described in detail. On this basis, please refer to Figure 3 , the step S10 of the foreground segmentation method further includes steps S11-S13:

[0078] In step S11, the intermediate state mask of each sample training image and each sample training image are input to an alignment condition extraction network in a to-be-trained segmentation model, and the image embedding module in the alignment condition extraction network is used to process each sample training image to obtain a plurality of first processed images.

[0079] It should be noted that, for each sample training image and its intermediate state mask at a time step: the sample training image and an intermediate state mask x t are input to the alignment condition extraction network, the image embedding module in the alignment condition extraction network divides and linearly projects the sample training image, and the sample training image is subjected to normalization processing to output a first processed image I p corresponding to the sample training image.

[0080] In step S12, the intermediate state mask alignment module in the alignment condition extraction network aligns each first processed image and the intermediate state mask of each sample training image to obtain to-be-encoded features of each sample training image.

[0081] It should be noted that, for each sample training image and its intermediate state mask at a time step: the intermediate state mask alignment module aligns the first processed image and the intermediate state mask based on the intermediate state mask alignment module, aligns the mask of the foreground object in the intermediate state under different degrees of noise with the rich scene context of the image, provides an aligned representation for the reverse denoising process, and outputs the aligned conditional representation P corresponding to the sample training image. out In this embodiment, the intermediate state mask alignment module outputs the aligned conditional representation P corresponding to the sample training image out , which is the to-be-encoded feature of the sample training image.

[0082] In a feasible implementation, step S12 can include steps B11-B13.

[0083] Step B11, by the intermediate state mask alignment module in the alignment condition extraction network, embedding operation is performed on the intermediate state mask of each sample training image according to the image dimension of each first processed image, to obtain the embedding feature of each sample training image.

[0084] It should be noted that, for each sample training image and its intermediate state mask at a time step: the intermediate state mask alignment module aligns the first processed image and the intermediate state mask based on the intermediate state mask alignment module, aligns the mask of the foreground object in the intermediate state under different degrees of noise with the rich scene context of the image, provides an aligned representation for the reverse denoising process, and outputs the aligned conditional representation P corresponding to the sample training image. t embedding operation, including convolution CONV and layer normalization LN, to obtain the embedding feature P of the sample training image. x , so as to be the same as the dimension of the first processed image I p . In this embodiment, P x = LN(CONV(x t )).

[0085] Step B12, flattening operation is performed on the embedding feature of each sample training image and the first processed image to obtain the first feature sequence and the second feature sequence of each sample training image.

[0086] It should be noted that, for each sample training image and its intermediate state mask at a time step: the intermediate state mask alignment module aligns the first processed image and the intermediate state mask based on the intermediate state mask alignment module, aligns the mask of the foreground object in the intermediate state under different degrees of noise with the rich scene context of the image, provides an aligned representation for the reverse denoising process, and outputs the aligned conditional representation P corresponding to the sample training image. x and the first processed image I p , to obtain the first feature sequence F(P x ) and the second feature sequence F(I p ) of the sample training image.

[0087] Step B13, fusion alignment is performed on the first feature sequence and the second feature sequence of each sample training image to obtain the to-be-encoded feature of each sample training image.

[0088] It should be noted that for each sample training image and its intermediate state mask at a time step: the intermediate state mask alignment module fuses and accurately aligns the first feature sequence and the second feature sequence, and performs residual connection to obtain the to-be-encoded feature P of the sample training image out .

[0089] In a feasible implementation, step B13 can include steps C11-C13.

[0090] Step C11, performing a fusion operation on the first feature sequence and the second feature sequence of each sample training image through a plurality of projection layers in the intermediate state mask alignment module to obtain a fused feature sequence of each sample training image.

[0091] Step C12, performing alignment on the fused feature sequence of each sample training image through a multi-layer perception and an activation layer in the intermediate state mask alignment module to obtain a fused and aligned feature of each sample training image.

[0092] Step C13, performing residual connection on the fused and aligned feature of each sample training image that has passed through the upper projection layer in the intermediate state mask alignment module and each first processing image to obtain the to-be-encoded feature of each sample training image.

[0093] It should be noted that for each sample training image and its intermediate state mask at a time step: through two projection layers L e in the intermediate state mask alignment module, the first feature sequence F(P x ) and the second feature sequence F(I p ) are linearly projected and fused to obtain a fused feature sequence P union = L e1 (F(P x ))+L e2 (F(I p )). Then, through a multi-layer perception MLP and an activation layer, the fused feature sequence is accurately aligned to obtain a fused and aligned feature P align =GELU(MLP(P union )). Finally, the fused and aligned feature L up (P up ) that has passed through the upper projection layer L align and the first processing image I p are residually connected to restore to the original dimension to obtain the aligned conditional representation P out . In this embodiment, P out =L up (P align )+I p .

[0094] Step S13, the multi-order cascaded encoding module in the alignment condition extraction network encodes the to-be-encoded features of each sample training image to obtain multiple alignment condition representations of each sample training image.

[0095] It should be noted that, for each sample training image and its intermediate state mask at a time step: the to-be-encoded features P out After that, the to-be-encoded features are passed to the multi-order cascaded encoding module, and the to-be-encoded features are down-sampled and encoded by the multi-order cascaded encoding module to extract features at different levels, and finally obtain the condition representations f1~f i In this embodiment, the condition representations at multiple scales output by the intermediate state mask of the sample training image are the multiple alignment condition representations of the sample training image.

[0096] In this embodiment, the intermediate state mask of each sample training image and each sample training image are input into the alignment condition extraction network in the to-be-trained segmentation model, the image embedding module in the alignment condition extraction network processes each sample training image to obtain multiple first processed images; the intermediate state mask alignment module in the alignment condition extraction network aligns each first processed image and the intermediate state mask of each sample training image to obtain the to-be-encoded features of each sample training image; and the multi-order cascaded encoding module in the alignment condition extraction network encodes the to-be-encoded features of each sample training image to obtain multiple alignment condition representations of each sample training image. Through the above manner, the multi-task generalization capability is improved.

[0097] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above introduction, and will not be repeated hereinafter. On this basis, please refer to Figure 4 , the step S20 in the foreground segmentation method further includes steps S21~S23:

[0098] Step S21, input the multiple alignment condition representations of each sample training image into the bidirectional cascaded fusion denoising network in the to-be-trained segmentation model, and fuse the multiple alignment condition representations of each sample training image in a first preset direction through the cascaded flow in the bidirectional cascaded fusion denoising network to output the cascaded features of each sample training image.

[0099] It should be noted that the first preset direction refers to a top-down direction, and for each sample training image and its intermediate state mask at a time step: the cascaded flow outputs the alignment condition representations f i fuse the multi-scale condition representations Fi+1 , specifically: the representation F i+1 is obtained by two convolutional layers, a ReLU activation layer CR 2 and a convolutional layer with batch normalization CB, and finally f i is concatenated to obtain F i , F i = concat(CB(CR 2 (F i+1 ), f i )), i = 1, 2, 3. In this embodiment, the representation F i output by the cascade flow is the cascade feature of the sample training image.

[0100] In step S22, the feature map of each sample training image is output by the fusion flow in the bidirectional cascade fusion denoising network according to the cascade feature of each sample training image and the intermediate state mask of each sample training image.

[0101] It should be noted that for each sample training image and its intermediate state mask at a time step: the cascade feature F i output by the cascade flow and the intermediate state mask x t input to the downsampling block are fully fused through a residual connection or the like to obtain the feature map SC i of the sample training image.

[0102] In a feasible implementation, step S22 can include steps D11-D14:

[0103] In step D11, the intermediate state of each sample training image is obtained by downsampling the intermediate state mask of each sample training image by the fusion flow in the bidirectional cascade fusion denoising network.

[0104] In step D12, the intermediate state of each sample training image and the cascade feature of each sample training image are fused to obtain the embedding representation of each sample training image.

[0105] In step D13, the embedding representation of each sample training image and the time step are gradually fused to obtain the cell layer output corresponding to each sample training image.

[0106] In step D14, the feature map of each sample training image is obtained according to the cell layer output corresponding to each sample training image and the embedding representation of each sample training image.

[0107] It should be noted that for each sample training image and its intermediate state mask at a time step: the intermediate state mask x t is downsampled by the downsampling block of the fusion flow to obtain the intermediate state D iIn the embodiment, the down-sampling block is to down-sample the input to the same scale as the output of the cascaded module. The residual fusion module processes the cascaded feature F i , the down-sampled intermediate state D i and the time step. By convolution, group normalization and SILU activation layer, the cascaded feature F i and the down-sampled intermediate state D i are fused to obtain the embedding representation E i In the embodiment, E i = CGA(F i ) + D i .

[0108] It can be understood that the embedding representation E i is fused step by step at the time step t, allowing the lightweight bidirectional cascaded fusion module to perceive the time step of the diffusion process, and taking it as the output of the layer unit by time embedding and scaling STE, to obtain the unit layer output D out = S TE(t) (CGA(E i )). Finally, based on the unit layer output D out = S TE(t) (CGA(E i )), two residual connections are used to further enhance the input SC i of the skip connection in the same layer up-sampling module. In the embodiment, SC i = CGA(D out + CGA(F i )) + E i , and the feature map of the sample training image refers to the output SC i of the fusion flow.

[0109] In step S23, the feature map of each sample training image is input to the up-sampling unit in the cascaded fusion denoising network for feature recovery, to obtain the target foreground segmentation network.

[0110] It should be noted that the output SC i of the fusion flow is input to the down-sampling block and the up-sampling unit, and the up-sampling unit performs feature recovery based on the feature map SC i , and the training process is repeated to finally obtain the target foreground segmentation network.

[0111] The embodiment inputs multiple alignment condition representations of each sample training image into a bidirectional cascaded fusion denoising network in the segmentation model to be trained, fuses the multiple alignment condition representations of each sample training image in a cascaded flow in the bidirectional cascaded fusion denoising network according to a first preset direction, and outputs cascaded features of each sample training image; the bidirectional cascaded fusion denoising network outputs feature mapping of each sample training image according to the cascaded features of each sample training image and an intermediate state mask of each sample training image according to a fusion flow in the bidirectional cascaded fusion denoising network; and the feature mapping of each sample training image is input into an up-sampling unit in the cascaded fusion denoising network for feature recovery to obtain a target foreground segmentation network. In the foregoing manner, the bootstrap strategy of guiding the subsequent denoising process in a self-recurrence manner based on the previous denoising result is used to gradually improve the sampling quality, and the U-shaped denoising network architecture based on bidirectional feature cascaded fusion is used to accurately receive and efficiently process the multi-level guidance conditions, thereby improving the performance of the model.

[0112] For example, to facilitate understanding of the implementation process of the foreground segmentation method obtained after the foregoing embodiment one, refer to Figure 5 , Figure 5 A brief flowchart of a foreground segmentation method is provided, and specifically,

[0113] The alignment condition extraction network and the lightweight bidirectional cascaded fusion denoising network of the foreground segmentation model in the embodiment include an image embedding module (i.e., an embedding layer), an intermediate state mask alignment module, and a multi-stage cascaded encoding module (i.e., an encoding layer). The lightweight bidirectional cascaded fusion denoising network includes an up-sampling unit and a down-sampling unit, the down-sampling unit includes three progressive fusion units, each progressive fusion unit includes a cascaded module, a down-sampling block, and a residual fusion module; and the up-sampling unit includes three small units, each small unit includes a residual module and an up-sampling block. In the embodiment, a progressive bidirectional feature flow, i.e., a cascaded flow and a fusion flow, is established among the three progressive fusion units: the cascaded flow fuses the condition representations output by the alignment condition extraction network in a top-down direction between each cascaded module; and the fusion flow is composed of a down-sampling block and a residual fusion module, and fully fuses the input of the cascaded flow and the input of the down-sampling block through a residual connection.

[0114] It can be understood that, in the training stage of the embodiment, a large number of sample training images and corresponding true value masks are used as inputs, and the training is performed through alignment of the images and the masks and denoising and restoration of the masks; in the bootstrap inference stage, a self-recurrence mode is adopted, and temporary intermediate sampling results are used as inputs to enhance the next sampling results. In the training stage of the embodiment, three main processes are involved: a forward process, in which the true value mask is forwardly added with noise by randomly sampling time steps to obtain diffusion intermediate state masks with different noise levels; a condition extraction process, in which the original image and the intermediate state mask are aligned to extract multi-scale condition features; and a denoising process, in which a U-shaped structure is used for step-by-step denoising to improve the mask.

[0115] It should be noted that the above examples are only used to understand the present application and do not limit the foreground segmentation method of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0116] The present application also provides a foreground segmentation device, which refers to Figure 6 , and the foreground segmentation device comprises:

[0117] The alignment module 10 is configured to input the intermediate state masks of the sample training images into an alignment condition extraction network in the to-be-trained segmentation model for alignment, to determine a plurality of aligned condition features of the sample training images, and the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module and a multi-stage cascaded encoding module.

[0118] The denoising module 20 is configured to input the plurality of aligned condition features of the sample training images into a bidirectional cascaded fusion denoising network in the to-be-trained segmentation model for denoising, to obtain a target foreground segmentation model, and the bidirectional cascaded fusion denoising network comprises a downsampling unit and an upsampling unit, and a cascaded flow and a fusion flow are established between a plurality of progressive fusion units in the downsampling unit.

[0119] The segmentation module 30 is configured to input a to-be-segmented foreground image into the target foreground segmentation model, to obtain a foreground segmentation mask of the to-be-segmented foreground image.

[0120] Optionally, the alignment module 10 is further configured to:

[0121] The intermediate state mask of each sample training image and each sample training image are input into an alignment condition extraction network in the segmentation model to be trained, the image embedding module in the alignment condition extraction network is used for processing each sample training image, and a plurality of first processing images are obtained; the intermediate state mask alignment module in the alignment condition extraction network is used for aligning each first processing image and the intermediate state mask of each sample training image, and obtaining the to-be-encoded feature of each sample training image; and the multi-order cascaded encoding module in the alignment condition extraction network is used for encoding the to-be-encoded feature of each sample training image, and obtaining a plurality of alignment condition representations of each sample training image.

[0122] Optionally, the alignment module 10 is further used for:

[0123] The intermediate state mask alignment module in the alignment condition extraction network is used for embedding each intermediate state mask of each sample training image according to the image dimension of each first processing image, obtaining the embedding feature of each sample training image; the embedding feature of each sample training image and each first processing image are flattened, obtaining the first feature sequence and the second feature sequence of each sample training image; and the first feature sequence and the second feature sequence of each sample training image are fused and aligned, obtaining the to-be-encoded feature of each sample training image.

[0124] Optionally, the alignment module 10 is further used for:

[0125] The plurality of projection layers in the intermediate state mask alignment module are used for performing fusion operation on the first feature sequence and the second feature sequence of each sample training image, obtaining the fusion feature sequence of each sample training image; the multi-layer perception and the activation layer in the intermediate state mask alignment module are used for aligning the fusion feature sequence of each sample training image, obtaining the fusion alignment feature of each sample training image; and the fusion alignment feature of each sample training image passing through the upper projection layer in the intermediate state mask alignment module and each first processing image are residual connected, obtaining the to-be-encoded feature of each sample training image.

[0126] Optionally, the denoising module 20 is further used for:

[0127] The multiple alignment condition representations of each sample training image are input into a bidirectional cascaded fusion denoising network in the segmentation model to be trained, the multiple alignment condition representations of each sample training image are fused in a cascaded flow in the bidirectional cascaded fusion denoising network, and the cascaded features of each sample training image are output in a first preset direction; the feature mapping of each sample training image is output according to the cascaded features of each sample training image and the intermediate state mask of each sample training image through a fusion flow in the bidirectional cascaded fusion denoising network; and the feature mapping of each sample training image is input into an upsampling unit in the cascaded fusion denoising network for feature recovery to obtain the target foreground segmentation network.

[0128] Optionally, the denoising module 20 is further configured to:

[0129] The intermediate state of each sample training image is obtained by downsampling the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascaded fusion denoising network; the embedding representation of each sample training image is obtained by fusing the intermediate state of each sample training image and the cascaded features of each sample training image; and the cell layer output corresponding to each sample training image is obtained by gradually fusing the embedding representation of each sample training image and the time step; and the feature mapping of each sample training image is obtained according to the cell layer output corresponding to each sample training image and the embedding representation of each sample training image.

[0130] Optionally, the alignment module 10 is further configured to:

[0131] The true value mask of each sample training image is obtained; and the intermediate state mask of each sample training image is obtained by forwardly adding noise to the true value mask of each sample training image at a randomly selected time step.

[0132] The foreground segmentation device provided in the present application adopts the foreground segmentation method in the above embodiments, and can solve the technical problem that the foreground segmentation model in the prior art cannot guarantee accuracy and generalization at the same time when performing a foreground segmentation task. Compared with the prior art, the foreground segmentation device provided in the present application has the same beneficial effects as the foreground segmentation method provided in the above embodiments, and other technical features in the foreground segmentation device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0133] The present application provides a foreground segmentation device, which comprises at least one processor and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the foreground segmentation method in the above embodiment one.

[0134] The following refers to Figure 7, which shows a schematic structural diagram of a foreground segmentation device suitable for implementing an embodiment of the present application. The foreground segmentation device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The foreground segmentation device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0135] like Figure 7 As shown, the foreground segmentation device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the foreground segmentation device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. The communication device 1009 can allow the foreground segmentation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a foreground segmentation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0136] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0137] The foreground segmentation device provided by the present application adopts the foreground segmentation method in the above-mentioned embodiments, and can solve the technical problem that the foreground segmentation model in the prior art cannot guarantee accuracy and generalization at the same time when performing a foreground segmentation task. Compared with the prior art, the foreground segmentation device provided by the present application has the same beneficial effects as the foreground segmentation method provided by the above-mentioned embodiments, and other technical features in the foreground segmentation device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0138] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0139] The above is merely specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0140] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the foreground segmentation method in the above-mentioned embodiments.

[0141] The computer readable storage medium provided in the application may be, for example, a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electrical wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0142] The computer readable storage medium described above may be contained in the foreground segmentation device, or may exist separately without being assembled into the foreground segmentation device.

[0143] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the foreground segmentation device, the foreground segmentation device: inputs the intermediate state mask of each sample training image into the alignment condition extraction network in the to-be-trained segmentation model for alignment, determines a plurality of alignment condition representations of each sample training image, the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascaded encoding module; inputs the plurality of alignment condition representations of each sample training image into the bidirectional cascaded fusion denoising network in the to-be-trained segmentation model for denoising to obtain a target foreground segmentation model, the bidirectional cascaded fusion denoising network includes a downsampling unit and an upsampling unit, a plurality of progressive fusion units in the downsampling unit establish a cascaded flow and a fusion flow; inputs the to-be-segmented foreground image into the target foreground segmentation model to obtain a foreground segmentation mask of the to-be-segmented foreground image.

[0144] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0145] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0146] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0147] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the foreground segmentation method described above, and can solve the technical problem that the foreground segmentation model in the prior art cannot guarantee accuracy and generalization when performing a foreground segmentation task. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the foreground segmentation method provided by the above embodiments, and will not be described here.

[0148] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the foreground segmentation method as described above.

[0149] The computer program product provided by the application can solve the technical problem that the foreground segmentation model in the prior art cannot guarantee both accuracy and generalization when performing a foreground segmentation task. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the foreground segmentation method provided by the above-described embodiments, and are not described herein.

[0150] The above only describes some embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or the like made by using the content of the specification and drawings within the technical concept of the application is included in the patent protection scope of the application.

Claims

1. A foreground segmentation method, characterized in that: The method comprises: Inputting the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determining multiple alignment condition representations of each sample training image, the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascade encoding module; Inputting multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, thereby obtaining a target foreground segmentation model, wherein the bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, and a cascade flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit; Inputting the foreground image to be segmented into the target foreground segmentation model to obtain a foreground segmentation mask of the foreground image to be segmented; The step of inputting the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment and determining multiple alignment condition representations of each sample training image includes: Inputting the intermediate state mask of each sample training image and each sample training image into the alignment condition extraction network in the segmentation model to be trained, and processing each sample training image through the image embedding module in the alignment condition extraction network to obtain a plurality of first processed images; Aligning the intermediate state masks of each first processed image and each sample training image through the intermediate state mask alignment module in the alignment condition extraction network to obtain the to-be-encoded features of each sample training image; The features to be encoded of each sample training image are encoded by the multi-stage cascade encoding module in the alignment condition extraction network to obtain multiple alignment condition representations of each sample training image.

2. The method according to claim 1, wherein The step of aligning the intermediate state mask alignment module in the alignment condition extraction network according to the intermediate state masks of each first processed image and each sample training image to obtain the to-be-encoded features of each sample training image includes: The intermediate mask alignment module in the alignment condition extraction network performs an embedding operation on the intermediate mask of each sample training image according to the image dimension of each first processed image to obtain an embedding feature of each sample training image; Performing a flattening operation on the embedded features of each sample training image and each first processed image to obtain a first feature sequence and a second feature sequence of each sample training image; The first feature sequence and the second feature sequence of each sample training image are fused and aligned to obtain the features to be encoded of each sample training image.

3. The method according to claim 2, wherein The step of fusing and aligning the first feature sequence and the second feature sequence of each sample training image to obtain the features to be encoded of each sample training image includes: Performing a fusion operation on the first feature sequence and the second feature sequence of each sample training image through multiple projection layers in the intermediate state mask alignment module to obtain a fused feature sequence of each sample training image; Aligning the fused feature sequences of each sample training image through the multi-layer perceptron and activation layer in the intermediate state mask alignment module to obtain the fused alignment features of each sample training image; The fused alignment features of each sample training image that has passed through the upper projection layer in the intermediate state mask alignment module and each first processed image are residually connected to obtain the features to be encoded of each sample training image.

4. The method according to claim 1, wherein The step of inputting the multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising to obtain the target foreground segmentation model includes: Inputting multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained, fusing the multiple alignment condition representations of each sample training image according to a first preset direction through the cascade flow in the bidirectional cascade fusion denoising network, and outputting cascade features of each sample training image; Outputting a feature map of each sample training image according to the cascade features of each sample training image and the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network; The feature map of each sample training image is input into the upsampling unit in the cascade fusion denoising network for feature recovery to obtain the target foreground segmentation network.

5. The method according to claim 4, wherein The step of outputting a feature map of each sample training image according to the cascade features of each sample training image and the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network includes: Downsampling the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network to obtain the intermediate state of each sample training image; The intermediate state of each sample training image and the cascade features of each sample training image are fused to obtain the embedded representation of each sample training image; The embedded representation and time step of each sample training image are gradually fused to obtain the unit layer output corresponding to each sample training image; According to the unit layer output corresponding to each sample training image and the embedded representation of each sample training image, the feature map of each sample training image is obtained.

6. The method according to any one of claims 1 to 5, characterized in that Before the step of inputting the multiple intermediate state masks of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment and determining the multiple alignment condition representations of each sample training image, the method further includes: Obtain each sample training image and the true value mask of each sample training image; The true value mask of each sample training image is forward-noised at a randomly selected time step to obtain the intermediate state mask of each sample training image.

7. A foreground segmentation device, characterized in that: The foreground segmentation device comprises: An alignment module is used to input the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determine multiple alignment condition representations of each sample training image. The alignment condition extraction network consists of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascade encoding module; a denoising module, configured to input multiple alignment condition representations of each sample training image into a bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, thereby obtaining a target foreground segmentation model, wherein the bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, and a cascade flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit; A segmentation module, configured to input the foreground image to be segmented into the target foreground segmentation model to obtain a foreground segmentation mask of the foreground image to be segmented; The alignment module is also used to input the intermediate state mask of each sample training image and each sample training image into the alignment condition extraction network in the segmentation model to be trained, and process each sample training image through the image embedding module in the alignment condition extraction network to obtain multiple first processed images; align each first processed image and the intermediate state mask of each sample training image through the intermediate state mask alignment module in the alignment condition extraction network to obtain the to-be-encoded features of each sample training image; encode the to-be-encoded features of each sample training image through the multi-stage cascade encoding module in the alignment condition extraction network to obtain multiple alignment condition representations of each sample training image.

8. A foreground segmentation device, characterized in that: The foreground segmentation device includes: a memory, a processor, and a foreground segmentation program stored in the memory and executable on the processor, wherein the foreground segmentation program is configured to implement the foreground segmentation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium stores a foreground segmentation program, which, when executed by a processor, implements the foreground segmentation method according to any one of claims 1 to 6.

Images