Foreground segmentation method and device, equipment and storage medium

Abstract

The invention discloses a foreground segmentation method, device and equipment and a storage medium, and relates to the technical field of images, and the method comprises the steps: inputting an intermediate state mask of each sample training image into an alignment condition extraction network in a to-be-trained segmentation model, determining a plurality of alignment condition representations, the network is composed of an image embedding module, an intermediate state mask alignment module and a multi-order cascade coding module. A plurality of alignment condition representations of the sample training images are input into a bidirectional cascade fusion denoising network in the to-be-trained segmentation model for denoising, a target foreground segmentation model is obtained, the denoising network comprises a down-sampling unit and an up-sampling unit, and the down-sampling unit comprises cascade streams and fusion streams; and inputting a foreground image to be segmented into the target foreground segmentation model to obtain a segmentation mask. Through the above mode, the target foreground segmentation model has excellent performance and cross-task generalization ability, and accuracy and generalization can be ensured in the foreground segmentation task at the same time.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to a foreground segmentation method, device, equipment and storage medium. Background Art

[0002] Traditional single-task foreground segmentation models often design complex and sophisticated network structures and customized training methods in order to obtain the characteristics within a single task domain, resulting in these models lacking generalization capabilities in the field of foreground segmentation. In addition, these models tend to overfit to the data distribution of specific tasks. In contrast, the current general model used to solve multiple tasks of foreground segmentation extracts low-level features such as frequency, multi-scale, texture, and edge from the perspective of obtaining the commonalities between multiple tasks; however, this approach easily ignores the characteristics between different tasks and cannot accurately distinguish the focus between different tasks. Discriminative networks face the dilemma of not being able to take into account both task performance and task generalization at the same time. In addition, some methods based on diffusion models use their ability to generate images to solve the task of segmentation from a generated perspective; however, due to the complex and large network architecture of the original diffusion model and the improper adaptation of the denoising mechanism in the segmentation task, the performance of such methods is far inferior to previous methods. Summary of the invention

[0003] The main purpose of the present application is to provide a foreground segmentation method, apparatus, device and storage medium, aiming to solve the technical problem that the foreground segmentation model in the prior art cannot guarantee both accuracy and generalization when performing foreground segmentation tasks.

[0004] To achieve the above object, the present application proposes a foreground segmentation method, which includes:

[0005] Inputting the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determining multiple alignment condition representations of each sample training image, wherein the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascade encoding module;

[0006] Inputting multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, to obtain a target foreground segmentation model, wherein the bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, and a cascade flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit;

[0007] The foreground image to be segmented is input into the target foreground segmentation model to obtain a foreground segmentation mask of the foreground image to be segmented.

[0008] In one embodiment, the step of inputting the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determining multiple alignment condition representations of each sample training image includes:

[0009] Inputting the intermediate state mask of each sample training image and each sample training image into the alignment condition extraction network in the segmentation model to be trained, and processing each sample training image through the image embedding module in the alignment condition extraction network to obtain a plurality of first processed images;

[0010] Aligning the intermediate state masks of each first processed image and each sample training image through the intermediate state mask alignment module in the alignment condition extraction network to obtain the to-be-encoded features of each sample training image;

[0011] The features to be encoded of each sample training image are encoded by the multi-stage cascade encoding module in the alignment condition extraction network to obtain multiple alignment condition representations of each sample training image.

[0012] In one embodiment, the step of aligning the intermediate mask alignment module in the alignment condition extraction network according to the intermediate masks of each first processed image and each sample training image to obtain the features to be encoded of each sample training image includes:

[0013] The intermediate state mask alignment module in the alignment condition extraction network performs an embedding operation on the intermediate state mask of each sample training image according to the image dimension of each first processed image, so as to obtain an embedding feature of each sample training image;

[0014] Flattening the embedded features of each sample training image and each first processed image to obtain a first feature sequence and a second feature sequence of each sample training image;

[0015] The first feature sequence and the second feature sequence of each sample training image are fused and aligned to obtain the to-be-encoded features of each sample training image.

[0016] In one embodiment, the step of fusing and aligning the first feature sequence and the second feature sequence of each sample training image to obtain the feature to be encoded of each sample training image includes:

[0017] Performing a fusion operation on the first feature sequence and the second feature sequence of each sample training image through the multiple projection layers in the intermediate state mask alignment module to obtain a fused feature sequence of each sample training image;

[0018] Aligning the fused feature sequences of each sample training image through the multi-layer perceptron and the activation layer in the intermediate state mask alignment module to obtain the fused alignment features of each sample training image;

[0019] The fused alignment features of each sample training image that has passed through the upper projection layer in the intermediate state mask alignment module and each first processed image are residually connected to obtain the features to be encoded of each sample training image.

[0020] In one embodiment, the step of inputting multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising to obtain the target foreground segmentation model includes:

[0021] Inputting multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained, fusing the multiple alignment condition representations of each sample training image according to a first preset direction through the cascade flow in the bidirectional cascade fusion denoising network, and outputting cascade features of each sample training image;

[0022] Outputting a feature map of each sample training image according to the cascade features of each sample training image and the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network;

[0023] The feature map of each sample training image is input into the upsampling unit in the cascade fusion denoising network for feature recovery to obtain a target foreground segmentation network.

[0024] In one embodiment, the step of outputting a feature map of each sample training image according to the cascade features of each sample training image and the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network includes:

[0025] Down-sampling the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network to obtain the intermediate state of each sample training image;

[0026] The intermediate state of each sample training image and the cascade features of each sample training image are fused to obtain an embedded representation of each sample training image;

[0027] The embedded representation and time step of each sample training image are gradually fused to obtain the unit layer output corresponding to each sample training image;

[0028] According to the unit layer output corresponding to each sample training image and the embedded representation of each sample training image, the feature map of each sample training image is obtained.

[0029] In one embodiment, before the step of inputting the multiple intermediate state masks of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment and determining the multiple alignment condition representations of each sample training image, the step further includes:

[0030] Obtain each sample training image and the true value mask of each sample training image;

[0031] The true value mask of each sample training image is forward-noised at a randomly selected time step to obtain the intermediate state mask of each sample training image.

[0032] In addition, to achieve the above purpose, the present application also proposes a foreground segmentation device, the foreground segmentation device comprising:

[0033] An alignment module is used to input the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determine multiple alignment condition representations of each sample training image, wherein the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascade encoding module;

[0034] A denoising module is used to input multiple alignment condition representations of each sample training image into a bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, so as to obtain a target foreground segmentation model, wherein the bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, and a cascade flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit;

[0035] The segmentation module is used to input the foreground image to be segmented into the target foreground segmentation model to obtain a foreground segmentation mask of the foreground image to be segmented.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a foreground segmentation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the foreground segmentation method as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the foreground segmentation method described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the foreground segmentation method described above are implemented.

[0039] The present application determines multiple alignment condition representations of each sample training image by inputting the intermediate state mask of each sample training image into an alignment condition extraction network in a segmentation model to be trained for alignment, wherein the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascade encoding module; multiple alignment condition representations of each sample training image are input into a bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, thereby obtaining a target foreground segmentation model, wherein the bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, wherein a cascade flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit; a foreground image to be segmented is input into the target foreground segmentation model, thereby obtaining a foreground segmentation mask of the foreground image to be segmented. Through the above methods, an alignment module based on the intermediate states in the mask diffusion process is used to improve the generalization ability of multiple tasks. The bootstrap strategy based on the previous denoising result autoregressively guides the subsequent denoising process to gradually improve the sampling quality. The U-shaped denoising network architecture based on the bidirectional feature cascade fusion realizes accurate reception and efficient processing of multi-level guidance conditions, which makes the target foreground segmentation model have excellent performance and cross-task generalization ability, ensuring the dilemma of both accuracy and generalization in the foreground segmentation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the foreground segmentation method of the present application;

[0043] Figure 2 A schematic diagram of the model architecture of the foreground segmentation method provided in Example 1 of the present application;

[0044] Figure 3 A schematic diagram of a flow chart provided for the second embodiment of the foreground segmentation method of the present application;

[0045] Figure 4 A flowchart of the third embodiment of the foreground segmentation method of the present application is provided;

[0046] Figure 5 A schematic diagram of a brief flow of a foreground segmentation method provided in Embodiment 3 of the present application;

[0047] Figure 6 This is a schematic diagram of the module structure of the foreground segmentation device according to an embodiment of the present application;

[0048] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the foreground segmentation method in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of the embodiment of the present application is: inputting the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determining multiple alignment condition representations of each sample training image, wherein the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module and a multi-stage cascade encoding module; inputting multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, and obtaining a target foreground segmentation model, wherein the bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, and cascade flows and fusion flows are established between multiple progressive fusion units in the downsampling unit; inputting the foreground image to be segmented into the target foreground segmentation model, and obtaining a foreground segmentation mask of the foreground image to be segmented.

[0053] Traditional single-task foreground segmentation models often design complex and sophisticated network structures and customized training methods in order to obtain the characteristics within a single task domain, resulting in these models lacking generalization capabilities in the field of foreground segmentation. In addition, these models tend to overfit to the data distribution of specific tasks. In contrast, the current general model used to solve multiple tasks of foreground segmentation extracts low-level features such as frequency, multi-scale, texture, and edge from the perspective of obtaining the commonalities between multiple tasks; however, this approach easily ignores the characteristics between different tasks and cannot accurately distinguish the focus between different tasks. Discriminative networks face the dilemma of not being able to take into account both task performance and task generalization at the same time. In addition, some methods based on diffusion models use their ability to generate images to solve the task of segmentation from a generated perspective; however, due to the complex and large network architecture of the original diffusion model and the improper adaptation of the denoising mechanism in the segmentation task, the performance of such methods is far inferior to previous methods.

[0054] The present application provides a solution, which improves the generalization ability of multiple tasks through an alignment module based on the intermediate state in the mask diffusion process, and guides the bootstrap strategy of the subsequent denoising process in an autoregressive manner based on the previous denoising result to gradually improve the sampling quality. The U-shaped denoising network architecture based on the bidirectional feature cascade fusion realizes the accurate reception and efficient processing of multi-level guidance conditions, so that the target foreground segmentation model has excellent performance and cross-task generalization ability, ensuring the dilemma of ensuring both accuracy and generalization in the foreground segmentation task.

[0055] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a foreground segmentation device capable of realizing the above functions, etc. The following takes the foreground segmentation device as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides a foreground segmentation method, referring to Figure 1 , Figure 1 FIG. 1 is a flow chart of the first embodiment of the foreground segmentation method of the present application.

[0057] In this embodiment, the foreground segmentation method includes steps S10 to S30:

[0058] Step S10, inputting the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determining multiple alignment condition representations of each sample training image, wherein the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module and a multi-stage cascade encoding module.

[0059] It should be noted that the segmentation model to be trained in this embodiment is an untrained foreground segmentation model. The segmentation model to be trained includes an alignment condition extraction network and a lightweight bidirectional cascade fusion denoising network. The specific architecture diagram is shown in FIG. Figure 2As shown. The alignment condition extraction network consists of an image embedding module (i.e., an embedding layer), an intermediate state mask alignment module, and a multi-stage cascade coding module (i.e., a coding layer). The lightweight bidirectional cascade fusion denoising network includes an upsampling unit and a downsampling unit. The downsampling unit includes three progressive fusion units, each of which consists of a cascade module, a downsampling block, and a residual fusion module; the upsampling unit includes three small units, each of which consists of a residual module and an upsampling block. In this embodiment, a progressive bidirectional feature flow is established between the three progressive fusion units: a cascade flow and a fusion flow. The cascade flow fuses the conditional representation of the output corresponding to the alignment condition extraction network with the conditional representation of the previous level in a bottom-up direction between each cascade module; the fusion flow consists of a downsampling block and a residual fusion module, and the corresponding input of the cascade flow and the input of the downsampling block are fully fused through residual connections and the like.

[0060] It can be understood that in the training phase of this embodiment, a large number of sample training images and their corresponding true value masks are used as input, and training is performed by aligning the image with the mask and denoising and restoring the mask; in the bootstrap inference phase, an autoregressive mode is adopted, and the temporary intermediate sampling results are used as input to enhance the next step sampling results. In this embodiment, there are three main processes involved in the training phase: the forward process, forward denoising the true value mask through random sampling time steps to obtain diffuse intermediate state masks with different noise levels; the conditional extraction process, aligning the original image and the intermediate state mask, and extracting multi-scale conditional features; the denoising process, using the U-shaped structure to gradually denoise and improve the mask.

[0061] In a specific implementation, the intermediate state mask is obtained by forward denoising the true value mask of each sample training image at any time step, and one sample training image corresponds to one intermediate state mask set. In this embodiment, a large number of sample training images are used to train the segmentation model to be trained.

[0062] It should be noted that, for each sample training image, in order to extract clues obtained by accurately aligning the intermediate state mask with sufficient scene context in the sample training image, the sample training image and the intermediate state mask are provided as input to the alignment condition extraction network.

[0063] It can be understood that for each sample training image and its intermediate state mask at one time step: the embedding layer in the alignment condition extraction network divides and linearly projects the sample training image, and after normalization, enters the intermediate state mask alignment module, which aligns the mask of the foreground object in the intermediate state under different degrees of noise with the rich scene context of the image, providing an aligned representation for the reverse denoising process. The above representation has a comprehensive understanding of the foreground objects and scene context in different diffusion states. When the forward intermediate diffusion state of the mask is reasonably aligned with the scene information of the image, it is able to provide a combination of image-specific cues and common cues in different forward stages, and then these cues are used as conditional representations of the reverse denoising process to reconstruct the mask.

[0064] In a specific implementation, the multi-stage cascade encoding module downsamples and encodes the features output by the intermediate mask alignment module to extract features at different levels, and finally obtains a multi-scale conditional representation corresponding to an intermediate mask of the sample training image. In this embodiment, the multi-scale conditional representation output based on an intermediate mask of the sample training image is a plurality of aligned conditional representations of the sample training image.

[0065] In a feasible implementation manner, before step S10, steps A11 to A12 may be included:

[0066] Step A11, obtaining each sample training image and the true value mask of each sample training image.

[0067] Step A12, forward denoising is performed on the true value mask of each sample training image at a randomly selected time step to obtain an intermediate state mask of each sample training image.

[0068] It should be noted that during training, for each sample training image: the sample training image and its corresponding true value mask are taken as input, and the true value mask is reconstructed through the forward process at different randomly selected time steps. The forward process outputs multiple intermediate state masks x of the sample training image. t , forming an intermediate state set, which includes a set of diffuse foreground object states under different noise levels.

[0069] Step S20, input multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, and obtain a target foreground segmentation model, the bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, and a cascade flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit.

[0070] It should be noted that in the reasoning stage of this embodiment, an improved bootstrap strategy is adopted to maximize the refinement ability in the autoregressive sampling process. The purpose of this strategy is to make full use of the ability of the intermediate state learned in the training stage to align with the image, and further enhance this ability during the reasoning by bootstrapping. Specifically, the bootstrap strategy not only uses the sampling results of the previous step as the input of the subsequent step, but more importantly, it uses these results as temporary intermediate states to simulate the mask alignment in the training process. The purpose of this is to extract specific clues from the previous sampling that are helpful for the refinement process, and use these clues to enhance the results of the next sampling. This method allows the system to gradually accumulate and refine information in each iteration, thereby achieving more refined and accurate mask prediction in the reasoning stage. Since the diffusion process involves random noise at different time steps, this embodiment must learn specific clues at different time points in the whole process from the noise-free clean mask to the Gaussian noise. However, the traditional diffusion model usually adopts an autoregressive method in the reasoning stage, and refines the previous sampling results through the reverse process many times to recover the final result from the Gaussian noise. While this paradigm works to some extent, it often does not fully exploit the alignment capabilities learned during the training phase.

[0071] It is understandable that in order to overcome the above limitations, the bootstrap strategy proposed in this embodiment uses the sampling result of the previous step as a temporary intermediate state, so that the model can continuously review and utilize the previously learned knowledge during the reasoning process. This method not only improves the model's ability to capture details, but also enhances the model's adaptability to complex scenes, thereby achieving better performance in various foreground segmentation tasks. When implementing this bootstrap strategy, the model will carefully analyze the results of the previous sampling, identify the key information therein, and use this information to guide the next step of the denoising process. In this way, each iteration is based on the previous iteration, gradually improving the accuracy and quality of the results. In this way, the model can make more full use of the knowledge learned in the training phase to achieve more efficient and accurate foreground segmentation.

[0072] In the specific implementation, in the model proposed in this embodiment, the alignment condition extraction network successfully captures the multi-scale conditional features aligned with the image, and these features are further used in the lightweight bidirectional cascade fusion denoising network to realize the reverse denoising process. In order to fully understand and effectively utilize these conditional representations, a lightweight bidirectional cascade fusion denoising network exists in the model of this embodiment. The design core of the lightweight bidirectional cascade fusion denoising network is to construct a step-by-step bidirectional feature flow, which allows the system to gradually cascade and fuse features during the reverse denoising process. Specifically, the three progressive fusion units work together in the downsampling unit of the network, and through the cascade module, the downsampling module and the residual fusion module, the deep integration of multi-scale conditional representation is realized. This integration process not only enhances the system's ability to integrate local and global information, but also improves the system's capture and expression of complex foreground structures through the interaction of bidirectional flow features.

[0073] It should be noted that, further, the lightweight bidirectional cascade fusion denoising network proposed in this embodiment effectively eliminates noise differences in a cost-effective manner while achieving a comprehensive understanding of feature representation. This is crucial to solving the difficulties encountered by traditional diffusion models during training. Complex diffusion models are often difficult to apply to practical problems due to their heavy computational burden and slow convergence speed. However, this embodiment significantly reduces the complexity of system training and speeds up the convergence speed through innovative network structure and algorithm optimization, so that the system can quickly and accurately perform foreground segmentation.

[0074] It can be understood that multiple alignment condition representations of each sample training image are input into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising. The bidirectional cascade fusion denoising network is also a lightweight bidirectional cascade fusion denoising network. In the downsampling unit of the lightweight bidirectional cascade fusion denoising network, through the cascade flow and the fusion flow, the model can integrate multi-scale information from the bottom up and refine the feature representation from the top down. This bidirectional information flow ensures that sufficient information support can be obtained at all levels from coarse to fine during the denoising process. The upsampling unit gradually restores the deep features obtained in the downsampling process to the resolution of the sample training image through the corresponding upsampling blocks and residual modules, and finally outputs an accurate prediction mask. Through training with a large number of samples, a target foreground segmentation model that can directly output an accurate foreground segmentation prediction mask is obtained.

[0075] Step S30: input the foreground image to be segmented into the target foreground segmentation model to obtain a foreground segmentation mask of the foreground image to be segmented.

[0076] In this embodiment, the intermediate state mask of each sample training image is input into the alignment condition extraction network in the segmentation model to be trained for alignment, and multiple alignment condition representations of each sample training image are determined, wherein the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascade encoding module; the multiple alignment condition representations of each sample training image are input into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, and a target foreground segmentation model is obtained, wherein the bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, and a cascade flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit; the foreground image to be segmented is input into the target foreground segmentation model, and a foreground segmentation mask of the foreground image to be segmented is obtained. Through the above methods, an alignment module based on the intermediate states in the mask diffusion process is used to improve the generalization ability of multiple tasks. The bootstrap strategy based on the previous denoising result autoregressively guides the subsequent denoising process to gradually improve the sampling quality. The U-shaped denoising network architecture based on the bidirectional feature cascade fusion realizes accurate reception and efficient processing of multi-level guidance conditions, which makes the target foreground segmentation model have excellent performance and cross-task generalization ability, ensuring the dilemma of both accuracy and generalization in the foreground segmentation task.

[0077] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 In the foreground segmentation method, step S10 further includes steps S11 to S13:

[0078] Step S11, inputting the intermediate state mask of each sample training image and each sample training image into the alignment condition extraction network in the segmentation model to be trained, and processing each sample training image through the image embedding module in the alignment condition extraction network to obtain multiple first processed images.

[0079] It should be noted that for each sample training image and its intermediate state mask at one time step: the sample training image and an intermediate state mask x t The image is input to the alignment condition extraction network. The image embedding module in the alignment condition extraction network divides and linearly projects the sample training image, and after normalization, outputs the first processed image I corresponding to the sample training image. p .

[0080] Step S12, aligning the intermediate masks of each first processed image and each sample training image through the intermediate mask alignment module in the alignment condition extraction network to obtain the to-be-encoded features of each sample training image.

[0081] It should be noted that for each sample training image and its intermediate state mask at one time step: the intermediate state mask alignment module performs alignment based on the first processed image and the intermediate state mask, aligns the mask of the foreground object in the intermediate state under different degrees of noise with the rich scene context of the image, provides an aligned representation for the reverse denoising process, and outputs the aligned conditional representation P corresponding to the sample training image out In this embodiment, the aligned conditional representation P corresponding to the sample training image output by the intermediate mask alignment module out That is, the feature to be encoded of the sample training image.

[0082] In a feasible implementation manner, step S12 may include steps B11 to B13:

[0083] Step B11, the intermediate mask alignment module in the alignment condition extraction network embeds the intermediate mask of each sample training image according to the image dimension of each first processed image, so as to obtain the embedding feature of each sample training image.

[0084] It should be noted that for each sample training image and its intermediate state mask at one time step: the intermediate state mask alignment module converts the intermediate state mask x t Perform embedding operations, including convolution CONV and layer normalization LN, to obtain the embedding features P of the sample training image x , so that it is consistent with the first processed image I p In this embodiment, P x =LN(CONV(x t )).

[0085] Step B12: Flatten the embedded features of each sample training image and each first processed image to obtain a first feature sequence and a second feature sequence of each sample training image.

[0086] It should be noted that for each sample training image and its intermediate state mask at one time step: the intermediate state mask alignment module aligns the embedded feature P x and the first processed image I p Perform a flattening operation to obtain the first feature sequence F(P x ) and the second characteristic sequence F(I p ).

[0087] Step B13, fusing and aligning the first feature sequence and the second feature sequence of each sample training image to obtain the features to be encoded of each sample training image.

[0088] It should be noted that for each sample training image and its intermediate state mask at one time step: the intermediate state mask alignment module fuses and accurately aligns the first feature sequence and the second feature sequence, and performs residual connection to obtain the feature P to be encoded of the sample training image out .

[0089] In a feasible implementation manner, step B13 may include steps C11 to C13:

[0090] Step C11, performing a fusion operation on the first feature sequence and the second feature sequence of each sample training image through multiple projection layers in the intermediate mask alignment module to obtain a fused feature sequence of each sample training image.

[0091] Step C12, aligning the fused feature sequences of each sample training image through the multi-layer perceptron and the activation layer in the intermediate state mask alignment module to obtain the fused alignment features of each sample training image.

[0092] Step C13, performing residual connection on the fused alignment features of each sample training image that has passed through the upper projection layer in the intermediate mask alignment module and each first processed image, to obtain the features to be encoded of each sample training image.

[0093] It should be noted that for each sample training image and its intermediate state mask at one time step: through the two projection layers L in the intermediate state mask alignment module e The first feature sequence F(P x ) and the second characteristic sequence F(I p ) are linearly projected and fused to obtain the fused feature sequence P union =L e1 (F(P x ))+L e2 (F(I p )). Then, the fused feature sequence is accurately aligned through the multi-layer perceptron MLP and activation layer to obtain the fused alignment feature P align =GELU(MLP(P union )). Finally, the projection layer L up The fusion alignment feature L up (P align ) and the first processed image I p Perform residual connection to restore the original dimension and obtain the aligned conditional representation P out In this embodiment, P out =L up (P align )+I p .

[0094] Step S13, encoding the to-be-encoded features of each sample training image through the multi-stage cascade encoding module in the alignment condition extraction network to obtain multiple alignment condition representations of each sample training image.

[0095] It should be noted that for each sample training image and its intermediate state mask at one time step: the intermediate state mask alignment module outputs the to-be-encoded feature P of the sample training image out Finally, the features to be encoded are passed to the multi-stage cascade encoding module, which downsamples and encodes the features to be encoded to extract features at different levels, and finally obtains the multi-scale conditional representation f1~f corresponding to an intermediate state mask of the sample training image. i In this embodiment, the conditional representations at multiple scales outputted based on an intermediate mask of the sample training image are multiple aligned conditional representations of the sample training image.

[0096] This embodiment inputs the intermediate state mask of each sample training image and each sample training image into the alignment condition extraction network in the segmentation model to be trained, processes each sample training image through the image embedding module in the alignment condition extraction network, and obtains multiple first processed images; aligns each first processed image and the intermediate state mask of each sample training image through the intermediate state mask alignment module in the alignment condition extraction network to obtain the features to be encoded of each sample training image; encodes the features to be encoded of each sample training image through the multi-stage cascade encoding module in the alignment condition extraction network to obtain multiple alignment condition representations of each sample training image. In the above manner, the multi-task generalization capability is improved.

[0097] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 4 In the foreground segmentation method, step S20 further includes steps S21 to S23:

[0098] Step S21, input multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained, and fuse the multiple alignment condition representations of each sample training image according to the first preset direction through the cascade flow in the bidirectional cascade fusion denoising network, and output the cascade features of each sample training image.

[0099] It should be noted that the first preset direction refers to the bottom-up direction. For each sample training image and its intermediate state mask at one time step: the cascade flow extracts the alignment condition representation f of the output of the corresponding layer of the alignment condition extraction network from the bottom to the top between each cascade module. i Fusion of multi-scale conditional representations Fi+1 Specifically: the previous level representation F i+1 Don't go through two convolutional layers, ReLU activation layer CR 2 And a convolution layer and batch normalization CB, and finally with f i Connect to get F i , F i =concat(CB(CR 2 (F i+1 ),f i )), i = 1, 2, 3. In this embodiment, the representation F output by the cascade flow i That is the cascade feature of the sample training image.

[0100] Step S22: outputting a feature map of each sample training image according to the cascade features of each sample training image and the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network.

[0101] It should be noted that for each sample training image and its intermediate state mask at one time step: the fusion flow converts the cascade feature F output by the cascade flow i and the intermediate state mask x input to the downsampling block t Through residual connection and other methods, the feature map SC of the sample training image is obtained. i .

[0102] In a feasible implementation, step S22 may include steps D11 to D14:

[0103] Step D11, down-sampling the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network to obtain the intermediate state of each sample training image.

[0104] Step D12, fusing the intermediate state of each sample training image and the cascade feature of each sample training image to obtain an embedded representation of each sample training image.

[0105] Step D13, gradually fuse the embedded representation and time step of each sample training image to obtain the unit layer output corresponding to each sample training image.

[0106] Step D14, obtaining a feature map of each sample training image according to the unit layer output corresponding to each sample training image and the embedded representation of each sample training image.

[0107] It should be noted that for each sample training image and its intermediate state mask at one time step: the intermediate state mask x is fused by the downsampling block of the fusion flow t Downsample to obtain the intermediate state D of the sample training image iIn this embodiment, the downsampling block downsamples the input to the same scale as the output of the cascade module. The residual fusion module processes the cascade feature F i , the intermediate state of downsampling D i And time steps. Through convolution, group normalization and SILU activation layers, the cascade features F are fused. i and the intermediate state D of downsampling i Fusion gets the embedded representation E i In this embodiment, E i =CGA(F i )+D i .

[0108] It can be understood that the embedding representation E i The time step t is gradually integrated, allowing the lightweight bidirectional cascade fusion module to perceive the time step of the diffusion process, and use it as the output of the unit of this layer through time embedding and scaling STE to obtain the unit layer output D out =S TE(t) (CGA(E i )). Finally, based on the unit layer output D out =S TE(t) (CGA(E i )), two residual connections are used to further enhance the input SC of the skip connection in the same-layer upsampling module i In this embodiment, SC i =CGA(D out +CGA(F i ))+E i , the feature map of the sample training image refers to the output SC of the fusion flow i .

[0109] Step S23, inputting the feature map of each sample training image into the upsampling unit in the cascade fusion denoising network for feature recovery to obtain a target foreground segmentation network.

[0110] It should be noted that the output SC of the fused stream i It will be input to the downsampling block and the upsampling unit, and the upsampling unit will be based on the feature map SC i Perform feature recovery and repeat the training process to finally obtain the target foreground segmentation network.

[0111] This embodiment inputs multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained, and fuses multiple alignment condition representations of each sample training image in a first preset direction through the cascade flow in the bidirectional cascade fusion denoising network, and outputs the cascade features of each sample training image; outputs the feature map of each sample training image according to the cascade features of each sample training image and the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network; inputs the feature map of each sample training image into the upsampling unit in the cascade fusion denoising network for feature recovery, and obtains the target foreground segmentation network. In the above manner, the bootstrap strategy of the subsequent denoising process is guided by autoregression based on the previous denoising result to gradually improve the sampling quality, and at the same time, the U-shaped denoising network architecture based on the bidirectional feature cascade fusion realizes accurate reception and efficient processing of multi-level guidance conditions, thereby improving the performance of the model.

[0112] For example, to help understand the implementation process of the foreground segmentation method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 5 , Figure 5 A brief flowchart of a foreground segmentation method is provided, specifically:

[0113] The foreground segmentation model of this embodiment is aligned with the conditional extraction network and lightweight bidirectional cascade fusion, and the denoising network includes an aligned conditional extraction network composed of an image embedding module (i.e., an embedding layer), an intermediate state mask alignment module, and a multi-stage cascade coding module (i.e., a coding layer). The lightweight bidirectional cascade fusion denoising network includes an upsampling unit and a downsampling unit. The downsampling unit includes three progressive fusion units, each of which is composed of a cascade module, a downsampling block, and a residual fusion module; the upsampling unit includes three small units, each of which is composed of a residual module and an upsampling block. In this embodiment, a progressive bidirectional feature flow is established between the three progressive fusion units: a cascade flow and a fusion flow. The cascade flow fuses the conditional representation of the output corresponding to the aligned conditional extraction network with the conditional representation of the previous level in a bottom-up direction between each cascade module; the fusion flow is composed of a downsampling block and a residual fusion module, and the corresponding input of the cascade flow and the input of the downsampling block are fully fused through residual connections and other methods.

[0114] It can be understood that in the training phase of this embodiment, a large number of sample training images and their corresponding true value masks are used as input, and training is performed by aligning the image with the mask and denoising and restoring the mask; in the bootstrap inference phase, an autoregressive mode is adopted, and the temporary intermediate sampling results are used as input to enhance the next step sampling results. In this embodiment, there are three main processes involved in the training phase: the forward process, forward denoising the true value mask through random sampling time steps to obtain diffuse intermediate state masks with different noise levels; the conditional extraction process, aligning the original image and the intermediate state mask, and extracting multi-scale conditional features; the denoising process, using the U-shaped structure to gradually denoise and improve the mask.

[0115] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the foreground segmentation method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0116] This application also provides a foreground segmentation device, please refer to Figure 6 , the foreground segmentation device comprises:

[0117] The alignment module 10 is used to input the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determine multiple alignment condition representations of each sample training image. The alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module and a multi-stage cascade encoding module.

[0118] The denoising module 20 is used to input multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, so as to obtain a target foreground segmentation model. The bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, and a cascade flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit.

[0119] The segmentation module 30 is used to input the foreground image to be segmented into the target foreground segmentation model to obtain a foreground segmentation mask of the foreground image to be segmented.

[0120] Optionally, the alignment module 10 is further used for:

[0121] The intermediate state mask of each sample training image and each sample training image are input into the alignment condition extraction network in the segmentation model to be trained, and each sample training image is processed by the image embedding module in the alignment condition extraction network to obtain a plurality of first processed images; the intermediate state mask alignment module in the alignment condition extraction network is used to align each first processed image and the intermediate state mask of each sample training image to obtain the feature to be encoded of each sample training image; the feature to be encoded of each sample training image is encoded by the multi-stage cascade encoding module in the alignment condition extraction network to obtain a plurality of alignment condition representations of each sample training image.

[0122] Optionally, the alignment module 10 is further used for:

[0123] The intermediate mask alignment module in the alignment condition extraction network embeds the intermediate mask of each sample training image according to the image dimension of each first processed image to obtain the embedded features of each sample training image; flattens the embedded features of each sample training image and each first processed image to obtain the first feature sequence and the second feature sequence of each sample training image; and performs fusion alignment based on the first feature sequence and the second feature sequence of each sample training image to obtain the features to be encoded of each sample training image.

[0124] Optionally, the alignment module 10 is further used for:

[0125] The first feature sequence and the second feature sequence of each sample training image are fused through the multiple projection layers in the intermediate state mask alignment module to obtain the fused feature sequence of each sample training image; the fused feature sequence of each sample training image is aligned through the multi-layer perceptron and the activation layer in the intermediate state mask alignment module to obtain the fused alignment features of each sample training image; the fused alignment features of each sample training image that has passed through the upper projection layer in the intermediate state mask alignment module and each first processed image are residually connected to obtain the features to be encoded of each sample training image.

[0126] Optionally, the denoising module 20 is further configured to:

[0127] The multiple alignment condition representations of each sample training image are input into the bidirectional cascade fusion denoising network in the segmentation model to be trained, and the multiple alignment condition representations of each sample training image are fused according to a first preset direction through the cascade flow in the bidirectional cascade fusion denoising network to output the cascade features of each sample training image; the feature map of each sample training image is output according to the cascade features of each sample training image and the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network; the feature map of each sample training image is input into the upsampling unit in the cascade fusion denoising network for feature recovery, so as to obtain the target foreground segmentation network.

[0128] Optionally, the denoising module 20 is further configured to:

[0129] The intermediate state mask of each sample training image is downsampled by the fusion flow in the bidirectional cascade fusion denoising network to obtain the intermediate state of each sample training image; the intermediate state of each sample training image and the cascade feature of each sample training image are fused to obtain the embedded representation of each sample training image; the embedded representation and the time step of each sample training image are gradually fused to obtain the unit layer output corresponding to each sample training image; according to the unit layer output corresponding to each sample training image and the embedded representation of each sample training image, the feature map of each sample training image is obtained.

[0130] Optionally, the alignment module 10 is further used for:

[0131] Obtain each sample training image and the true value mask of each sample training image; perform forward noise addition on the true value mask of each sample training image at a randomly selected time step to obtain an intermediate state mask of each sample training image.

[0132] The foreground segmentation device provided by the present application adopts the foreground segmentation method in the above embodiment, which can solve the technical problem that the foreground segmentation model in the prior art cannot ensure both accuracy and generalization when performing the foreground segmentation task. Compared with the prior art, the beneficial effects of the foreground segmentation device provided by the present application are the same as the beneficial effects of the foreground segmentation method provided by the above embodiment, and the other technical features in the foreground segmentation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0133] The present application provides a foreground segmentation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the foreground segmentation method in the above-mentioned embodiment 1.

[0134] Reference below Figure 7, which shows a schematic diagram of the structure of a foreground segmentation device suitable for implementing the embodiment of the present application. The foreground segmentation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The foreground segmentation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0135] like Figure 7 As shown, the foreground segmentation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the foreground segmentation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the foreground segmentation device to communicate with other devices wirelessly or by wire to exchange data. Although the foreground segmentation device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0136] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0137] The foreground segmentation device provided by the present application adopts the foreground segmentation method in the above embodiment, which can solve the technical problem that the foreground segmentation model in the prior art cannot simultaneously ensure accuracy and generalization when performing the foreground segmentation task. Compared with the prior art, the beneficial effects of the foreground segmentation device provided by the present application are the same as the beneficial effects of the foreground segmentation method provided by the above embodiment, and the other technical features in the foreground segmentation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0138] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0140] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the foreground segmentation method in the above-mentioned embodiment.

[0141] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0142] The computer-readable storage medium may be included in the foreground segmentation device; or may exist independently without being assembled into the foreground segmentation device.

[0143] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the foreground segmentation device, the foreground segmentation device: inputs the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determines multiple alignment condition representations of each sample training image, wherein the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascade encoding module; inputs the multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, and obtains a target foreground segmentation model, wherein the bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, and a cascade flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit; inputs the foreground image to be segmented into the target foreground segmentation model, and obtains a foreground segmentation mask of the foreground image to be segmented.

[0144] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0147] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the foreground segmentation method described above, and can solve the technical problem that the foreground segmentation model in the prior art cannot simultaneously ensure accuracy and generalization when performing the foreground segmentation task. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the foreground segmentation method provided by the above embodiment, and will not be described in detail here.

[0148] The present application also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the foreground segmentation method as described above are implemented.

[0149] The computer program product provided by the present application can solve the technical problem that the foreground segmentation model in the prior art cannot guarantee both accuracy and generalization when performing the foreground segmentation task. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the foreground segmentation method provided by the above embodiment, and will not be described in detail here.

[0150] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A foreground segmentation method, characterized in that: The method comprises: Inputting the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determining multiple alignment condition representations of each sample training image, wherein the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascade encoding module; Inputting multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, to obtain a target foreground segmentation model, wherein the bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, and a cascade flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit; The foreground image to be segmented is input into the target foreground segmentation model to obtain a foreground segmentation mask of the foreground image to be segmented.

2. The method according to claim 1, characterized in that The step of inputting the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determining multiple alignment condition representations of each sample training image comprises: Inputting the intermediate state mask of each sample training image and each sample training image into the alignment condition extraction network in the segmentation model to be trained, and processing each sample training image through the image embedding module in the alignment condition extraction network to obtain a plurality of first processed images; Aligning the intermediate state masks of each first processed image and each sample training image through the intermediate state mask alignment module in the alignment condition extraction network to obtain the to-be-encoded features of each sample training image; The features to be encoded of each sample training image are encoded by the multi-stage cascade encoding module in the alignment condition extraction network to obtain multiple alignment condition representations of each sample training image.

3. The method according to claim 2, characterized in that The step of aligning the intermediate state mask alignment module in the alignment condition extraction network according to the intermediate state masks of each first processed image and each sample training image to obtain the features to be encoded of each sample training image comprises: The intermediate state mask alignment module in the alignment condition extraction network performs an embedding operation on the intermediate state mask of each sample training image according to the image dimension of each first processed image, so as to obtain an embedding feature of each sample training image; Flattening the embedded features of each sample training image and each first processed image to obtain a first feature sequence and a second feature sequence of each sample training image; The first feature sequence and the second feature sequence of each sample training image are fused and aligned to obtain the to-be-encoded feature of each sample training image.

4. The method according to claim 3, characterized in that The step of fusing and aligning the first feature sequence and the second feature sequence of each sample training image to obtain the features to be encoded of each sample training image comprises: Performing a fusion operation on the first feature sequence and the second feature sequence of each sample training image through the multiple projection layers in the intermediate state mask alignment module to obtain a fused feature sequence of each sample training image; Aligning the fused feature sequences of each sample training image through the multi-layer perceptron and the activation layer in the intermediate state mask alignment module to obtain the fused alignment features of each sample training image; The fused alignment features of each sample training image that has passed through the upper projection layer in the intermediate state mask alignment module and each first processed image are residually connected to obtain the features to be encoded of each sample training image.

5. The method according to claim 1, characterized in that The step of inputting multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising to obtain the target foreground segmentation model includes: Inputting multiple alignment condition representations of each sample training image into the bidirectional cascade fusion denoising network in the segmentation model to be trained, fusing the multiple alignment condition representations of each sample training image according to a first preset direction through the cascade flow in the bidirectional cascade fusion denoising network, and outputting cascade features of each sample training image; Outputting a feature map of each sample training image according to the cascade features of each sample training image and the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network; The feature map of each sample training image is input into the upsampling unit in the cascade fusion denoising network for feature recovery to obtain a target foreground segmentation network.

6. The method according to claim 5, characterized in that The step of outputting the feature map of each sample training image according to the cascade features of each sample training image and the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network comprises: Down-sampling the intermediate state mask of each sample training image through the fusion flow in the bidirectional cascade fusion denoising network to obtain the intermediate state of each sample training image; The intermediate state of each sample training image and the cascade features of each sample training image are fused to obtain an embedded representation of each sample training image; The embedded representation and time step of each sample training image are gradually fused to obtain the unit layer output corresponding to each sample training image; According to the unit layer output corresponding to each sample training image and the embedded representation of each sample training image, the feature map of each sample training image is obtained.

7. The method according to any one of claims 1 to 6, characterized in that Before the step of inputting the multiple intermediate state masks of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment and determining the multiple alignment condition representations of each sample training image, the method further includes: Obtain each sample training image and the true value mask of each sample training image; The true value mask of each sample training image is forward-noised at a randomly selected time step to obtain the intermediate state mask of each sample training image.

8. A foreground segmentation device, characterized in that: The foreground segmentation device comprises: An alignment module is used to input the intermediate state mask of each sample training image into the alignment condition extraction network in the segmentation model to be trained for alignment, and determine multiple alignment condition representations of each sample training image, wherein the alignment condition extraction network is composed of an image embedding module, an intermediate state mask alignment module, and a multi-stage cascade encoding module; A denoising module is used to input multiple alignment condition representations of each sample training image into a bidirectional cascade fusion denoising network in the segmentation model to be trained for denoising, so as to obtain a target foreground segmentation model, wherein the bidirectional cascade fusion denoising network includes a downsampling unit and an upsampling unit, and a cascade flow and a fusion flow are established between multiple progressive fusion units in the downsampling unit; The segmentation module is used to input the foreground image to be segmented into the target foreground segmentation model to obtain a foreground segmentation mask of the foreground image to be segmented.

9. A foreground segmentation device, characterized in that: The foreground segmentation device comprises: a memory, a processor, and a foreground segmentation program stored in the memory and executable on the processor, wherein the foreground segmentation program is configured to implement the foreground segmentation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a foreground segmentation program, and when the foreground segmentation program is executed by the processor, the foreground segmentation method according to any one of claims 1 to 7 is implemented.

Images