Target elimination method and device, electronic equipment and storage medium

Abstract

The application provides a target elimination method and device, electronic equipment and storage medium, and relates to the field of artificial intelligence. The method comprises the following steps: encoding mask visual data to obtain a first latent representation; wherein the mask visual data is obtained by masking a target object to be eliminated in input visual data based on first mask information, and the first mask information is used for indicating the position of the target object in the input visual data; splicing the first latent representation and the first mask information to obtain a first spliced feature, and extracting a context of the first spliced feature to obtain target context information; and performing denoising processing according to the target context information to generate target visual data after the target object is eliminated. Therefore, the denoising process can generate repair content that is coordinated with the surrounding environment in structure, texture and dynamic characteristics on the premise of preserving spatiotemporal consistency, thereby significantly improving the integrity of target elimination and the realism of background reconstruction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a target elimination method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the explosive growth of digital video content, the demand for efficient and intelligent video editing technologies is becoming increasingly urgent. Object removal, as a core function, aims to automatically remove specified target objects from videos and generate visually coherent and content-appropriate alternative backgrounds. However, related technologies still commonly suffer from problems such as unnatural results, inter-frame flickering, structural distortion, or loss of detail when processing dynamic videos. Summary of the Invention

[0003] This application proposes a target elimination method, apparatus, electronic device, and storage medium, which aims to at least partially solve one of the technical problems in the related art.

[0004] One embodiment of this application proposes a target elimination method, including:

[0005] The masked visual data is encoded to obtain a first latent representation; wherein the masked visual data is obtained by masking the target object to be eliminated in the input visual data based on the first mask information, and the first mask information is used to indicate the position of the target object in the input visual data; The first latent representation is concatenated with the first mask information to obtain a first concatenated feature, and the first concatenated feature is subjected to context extraction to obtain target context information. Denoising is performed based on the target context information to generate target visual data after the target object is eliminated.

[0006] Another embodiment of this application proposes a target elimination device, comprising: An encoding module is used to encode masked visual data to obtain a first latent representation; wherein the masked visual data is obtained by masking the target object to be eliminated in the input visual data based on the first mask information, and the first mask information is used to indicate the position of the target object in the input visual data; The splicing module is used to splice the first latent representation with the first mask information to obtain the first splicing feature; The extraction module is used to extract the context of the first spliced ​​features to obtain the target context information; The denoising module is used to perform denoising processing based on the target context information to generate target visual data after the target object is eliminated.

[0007] In another aspect of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the target elimination method as described in the foregoing aspect.

[0008] Another aspect of this application provides a non-transitory computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the target elimination method as described in the foregoing aspect.

[0009] In another aspect, this application provides a computer program product having a computer program stored thereon, which, when executed by a processor, implements the target elimination method as described in the foregoing aspect.

[0010] The target removal method, apparatus, electronic device, and storage medium proposed in this application obtain a first latent representation by encoding mask visual data, so that the object region corresponding to the target object to be removed is explicitly "excavated" in the latent space, thereby providing an initial representation for background reconstruction. The first latent representation is explicitly concatenated with the first mask information (i.e., the spatial location indication of the target object), which simultaneously preserves the positional prior of the missing region and the semantic features of the surrounding visible content in the latent space. Context extraction is performed on the concatenated features to generate target context information that reflects the surrounding structure, texture, and motion patterns of the target object. This context information is used as a conditional input for the denoising process, enabling the denoising process to generate repair content that is coordinated with the surrounding environment in terms of structure, texture, and dynamic characteristics while preserving spatiotemporal consistency. Since context modeling and denoising generation are designed as two ordered and collaborative stages, the guiding role of spatial context is fully utilized, while semantic confusion under noise interference is avoided. Ultimately, typical artifacts such as boundary blurring, structural breaks, and inter-frame flicker are effectively suppressed. While maintaining temporal continuity and visual naturalness, the integrity of target removal and the realism of background reconstruction are significantly improved.

[0011] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a target elimination method provided for an exemplary embodiment of this application; Figure 2 A flowchart illustrating another target elimination method provided for an exemplary embodiment of this application; Figure 3 A flowchart illustrating a training method for a target elimination model provided for an exemplary embodiment of this application; Figure 4 A flowchart illustrating another training method for a target elimination model provided for an exemplary embodiment of this application; Figure 5 A flowchart illustrating another training method for a target elimination model provided for an exemplary embodiment of this application; Figure 6 A schematic diagram illustrating the model structure and training principle of a target elimination model provided for an exemplary embodiment of this application; Figure 7 A schematic diagram of the structure of a target elimination device provided for an exemplary embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. Detailed Implementation

[0013] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0014] With the continuous development of deep learning, such as diffusion models and vision transformers (ViT), video generation and editing technologies have made significant progress. Early video object removal methods mainly relied on non-generative means, such as optical flow-based pixel propagation or local context reasoning using 3D convolutional networks.

[0015] However, these methods are essentially "copy-paste" pixel migrations, lacking true content generation capabilities, and are prone to problems such as blurring, artifacts, and time flickering when faced with large-area occlusion, complex motion, or missing textures.

[0016] To overcome the aforementioned limitations, related technologies propose using generative video diffusion models to remove targets from videos. These video diffusion models are typically based on the Transformer architecture (DiffusionTransformer, or DiT for short), utilizing causal attention mechanisms to model long-range temporal dependencies, thereby improving the model's spatiotemporal modeling capabilities and high-fidelity video generation performance.

[0017] In related technologies, generative video diffusion models face the dual tasks of extracting features from the input non-elimination region and predicting the elimination region. This makes them ineffective at eliminating artifacts and failing to effectively remove negative effects from objects such as shadows, reflections, and lighting. Furthermore, during training, they heavily rely on paired data (videos before and after object elimination), but collecting such data is costly and difficult. While simulation engines can be used to construct paired data, the differences between simulated data and real-world scenarios can also lead to noticeable artifacts and unnatural background restoration in the video diffusion model's output after object elimination.

[0018] In view of at least one of the problems existing in the above-mentioned related technologies, this application proposes a target elimination method, apparatus, electronic device and storage medium.

[0019] The target elimination method, apparatus, electronic device, and storage medium of this application are described below with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a target elimination method provided for an exemplary embodiment of this application.

[0021] It should be noted that the target elimination method of this application embodiment can be applied to a target elimination device. In some possible embodiments, the target elimination device can be configured in an electronic device so that the electronic device can perform the target elimination function. In addition, in some possible embodiments, the target elimination device can also be software in the electronic device.

[0022] In any embodiment of this application, the electronic device includes, but is not limited to, a terminal, a personal computer, a server (or a server-side device, cloud computing device), etc.

[0023] In this context, a terminal is a user-side entity used to receive or transmit signals, such as a mobile phone. A terminal can also be called a terminal device (terminal), user equipment (UE), mobile station (MS), mobile terminal device (MT), etc. Terminals can be communication-enabled vehicles, smart cars, mobile phones, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) terminals, augmented reality (AR) terminals, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on. The embodiments in this application do not limit the specific technology or device form used in the terminal.

[0024] like Figure 1 As shown, the target elimination method may include the following steps S101 to S103: Step S101: Encode the masked visual data to obtain a first latent representation; wherein, the masked visual data is obtained by masking the target object to be eliminated in the input visual data based on the first mask information, and the first mask information is used to indicate the position of the target object in the input visual data.

[0025] The visual data includes images and videos. That is, the input visual data includes input images and input videos.

[0026] The input visual data displays target objects to be eliminated, including but not limited to: people, plants, animals, vehicles, etc.

[0027] It should be noted that the target object can be a user-specified target to be eliminated, or it can be a preset target to be eliminated. This application embodiment does not limit this.

[0028] It should also be noted that this application does not limit the method of obtaining input visual data. For example, input visual data can be real visual data taken or recorded by the user, or it can be artificially synthesized visual data, visual data collected online, visual data obtained from training sets or test sets, or visual data generated using Artificial Intelligence Generated Content (AIGC) technology, etc.

[0029] The first mask information is used to indicate the position of the target object in the input visual data. Taking the input visual data as an input image as an example, the first mask information can also be called a first mask image, which is used to indicate the position of the target object in the input image; taking the input visual data as an input video as an example, the first mask information can also be called a first mask sequence, which is used to indicate the position of the target object in each video frame of the input video.

[0030] The first mask information can be obtained by semantic segmentation of the input visual data. For example, taking the input visual data as an input video, semantic segmentation can be performed on the input video to obtain a first mask sequence. The first mask sequence includes mask images of each video frame in the input video. The mask image can be a binary image, where each pixel has a value of 0 or 1. A pixel value of 1 indicates that the corresponding pixel belongs to the target object, and a pixel value of 0 indicates that the corresponding pixel does not belong to the target object.

[0031] The masked visual data is obtained by masking the target object in the input visual data based on the first mask information. For example, taking the input visual data as an input video and the masked visual data as a masked video, the pixels belonging to the target object in each video frame of the input video can be determined according to the first mask sequence, and each determined pixel can be assigned an equal pixel value to obtain the masked video. For instance, the pixel value of each pixel belonging to the target object in the video frame can be assigned to 128 (gray) to obtain the masked video.

[0032] In this embodiment, the masked visual data can be encoded using visual coding techniques to obtain a first latent representation. For example, taking video as the visual data, the first latent representation can also be called a masked video latent representation.

[0033] As an example, a variational autoencoder (VAE) can be used to encode the masked visual data to obtain a first latent representation.

[0034] Step S102: Concatenate the first latent representation with the first mask information to obtain the first concatenated feature, and extract the context from the first concatenated feature to obtain the target context information.

[0035] In this embodiment, the first latent representation and the first mask information can be concatenated (concatenated) to obtain a first concatenated feature, and the first concatenated feature can be used to extract the context to obtain the target context information. For example, deep learning techniques, such as a context network, can be used to extract the context from the first concatenated feature to obtain the target context information.

[0036] Step S103: Denoising is performed based on the target context information to generate target visual data after the target object is eliminated.

[0037] In this embodiment of the application, denoising can be performed based on target context information to generate a target video after the target object has been removed.

[0038] As an application scenario, the solution provided in this application can be applied to video content editing and processing scenarios, especially suitable for mobile phone photo albums, video editing applications, smart terminals, or cloud video processing systems, to automatically remove foreground objects and restore backgrounds in user-captured videos or images. For example, taking video as visual data, if unwanted foreground objects such as pedestrians, animals, vehicles, or obstructions appear in a user-captured video, the solution provided in this application can analyze and process the video frame by frame to remove the specified foreground objects (referred to as the target objects to be removed in this application) and restore reasonable background content, thereby removing redundant foreground objects while maintaining the temporal consistency and visual naturalness of the video.

[0039] The target removal method in this application encodes the mask visual data to obtain a first latent representation, so that the object region corresponding to the target object to be removed is explicitly "excavated" in the latent space, thereby providing an initial representation for background reconstruction. The first latent representation is then explicitly concatenated with the first mask information (i.e., the spatial location indication of the target object), which simultaneously preserves the positional prior of the missing region and the semantic features of the surrounding visible content in the latent space. Context extraction is performed on the concatenated features to generate target context information that reflects the surrounding structure, texture, and motion patterns of the target object. This context information is then used as the conditional input for the denoising process, enabling the denoising process to generate repair content that is coordinated with the surrounding environment in terms of structure, texture, and dynamic characteristics while preserving spatiotemporal consistency. Since context modeling and denoising generation are designed as two ordered and collaborative stages, the guiding role of spatial context is fully utilized, while semantic confusion under noise interference is avoided. Ultimately, typical artifacts such as boundary blurring, structural breaks, and inter-frame flicker are effectively suppressed. While maintaining temporal continuity and visual naturalness, the completeness of target removal and the realism of background reconstruction are significantly improved.

[0040] As one possible implementation method, Figure 2 A flowchart illustrating another target elimination method provided for an exemplary embodiment of this application.

[0041] It should be noted that the target elimination method can be executed alone, or it can be executed together with any embodiment of this application or any possible implementation in the embodiment, or it can be executed together with any technical solution in the related technology. The embodiments of this application do not limit this.

[0042] like Figure 2 As shown, the target elimination method may include the following steps S201 to S205: Step S201: Encode the masked visual data to obtain a first latent representation; wherein, the masked visual data is obtained by masking the target object to be eliminated in the input visual data based on the first mask information, and the first mask information is used to indicate the position of the target object in the input visual data.

[0043] Step S202: The first latent representation is concatenated with the first mask information to obtain the first concatenated feature.

[0044] Step S203: Extract the context of the first spliced ​​feature to obtain the target context information.

[0045] It should be noted that the explanations of steps S201 to S203 can be found in the relevant descriptions in any embodiment of this application, and will not be repeated here.

[0046] For example, the first concatenated features can be input into the context net in the trained target elimination model to extract the target context information.

[0047] There are no restrictions on the network structure of the context network; it can be a network structure based on the Transformer architecture, or it can be a network structure based on a non-Transformer architecture.

[0048] Step S204: Denoising is performed based on the target context information and the first target noise to obtain the target potential representation.

[0049] The first target noise is obtained by noise extraction using a set method. For example, the first target noise can be extracted using a random sampling noise extraction method.

[0050] In this embodiment, the target context information and the first target noise can be combined for denoising to obtain the latent target representation. For example, the target context information and the first target noise are input into the denoising network of the target elimination model for denoising to obtain the latent target representation.

[0051] There are no restrictions on the network structure of the denoising network. It can be a network structure based on the Transformer architecture, or it can be a network structure based on a non-Transformer architecture. For example, the denoising network can be a diffusion model (DiT) based on the Transformer architecture.

[0052] Taking DiT as an example of a denoising network, the target context information can be used as a conditional feature, and together with the first target noise, it can be used as the input of DiT to obtain the target latent representation output by DiT.

[0053] In summary, the context network performs structured semantic extraction on the first concatenated features that incorporate mask information, generating target context information rich in prior knowledge of the surrounding environment. This information is then used as a conditional input to the denoising network to guide the refined reconstruction of noisy latent representations. This explicit stage decoupling design ensures that context modeling is not affected by noise interference during the denoising process, while ensuring that the denoising process is always guided by high-fidelity spatial semantics. This effectively avoids feature confusion and gradient conflicts common in end-to-end joint optimization, thereby significantly improving the structural consistency and temporal stability of background restoration.

[0054] Step S205: Decode the latent representation of the target to obtain the visual data of the target after the target object is eliminated.

[0055] In this embodiment, visual decoding technology can be used to decode the latent representation of the target to obtain the target visual data after the target object is eliminated. That is, the target visual data is a video obtained by eliminating the object region to which the target object belongs in the input visual data and optimizing the background of the eliminated object region. For example, a VAE decoder can be used to decode the latent representation of the target to obtain the target visual representation after the target object is eliminated.

[0056] As an example, a VAE encoder can be used to encode the mask visual data to obtain a first latent representation. The first latent representation is then concatenated with a first mask sequence to obtain a first concatenated feature. Subsequently, the first concatenated feature is input into the context network in the target elimination model for context extraction to obtain the target context information output by the context network. The target context information and the first target noise are then input into the denoising network in the target elimination model for denoising processing to obtain the target latent representation output by the denoising network. Finally, a VAE decoder is used to decode the target latent representation to obtain the target visual representation.

[0057] The target elimination method in this application uses target context information as a conditional feature of the denoising process to guide the denoising network to gradually reconstruct background content that is semantically consistent with the surrounding environment from the first target noise in the latent space. This makes the denoising process not only dependent on global priors, but also subject to precise constraints from local structure, texture and motion laws. On this basis, decoding the target latent representation output by the denoising network can generate visually coherent and detailed target visual data.

[0058] It should be noted that the above are various embodiments corresponding to the application method of the target elimination model (i.e., the target elimination method). This application also proposes a training method for the target elimination model. Figure 3 This is a flowchart illustrating a training method for a target elimination model provided for an exemplary embodiment of this application.

[0059] It should be noted that the training method of the target elimination model can be executed alone, or it can be executed together with any embodiment of this application or any possible implementation in the embodiment, or it can be executed together with any technical solution in the related technology. The embodiments of this application do not limit this.

[0060] like Figure 3 As shown, the training method for this target elimination model may include the following steps S301 to S303: Step S301: Obtain first sample visual data, second sample visual data, third sample visual data, and second mask information displaying sample objects; wherein, the second sample visual data is obtained by eliminating sample objects in the first sample visual data, the second mask information is used to indicate the position of sample objects in the first sample visual data, and the third sample visual data is obtained by masking sample objects in the first sample visual data based on the second mask information.

[0061] The first sample visual data displays sample objects to be eliminated. This first sample visual data can also be referred to as visual data before target elimination, and includes images and videos. It should be noted that this application does not limit the method of acquiring the first sample visual data. For example, the first sample visual data can be simulated data, real visual data captured or recorded by a user, artificially synthesized visual data, visual data collected online, visual data obtained from a training set, or visual data generated using AIGC technology, etc.

[0062] The second sample visual data is obtained by eliminating the relevant regions of the sample objects in the first sample visual data and then optimizing the background of the eliminated relevant regions. The relevant regions include the object region to which the sample object belongs, as well as the influence region of the sample object (such as shadows, reflections, self-illuminating halos, etc.). The second sample visual data can also be referred to as the visual data after target elimination. It should be noted that this application does not limit the method of obtaining the second sample visual data. For example, the second sample visual data and the first sample visual data can be a pair of visual data obtained from the training set, or they can be obtained by eliminating targets in the first sample visual data in a simulation environment, etc.

[0063] The second mask information is used to indicate the position of the sample object in the first sample visual data. Taking the first sample visual data as a first sample image as an example, the second mask information can also be called a second mask image, which is used to indicate the position of the sample object in the first sample image; taking the first sample visual data as a first sample video as an example, the second mask information can also be called a second mask sequence, which is used to indicate the position of the sample object in each video frame of the first sample video.

[0064] The second mask information can be obtained by semantic segmentation of the first sample visual data. For example, taking the first sample visual data as a first sample video, semantic segmentation can be performed on the first sample video to obtain a second mask sequence. The second mask sequence includes mask images of each video frame in the first sample video. The mask image can be a binary image, where each pixel has a value of 0 or 1. A pixel value of 1 indicates that the corresponding pixel belongs to the sample object, and a pixel value of 0 indicates that the corresponding pixel does not belong to the sample object.

[0065] The third sample visual data is obtained by masking the sample objects in the first sample visual data based on the second mask information. For example, taking the first sample visual data as the first sample video and the third sample visual data as the third sample video, the pixels belonging to the sample objects in each video frame of the first sample video can be determined according to the second mask sequence, and the determined pixels can be assigned equal pixel values ​​to obtain the third sample video. For instance, the pixel value of each pixel belonging to the sample object in the video frame can be assigned to 128 (gray) to obtain the third sample video.

[0066] Step S302: Generate light and shadow mask information based on the difference between the first sample visual data and the second sample visual data; wherein, the light and shadow mask information is used to indicate the range of visual and light and shadow influence of the sample object on the first sample visual data.

[0067] The effect-aware mask information is used to indicate the range of visual and lighting effects of the sample object on the first sample visual data. For example, if the visual data is an image, the effect-aware mask information can also be called an effect-aware mask image, which indicates the range of visual and lighting effects of the sample object on the first sample image; if the visual data is video, the effect-aware mask information can also be called an effect-aware mask sequence, which indicates the range of visual and lighting effects (such as illumination, shadows, and reflections) of the sample object on each video frame in the first sample video.

[0068] That is, the light and shadow mask information is used to indicate the elimination of various visual accompanying effects of the sample object on the first sample visual data.

[0069] In this embodiment, a first sample visual data and a second sample visual data can be compared to generate a light and shadow mask information based on the difference between the two samples. For example, taking video as the visual data, a first sample video and a second sample video can be compared to generate a light and shadow mask sequence based on the difference between corresponding video frames in the two videos. The light and shadow mask sequence includes a light and shadow mask image corresponding to each video frame, which indicates the visual and light and shadow influence range of the sample object on the corresponding video frame.

[0070] Step S303: Based on the second mask information, the light and shadow mask information, the second sample visual data, and the third sample visual data, jointly train the context network and the denoising network in the target elimination model.

[0071] In this embodiment of the application, the context network and the denoising network in the target elimination model can be jointly trained by integrating the second mask information, the light and shadow mask information, the second sample visual data and the third sample visual data.

[0072] The training method of the target elimination model in this application embodiment uses second mask information to indicate the position of the sample object to be eliminated, third sample visual data to provide the input form of the occluded background, second sample visual data as the ideal reconstruction target, and light and shadow mask information to explicitly characterize the range of indirect visual influences such as lighting, shadows and reflections after the sample object is removed. By combining these four factors, the context network and the denoising network are jointly trained, so that the target elimination model not only learns to fill the geometrically missing areas, but also synchronously restores the lighting environment that has changed due to the presence of the object. This significantly improves the realism of the target elimination results in terms of structure, texture and lighting consistency, effectively alleviates the artifact problems such as "hollowness", lighting mismatch or residual shadows that are common in related technologies, and achieves a more natural and high-fidelity target elimination effect.

[0073] As one possible implementation method, Figure 4 A flowchart illustrating another method for training a target elimination model provided for an exemplary embodiment of this application.

[0074] It should be noted that the training method of the target elimination model can be executed alone, or it can be executed together with any embodiment of this application or any possible implementation in the embodiment, or it can be executed together with any technical solution in the related technology. The embodiments of this application do not limit this.

[0075] like Figure 4 As shown, the training method for this target elimination model may include the following steps S401 to S406: Step S401: Obtain the first sample visual data, the second sample visual data, the third sample visual data, and the second mask information of the sample object being displayed.

[0076] The second sample visual data is obtained by eliminating sample objects in the first sample visual data. The second mask information is used to indicate the position of the sample object in the first sample visual data. The third sample visual data is obtained by masking the sample objects in the first sample visual data based on the second mask information.

[0077] Step S402: Generate light and shadow mask information based on the difference between the first sample visual data and the second sample visual data.

[0078] Among them, the light and shadow mask information is used to indicate the range of visual and light and shadow effects of eliminating the sample object on the visual data of the first sample.

[0079] It should be noted that the explanations of steps S401 to S402 can be found in the relevant descriptions in any embodiment of this application, and will not be repeated here.

[0080] Step S403: The context network in the target elimination model is used to extract the context of the second splicing feature to obtain the first context information; wherein, the second splicing feature is obtained by encoding the third sample visual data to obtain the second latent representation, and splicing the second latent representation and the second mask sequence.

[0081] In this embodiment of the application, visual coding technology (such as VAE encoder) can be used to encode the third sample visual data to obtain a second latent representation. The second latent representation and the second mask information are then concatenated to obtain a second concatenated feature. The second concatenated feature is then input into a context network for context extraction to obtain the first context information.

[0082] Step S404: Input the first context information into the mask prediction network in the target elimination model to perform mask prediction and obtain the predicted mask information.

[0083] There are no restrictions on the network structure of the MaskPred network. For example, the MaskPred network can be a segmentation prediction head composed of a Multilayer Perceptron (MLP).

[0084] Among them, the prediction mask information is used to indicate the range of visual and light and shadow effects of eliminating the sample object on the visual data of the first sample.

[0085] In this embodiment of the application, the first context information output by the context network can be input into the mask prediction network to perform mask prediction and obtain the predicted mask information.

[0086] Step S405: A denoising network is used to predict noise based on the first noise latent representation and the first context information to obtain the first predicted noise; wherein, the first noise latent representation is obtained by encoding the second sample visual data to obtain the third latent representation, and the third latent representation is denoised by the second target noise.

[0087] The second target noise can be different from the first target noise. The second target noise is also obtained by noise extraction using a set method. For example, the second target noise can be extracted using a random sampling noise extraction method.

[0088] In this embodiment of the application, visual coding technology (such as VAE encoder) can be used to encode the second sample visual data to obtain a third latent representation, and the third latent representation can be denoised by using second target noise to obtain a first noisy latent representation. The first noisy latent representation and the first context information are then input into a denoising network for noise prediction to obtain the first predicted noise.

[0089] Step S406: Based on the first difference between the predicted mask information and the light and shadow mask information, and the second difference between the first predicted noise and the second target noise, the context network, the denoising network and the mask prediction network are jointly trained.

[0090] In this embodiment of the application, the context network, the denoising network and the mask prediction network can be jointly trained based on the first difference between the predicted mask information and the light and shadow mask information, and based on the second difference between the first predicted noise and the second target noise.

[0091] In any embodiment of this application, a first loss value for the mask prediction network and the context network can be determined based on a first difference, and a second loss value for the denoising network and the context network can be determined based on a second difference. The first loss value is positively correlated with the first difference, that is, the larger the first difference, the larger the first loss value, and vice versa. Similarly, the second loss value is positively correlated with the second difference. Thus, in this application, the context network, the denoising network, and the mask prediction network can be jointly trained based on the first loss value and the second loss value.

[0092] As an example, a target loss value can be determined based on a first loss value and a second loss value, and the context network, denoising network, and mask prediction network can be jointly trained based on the target loss value.

[0093] For example, the first loss value is labeled as L seg The second loss value is L diff1 The target loss value is L total Then we have: (1) Where, is the set weight coefficient, for example, 0.1 is acceptable.

[0094] As another example, the model parameters in the context network and mask prediction network can be updated using backpropagation based on the first loss value, and the model parameters in the denoising network and context network can be updated using backpropagation based on the second loss value.

[0095] In summary, backpropagation updates based on the first loss value enable the context network to better generate contextual information that helps the mask prediction network accurately predict the mask, while also improving the prediction accuracy of the mask prediction network itself. Backpropagation updates based on the second loss value can prompt the context network to generate inputs that better meet the needs of the denoising network, thereby improving the output accuracy of the denoising network.

[0096] The training method of the target removal model in this application extracts environmental semantics through a context network. On the one hand, this guides the denoising network to more accurately restore the background content (through noise prediction supervision). On the other hand, it drives the mask prediction network to reconstruct the visually influential area when the sample object is present (through comparison with the light and shadow mask information supervision). This not only enables the target removal model to learn a more refined spatial context representation, but also explicitly models the range of influence of the sample object removal on indirect visual effects such as lighting and shadows. By simultaneously minimizing the mask prediction error and the noise prediction error, the target removal model can take into account both geometric structure repair and lighting consistency restoration during training, thereby significantly improving the realism of the target removal results, effectively suppressing problems such as residual artifacts, lighting breaks, or unnatural boundaries, and achieving high-fidelity visual editing effects.

[0097] As one possible implementation method, Figure 5 A flowchart illustrating another method for training a target elimination model provided as an exemplary embodiment of this application.

[0098] It should be noted that the training method of the target elimination model can be executed alone, or it can be executed together with any embodiment of this application or any possible implementation in the embodiment, or it can be executed together with any technical solution in the related technology. The embodiments of this application do not limit this.

[0099] like Figure 5 As shown, in Figure 4 and / or Figure 3 Based on the illustrated embodiment, the target elimination model can be a pre-trained model, wherein the target elimination model can be pre-trained using the following steps S501 to S503: Step S501: Obtain first background visual data where no foreground target is displayed, and motion mask information associated with the foreground target; wherein, the motion mask information is used to indicate the size change information and position change information of the foreground target in the first background visual data.

[0100] The foreground target, also known as the foreground object, refers to the main subject or main point of interest that is closer to the camera than the background, including but not limited to: animals, plants, vehicles, etc.

[0101] The first background visual data shows background targets but not foreground targets. It should be noted that this application does not limit the method of obtaining the first background visual data. For example, the first background visual data can be real visual data taken or recorded by the user, or it can be artificially synthesized visual data, online collected visual data, visual data obtained from the training set, or visual data generated using AIGC technology, etc.

[0102] Among them, the motion mask information associated with the background target is used to indicate the size change information and position change information of the foreground target in the first background visual data.

[0103] As one possible implementation, a random mask generator can be used to generate motion mask information for the foreground target; wherein the size change information indicated by the motion mask information includes fixed size or non-fixed size; wherein the position change information indicated by the motion mask information includes any one of the following: linear position change, random position change, or fixed position.

[0104] For example, taking visual data as video, the motion mask information can also be called a motion mask sequence. Each mask image in the motion mask sequence can be used to indicate the size and position information of the foreground target in the corresponding video frame in the first background video.

[0105] Understandably, by introducing a random mask generator, it is possible to flexibly simulate diverse motion patterns of foreground targets—including scale changes with fixed or non-fixed dimensions, as well as spatial dynamic characteristics such as linear, random, or fixed position changes, thereby generating rich and realistic motion mask information. This mechanism can significantly enhance the diversity and challenge of training data, enabling the target elimination model to fully learn to handle various dynamic occlusions and complex spatiotemporal changes during the pre-training stage. This effectively improves its generalization and robustness to real-world moving targets (such as pedestrians and vehicles) in practical applications, avoids overfitting caused by a single motion pattern, and ultimately achieves a more stable, adaptive, and high-quality target elimination effect.

[0106] Step S502: Using motion masking information, the pixels corresponding to the foreground target in the first background visual data are masked to obtain the second background visual data.

[0107] In this embodiment of the application, motion masking information can be used to mask the pixels in the first background visual data that correspond to the foreground target in order to obtain the second background visual data.

[0108] For example, taking visual data as video, the pixels corresponding to the foreground target in each video frame of the first background video can be determined according to the motion mask information, and the determined pixels can be assigned equal pixel values ​​to obtain the second background video. For example, the pixels corresponding to the foreground target in the video frame can be assigned the value 128 (gray) to obtain the second background video.

[0109] Step S503: Based on the second background visual data and the first background visual data, pre-train the context network and the denoising network in the target elimination model.

[0110] In this embodiment of the application, the context network and denoising network in the target elimination model can be pre-trained based on the second background visual data and the first background visual data.

[0111] As one possible implementation, the pre-training method is as follows: First, based on visual encoding techniques (such as a VAE encoder), the first background visual data is encoded to obtain a fourth latent representation. Then, the fourth latent representation is denoised using third target noise to obtain a second noise latent representation. Next, the second background visual data is encoded using visual encoding techniques to obtain a fifth latent representation. Finally, the fifth latent representation is concatenated with motion mask information to obtain a third concatenated feature. Then, the third concatenated feature is input into a context network for context extraction to obtain second context information. Then, the second context information and the second noise latent representation are input into a denoising network for noise prediction to obtain second predicted noise. Thus, in this application, the denoising network and context network in the target elimination model can be pre-trained based on the third difference between the second predicted noise and the third target noise.

[0112] The third target noise can be different from the first target noise and the second target noise. The third target noise is also obtained by noise extraction using a set method. For example, the third target noise can be extracted by random sampling noise extraction.

[0113] As an example, a third loss value can be determined based on the third difference, where the third loss value is positively correlated with the third difference. Therefore, in this application, the denoising network and the context network can be pre-trained based on the third loss value. Taking DiT as the denoising network and standard Gaussian noise as the third target noise as an example, the third loss value can be calculated using the following formula: (2) Where x represents the original background visual data (first background video data). The noise is standard Gaussian noise, and t is the diffusion time step. The predicted noise output by DiT. Let be the potential noise representation at the t-th diffusion time step.

[0114] The target elimination model training method of this application embodiment can effectively reduce the dependence of the target elimination model on paired visual data (such as first visual data containing the sample object to be eliminated and the corresponding second visual data of the sample object that has been eliminated) during the formal training stage by using background visual data to pre-train the target elimination model. This alleviates the training bottleneck caused by the scarcity of high-quality paired visual data, thereby significantly reducing the training difficulty of the target elimination model and improving convergence stability and generalization ability.

[0115] In any embodiment of this application, taking video as the visual data and DiT as the denoising network as an example, the structure of the target elimination model can be as follows: Figure 6As shown, the model structure consists of three parts: DiT (Denoising Transformation), a side network—context net—responsible for extracting contextual information and injecting it as conditional features into DiT, and MaskPred (e.g., composed of multiple MLP layers) responsible for predicting the mask of the target to be eliminated and its related effects. The Context net can be composed of copies of some blocks from DiT. The model input consists of three parts: noise latent (noise latent representation, or noisy latent representation), mask video latent (mask video latent representation), and mask (mask sequence). The noise latent is obtained by adding noise to the original video after passing it through a 3D VAE encoder. The mask video latent is obtained by filling the object region of the target to be eliminated in the original video with gray (pixel value 128) and then passing it through a 3D VAE encoder. The mask is the mask sequence corresponding to the target to be eliminated, interpolated to maintain the latent scale. The mask video latent and the mask are concatenated and input into the context net for context extraction, while the noise latent is input into DiT for noise prediction. MaskPred predicts the mask sequence of the target to be eliminated and its associated effects, and is explicitly strongly supervised during training on pairwise data.

[0116] For example, the training process is divided into the following two stages: 1. Phase One: Background Restoration Warm-up of Unpaired Data in Real-World Scenarios Video generation models typically use unbiased data for large-scale pre-training, and the generated content often favors foreground objects such as tasks, animals, and vehicles, which conflicts with the target elimination task. Therefore, in Phase 1, background videos will be used for warm-up, shifting the restoration preference to reasonably restore the background rather than the foreground. First, in this Phase 1, background videos without foreground objects can be selected from an open-source real video set as training data through a designed pipeline, and a random mask generator ( Figure 7 The A module simulates the motion of foreground targets in real-world scenarios, thereby eliminating the dependence on segmentation and effectively utilizing real data for pre-training.

[0117] The noise prediction learning paradigm employs a standard diffusion model, with the training objective of recovering the original clean video content corresponding to the occluded region given a mask. Specifically, the target removal model uses the noisy video latent representation (i.e., the noise latent representation z) as the training vector. t The noise is predicted by taking the predicted noise and the corresponding mask sequence m as input. The optimization objective is to minimize the L2 reconstruction error between the predicted noise and the actual added noise, and the loss value can be expressed as shown in formula (2).

[0118] 2. Phase Two: Training of Elimination Ability under Simulated Paired Data Supervision To effectively improve the target removal capabilities of the target removal model in complex scenes, especially its ability to simultaneously remove visual side effects such as shadows, reflections, and self-illuminating halos, this application employs high-quality simulated paired data for supervised training in stage two. The simulated paired data includes triplet-style annotations: the original video V (containing the target to be removed and its environmental interactions), and the edited video... (A clean video after the target to be removed is generated), and the corresponding mask sequence M. All samples are generated based on physically realistic lighting models, which can accurately simulate the global impact of the target to be removed on the scene, such as dynamic shadow casting, specular distortion, and light source attenuation, and have high semantic integrity and physical consistency. At the same time, by comparing the videos before and after editing, an "effect-aware mask" sequence can be obtained to construct the influence domain of the target to be removed, which is used to explicitly supervise the model's repair behavior in the side effect region.

[0119] The training objective of Phase Two is to further enhance the model's target removal capability based on the background prior established in Phase One, and to enable it to simultaneously identify and repair various visual accompanying effects related to the target to be removed. Specifically, given a video and a corresponding mask sequence, the target removal model needs to predict the clean video after the target to be removed is removed, optimizing the standard noise prediction loss (diffusion loss) of the target-to-diffusion model, i.e., minimizing the residual between the predicted noise output by the model and the actual added noise. This process relies on paired supervision signals, enabling the model to learn an end-to-end mapping from "observation containing the target to be removed" to "ideal removal result," significantly improving the visual consistency and physical plausibility of the removal result. Simultaneously, by supervising the mask sequence predicted by MaskPred, the model is guided to focus on the affected region corresponding to the target to be removed for simultaneous removal.

[0120] Therefore, the loss in stage two is divided into two parts: one is the diffusion loss (referred to as the second loss value in this application), which is the same as in stage one, and the other is the segmentation prediction loss (referred to as the first loss value in this application). The segmentation prediction loss can be calculated using binary cross-entropy loss (BCE) on its output prediction mask sequence M. pred With the downsampled light and shadow mask sequence D down To conduct oversight.

[0121] For example, the application process (or reasoning process) is as follows: The user uploads the input video of the target to be removed and the corresponding mask sequence. The object region in the input video is filled with gray using the mask sequence and then encoded to obtain the maskvideo latent. The latent ...

[0122] In summary, the solution provided in this application has at least the following advantages: through two-stage training and network structure design, the model's elimination ability is significantly improved, including the ability to eliminate corresponding light and shadow; at the same time, the model's background recovery ability is improved, reducing the occurrence of artifacts and blurring; the two-stage training reduces the dependence on a large amount of paired data, and the use of real background videos for pre-training of elimination ability can reduce the problem of discrepancies between the model trained on subsequent simulated paired data and the real world.

[0123] To achieve the above embodiments, this application also proposes a target elimination device.

[0124] Figure 7 This is a schematic diagram of the structure of a target elimination device provided for an exemplary embodiment of this application.

[0125] like Figure 7 As shown, the target elimination device 700 may include: an encoding module 710, a splicing module 720, an extraction module 730, and a noise reduction module 740.

[0126] The encoding module 710 is used to encode the masked visual data to obtain a first latent representation; wherein the masked visual data is obtained by masking the target object to be eliminated in the input visual data based on the first mask information, and the first mask information is used to indicate the position of the target object in the input visual data. The splicing module 720 is used to splice the first latent representation with the first mask information to obtain the first splicing feature; The extraction module 730 is used to extract the context of the first concatenated features to obtain the target context information; The denoising module 740 is used to perform denoising processing based on the target context information to generate target visual data after the target object is eliminated.

[0127] In one implementation of this application, the denoising module 740 is used to: perform denoising processing based on target context information and first target noise to obtain a target latent representation; and decode the target latent representation to obtain target visual data after the target object is eliminated.

[0128] In one implementation of this application, the extraction module 730 is used to: input the first concatenation feature into the context network of the trained target elimination model to extract the context and obtain target context information; The denoising module 740 is used to: input the target context information and the first target noise into the denoising network in the target elimination model for denoising processing to obtain the target potential representation.

[0129] In one implementation of this application, the target elimination model is trained using the following modules: The first acquisition module is used to acquire first sample visual data, second sample visual data, third sample visual data and second mask information, which display sample objects; wherein, the second sample visual data is obtained by removing the sample objects in the first sample visual data, the second mask information is used to indicate the position of the sample objects in the first sample visual data, and the third sample visual data is obtained by masking the sample objects in the first sample visual data based on the second mask information. The generation module is used to generate light and shadow mask information based on the difference between the first sample visual data and the second sample visual data; wherein, the light and shadow mask information is used to indicate the range of visual influence and light and shadow influence of the sample object on the first sample visual data to be eliminated; The training module is used to jointly train the context network and the denoising network based on the second mask information, the light and shadow mask information, the second sample visual data, and the third sample visual data.

[0130] In one implementation of this application, the target elimination model further includes a mask prediction network and a training module, configured to: extract context from the second splicing features using a context network to obtain first context information; wherein the second splicing features are obtained by encoding third sample visual data to obtain a second latent representation, and splicing the second latent representation with the second mask information; input the first context information into the mask prediction network for mask prediction to obtain predicted mask information; use a denoising network to predict noise based on the first noise latent representation and the first context information to obtain first predicted noise; wherein the first noise latent representation is obtained by encoding second sample visual data to obtain a third latent representation, and adding noise to the third latent representation using second target noise; and jointly train the context network, the denoising network, and the mask prediction network based on the first difference between the predicted mask information and the light and shadow mask information, and the second difference between the first predicted noise and the second target noise.

[0131] In one implementation of this application, the training module is configured to: determine a first loss value for the mask prediction network and the context network based on a first difference; determine a second loss value for the denoising network and the context network based on a second difference; and jointly train the context network, the denoising network, and the mask prediction network based on the first loss value and the second loss value.

[0132] In one implementation of this application, the training module is used to: update the model parameters in the context network and the mask prediction network using a backpropagation mechanism based on a first loss value; and update the model parameters in the denoising network and the context network using a backpropagation mechanism based on a second loss value.

[0133] In one implementation of this application, the target elimination model is a pre-trained model, which is pre-trained using the following modules: The second acquisition module is used to acquire first background visual data in which no foreground target is displayed, and motion mask information associated with the foreground target; wherein, the motion mask information is used to indicate the size change information and position change information of the foreground target in the first background visual data; The masking module is used to mask the pixels corresponding to the foreground target in the first background visual data using motion masking information to obtain the second background visual data. The pre-training module is used to pre-train the context network and denoising network in the target elimination model based on the second background visual data and the first background visual data.

[0134] In one implementation of this application, the pre-training module is configured to: encode the first background visual data to obtain a fourth latent representation, and add noise to the fourth latent representation using third target noise to obtain a second noisy latent representation; encode the second background visual data to obtain a fifth latent representation, and concatenate the fifth latent representation with motion mask information to obtain a third concatenated feature; extract the context of the third concatenated feature using a context network to obtain second context information; input the second context information and the second noisy latent representation into a denoising network for noise prediction to obtain second predicted noise; and pre-train the denoising network and the context network based on a third difference between the second predicted noise and the third target noise.

[0135] In one implementation of this application, the method for obtaining motion mask information includes: generating motion mask information of the foreground target using a random mask generator; wherein the size change information indicated by the motion mask information includes fixed size or non-fixed size; wherein the position change information indicated by the motion mask information includes any one of the following: linear position change, random position change, or fixed position.

[0136] It should be noted that the explanations and descriptions of the embodiments corresponding to the training methods of any target elimination method or target elimination model described above also apply to the target elimination device of this embodiment, and will not be repeated here.

[0137] In the target elimination device of this application embodiment, a first latent representation is obtained by encoding the mask visual data, so that the object region corresponding to the target object to be eliminated is explicitly "excavated" in the latent space, thereby providing an initial representation for background reconstruction. The first latent representation and the first mask information (i.e., the spatial position indication of the target object) are explicitly spliced ​​together, so that the positional prior of the missing region and the semantic features of the surrounding visible content can be preserved in the latent space at the same time. The spliced ​​features are extracted into context to generate target context information that can reflect the surrounding structure, texture and motion law of the target object, and this is used as the conditional input of the denoising process, so that the denoising process can generate repair content that is coordinated with the surrounding environment in terms of structure, texture and dynamic characteristics while preserving spatiotemporal consistency. Since context modeling and denoising generation are designed as two orderly and collaborative stages, the guiding role of spatial context can be fully explored, and semantic confusion under noise interference can be avoided. In the end, typical artifacts such as boundary blurring, structural breakage and inter-frame flicker are effectively suppressed. While maintaining temporal continuity and visual naturalness, the integrity of target elimination and the realism of background reconstruction are significantly improved.

[0138] To implement the above embodiments, this application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the target elimination method or the target elimination model training method as described in any of the foregoing embodiments.

[0139] Figure 8 This is a schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this application. For example, the electronic device 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0140] Reference Figure 8 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0141] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0142] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0143] Power component 806 provides power to various components of electronic device 800. Power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0144] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0145] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0146] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0147] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in the position of a component of electronic device 800, the presence or absence of user contact with electronic device 800, the orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0148] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), Bluetooth, and other technologies.

[0149] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0150] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0151] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the target elimination method or the target elimination model training method as described in any of the foregoing method embodiments.

[0152] To implement the above embodiments, this application also proposes a computer program product having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the target elimination method or the target elimination model training method as described in any of the foregoing method embodiments.

[0153] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0154] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0155] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0156] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and compact disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0157] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0158] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0159] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0160] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A target elimination method characterized by comprising: include: The masked visual data is encoded to obtain a first latent representation; wherein the masked visual data is obtained by masking the target object to be eliminated in the input visual data based on the first mask information, and the first mask information is used to indicate the position of the target object in the input visual data; The first latent representation is concatenated with the first mask information to obtain a first concatenated feature, and the first concatenated feature is subjected to context extraction to obtain target context information. Denoising is performed based on the target context information to generate target visual data after the target object is eliminated.

2. The method of claim 1, wherein, The step of performing denoising processing based on the target context information to generate target visual data after the target object is eliminated includes: Denoising is performed based on the target context information and the first target noise to obtain the target potential representation; The latent representation of the target is decoded to obtain the target visual data after the target object is eliminated.

3. The method according to claim 2, characterized in that, The step of extracting the context of the first spliced ​​feature to obtain the target context information includes: The first concatenated feature is input into the context network of the trained target elimination model to extract the context information of the target. The step of performing denoising processing based on the target context information and the first target noise to obtain the target latent representation includes: The target context information and the first target noise are input into the denoising network in the target elimination model for denoising processing to obtain the target latent representation.

4. The method according to claim 3, characterized in that, The training methods for the target elimination model include: Acquire first sample visual data, second sample visual data, third sample visual data, and second mask information displaying sample objects; wherein, the second sample visual data is obtained by removing the sample objects in the first sample visual data, the second mask information is used to indicate the position of the sample objects in the first sample visual data, and the third sample visual data is obtained by masking the sample objects in the first sample visual data based on the second mask information. Based on the difference between the first sample visual data and the second sample visual data, a light and shadow mask information is generated; wherein, the light and shadow mask information is used to indicate the range of visual and light and shadow influence of the sample object on the first sample visual data to be eliminated; The context network and the denoising network are jointly trained based on the second mask information, the light and shadow mask information, the second sample visual data, and the third sample visual data.

5. The method according to claim 4, characterized in that, The target elimination model further includes a mask prediction network. The joint training of the context network and the denoising network based on the second mask information, the light and shadow mask information, the second sample visual data, and the third sample visual data includes: The context network is used to extract the context of the second splicing feature to obtain the first context information; wherein, the second splicing feature is obtained by encoding the third sample visual data to obtain the second latent representation, and then splicing the second latent representation and the second mask information. The first context information is input into the mask prediction network to perform mask prediction and obtain the predicted mask information. The denoising network is used to predict noise based on the first noise latent representation and the first context information to obtain the first predicted noise; wherein the first noise latent representation is obtained by encoding the second sample visual data to obtain the third latent representation, and then adding noise to the third latent representation using the second target noise. Based on the first difference between the predicted mask information and the light and shadow mask information, and the second difference between the first predicted noise and the second target noise, the context network, the denoising network, and the mask prediction network are jointly trained.

6. The method according to claim 5, characterized in that, The step of jointly training the context network, the denoising network, and the mask prediction network based on the first difference between the predicted mask information and the light and shadow mask information, and the second difference between the first predicted noise and the second target noise, includes: Based on the first difference, a first loss value is determined for the mask prediction network and the context network; Based on the second difference, a second loss value is determined for the denoising network and the context network; The context network, the denoising network, and the mask prediction network are jointly trained based on the first loss value and the second loss value.

7. The method according to claim 6, characterized in that, The step of jointly training the context network, the denoising network, and the mask prediction network based on the first loss value and the second loss value includes: Based on the first loss value, the model parameters in the context network and the mask prediction network are updated using a backpropagation mechanism; Based on the second loss value, the model parameters in the denoising network and the context network are updated using a backpropagation mechanism.

8. The method according to any one of claims 4-7, characterized in that, The target elimination model is a pre-trained model, and the pre-training method of the target elimination model includes: Acquire first background visual data in which no foreground target is displayed, and motion mask information associated with the foreground target; wherein the motion mask information is used to indicate the size change information and position change information of the foreground target in the first background visual data; Using the motion mask information, the pixels in the first background visual data corresponding to the foreground target are masked to obtain the second background visual data; Based on the second background visual data and the first background visual data, the context network and denoising network in the target elimination model are pre-trained.

9. The method according to claim 8, characterized in that, The step of pre-training the context network and denoising network in the target elimination model based on the second background visual data and the first background visual data includes: The first background visual data is encoded to obtain a fourth latent representation, and the fourth latent representation is noise-added with third target noise to obtain a second noisy latent representation; The second background visual data is encoded to obtain a fifth latent representation, and the fifth latent representation is concatenated with the motion mask information to obtain a third concatenated feature; The context network is used to extract the context of the third concatenation feature to obtain the second context information; The second context information and the second noise latent representation are input into the denoising network to perform noise prediction, thereby obtaining the second predicted noise. The denoising network and the context network are pre-trained based on the third difference between the second predicted noise and the third target noise.

10. The method according to claim 8, characterized in that, The methods for obtaining the motion mask information include: The motion mask information of the foreground target is generated using a random mask generator; The size change information indicated by the motion mask information includes fixed size or non-fixed size; The position change information indicated by the motion mask information includes any one of the following: linear position change, random position change, or fixed position.

11. A target elimination device, characterized in that, include: An encoding module is used to encode masked visual data to obtain a first latent representation; wherein the masked visual data is obtained by masking the target object to be eliminated in the input visual data based on the first mask information, and the first mask information is used to indicate the position of the target object in the input visual data; The splicing module is used to splice the first latent representation with the first mask information to obtain the first splicing feature; The extraction module is used to extract the context of the first spliced ​​features to obtain the target context information; The denoising module is used to perform denoising processing based on the target context information to generate target visual data after the target object is eliminated.

12. The apparatus according to claim 11, characterized in that, The noise reduction module is used for: Denoising is performed based on the target context information and the first target noise to obtain the target potential representation; The latent representation of the target is decoded to obtain the target visual data after the target object is eliminated.

13. The apparatus according to claim 12, characterized in that, The extraction module is used for: The first concatenated feature is input into the context network of the trained target elimination model to extract the context information of the target. The noise reduction module is used for: The target context information and the first target noise are input into the denoising network in the target elimination model for denoising processing to obtain the target latent representation.

14. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 10.

15. A non-transitory computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method according to any one of claims 1 to 10.

16. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images