Image processing method and device, model training method and device and electronic equipment

Abstract

The invention discloses an image processing method and device, a model training method and device and electronic equipment, and belongs to the technical field of electronic equipment, the image processing method comprises the steps that a first image and reference information corresponding to the first image are acquired, and the reference information comprises at least one of a depth image and environment noise; according to the first image and the reference information, performing segmentation processing on the first image to obtain a Mask image; and based on the Mask image, performing content elimination processing on the first image to obtain a second image.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to an image processing method, a model training method, a device and an electronic device. Background Art

[0002] With the rapid development of electronic device photography technology, users are increasingly demanding image editing capabilities. In particular, the ability to remove distracting elements (such as pedestrians and debris) from images has become a critical requirement for image processing in intelligent electronic devices. Currently, removing distracting elements from images typically involves two core steps: mask segmentation (also known as object segmentation) and content removal. Mask segmentation generates a precise mask to identify the removal area in the image where the distracting elements are to be removed. Content removal, based on the mask generated by mask segmentation, removes the distracting elements from the removal area and generates a natural and reasonable background fill.

[0003] In related technologies, mask segmentation is implemented based on a single image modality (such as RGB image). Although this mask segmentation method can achieve image elimination to a certain extent, it also has obvious defects: for example, when the image has insufficient resolution, motion blur, or complex occlusion, the segmentation accuracy will drop sharply, resulting in blurred edges or misjudgment of the generated mask area, which in turn affects the accuracy of subsequent image content elimination. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, a model training method, an apparatus and an electronic device that can improve the accuracy and robustness of image content elimination.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Acquire a first image and reference information corresponding to the first image, wherein the reference information includes at least one of the following: a depth image and environmental noise;

[0007] Segmenting the first image to obtain a mask image according to the first image and the reference information;

[0008] Based on the Mask image, content removal processing is performed on the first image to obtain a second image.

[0009] In a second aspect, an embodiment of the present application provides a model training method, the method comprising:

[0010] Perform panoptic segmentation on the foreground sample image, extract foreground content with clear semantic categories, and generate the corresponding foreground mask image; different semantic categories use different mask encoding methods;

[0011] Perform panorama segmentation on the background sample image, extract the background content with complex semantics, and generate the corresponding background mask image; all background sample images use the same mask encoding method;

[0012] Build a training model including an image encoder, a text feature encoding network, a backbone network corresponding to the image restoration network, and a denoiser;

[0013] Alternately training the model to be trained using foreground training data and background training data, updating the parameters of the backbone network until convergence, and obtaining an elimination model and its prompt word;

[0014] The foreground training data includes: the foreground sample image, the foreground mask image, the foreground masked image, and randomly initialized foreground prompt words; the foreground masked image is an image obtained by performing a pixel-by-pixel product operation at the same pixel position of the foreground sample image and the foreground mask image;

[0015] The background training data includes: the background sample image, the background mask image, the background masked image and randomly initialized background prompt words; the background masked image is an image obtained by performing a pixel-by-pixel product operation on the same pixel position of the background sample image and the background mask image;

[0016] The prompt words of the elimination model are composed of foreground prompt words and background prompt words at the time of convergence.

[0017] In a third aspect, an embodiment of the present application provides an image processing device, comprising:

[0018] an acquisition module, configured to acquire a first image and reference information corresponding to the first image, wherein the reference information includes at least one of the following: a depth image and environmental noise;

[0019] a segmentation module, configured to segment the first image according to the first image and the reference information to obtain a mask image;

[0020] The elimination module is used to perform content elimination processing on the first image based on the mask image to obtain a second image.

[0021] In a fourth aspect, an embodiment of the present application provides a model training device, comprising:

[0022] The first generation module is used to perform panorama segmentation on the foreground sample image, extract foreground content with clear semantic categories, and generate corresponding foreground mask images; different semantic categories use different mask encoding methods;

[0023] The second generation module is used to perform panoramic segmentation on the background sample image, extract the background content with complex semantics, and generate the corresponding background mask image; wherein all background samples use the same mask encoding method;

[0024] A construction module is used to construct a model to be trained, including an image encoder, a text feature encoding network, a backbone network corresponding to the image restoration network, and a denoiser;

[0025] A training module is used to alternately train the model to be trained using foreground training data and background training data, update the parameters of the backbone network until convergence, and obtain an elimination model and its prompt words; wherein the foreground training data includes: the foreground sample image, the foreground mask image, the foreground masked image and randomly initialized foreground prompt words; the foreground masked image is an image obtained by performing a pixel-by-pixel product operation on the foreground sample image and the foreground mask image at the same pixel position; the background training data includes: the background sample image, the background mask image, the background masked image and randomly initialized background prompt words; the background masked image is an image obtained by performing a pixel-by-pixel product operation on the background sample image and the background mask image at the same pixel position; the prompt words of the elimination model are composed of the foreground prompt words and the background prompt words at convergence.

[0026] In a fifth aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the image processing method described in the first aspect are implemented, or the steps of the model training method described in the second aspect are implemented.

[0027] In a sixth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the image processing method described in the first aspect are implemented, or the steps of the model training method described in the second aspect are implemented.

[0028] In the seventh aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the image processing method described in the first aspect, or to implement the steps of the model training method described in the second aspect.

[0029] In an eighth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the image processing method described in the first aspect, or to implement the steps of the model training method described in the second aspect.

[0030] In an embodiment of the present application, a first image and reference information corresponding to the first image are obtained, wherein the reference information includes at least one of the following: a depth image and environmental noise; based on the first image and the reference information, the first image is segmented to obtain a mask image; based on the mask image, the first image is subjected to content elimination processing to obtain a second image. It can be seen that, unlike the related art in which mask segmentation is performed based only on image data of a single modality, in an embodiment of the present application, in addition to image data, mask segmentation can also be performed with reference to depth data and / or environmental noise data. Since depth data can provide the relative distance or absolute depth between objects, clarify the spatial hierarchy (such as foreground / background separation), and identify occlusion relationships, and environmental noise data can identify the scene in which the image is located, scene recognition helps to identify objects, backgrounds or semantic areas (such as the sky, roads, faces, etc.) in the image. Therefore, for images with insufficient resolution, motion blur or complex occlusion, the multimodal mask segmentation method can accurately lock the content to be eliminated, thereby improving the accuracy and robustness of image content elimination.

[0031] In the embodiment of the present application, a panorama segmentation is performed on the foreground sample image, and the foreground content with a clear semantic category is extracted to generate a corresponding foreground Mask image; wherein different semantic categories use different Mask encoding methods; a panorama segmentation is performed on the background sample image, and the background content with complex semantics is extracted to generate a corresponding background Mask image; wherein all background sample images use the same Mask encoding method; a training model including an image encoder, a text feature encoding network, a backbone network corresponding to the image restoration network, and a denoiser is constructed; the training model is alternately trained using foreground training data and background training data, and the parameters of the backbone network are updated until convergence, to obtain an elimination model and Its prompt words; wherein, the foreground training data includes: foreground sample images, foreground mask images, foreground masked images and randomly initialized foreground prompt words; the foreground masked image is an image obtained by performing a pixel-by-pixel product operation on the foreground sample image and the foreground mask image at the same pixel position; the background training data includes: background sample images, background mask images, background masked images and randomly initialized background prompt words; the background masked image is an image obtained by performing a pixel-by-pixel product operation on the background sample image and the background mask image at the same pixel position; the prompt words of the elimination model are composed of the foreground prompt words and the background prompt words at the time of convergence. The above-mentioned training of the elimination model by alternating the use of foreground training data and background training data can significantly improve the robustness and generation quality of the elimination model, thereby improving the accuracy and robustness of image content elimination. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is an example diagram of the working principle of the diffusion model in the related art;

[0033] Figure 2 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0034] Figure 3 This is a flowchart of an implementation of step 203 provided in an embodiment of the present application;

[0035] Figure 4 This is a diagram illustrating an example of the structure of an elimination model provided in an embodiment of the present application;

[0036] Figure 5 This is a flowchart of an implementation of step 301 provided in an embodiment of the present application;

[0037] Figure 6 This is an example diagram of the reasoning process of the elimination model provided in the embodiment of the present application;

[0038] Figure 7 This is a flow chart of a model training method provided in an embodiment of the present application;

[0039] Figure 8 This is a flowchart of an implementation of step 704 provided in an embodiment of the present application;

[0040] Figure 9 This is an example diagram of the training process of the elimination model provided in an embodiment of the present application;

[0041] Figure 10 This is a structural block diagram of an image processing device provided by an embodiment of the present application;

[0042] Figure 11 This is a structural block diagram of a model training device provided in an embodiment of the present application;

[0043] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0044] Figure 13 It is a schematic diagram of the hardware structure of an electronic device that implements various embodiments of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0046] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein. Furthermore, the objects distinguished by "first," "second," and the like generally refer to a class of objects and do not limit the number of objects. For example, the first object may be one or more. Furthermore, the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0047] To facilitate understanding, some concepts and application scenarios involved in the embodiments of this application are first introduced.

[0048] 1. Concept

[0049] Mask segmentation is a key task in computer vision and a subdivision of image segmentation. Its core goal is to generate accurate pixel-level masks for target instances in an image, thereby distinguishing the boundaries and shapes of different instances. Unlike semantic segmentation, which only distinguishes between different categories, mask segmentation can distinguish between different individuals of the same category. For example, in an image containing multiple people, a separate mask can be generated for each person.

[0050] A mask is a binary or probability matrix in digital image processing that is used to accurately specify pixel areas in an image that need to be processed, retained, or ignored. Its core function is to guide algorithms to operate on specific areas through pixel-level labeling.

[0051] A mask image is a binary or probability matrix used to identify specific areas in an image. Its core function is to guide the processing scope of computer vision algorithms through pixel-level annotation.

[0052] A masked image is an image obtained by performing a pixel-by-pixel product operation on the original image and the mask image at the same pixel position. Its core purpose is to highlight or isolate the target area while retaining or suppressing the background information.

[0053] A depth image is a special two-dimensional image in which each pixel value represents the physical distance from the corresponding scene point to the camera (rather than color information).

[0054] The Cross-Modal Interaction Module (CMI-Module) is a neural network component used to achieve deep semantic fusion between multimodal data (such as text, images, audio, etc.). Its core goal is to establish alignment relationships between modalities and extract complementary information.

[0055] Multi-Head Attention (MHA), a core component of the Transformer architecture, captures different types of dependencies in the data by learning multiple sets of attention patterns in parallel.

[0056] The Depth-Guided U-Net is an improved U-Net architecture that integrates depth information. By introducing depth images as additional guiding conditions, it significantly improves the segmentation task's ability to model geometric structures and occlusion relationships.

[0057] Diffusion Probabilistic Models (DPMs) are a class of generative models that define a Markov chain of diffusion steps to slowly add random noise to the data, and then learn a reverse diffusion process to gradually transform the Gaussian noise distribution into the target data. Figure 1 As shown, the application of the diffusion model is usually divided into two stages: the forward diffusion process (indicated by the dotted line) and the reverse denoising process (indicated by the solid line). The input image is x0. The forward diffusion process of the diffusion model is performed by the forward denoising algorithm q(x t |x t-1 ), add Gaussian noise to x0 at each time step {1,2,…,t-1,t,…,T}, and obtain noisy data (referred to as “noisy data”) x1,x2,…,x t-1 ,x t ,…,x T The reverse denoising process of the diffusion model is to denoise x at each time step {T, T-1, T-2, …, 2, 1} by the denoiser. T Perform denoising to obtain the final output image.

[0058] The denoiser, also known as a denoising autoencoder, is a key component of the diffusion model. It takes noisy data as input and, through training, predicts the original, noise-free data as output. This process is similar to the inverse of the diffusion model, which involves recovering the original data from noised data. This process, through learning to recover the original data from noisy data, is one of the key mechanisms by which the diffusion model successfully generates clear images from noise.

[0059] A variational autoencoder (VAE) is a deep generative model that combines techniques from autoencoders and variational inference. VAEs learn the latent space distribution of input data and are able to generate new samples similar to the input data. It works primarily through two components: an encoder and a decoder. The encoder converts the input data into a latent form, while the decoder attempts to reconstruct the original data based on this latent representation. The advantage of this model is that it can generate new data samples similar to the training data, and because the latent space is continuous, the decoder can generate new data points that seamlessly interpolate between training data points.

[0060] An image encoder is a deep learning model used for image processing. Its primary purpose is to compress image data into a low-dimensional code while preserving the image's key features. Its working principle involves multiple layers, including convolutional layers, pooling layers, and fully connected layers. Convolutional layers extract features by sliding convolution kernels over the input data, pooling layers reduce computational effort by downsampling feature maps, and fully connected layers map the extracted features to the final output categories. This model is characterized by its hierarchical structure, local perception, and weight sharing, making it particularly well-suited for processing image data. Image encoders can be applied to tasks such as data dimensionality reduction and feature extraction. Through training, the reconstructed data is made as close as possible to the original input data, thereby learning to represent the important features of the input data.

[0061] An image decoder is a deep learning model used for image processing. Its main purpose is to restore encoded image data to its original image format or displayable pixel data. Its core function is to reverse the image encoding process and restore the original visual information of the image so that it can be presented on a display, printer or other output device.

[0062] The text feature encoding network is a neural network architecture that converts raw text data into numerical feature representations that can be understood and processed by machine learning models. Its core function is to extract text semantic information and generate structured feature vectors through specific algorithms.

[0063] EmbeddingNet is a neural network architecture that maps discrete or high-dimensional data (such as text, images, user behavior, etc.) to a low-dimensional continuous vector space. Its core goal is to capture the semantic or structural relationships between data and convert raw data into numerical feature representations for efficient processing by machine learning models.

[0064] A text encoder is a tool that converts text into numerical representations for use in machine learning models. Its primary function involves breaking text into subwords and converting each subword into numerical representations so that machine learning models can understand and process the text. Text encoders play an important role in machine learning. For example, in diffusion models, text encoders are used to convert text into embeddings, which are then used to control image generation.

[0065] BrushNet is a plug-and-play dual-branch image inpainting network based on deep learning technology, primarily used for local image repainting and restoration. Using a "decomposed dual-branch diffusion" technique, BrushNet embeds pixel-level mask image features into a pre-trained diffusion model, generating consistent and high-quality restoration results.

[0066] Panoptic segmentation is a computer vision task that aims to simultaneously perform semantic classification and instance discrimination on each pixel in an image, dividing the image into coherent regions with semantic labels and distinguishing between different individual instances of the same category. The core goal of panoptic segmentation is to generate a complete panoptic segmentation map, in which each pixel belongs to both a semantic category (such as person, car, sky) and a unique instance identifier (such as "person 1" and "car 2").

[0067] Prompts are the source of content for images generated by the diffusion model. In the diffusion model, prompts play a vital role. They are not only the starting point or guide for the model to generate images, but also the key for the model to understand the input intent. By providing a clear and specific prompt, the user can guide the model to generate images that meet expectations. This prompt can be a question, a text description, a conversation, or any form of text input. The model will generate corresponding outputs based on the provided context and semantic information. For example, for a language model, a prompt may be "The cat sat on the", and the model will predict the next word based on this prompt and generate different outputs such as "mat", "chair", "sofa", etc.

[0068] 2. Application Scenarios

[0069] With the rapid development of electronic device photography technology, the ability to remove distracting elements (such as pedestrians and debris) from images has become a critical requirement for image processing in intelligent electronic devices. Currently, removing distracting elements from images typically involves two core steps: mask segmentation and content removal. Mask segmentation generates a precise mask to identify the removal area in the image where the distracting elements are to be removed. Content removal uses the mask generated by mask segmentation to remove the distracting elements from the removal area and generate a natural and reasonable background fill.

[0070] In the mask segmentation stage, related technologies use semantic segmentation models based on deep learning (such as U-Net and MaskR-CNN), which rely solely on RGB images for target detection. This single-modal processing method has significant drawbacks: when the image has insufficient resolution, motion blur, or complex occlusion, the segmentation accuracy will drop sharply, resulting in blurred edges or misjudgment of the generated mask area, which in turn affects the accuracy of subsequent content removal. To solve the above technical problems, the embodiments of the present application provide an image processing method, a model training method, an apparatus, and an electronic device.

[0071] It should be noted that the image processing method and model training method provided in the embodiments of the present application are applicable to electronic devices. In practical applications, the electronic devices include but are not limited to: smart phones, tablet computers, personal digital assistants and other mobile terminals, and the embodiments of the present application do not limit this.

[0072] Figure 2 This is a flow chart of an image processing method provided by an embodiment of the present application. Figure 2 As shown, the method may include the following steps: step 201, step 202 and step 203.

[0073] In step 201, a first image and reference information corresponding to the first image are obtained, wherein the reference information includes at least one of the following: a depth image and environmental noise.

[0074] In the embodiment of the present application, the first image is an image that needs to be eliminated.

[0075] On the one hand, considering that the relevant technologies only use image information and do not use the surrounding environmental noise to identify cluttered scenes (that is, the environmental noise information is not modeled), and the environmental noise can often reflect the state of the surrounding environment of the photo, it can effectively help identify the scene in which the image is located, such as urban streets, indoor environments, outdoor scenes, etc. Scene recognition can quickly identify objects, backgrounds or semantic areas (such as the sky, roads, faces, etc.) in the image, thereby accurately locking the content area that needs to be eliminated.

[0076] On the other hand, considering that the relevant technologies only use image information without combining depth images and lack spatial information, they cannot distinguish between real occlusions (such as railings behind pedestrians) and objects of similar color / texture (such as overlapping objects, shadow interference, etc.), resulting in insufficient segmentation mask accuracy. The depth information in the depth image can provide the relative distance or absolute depth between objects, clarify the spatial hierarchy (such as foreground / background separation), and combine the image depth information to use the physical geometric characteristics of parallax to locate the real occlusion boundaries and texture differences of objects, thereby accurately locking the content area that needs to be eliminated.

[0077] In the embodiment of the present application, based on the above two considerations, when performing the image removal task, in addition to image information, depth information and / or environmental noise may also be referred to.

[0078] In an embodiment of the present application, a first image can be acquired by a camera of an electronic device, wherein the first image is typically an image in RGB format (hereinafter referred to as an RGB image). A depth image corresponding to the first image can be acquired by a binocular vision system or a monocular vision system of the electronic device. For example, a binocular vision system can typically provide depth information with millimeter-level accuracy. A microphone of the electronic device can be used to collect ambient noise corresponding to the first image.

[0079] In step 202, based on the first image and reference information, the first image is segmented to obtain a mask image.

[0080] In the embodiment of the present application, the Mask image of the first image is used to indicate the elimination area in the first image.

[0081] In an embodiment of the present application, in the process of performing Mask segmentation on the first image, multimodal information such as image information, depth information and / or environmental noise is integrated. This integration method improves the accuracy, robustness and adaptability of Mask segmentation in complex environments.

[0082] In step 203, based on the mask image, content removal processing is performed on the first image to obtain a second image.

[0083] In an embodiment of the present application, the elimination area in the first image can be determined based on the Mask image, the original object in the elimination area can be eliminated, and after the original object is eliminated, background content is generated in the elimination area to obtain a second image.

[0084] As can be seen from the above embodiment, in this embodiment, a first image and reference information corresponding to the first image are obtained, wherein the reference information includes at least one of the following: a depth image and environmental noise; based on the first image and the reference information, the first image is segmented to obtain a Mask image; based on the Mask image, the first image is subjected to content elimination processing to obtain a second image. It can be seen that in the embodiment of the present application, in addition to image data, depth data and / or environmental noise data can also be used for Mask segmentation. Since depth data can provide the relative distance or absolute depth between objects, clarify the spatial hierarchy (such as foreground / background separation), and identify occlusion relationships, and environmental noise data can identify the scene in which the image is located, scene recognition helps to identify objects, backgrounds or semantic areas (such as the sky, roads, faces, etc.) in the image. Therefore, for images with insufficient resolution, motion blur or complex occlusion, the multimodal Mask segmentation method can accurately lock the content to be eliminated, thereby improving the accuracy and robustness of image content elimination.

[0085] In some embodiments provided herein, considering that the first image (e.g., RGB image), its corresponding depth image, and ambient noise belong to information of different modalities, these modalities have significant differences in data representation, feature distribution, and semantic meaning, and therefore cannot be directly jointly processed or fused. In order to effectively utilize these multimodal information, when the reference information includes: depth image and ambient noise, the above step 202 may specifically include the following steps:

[0086] Step 2021, step 2022, step 2023 and step 2024.

[0087] In step 2021, image features of the first image are obtained.

[0088] In the embodiment of the present application, image features are used to characterize the texture and color details of the image.

[0089] In an embodiment of the present application, a classic convolutional neural network (CNN) such as ResNet or VGG can be used to extract image features of the first image.

[0090] In an embodiment of the present application, the above-mentioned convolutional neural network may include: ResNet / VGG Blocks, a global average pooling layer and a fully connected layer; wherein, ResNet / VGG Blocks is used to extract high-level features and provide depth and RGB information representation; global average pooling is used to reduce feature dimensions and retain important information; and the fully connected layer is used for the final feature output.

[0091] In step 2022, depth features of the depth image corresponding to the first image are obtained.

[0092] In the embodiment of the present application, the depth feature is used to clarify the spatial relationship and occlusion level of objects.

[0093] In the embodiment of the present application, a classic convolutional neural network (CNN) such as ResNet or VGG may also be used to extract depth features of the depth image.

[0094] In step 2023, semantic features of scene description information corresponding to the first image are obtained based on the environmental noise corresponding to the first image.

[0095] In the embodiment of the present application, semantic features are used to represent high-level semantic understanding (such as object categories / attributes).

[0096] In this embodiment, a noise-aware scene classification module can be used to pre-build a noise-scene mapping network based on the physical correlation between voiceprint features and scenes (e.g., the difference between 60dB-70dB low- to medium-frequency continuous noise on city streets and 40dB-50dB discrete medium-frequency noise in indoor scenes). In actual use, the ambient noise is input into the noise-scene mapping network, which outputs scene description information corresponding to the first image.

[0097] In an embodiment of the present application, the scene description information may include: description text of the scene type and description text of the scene clutter level, wherein the scene type includes but is not limited to: indoor, outdoor, outdoors, concert, etc., and the scene clutter level includes but is not limited to: quiet, noisy, clamorous, roaring, etc.

[0098] In the embodiment of the present application, the semantic features of the scene description information can be extracted through a text encoder.

[0099] In step 2024, the first image is segmented according to the image features, depth features, and semantic features to obtain a mask image.

[0100] It can be seen that for the first image, its corresponding depth image, and environmental noise, which belong to information of different modalities, in an embodiment of the present application, the above-mentioned information of different modalities can be converted into image features, depth features, and semantic features that can be jointly processed or fused. By combining the above-mentioned image features, depth features, and semantic features, the first image can be subjected to Mask segmentation processing, which can effectively utilize multimodal information and obtain a high-precision Mask image.

[0101] In some embodiments provided in the present application, in order to further improve the accuracy of Mask segmentation, the above-mentioned step 2024 may specifically include the following steps: step 20241 and step 20242.

[0102] In step 20241, the image features, depth features and semantic features are fused and enhanced to obtain multimodal mixed features.

[0103] Taking into account the limitations of a single feature, in an embodiment of the present application, the image features, depth features and semantic features can be fused and enhanced to play a role in feature fusion compensation, so that when Mask segmentation processing is performed based on multimodal mixed features, the obtained Mask image conforms to both physical space constraints (depth) and semantic logic (text description).

[0104] In an embodiment of the present application, image features, depth features and semantic features can be input into a cross-modal interaction module to output fusion features; wherein, the cross-modal interaction module is used to establish semantic alignment and feature fusion between features of different modalities; through the multi-head attention mechanism, the above-mentioned fusion features are enhanced to obtain multimodal mixed features.

[0105] In an embodiment of the present application, a cross-modal interaction module can realize the complementary fusion of cross-modal features. Specifically, the module first forms three feature spaces from the semantic features of image features, depth features and environmental noise respectively; then, a cross-modal attention mechanism is designed, and each modality interacts with other modalities after being projected through the query, key and value matrices. For example, image features can aggregate relevant information from noise and depth modalities through attention weights, thereby enhancing their representation capabilities. Each attention head focuses on a different feature subspace, and finally merges the multi-head outputs through a linear projection layer. This mechanism enables the three modalities to enhance each other. The semantic features of environmental noise provide environmental complexity information, the depth features provide geometric structure constraints, and the image features provide visual semantic information. The synergistic effect of the three significantly improves the ability to distinguish between targets and backgrounds in cluttered scenes.

[0106] In an embodiment of the present application, the multi-head attention mechanism may include: 1) a multi-head attention layer: used for interaction between features of different modalities; 2) position encoding: adding position encoding information to the features of each modality, using the sine-cosine position encoding method to enhance the model's perception of sequence position, and the position encoding is directly added to the features to keep the feature dimension unchanged; 3) multi-head attention interaction: using a multi-head attention mechanism to realize information interaction between different modalities, each modality is used as a query, and cross-attention calculation is performed with other modalities. Multiple attention heads work in parallel, focusing on different feature subspaces. Through attention weights, each modality can selectively obtain complementary information of other modalities; 4) self-attention enhancement: applying a self-attention layer to each modality to enhance the feature representation within the modality. Self-attention focuses on long-distance dependencies within the modality, using residual connections and layer normalization to improve training stability; 5) feature fusion: using a dynamic weight method to fuse the enhanced features of the three modalities, using a modal attention mechanism or a gated fusion mechanism, calculating the contribution weight of each modality, adding cross-modal residual connections, retaining the original feature information, and outputting the fused multimodal feature representation.

[0107] In the embodiments of the present application, through feature extraction and cross-modal interaction, the semantic features of different scenes can be effectively analyzed and understood, providing more accurate input for subsequent image processing steps.

[0108] In step 20242, the first image is segmented according to the multimodal mixed features to obtain a mask image.

[0109] In an embodiment of the present application, the multimodal mixed features can be input into a depth-guided segmentation network to output a mask image of the first image; wherein the depth-guided segmentation network is a mask segmentation network based on depth information.

[0110] In an embodiment of the present application, the depth-guided segmentation network can be an improved version of the U-Net network using Depth-guided, mainly including: an encoder-decoder, a skip connection layer and a depth information guidance layer; wherein the encoder-decoder is used for image segmentation, the skip connection layer is used to maintain high-precision spatial information, and the depth information guidance layer is used to introduce depth information into the above layers to improve segmentation accuracy.

[0111] In an embodiment of the present application, the depth-guided segmentation network may include a dual-stream architecture: an image branch network and a depth branch network; wherein the image branch network is used to extract texture semantic features, and the depth branch network is used to learn geometric constraint features through deformable convolution, and adopt a deep weighted strategy to enhance the physical consistency of the segmentation boundary and improve the segmentation accuracy of the occluded area, so that the occlusion boundary is clearer (such as the junction of the chair legs and the ground), and the low-texture area (such as the door frame on the white wall) is segmented more accurately.

[0112] It can be seen that in the embodiment of the present application, Mask segmentation is performed on the first image based on multimodal reference information such as the first image, depth image and environmental noise, which breaks through the limitations of pure visual segmentation and improves the accuracy of Mask segmentation.

[0113] In some embodiments provided in the present application, when the reference information includes a depth image, the above step 202 may specifically include the following steps: step 2025 , step 2026 , and step 2027 .

[0114] In step 2025, image features of the first image are obtained.

[0115] In step 2026 , depth features of the depth image corresponding to the first image are obtained.

[0116] In the embodiment of the present application, the contents of step 2025 and step 2026 are similar to the contents of the above-mentioned step 2021 and step 2022, and will not be repeated here. For details, please refer to the contents of the above-mentioned step 2021 and step 2022.

[0117] In step 2027, the first image is segmented according to the image features and the depth features to obtain a Mask image.

[0118] In an embodiment of the present application, the image features and depth features can be fused and enhanced to obtain multimodal mixed features; based on the multimodal mixed features, the first image is segmented to obtain a mask image.

[0119] In an embodiment of the present application, image features and depth features can be input into a cross-modal interaction module to output fused features; the fused features can be enhanced through a multi-head attention mechanism to obtain multimodal mixed features; the multimodal mixed features can be input into a depth-guided segmentation network to output a mask image of the first image.

[0120] It can be seen that for the first image, its corresponding depth image and other information belonging to different modalities, in an embodiment of the present application, the above-mentioned information of different modalities can be converted into image features and depth features that can be jointly processed or fused. By combining the above-mentioned image features and depth features, the first image can be subjected to Mask segmentation processing, which can effectively utilize multimodal information and obtain a high-precision Mask image.

[0121] In some embodiments provided in the present application, when the reference information includes environmental noise, the above step 202 may specifically include the following steps: step 2028, step 2029 and step 20210.

[0122] In step 2028, image features of the first image are obtained.

[0123] In step 2029, semantic features of scene description information corresponding to the first image are obtained based on the environmental noise corresponding to the first image.

[0124] In the embodiment of the present application, the contents of step 2028 and step 2029 are similar to the contents of the above-mentioned step 2021 and step 2023, and will not be repeated here. For details, please refer to the contents of the above-mentioned step 2021 and step 2023.

[0125] In step 20210, the first image is segmented based on image features and semantic features to obtain a mask image.

[0126] In an embodiment of the present application, the image features and semantic features can be fused and enhanced to obtain multimodal mixed features; based on the multimodal mixed features, the first image is segmented to obtain a mask image.

[0127] In an embodiment of the present application, image features and semantic features can be input into a cross-modal interaction module to output fused features; the fused features can be enhanced through a multi-head attention mechanism to obtain multimodal mixed features; the multimodal mixed features can be input into a Mask segmentation network to output a Mask image of the first image.

[0128] It can be seen that for the first image, its corresponding environmental noise and other information belonging to different modalities, in an embodiment of the present application, the above-mentioned information of different modalities can be converted into image features and semantic features that can be jointly processed or fused. Combined with the above-mentioned image features and semantic features, the first image is subjected to Mask segmentation processing, which can effectively utilize multimodal information and obtain a high-precision Mask image.

[0129] At present, in the content elimination stage, the scene elimination algorithms in related technologies also have obvious defects: insufficient elimination capabilities lead to residual problems, new debris and passers-by are generated in the eliminated area, and the generated area is not integrated enough with the original area. These problems make the processed image still visually inconsistent, affecting the overall elimination effect. For example, after eliminating passers-by or debris, there are often residual outlines or inconsistent colors, making the elimination results unnatural. At the same time, the generated content used to fill the elimination area often has color and texture differences from the surrounding environment, affecting the final visual effect. The scene elimination algorithms in related technologies lack a full understanding of the context, making it difficult to generate content that seamlessly integrates with the surrounding environment, and ultimately it is difficult to achieve accurate and effective elimination of cluttered scenes.

[0130] In order to solve the above technical problems, the following Figure 3 The illustrated method embodiment improves the ability to remove image content, thereby achieving accurate and effective clutter removal.

[0131] Figure 3 This is a flowchart of an implementation of step 203 provided in an embodiment of the present application. Figure 3 As shown, the above step 203 may specifically include the following steps: step 301 and step 302.

[0132] In step 301, the first image, the mask image and the prompt word are input into the elimination model, and the elimination image of the elimination area indicated by the mask image is output; wherein the elimination model is a model obtained by training the diffusion model based on the foreground sample image and the background sample image; the prompt word is used to suppress the appearance of the original object or interference elements in the elimination area, and to guide the generated content in the elimination area to be consistent with the background semantic style of the image where the elimination area is located.

[0133] The current scene elimination capability mainly depends on the elimination capability of the segmentation and elimination model. The elimination model can be roughly divided into traditional elimination methods and elimination methods based on deep learning. In recent years, the diffusion model has had a huge impact on image generation, and elimination based on the diffusion model has shown great prospects. In the embodiment of the present application, the elimination model is a model obtained by training the diffusion model based on a large number of foreground sample images and background sample images. Through the probabilistic generation framework and large-scale data priors, it achieves the naturalness of elimination that is difficult to achieve with traditional methods. Its core advantage lies in converting the elimination task into "semantic-based generative completion" rather than a simple "local pixel replacement", which can improve the fusion effect and elimination capability of the elimination area.

[0134] In the embodiments of this application, foreground sample images refer to a type of data specifically used to train image elimination or segmentation models. Their core characteristic is that they contain prominent objects (i.e., foreground elements) that need to be identified, separated, or eliminated. The goal of model training is to teach the model to recognize the visual features of common foreground elements (such as human body structure and vehicle outlines) and to enhance the model's ability to fine-tune the foreground-background transition area.

[0135] In the embodiments of this application, background sample images refer to key data specifically used to train image removal, restoration, or generation models. Their core characteristic is that they do not contain foreground objects that need to be removed, retaining only the original background content of the scene. The goal of model training is to teach the model to generate reasonable background content based on context (such as obscured wall textures, ground extensions, etc.), while maintaining the lighting, perspective, and material of the generated area to match the original background.

[0136] In an embodiment of the present application, the diffusion model is trained based on a large number of foreground sample images and background sample images. On the one hand, the trained elimination model can thoroughly learn the visual features and occlusion patterns of common foreground objects (such as people, vehicles, and debris), reduce background information interference, and enhance sensitivity to foreground edges / textures; on the other hand, the trained elimination model learns to generate reasonable background content based on the context, thereby improving the illumination / geometric consistency between the generated content and the surrounding background, thereby significantly improving the elimination capability and generation quality of the elimination model in complex scenes.

[0137] In an embodiment of the present application, prompt words are obtained by training along with the elimination model, and the prompt words are composed of two parts: background prompt words and foreground prompt words; among them, background prompt words (also called "positive prompt words", "background repair prompt") are used to guide the generated content in the elimination area to be consistent with the background semantic style of the image where the elimination area is located; foreground prompt words (also called "negative prompt words", "foreground repair prompt") are used to suppress the appearance of original objects or interference elements in the elimination area.

[0138] In an embodiment of the present application, in order to improve the generalization ability of the model, the background cue word is a single high-level semantic word (such as P_bg), which refers to common background elements such as the sky, floor, wall, and street. The features of all background categories share a subspace in the latent space of the diffusion model and are activated by "P_bg"; similarly, the foreground cue word is also a single high-level semantic word (such as P_fg), which refers to common foreground elements such as people, animals, tables, and footballs. The features of all foreground categories share a subspace in the latent space of the diffusion model and are activated by "P_fg", thereby enabling the elimination model to adapt to a wider range of inputs.

[0139] In an embodiment of the present application, the first image and the prompt word are used as the guiding control conditions of the elimination model, the elimination area in the first image is determined by the first image and the Mask image, the original image of the elimination area is used as noisy data, and the original image of the elimination area is denoised by the elimination model to obtain an eliminated image of the elimination area. The eliminated image is generated based on the contextual background information such as the lighting and texture of the pixels around the elimination area, which can ensure the consistency of lighting and texture between the elimination area and the surrounding pixels, reduce color difference, and enhance the elimination ability.

[0140] For example, the first image depicts a girl, A, standing on the shore of West Lake, with a boy, B, interspersed in the image. The goal of content removal is to remove boy B from the first image and generate a natural background image in the image region where boy B resides. In this case, the removal region is the image region where boy B resides, and boy B's image is the original image of the removal region. The removal image of the removal region is generated based on contextual information such as lighting and texture of the pixels surrounding boy B. For example, the removal image represents the water or shore of West Lake.

[0141] In step 302, the original image of the eliminated area in the first image is replaced with the eliminated image to obtain a second image.

[0142] In the embodiment of the present application, the second image is an image obtained by eliminating interference elements in the first image.

[0143] It can be seen that in the embodiment of the present application, since the elimination model is obtained by training the diffusion model based on a large number of background sample images and foreground sample images, it can fully learn the prior knowledge from the original image to the eliminated image, and the prompt word can guide the elimination model to suppress the appearance of the original object or interference elements in the elimination area, and guide the generated content in the elimination area to be consistent with the background semantic style of the image where the elimination area is located. Therefore, based on the elimination model, it is possible to accurately and effectively eliminate the interference elements in the image, reduce the low-quality output caused by incomplete elimination or erroneous elimination, and improve the accuracy and robustness of image content elimination.

[0144] In some embodiments provided in this application, Figure 4 As shown, the elimination model 40 may include: an image encoder 41, a text feature encoding network 42, an image restoration network 43 based on a diffusion model, a denoiser based on a diffusion model 44 and an image decoder 45;

[0145] In an embodiment of the present application, considering that the dimension of feature data is much lower than the dimension of pixel space, in order to reduce the amount of computation, an image encoder is deployed in the elimination model to perform image encoding on the image input to the elimination model into low-dimensional feature data through the image encoder.

[0146] In the embodiment of the present application, the above-mentioned image encoder is a pre-trained image encoder.

[0147] Considering that the VAE encoder can extract high-level semantic features of the image, filter out irrelevant details (such as noise and texture), and make the diffusion process more stable, in an embodiment of the present application, the above-mentioned image encoder can be a VAE encoder.

[0148] In an embodiment of the present application, a text feature encoding network is used to convert text descriptions into numerical representations (embedded vectors) with semantic meaning, thereby guiding the model to generate images aligned with the text content.

[0149] In the embodiment of the present application, the above-mentioned text feature encoding network is a pre-trained text feature encoding network.

[0150] In an embodiment of the present application, the above-mentioned text feature encoding network can be a contrastive language-image pretraining (CLIP) text encoder or a lightweight EmbeddingNet.

[0151] In this embodiment, the image inpainting network is a local editing control network designed specifically for diffusion models. Its core function is to precisely constrain the visual content of specific areas during image generation (for example, preserving a portion of the original image's structure or style) while allowing other areas to be freely generated based on text prompts. Introducing the image inpainting network into the elimination model improves the lighting and texture consistency between the eliminated area and surrounding pixels, reduces color difference, and enhances the elimination capability.

[0152] Considering that BrushNet has precise local control capabilities and can repair or edit images with high quality without affecting non-target areas, in an embodiment of the present application, the image repair network can be BrushNet.

[0153] In an embodiment of the present application, the denoiser may be a pre-trained UNet denoiser.

[0154] In the embodiment of the present application, the image decoder is used to convert the data of the feature dimension into an image. In practical applications, the image decoder can be a decoder of VAE.

[0155] Accordingly, if Figure 5 As shown, the above step 301 may specifically include the following steps: step 3011, step 3012, step 3013, step 3014 and step 3015.

[0156] In step 3011, the first image, the mask image and the masked image are respectively encoded by an image encoder to obtain feature data of the first image, feature data of the mask image and feature data of the masked image; wherein the masked image is an image obtained by performing a pixel-by-pixel product operation on the first image and the mask image at the same pixel position.

[0157] In the embodiment of the present application, since other image areas outside the elimination area in the first image do not want to be changed, only the image content in the elimination area is to be eliminated, a masked image is obtained by performing a pixel-by-pixel product operation on the first image and the mask image at the same pixel position.

[0158] In this embodiment of the application, the image encoder compresses the input image into a low-dimensional latent space, preserving the image's structure and semantic information while avoiding the computational burden and detail loss caused by direct pixel-space operations. In deep learning and generative AI, latent features refer to the compressed encoding of data in a low-dimensional embedding space and serve as a critical bridge between raw data and high-level semantics.

[0159] In the embodiment of the present application, the feature data of the first image refers to the Latent corresponding to the first image, the feature data of the mask image refers to the Latent corresponding to the mask image, and the feature data of the masked image refers to the Latent corresponding to the masked image.

[0160] In step 3012, the prompt word is encoded through a text feature encoding network to obtain feature data of the prompt word.

[0161] In the embodiment of the present application, the text feature encoding network encodes text prompt words into semantic vectors, provides semantic guidance for eliminating intentions, and makes the results more in line with user expectations.

[0162] In the embodiment of the present application, the feature data of the prompt word refers to the Latent corresponding to the prompt word.

[0163] In step 3013, the feature data of the prompt word and the spliced ​​feature data are fused and enhanced through the image restoration network to obtain enhanced feature data; wherein, the spliced ​​feature data is obtained by feature splicing the feature data of the mask image and the feature data of the masked image.

[0164] In an embodiment of the present application, the image restoration network iteratively denoises and reconstructs the masked area in the latent space, generates reasonable content based on the text and image context, and avoids the repeated texture or blurring problems of traditional methods (such as PatchMatch).

[0165] In the embodiment of the present application, the spliced ​​feature data refers to the Latent obtained by splicing the Latent corresponding to the Mask image and the Latent corresponding to the Masked image.

[0166] In the embodiment of the present application, the introduction of an image restoration network for feature fusion enhancement processing can improve the transition between the elimination area and the background area, making the image processing result more natural and significantly reducing common problems such as color difference and edge discontinuity.

[0167] Exemplarily, when the image restoration network is BrushNet, the splicing feature data is input into BrushNet as a control condition. After being processed by BrushNet, the obtained enhanced feature data consists of the following three parts: down_block_res_samples, mid_block_res_sample and up_block_res_samples.

[0168] In step 3014, the feature data of the first image and the feature data of the prompt word are used as guiding conditions of the denoiser. The denoiser under the guiding conditions is used to denoise the enhanced feature data to obtain noise data corresponding to the feature data of the masked image.

[0169] In the embodiment of the present application, the noise is gradually eliminated by the denoiser, which can improve the authenticity and detail quality (such as natural edge transition) of the generated area.

[0170] In an embodiment of the present application, the original image of the elimination area in the first image is used as noisy data, and the elimination area is denoised time-step by time-step through a denoiser. The noise data corresponding to the feature data of the masked image refers to the noise data of the elimination area in the first image. By removing the noise data from the feature data of the masked image, denoised data of the elimination area can be obtained. Image decoding of the denoised data can be performed to obtain an eliminated image of the elimination area indicated by the masked image.

[0171] For example, when the image restoration network is BrushNet, the spliced ​​feature data is input into BrushNet as a control condition. After BrushNet processing, the enhanced feature data obtained is composed of the following three parts: down_block_res_samples, mid_block_res_samples, and up_block_res_samples. Subsequently, down_block_res_samples, mid_block_res_samples, and up_block_res_samples are used as the down_block_res_samples, mid_block_res_samples, and up_block_res_samples of the denoiser, respectively, and input into the denoiser. Denoising is performed time-step by time-step to improve the fusion effect of the mask area and the background area and the image quality.

[0172] In step 3015, the denoising data corresponding to the feature data of the masked image is decoded by an image decoder to obtain an eliminated image of the eliminated area indicated by the masked image; wherein the denoising data corresponding to the feature data of the masked image is obtained by removing the noise data from the feature data of the masked image.

[0173] In an embodiment of the present application, the image decoder decodes the latent features back to the pixel space and outputs a high-resolution restoration result that is compatible with input images of different sizes.

[0174] For ease of understanding, combined Figure 6To illustrate the example shown, first, the first image, mask image, and masked image are input into the image encoder 41, which outputs feature data for the first image, mask image, and masked image, respectively. A prompt word is input into the text encoding feature network 42, which outputs feature data for the prompt word.

[0175] Afterwards, the feature data of the masked image and the feature data of the masked image are spliced ​​to obtain spliced ​​feature data, and the spliced ​​feature data and the feature data of the prompt word are input into the image restoration network 43, and the image restoration network 43 outputs the enhanced feature data after fusion enhancement.

[0176] The feature data of the first image, the feature data of the prompt word and the enhanced feature data are input into the denoiser 44, and the feature data of the first image and the feature data of the prompt word are used as guiding conditions of the denoiser 44 to guide the denoiser 44 to perform denoising on the enhanced feature data. The denoiser 44 outputs the noise data corresponding to the feature data of the masked image.

[0177] The noise data in the feature data of the masked image is removed to obtain denoised data corresponding to the feature data of the masked image, and the denoised data is input into the image decoder 45 , which outputs an eliminated image of the eliminated area indicated by the mask image.

[0178] It can be seen that in the embodiment of the present application, in the image elimination task, the elimination model is modularly designed with an image encoder, a text feature encoding network, a diffusion model-based restoration network, a denoiser and an image decoder. Through the collaboration of semantic guidance (text), structure retention (image encoding) and generation capability (diffusion model), accurate elimination of target content and natural generation of context-matched restoration content are achieved, which can significantly improve the quality and controllability of the elimination effect.

[0179] Figure 7 This is a flow chart of a model training method provided in an embodiment of the present application. Figure 7 As shown, the method may include the following steps: step 701, step 702, step 703 and step 704.

[0180] In step 701, a panoptic segmentation is performed on the foreground sample image to extract foreground content with clear semantic categories and generate a corresponding foreground Mask image; wherein different Mask encoding methods are used for different semantic categories.

[0181] In an embodiment of the present application, each foreground sample image can be panoramically segmented to extract foreground content with a clear semantic category in each foreground sample image, and a foreground Mask image corresponding to the foreground content in each foreground sample image is generated based on the semantic category of the foreground content; wherein, foreground content of different semantic categories corresponds to different Mask encoding methods.

[0182] In an embodiment of the present application, in order to improve the elimination ability of the elimination model, a large number of foreground sample images can be used when training the elimination model, so as to meet the complex and changeable characteristics of the foreground in the elimination task (such as pedestrians / vehicles, etc.).

[0183] Exemplarily, the number of foreground sample images is 15 million, and the foreground sample images may be from the SA-1B dataset and / or the coconut dataset.

[0184] In an embodiment of the present application, a Mask2former network can be used to perform panoramic segmentation on each foreground sample image, and foreground content with clear semantic categories in each foreground sample image can be extracted.

[0185] In an embodiment of the present application, when generating a foreground mask image corresponding to the foreground content in each foreground sample image, each foreground category corresponds to an independent mask to help the network learn the foreground representation of a specific category. Each foreground category has an independent mask, and the network can focus more on the characteristics of the category during training, such as shape, texture, color, etc., thereby more accurately identifying and generating foreground objects of that category. For example, if the foreground category is "car", an independent mask can allow the network to specifically learn the characteristics of the car, improving the recognition and generation capabilities of the car category.

[0186] In step 702, panorama segmentation is performed on the background sample image to extract background content with complex semantics and generate a corresponding background mask image; wherein all background sample images adopt the same mask encoding method.

[0187] In an embodiment of the present application, each background sample image can be panoramically segmented, the background content with complex semantics in each background sample image can be extracted, and a background mask image corresponding to the background content in each background sample image is generated; wherein the background content of all background sample images corresponds to the same mask encoding method.

[0188] In an embodiment of the present application, in order to improve the elimination capability of the elimination model, a large number of background sample images can be used when training the elimination model, thereby meeting the requirement of relatively stable background in the elimination task.

[0189] Exemplarily, the number of background sample images is 3 million, and the background sample images may also come from the SA-1B dataset and / or the coconut dataset.

[0190] In the embodiment of the present application, the Mask2former network can also be used to perform panoramic segmentation on each background sample image to extract background content with complex semantics in each background sample image.

[0191] In an embodiment of the present application, when generating a background mask image corresponding to the background content in each background sample image, all background categories are merged into a single mask, prompting the network to learn to generate the content of the occluded area based on the contextual semantics. When the background categories are merged into a single mask, the network no longer needs to distinguish between different background categories, but instead pays more attention to the overall relationship and contextual information between the foreground object and the background. For example, when there is a partially occluded object in the image, the network can use the learned contextual semantics to infer the reasonable content of the occluded area based on the surrounding known foreground objects and the overall scene information, so that the generated result is more consistent with the actual scene.

[0192] In an embodiment of the present application, a background mask image corresponding to the background content of the background sample image can be generated based on a diversified mask generation strategy; wherein the diversified mask generation strategy includes at least two of the following: a random brush strategy, a Bezier curve strategy, and a polygon strategy.

[0193] In an embodiment of the present application, the random brush strategy can be used to randomly paint using circular brushes of different sizes / hardnesses to simulate irregular occlusion (such as graffiti).

[0194] In the embodiment of the present application, the Bezier curve strategy can be used to generate and fill a smooth curve of 3-5 control points to create an organic shape (such as the outline of a natural object).

[0195] In this embodiment of the present application, the polygon strategy can be used to randomly generate 4-8 polygons and apply affine transformations (rotation / scaling) to simulate geometric structure occlusion (such as buildings). In addition, the intersection with the original mask can be taken to ensure that the original area to be eliminated is not exceeded.

[0196] It can be seen that in the embodiments of the present application, a diversified mask generation strategy is adopted, so that the generated mask image can contain smooth / jagged / irregular edges, cover complex occlusions in the real world, improve adaptability to masks of arbitrary shapes, and enhance the model's robustness and generalization ability to complex mask shapes.

[0197] It is understandable that during panorama segmentation, a targeted category division method can be adopted according to the actual application scenario requirements, and objects that need to be suppressed (such as people, animals, cars, etc.) are classified as foregrounds, while elements that need to be generated (such as the sky, grass, walls, etc.) are classified as backgrounds.

[0198] In step 703, a model to be trained is constructed including an image encoder, a text feature encoding network, a backbone network corresponding to the image restoration network, and a denoiser.

[0199] In the embodiments of the present application, the image encoder, text feature encoding network, and denoiser in the elimination model are all pre-trained networks. The image inpainting network in the elimination model needs to be trained. That is, during the model training process, the image encoder, text feature encoding network, and denoiser in the to-be-trained model are fixed, and only the backbone network corresponding to the image inpainting network is trained, ultimately obtaining the image inpainting network and thus the elimination model.

[0200] In step 704, the foreground training data and the background training data are used to alternately train the model to be trained, and the parameters of the backbone network are updated until convergence, thereby obtaining an elimination model and its prompt words; wherein, the foreground training data includes: a foreground sample image, a foreground mask image, a foreground masked image, and randomly initialized foreground prompt words; the foreground masked image is an image obtained by performing a pixel-by-pixel product operation on the foreground sample image and the foreground mask image at the same pixel position; the background training data includes: a background sample image, a background mask image, a background masked image, and randomly initialized background prompt words; the background masked image is an image obtained by performing a pixel-by-pixel product operation on the background sample image and the background mask image at the same pixel position; the prompt words of the elimination model are composed of the foreground prompt words and the background prompt words at the time of convergence.

[0201] In an embodiment of the present application, the model to be trained is alternately trained using multiple sets of foreground training data and multiple sets of background training data to update the network parameters of the backbone network in the model to be trained until the model convergence conditions are met, thereby obtaining an elimination model and its prompt words.

[0202] In the embodiment of the present application, each set of foreground training data includes: a foreground sample image, a foreground mask image corresponding to the foreground sample image, a foreground masked image, and a randomly initialized foreground prompt word.

[0203] In the embodiment of the present application, each set of background training data includes: a background sample image, a background mask image corresponding to the background sample image, a background masked image and a randomly initialized background prompt word.

[0204] In this embodiment of the present application, the Adam optimizer can be used to train the elimination model, with a learning rate set to 1e-5 and an image resolution set to 768 × 768. The training process is performed on 8 A100 GPUs, with a batch size of 1 for each batch, and a total of approximately 250,000 training steps.

[0205] In the embodiment of the present application, the principle of training the elimination model is mainly to guide the model to be trained to generate background content and suppress the model from generating foreground. In order to achieve such a task, two prompts are designed, namely background prompt words and foreground prompt words. The background prompt words are used to guide the denoiser to generate background categories, and the foreground prompt words are used to suppress the denoiser from generating foreground categories.

[0206] In the embodiment of the present application, to simplify the model and improve inference speed, during training, the randomly initialized foreground and background cue words are both learnable semantic tensors; the text feature encoding network is a lightweight EmbeddingNet. Specifically, the background cue words and foreground cue words are initialized as different Tensors, then encoded into different feature vectors through EmbeddingNet. Finally, these feature vectors are injected into the backbone network and denoiser of the subsequent image restoration network. The lightweight EmbeddingNet replaces the functions of the tokenizer and text encoder, reducing the time consumption during training and inference.

[0207] It should be noted that a more primitive method can also be used to randomly initialize background prompt words and foreground prompt words using other letters or words, then perform word segmentation through Tokenizer, and then encode them into features through Text_Enocder. Since Text_Enocder is already trained, in order to make Text_Enocder represent the meaning of these two words, it is only necessary to fine-tune the token_embedding layer under the Text_Model module Embeddings of Text_Enocder. The other layers are frozen and not trained. Similarly, the features after Text_Enocder are injected into the backbone network and denoiser of the image restoration network to guide model generation.

[0208] It can be seen that in the embodiment of the present application, by alternately using foreground training data and background training data to train the elimination model, the robustness and generation quality of the elimination model can be significantly improved, thereby improving the accuracy and robustness of image content elimination.

[0209] In some embodiments provided herein, the process of alternately training the training model using foreground training data and background training data is similar to the inference process of the elimination model described above, but there are some differences: it includes the forward noise addition process, the loss calculation process, and the backpropagation process to update the network parameters. Figure 8 As shown, the above step 704 may specifically include the following steps: step 7041, step 7042 and step 7043.

[0210] In step 7041, a batch of foreground training data is input into the model to be trained, the first loss is calculated and the parameters of the backbone network in the model to be trained are updated.

[0211] In an embodiment of the present application, a batch of foreground training data is input into the model to be trained, the foreground training data is processed by the model to be trained to obtain a first processing result, a first loss value is calculated based on the first processing result and the loss function, and based on the first loss value, the network parameters of the backbone network in the model to be trained are updated.

[0212] In an embodiment of the present application, the loss function of the model to be trained can be composed of a mean square error loss function and a noise consistency loss function.

[0213] In the embodiment of the present application, the mean squared error loss (MSE Loss) function is a basic loss function used in deep learning to measure the difference between the predicted value and the target value, and the model is optimized by calculating the average of the pixel-level square difference between the two. In the diffusion model, the above-mentioned predicted value refers to the model prediction noise, and the above-mentioned target value refers to the real noise added by the noise addition process. MSE Loss has become the benchmark loss function for computer vision tasks because of its simplicity and efficiency. In the diffusion model, it provides a basic guarantee for the quality of image generation by directly constraining the accuracy of noise prediction.

[0214] In the embodiment of the present application, the noise consistency loss function is a special loss function in the diffusion model (especially for image restoration / elimination tasks). Its core function is to improve the seamless fusion of the generated content and the original background by constraining the spatial consistency of the noise prediction. Noise Consistency Loss avoids abrupt differences in color / lighting / texture between the generated content and the background by forcing the predicted noise of the elimination area to be consistent with the noise distribution of the surrounding real background area. It also ensures that the generated content conforms to the physical logic of the scene by guiding the generation of the noise statistical characteristics (mean, variance) of the background area, thereby maintaining the authenticity of the generated content and achieving seamless visual fusion.

[0215] In the embodiment of the present application, during the training process, the combination of MSE Loss and Noise Consistency Loss can balance the detail retention and overall consistency in the elimination area of ​​the image.

[0216] In some embodiments, a batch of foreground training data can be input into the model to be trained, and the foreground sample image, foreground mask image and foreground masked image are encoded respectively through the image encoder to obtain the first, second and third foreground features; the forward denoising algorithm of the model to be trained is used to add noise to the first and third foreground features respectively to obtain the first and second foreground noisy data; the randomly initialized foreground prompt words are encoded through the text feature encoding network to obtain the fourth foreground feature; the second foreground feature and the second foreground noisy data are spliced ​​into the fifth foreground feature; the fourth and fifth foreground features are fused and enhanced through the backbone network to obtain the sixth foreground feature; the first foreground noisy data and the fourth foreground feature are used as guiding conditions of the denoiser, and the sixth foreground feature is denoised by the denoiser under the guiding conditions to obtain the first predicted noise; the first loss is calculated based on the first predicted noise and the real noise added to the third foreground feature, and the parameters of the backbone network are updated.

[0217] In step 7042, a batch of background training data is input into the updated model to be trained, the second loss is calculated and the parameters of the backbone network are updated.

[0218] In an embodiment of the present application, a batch of background training data is input into the model to be trained after parameter update, the background training data is processed by the model to be trained to obtain a second processing result, a second loss value is calculated based on the second processing result and the loss function, and the network parameters of the backbone network in the model to be trained are updated based on the second loss value.

[0219] In some embodiments, a batch of background training data can be input into the updated model to be trained, and the background sample image, background mask image and background masked image are encoded respectively through the image encoder to obtain the first, second and third background features; the forward denoising algorithm of the model to be trained is used to add noise to the first and third background features respectively to obtain the first and second background noisy data; the randomly initialized background prompt words are encoded through the text feature encoding network to obtain the fourth background feature; the second background feature and the second background noisy data are spliced ​​into the fifth background feature; the fourth and fifth background features are fused and enhanced through the backbone network to obtain the sixth background feature; the first background noisy data and the fourth background feature are used as guiding conditions of the denoiser, and the sixth background feature is denoised by the denoiser under the guiding conditions to obtain the second predicted noise; the second loss is calculated based on the second predicted noise and the real noise added to the third background feature, and the parameters of the backbone network are updated.

[0220] In step 7043, the above alternating training process is repeated for the remaining batches of foreground training data and background training data until the model converges to obtain the elimination model and its prompt words.

[0221] In an embodiment of the present application, the remaining batches of foreground training data and background training data can be alternately input into the model to be trained after parameter update, and the model to be trained can be trained to update the network parameters of the backbone network in the model to be trained until the model convergence conditions are met to obtain the elimination model and its prompt words.

[0222] For ease of understanding, take the training process of a batch of foreground training data as an example, combined with Figure 9 The example shown is used to illustrate this: First, the foreground sample image, foreground mask image, and foreground masked image are respectively input into the image encoder 41 of the model to be trained. The image encoder 41 outputs the first foreground feature of the foreground sample image, the second foreground feature of the foreground mask image, and the third foreground feature of the foreground masked image, respectively. Simultaneously, a foreground cue word is randomly initialized and then input into the text encoding feature network 42 of the model to be trained. The text encoding feature network 42 outputs the fourth foreground feature of the foreground cue word.

[0223] Afterwards, random noise is added to the first foreground feature and the third foreground feature respectively through the forward denoising algorithm of the model to be trained, so as to obtain first foreground noisy data corresponding to the first foreground feature and second foreground noisy data corresponding to the third foreground feature.

[0224] Afterwards, the second foreground feature and the second foreground noisy data are feature spliced ​​to obtain the fifth foreground feature, and the fourth foreground feature and the fifth foreground feature are input into the backbone network 50 corresponding to the image restoration network of the model to be trained, and the backbone network 50 outputs the fused and enhanced sixth foreground feature.

[0225] The first foreground noisy data, the fourth foreground feature and the sixth foreground feature are input into the denoiser 44 of the model to be trained. The first foreground noisy data and the fourth foreground feature are used as guiding conditions of the denoiser 44 to guide the denoiser 44 to perform denoising on the sixth foreground feature. The denoiser 44 outputs the first predicted noise corresponding to the third foreground feature.

[0226] Based on the first predicted noise, the actual noise data added to the third foreground feature by the forward noise algorithm, and the loss function, a first loss is calculated. Based on the first loss, the network parameters of the backbone network in the model to be trained are updated, completing one training cycle. The training process for the background training data is similar and will not be repeated here.

[0227] It can be seen that in the embodiment of the present application, by alternately using foreground training data and background training data to train the elimination model, the robustness and generation quality of the model can be significantly improved.

[0228] It should be noted that in the above training process, in order to balance the training system overhead and the model's elimination ability, the denoiser was fixed and the image inpainting network was trained. In actual applications, the denoiser and image inpainting network can also be trained together to further improve the elimination ability of the model.

[0229] In some embodiments provided herein, the elimination model may include: an image encoder, a text feature encoding network, a denoiser based on a diffusion model, and an image decoder. Figure 4 The elimination model shown in the embodiment of the present application removes the image restoration network, and the contents of other parts are similar and will not be repeated here. The elimination model can improve the reasoning speed.

[0230] In summary, in the embodiment of the present application, the elimination effect is improved from two directions. First, the following Figure 2 The method embodiment shown improves the accuracy of Mask segmentation, and then adopts Figure 3 The illustrated method embodiment improves the ability to remove image content, thereby achieving accurate and effective clutter removal.

[0231] In the embodiment of the present application, the image information, depth information and environmental noise are used to realize the collection and processing of multimodal data. This integration method improves the robustness and adaptability of the system in complex environments. Through feature extraction and cross-modal interaction, the semantic features of different scenes can be effectively analyzed and understood, providing more accurate input for subsequent image processing steps. The use of a deep guided segmentation network for segmentation can provide a more accurate Mask image in the first stage, which helps to accurately identify and locate the target object in image processing. In the second stage, foreground prompt words and background prompt words are introduced to guide the elimination model denoising to realize intelligent elimination of cluttered scenes, such as removing passers-by and debris. This method reduces the low-quality output caused by incomplete elimination or erroneous elimination. The introduction of an image restoration network for regional fusion improves the transition between the elimination area and the background area, making the image processing results more natural, significantly reducing common problems such as color difference and edge discontinuity, and improving the elimination ability of the model.

[0232] In the embodiment of the present application, by clearly distinguishing foreground data from background data, the correspondence between foreground prompt words and background prompt words is strengthened. This processing strategy enables the model to more effectively suppress the generation of foreground elements, while generating appropriate background content based on the contextual content, which not only enhances the model's ability to recognize and distinguish foreground and background, but also improves the overall elimination performance.

[0233] In the embodiments of this application, existing smartphone devices can be used to automatically clean up complex scenes and process high-quality images without the need for additional hardware support. This method can efficiently process and eliminate clutter (such as passers-by, debris, etc.), and accurately segment and process the scene, ensuring that the processed image blends naturally with the original life scene. This not only improves the post-processing limitations of traditional image capture, greatly improves the quality of image processing and user satisfaction, but also significantly reduces the cost of post-editing and processing. Users can achieve direct output of multiple effects through simple operations, improving the image's aesthetic and artistic value.

[0234] The image processing method and model training method provided in the embodiments of the present application can be executed by an image processing device and a model training device, respectively. In the embodiments of the present application, an image processing device executing the image processing method is used as an example to illustrate the image processing device provided in the embodiments of the present application, and a model training device executing the model training method is used as an example to illustrate the model training device provided in the embodiments of the present application.

[0235] Figure 10 This is a structural block diagram of an image processing device provided by an embodiment of the present application. Figure 10 As shown, the image processing device 1000 may include: an acquisition module 1001, a segmentation module 1002 and an elimination module 1003;

[0236] The acquisition module 1001 is configured to acquire a first image and reference information corresponding to the first image, wherein the reference information includes at least one of the following: a depth image and environmental noise;

[0237] The segmentation module 1002 is configured to segment the first image to obtain a mask image based on the first image and the reference information;

[0238] The elimination module 1003 is configured to perform content elimination processing on the first image based on the mask image to obtain a second image.

[0239] It can be seen that, unlike the related art in which Mask segmentation is performed based only on image data of a single modality, in the embodiments of the present application, in addition to image data, Mask segmentation can also be performed with reference to depth data and / or environmental noise data. Since depth data can provide the relative distance or absolute depth between objects, clarify the spatial hierarchy (such as foreground / background separation), and identify occlusion relationships, and environmental noise data can identify the scene in which the image is located, scene recognition helps to identify objects, backgrounds or semantic areas (such as the sky, roads, faces, etc.) in the image. Therefore, for images with insufficient resolution, motion blur or complex occlusion, the multi-modal Mask segmentation method can accurately lock the content to be eliminated, thereby improving the accuracy and robustness of image content elimination.

[0240] Optionally, as an embodiment, the reference information includes: a depth image and environmental noise;

[0241] The segmentation module 1002 is specifically used to obtain image features of the first image; obtain depth features of the depth image corresponding to the first image; obtain semantic features of the scene description information corresponding to the first image based on the environmental noise corresponding to the first image; and segment the first image according to the image features, the depth features and the semantic features to obtain a mask image.

[0242] Optionally, as an embodiment, the segmentation module 1002 is specifically used to perform fusion enhancement processing on the image features, the depth features and the semantic features to obtain multimodal mixed features; and based on the multimodal mixed features, the first image is segmented to obtain a mask image.

[0243] Optionally, as an embodiment, the segmentation module 1002 is specifically used to input the image features, the depth features and the semantic features into a cross-modal interaction module, and output fusion features; wherein the cross-modal interaction module is used to establish semantic alignment and feature fusion between different modal features; through a multi-head attention mechanism, the fusion features are enhanced to obtain multimodal mixed features.

[0244] Optionally, as an embodiment, the segmentation module 1002 is specifically used to input the multimodal mixed features into a depth-guided segmentation network and output a mask image of the first image; wherein the depth-guided segmentation network is a mask segmentation network based on depth information.

[0245] Optionally, as an embodiment, the elimination module 1003 is specifically used to input the first image, the mask image and the prompt word into the elimination model, and output the elimination image of the elimination area indicated by the mask image; replace the original image of the elimination area in the first image with the elimination image to obtain a second image; wherein, the elimination model is a model obtained by training a diffusion model based on the foreground sample image and the background sample image; the prompt word is used to suppress the appearance of the original object or interference elements in the elimination area, and to guide the generated content in the elimination area to be consistent with the background semantic style of the image where the elimination area is located.

[0246] Optionally, as an embodiment, the elimination model includes: an image encoder, a text feature encoding network, an image restoration network based on a diffusion model, a denoiser based on a diffusion model and an image decoder;

[0247] The segmentation module 1002 is specifically configured to encode the first image, the mask image and the masked image respectively through the image encoder to obtain feature data of the first image, feature data of the mask image and feature data of the masked image; wherein the masked image is an image obtained by performing a pixel-by-pixel product operation on the first image and the mask image at the same pixel position; encode the prompt word through the text feature encoding network to obtain feature data of the prompt word; and fuse and enhance the feature data of the prompt word and the spliced ​​feature data through the image restoration network to obtain enhanced feature data; wherein the spliced ​​feature data is The feature data of the masked image and the feature data of the masked image are obtained by feature splicing; the feature data of the first image and the feature data of the prompt word are used as guiding conditions of the denoiser, and the denoiser under the guiding conditions is used to denoise the enhanced feature data to obtain noise data corresponding to the feature data of the masked image; the denoised data corresponding to the feature data of the masked image is decoded by the image decoder to obtain an eliminated image of the eliminated area indicated by the masked image; wherein the denoised data corresponding to the feature data of the masked image is obtained by removing the noise data from the feature data of the masked image.

[0248] Optionally, as an embodiment, the image restoration network is BrushNet, and the denoiser is a pre-trained UNet denoiser.

[0249] Figure 11 This is a structural block diagram of a model training device provided in an embodiment of the present application. Figure 11As shown, the model training device 1100 may include: a first generating module 1101, a second generating module 1102, a construction module 1103 and a training module 1104;

[0250] The first generation module 1101 is used to perform panoptic segmentation on the foreground sample image, extract foreground content with clear semantic categories, and generate corresponding foreground mask images; wherein different mask encoding methods are used for different semantic categories;

[0251] The second generation module 1102 is used to perform panorama segmentation on the background sample image, extract the background content with complex semantics, and generate a corresponding background mask image; wherein all background samples use the same mask encoding method;

[0252] The construction module 1103 is used to construct a model to be trained including an image encoder, a text feature encoding network, a backbone network corresponding to the image restoration network, and a denoiser;

[0253] The training module 1104 is used to alternately train the model to be trained using foreground training data and background training data, update the parameters of the backbone network until convergence, and obtain an elimination model and its prompt words; wherein the foreground training data includes: the foreground sample image, the foreground mask image, the foreground masked image and randomly initialized foreground prompt words; the foreground masked image is an image obtained by performing a pixel-by-pixel product operation on the foreground sample image and the foreground mask image at the same pixel position; the background training data includes: the background sample image, the background mask image, the background masked image and randomly initialized background prompt words; the background masked image is an image obtained by performing a pixel-by-pixel product operation on the background sample image and the background mask image at the same pixel position; the prompt words of the elimination model are composed of the foreground prompt words and the background prompt words at convergence.

[0254] It can be seen that in the embodiment of the present application, by alternately using foreground training data and background training data to train the elimination model, the robustness and generation quality of the elimination model can be significantly improved, thereby improving the accuracy and robustness of image content elimination.

[0255] Optionally, as an embodiment, the randomly initialized foreground prompt words and background prompt words are both learnable semantic tensors; and the text feature encoding network is a lightweight EmbeddingNet.

[0256] Optionally, as an embodiment, the second generation module 1102 is specifically used to generate a background mask image corresponding to the background content of the background sample image based on a diversified mask generation strategy; wherein the diversified mask generation strategy includes at least two of the following: a random brush strategy, a Bezier curve strategy, and a polygon strategy.

[0257] Optionally, as an embodiment, the training module 1104 is specifically used to input a batch of foreground training data into the model to be trained, calculate the first loss and update the parameters of the backbone network in the model to be trained; input a batch of background training data into the updated model to be trained, calculate the second loss and update the parameters of the backbone network; for the remaining batches of foreground training data and background training data, repeat the above alternating training process until the model converges to obtain the elimination model and its prompt word.

[0258] Optionally, as an embodiment, the training module 1104 is specifically used to input a batch of foreground training data into the model to be trained, and encode the foreground sample image, the foreground mask image and the foreground masked image respectively through the image encoder to obtain the first, second and third foreground features; add noise to the first and third foreground features respectively through the forward denoising algorithm of the model to be trained to obtain the first and second foreground noisy data; encode the randomly initialized foreground prompt word through the text feature encoding network to obtain the fourth foreground feature; splice the second foreground feature and the second foreground noisy data into a fifth foreground feature; fuse and enhance the fourth and fifth foreground features through the backbone network to obtain the sixth foreground feature; use the first foreground noisy data and the fourth foreground feature as the guiding conditions of the denoiser, and denoise the sixth foreground feature through the denoiser under the guiding conditions to obtain the first predicted noise; calculate the first loss based on the first predicted noise and the real noise added to the third foreground feature, and update the parameters of the backbone network.

[0259] Optionally, as an embodiment, the training module 1104 is specifically used to input a batch of background training data into the updated model to be trained, and encode the background sample image, the background mask image and the background masked image respectively through the image encoder to obtain the first, second and third background features; add noise to the first and third background features respectively through the forward denoising algorithm of the model to be trained to obtain the first and second background noisy data; encode the randomly initialized background prompt words through the text feature encoding network to obtain the fourth background feature; splice the second background feature and the second background noisy data into the fifth background feature; fuse and enhance the fourth and fifth background features through the backbone network to obtain the sixth background feature; use the first background noisy data and the fourth background feature as the guiding conditions of the denoiser, and denoise the sixth background feature through the denoiser under the guiding conditions to obtain the second predicted noise; calculate the second loss based on the second predicted noise and the real noise added to the third background feature, and update the parameters of the backbone network.

[0260] Optionally, as an embodiment, the loss function of the model to be trained consists of a mean square error loss function and a noise consistency loss function.

[0261] The image processing device and the model training device in the embodiment of the present application can be an electronic device that works independently, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, a car-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) / virtual reality (Virtual Reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (Ultra-Mobile Personal Computer, UMPC), a netbook, or a personal digital assistant (Personal Digital Assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (Personal Computer, PC), a television (Television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0262] The image processing device and model training device in the embodiments of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0263] The image processing device provided in the embodiment of the present application can achieve Figure 2 、 Figure 3 or Figure 5 The various processes implemented by the method embodiment shown in the figure can be realized by the model training device provided in the embodiment of the present application. Figure 7 、 Figure 8 To avoid repetition, the various processes implemented in the illustrated method embodiment will not be described again here.

[0264] Alternatively, as Figure 12 As shown, the embodiment of the present application further provides an electronic device 1200, including a processor 1201 and a memory 1202, wherein the memory 1202 stores a program or instruction that can be run on the processor 1201, and when the program or instruction is executed by the processor 1201, each step of the above-mentioned image processing method or model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not described here. It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and non-mobile electronic device described above.

[0265] Figure 13 This is a schematic diagram of the hardware structure of an electronic device that implements various embodiments of the present application. The electronic device 1300 includes, but is not limited to, a radio frequency unit 1301, a network module 1302, an audio output unit 1303, an input unit 1304, a sensor 1305, a display unit 1306, a user input unit 1307, an interface unit 1308, a memory 1309, and a processor 1310.

[0266] Those skilled in the art will understand that the electronic device 1300 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 1310 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 13 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0267] Processor 1310 is used to obtain a first image and reference information corresponding to the first image, wherein the reference information includes at least one of the following: a depth image and environmental noise; based on the first image and the reference information, the first image is segmented to obtain a mask image; based on the mask image, the first image is content-eliminated to obtain a second image.

[0268] It can be seen that in the embodiments of the present application, in addition to image data, Mask segmentation can also be performed with reference to depth data and / or environmental noise data. Since depth data can provide the relative distance or absolute depth between objects, clarify the spatial hierarchy (such as foreground / background separation), and identify occlusion relationships, and environmental noise data can identify the scene in which the image is located, scene recognition helps to identify objects, backgrounds or semantic areas (such as the sky, roads, faces, etc.) in the image. Therefore, for images with insufficient resolution, motion blur or complex occlusion, the multimodal Mask segmentation method can accurately lock the content to be eliminated, thereby improving the accuracy and robustness of image content elimination.

[0269] Optionally, as an embodiment, the reference information includes: a depth image and environmental noise; the processor 1310 is specifically used to obtain image features of the first image; obtain depth features of the depth image corresponding to the first image; obtain semantic features of the scene description information corresponding to the first image based on the environmental noise corresponding to the first image; and perform segmentation processing on the first image based on the image features, the depth features and the semantic features to obtain a mask image.

[0270] Optionally, as an embodiment, the processor 1310 is specifically used to perform fusion enhancement processing on the image features, the depth features and the semantic features to obtain multimodal mixed features; and based on the multimodal mixed features, the first image is segmented to obtain a mask image.

[0271] Optionally, as an embodiment, the processor 1310 is specifically used to input the image features, the depth features and the semantic features into a cross-modal interaction module, and output fusion features; wherein the cross-modal interaction module is used to establish semantic alignment and feature fusion between different modal features; through a multi-head attention mechanism, the fusion features are enhanced to obtain multimodal mixed features.

[0272] Optionally, as an embodiment, the processor 1310 is specifically configured to input the multimodal mixed features into a depth-guided segmentation network and output a mask image of the first image; wherein the depth-guided segmentation network is a mask segmentation network based on depth information.

[0273] Optionally, as an embodiment, the processor 1310 is specifically used to input the first image, the mask image and the prompt word into the elimination model, and output the elimination image of the elimination area indicated by the mask image; replace the original image of the elimination area in the first image with the elimination image to obtain a second image; wherein, the elimination model is a model obtained by training a diffusion model based on the foreground sample image and the background sample image; the prompt word is used to suppress the appearance of the original object or interference elements in the elimination area, and to guide the generated content in the elimination area to be consistent with the background semantic style of the image where the elimination area is located.

[0274] Optionally, as an embodiment, the elimination model includes: an image encoder, a text feature encoding network, an image restoration network based on a diffusion model, a denoiser based on a diffusion model and an image decoder; the processor 1310 is specifically used to encode the first image, the mask image and the masked image respectively through the image encoder to obtain feature data of the first image, feature data of the mask image and feature data of the masked image; wherein the masked image is an image obtained by performing a pixel-by-pixel product operation on the first image and the mask image at the same pixel position; the prompt word is encoded through the text feature encoding network to obtain the feature data of the prompt word; the feature data of the prompt word and the splicing feature data are obtained through the image restoration network. The feature data is fused and enhanced to obtain enhanced feature data; wherein, the spliced ​​feature data is obtained by feature splicing the feature data of the masked image and the feature data of the masked image; the feature data of the first image and the feature data of the prompt word are used as guiding conditions of the denoiser, and the enhanced feature data is denoised by the denoiser under the guiding conditions to obtain noise data corresponding to the feature data of the masked image; the denoised data corresponding to the feature data of the masked image is decoded by the image decoder to obtain an eliminated image of the eliminated area indicated by the masked image; wherein, the denoised data corresponding to the feature data of the masked image is obtained by removing the noise data from the feature data of the masked image.

[0275] Optionally, as an embodiment, the image restoration network is BrushNet, and the denoiser is a pre-trained UNet denoiser.

[0276] Optionally, as an embodiment, the processor 1310 is further used to perform panoramic segmentation on the foreground sample image, extract the foreground content with clear semantic categories, and generate a corresponding foreground Mask image; wherein different semantic categories use different Mask encoding methods; perform panoramic segmentation on the background sample image, extract the background content with complex semantics, and generate a corresponding background Mask image; wherein all background sample images use the same Mask encoding method; construct a training model including an image encoder, a text feature encoding network, a backbone network corresponding to the image restoration network, and a denoiser; use foreground training data and background training data to alternately train the training model, update the parameters of the backbone network until convergence, and obtain an elimination model and its prompt words; wherein, the foreground training data includes: the foreground sample image, the foreground mask image, the foreground masked image and randomly initialized foreground prompt words; the foreground masked image is an image obtained by performing a pixel-by-pixel product operation on the foreground sample image and the foreground mask image at the same pixel position; the background training data includes: the background sample image, the background mask image, the background masked image and randomly initialized background prompt words; the background masked image is an image obtained by performing a pixel-by-pixel product operation on the background sample image and the background mask image at the same pixel position; the prompt words of the elimination model are composed of the foreground prompt words and the background prompt words at the time of convergence.

[0277] Optionally, as an embodiment, the randomly initialized foreground prompt words and background prompt words are both learnable semantic tensors; and the text feature encoding network is a lightweight EmbeddingNet.

[0278] Optionally, as an embodiment, the processor 1310 is specifically used to generate a background mask image corresponding to the background content of the background sample image based on a diversified mask generation strategy; wherein the diversified mask generation strategy includes at least two of the following: a random brush strategy, a Bezier curve strategy, and a polygon strategy.

[0279] Optionally, as an embodiment, the processor 1310 is specifically used to input a batch of foreground training data into the model to be trained, calculate the first loss and update the parameters of the backbone network in the model to be trained; input a batch of background training data into the updated model to be trained, calculate the second loss and update the parameters of the backbone network; for the remaining batches of foreground training data and background training data, repeat the above alternating training process until the model converges to obtain the elimination model and its prompt word.

[0280] Optionally, as an embodiment, the processor 1310 is specifically configured to input a batch of foreground training data into the model to be trained, encode the foreground sample image, the foreground mask image and the foreground masked image respectively through the image encoder to obtain first, second and third foreground features; add noise to the first and third foreground features respectively through the forward denoising algorithm of the model to be trained to obtain first and second foreground noisy data; encode the randomly initialized foreground prompt word through the text feature encoding network to obtain a fourth foreground feature; concatenate the second foreground feature and the second foreground noisy data into a fifth foreground feature; fuse and enhance the fourth and fifth foreground features through the backbone network to obtain a sixth foreground feature; use the first foreground noisy data and the fourth foreground feature as guiding conditions of the denoiser, and denoise the sixth foreground feature through the denoiser under the guiding conditions to obtain a first predicted noise; calculate a first loss based on the first predicted noise and the real noise added to the third foreground feature, and update the parameters of the backbone network.

[0281] Optionally, as an embodiment, the processor 1310 is specifically used to input a batch of background training data into the updated model to be trained, and encode the background sample image, the background mask image and the background masked image respectively through the image encoder to obtain the first, second and third background features; add noise to the first and third background features respectively through the forward denoising algorithm of the model to be trained to obtain the first and second background noisy data; encode the randomly initialized background prompt words through the text feature encoding network to obtain the fourth background feature; splice the second background feature and the second background noisy data into the fifth background feature; fuse and enhance the fourth and fifth background features through the backbone network to obtain the sixth background feature; use the first background noisy data and the fourth background feature as the guiding conditions of the denoiser, and denoise the sixth background feature through the denoiser under the guiding conditions to obtain the second predicted noise; calculate the second loss based on the second predicted noise and the real noise added to the third background feature, and update the parameters of the backbone network.

[0282] Optionally, as an embodiment, the loss function of the model to be trained consists of a mean square error loss function and a noise consistency loss function.

[0283] It should be understood that in an embodiment of the present application, the input unit 1304 may include a graphics processing unit (GPU) 13041 and a microphone 13042, and the graphics processor 13041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1306 may include a display panel 13061, and the display panel 13061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1307 includes a touch panel 13071 and at least one of the other input devices 13072. The touch panel 13071 is also called a touch screen. The touch panel 13071 may include two parts: a touch detection device and a touch controller. Other input devices 13072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0284] The memory 1309 can be used to store software programs and various data. The memory 1309 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1309 may include a volatile memory or a non-volatile memory, or the memory 1309 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1309 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0285] Processor 1310 may include one or more processing units. Optionally, processor 1310 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1310.

[0286] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, each process of the above-mentioned image processing method or model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not described here. Wherein, the processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a disk or an optical disk, etc.

[0287] The embodiment of the present application further provides a chip, the chip including a processor and a communication interface, the communication interface and the processor being coupled, the processor being used to run a program or instruction to implement the various processes of the above-mentioned image processing method or model training method embodiment, and to achieve the same technical effect. To avoid repetition, it will not be described here. It should be understood that the chip mentioned in the embodiment of the present application can also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip chip, etc.

[0288] An embodiment of the present application also provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the various processes of the above-mentioned image processing method or model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0289] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0290] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (such as a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0291] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image and reference information corresponding to the first image, wherein the reference information includes at least one of the following: a depth image and environmental noise; Segmenting the first image to obtain a mask image according to the first image and the reference information; Based on the Mask image, content removal processing is performed on the first image to obtain a second image.

2. The method according to claim 1, characterized in that The reference information includes: a depth image and environmental noise; The step of segmenting the first image according to the first image and the reference information to obtain a mask image includes: acquiring image features of the first image; Obtaining depth features of the depth image corresponding to the first image; acquiring semantic features of scene description information corresponding to the first image according to the environmental noise corresponding to the first image; The first image is segmented according to the image features, the depth features and the semantic features to obtain a Mask image.

3. The method according to claim 2, characterized in that The step of segmenting the first image according to the image feature, the depth feature, and the semantic feature to obtain a mask image includes: Performing fusion enhancement processing on the image features, the depth features, and the semantic features to obtain multimodal mixed features; According to the multimodal mixed features, the first image is segmented to obtain a Mask image.

4. The method according to claim 3, characterized in that The fusing and enhancing the image features, the depth features, and the semantic features to obtain multimodal mixed features includes: Inputting the image features, the depth features, and the semantic features into a cross-modal interaction module and outputting fusion features; wherein the cross-modal interaction module is used to establish semantic alignment and feature fusion between features of different modalities; Through the multi-head attention mechanism, the fusion features are enhanced to obtain multimodal mixed features.

5. The method according to claim 3, characterized in that The step of segmenting the first image according to the multimodal mixed feature to obtain a mask image includes: The multimodal hybrid features are input into a depth-guided segmentation network, and a mask image of the first image is output; wherein the depth-guided segmentation network is a mask segmentation network based on depth information.

6. The method according to any one of claims 1 to 5, characterized in that The performing content elimination processing on the first image based on the mask image to obtain the second image includes: Inputting the first image, the mask image and the prompt word into the elimination model, and outputting an eliminated image of the elimination area indicated by the mask image; replacing the original image of the eliminated area in the first image with the eliminated image to obtain a second image; Among them, the elimination model is a model obtained by training a diffusion model based on foreground sample images and background sample images; the prompt word is used to suppress the appearance of original objects or interference elements in the elimination area, and to guide the generated content in the elimination area to be consistent with the background semantic style of the image where the elimination area is located.

7. The method according to claim 6, characterized in that The elimination model includes: an image encoder, a text feature encoding network, an image restoration network based on a diffusion model, a denoiser based on a diffusion model and an image decoder; The step of inputting the first image, the mask image, and the prompt word into the elimination model and outputting an eliminated image of the elimination area indicated by the mask image comprises: Encoding the first image, the masked image, and the masked image, respectively, by the image encoder to obtain feature data of the first image, feature data of the masked image, and feature data of the masked image; wherein the masked image is an image obtained by performing a pixel-by-pixel product operation at the same pixel position of the first image and the masked image; Encoding the prompt word through the text feature encoding network to obtain feature data of the prompt word; The feature data of the prompt word and the spliced ​​feature data are fused and enhanced by the image restoration network to obtain enhanced feature data; wherein the spliced ​​feature data is obtained by splicing the feature data of the mask image and the feature data of the masked image; Using the feature data of the first image and the feature data of the prompt word as guiding conditions of the denoiser, and denoising the enhanced feature data by the denoiser under the guiding conditions to obtain noise data corresponding to the feature data of the masked image; The denoising data corresponding to the feature data of the masked image is decoded by the image decoder to obtain an eliminated image of the eliminated area indicated by the masked image; wherein the denoising data corresponding to the feature data of the masked image is obtained by removing the noise data from the feature data of the masked image.

8. The method according to claim 7, characterized in that The image restoration network is BrushNet, and the denoiser is a pre-trained UNet denoiser.

9. A model training method, characterized in that: The method comprises: Perform panoptic segmentation on the foreground sample image, extract foreground content with clear semantic categories, and generate the corresponding foreground mask image; different semantic categories use different mask encoding methods; Perform panorama segmentation on the background sample image, extract the background content with complex semantics, and generate the corresponding background mask image; all background sample images use the same mask encoding method; Build a training model including an image encoder, a text feature encoding network, a backbone network corresponding to the image restoration network, and a denoiser; Alternately training the model to be trained using foreground training data and background training data, updating the parameters of the backbone network until convergence, and obtaining an elimination model and its prompt word; The foreground training data includes: the foreground sample image, the foreground mask image, the foreground masked image, and randomly initialized foreground prompt words; the foreground masked image is an image obtained by performing a pixel-by-pixel product operation at the same pixel position of the foreground sample image and the foreground mask image; The background training data includes: the background sample image, the background mask image, the background masked image and randomly initialized background prompt words; the background masked image is an image obtained by performing a pixel-by-pixel product operation on the same pixel position of the background sample image and the background mask image; The prompt words of the elimination model are composed of foreground prompt words and background prompt words at the time of convergence.

10. The method according to claim 9, characterized in that The randomly initialized foreground prompt words and background prompt words are both learnable semantic tensors; the text feature encoding network is a lightweight EmbeddingNet.

11. The method according to claim 9, characterized in that The generating of the corresponding background Mask image includes: Based on a diversified Mask generation strategy, generating a background Mask image corresponding to the background content of the background sample image; The diversified mask generation strategies include at least two of the following: a random brush strategy, a Bezier curve strategy, and a polygon strategy.

12. The method according to claim 9, characterized in that The method of alternately training the model to be trained using the foreground training data and the background training data, updating the parameters of the backbone network until convergence, and obtaining the elimination model and its prompt words includes: Inputting a batch of foreground training data into the model to be trained, calculating a first loss and updating the parameters of the backbone network in the model to be trained; Inputting a batch of background training data into the updated model to be trained, calculating a second loss and updating the parameters of the backbone network; For the remaining batches of foreground training data and background training data, repeat the above alternating training process until the model converges to obtain the elimination model and its prompt words.

13. The method according to claim 12, characterized in that Inputting a batch of foreground training data into the to-be-trained model, calculating a first loss, and updating the parameters of the backbone network in the to-be-trained model include: Inputting a batch of foreground training data into the to-be-trained model, and respectively encoding the foreground sample image, the foreground mask image, and the foreground masked image through the image encoder to obtain first, second, and third foreground features; adding noise to the first and third foreground features respectively using the forward noise addition algorithm of the model to be trained to obtain first and second foreground noisy data; encoding the randomly initialized foreground prompt word through the text feature encoding network to obtain a fourth foreground feature; splicing the second foreground feature and the second foreground noisy data into a fifth foreground feature; fusing and enhancing the fourth and fifth foreground features through the backbone network to obtain a sixth foreground feature; using the first foreground noisy data and the fourth foreground feature as guiding conditions of the denoiser, and denoising the sixth foreground feature by the denoiser under the guiding conditions to obtain a first predicted noise; A first loss is calculated according to the first predicted noise and the real noise added to the third foreground feature, and the parameters of the backbone network are updated.

14. The method according to claim 12, characterized in that Inputting a batch of background training data into the updated to-be-trained model, calculating the second loss and updating the parameters of the backbone network, includes: Inputting a batch of background training data into the updated to-be-trained model, and respectively encoding the background sample image, the background mask image, and the background masked image through the image encoder to obtain first, second, and third background features; Adding noise to the first and third background features respectively using the forward noise addition algorithm of the model to be trained to obtain first and second background noisy data; Encoding the randomly initialized background prompt words through the text feature encoding network to obtain a fourth background feature; splicing the second background feature and the second background noisy data into a fifth background feature; fusing and enhancing the fourth and fifth background features through the backbone network to obtain a sixth background feature; Using the first background noisy data and the fourth background feature as guiding conditions of the denoiser, and denoising the sixth background feature by the denoiser under the guiding conditions to obtain a second predicted noise; A second loss is calculated according to the second predicted noise and the real noise added by the third background feature, and the parameters of the backbone network are updated.

15. The method according to claim 12, characterized in that The loss function of the model to be trained consists of a mean square error loss function and a noise consistency loss function.

16. An image processing device, characterized in that: The device comprises: an acquisition module, configured to acquire a first image and reference information corresponding to the first image, wherein the reference information includes at least one of the following: a depth image and environmental noise; a segmentation module, configured to segment the first image according to the first image and the reference information to obtain a mask image; The elimination module is used to perform content elimination processing on the first image based on the mask image to obtain a second image.

17. A model training device, characterized in that: The device comprises: The first generation module is used to perform panorama segmentation on the foreground sample image, extract foreground content with clear semantic categories, and generate corresponding foreground mask images; different semantic categories use different mask encoding methods; The second generation module is used to perform panoramic segmentation on the background sample image, extract the background content with complex semantics, and generate the corresponding background mask image; wherein all background samples use the same mask encoding method; A construction module is used to construct a model to be trained, including an image encoder, a text feature encoding network, a backbone network corresponding to the image restoration network, and a denoiser; A training module, configured to alternately train the model to be trained using foreground training data and background training data, update the parameters of the backbone network until convergence, and obtain an elimination model and its prompt words; The foreground training data includes: the foreground sample image, the foreground mask image, the foreground masked image, and randomly initialized foreground prompt words; the foreground masked image is an image obtained by performing a pixel-by-pixel product operation at the same pixel position of the foreground sample image and the foreground mask image; The background training data includes: the background sample image, the background mask image, the background masked image and randomly initialized background prompt words; the background masked image is an image obtained by performing a pixel-by-pixel product operation on the same pixel position of the background sample image and the background mask image; The prompt words of the elimination model are composed of foreground prompt words and background prompt words at the time of convergence.

18. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory storing programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, implementing the steps of the image processing method as described in any one of claims 1 to 8, or implementing the steps of the model training method as described in any one of claims 9 to 15.

19. A readable storage medium, characterized in that The readable storage medium stores a program or instruction, which, when executed by a processor, implements the steps of the image processing method as described in any one of claims 1 to 8, or implements the steps of the model training method as described in any one of claims 9 to 15.

Images