A method and device for eliminating objects associated with environmental interactive perception

Abstract

The present invention discloses a method and device for eliminating objects associated with environmental interactive perception. The present invention uses multiple self-attention features A extracted during any one of the iterations of 0.4T to 0.6T as self-attention features that can accurately reflect objects and their influences, and extracts the self-attention features A corresponding to the object based on the obtained object mask, thereby separating the object from the background and retaining more environmental information, and performs multi-scale fusion and thresholding on the averaged self-attention features A to obtain the mask of the object and its influence. The present invention performs a mask suppression operation on the self-attention layer of the Unet module based on the mask of the object and its influence, so that the object and its influence part in the second process latent vector obtained by suppressing the Unet module are eliminated, thereby eliminating the object and its influence part in the final noise latent code obtained by multiple sampling.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a method and device for eliminating objects associated with environmental interactive perception. Background Art

[0002] With the rapid development of artificial intelligence technology, applying artificial intelligence to the field of artistic creation has become a hot research direction. Especially in film and television art creation, how to use artificial intelligence technology to assist in film and television scene construction, emotional expression, etc. is of great significance to improving the creation efficiency and quality of film and television art.

[0003] At present, the construction of film and television scenes mainly relies on manual design and construction, which is time-consuming and costly. In addition, film and television scenes often contain a large number of object elements that have an important impact on emotional expression and aesthetic presentation. However, there are often intricate connections between these object elements and the scenes, which brings many inconveniences to scene construction and post-editing.

[0004] An important application in film and television art creation is to use artificial intelligence to automatically extract object information from stock footage. However, most current methods only focus on the segmentation and recognition of the object itself, ignoring the relationship between the object and the surrounding environment. In fact, objects in film and television scenes often have complex effects on the environment, such as blocking the background, casting shadows, reflecting light, etc.

[0005] The invention patent application with publication number CN112184738A discloses an image segmentation method, device, equipment and storage medium. The method includes: fusing the visual features corresponding to the original image and the text features corresponding to the description language to obtain multimodal features, and the description language is used to specify the target object to be segmented in the original image; determining the visual area of ​​the target object according to the image corresponding to the multimodal features, and recording the image corresponding to the visual area as a response heat map; determining the segmentation result of the target object according to the image corresponding to the multimodal features and the response heat map. The above scheme decomposes the image segmentation process, firstly determines the visual area of ​​the target object based on the image corresponding to the fused multimodal features, obtains the response heat map, and then further determines the segmentation result according to the image corresponding to the multimodal features and the response heat map, effectively eliminating the semantic gap between the image and the description language, and segmenting the target object specified by the description language. However, the method disclosed in the patent cannot eliminate the complex impact of objects on the environment.

[0006] Therefore, there is an urgent need for an intelligent technology that can accurately model the relationship between objects and environment in film and television scenes, support flexible object editing and scene reconstruction, and provide technical support for applications such as object cutouts and background fusion.

[0007] In recent years, generative AI models represented by the Stable Diffusion model have made breakthrough progress in the field of image generation and editing. In particular, the Stable Diffusion XL (SDXL) model has a more powerful learning ability and can generate high-resolution, high-quality, and detailed realistic images. Stable Diffusion XL also supports local image editing, providing new possibilities for object editing in film and television scenes.

[0008] The method of removing objects from film and television images based on the stable diffusion XL model was proposed in this context. This method uses the powerful learning ability of the stable diffusion XL model to model the complex relationship between objects and the environment in film and television scenes, and realizes intelligent and refined object removal and scene restoration. This will provide strong technical support for film and television art creation, help intelligent film and television production, and promote the deep integration of audio-visual art and artificial intelligence technology, and has broad application prospects. Summary of the invention

[0009] The present invention provides an environment interactive perception-related object elimination method, and the image object elimination method can eliminate objects and their influences more accurately.

[0010] The present invention provides a method for eliminating objects associated with environment interactive perception, comprising:

[0011] The obtained original image is converted into an original latent vector, and based on the original latent vector, a first stable diffusion model is used to perform T sampling to obtain a plurality of first process latent vectors and a first noise latent code obtained by the last sampling;

[0012] Obtain multiple self-attention features A output by the upsampling layers of different sizes in the Unet module of the first stable diffusion model in any iteration of 0.4T to 0.6T, extract the corresponding self-attention features from each self-attention feature A based on the obtained object mask, and take the average to obtain self-attention features of multiple sizes ;

[0013] Multiple scales of self-attention features Multi-scale fusion is performed from low scale to high scale to obtain the mask of the object and its influence. In the tensor of the current fusion feature obtained by each scale fusion, the value below the current scale mask threshold is replaced with 0 to obtain the current fusion threshold feature. The current fusion threshold feature is smoothed and upsampled before fusion at the next scale.

[0014] A second stable diffusion model is constructed, wherein the second stable diffusion model includes a suppressed Unet module, wherein the suppressed Unet module is obtained by performing a mask suppression operation on the self-attention layer of the Unet module based on the mask of the object and its influence, and the first noise hidden code is sampled T times by the second stable diffusion model to obtain multiple process fusion hidden vectors and a final noise hidden code, wherein the fusion hidden vector obtained in each sampling process includes data within the mask range of the object and its influence in the second process hidden vector of the current sampling output by suppressing the Unet module and data outside the mask range of the object and its influence in the first process hidden vector of T-current sampling, and the final noise hidden code is decoded to obtain a final image with the image object eliminated.

[0015] Preferably, based on the obtained object mask, the corresponding self-attention feature is extracted from each self-attention feature A and the average is taken to obtain self-attention features of multiple sizes ,include:

[0016] The obtained object mask is downsampled so that the size of the object mask is consistent with the size of the self-attention feature A;

[0017] Based on the non-zero point coordinates of the object mask that keeps the same size as the self-attention feature A, the self-attention features within the object range are extracted from each self-attention feature A, the extracted self-attention features are averaged to obtain the self-attention feature B, and the self-attention features B of the same size are averaged to obtain self-attention features of multiple sizes. .

[0018] Preferably, the self-attention features of multiple sizes are Multi-scale fusion from low scale to high scale is used to obtain masks of objects and their effects, including:

[0019] S1. The self-attention feature of the current size Perform weighted fusion with the upsampled features obtained by the previous iteration to obtain the current fused features;

[0020] S2. In the tensor of the current fused feature, replace the values ​​below the current size mask threshold with 0 to obtain the current fused threshold feature, use a 2D Gaussian filter kernel to smooth the current fused threshold feature, and use bilateral linear interpolation to upsample the smoothed current fused threshold feature to obtain the current upsampled feature;

[0021] S3, iterate steps S1-S2 until self-attention features of multiple sizes The fusion is done, resulting in a mask of the object and its influence.

[0022] Preferably, in the tensor of the current fused feature, values ​​below the current size mask threshold are replaced with 0, wherein the current size mask threshold is a constant value or a locally varying value;

[0023] When the current size mask threshold is a local change value, a method for constructing a local change value includes:

[0024] Set the threshold constant C, sliding window size s, convolution kernel size to (1, 1, s, s), and convolution step size to s to ensure that there is no overlap between sliding windows;

[0025] Using the convolution kernel to perform sliding window calculation on the tensor of the current fusion feature, obtain the local mean of the pixels in each window, construct a local mean tensor based on the local mean of the pixels in each window, and make the size of the local mean tensor the same as the size of the tensor of the current fusion feature through interpolation operation;

[0026] The local change value is obtained by subtracting the local mean tensor after the weighted interpolation operation from the threshold constant.

[0027] Preferably, performing a mask suppression operation on the self-attention layer of the Unet module based on the mask of the object and its influence to obtain a suppressed Unet module comprises:

[0028] The mask input channel is extended to the Unet module, and the object and its affected mask are input into the self-attention layers of different sizes through the mask input channel, so that the self-attention scores output by the self-attention layers of different sizes are suppressed by the mask;

[0029] Mask-suppressed self-attention score for:

[0030] , ,in, , They are the Query vector and Key vector in the conventional self-attention calculation, d is the dimension of the Key vector, the Query vector in the conventional self-attention calculation, the Key vector, d is the dimension of the Key vector, is the mask of the object and its influence, To perform the minimum operation on each Q vector, copy the minimum value obtained along the dimension direction corresponding to the K vector. is element-wise multiplication.

[0031] Preferably, sampling the first noise hidden code T times by using the second stable diffusion model to obtain the final noise hidden code includes:

[0032] S1, the first noise hidden code is passed through the suppression Unet module to obtain the second process hidden vector of the current sub-sampling;

[0033] S2, constructing a fused latent vector for the current sampling by combining the data within the mask range of the object and its influence in the second process latent vector of the current sampling with the corresponding data outside the mask range of the object and its influence in the first process latent vector of the current sampling T-current sampling;

[0034] S3, iterate steps S1-S2 until T samplings are completed to obtain the final noise hidden code.

[0035] Preferably, the method for constructing a fusion vector includes:

[0036] Multiply the second-pass latent vector of the current sample by the binary object and its affected mask to obtain the second-pass suppressed latent vector;

[0037] The first-process suppressed latent vector is obtained by multiplying the first-process latent vector with the reflexive binary object and its affected mask with the same noise level;

[0038] The first process suppressed latent vector and the second process suppressed latent vector are added together to obtain the fusion vector of the current sample.

[0039] Preferably, the fusion latent vector of the s-th sampling for: ,in, is the hidden vector of the second process obtained by sampling for the sth time, For the T The first process hidden vector obtained by -s sampling, is a binarization operation, A mask for the object and its influence.

[0040] The present invention also provides an environment interactive perception-related object elimination device, comprising: a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement any one of the environment interactive perception-related object elimination methods.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention thus extracts multiple self-attention features A during any one of 0.4T to 0.6T iterations to obtain self-attention features that can more accurately reflect the object and its influence, and extracts the self-attention feature A corresponding to the object based on the obtained object mask, thereby separating the object from the background and retaining more environmental information, and performs multi-scale fusion and thresholding on the averaged self-attention feature A to obtain a mask of the object and its influence.

[0043] The present invention performs a mask suppression operation on the self-attention layer of the Unet module based on the mask of the object and its influence, so that the object and its influence part in the second process latent vector obtained by suppressing the Unet module are eliminated, and thus the object and its influence part in the final noise latent code obtained by multiple sampling are eliminated. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flow chart of a method for eliminating objects associated with environment interactive perception provided by a specific embodiment of the present invention;

[0045] Figure 2 This is a diagram of the processing effect provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0046] A specific embodiment of the present invention provides a method and system for eliminating image objects and their impact on the environment based on the stable diffusion XL (SDXL) model, aiming to automatically extract the impact of a specified object (such as a person, prop, etc.) on the surrounding environment from an image, such as mirror reflection, shadow, etc., create a mask containing the object and its impact, and repair the image on this basis, thereby eliminating the object and its impact on the environment and generating a "clean" background image without object interference. The present invention can achieve environmental interactive perception association through the constructed mask of the object and its impact.

[0047] A specific embodiment of the present invention provides an image object removal method based on a stable diffusion XL (SDXL) model, such as Figure 1 As shown, the following steps are included:

[0048] S1: Convert the obtained original image into an original latent vector, and perform T samplings based on the original latent vector through a first stable diffusion model to obtain multiple first process latent vectors and a first noise latent code obtained by the last sampling.

[0049] The specific embodiment of the present invention provides a method of performing T samplings based on the original latent vector through a first stable diffusion model to obtain a plurality of first process latent vectors and a first noise latent code obtained by the last sampling, including:

[0050] S11. Project the obtained original image into the latent space for encoding to obtain the original latent vector.

[0051] S12, performing reverse diffusion on the original latent vector through the diffusion implicit inversion sampler of the first stable diffusion model to obtain noise prediction information.

[0052] S13. Subtract the noise prediction information from the original latent vector to obtain the first process latent vector.

[0053] S14. Based on the preset number of sampling times T, iteratively execute S11-S13 to obtain the first process latent vector obtained from the first sampling to the T-1th sampling, the first noise latent code obtained from the last sampling, and the multiple self-attention features A output by the sampling layer in different dimensions in the Unet module of the first stable diffusion model in any iteration of 0.4T to 0.6T iterations. The self-attention feature A obtained at this number of iterations has the highest semantics, that is, it more significantly models the association between the area where the object is located and the area of ​​its influence.

[0054] In a specific embodiment, the sampling number T of the first stable diffusion model is 50 times. When the model reaches half of the set sampling number (T / 2) at the time step, the self-attention features generated by the sampling layers of all different dimensions of the Unet module of the first stable diffusion model in this time step are obtained, that is, the self-attention feature A; when the model reaches the set sampling number at the time step, the noise hidden code in this time step is obtained.

[0055] In a specific embodiment, there are 36 self-attention features A provided by the specific embodiment of the present invention, of which the resolution of 30 self-attention features A is , the resolution of the 6 self-attention features A is .

[0056] The latent space encoding provided in the specific embodiment of the present invention uses a VAE automatic encoder; the latent space decoding uses a VAE automatic decoder.

[0057] S2. Obtain multiple self-attention features A output by the sampling layer of different dimensions in the Unet module of the first stable diffusion model in any iteration of the intermediate iteration, i.e., 0.4T to 0.6T iterations, and extract the corresponding self-attention features from each self-attention feature A based on the obtained object mask and take the average to obtain self-attention features of multiple dimensions. ,include:

[0058] The obtained object mask is downsampled so that the size of the object mask is consistent with the size of the self-attention feature A;

[0059] And based on the non-zero point coordinates of the object mask that maintains the same size as the self-attention feature A, the self-attention features within the image object range are extracted from each self-attention feature A, and the extracted self-attention features are averaged to obtain the self-attention feature B.

[0060] The kth self-attention feature B provided in the specific embodiment of the present invention k for:

[0061] ,in, is the self-attention feature Bk The corresponding k-th self-attention feature A has a resolution of , For The corresponding object mask has a resolution of , calculate The coordinates of the N non-zero points in , this coordinate is all the non-zero coordinates in the object mask, and then all N self-attention features of the above coordinates of the same size are selected ,Right now .

[0062] In a specific embodiment, there are 36 self-attention features B obtained in the specific embodiment of the present invention, of which 30 self-attention features B are Resolution, 6 self-attention features B are Resolution.

[0063] Specifically, the object mask is a mask input by the user, which is downsampled in advance to the same size as the self-attention feature A.

[0064] In a specific embodiment of the present invention, the self-attention features B of the same size are averaged to obtain self-attention features of multiple sizes. In one embodiment, the self-attention feature There are two in total, one self-attention feature The resolution is , another self-attention feature The resolution is The self-attention features obtained here are , which is the average value of the self-attention distribution of the region where the object specified by the object mask is located to all other regions. The above features explicitly quantify the relevance of the object region to other regions.

[0065] S3. Self-attention features of multiple sizes Multi-scale fusion is performed from low to high scale to obtain the mask of the object and its influence. The current fusion feature obtained based on each scale fusion is thresholded, the current size mask threshold is calculated, and the value in the tensor of the current fusion feature below the threshold is replaced with 0 to obtain the current fusion threshold feature. The current fusion threshold feature is smoothed and upsampled before fusion at the next size.

[0066] The specific embodiment of the present invention combines self-attention features of multiple sizes Multi-scale fusion from low scale to high scale is used to obtain masks of objects and their effects, including:

[0067] S31. The self-attention feature of the current size The current fused features are obtained by weighted fusion with the upsampled features obtained by the previous iteration.

[0068] S32. In the tensor of the current fused feature, replace the values ​​below the current size mask threshold with 0 to obtain the current fused threshold feature, use a 2D Gaussian filter kernel to smooth the current fused threshold feature, and use bilateral linear interpolation upsampling on the smoothed current fused threshold feature to obtain the current upsampled feature.

[0069] S33, iterate steps S1-S2 until self-attention features of multiple sizes are obtained The fusion is done, resulting in a mask of the object and its influence.

[0070] In a specific embodiment, l The current fusion feature of the iteration for: ,in, For the l -1 Iteration obtained upsampled features, is the weight. In one specific embodiment, is 0.8, For the l Iteration corresponding to the self-attention feature.

[0071] The current size mask threshold calculated when performing threshold processing based on the current fusion feature obtained by each scale fusion according to the specific embodiment of the present invention is a constant value or a local variation value;

[0072] The method for constructing a local change value provided by a specific embodiment of the present invention includes: setting a threshold constant C, a sliding window size s, a convolution kernel size of (1, 1, s, s), and a convolution step size of s to ensure that the sliding windows do not overlap.

[0073] The convolution kernel is used to perform sliding window calculation on the tensor of the current fusion feature to obtain the local mean of the pixels in each window, and a local mean tensor is constructed based on the local mean of the pixels in each window. The size of the local mean tensor is made the same as the size of the tensor of the current fusion feature through interpolation operation.

[0074] In a specific embodiment of the present invention, the local change value is obtained by subtracting the local mean tensor m after the weighted interpolation operation from the threshold constant C. In a specific embodiment, the local change value is Cm / 5. Specifically, the local change value is limited to the range of 0 to 1, that is, a value less than 0 is assigned to 0, and a value greater than 1 is assigned to 1, so as to ensure the effectiveness of the threshold. In a specific embodiment, the threshold constant C is 0.05-0.20.

[0075] The larger the value of the object and the mask affected by it obtained in step S33 of the specific embodiment of the present invention, that is, the reference of the object, the value of the entire area, during use, the original image tensor is compared with the calculated local threshold using the mask affected by the object and its influence. For pixels less than the threshold, their values ​​are set to 0, and pixels greater than or equal to the threshold remain unchanged.

[0076] S4. Perform multiple sampling based on the constructed second stable diffusion model to obtain the final noise hidden code, and decode the final noise hidden code to obtain the final image with the image object eliminated: construct a second stable diffusion model, the second stable diffusion model includes a suppression Unet module, the suppression Unet module is obtained by performing a mask suppression operation on the self-attention layer of the Unet module based on the mask of the object and its influence, and perform T sampling on the first noise hidden code through the second stable diffusion model to obtain multiple process fusion hidden vectors and the final noise hidden code, wherein the fusion hidden vector obtained in each sampling process includes the data within the mask range of the object and its influence in the second process hidden vector obtained by suppressing the Unet module and the data outside the mask range of the object and its influence in the first process hidden vector corresponding to the current sampling, the corresponding first process hidden vector described in step S4 is the first process hidden vector output by the first stable diffusion model with the same degree of noise as the second process hidden vector, if the second stable diffusion model outputs the second process hidden vector of the sth sampling, then the corresponding first process hidden vector is the first process hidden vector of the T-current sampling output by the first stable diffusion model, and the final noise hidden code is decoded to obtain the final image with the image object eliminated.

[0077] In a specific embodiment, a mask suppression operation is performed on the self-attention layer of the Unet module based on the mask of the object and its influence to obtain a suppressed Unet module, including:

[0078] The mask input channel is extended to the Unet module, and the object and its affected mask are input into self-attention layers of different sizes through the mask input channel, so that the self-attention scores output by self-attention layers of different sizes are suppressed by the mask.

[0079] Mask-suppressed self-attention score for:

[0080] , ,in, , They are the Query vector and Key vector in the conventional self-attention calculation, d is the dimension of the Key vector, is the mask of the object and its influence, is element-wise multiplication, To perform the minimum operation on each Q vector, it copies the minimum value along the dimension corresponding to the K vector, and finally generates a vector aligned with the original dimension.

[0081] In a specific embodiment, the first noise hidden code is sampled T times by the second stable diffusion model to obtain the final noise hidden code, including:

[0082] S41. The first noise hidden code is encoded through the suppression Unet module to obtain the second process hidden vector of the current sub-sampling.

[0083] S42, constructing a fused latent vector for the current sampling by combining the data within the mask range of the object and its influence in the second process latent vector of the current sampling with the corresponding data outside the mask range of the object and its influence in the first process latent vector of the current sampling T-current sampling.

[0084] S43, iterate steps S41-S42 until T samplings are completed to obtain the final noise hidden code.

[0085] In a specific embodiment, the sampling times of the second stable diffusion model is 50 times.

[0086] In a specific embodiment, a method for constructing a fusion vector includes:

[0087] Multiply the second-pass latent vector of the current sample by the binary object and its affected mask to obtain the second-pass suppressed latent vector;

[0088] Multiply the first-process latent vector of the sampling with the same noise level, i.e. the corresponding T-current time, by the reflexive object and its affected mask to obtain the first-process suppressed latent vector;

[0089] The first process suppressed latent vector and the second process suppressed latent vector are added together to obtain the fusion vector of the current sample.

[0090] The fusion latent vector of the sth sampling provided by the specific embodiment of the present invention is for: ,in, is the hidden vector of the second process obtained by sampling for the sth time, For the T The first process hidden vector obtained by -s sampling, is a binarization operation, is the mask of the object and its influence, The mask for the reflexive binarized object and its influence.

[0091] A specific embodiment of the present invention also provides an environment interactive perception-related object elimination device, which is characterized by comprising: a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the environment interactive perception-related object elimination method.

[0092] The first stable diffusion model provided in the specific embodiment of the present invention is a stable diffusion XL implicit inversion model (SDXL DDIM Inversion) obtained by modifying the stable diffusion XL model denoising process;

[0093] The second stable diffusion model provided in the specific embodiment of the present invention is a stable diffusion XL repair model (SDXL) that performs a mask suppression operation on the self-attention mechanism of the Unet module in the stable diffusion XL model;

[0094] Furthermore, a specific embodiment of the present invention provides a method for obtaining a second stable diffusion model, which is: building a stable diffusion XL model, expanding the Unet module of the model with additional mask input channels, and performing mask suppression on the self-attention scores during the reasoning process to obtain a stable diffusion XL repair model.

[0095] The method and device for eliminating objects associated with environment interaction perception provided by the specific embodiment of the present invention can be used in the following aspects:

[0096] 1. Intelligent film and television post-production. By automatically modeling object-environment associations and achieving object elimination, the efficiency of film and television post-production can be greatly improved and labor costs can be reduced. Post-production staff can easily edit and modify scenes, such as replacing / removing characters and props, changing backgrounds, etc.

[0097] 2. Virtual film and television production. In virtual film and television production, it is necessary to seamlessly integrate foreground elements such as real people and props with virtual scenes. Accurately modeling and eliminating the impact of foreground objects on the original shooting environment can better combine them with the virtual background and enhance visual realism.

[0098] 3. Visual effects production. Many visual effects require object editing and scene reconstruction based on the original stock footage. Object-environment association modeling based on SDXL can intelligently separate the required objects or background elements, providing more convenient and efficient materials for special effects synthesis.

[0099] 4. Film and television data processing. Object-environment association modeling of massive film and television footage can obtain rich multimodal data, including object masks, background images, environmental features, etc. These data can be used to construct data sets in the fields of film and television scene understanding, audio-visual information fusion, etc., and promote the research and development of artificial intelligence in the field of film and television art.

[0100] 5. Assistance in artistic creation. Object-environment association information can be used as a source of creative inspiration and material to help artists better understand the composition of film and television scenes. For example, scene combinations can be intelligently recommended based on the object-environment relationship, or objects can be flexibly replaced through background repair, inspiring more creative possibilities.

[0101] Embodiment 1: The specific steps of this embodiment are:

[0102] First, the original image is input into the first stable diffusion model (stable diffusion XL implicit inversion model), the self-attention features A of all upsampling layers are obtained at the middle time step (step 25), and the inverted noise hidden code is obtained at the final time step (step 50). There are 36 self-attention features A, 30 of which are Resolution, 6 Resolution.

[0103] Then, according to the object mask input by the user, the self-attention feature A is selected and the mean is calculated to obtain the self-attention feature B. Specifically, the object mask is first downsampled to the same size as the self-attention feature A, and then the coordinates of the non-zero points in the object mask at each resolution are calculated, and the slices corresponding to the points of the self-attention feature A are selected according to these coordinates. Finally, the mean of the selected self-attention feature A is calculated to obtain self-attention features B of different sizes, a total of 36, 30 of which are Resolution, 6 Resolution. Then take the average of the self-attention features B of the same size to get the self-attention features of 2 sizes , which is a self-attention feature The resolution is , another self-attention feature The resolution is .

[0104] Next, for self-attention features of different sizes Perform iterative multi-scale fusion. From the smallest size self-attention feature Start by initializing the fusion feature. For the self-attention feature B of size, the current self-attention feature The upsampled features obtained by the previous iteration are weighted and fused, and the fused features are adaptively thresholded and Gaussian smoothed to remove the parts smaller than the threshold. The fused features are then upsampled to obtain the current upsampled features, and the above process is repeated until multiple sizes of self-attention features are obtained. The fusion is completed. The final fused feature is the unified object and its affected mask.

[0105] Finally, the mask of the object and its influence and the noise hidden code are input into the second stable diffusion model (stable diffusion XL repair model) together. This model inputs the noisy hidden code into the Unet module for denoising. During the denoising process, the self-attention map generated by all the self-attention layers of the Unet module is subjected to mask suppression. Through the minimum value of the self-attention map, the position specified by the mask of the object and its influence is suppressed, thereby completing the elimination of the specified object and its influence position. After each denoising step, the denoising result and the original noise hidden code are masked and added to ensure the consistency of the non-edited area. Repeat the above denoising process 50 times to obtain the final result latent vector. Projecting the resulting latent vector back to the image space can obtain a repaired image with the object and its influence eliminated. The specific effect is as follows. Figure 2 As shown, the original image and the object mask are provided to the user. Using the method provided in this embodiment, the object and its impact mask are used to specify the area where the object is located and the area where the object has an impact, and the area with stronger correlation is closer to white in color; the output image represents the output after image elimination, in which the object and the impact of the object are eliminated and filled with reasonable background pixels.

[0106] This embodiment uses the inversion and repair capabilities of the stable diffusion XL model, extracts and fuses the complex relationship between the modeling object and the environment through self-attention features, and achieves high-quality object removal and scene repair. This method can be widely used in film and television post-production, visual effects, virtual production and other fields, greatly improving content production efficiency and creative space.

Claims

1. A method for eliminating objects associated with environmental interactive perception, characterized in that: include: The obtained original image is converted into an original latent vector, and based on the original latent vector, a first stable diffusion model is used to perform T sampling to obtain a plurality of first process latent vectors and a first noise latent code obtained by the last sampling; Obtain multiple self-attention features A output by the upsampling layers of different sizes in the Unet module of the first stable diffusion model in any iteration of 0.4T to 0.6T, extract the corresponding self-attention features from each self-attention feature A based on the obtained object mask, and take the average to obtain self-attention features of multiple sizes ; The self-attention features of multiple sizes Multi-scale fusion is performed from low scale to high scale to obtain the mask of the object and its influence. In the tensor of the current fusion feature obtained by each scale fusion, the value below the current scale mask threshold is replaced with 0 to obtain the current fusion threshold feature. The current fusion threshold feature is smoothed and upsampled before fusion at the next scale. A second stable diffusion model is constructed, wherein the second stable diffusion model includes a suppressed Unet module, wherein the suppressed Unet module is obtained by performing a mask suppression operation on the self-attention layer of the Unet module based on the mask of the object and its influence, and the first noise hidden code is sampled T times by the second stable diffusion model to obtain multiple process fusion hidden vectors and a final noise hidden code, wherein the fusion hidden vector obtained in each sampling process includes data within the mask range of the object and its influence in the second process hidden vector of the current sampling output by suppressing the Unet module and data outside the mask range of the object and its influence in the first process hidden vector of T-current sampling, and the final noise hidden code is decoded to obtain a final image with the image object eliminated.

2. The method for eliminating objects associated with environment interactive perception according to claim 1, characterized in that: Based on the obtained object mask, the corresponding self-attention feature is extracted from each self-attention feature A and the average is taken to obtain self-attention features of multiple sizes. ,include: The obtained object mask is downsampled so that the size of the object mask is consistent with the size of the self-attention feature A; Based on the non-zero point coordinates of the object mask that keeps the same size as the self-attention feature A, the self-attention features within the object range are extracted from each self-attention feature A, the extracted self-attention features are averaged to obtain the self-attention feature B, and the self-attention features B of the same size are averaged to obtain self-attention features of multiple sizes. .

3. The method for eliminating objects associated with environment interactive perception according to claim 1, characterized in that: The self-attention features of multiple sizes Multi-scale fusion from low scale to high scale is used to obtain masks of objects and their effects, including: S1. The self-attention feature of the current size Perform weighted fusion with the upsampled features obtained by the previous iteration to obtain the current fused features; S2. In the tensor of the current fused feature, replace the values ​​below the current size mask threshold with 0 to obtain the current fused threshold feature, use a 2D Gaussian filter kernel to smooth the current fused threshold feature, and use bilateral linear interpolation to upsample the smoothed current fused threshold feature to obtain the current upsampled feature; S3, iterate steps S1-S2 until self-attention features of multiple sizes The fusion is done, resulting in a mask of the object and its influence.

4. The method for eliminating objects associated with environment interactive perception according to claim 1 or 3, characterized in that: In the tensor of the current fused feature, the values ​​below the current size mask threshold are replaced with 0, where the current size mask threshold is a constant value or a locally varying value; When the current size mask threshold is a local change value, a method for constructing a local change value includes: Set the threshold constant C, sliding window size s, convolution kernel size to (1, 1, s, s), and convolution step size to s to ensure that there is no overlap between sliding windows; Using the convolution kernel to perform sliding window calculation on the tensor of the current fusion feature, obtain the local mean of the pixels in each window, construct a local mean tensor based on the local mean of the pixels in each window, and make the size of the local mean tensor the same as the size of the tensor of the current fusion feature through interpolation operation; The local change value is obtained by subtracting the local mean tensor after the weighted interpolation operation from the threshold constant.

5. The method for eliminating objects associated with environment interactive perception according to claim 1, characterized in that: The mask suppression operation is performed on the self-attention layer of the Unet module based on the mask of the object and its influence to obtain the suppressed Unet module, including: The mask input channel is extended to the Unet module, and the object and its affected mask are input into the self-attention layers of different sizes through the mask input channel, so that the self-attention scores output by the self-attention layers of different sizes are suppressed by the mask; Self-attention score for mask suppression for: , ,in, , They are the Query vector and Key vector in the conventional self-attention calculation, d is the dimension of the Key vector, is the mask of the object and its influence, To perform the minimum operation on each Q vector, copy the minimum value obtained along the dimension direction corresponding to the K vector. is element-wise multiplication.

6. The method for eliminating objects associated with environment interaction perception according to claim 1, characterized in that: The first noise hidden code is sampled T times by the second stable diffusion model to obtain the final noise hidden code, including: S1, the first noise hidden code is passed through the suppression Unet module to obtain the second process hidden vector of the current sub-sampling; S2, constructing a fused latent vector for the current sampling by combining the data within the mask range of the object and its influence in the second process latent vector of the current sampling with the corresponding data outside the mask range of the object and its influence in the first process latent vector of the current sampling T-current sampling; S3, iterate steps S1-S2 until T samplings are completed to obtain the final noise hidden code.

7. The method for eliminating objects associated with environment interactive perception according to claim 1 or 6, characterized in that: The method of constructing the fusion vector includes: Multiply the second-pass latent vector of the current sample by the binary object and its affected mask to obtain the second-pass suppressed latent vector; The first-process suppressed latent vector is obtained by multiplying the first-process latent vector with the reflexive binary object and its affected mask with the same noise level; The first process suppressed latent vector and the second process suppressed latent vector are added together to obtain the fusion vector of the current sample.

8. The method for eliminating objects associated with environment interaction perception according to claim 7, characterized in that: The fused latent vector of the sth sampling for: ,in, is the hidden vector of the second process obtained by sampling for the sth time, For the T The first process hidden vector obtained by -s sampling, is a binarization operation, A mask for the object and its influence.

9. An object elimination device for environment interactive perception association, characterized in that: include: It comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement the method for eliminating objects associated with environmental interactive perception as described in any one of claims 1-8.

Images