Shielding object moving and editing method and system based on diffusion model

Abstract

The invention discloses a shielding object moving and editing method and system based on a diffusion model, and belongs to the technical field of computer vision and image processing. Comprising the following steps: acquiring an original image and a visible area mask and performing preprocessing to generate a standard input image and a standard mask; processing the standard input image and the standard mask through a de-occlusion branch, guiding a diffusion process by using a cross attention mechanism and a self-attention mechanism, and generating a de-occlusion object image; and obtaining a target position, inputting the original image, the visible area mask and the target position into a moving branch for processing, introducing a potential space adjustment operation, and generating a shielding object moving image under the guidance of a local text condition. According to the method, the integrity and authenticity of the shielded object after editing can be kept, natural fusion of the target position and traceless reconstruction of the original position are realized, and the problems of insufficient content completion, unnatural fusion of the target position and residual artifacts of the original position in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a method and system for moving and editing occluded objects based on a diffusion model. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] The movement of objects in image editing is a common requirement. However, when processing occluded objects, the occluded parts of the objects need to be completed. After the objects are moved to the target position, they need to be naturally integrated with the surrounding environment, and the background content of the original position area needs to be reasonably reconstructed. This makes the movement of objects in image editing still face great challenges.

[0004] In the existing technology, two independent networks are relied on to predict the complete mask of the occluded object and fill in the content, but the generated content usually lacks details and naturalness. In recent years, the diffusion model has provided new opportunities to solve this problem with its powerful generation ability; however, existing diffusion-based editing methods mainly focus on content dragging, which controls the movement trajectory of the object through the drag points set by the user, which is a highly interactive method.

[0005] For example, the drag point guidance mechanism is extended to the diffusion model to enhance the generation capability and robustness; or, the editing flexibility and the realism of the generation are further improved to achieve high-quality content operations in complex scenes. However, these methods are mainly aimed at content deformation or local dragging, and there are still significant deficiencies in the movement of the entire object, especially the movement of occluded objects. When the moving object is partially occluded, these methods have difficulty in reconstructing the occluded part, and they also perform poorly in terms of fusion of the target position with the background.

[0006] Although existing text-guided editing, drag-and-drop editing, and occlusion completion methods have made breakthroughs in their respective tasks, they still have the following problems when dealing with the task of moving occluded objects:

[0007] (1) Separation of content completion and movement tasks: Current methods find it difficult to effectively combine the reconstruction of occluded parts with the object movement task, resulting in incomplete or unnatural generated results.

[0008] (2) Reasonableness of target position: When the object fusion is completed at the target position, there is a lack of adaptability to the surrounding environment, resulting in the inconsistency between the moved object and the background.

[0009] (3) Residual problem of original position: After the object moves, the background reconstruction of the original position usually has residual artifacts, which reduces the authenticity of the overall visual effect. Summary of the invention

[0010] In order to address the deficiencies of the prior art, the present invention provides a method, system, electronic device, computer-readable storage medium and computer program product for moving and editing occluded objects based on a diffusion model. Through a dual-branch design, the de-occlusion and movement tasks of the occluded objects are simultaneously completed to achieve high-quality image editing effects.

[0011] In a first aspect, the present invention provides a method for moving and editing occluded objects based on a diffusion model;

[0012] A method for moving and editing occluded objects based on a diffusion model, comprising:

[0013] Obtain the original image containing the occluded object and the corresponding visible area mask and perform preprocessing to generate a standard input image and a standard mask;

[0014] The standard input image and the standard mask are processed by the trained deocclusion branch, the visible area is initialized by the background color filling strategy, and the diffusion process is guided by the cross attention mechanism and the self-attention mechanism to generate a deoccluded object image;

[0015] The target position is obtained, the original image, the visible area mask and the target position are input into the trained moving branch for processing, a latent space adjustment operation is introduced, and a moving image of the occluded object is generated under the guidance of local text conditions.

[0016] In some embodiments, the de-occlusion branch includes a variational autoencoder, a DDIM inversion model, a denoising diffusion model U-net, a latent variable preserving layer and a variational autodecoder connected in sequence, and the denoising diffusion model U-net includes a cross-attention mechanism and a self-attention mechanism.

[0017] In some embodiments, the mobile branch includes a variational autoencoder, a DDIM inversion model, a denoising diffusion model U-net, a latent variable preserving layer, a local text guidance layer, a latent variable optimization layer and a variational autodecoder connected in sequence, wherein the denoising diffusion model U-net includes a cross-attention mechanism and a self-attention mechanism.

[0018] In some embodiments, processing the standard input image and the standard mask by the trained deocclusion branch includes:

[0019] Encoding the standard input image using a pre-trained variational autoencoder to generate a latent variable; inverting the latent variable using a denoising diffusion implicit model to obtain an intermediate noise latent variable generated by the inversion;

[0020] Initializing and filling the latent variable using the standard mask, processing the filling result through the denoising diffusion model U-net, and obtaining the generated latent variable through the guidance of the cross attention mechanism and the self-attention mechanism;

[0021] A latent variable preservation strategy is introduced to replace the visible area in the generated latent variable with the inversion result using the intermediate noise latent variable and the standard mask generated by inversion.

[0022] The replaced generated latent variables are decoded into the deoccluded object image through a variational self-decoder.

[0023] In some embodiments, inputting the original image, the visible area mask, and the target position into the trained movement branch for processing includes:

[0024] The original image is encoded by a pre-trained variational autoencoder to generate an initial latent variable; the initial latent variable is inverted by a denoising diffusion implicit model and an intermediate key-value pair is stored;

[0025] The inverted initial latent variables are input into the denoising diffusion model U-net, the initial latent variables are processed by the cross attention mechanism, the local text guidance strategy is introduced, and the latent variable generation content of the target position is adjusted; the query vector of the adjusted initial latent variable is replaced with the intermediate key-value pair by the self-attention mechanism to update the background information; the latent variables are introduced, and the latent variable gap between the target object and the target area is minimized by the latent variable optimization strategy;

[0026] A noise filling strategy is used to guide the background information to perform regional repair at the original position of the target object. The repaired image is decoded by a variational self-decoder to obtain a moving image of the occluded object.

[0027] In some embodiments, the de-occlusion branch introduces a low-rank adaptation technique to perform style consistency supervision on the visible region.

[0028] In a second aspect, the present invention provides an occluded object movement and editing system based on a diffusion model;

[0029] A diffusion model-based occlusion object movement and editing system, comprising:

[0030] The preprocessing module is configured to: obtain an original image containing an occluded object and a corresponding visible area mask and perform preprocessing to obtain a standard input image and a standard mask;

[0031] A deocclusion module is configured to: process the standard input image and the standard mask through the trained deocclusion branch, initialize the visible area through the background color filling strategy, and use the cross attention mechanism and the self-attention mechanism to guide the diffusion process to generate a deoccluded object image;

[0032] The moving module is configured to: obtain the target position, input the original image, the visible area mask and the target position into the trained moving branch for processing, introduce the latent space adjustment operation, and generate a moving image of the occluded object guided by the local text condition. In a third aspect, the present invention provides an electronic device;

[0033] An electronic device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned method for moving and editing occluding objects based on a diffusion model.

[0034] In a fourth aspect, the present invention provides a computer-readable storage medium;

[0035] A computer-readable storage medium stores a computer program / instruction thereon, which, when executed by a processor, implements the steps of the above-mentioned method for moving and editing occluded objects based on a diffusion model.

[0036] In a fifth aspect, the present invention provides a computer program product;

[0037] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned diffusion model-based occluding object movement and editing method.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The technical solution provided by the present invention generates high-quality content completion through a de-occlusion branch; applies a background color filling strategy in the initial stage of the diffusion process to fill the potential representation of the occluded area, thereby generating consistent target content; uses a progressive update mask mechanism to ensure that the content generated by the model gradually fits the true shape of the object; introduces a LoRA module (low-rank adaptation) to optimize the attention module of the diffusion model so that the generated occluded part is consistent with the visible part of the object; and ensures that the occluded object maintains integrity and authenticity after editing.

[0040] 2. The technical solution provided by the present invention sets local text condition guidance at the target position, and makes the mobile content generated by the model match the background of the target position by adding semantic description. The latent space optimization technology is used to minimize the difference between the latent representation of the object and the target position, ensuring the natural fusion of the generated content and the target area, eliminating the sense of incongruity.

[0041] 3. The technical solution provided by the present invention utilizes the mask constrained diffusion process and, through the mask constraint of the self-attention module, limits the model to focus only on the relevant area at the target position to avoid generating redundant content or artifacts; effectively avoids residual artifacts at the original position after the object moves, makes the background of the reconstructed area consistent with the surrounding environment, and realizes seamless reconstruction of the original position. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0043] Figure 1 A schematic flow chart of a method for moving and editing an occluding object based on a diffusion model provided in an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a network architecture of a diffusion model-based occluded object moving and editing method provided in an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of the effect of the processing process and key mechanisms of the de-occlusion branch provided by an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of the key mechanism effect of the mobile branch in the object movement task provided by the embodiment of the present invention;

[0047] Figure 5 A schematic diagram of the actual application effect provided by an embodiment of the present invention;

[0048] Figure 6 The present invention is another practical application effect diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0050] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0052] Embodiment 1

[0053] The existing technology has problems in the task of moving occluded objects, such as insufficient content completion, unnatural target position fusion, and residual artifacts in the original position; therefore, this embodiment provides an occluded object movement and editing method based on a diffusion model, constructs a unified framework combining de-occlusion and object movement, and fully utilizes the generation capability of the diffusion model to achieve high-quality occluded object movement tasks.

[0054] Next, combine Figure 1-Figure 6 , a method for moving and editing an occluding object based on a diffusion model disclosed in this embodiment is described in detail. The method for moving and editing an occluding object based on a diffusion model comprises the following steps:

[0055] S1. Obtain an original image containing an occluded object, a corresponding visible area mask, and a target position, preprocess the original image containing the occluded object and the corresponding visible area mask, and obtain a standard input image and a standard mask.

[0056] Here, the original image of the occluded object, the corresponding visible area mask and the target position are input by the user, the occlusion range of the object is marked by the original image of the occluded object and the corresponding visible area mask, and the end point of the object movement is determined by the target position set by the user.

[0057] In this embodiment, the preprocessing of the original image containing the blocked object and the corresponding visible area mask is specifically as follows:

[0058] Through the original image I s and visible area mask M v , calculate the cropping center b and side length r of the occluded object (target object), perform cropping operations on the original image containing the occluded object and the corresponding visible area mask, and generate a standard input image and standard mask It is expressed as:

[0059]

[0060] S2. The standard input image and standard mask are processed through the trained deocclusion branch, the occluded area is initialized through the background color filling strategy, and the cross-attention mechanism and self-attention mechanism are used to guide the diffusion process to generate a deoccluded object image.

[0061] In this embodiment, the missing part of the occluded object is completed by the deocclusion branch, and the deocclusion branch includes a variational autoencoder, a DDIM inversion model, a denoising diffusion model U-net, a latent variable preserving layer and a variational autodecoder connected in sequence. The denoising diffusion model U-net includes a cross-attention mechanism and a self-attention mechanism.

[0062] Here, the variational autoencoder, the denoising diffusion model U-net and the variational autodecoder are all provided by Stable Diffusion, and this embodiment does not make any improvements to them, so they will not be described here.

[0063] The visible area is initialized by the background color filling strategy, and the cross-attention and self-attention mechanisms are used to guide the diffusion process to gradually generate content consistent with the target object. At the same time, the deocclusion branch introduces a mask update mechanism and a potential hold operation to ensure that the generated result is gradually aligned with the true shape of the object during the entire diffusion process. In addition, this branch introduces the Low-Rank Adaptation (LoRA) technology to fine-tune the attention module in the diffusion model and supervise the style consistency of the visible area to ensure that the generated content is consistent with the features of the visible part.

[0064] As an implementation mode, S2 specifically includes:

[0065] S201, use the pre-trained variational autoencoder E to the standard input image Encode and generate latent variables The latent variables are denoised by the DDIM Perform inversion to obtain the inversion intermediate noise latent variable

[0066] Among them, the latent variable It is expressed as:

[0067]

[0068] S202, using standard mask The latent variables are filled by background color strategy Initialize and fill the results generated by the inversion to obtain the intermediate noise latent variables It is expressed as:

[0069]

[0070] In the formula, J T represents a random pure color background plus the noise corresponding to T steps, {t=1, 2, 3, ...., T}, T represents the maximum number of iterations, i.e. the first iteration.

[0071] S203, the intermediate noise latent variable The input is processed by the denoising diffusion model U-net, guided by the cross attention mechanism and self-attention mechanism, to generate the complete shape of the occluded object, and the output generates the latent variable

[0072] Specifically, in During the denoising process, the mean of the cross attention map is gradually updated and the mean of the self-attention map Calculation area Guides the generation of the complete shape of the target object.

[0073] Here, the mean of the cross attention map is the mean of the cross attention map generated by the cross attention mechanism in the denoising diffusion model U-net after processing the input features, and the mean of the self-attention map It is the mean of the self-attention map generated after the self-attention mechanism in the denoising diffusion model U-net processes the input features. In this embodiment, the cross attention mechanism and the self-attention mechanism are not improved, and the specific calculation process is not repeated here.

[0074] Among them, the area It is expressed as:

[0075]

[0076] By extracting the last time step To guide the self-attention mechanism in the denoising model U-net in the next time step, forcing the self-attention to only Calculate the inner area.

[0077] S204, introduce a latent variable preservation strategy, using the intermediate noise latent variable generated by inversion and standard mask Generate latent variables at each step The visible area in is replaced by the inversion result; it is expressed as:

[0078]

[0079] Based on this, the consistency of the visible part can be ensured.

[0080] S205, decoding the replaced latent variable into a de-occluded object image through a variational self-decoder It is expressed as:

[0081]

[0082] In this embodiment, the image patch where the target object is located is cropped and adjusted to a standard size as input, and the latent variable of the target object is initialized and filled using the visible area mask. At the same time, the consistency of the visible part is ensured by introducing a latent variable preservation strategy. During the diffusion process, the gradually updated cross-attention map is used to guide the generation of the shape of the complete object, and a unified color filling strategy is used to reduce background interference. In order to ensure that the style of the generated result is consistent with the original object, LoRA is used to fine-tune the style of the diffusion model, and finally decoded to generate a complete de-occluded object image.

[0083] Through the above process, the deocclusion branch can generate a deoccluded target object image with consistent style and complete shape.

[0084] For example, this embodiment is Figure 3 The processing and key mechanisms of the deocclusion branch are shown. Figure 3 (a) and (b) in Figure 3 show the process of obtaining the cropped object image and mask, where: Figure 3 The white area in (b) represents the occluded area; Figure 3 (c) to (f) show the results of different variants of the deocclusion branch on completing the occluded area. Figure 3 (c) is the initial generation result after filling the occluded area with random noise, showing the problem of inconsistent content and rough texture; Figure 3 (d) introduces a background color filling strategy, which significantly improves the initial generation quality of the occluded area content; Figure 3 (e) shows that under the guidance of the gradually updated mask, the generated result gradually fits the real shape of the object, and the generated content is more accurate and detailed; Figure 3 (f) is the complete deocclusion branch generation result. By introducing mask constraints and cross-attention mechanism, high-quality completion content that is consistent with the visible area is finally generated; Figure 3 (g) in the figure shows the gradual update of the mask based on the refined cross-attention map as the diffusion process progresses in time, illustrating the key role of the progressive guidance mechanism of the mask in the de-occlusion task.

[0085] S3. The original image, visible area mask and target position are input into the trained moving branch for processing, and the latent space adjustment operation is introduced to generate a moving image of the occluded object guided by the local text condition.

[0086] In this embodiment, the de-occluded object is moved to the target position by moving branches, and the moving branches include a variational autoencoder, a DDIM inversion model, a denoising diffusion model U-net, a latent variable preserving layer, a local text guidance layer, a latent variable optimization layer and a variational autodecoder connected in sequence, wherein the denoising diffusion model U-net includes a cross-attention extreme and a self-attention mechanism.

[0087] The moving branch uses a local text conditional guidance mechanism to make the model focus on the target area during the generation process and generate content that naturally blends with the surrounding background; a latent space optimization strategy is adopted to gradually adjust the generation results to ensure that the moved object matches the background of the target area and eliminate the sense of incongruity; finally, by combining de-occlusion with the moving results, a complete and natural target image is output, and at the same time, a reasonable background reconstruction of the original position area of ​​the object is performed to eliminate artifacts and generate a high-quality final result.

[0088] As an implementation method, S3 includes:

[0089] S301, the original image I is trained by the pre-trained variational autoencoder s Encode to generate the initial latent variable Z s ; Invert the initial latent variables through the DDIM inversion model and store the intermediate key-value pairs used to guide the diffusion process

[0090] Among them, the initial latent variable after inversion is expressed as:

[0091]

[0092] In the formula, ∈ represents random noise, Z inv,T represents the noise latent variable of the T-step obtained by inversion, M v Represents the visible area mask.

[0093] Intermediate key-value pairs The DDIM inversion model is automatically generated during the inversion process.

[0094] S302, Z T Input denoising diffusion model U-net, and use the cross attention mechanism in U-net to invert the initial latent variable Z T Processing is performed, and a local text guidance strategy is introduced to freely guide the classifier to adjust the latent variables at the target position to generate content; it is expressed as:

[0095]

[0096] In the formula, ∈ pred，t represents the noise map predicted under the local text guidance strategy, ∈ θ (Z t , t) represents the noise map predicted without text prompts, ω represents the weight coefficient, Q represents the local mask, ∈ θ (Z t , t, c) represents the noise map predicted by the text prompt.

[0097] S303, replace the query vector in the self-attention mechanism with the intermediate key-value pair obtained by inversion, and use the replaced self-attention mechanism to ∈ pred，t Processing is performed to update the background information of the input latent variables.

[0098] S304: Introduce the latent variables generated in S204 Through the latent variable optimization strategy, the latent variable Z′ is obtained with the goal of minimizing the latent variable gap between the target object and the target area. t . It is expressed as:

[0099]

[0100] In the formula, represents the optimization objective, Crop(Z′ t , g, r′) represents the latent variable of the target area, L-Resize represents the resized incomplete object latent variable, Represents the complete shape of the generated target.

[0101] S305, guide the background information to perform regional repair at the original position of the target object through the noise filling strategy, decode the repaired image Zt through the decoder D, and obtain the moving image of the occluded object; expressed as:

[0102]

[0103] The moving branch achieves precise movement of the target object and peak-free fusion of the background by combining latent variable initialization, optimization and local guidance.

[0104] In this embodiment, the moving branch and the removing occlusion branch are arranged in parallel.

[0105] For example, in this embodiment, Figure 4 We demonstrate different variations of the movement branch in an object movement task and their effects on the results. Figure 4 (a) and (d) are the input source images, showing the movement of the object between the initial position and the target position, respectively. The yellow arrows indicate the starting and ending positions of the object. Figure 4 (b) and (e) show the results of directly adjusting the position of the object. At this time, there is an obvious sense of disharmony between the generated object and the background, especially blurring and artifacts appearing in the edge area; Figure 4 (c) in the figure effectively alleviates the problem of edge degradation after the object moves by introducing the latent space adjustment operation, so that the quality of the object at the target position is significantly improved; Figure 4 (e) and (f) in the figure further compare the effect of using local text conditional guidance. Figure 4 (e) in Figure 1 does not use text guidance, and the object appears unnatural against the background. Figure 4 In (f), under the guidance of local text conditions, the moved object blends naturally with the surrounding environment, the edge details are clearer and the overall visual effect is more realistic and coordinated.

[0106] pass Figure 5 and Figure 6 The final effect generated by the method and its performance in different scenarios are shown in the task of moving occluded objects. Figure 5 In , the originally occluded object is processed by the deocclusion branch, and the completed part is highly consistent with the actual shape and texture of the object, with rich and natural details and no obvious artifacts or missing areas; Figure 6 The moved object blends naturally with the surrounding background at the target location. Through the guidance of local text conditions and latent space optimization, the color, lighting and edge transition of the object are consistent with the target location environment, which is highly coordinated visually without any abruptness. At the same time, after the object moves, the background of its original location area is reconstructed through mask constraints. The generated background content is reasonable and natural, and can be seamlessly connected with the surrounding environment without any traces or artifacts of residual objects.

[0107] Embodiment 2

[0108] This embodiment discloses a diffusion model-based occluding object movement and editing system, including:

[0109] A preprocessing module is configured to: obtain an original image containing an occluded object and a corresponding occluded region mask and perform preprocessing to obtain a potential representation of the object;

[0110] A deocclusion module is configured to: process the object potential representation through the trained deocclusion branch, initialize the occlusion area through the background color filling strategy, and use the cross attention mechanism and the self-attention mechanism to guide the diffusion process to generate a deoccluded object image;

[0111] The moving module is configured to: obtain the target position, input the de-occluded object image and the target position into the trained moving branch for processing, introduce a latent space adjustment operation, and generate an occluded object moving image guided by local text conditions.

[0112] It should be noted that the above-mentioned preprocessing module, de-occlusion module and movement module correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the above-mentioned modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Embodiment 1. It should be noted that the above-mentioned modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0113] Embodiment 3

[0114] Embodiment 3 of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above-mentioned method for moving and editing occluded objects based on the diffusion model are completed.

[0115] Embodiment 4

[0116] A fourth embodiment of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned method for moving and editing occluding objects based on a diffusion model are completed.

[0117] Embodiment 5

[0118] Embodiment 5 of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned diffusion model-based occluding object movement and editing method.

[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0122] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for moving and editing occluded objects based on a diffusion model, characterized in that: include: Obtain the original image containing the occluded object and the corresponding visible area mask and perform preprocessing to generate a standard input image and a standard mask; The standard input image and the standard mask are processed by the trained deocclusion branch, the visible area is initialized by the background color filling strategy, and the diffusion process is guided by the cross attention mechanism and the self-attention mechanism to generate a deoccluded object image; The target position is obtained, the original image, the visible area mask and the target position are input into the trained moving branch for processing, a latent space adjustment operation is introduced, and a moving image of the occluded object is generated under the guidance of local text conditions.

2. The method for moving and editing occluded objects based on a diffusion model as claimed in claim 1, characterized in that: The de-occlusion branch includes a variational autoencoder, a DDIM inversion model, a denoising diffusion model U-net, a latent variable preserving layer and a variational autodecoder connected in sequence, and the denoising diffusion model U-net includes a cross-attention mechanism and a self-attention mechanism.

3. The method for moving and editing occluded objects based on a diffusion model as claimed in claim 1, characterized in that: The mobile branch includes a variational autoencoder, a DDIM inversion model, a denoising diffusion model U-net, a latent variable preserving layer, a local text guidance layer, a latent variable optimization layer and a variational autodecoder connected in sequence, wherein the denoising diffusion model U-net includes a cross-attention mechanism and a self-attention mechanism.

4. The method for moving and editing occluded objects based on a diffusion model as claimed in claim 1, characterized in that: Processing the standard input image and the standard mask by the trained deocclusion branch includes: Encoding the standard input image using a pre-trained variational autoencoder to generate a latent variable; inverting the latent variable using a denoising diffusion implicit model to obtain an intermediate noise latent variable generated by the inversion; Initializing and filling the latent variable using the standard mask, processing the filling result through the denoising diffusion model U-net, and obtaining the generated latent variable through the guidance of the cross attention mechanism and the self-attention mechanism; A latent variable preservation strategy is introduced to replace the visible area in the generated latent variable with the inversion result using the intermediate noise latent variable and the standard mask generated by inversion. The replaced generated latent variables are decoded into the deoccluded object image through a variational self-decoder.

5. The method for moving and editing occluded objects based on a diffusion model as claimed in claim 1, characterized in that: Inputting the original image, the visible area mask and the target position into the trained moving branch for processing comprises: The original image is encoded by a pre-trained variational autoencoder to generate an initial latent variable; the initial latent variable is inverted by a denoising diffusion implicit model and an intermediate key-value pair is stored; The inverted initial latent variables are input into the denoising diffusion model U-net, the initial latent variables are processed by the cross attention mechanism, the local text guidance strategy is introduced, and the latent variable generation content of the target position is adjusted; the query vector of the adjusted initial latent variable is replaced with the intermediate key-value pair by the self-attention mechanism to update the background information; the latent variables are introduced, and the latent variable gap between the target object and the target area is minimized by the latent variable optimization strategy; A noise filling strategy is used to guide the background information to perform regional repair at the original position of the target object. The repaired image is decoded by a variational self-decoder to obtain a moving image of the occluded object.

6. The method for moving and editing occluded objects based on a diffusion model as claimed in claim 1, characterized in that: The deocclusion branch introduces a low-rank adaptation technique to supervise the style consistency of the visible region.

7. The occluded object moving and editing system based on the diffusion model is characterized by: include: The preprocessing module is configured to: obtain an original image containing an occluded object and a corresponding visible area mask and perform preprocessing to obtain a standard input image and a standard mask; A deocclusion module is configured to: process the standard input image and the standard mask through the trained deocclusion branch, initialize the visible area through the background color filling strategy, and use the cross attention mechanism and the self-attention mechanism to guide the diffusion process to generate a deoccluded object image; The moving module is configured to: obtain the target position, input the original image, the visible area mask and the target position into the trained moving branch for processing, introduce the latent space adjustment operation, and generate a moving image of the occluded object guided by the local text condition.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method for moving and editing occluding objects based on a diffusion model as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the method for moving and editing occluded objects based on a diffusion model as described in any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method for moving and editing occluded objects based on a diffusion model as described in any one of claims 1 to 6 are implemented.

Images