Irregular missing region image completion method based on fuzzy diffusion model

Abstract

The invention discloses an irregular missing region image completion method based on a fuzzy diffusion model, and the method comprises the following steps: S1, generating a noise image from random Gaussian noise, taking an input image and a binary mask image as the initial input of a denoising network, and setting the total denoising time step T of the model; s2, setting a time step S to be obtained in the next step; s3, adding noise to the input image according to a forward process to obtain a noise map of a known region; s4, integrating the noise image of the known area into the noise image through the binary mask image; s5, inputting the combined features of the noise image, the input image and the binary mask image into a denoising network to obtain a noise image of the next step; s6, taking the next step as the current step, and performing the next round of iteration under the condition; and S7, taking the final noise image as output to obtain a completion result. According to the image completion method, the precision and robustness of image completion are improved, and the image completion method has relatively high flexibility and practicability in practical application.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to an irregular missing region image completion method based on a fuzzy diffusion model. Background Art

[0002] Image completion technology aims to repair missing areas in images caused by various reasons in order to restore the integrity of the image. This technology has been widely used in digital image processing, computer vision and other fields, such as old photo restoration, medical image processing, video denoising and so on. However, existing image completion methods often face the problem of poor completion effect when facing missing areas with irregular shapes. Traditional image completion methods usually rely on the information of local neighborhoods to fill in the missing parts. This method performs well when dealing with small-scale missing areas with clear boundaries and regular shapes, but when dealing with missing areas with complex shapes and large areas, it is prone to problems such as texture discontinuity and unreasonable structure. In addition, existing technologies often ignore the information of images in the frequency domain, resulting in the failure to effectively remove high-frequency noise during the completion process, which in turn affects the final image quality. At the same time, due to the lack of full consideration of the multi-level fusion of image features and the differences between different missing areas, the overall consistency of the image is poor and the completion effect is not natural enough. Therefore, it is particularly important to develop an efficient image completion method that can overcome the above shortcomings and is suitable for irregular missing areas. Summary of the invention

[0003] 1. Technical Problems Solved

[0004] The technical problems to be solved by the present invention are the various problems mentioned in the above background technology, and a method for completing an irregular missing area image based on a fuzzy diffusion model is provided.

[0005] 2. Technical Solution

[0006] In order to solve the above technical problems, the present invention provides a technical solution: an irregular missing area image completion method based on a fuzzy diffusion model, comprising the following steps:

[0007] S1: Generate a noisy image from random Gaussian noise, and combine the input image and the binary mask image as the initial input of the denoising network, setting the total denoising time step T of the model;

[0008] S2: Set the time step S you want to get in the next step;

[0009] S3: adding noise to the input image according to the forward process to obtain a noise map of the known area;

[0010] S4: Integrate the noise map of the known area into the noise image through the binary mask image;

[0011] S5: Input the combined features of the noise image, input image, and binary mask image into the denoising network to obtain the noise map for the next step;

[0012] S6: Take the next step as the current step and perform the next round of iteration under this condition; determine whether the current step is the last step. If so, execute S7; otherwise, repeat S2 to S5 until the last step;

[0013] S7: Take the final noisy image as output to obtain the completion result.

[0014] As an improvement, in step S3, while adding noise, high-frequency information is suppressed in the frequency domain, and high-frequency information is suppressed in the frequency domain through discrete cosine transform (DCT). Specifically, DCT is applied to the original data, and then fixed noise is added in the frequency domain, and then IDCT is applied back to the spatial domain.

[0015] As an improvement, it also includes a multi-scale feature fusion module for enhancing the consistency of generation of complete areas and missing areas. The module divides the input features into two equal features according to the number of feature channels, extracts shallow features and deep features respectively, concatenates and fuses the features of the two scales, and uses jump connections to add the original input features and the fused features to obtain the final output features.

[0016] As an improvement, it also includes a visual linear space module for solving the problem of differences in different missing areas. The module converts two-dimensional image information into one-dimensional sequence information, and models the one-dimensional sequence information from four different directions according to the rows and columns in the original image. The sequence modeling information in four different directions is then stacked and reorganized into one sequence modeling.

[0017] As an improvement, the fusion steps of the multi-scale feature fusion module are as follows:

[0018] a51, divide the input features into two equal parts;

[0019] a52, perform different convolution processing on the two parts of features respectively;

[0020] a53, stitching the processed features;

[0021] a54. Use skip connection to add the concatenated features to the original input features.

[0022] As an improvement, the implementation steps of the visual linear space module are as follows:

[0023] a61, converting the input image into a one-dimensional sequence;

[0024] a62, modeling along the row direction, along the column direction, in the reverse row direction, and in the reverse column direction respectively;

[0025] a63. Stack and reorganize the modeling information in four directions.

[0026] 3. Beneficial Effects

[0027] The advantages of the present invention compared with the prior art are:

[0028] The image completion method of the present invention can effectively restore details and maintain the consistency and naturalness of the image when processing irregular missing areas in the image by introducing a denoising network and a multi-scale feature fusion module. The method first uses Gaussian noise to initialize the image completion process, and gradually removes the noise through a set denoising time step, and finally achieves the purpose of image completion. In particular, by suppressing the operation of high-frequency information in the frequency domain, unnecessary interference can be reduced in the completion process, thereby improving the quality of the completed image. The multi-scale feature fusion module fuses shallow and deep features and uses a skip connection mechanism so that the completed image retains the edge and texture information of the original image and can fill the missing part, thereby enhancing the realism of the completion result. In addition, the visual linear space module takes into account the differences between different missing areas, and by modeling and reorganizing the image information in multiple directions, the algorithm can better adapt to various missing modes, thereby providing accurate image completion functions in a wider range of scenarios. In summary, the method not only improves the accuracy and robustness of image completion, but also has high flexibility and practicality in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the irregular missing area image completion method based on the fuzzy diffusion model of the present invention.

[0030] Figure 2 It is a schematic diagram of the denoising network architecture of the irregular missing area image completion method based on the fuzzy diffusion model of the present invention.

[0031] Figure 3 This is a restoration effect diagram of the irregular missing area image completion method based on the fuzzy diffusion model of the present invention. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] Combined with Figure 1 and attached Figure 2First, a noise image is generated from random Gaussian noise, and the input image and the corresponding binary mask image are combined as the initial input of the denoising network, and the total denoising time step T of the model is set (step S1). Next, the time step S that is desired to be obtained in the next step is set (step S2). Then, the input image is added with noise according to the forward process to obtain the noise map of the known area. While adding noise, high-frequency information is suppressed in the frequency domain. Specifically, by applying discrete cosine transform (DCT) to the original data, fixed noise is added in the frequency domain, and then the inverse discrete cosine transform (IDCT) is applied back to the spatial domain (step S3). After that, the noise map of the known area is integrated into the noise image through the binary mask image (step S4). Subsequently, the combined features of the noise image, the input image, and the binary mask image are input into the denoising network to obtain the noise map of the next step (step S5). On this basis, the next step is taken as the current step, and the next round of iteration is performed under this condition to determine whether the current step is the last step. If so, step S7 is entered, otherwise steps S2 to S5 are repeated until the last step (step S6). Finally, the final noise image is taken as output to obtain the completion result (step S7).

[0034] In order to enhance the generation consistency of complete areas and missing areas, the present invention also introduces a multi-scale feature fusion module. This module divides the input features into two equal parts according to the number of feature channels, extracts shallow features and deep features respectively, and splices and fuses the features of these two scales, and then uses a jump connection to add the original input features and the fused features to obtain the final output features. The specific steps are: first divide the input features into two equal parts (step a51), then perform different convolution processing on the two parts of the features respectively (step a52), splice the processed features (step a53), and use a jump connection to add the spliced ​​features to the original input features (step a54).

[0035] In addition, in order to overcome the difference problem of different missing areas, the present invention also uses a visual linear space module. This module converts the two-dimensional image information into one-dimensional sequence information, and models the one-dimensional sequence information from four different directions according to the rows and columns in the original image, and then stacks and reorganizes the sequence modeling information in four different directions into one sequence modeling. The specific steps are: first convert the input image into a one-dimensional sequence (step a61), then model along the row direction, along the column direction, inverse row direction, and inverse column direction (step a62), and finally stack and reorganize the modeling information in the four directions (step a63).

[0036] Through the above steps, the method of the present invention can effectively complete the image of irregular missing areas, thereby improving the quality and consistency of the completed image.

[0037] Embodiment 1

[0038] Reference Figure 3 In the first row, for facial images with relatively fixed facial features, the restoration method of the present invention can generate a facial structure that is harmonious with the original image in the missing area based on limited information. In the second and third rows, for the changeable and flexible edge structures of indoor and outdoor buildings, the method of the present invention combines other complete areas to infer the restoration results that are consistent with the overall image style. In all these scenes, it can be seen that the method of the present invention can effectively combine other complete areas when targeting irregular missing areas to obtain harmonious and realistic restoration results.

[0039] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0040] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0041] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. An irregular missing region image completion method based on a fuzzy diffusion model, characterized in that: The following steps are involved: S1: Generate a noisy image from random Gaussian noise, and combine the input image and the binary mask image as the initial input of the denoising network, setting the total denoising time step T of the model; S2: Set the time step S you want to get in the next step; S3: adding noise to the input image according to the forward process to obtain a noise map of the known area; S4: Integrate the noise map of the known area into the noise image through the binary mask image; S5: Input the combined features of the noise image, the input image, and the binary mask image into the denoising network to obtain the noise map for the next step; S6: Take the next step as the current step and perform the next round of iteration under this condition; determine whether the current step is the last step. If so, execute S7; otherwise, repeat S2 to S5 until the last step; S7: Take the final noisy image as output to obtain the completion result.

2. The irregular missing area image completion method based on fuzzy diffusion model according to claim 1, characterized in that: In step S3, while adding noise, high-frequency information is suppressed in the frequency domain, and the high-frequency information is suppressed in the frequency domain by discrete cosine transform (DCT). Specifically, DCT is applied to the original data, and then fixed noise is added in the frequency domain, and then IDCT is applied back to the spatial domain.

3. The irregular missing area image completion method based on fuzzy diffusion model according to claim 1, characterized in that: It also includes a multi-scale feature fusion module for enhancing the consistency of generating complete areas and missing areas. The module divides the input features into two equal features according to the number of feature channels, extracts shallow features and deep features respectively, concatenates and fuses the features of the two scales, and uses jump connections to add the original input features and the fused features to obtain the final output features.

4. The irregular missing area image completion method based on fuzzy diffusion model according to claim 1, characterized in that: It also includes a visual linear space module for solving the problem of differences in different missing areas. The module converts two-dimensional image information into one-dimensional sequence information, and models the one-dimensional sequence information from four different directions according to the rows and columns in the original image. The sequence modeling information in four different directions is then stacked and reorganized into a sequence modeling.

5. The irregular missing area image completion method based on fuzzy diffusion model according to claim 3 is characterized in that: The fusion steps of the multi-scale feature fusion module are as follows: a51, divide the input features into two equal parts; a52, perform different convolution processing on the two parts of features respectively; a53, stitching the processed features; a54. Use skip connection to add the concatenated features to the original input features.

6. The irregular missing region image completion method based on fuzzy diffusion model according to claim 4, characterized in that: The implementation steps of the visual linear space module are as follows: a61. Convert the input image into a one-dimensional sequence; a62, modeling along the row direction, along the column direction, in the reverse row direction, and in the reverse column direction respectively; a63. Stack and reorganize the modeling information in four directions.

Images