Image restoration method and system guided by image structure and texture information
Through an integrated three-encoder network architecture, combining texture and structural information, the distortion problem of neural networks when repairing large-area damaged areas is solved, the image repair quality is improved, and the repaired image is ensured to be natural and realistic.
Patent Information
- Application Number
- CN202411203742.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-30
AI Technical Summary
When neural network models are repaired in large areas of damaged areas or complex images, they are prone to distorted structures and blurred textures, resulting in a degradation of repair quality.
Using an integrated three-encoder network architecture, the structure-oriented texture synthesis and texture-oriented structure repair are synchronized through the damaged area guidance of the image, and the STIM module, GIU unit and MLRN network are used to combine the Texture information encoder and structure information encoder, remapping and interaction to extract image features.
It effectively improves the quality of image repair, ensuring that the restored image is visually highly unified with the undamaged parts, and appears more natural and realistic.
Smart Images

Figure CN118710557B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image restoration method and system guided by image structure and texture information. Background Art
[0002] Image restoration technology is a highly complex and creative computational process that goes beyond simply "painting" damaged parts, but requires a deep understanding and intelligent analysis of damaged images. The core of this technology is the ability to identify and simulate patterns and structures in images to create pixels that are visually perfect with the original material, restoring the integrity and beauty of the image.
[0003] Traditional image restoration methods include manual hand-drawn image restoration and the use of traditional mathematical algorithms to process images. Manual hand-drawn image restoration usually requires a considerable amount of time and professional skills. At the same time, the subjective judgment and personalized style of the restorer may lead to differences between different restoration results; some use mathematical algorithms to restore images. These methods find it difficult to capture high-level semantic information in the image. When faced with noise, distortion or large deformation in the image, the robustness of traditional methods may be relatively poor, and they are prone to produce unsatisfactory restoration results.
[0004] With the rapid development of artificial intelligence, image restoration technology based on deep learning has made significant progress. In particular, the application of convolutional neural network (CNN) in this field has greatly promoted the advancement of technology. CNN can capture the deep features of the image by learning a large number of data samples, which makes the restoration process more accurate and efficient.
[0005] In practical applications, this technology can be used to restore historical documentary photos that have faded or been damaged over time, or to restore images damaged by accidents (such as floods or fires) to their original state as much as possible. In addition, it also provides great convenience for the digital media industry, such as restoring old films in film production or repairing defects in scanned images in digital publishing.
[0006] One of the current challenges facing image restoration technology is how to deal with damage in large areas or critical areas. The restoration of these areas requires not only reconstructing the appearance of the image, but also taking into account the semantic content and logical structure of the image. Summary of the invention
[0007] The present invention provides an image restoration method and system guided by image structure and texture information, which is used to solve the technical problem that a neural network model still produces distorted structures and blurred textures when repairing large damaged areas or complex images, resulting in a serious reduction in restoration quality.
[0008] In a first aspect, the present invention provides an image restoration method guided by image structure and texture information, comprising:
[0009] An image restoration model with image structure and texture information interaction capability is constructed, wherein the structure of the image restoration model includes a generator and a discriminator, wherein the generator includes a STIM module, a GIU unit, and an MLRN network, wherein the STIM module includes an upper branch of a texture information encoder, a lower branch of a structure information encoder, and an intermediate encoder, wherein the output results of the upper branch of the texture information encoder and the lower branch of the structure information encoder are respectively remapped with the output results of the intermediate encoder, and the expression of the remapping is:
[0010] ,
[0011] In the formula, is the mapping function, is the input feature value, is a logarithmic function, is a learnable hyperparameter, is a very small value set. is the mask feature fragment 1, is the mask feature fragment 2, is a variable;
[0012] Preprocessing the damaged image samples to obtain mask image samples and edge structure image samples;
[0013] Inputting the damaged image sample, the mask image sample and the edge structure image sample into the image restoration model, training and generating weight information of the image restoration model, and obtaining a target image restoration model;
[0014] The damaged image to be repaired is obtained, and the damaged image to be repaired is preprocessed to obtain a mask image and an edge structure image, and the damaged image to be repaired, the mask image and the edge structure image are respectively input into the target image repair model, and the target image repair model outputs a repaired image.
[0015] In a second aspect, the present invention provides an image restoration system guided by image structure and texture information, comprising:
[0016] A construction module is configured to construct an image restoration model with image structure and texture information interaction capabilities, wherein the structure of the image restoration model includes a generator and a discriminator, wherein the generator includes a STIM module, a GIU unit, and an MLRN network, wherein the STIM module includes an upper branch of a texture information encoder, a lower branch of a structure information encoder, and an intermediate encoder, wherein the output results of the upper branch of the texture information encoder and the lower branch of the structure information encoder are respectively remapped with the output results of the intermediate encoder, and the expression of the remapping is:
[0017] ,
[0018] In the formula, is the mapping function, is the input feature value, is a logarithmic function, is a learnable hyperparameter, is a very small value set. is the mask feature fragment 1, is the mask feature fragment 2, is a variable;
[0019] A preprocessing module is configured to preprocess the damaged image samples to obtain mask image samples and edge structure image samples;
[0020] A training module, configured to input the damaged image sample, the mask image sample and the edge structure image sample into the image restoration model, train and generate weight information of the image restoration model, and obtain a target image restoration model;
[0021] The output module is configured to obtain a damaged image to be repaired, preprocess the damaged image to be repaired to obtain a mask image and an edge structure image, and input the damaged image to be repaired, the mask image and the edge structure image into the target image repair model respectively, and the target image repair model outputs a repaired image.
[0022] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the image restoration method guided by image structure and texture information of any embodiment of the present invention.
[0023] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor executes the steps of the image restoration method guided by image structure and texture information according to any embodiment of the present invention.
[0024] The image restoration method and system guided by image structure and texture information in this application proposes an integrated three-encoder network architecture, which uses the damaged area of the image as a guide to synchronously couple modeling structure-guided texture synthesis and texture-guided structure restoration, thereby exploring image features more accurately. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0026] Figure 1 A flowchart of an image restoration method guided by image structure and texture information provided by an embodiment of the present invention;
[0027] Figure 2 A principle block diagram of image restoration guided by image structure and texture information is provided for one embodiment of the present invention;
[0028] Figure 3 A schematic diagram of the first stage working process of a STIM unit of a specific embodiment provided by an embodiment of the present invention;
[0029] Figure 4 A schematic diagram of the second stage working process of a STIM unit of a specific embodiment provided by an embodiment of the present invention;
[0030] Figure 5 A schematic diagram of the working process of a GIU unit of a specific embodiment provided for an embodiment of the present invention;
[0031] Figure 6 A schematic diagram of the MLRN network working process of a specific embodiment provided by an embodiment of the present invention;
[0032] Figure 7 A practical application effect diagram of an image restoration method guided by image structure and texture information in restoring old photos, removing objects and converting styles according to a specific embodiment of an embodiment of the present invention;
[0033] Figure 8A structural block diagram of an image restoration system guided by image structure and texture information provided by an embodiment of the present invention;
[0034] Fig. 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0036] See also Figure 1 , which shows a flowchart of an image restoration method guided by image structure and texture information of the present application,
[0037] like Figure 1 As shown, the image restoration method guided by image structure and texture information specifically includes the following steps:
[0038] Step S101, construct an image restoration model with the ability to interact with image structure and texture information, the structure of the image restoration model includes a generator and a discriminator, the generator includes a STIM module, a GIU unit and an MLRN network, and the STIM module includes an upper branch of a texture information encoder, a lower branch of a structure information encoder and an intermediate encoder.
[0039] In this step, the generator is an AE-like encoder-decoder structure, which is composed of a STIM module, a GIU unit, and an MLRN network in series. The model automatically performs forward propagation operations, and the input damaged image features pass through the STIM module, the GIU unit, and the MLRN network in sequence. The final output is the repaired image generated by the image repair model.
[0040] The loss function of the discriminator in the image restoration model includes step loss sub-function, perceptual loss sub-function, style loss sub-function and adversarial loss sub-function;
[0041] The loss function of the image restoration model is expressed as:
[0042] ,
[0043] In the formula, is the loss function of the image restoration model, is the step loss subfunction, is the perceptual loss subfunction, is the style loss sub-function, To counter the loss subfunction, , , and They are the weight of the step loss sub-function, the weight of the perceptual loss sub-function, the weight of the style loss sub-function and the weight of the adversarial loss sub-function.
[0044] Among them, the expression of the step loss subfunction is:
[0045] ,
[0046] In the formula, is the weighting coefficient, For expectations, is the k-th stage output image of the MLRN network, is the original thumbnail image of the corresponding image size at stage k when training the model;
[0047] The expression of the perceptual loss subfunction is:
[0048] ,
[0049] In the formula, is the activation of the ith pooling layer, Output image for the first stage of the MLRN network, is the original thumbnail image of the corresponding image size in the first stage when training the model;
[0050] The expression of the style loss subfunction is:
[0051] ,
[0052] In the formula, is the style loss based on the Gram matrix, ;
[0053] The expression of the adversarial loss subfunction is:
[0054] ,
[0055] In the formula, For the generator, is the discriminator, and It is the edge structure image corresponding to the original thumbnail image of the corresponding image size in the kth stage when training the model, and the edge structure image corresponding to the kth stage output image of the MLRN network.
[0056] It should be noted that the damaged image samples, mask image samples and edge structure image samples are used as inputs of the STIM module, and the STIM module outputs feature results, specifically:
[0057] The mask image sample is input into the intermediate encoder, and after convolution operation, normalization operation and Sigmoid activation function operation, the intermediate encoder outputs the first mask feature fragment and the second mask feature fragment;
[0058] The preprocessed image sample combination is input into the upper branch of the texture information encoder, and the upper branch of the texture information encoder outputs the texture features corresponding to the image sample combination, and the texture features are convolved, normalized, and LeakyReLU activated, and then added to the remapped first mask feature fragment in the channel dimension, and then multiplied with the structural feature output by the lower branch of the structure information encoder in the channel dimension to obtain the first feature result, wherein the image sample combination includes damaged image samples, mask image samples, and edge structure image samples;
[0059] The preprocessed image sample combination is input into the lower branch of the structure information encoder. The lower branch of the structure information encoder outputs the structural features corresponding to the image sample combination, and performs convolution, normalization and LeakyReLU activation function on the structural features, and then adds them to the remapped second mask feature fragment in the channel dimension, and then multiplies them with the texture features output by the upper branch of the texture information encoder in the channel dimension to obtain the second feature result.
[0060] Step S102: pre-process the damaged image samples to obtain mask image samples and edge structure image samples.
[0061] In this step, the damaged image sample is input into a Canny edge detector, and the Canny edge detector outputs an edge structure image sample corresponding to the damaged image sample.
[0062] Step S103: input the damaged image sample, the mask image sample and the edge structure image sample into the image restoration model, train and generate weight information of the image restoration model, and obtain a target image restoration model.
[0063] In this step, the encoder of the generator has three layers and is mainly composed of three branches. In the encoder part, the work of the STIM module is to send the damaged image samples and structural image samples after preprocessing operations to the upper branch of the texture information encoder and the lower branch of the structural information encoder respectively for extracting image features separately. The middle encoder branch additionally sends the 0-1 mask of the image damaged area information to guide feature extraction.
[0064] The bottleneck layer of the generator is mainly composed of four layers of GIU units, which can focus on extracting effective features. In the bottleneck layer, the work of the GIU unit is to input the feature information from the encoder STIM, perform residual convolution, and the residual branch additionally calculates the attention score for the spatial dimension and channel dimension of the feature, and weights it to the backbone network of the forward channel to extract useful information and filter out useless information.
[0065] The decoder of the generator is mainly composed of a five-level ladder-style MLRN network, which gradually diffuses from low-pixel repair images to high-pixel ones. The work of the MLRN network is to input the feature information from the bottleneck layer GIU, perform convolution and deconvolution in sequence, first restore the low-pixel image with a smaller feature size, gradually expand and pass it to restore the medium-pixel image, and then restore the full-size image.
[0066] It should be noted that the texture information encoder upper branch and the structure information encoder lower branch blocks of the STIM module each include a gated convolution operation, an instance normalization operation, and a LeakyReLU activation function operation. The intermediate encoder branch can be further divided into two stages. The first stage is the mask update, which includes a regular convolution operation, an instance normalization operation, and a Sigmoid activation function operation. The second stage is the mask-guided bidirectional information interaction between structural features and texture features. The output features of the first stage are used as the input of the second stage. The input features are divided into two according to the channel level, namely, mask feature fragment one and mask feature fragment two, which are sent to the texture interactor and structure interactor respectively. A small-range logarithmic function is introduced as the mapping function of both the texture interactor and the structure interactor to remap the input features, which is expressed as:
[0067] ,
[0068] In the formula, is the mapping function, is the input feature value, is a logarithmic function, is a learnable hyperparameter, is a very small value set. is the mask feature fragment 1, is the mask feature fragment 2, is a variable;
[0069] In the texture interactor, structural features are used to guide texture feature extraction. The texture features output by the upper branch of the texture information encoder are subjected to a conventional convolution operation, an instance normalization operation, and a LeakyReLU activation function operation, and then added to the remapped mask feature fragment in the channel dimension, and then multiplied with the structural features output by the lower branch of the structural information encoder in the channel dimension as the output of the texture interactor.
[0070] In the structure interactor, texture features are used to guide the extraction of structure features. The structure features output by the lower branch of the structure information encoder are subjected to a conventional convolution operation, an instance normalization operation, and a LeakyReLU activation function operation, and then added to the remapped mask feature fragment 2 in the channel dimension, and then multiplied with the texture features output by the upper branch of the texture information encoder in the channel dimension as the output of the structure interactor.
[0071] The GIU unit consists of a forward connection and a residual connection. The forward connection includes a regular convolution operation, an instance normalization operation, and a LeakyReLU activation function. The residual connection performs strip pooling operations in the feature height direction and strip pooling operations in the feature width direction, respectively, and performs channel expansion through a 3x1 convolution, and then performs channel dimension feature concatenation to restore the original height and width of the GIU unit input feature. Then, a 3x1 convolution using a Sigmoid activation function is used as the image gated feature channel dimension and added to the forward connection. It is the final output of the SCGIU unit.
[0072] The five levels of the MLRN network repair the original image resolution of 1 / 16, 1 / 8, 1 / 4, 1 / 2, and 1 / 1 respectively. The feature results of each level have two forward outputs, a pixel repair output that repairs the repaired image of the corresponding size through a 1x1 mapping convolution, and a feature transfer output that transfers low-pixel features to high-pixel features for guiding high-pixel repair.
[0073] Furthermore, the structure-texture dual-branch discriminator has two feature transfer branches. The texture branch and the structure branch each have five symmetrical convolutional layers, each of which includes a regular convolution operation Sigmoid activation function. The structure branch has an additional residual block detection at the input to map the features to the edges of the image structure contours. The outputs of the two branches use the Sigmoid nonlinear activation function to convert the feature values to 0-1, and the generative adversarial network loss is calculated based on this.
[0074] Step S104, obtain the damaged image to be repaired, preprocess the damaged image to be repaired to obtain a mask image and an edge structure image, and input the damaged image to be repaired, the mask image and the edge structure image into the target image repair model respectively, and the target image repair model outputs the repaired image.
[0075] In summary, the existing methods have some erroneous structural and texture information that will affect each other and cause serious distortion and degradation of the image restoration results. The present invention designs an image restoration method guided by image structure and texture information, and proposes an integrated three-encoder network architecture. Guided by the damaged area of the image, the structure-guided texture synthesis and texture-guided structure restoration are coupled modeled synchronously to explore the image features more accurately; the focus is on the enhancement of key area features in the image, which effectively improves the quality of restoration; a layer-by-layer restoration strategy is adopted to gradually restore the structure and texture information from low to high resolution. This method can ensure a high degree of visual unity between the damaged and undamaged parts of the image, making the restored image appear more natural and real.
[0076] In a specific embodiment, see Figure 2 ,The principles of image restoration guided by image structure and texture information include:
[0077] Step S1: preprocessing the damaged image to be input into the image restoration model, wherein the edge structure diagram of the damaged image is generated using a Canny edge detector.
[0078] In this embodiment, three public dataset images, CelebA, Paris Street View, and Places2, are used. Irregular masks are used to replace damaged parts of the image, with a total of 12,000 mask images. The masks are divided into three groups of 0%-20%, 20%-40%, and 40%-60% for testing according to the proportion of covering the entire image. In the CelebA dataset, the embodiment divides 30,000 original images into 29,900 for training and 100 for testing. For the Places2 dataset, the embodiment uses 1.8 million images from 365 scene categories in the original training set for training, and randomly selects 200 images from the original test set for testing. For the Paris Street View dataset, the embodiment uses 14,900 pre-divided training images and 100 test images.
[0079] Step S2: The damaged image, mask image and corresponding edge structure map are input into the image restoration model generator. The input image combination first passes through the three-layer STIM module of the model. The work of each layer of STIM module is to send the damaged map and structure map after preprocessing operation to the upper branch of texture information encoder and the lower branch of structure information encoder respectively, for extracting image features separately, and the middle encoder branch additionally sends the 0-1 mask of the damaged area information of the image to guide feature extraction.
[0080] The STIM module workflow is as follows Figure 3 , Figure 4The convolution operation, normalization operation and activation function operation are combined into Figure 3 , Figure 4 A convolutional layer in .
[0081] The texture information encoder upper branch and structure information encoder lower branch blocks of the STIM module each include a gated convolution operation, an instance normalization operation, and a LeakyReLU activation function operation. The intermediate encoder branch can be further divided into two stages. The first stage is the mask update, such as Figure 3 , including a regular convolution operation, an instance normalization operation and a Sigmoid activation function operation. The second stage is the mask-guided bidirectional information interaction between structural features and texture features, such as Figure 4 The output features of the first stage are used as the input of the second stage. The input features are divided into two according to the channel level, namely, mask feature fragment 1 and mask feature fragment 2, and sent to the texture interactor and structure interactor respectively. A small-range logarithmic function is introduced as the mapping function of both the texture interactor and the structure interactor to remap the input features.
[0082] In the texture interactor, structural features are used to guide texture feature extraction. The texture features output by the upper branch of the texture information encoder are subjected to a conventional convolution operation, an instance normalization operation, and a LeakyReLU activation function operation, and then added to the remapped mask feature fragment in the channel dimension, and then multiplied with the structural features output by the lower branch of the structural information encoder in the channel dimension as the output of the texture interactor.
[0083] In the structure interactor, texture features are used to guide the extraction of structure features. The structure features output by the lower branch of the structure information encoder are subjected to a conventional convolution operation, an instance normalization operation, and a LeakyReLU activation function operation, and then added to the remapped mask feature fragment 2 in the channel dimension, and then multiplied with the texture features output by the upper branch of the texture information encoder in the channel dimension as the output of the structure interactor.
[0084] Step S3: The final output of the STIM module is used as the input of the four-layer GIU unit. The work of the GIU unit is to input the feature information from the encoder STIM, perform residual convolution, and the residual branch additionally calculates the attention score for the spatial dimension and channel dimension of the feature, and weights it to the backbone network of the forward channel to extract useful information and filter out useless information.
[0085] GIU unit workflow Figure 5 The convolution operation, normalization operation and activation function operation are combined into one convolution layer in the figure.
[0086] The GIU unit consists of a forward connection and a residual connection. The forward connection includes a regular convolution operation, an instance normalization operation, and a LeakyReLU activation function. The residual connection performs strip pooling operations in the feature height direction and strip pooling operations in the feature width direction, respectively, and performs channel expansion through a 3x1 convolution, and then performs channel dimension feature concatenation to restore the original height and width of the GIU unit input feature. Then, a 3x1 convolution using a Sigmoid activation function is used as the image gated feature channel dimension and added to the forward connection. It is the final output of the GIU unit.
[0087] Step S4: The final output of the GIU unit is used as the input of the five-level ladder MLRN restoration network. The work of the MLRN network is to input the feature information from the bottleneck layer GIU, perform convolution and deconvolution in sequence to first restore the low-pixel image with a smaller feature size, gradually expand and transfer it to restore the medium-pixel image, and then restore the full-size image.
[0088] MLRN network workflow Figure 6 The convolution operation, normalization operation and activation function operation are combined into Figure 6 A convolutional layer in .
[0089] The five levels of the MLRN network repair the original image resolution of 1 / 16, 1 / 8, 1 / 4, 1 / 2, and 1 / 1 in turn. The feature results of each level have two forward outputs, a pixel repair output that repairs the repaired image of the corresponding size through a 1x1 mapping convolution, and a feature transfer output that transfers low-pixel features to high-pixel features to guide high-pixel repair. The generated image of the original image resolution size is used as the final output for training or application.
[0090] Step S5: Input the image finally generated in step S4 into the discriminator for discriminant training. The structure-texture dual-branch discriminator has two feature transfer branches, and the texture branch and the structure branch each have five symmetrical convolution layers, each of which includes a conventional convolution operation Sigmoid activation function. The structure branch has an additional residual block detection at the input end, which is used to map the feature to the edge of the image structure contour. The output of the two branches uses the Sigmoid nonlinear activation function to convert the feature value to 0-1.
[0091] Step S6: Repeat steps S1 to S5 to train the image restoration model and generate a weight file corresponding to the model for image restoration.
[0092] Step S7: Load the weight file trained by the model, repeat steps S1 to S4, and output the repaired image.
[0093] Figure 7In the figure, the actual restoration effect diagram of the damaged photo by the present invention is demonstrated, which covers application scenarios such as crease repair of old photos, removal of image occlusion and image style conversion.
[0094] The simulation experiments in this application are implemented under Pytorch, and NVIDIA(R) Tesla(R) V100 GPU is used for training and testing. For the convenience of comparison, all images and damage masks are resized to 256x256 pixels, the batch size is set to 6, and the Adam optimizer is used. The initial learning rate is 0.0002, the fine-tuning learning rate is 0.00005, and all normalization layers in the generator are frozen.
[0095] The superior performance of the proposed method is verified by comparing it with five other representative image restoration algorithms. The five algorithms are EC, LBAM, MED, RFR, and CTSDG. For image quality evaluation indicators, several common indicators in image restoration tasks are used: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learning-perceptual image patch similarity (LPIPS). The experimental results on the Paris dataset are shown in Table 1.
[0096] ,
[0097] From the image restoration index results given in Table 1, it can be seen that in the process of repairing damaged images, the method of the present application is superior to the comparison method. At the same time, the error between the pixel values of the experimental image and the real image obtained is the smallest among all experimental methods, and the image restoration effect is the best.
[0098] See also Figure 8 , which shows a structural block diagram of an image restoration system guided by image structure and texture information of the present application.
[0099] like Figure 8 As shown, the image restoration system 200 includes a construction module 210 , a preprocessing module 220 , a training module 230 and an output module 240 .
[0100] Among them, the construction module 210 is configured to construct an image restoration model with image structure and texture information interaction capabilities, the structure of the image restoration model includes a generator and a discriminator, the generator includes a STIM module, a GIU unit and an MLRN network, the STIM module includes an upper branch of a texture information encoder, a lower branch of a structure information encoder and an intermediate encoder, wherein the output results of the upper branch of the texture information encoder and the lower branch of the structure information encoder are respectively remapped with the output results of the intermediate encoder, and the remapping expression is:
[0101] ,
[0102] In the formula, is the mapping function, is the input feature value, is a logarithmic function, is a learnable hyperparameter, is a very small value set. is the mask feature fragment 1, is the mask feature fragment 2, is a variable;
[0103] A preprocessing module 220 is configured to preprocess the damaged image samples to obtain mask image samples and edge structure image samples;
[0104] The training module 230 is configured to input the damaged image sample, the mask image sample and the edge structure image sample into the image restoration model, train and generate weight information of the image restoration model, and obtain a target image restoration model;
[0105] The output module 240 is configured to obtain the damaged image to be repaired, preprocess the damaged image to be repaired to obtain a mask image and an edge structure image, and input the damaged image to be repaired, the mask image and the edge structure image into the target image repair model respectively, and the target image repair model outputs the repaired image.
[0106] It should be understood that Figure 8 Modules and references documented in Figure 1 Therefore, the operations and features described above for the method and the corresponding technical effects are also applicable to Figure 8 The modules in it will not be described in detail here.
[0107] In some other embodiments, the embodiments of the present invention further provide a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor is caused to execute the image restoration method guided by image structure and texture information in any of the above method embodiments;
[0108] As an implementation mode, the computer-readable storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows:
[0109] An image restoration model with image structure and texture information interaction capability is constructed, wherein the structure of the image restoration model includes a generator and a discriminator, wherein the generator includes a STIM module, a GIU unit, and an MLRN network, wherein the STIM module includes an upper branch of a texture information encoder, a lower branch of a structure information encoder, and an intermediate encoder, wherein the output results of the upper branch of the texture information encoder and the lower branch of the structure information encoder are respectively remapped with the output results of the intermediate encoder, and the expression of the remapping is:
[0110] ,
[0111] In the formula, is the mapping function, is the input feature value, is a logarithmic function, is a learnable hyperparameter, is a very small value set. is the mask feature fragment 1, is the mask feature fragment 2, is a variable;
[0112] Preprocessing the damaged image samples to obtain mask image samples and edge structure image samples;
[0113] Inputting the damaged image sample, the mask image sample and the edge structure image sample into the image restoration model, training and generating weight information of the image restoration model, and obtaining a target image restoration model;
[0114] The damaged image to be repaired is obtained, and the damaged image to be repaired is preprocessed to obtain a mask image and an edge structure image, and the damaged image to be repaired, the mask image and the edge structure image are respectively input into the target image repair model, and the target image repair model outputs a repaired image.
[0115] The computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the image restoration system guided by the image structure and texture information, etc. In addition, the computer-readable storage medium may include a high-speed random access memory, and may also include a memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the image restoration system guided by the image structure and texture information via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0116] Fig. 9 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig. 9 As shown, the device includes: a processor 310 and a memory 320. The electronic device may also include: an input device 330 and an output device 340. The processor 310, the memory 320, the input device 330 and the output device 340 may be connected via a bus or other means. Fig. 9 The example of the connection via bus is taken. The memory 320 is the above-mentioned computer-readable storage medium. The processor 310 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 320, that is, the image restoration method guided by the image structure and texture information in the above-mentioned method embodiment is implemented. The input device 330 can receive input digital or character information, and generate key signal input related to user settings and function control of the image restoration system guided by the image structure and texture information. The output device 340 may include a display device such as a display screen.
[0117] The electronic device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided by the embodiment of the present invention.
[0118] As an implementation mode, the electronic device is applied to an image restoration system guided by image structure and texture information, and is used for a client, and includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can:
[0119] An image restoration model with image structure and texture information interaction capability is constructed, wherein the structure of the image restoration model includes a generator and a discriminator, wherein the generator includes a STIM module, a GIU unit, and an MLRN network, wherein the STIM module includes an upper branch of a texture information encoder, a lower branch of a structure information encoder, and an intermediate encoder, wherein the output results of the upper branch of the texture information encoder and the lower branch of the structure information encoder are respectively remapped with the output results of the intermediate encoder, and the expression of the remapping is:
[0120] ,
[0121] In the formula, is the mapping function, is the input feature value, is a logarithmic function, is a learnable hyperparameter, is a very small value set. is the mask feature fragment 1, is the mask feature fragment 2, is a variable;
[0122] Preprocessing the damaged image samples to obtain mask image samples and edge structure image samples;
[0123] Inputting the damaged image sample, the mask image sample and the edge structure image sample into the image restoration model, training and generating weight information of the image restoration model, and obtaining a target image restoration model;
[0124] The damaged image to be repaired is obtained, and the damaged image to be repaired is preprocessed to obtain a mask image and an edge structure image, and the damaged image to be repaired, the mask image and the edge structure image are respectively input into the target image repair model, and the target image repair model outputs a repaired image.
[0125] Through the description of the above implementation modes, those skilled in the art can clearly understand that each implementation mode can be implemented by means of software plus a necessary general hardware platform, or of course by hardware. Based on such an understanding, the above technical solution can essentially or in other words be embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.
[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An image restoration method guided by image structure and texture information, characterized in that: include: An image restoration model with image structure and texture information interaction capability is constructed, wherein the structure of the image restoration model includes a generator and a discriminator, wherein the generator includes a STIM module, a GIU unit, and an MLRN network, wherein the STIM module includes an upper branch of a texture information encoder, a lower branch of a structure information encoder, and an intermediate encoder, wherein the output results of the upper branch of the texture information encoder and the lower branch of the structure information encoder are respectively remapped with the output results of the intermediate encoder, and the expression of the remapping is: , In the formula, is the mapping function, is the input feature value, is a logarithmic function, is a learnable hyperparameter, is a very small value set. is the mask feature fragment 1, is the mask feature fragment 2, is a variable; The damaged image samples, mask image samples and edge structure image samples are used as inputs of the STIM module, and the STIM module outputs feature results, specifically: Inputting the mask image sample into an intermediate encoder, after a convolution operation, a normalization operation and a Sigmoid activation function operation, the intermediate encoder outputs a first mask feature segment and a second mask feature segment; The preprocessed image sample combination is inputted into the upper branch of the texture information encoder, and the upper branch of the texture information encoder outputs the texture features corresponding to the image sample combination, and the texture features are convolved, normalized, and LeakyReLU activated, and then added to the remapped first mask feature fragment in the channel dimension, and then multiplied with the structural feature output by the lower branch of the structure information encoder in the channel dimension to obtain a first feature result, wherein the image sample combination includes damaged image samples, mask image samples, and edge structure image samples; The preprocessed image sample combination is inputted into the lower branch of the structure information encoder, and the lower branch of the structure information encoder outputs the structure feature corresponding to the image sample combination, and the structure feature is convolved, normalized and LeakyReLU activated, and then added to the remapped second mask feature fragment in the channel dimension, and then multiplied with the texture feature outputted by the upper branch of the texture information encoder in the channel dimension to obtain the second feature result; Preprocessing the damaged image samples to obtain mask image samples and edge structure image samples; Inputting the damaged image sample, the mask image sample and the edge structure image sample into the image restoration model, training and generating weight information of the image restoration model, and obtaining a target image restoration model; The damaged image to be repaired is obtained, and the damaged image to be repaired is preprocessed to obtain a mask image and an edge structure image, and the damaged image to be repaired, the mask image and the edge structure image are respectively input into the target image repair model, and the target image repair model outputs a repaired image.
2. The image restoration method guided by image structure and texture information according to claim 1, characterized in that: The loss function of the discriminator in the image restoration model includes a step loss sub-function, a perceptual loss sub-function, a style loss sub-function and an adversarial loss sub-function; The expression of the loss function of the image restoration model is: , In the formula, is the loss function of the image restoration model, is the step loss subfunction, is the perceptual loss subfunction, is the style loss sub-function, To counter the loss subfunction, , , and They are the weight of the step loss sub-function, the weight of the perceptual loss sub-function, the weight of the style loss sub-function and the weight of the adversarial loss sub-function.
3. The image restoration method guided by image structure and texture information according to claim 2, characterized in that: in, The expression of the step loss subfunction is: , In the formula, is the weighting coefficient, For expectations, is the k-th stage output image of the MLRN network, is the original thumbnail image of the corresponding image size at stage k when training the model; The expression of the perceptual loss subfunction is: , In the formula, is the activation of the ith pooling layer, Output image for the first stage of the MLRN network, is the original thumbnail image of the corresponding image size in the first stage when training the model; The expression of the style loss subfunction is: , In the formula, is the style loss based on the Gram matrix, ; The expression of the adversarial loss subfunction is: , In the formula, For the generator, is the discriminator, and It is the edge structure image corresponding to the original thumbnail image of the corresponding image size in the kth stage when training the model, and the edge structure image corresponding to the kth stage output image of the MLRN network.
4. The image restoration method guided by image structure and texture information according to claim 1, characterized in that: The GIU unit includes a forward connection and a residual connection, and the forward connection includes a convolution operation, a normalization operation and a LeakyReLU activation function; The residual connection performs strip pooling operations in the feature height direction and strip pooling operations in the feature width direction on the input features, performs channel expansion through 3x1 convolution, and then performs channel dimension feature splicing to restore the original height and width of the input features of the GIU unit; After a 3x1 convolution using a Sigmoid activation function as the image gated feature channel dimension, it is added to the forward connection as the final output of the GIU unit.
5. The image restoration method guided by image structure and texture information according to claim 1, characterized in that: The MLRN network comprises five steps, which successively repair the original image resolution of 1 / 16, 1 / 8, 1 / 4, 1 / 2 and 1 / 1. The feature result of each step has two forward outputs, a pixel repair output that repairs the repaired image of the corresponding size through a 1x1 mapping convolution, and a feature transfer output that transfers the low-pixel feature to the high-pixel feature for guiding the high-pixel repair, thereby generating an image of the original image resolution size.
6. The image restoration method guided by image structure and texture information according to claim 1, characterized in that: The preprocessing of the damaged image samples to obtain the mask image samples and the edge structure image samples comprises: The damaged image sample is input into a Canny edge detector, and the Canny edge detector outputs an edge structure image sample corresponding to the damaged image sample.
7. An image restoration system guided by image structure and texture information, characterized in that: include: A construction module is configured to construct an image restoration model with image structure and texture information interaction capabilities, wherein the structure of the image restoration model includes a generator and a discriminator, wherein the generator includes a STIM module, a GIU unit, and an MLRN network, wherein the STIM module includes an upper branch of a texture information encoder, a lower branch of a structure information encoder, and an intermediate encoder, wherein the output results of the upper branch of the texture information encoder and the lower branch of the structure information encoder are respectively remapped with the output results of the intermediate encoder, and the expression of the remapping is: , In the formula, is the mapping function, is the input feature value, is a logarithmic function, is a learnable hyperparameter, is a very small value set. is the mask feature fragment 1, is the mask feature fragment 2, is a variable; The damaged image samples, mask image samples and edge structure image samples are used as inputs of the STIM module, and the STIM module outputs feature results, specifically: Inputting the mask image sample into an intermediate encoder, after a convolution operation, a normalization operation and a Sigmoid activation function operation, the intermediate encoder outputs a first mask feature segment and a second mask feature segment; The preprocessed image sample combination is inputted into the upper branch of the texture information encoder, and the upper branch of the texture information encoder outputs the texture features corresponding to the image sample combination, and the texture features are convolved, normalized, and LeakyReLU activated, and then added to the remapped first mask feature fragment in the channel dimension, and then multiplied with the structural feature output by the lower branch of the structure information encoder in the channel dimension to obtain a first feature result, wherein the image sample combination includes damaged image samples, mask image samples, and edge structure image samples; The preprocessed image sample combination is inputted into the lower branch of the structure information encoder, and the lower branch of the structure information encoder outputs the structure feature corresponding to the image sample combination, and the structure feature is convolved, normalized and LeakyReLU activated, and then added to the remapped second mask feature fragment in the channel dimension, and then multiplied with the texture feature outputted by the upper branch of the texture information encoder in the channel dimension to obtain the second feature result; A preprocessing module is configured to preprocess the damaged image samples to obtain mask image samples and edge structure image samples; A training module, configured to input the damaged image sample, the mask image sample and the edge structure image sample into the image restoration model, train and generate weight information of the image restoration model, and obtain a target image restoration model; The output module is configured to obtain a damaged image to be repaired, preprocess the damaged image to be repaired to obtain a mask image and an edge structure image, and input the damaged image to be repaired, the mask image and the edge structure image into the target image repair model respectively, and the target image repair model outputs a repaired image.
8. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
