Image restoration method, device, equipment and storage medium

Abstract

The present invention discloses an image restoration method, device, equipment and storage medium, including: extracting feature information of an image containing a curled target object, and performing feature equalization processing on the feature information to obtain equalized feature information; performing restoration processing on the equalized feature to obtain restoration features, and connecting the equalized feature and the restoration feature to obtain a target restored image. The above technical scheme, through the equalized feature obtained by equalizing the feature information, pays more attention to extracting the local features of the image containing the curled target object, and can obtain more local feature information about the curled target. The target restored image obtained by connecting the equalized feature and the restoration feature can contain more detailed information of the curled target. The target restored image obtained based on the equalized feature and the restoration feature is closer to the image containing the curled target, and the restoration degree and restoration effect of the curled target are further improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to image processing technology, and in particular to an image restoration method, device, equipment and storage medium. Background Art

[0002] During the image acquisition process, various factors often cause occlusion or deviation of the target objects in the acquired images. At this time, it is necessary to repair the occluded or deviated images based on image restoration technology.

[0003] For images containing curled objects, if their image contours and image textures need to be restored, existing image restoration methods can perform edge repair and content repair on the images that have been partially removed based on edge texture and content texture, but they cannot unfold images containing curled objects, and the restoration degree is insufficient and the restoration effect is also poor.

[0004] Therefore, there is an urgent need for an image restoration method that can restore curled objects and improve the restoration effect. Summary of the invention

[0005] The present invention provides an image restoration method, device, equipment and storage medium to restore an image containing a curled target object and improve the restoration effect.

[0006] In a first aspect, an embodiment of the present invention provides an image restoration method, comprising:

[0007] Extracting feature information of an image containing a curled target object, and performing feature equalization processing on the feature information to obtain a balanced feature;

[0008] The equalization feature is restored to obtain a restoration feature, and the equalization feature and the restoration feature are connected to obtain a target restored image.

[0009] Furthermore, feature information of the image containing the curled target object is extracted, and feature equalization processing is performed on the feature information to obtain equalized features, including:

[0010] A generator based on a preset generative adversarial model extracts feature information of an image containing a curled target object, and performs feature equalization processing on the feature information to obtain a balanced feature.

[0011] Further, the generator of the preset generative adversarial model includes an encoding module;

[0012] Accordingly, the generator based on the preset generative adversarial model extracts feature information of the image containing the curled target object, and performs feature equalization processing on the feature information to obtain equalized features, including:

[0013] Extracting at least one layer of feature information of the image containing the curled target object through the encoding module;

[0014] The feature information is feature balanced in the channel dimension and the spatial dimension according to the attention mechanism to obtain balanced features.

[0015] Furthermore, extracting at least one layer of feature information of the image containing the curled target object by the encoding module includes:

[0016] Performing feature fusion on the feature information extracted by the deep convolutional layer of the encoding module to obtain deep features;

[0017] The feature information extracted by the shallow convolutional layer of the encoding module is subjected to feature fusion to obtain shallow features.

[0018] Furthermore, after extracting at least one layer of feature information of the image containing the curled target object through the encoding module, the method further includes:

[0019] Splicing the deep features and the shallow features to obtain spliced ​​features;

[0020] Performing convolution processing on the splicing features to obtain fusion features;

[0021] Among them, the fusion features are feature balanced in the channel dimension and the spatial dimension according to the attention mechanism to obtain balanced features.

[0022] Furthermore, the preset generative adversarial model is determined in the following manner:

[0023] Acquire a training image containing a curled target object and a real unfolded image corresponding to the training image;

[0024] Using the training image and the real expanded image as training data to perform model training, and calculating a loss function;

[0025] The model is optimized based on the back-propagation update algorithm until the loss function converges to obtain a preset generative adversarial model.

[0026] Furthermore, the preset generative adversarial model also includes a discriminator,

[0027] Accordingly, the training image and the real expanded image are used as training data to perform model training, and a loss function is calculated, including:

[0028] Inputting the training image and the real expanded image as training data into the preset generative adversarial model;

[0029] Determining an initial restored image corresponding to the training image by using the generator of the preset generative adversarial model;

[0030] Determining the image category of the initial restored image based on a discriminator of a preset generative adversarial model;

[0031] Determine the loss function according to the determination result;

[0032] Among them, the image categories include training restored images and real expanded images.

[0033] In a second aspect, an embodiment of the present invention further provides an image restoration device, comprising:

[0034] An extraction module, used for extracting feature information of an image containing a curled target object, and performing feature equalization processing on the feature information to obtain a balanced feature;

[0035] The connection module is used to perform restoration processing on the equalization feature to obtain a restoration feature, and connect the equalization feature and the restoration feature to obtain a target restored image.

[0036] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an image restoration method as described in any one of the first aspects is implemented.

[0037] In a fourth aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to perform the image restoration method as described in any one of the first aspects.

[0038] An image restoration method provided by an embodiment of the present invention extracts feature information of an image containing a curled target object, performs feature equalization processing on the feature information, and obtains equalized feature information; performs restoration processing on the equalized feature to obtain restoration features, and connects the equalized feature and the restoration features to obtain a target restored image. The above technical scheme, through the equalized feature information obtained by equalizing the feature information, pays more attention to extracting local features of the image containing the curled target object, and can obtain more local feature information about the curled target. The target restored image obtained by connecting the equalized feature and the restoration feature can contain more detailed information of the curled target object. Further, the target restored image obtained according to the equalized feature and the restoration feature is closer to the image containing the curled target object, thereby improving the restoration degree and restoration effect of the curled target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1A flowchart of an image restoration method provided in Embodiment 1 of the present invention;

[0040] Figure 2 A flowchart of an image restoration method provided in Embodiment 2 of the present invention;

[0041] Figure 3 A flowchart of step 210 in an image restoration method provided in Embodiment 2 of the present invention;

[0042] Figure 4 A flowchart of determining a preset generative adversarial model in an image restoration method provided in Embodiment 2 of the present invention;

[0043] Figure 5 A structural diagram of an image restoration device provided in Embodiment 3 of the present invention;

[0044] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0046] It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe various operations (or steps) as sequential processes, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of various operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc. In addition, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0047] Embodiment 1

[0048] Figure 1 This is a flow chart of an image restoration method provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation where a curled image needs to be restored. The method can be executed by a computer device and specifically includes the following steps:

[0049] Step 110: extract feature information of the image containing the curled target object, and perform feature equalization processing on the feature information to obtain equalized feature information.

[0050] The curled target is not specifically limited here. The image restoration method provided in the embodiment of the present invention can restore an image containing any curled target. For example, the curled target may include curled tea leaves or curled leaves. The image containing the curled target can be obtained by an image acquisition device, can be downloaded from a server, or can be input into a computer device through an input device.

[0051] By convolving an image containing a curled target object, a feature map containing feature information corresponding to the image containing the curled target object can be obtained. Then, by convolving a certain range of feature maps, the feature information contained in multiple feature maps can be extracted into one feature information to obtain the next feature map. Then, the feature map is further convolved, and the feature information is continuously combined to obtain a more complex feature map, thereby obtaining richer feature information.

[0052] Feature equalization can include equalization in the channel dimension and equalization in the spatial dimension. It can extract more local feature information and pay more attention to the detail information of the curled target. It can also make the extracted feature information pay more attention to the curled target itself, greatly reducing the attention to the background information in the image containing the curled target, so that the extracted local features related to the curled target are more accurate.

[0053] It should be noted that the balance in the channel dimension and the balance in the spatial dimension can be achieved through the attention mechanism module. The attention mechanism module processes the input feature information, that is, it can achieve the balance of the feature information in the channel dimension and the balance in the spatial dimension.

[0054] Specifically, the convolution layer can be used to extract the feature information of the image containing the curled target. When the convolution layer is shallow, the texture information and detail information of the image containing the curled target can be extracted; when the convolution layer is deep, the high-level semantic information of the image containing the curled target, such as structural information, can be extracted. If it is necessary to restore a more detailed and realistic target restoration image, the feature information of the curled image can be extracted based on multiple layers of convolution layers. The first several layers can extract the texture features of the image containing the curled target, and the latter several layers can extract the structural features of the image containing the curled target. For example, there can be 6 convolution layers for feature information extraction, the first 3 layers can extract the texture features of the image containing the curled target, and the latter 3 layers can extract the structural features of the image containing the curled target.

[0055] It should be noted that the encoding module may include multiple convolutional layers, and the warped image input to the encoding module is convolved through the multiple convolutional layers of the encoding module to extract corresponding feature information.

[0056] In an embodiment of the present invention, after the first several convolutional layers and the last several convolutional layers extract texture feature information and structural feature information containing the curled target object, the texture feature information and the structural feature information can be fused, and then feature equalization processing can be performed on the fused feature information to obtain equalized features.

[0057] Step 120: Perform restoration processing on the equalization feature to obtain a restoration feature, and connect the equalization feature and the restoration feature to obtain a target restored image.

[0058] Specifically, the equalization feature can be input as input information to the decoding module, and the transposed convolution layer of the decoding module can perform deconvolution processing on the equalization feature to restore the image and obtain the restoration feature. Then, the output information of the decoding module can be connected with the equalization feature to generate the target restoration image.

[0059] The decoding module may include multiple layers of transposed convolutional layers, which may sequentially deconvolve the equalized features, and the output of each layer may also be jump-connected so that the determined initial restored image is closer to the real restored image.

[0060] In addition, the connection between the output information of the decoding module and the equalization feature can be a skip connection.

[0061] An image restoration method provided by an embodiment of the present invention extracts feature information of an image containing a curled target object, performs feature equalization processing on the feature information, and obtains equalized feature information; performs restoration processing on the equalized feature to obtain restoration features, and connects the equalized feature and the restoration features to obtain a target restored image. The above technical scheme, through the equalized feature information obtained by equalizing the feature information, pays more attention to extracting local features of the image containing the curled target object, and can obtain more local feature information about the curled target. The target restored image obtained by connecting the equalized feature and the restoration feature can contain more detailed information of the curled target object. Further, the target restored image obtained according to the equalized feature and the restoration feature is closer to the image containing the curled target object, thereby improving the restoration degree and restoration effect of the curled target object.

[0062] Embodiment 2

[0063] Figure 2 This is a flowchart of an image restoration method provided in Embodiment 2 of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, the method may also include:

[0064] Step 210: extract feature information of the image containing the curled target object, and perform feature equalization processing on the feature information to obtain equalized features.

[0065] In one implementation, step 210 may specifically include:

[0066] A generator based on a preset generative adversarial model extracts feature information of an image containing a curled target object, and performs feature equalization processing on the feature information to obtain a balanced feature.

[0067] The generator of the preset generative adversarial model can realize the encoding and decoding process of the image containing the curled target object. The generator of the preset generative adversarial model can include an encoding module and an attention mechanism module, and the encoding module can include a multi-scale fusion unit.

[0068] Before extracting feature information of the image containing the curled target object, the image containing the curled target object may be processed, wherein the image processing may include image rotation, image bottom flip, image horizontal flip, image brightness adjustment, etc.

[0069] In one embodiment, the generator of the preset generative adversarial model includes an encoding module; accordingly, Figure 3 This is a flowchart of step 210 in an image restoration method provided in Embodiment 2 of the present invention, as shown in FIG. Figure 3 As shown, step 210 may include:

[0070] Step 2110: extract at least one layer of feature information of the image containing the curled target object through the encoding module.

[0071] The encoding module may include multiple convolutional layers, and the multiple convolutional layers of the encoding module may extract feature information of an image containing a curled target object through convolution.

[0072] Specifically, as described in the first embodiment, the first several layers can extract the texture features of the image containing the curled target object, and the latter several layers can extract the structural features of the image containing the curled target object. For example, there can be 6 convolution layers for extracting feature information, the first 3 layers can extract the texture features of the image containing the curled target object, and the latter 3 layers can extract the structural features of the image containing the curled target object. This allows the decoding module to generate a target restoration image with more realistic details after decoding the feature information.

[0073] In one implementation, step 2110 may specifically include:

[0074] The feature information extracted by the deep convolution layer of the encoding module is subjected to feature fusion to obtain deep features; the feature information extracted by the shallow convolution layer of the encoding module is subjected to feature fusion to obtain shallow features.

[0075] Specifically, the feature information extracted by the deep convolution layer of the encoding module can be fused through the multi-scale fusion module to obtain deep features; and the feature information extracted by the shallow convolution layer of the encoding module can be fused to obtain shallow features.

[0076] The multi-scale fusion module can integrate the feature information output by different convolutional layers, and fuse the semantic feature information and spatial feature information of multiple convolutional layers to obtain deep features and shallow features with richer semantic feature information and spatial feature information.

[0077] Step 2120: Concatenate the deep features and the shallow features to obtain concatenated features.

[0078] Specifically, after obtaining the deep features and the shallow features, the deep features and the shallow features can be concatenated to obtain concatenated features. The concatenated features obtained have more channels and richer semantic feature information and spatial feature information.

[0079] Step 2130: Perform convolution processing on the splicing features to obtain fusion features.

[0080] The concatenated features can be further convolved to obtain fused features. The convolution processing here can be a 1×1 convolution, which can reduce the parameters of the generator, increase the depth of the generator, and improve the representation ability of the generator to a certain extent.

[0081] Step 2140: Perform feature equalization on the fused features in the channel dimension and the spatial dimension through the attention mechanism to obtain equalized features.

[0082] The fused features may be feature balanced in the channel dimension and the spatial dimension through an attention mechanism module to obtain balanced features.

[0083] The specific manner and steps of performing feature equalization on the fused features are consistent with the method and steps of performing feature equalization on the feature information in the aforementioned embodiment 1, and the effects are also the same, and will not be repeated here.

[0084] In the embodiment of the present invention, the obtained equalization features may contain richer structural feature information and texture feature information, thereby being able to restore a more accurate target restored image.

[0085] Figure 4 A flowchart of determining a preset generative adversarial model in an image restoration method provided in Embodiment 2 of the present invention, such as Figure 4 As shown, the preset generative adversarial model is determined in the following manner:

[0086] Step 410: Acquire a training image containing a curled target and a real unfolded image corresponding to the training image. The training image containing the curled target and the real unfolded image corresponding to the training image are in one-to-one correspondence. The curled target in the training image containing the curled target is unfolded and then image acquisition is performed to obtain the real unfolded image. In addition, when acquiring the training image and the real unfolded image, in order to reduce the occurrence of the phenomenon that the training image and the real unfolded image do not correspond to each other due to the movement of the curled target, the curled target can be pasted to the same position of the same background board with transparent tape, and the training image and the real unfolded image can be acquired separately.

[0087] An image acquisition device may be used to acquire the training image and the real expanded image, and the image acquisition device may include a line scan camera. In practical applications, the image size acquired by the line scan camera may be 4k*1k, and the format may be RGB format. In an embodiment of the present invention, the 4k*1k images acquired multiple times may be spliced ​​to form a single picture of a complete training image and a real expanded image, and then the size of the single image may be adjusted to 256*256 to obtain a training image and a real expanded image suitable for image restoration.

[0088] Step 420: Perform image processing on the training image and the real expanded image to obtain a processed training image and a processed real expanded image.

[0089] The image processing may include image calibration, image rotation, image bottom flip, image horizontal flip and image brightness adjustment. The image calibration may include calibration of the position of the curled target in the training image and the real unfolded image to obtain the accurately corresponding training image and the real unfolded image.

[0090] Image processing can emphasize the local characteristics of the image, make the originally unclear image clear and emphasize the characteristics of the curled target, expand the difference between the characteristics of the curled target and the background in the training image and the real unfolded image, improve the image quality, enrich the amount of information, enhance the image interpretation and recognition effect, and reduce the impact of the image inaccuracy on the image restoration effect.

[0091] Specifically, a normalization operation may be performed on the collected training image and the real expanded image based on image processing to obtain a processed training image and a processed real expanded image.

[0092] Step 430: Use the training image and the real expanded image as training data to perform model training, and calculate the loss function.

[0093] Among them, the preset generative adversarial model may include a generator and a discriminator. The training process of the preset generative adversarial model is similar to a game process. The generator can generate training restored images that are infinitely realistic than real restored images to confuse the judgment of the discriminator. The discriminator can distinguish the generated training restored images from the real restored images as much as possible, and the classification results of the discriminator can be transmitted back to stimulate the generator to optimize the generator performance.

[0094] Specifically, after the training image containing the curled target object is input into the preset generative adversarial model as input information, the output result obtained can be a training restored image. In order to make the obtained training restored image as close as possible to the real unfolded image, the preset generative adversarial model needs to be trained.

[0095] In addition, the loss function can include adversarial loss and reconstruction loss.

[0096] In an embodiment of the present invention, the loss functions of the output information and real information of the generator and the discriminator are not separately distinguished. The loss functions of the training restored images and the real expanded images output by the preset generative adversarial model can be directly obtained, so that the performance of the generator can be optimized while optimizing the performance of the discriminator.

[0097] In one implementation, step 420 may specifically include:

[0098] The training image and the real expanded image are input into the preset generative adversarial model as training data; the initial restored image corresponding to the training image is determined by the generator of the preset generative adversarial model; the image category of the initial restored image is determined based on the discriminator of the preset generative adversarial model; the loss function is determined according to the determination result; wherein the image category includes the training restored image and the real expanded image.

[0099] Specifically, the generator of the preset generative adversarial model can obtain a training restored image corresponding to the training image based on the training image, that is, the initial restored image may include the training image and the display expanded image.

[0100] Optionally, the discriminator based on the preset generative adversarial model determines the image category of the initial restored image, including: extracting feature information of the initial restored image at multiple scales to obtain initial restored features of each scale; respectively connecting the initial restored features of each scale to obtain connection features; classifying the connection feature information to obtain a classification result; and determining the image category of the initial restored image according to the classification result.

[0101] Optionally, the extracting feature information of the initial restored image at multiple scales to obtain the initial restored features of each scale includes: extracting the feature information of the initial restored image at multiple scales through a convolution module including multiple convolution layers; and performing jump connections on the feature information extracted by different convolution layers at each scale to obtain the initial restored features of each scale.

[0102] Specifically, the discriminator of the preset generative adversarial model may include a convolution module, and the convolution module may include a convolution layer, a fully connected layer, and a normalization layer.

[0103] Multi-layer convolutional layers can extract feature information of the initial restored image at multiple scales to obtain the initial restored features of each scale; the fully connected layer and the normalized layer can perform jump connections on the initial restored features to obtain connection features.

[0104] It should be noted that the initial restored image can be input into the discriminator in blocks, and the initial restored image can be divided into multiple image blocks according to the size to obtain a set of image blocks of the size, and the image block sets of different sizes can be input into the discriminator. For example, the initial restored image can be divided into N image blocks, and the size of each image block can be S*S, where S can be 70. Of course, the number of image blocks and the size of each image block are not specifically limited here, and can be determined based on the size of the acquired training image and the real unfolded image and the size of the curled target. Inputting the initial restored image into the discriminator in blocks can improve the recognition function of the discriminator.

[0105] The discriminator of the preset generative adversarial model may also include a classification module, which can classify the connection features to determine the image category of the initial restored image; based on the image category of the initial restored image, determine whether the initial restored image is a training restored image or a real expanded image.

[0106] In the embodiment of the present invention, the discriminator of the preset generative adversarial model can extract feature information of the initial restored image based on the convolution module to obtain the initial restored features, and perform jump connection on the initial restored features to obtain connection features, so that the discriminator can pay more attention to the extraction of features of the curled target object contained in the initial restored image. The classification module can also be used to realize the classification of the training restored image or the real unfolded image.

[0107] Step 440: Optimize the model based on the back-propagation update algorithm until the loss function converges to obtain a preset generative adversarial model.

[0108] Specifically, when the loss function converges, the corresponding parameter information can be determined as the parameter information of the preset generative adversarial model to update the preset generative adversarial model, and the updated preset generative adversarial model can also be made more suitable for restoring the current image containing the curled target object.

[0109] Step 220: Perform restoration processing on the equalization feature to obtain a restoration feature, and connect the equalization feature and the restoration feature to obtain a target restored image.

[0110] In one embodiment, the generator of the preset generative adversarial model further includes a decoding module.

[0111] Accordingly, the equalization feature is restored to obtain the restored feature, including:

[0112] The equalization feature is restored based on the decoding module to obtain a restored feature.

[0113] Specifically, the equalization features can be input as input information into the decoding module, and the transposed convolution layer of the decoding module can perform deconvolution processing on the equalization features to restore the image, and then the output information of the decoding module can be connected with the equalization features to generate the target restored image.

[0114] In the embodiment of the present invention, the feature information obtained by connecting the equalization feature and the restoration feature is richer and more detailed, and the obtained target restored image can be closer to the real unfolded image, thereby improving the accuracy of image restoration.

[0115] A second embodiment of the present invention provides an image restoration method, which extracts feature information of an image containing a curled target object, performs feature equalization processing on the feature information, and obtains equalized feature information; performs restoration processing on the equalized feature to obtain restoration features, and connects the equalized feature and the restoration features to obtain a target restored image. The above technical scheme, by equalizing the feature information obtained by equalizing the feature information, pays more attention to extracting local features of the target image, and can obtain more local feature information about the curled target object. The target restored image obtained by connecting the equalized feature and the restoration feature can contain more detailed information of the curled target object. Further, the target restored image obtained based on the equalized feature and the restoration feature is closer to the image containing the curled target object, thereby improving the restoration degree and restoration effect of the image containing the curled target object.

[0116] In addition, before restoring an image containing a curled target object based on a preset generative adversarial model, an image restoration method provided by an embodiment of the present invention may also include a training process for the preset generative adversarial model to improve the adaptability of the preset generative adversarial model to the curled target object and the image containing the curled target object, so that the target restored image is more accurate.

[0117] Furthermore, an image restoration method provided by an embodiment of the present invention extracts at least one layer of feature information of an image containing a curled target object through the encoding module, performs feature fusion on the feature information extracted by the deep convolution layer of the encoding module through the multi-scale fusion module to obtain deep features, performs feature fusion on the feature information extracted by the shallow convolution layer of the encoding module through the multi-scale fusion module to obtain shallow features, splices the deep features and the shallow features to obtain spliced ​​features, performs convolution processing on the spliced ​​features to obtain fused features, and performs feature equalization on the fused features in the channel dimension and the spatial dimension through the attention mechanism module to obtain equalized features, so as to obtain richer structural feature information and texture feature information, and thus restore a more accurate target restoration image.

[0118] Embodiment 3

[0119] Figure 5 This is a structural diagram of an image restoration device provided in Embodiment 3 of the present invention, which can be applied to situations where a curled image needs to be restored to improve the image restoration effect. The device can be implemented by software and / or hardware and is generally integrated in a computer device.

[0120] like Figure 5 As shown, the device comprises:

[0121] An extraction module 510 is used to extract feature information of an image containing a curled target object, and perform feature equalization processing on the feature information to obtain a balanced feature;

[0122] The connection module 520 is used to perform restoration processing on the equalization feature to obtain a restoration feature, and connect the equalization feature and the restoration feature to obtain a target restored image.

[0123] The third embodiment of the present invention provides an image restoration device, which extracts feature information of an image containing a curled target object, performs feature equalization processing on the feature information to obtain a balanced feature; performs restoration processing on the balanced feature to obtain a restored feature, and connects the balanced feature and the restored feature to obtain a target restored image. The above technical scheme, through the balanced feature obtained by equalizing the feature information, pays more attention to the extraction of local features of the image containing the curled target object, and can obtain more local feature information about the curled target object. The target restored image obtained by connecting the balanced feature and the restored feature can contain more detailed information of the curled target object. Further, the target restored image obtained based on the balanced feature and the restored feature is closer to the image containing the curled target object, thereby improving the restoration degree and restoration effect of the image containing the curled target object.

[0124] Based on the above embodiment, the extraction module 510 is specifically used for:

[0125] A generator based on a preset generative adversarial model extracts feature information of an image containing a curled target object, and performs feature equalization processing on the feature information to obtain a balanced feature.

[0126] Based on the above embodiment, the generator of the preset generative adversarial model includes an encoding module,

[0127] Accordingly, the extraction module 510 is more specifically configured to:

[0128] Extracting at least one layer of feature information of the image containing the curled target object through the encoding module;

[0129] The feature information is feature balanced in the channel dimension and the spatial dimension according to the attention mechanism to obtain balanced features.

[0130] Furthermore, extracting at least one layer of feature information of the image containing the curled target object by the encoding module includes:

[0131] Performing feature fusion on the feature information extracted by the deep convolutional layer of the encoding module to obtain deep features;

[0132] The feature information extracted by the shallow convolutional layer of the encoding module is subjected to feature fusion to obtain shallow features.

[0133] In one embodiment, after extracting at least one layer of feature information of the image containing the curled target object through the encoding module, the extraction module 510 is further configured to:

[0134] Splicing the deep features and the shallow features to obtain spliced ​​features;

[0135] Performing convolution processing on the splicing features to obtain fusion features;

[0136] Among them, the fusion features are feature balanced in the channel dimension and the spatial dimension according to the attention mechanism to obtain balanced features.

[0137] Based on the above embodiment, the generator of the preset generative adversarial model further includes a decoding module,

[0138] Accordingly, the connection module 520 is specifically used for:

[0139] The equalization feature is restored based on the decoding module to obtain a restored feature.

[0140] Based on the above embodiment, the preset generative adversarial model is determined in the following manner:

[0141] Acquire a training image containing a curled target object and a real unfolded image corresponding to the training image;

[0142] Using the training image and the real expanded image as training data to perform model training, and calculating a loss function;

[0143] The model is optimized based on the back-propagation update algorithm until the loss function converges to obtain a preset generative adversarial model.

[0144] In one embodiment, the preset generative adversarial model further includes a discriminator,

[0145] Accordingly, the training image and the real expanded image are used as training data to perform model training, and a loss function is calculated, including:

[0146] Inputting the training image and the real expanded image as training data into the preset generative adversarial model;

[0147] Determining an initial restored image corresponding to the training image by using the generator of the preset generative adversarial model;

[0148] Determining the image category of the initial restored image based on a discriminator of a preset generative adversarial model;

[0149] Determine the loss function according to the determination result;

[0150] Among them, the image categories include training restored images and real expanded images.

[0151] In one implementation, the discriminator based on the preset generative adversarial model determines the image category of the initial restored image, including:

[0152] Extracting feature information of the initial restored image at multiple scales to obtain initial restored features at each scale;

[0153] Respectively connecting the initial restored features of each scale to obtain a connected feature;

[0154] Classifying the connection feature information to obtain a classification result;

[0155] The image category of the initial restored image is determined according to the classification result.

[0156] In one implementation, extracting feature information of the initial restored image at multiple scales to obtain initial restored features at each scale includes:

[0157] Extracting feature information of the initial restored image at multiple scales through a convolution module including multiple convolution layers;

[0158] The feature information extracted by different convolutional layers at each scale is skipped and connected to obtain the initial restored features of each scale.

[0159] On the basis of the above embodiment, before using the training image and the real expanded image as training data for model training, the method further includes:

[0160] The training image and the real expanded image are subjected to image processing to obtain a processed training image and a processed real expanded image.

[0161] The image restoration device provided in the embodiment of the present invention can execute the image restoration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0162] Embodiment 4

[0163] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention is shown in FIG. Figure 6 As shown, the computer device includes a processor 610 and a memory 620; the number of processors 610 in the computer device can be one or more. Figure 6 A processor 610 is taken as an example; the processor 610 and the memory 620 in the computer device can be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0164] The memory 620, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the image restoration method in the embodiment of the present invention (for example, the extraction module 510 and the connection module 520 in the image restoration device). The processor 610 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 620, that is, implements the above-mentioned image restoration method.

[0165] The memory 620 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 620 may further include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0166] The computer device provided by the embodiment of the present invention can execute the image restoration method provided by the above embodiment and has corresponding functions and beneficial effects.

[0167] Embodiment 5

[0168] Embodiment 5 of the present invention further provides a storage medium including computer executable instructions, wherein the computer executable instructions are used to execute an image restoration method when executed by a computer processor, the method comprising:

[0169] Extracting feature information of an image containing a curled target object, and performing feature equalization processing on the feature information to obtain a balanced feature;

[0170] The equalization feature is restored to obtain a restoration feature, and the equalization feature and the restoration feature are connected to obtain a target restored image.

[0171] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided by an embodiment of the present invention are not limited to the operations of the method described above, and can also execute related operations in the image restoration method provided by any embodiment of the present invention.

[0172] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0173] It is worth noting that in the embodiment of the above-mentioned image restoration device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0174] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image restoration method, characterized in that: include: Extracting feature information of an image containing a curled target object based on a generator of a preset generative adversarial model, and performing feature equalization processing on the feature information to obtain a balanced feature; Performing restoration processing on the equalization feature to obtain a restoration feature, and connecting the equalization feature and the restoration feature to obtain a target restored image; Among them, the generator of the preset generative adversarial model includes an encoding module; the generator based on the preset generative adversarial model extracts feature information of an image containing a curled target object, and performs feature equalization on the feature information to obtain equalized features, including: extracting at least one layer of feature information of the image containing the curled target object through the encoding module; performing feature equalization on the feature information in the channel dimension and the spatial dimension according to the attention mechanism to obtain equalized features.

2. The image restoration method according to claim 1, characterized in that: Extracting at least one layer of feature information of the image containing the curled target object by the encoding module includes: Performing feature fusion on the feature information extracted by the deep convolutional layer of the encoding module to obtain deep features; The feature information extracted by the shallow convolutional layer of the encoding module is subjected to feature fusion to obtain shallow features.

3. The image restoration method according to claim 2, characterized in that: After extracting at least one layer of feature information of the image containing the curled target object through the encoding module, the method further includes: Splicing the deep features and the shallow features to obtain spliced ​​features; Performing convolution processing on the splicing features to obtain fusion features; Among them, the fusion features are feature balanced in the channel dimension and the spatial dimension according to the attention mechanism to obtain balanced features.

4. The image restoration method according to claim 1, characterized in that: The generator of the preset generative adversarial model also includes a decoding module, Accordingly, the equalization feature is restored to obtain the restored feature, including: The equalization feature is restored based on the decoding module to obtain a restored feature.

5. The image restoration method according to claim 1, characterized in that: The preset generative adversarial model is determined in the following way: Acquire a training image containing a curled target object and a real unfolded image corresponding to the training image; Using the training image and the real expanded image as training data to perform model training, and calculating a loss function; The model is optimized based on the back-propagation update algorithm until the loss function converges to obtain a preset generative adversarial model.

6. The image restoration method according to claim 5, characterized in that: The preset generative adversarial model also includes a discriminator, Accordingly, the training image and the real expanded image are used as training data to perform model training, and a loss function is calculated, including: Inputting the training image and the real expanded image as training data into the preset generative adversarial model; Determining an initial restored image corresponding to the training image by using the generator of the preset generative adversarial model; Determining the image category of the initial restored image based on a discriminator of a preset generative adversarial model; Determine the loss function according to the determination result; The image categories include training restored images and real expanded images.

7. The image restoration method according to claim 6, characterized in that: The discriminator based on the preset generative adversarial model determines the image category of the initial restored image, including: Extracting feature information of the initial restored image at multiple scales to obtain initial restored features at each scale; Respectively connecting the initial restored features of each scale to obtain a connected feature; Classifying the connection feature information to obtain a classification result; The image category of the initial restored image is determined according to the classification result.

8. The image restoration method according to claim 7, characterized in that: The step of extracting feature information of the initial restored image at multiple scales to obtain initial restored features at each scale includes: Extracting feature information of the initial restored image at multiple scales through a convolution module including multiple convolution layers; The feature information extracted by different convolutional layers at each scale is skipped and connected to obtain the initial restored features of each scale.

9. The image restoration method according to claim 5, characterized in that: Before using the training image and the real expanded image as training data for model training, the method further includes: Image processing is performed on the training image and the real expanded image to obtain a processed training image and a processed real expanded image.

10. An image restoration device, characterized in that: include: An extraction module, configured to extract feature information of an image containing a curled target object based on a generator of a preset generative adversarial model, and perform feature equalization processing on the feature information to obtain equalized features; A connection module, used for performing restoration processing on the equalization feature to obtain a restoration feature, and connecting the equalization feature and the restoration feature to obtain a target restored image; Among them, the generator of the preset generative adversarial model includes an encoding module; the extraction module is specifically used to: extract at least one layer of feature information of the image containing the curled target object through the encoding module; and perform feature equalization on the feature information in the channel dimension and the spatial dimension according to the attention mechanism to obtain equalized features.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the image restoration method as described in any one of claims 1 to 9 is implemented.

12. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the image restoration method according to any one of claims 1 to 9 when executed by a computer processor.

Images