Image restoration model training method and image restoration method, device and equipment

Abstract

The present application discloses a training method, device and equipment for an image restoration model, which performs a random mask operation on an original image to generate a defective image. The defective image is separated into multiple single-channel images, and the multiple single-channel images are respectively input into a single-channel image restoration module to obtain a single-channel predicted image corresponding to each single-channel image. The image restoration model includes a single-channel image restoration module. The single-channel predicted images corresponding to the multiple single-channel images are merged to obtain a predicted image. The image restoration model is trained by the loss value of the image restoration model obtained from the original image and the predicted image until a preset condition is met. Based on the hierarchical restoration of multiple single-channel images, the semantic information of the image can be fully learned. The overall pixel value of the predicted image can reduce the occurrence of distortions such as ripples and distortions in the repaired area. In this way, the trained image restoration model has a better restoration effect on defective images.

Description

Technical Field

[0001] The present application relates to the field of image processing, and specifically to a training method, device and equipment for an image restoration model, and an image restoration method, device and equipment. Background Art

[0002] The main purpose of image restoration is to repair the incomplete parts of an image. Currently, the restoration model is usually used to directly predict the pixel values ​​of the incomplete parts to achieve the purpose of restoring the image. However, this image restoration method is prone to cause distortion such as ripples and distortions in the repaired area. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a training method, device and equipment for an image restoration model, and an image restoration method, device and equipment, which can improve the image restoration effect and reduce the restoration distortion.

[0004] To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for training an image restoration model, the method comprising:

[0006] Perform random masking operations on the original image to generate a defect image;

[0007] Separating the defect image into a plurality of single-channel images;

[0008] Inputting the multiple single-channel images into the single-channel image restoration module respectively to obtain the single-channel prediction image corresponding to each single-channel image; the image restoration model includes the single-channel image restoration module;

[0009] Merging the single-channel prediction images corresponding to the multiple single-channel images to obtain a prediction image;

[0010] Obtaining a loss value of the image restoration model according to the original image and the predicted image;

[0011] The image restoration model is trained according to the loss value, and the steps of respectively inputting the multiple single-channel images into the single-channel image restoration module to obtain the single-channel prediction image corresponding to each single-channel image and subsequent steps are repeatedly performed until the preset conditions are met.

[0012] In a second aspect, an embodiment of the present application provides an image restoration method, the method comprising:

[0013] Separate the image to be repaired into multiple single-channel images to be repaired;

[0014] Inputting the multiple single-channel images to be repaired into the single-channel image repair module of the image repair model respectively to obtain multiple repaired single-channel images; the image repair model is trained according to any one of the above-mentioned image repair model training methods;

[0015] The multiple restored single-channel images are merged to obtain a restored image.

[0016] In a third aspect, an embodiment of the present application provides a training device for an image restoration model, the device comprising:

[0017] A generating unit, used for performing a random mask operation on the original image to generate a defect image;

[0018] A separation unit, used for separating the defect image into a plurality of single-channel images;

[0019] An input unit, used to input the multiple single-channel images into the single-channel image restoration module respectively, to obtain a single-channel prediction image corresponding to each single-channel image; the image restoration model includes the single-channel image restoration module;

[0020] a merging unit, configured to merge the single-channel prediction images respectively corresponding to the multiple single-channel images to obtain a prediction image;

[0021] An acquisition unit, used for acquiring a loss value of the image restoration model according to the original image and the predicted image;

[0022] A training unit is used to train the image restoration model according to the loss value, repeatedly input the multiple single-channel images into the single-channel image restoration module respectively, obtain the single-channel prediction image corresponding to each single-channel image respectively, and subsequent steps until the preset conditions are met.

[0023] In a fourth aspect, an embodiment of the present application provides an image restoration device, the device comprising:

[0024] A separation unit, used for separating the image to be restored into a plurality of single-channel images to be restored;

[0025] An input unit, used to input the multiple single-channel images to be repaired into the single-channel image repair module of the image repair model respectively, to obtain multiple repaired single-channel images; the image repair model is trained according to any one of the above-mentioned image repair model training methods;

[0026] The merging unit is used to merge the multiple restored single-channel images to obtain a restored image.

[0027] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0028] one or more processors;

[0029] a storage device having one or more programs stored thereon,

[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-described image restoration model training methods or any of the above-described image restoration methods.

[0031] In a sixth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, it implements a training method for an image restoration model as described in any one of the above, or an image restoration method as described in any one of the above.

[0032] In the seventh aspect, an embodiment of the present application provides a computer program product, which includes a computer program / instructions, and when the computer program / instructions are executed by a processor, implements a training method for an image restoration model as described in any of the above, or an image restoration method as described in any of the above.

[0033] It can be seen that the embodiments of the present application have the following beneficial effects:

[0034] The embodiment of the present application provides a training method, device and equipment for an image restoration model, first performing a random mask operation on the original image to generate a defective image. Then the defective image is separated into multiple single-channel images, and the multiple single-channel images are respectively input into the single-channel image restoration module to obtain the single-channel prediction images corresponding to each single-channel image. The image restoration model includes a single-channel image restoration module. Then, the single-channel prediction images corresponding to the multiple single-channel images are merged to obtain the prediction image. The loss value of the image restoration model is obtained from the original image and the prediction image. The image restoration model is trained using the loss value until the preset condition is reached. It can be seen that in the embodiment of the present application, based on the hierarchical restoration of multiple single-channel images, the semantic information of the image can be fully learned. In addition, instead of only predicting the pixel values ​​of the defective part of the image, the training method can predict the overall pixel value of the image, which can reduce the occurrence of distortion such as ripples and distortions in the repaired area. In this way, the training method of the image restoration model provided by the embodiment of the present application can make the trained image restoration model have a better restoration effect on the defective image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a framework of an exemplary application scenario provided in an embodiment of the present application;

[0036] Figure 2A flowchart of a method for training an image restoration model provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of multiple single-channel images provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of the structure of a single-channel image restoration module provided in an embodiment of the present application;

[0039] Figure 5 A flowchart of an image restoration method provided in an embodiment of the present application;

[0040] Figure 6 A schematic diagram of the structure of a training device for an image restoration model provided in an embodiment of the present application;

[0041] Figure 7 A schematic diagram of the structure of an image restoration device provided in an embodiment of the present application;

[0042] Figure 8 A schematic diagram of the basic structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0044] In order to facilitate the understanding and explanation of the technical solutions provided by the embodiments of the present application, the background technology involved in the embodiments of the present application is first introduced.

[0045] The main purpose of image restoration is to repair the incomplete parts of an image. Currently, the restoration model is usually used to directly predict the pixel values ​​of the incomplete parts to achieve the purpose of restoring the image. However, this image restoration method is prone to cause distortion such as ripples and distortions in the repaired area.

[0046] Based on this, the embodiment of the present application provides a training method, device and equipment for an image restoration model, first performing a random mask operation on the original image to generate a defective image. Then the defective image is separated into multiple single-channel images, and the multiple single-channel images are respectively input into the single-channel image restoration module to obtain the single-channel prediction images corresponding to each single-channel image. The image restoration model includes a single-channel image restoration module. Furthermore, the single-channel prediction images corresponding to the multiple single-channel images are merged to obtain the prediction image. The loss value of the image restoration model is obtained from the original image and the prediction image. The image restoration model is trained using the loss value until the preset conditions are met. It can be seen that the semantic information of the image can be fully learned based on the hierarchical restoration of multiple single-channel images. In addition, instead of only predicting the pixel values ​​of the defective part, the training method can predict the overall pixel values ​​of the image, which can reduce the occurrence of distortions such as ripples and distortions in the repaired area. In this way, the training method of the image restoration model provided by the embodiment of the present application can make the trained image restoration model have a better restoration effect on the defective image.

[0047] In order to facilitate understanding of the training method of the image restoration model provided in the embodiment of the present application, Figure 1 See the example scenario shown in the figure. Figure 1 As shown, this figure is a framework diagram of an exemplary application scenario provided in an embodiment of the present application.

[0048] In practical applications, the original image is first obtained. The original image can be understood as a real image or a complete image, that is, an image without defects. A random mask operation is performed on the original image to generate a defect image. The defect image is the damaged image to be repaired. The original image can be regarded as a label image of the defect image.

[0049] Then, the defect image is separated into multiple single-channel images. Figure 1 As shown, as an optional example, the multiple single-channel images may be an R single-channel image, a G channel image, and a B channel image.

[0050] After acquiring multiple single-channel images, the multiple single-channel images are respectively input into the single-channel image restoration module of the image restoration model to obtain the single-channel prediction image corresponding to each single-channel image. For example, the R single-channel image is input into the single-channel image restoration module to obtain the R single-channel prediction image output by the single-channel image restoration module; the G single-channel image is input into the single-channel image restoration module to obtain the G single-channel prediction image output by the single-channel image restoration module; the B single-channel image is input into the single-channel image restoration module to obtain the B single-channel prediction image output by the single-channel image restoration module. The single-channel prediction image can be understood as the image obtained after the single-channel image is restored.

[0051] The single-channel predicted images corresponding to multiple single-channel images are merged to obtain a predicted image. The predicted image can be understood as the image obtained after the defective image is repaired. The predicted image is the output of the image repair model.

[0052] According to the original image and the predicted image, the loss value of the image restoration model is obtained. In order to make the loss values ​​of the predicted image and the original image within a preset range, it is necessary to train the image restoration model according to the loss value, and repeatedly input multiple single-channel images into the single-channel image restoration module, obtain the single-channel predicted image corresponding to each single-channel image, and the subsequent steps until the preset condition is met. As an optional example, the preset condition may be that the loss values ​​of the predicted image and the original image are within a preset range.

[0053] Those skilled in the art will understand that Figure 1 The framework diagram shown is only an example in which the embodiments of the present application can be implemented. The scope of application of the embodiments of the present application is not limited by any aspect of the framework.

[0054] To facilitate understanding of the present application, a training method for an image restoration model provided in an embodiment of the present application is described below in conjunction with the accompanying drawings.

[0055] See also Figure 2 As shown in FIG. 1 , this figure is a flow chart of a training method for an image restoration model provided in an embodiment of the present application, such as Figure 2 As shown, the method may include S201-S206:

[0056] S201: Perform a random masking operation on the original image to generate a defect image.

[0057] The original image is obtained, and the original image can be understood as a real image or a complete image, that is, an image without defects. In one or more embodiments, the original image is a color image. In the embodiment of the present application, the image restoration model is trained in a self-supervised manner, that is, the original image itself is used as a supervision signal of the image restoration model.

[0058] In order to train the image restoration model, it is also necessary to obtain a damaged image. The damaged image is used as the input of the image restoration model so that the image restoration model can restore the damaged image and obtain the output of the image restoration model. The image restoration model is trained using the output of the image restoration model and the original image as a supervisory signal. As an optional example, a damaged image is obtained based on the original image. In specific implementation, a random mask operation can be performed on the original image to generate a defective image. The generated defective image is the damaged image to be restored. It can be understood that the original image can be regarded as a label image of the defective image.

[0059] The random mask operation is an operation of completely or partially blocking an image, and the blocked portion of the image is random.

[0060] See also Figure 3 , Figure 3 A schematic diagram of multiple single-channel images provided in an embodiment of the present application. Figure 3 In a possible implementation, the present application embodiment provides a specific implementation method of performing a random mask operation on an original image to generate a defect image, including:

[0061] A1: Divide the original image according to a preset size to obtain a first number of original image blocks; the size of each original image block is the preset size.

[0062] In a possible implementation, in order to facilitate processing of the original image, the size of the original image may be first changed to a set standard size. For example, if the standard size is 256*256, the size of the original image is first changed to 256*256. It should be noted that Figure 3 The original image shown is a color image (not shown in the figure).

[0063] Then, a preset size is determined, and the original image changed to the standard size is divided according to the preset size to obtain a first number of original image blocks. For example, the preset size is 16*16. The 256*256 original image is divided according to the preset size 16*16 to obtain 256 original image blocks. Among them, the first number is 256. The size of the original image blocks is the preset size, that is, 16*16.

[0064] It is understandable that the embodiments of the present application do not limit the standard size and the preset size, and can be set according to actual conditions.

[0065] A2: A first number of original image blocks are randomly masked according to a preset mask ratio to generate a defect image; the defect image is composed of a second number of original image blocks and a third number of defect image blocks; the sum of the second number and the third number is the first number, and the third number is the product of the first number and the preset mask ratio.

[0066] After obtaining a first number of original image blocks, a preset mask ratio is determined, and a random mask operation is performed on the first number of original image blocks according to the preset mask ratio to generate a defect image.

[0067] Specifically, a preset mask ratio is set first. The number of original image blocks subjected to random masking operation is the product of the first number and the preset mask ratio, which can be represented by a third number. The number of remaining original image blocks is the second number, and the sum of the first number and the second number is the first number.

[0068] Then, a third number of original image blocks are randomly selected from the first number of original image blocks for masking operation to obtain a third number of defect image blocks. Based on this, the defect image is composed of the second number of original image blocks and the third number of defect image blocks.

[0069] It can be understood that the random mask operation is to randomly select a third number of original image blocks from the first number of original image blocks to perform the mask operation. Figure 3 As shown, different from the image blocks of the original image, the image blocks displayed in gray in the defect image are obtained after performing a random mask operation on the image blocks of the original image.

[0070] In one or more embodiments, the preset mask ratio is preferably 35%-75%. For example, the preset mask ratio is set to 50%. Then, when there are 256 original image blocks, the third number is determined to be 256*50%=128 according to the preset mask ratio. Then, 128 original image blocks are randomly selected from the 256 original image blocks for masking operations, and 128 defective image blocks are obtained.

[0071] Among them, the mask operation refers to setting the pixel values ​​in the original image block to a uniform fixed value. It is understandable that the fixed value can be any value between the pixel values ​​0-255. The present application does not limit the fixed value and can be set according to actual needs. For example, the fixed value can be 0 (indicating black), that is, the pixel values ​​in the 128 original image blocks are all set to 0. After the random mask operation, the 128 original image blocks are displayed as black. For another example, the fixed value can also be 255 (indicating white), that is, the pixel values ​​in the 128 original image blocks are all set to 255. After the random mask operation, the 128 original image blocks are displayed as white.

[0072] S202: Separate the defect image into multiple single-channel images.

[0073] After acquiring the defect image, the defect image is separated into multiple single-channel images. As an optional example, Figure 3 As shown, the multiple single-channel images may be an R single-channel image, a G channel image, and a B channel image.

[0074] It can be understood that, based on A1-A2, the single-channel image is composed of a second number of single-channel original image blocks and a third number of single-channel defect image blocks, wherein the single-channel original image blocks are generated after the original image blocks are separated into RGB channels, and the single-channel defect image blocks are generated after the defect images are separated into RGB channels.

[0075] S203: Inputting a plurality of single-channel images into a single-channel image restoration module respectively to obtain a single-channel predicted image corresponding to each single-channel image; the image restoration model includes a single-channel image restoration module.

[0076] After obtaining a plurality of single-channel images, the plurality of single-channel images are respectively input into the single-channel image restoration module, so that the single-channel image restoration module performs restoration on each single-channel image, and obtains a single-channel prediction image corresponding to each single-channel image. The restoration of the single-channel image can be understood as predicting the pixel value in the single-channel image.

[0077] It should be noted that the image restoration model includes a single-channel image restoration module. In a possible implementation, the embodiment of the present application provides a specific implementation method of inputting multiple single-channel images into the single-channel image restoration module to obtain a single-channel prediction image corresponding to each single-channel image. For details, please see B1-B6 below.

[0078] S204: Merge the single-channel prediction images corresponding to the multiple single-channel images to obtain a prediction image.

[0079] After obtaining the single-channel prediction image corresponding to each single-channel image, the single-channel prediction images corresponding to multiple single-channel images are merged to obtain a prediction image. The prediction image is the output of the image restoration model. Each pixel value in the prediction image is a predicted value of the pixel value at the corresponding position in the defect image. The prediction image is the image obtained after the defect image is restored using the image restoration model.

[0080] It can be understood that the predicted image is an image composed of the predicted values ​​of the overall pixel values ​​in the defective image, which can reflect the global characteristics of the image. In this way, instead of only predicting the pixel values ​​of the incomplete part of the image, the prediction of the overall pixel values ​​in the defective image in the embodiment of the present application can reduce the occurrence of distortion such as ripples and distortions in the repaired image, and improve the repair effect of the image repair model.

[0081] In a possible implementation, the embodiment of the present application provides a specific implementation method of merging the single-channel prediction images corresponding to the multiple single-channel images in S204 to obtain the prediction image, which is specifically described in C1-C2 below.

[0082] S205: Obtain a loss value of the image restoration model according to the original image and the predicted image.

[0083] After obtaining the predicted image, the loss value of the image restoration model is obtained according to the original image and the predicted image. The original image is the supervision signal for training the image restoration model in a self-supervised manner. The purpose of training the image restoration model is to reduce the loss value until it meets the preset range.

[0084] S206: Train the image restoration model according to the loss value, and repeatedly input multiple single-channel images into the single-channel image restoration module to obtain the single-channel prediction image corresponding to each single-channel image and subsequent steps until the preset conditions are met.

[0085] After obtaining the loss value of the image restoration model, the image restoration model is trained according to the loss value, and it is determined whether the preset conditions are met. When the preset conditions are met, the training of the image restoration model is stopped. When the preset conditions are not met, the steps of inputting multiple single-channel images into the single-channel image restoration module are repeated to obtain the single-channel prediction images corresponding to each single-channel image and the subsequent steps until the preset conditions are met.

[0086] As an optional example, the preset condition may be that the loss value of the predicted image and the original image is within a preset range. As another optional example, the preset condition may also be a preset number of training times achieved. It is understood that the embodiments of the present application do not limit the preset range and the preset number of training times, and may be set according to actual conditions.

[0087] It can be understood that when the image restoration model only includes a single-channel image restoration module, training the image restoration model is actually training the single-channel image restoration module.

[0088] Based on the contents of S201-S206, it can be known that the embodiment of the present application provides an image restoration model training method based on a self-supervision mechanism. In the image restoration model training method, multiple single-channel images of defective images are restored in layers, which can fully learn the semantic information of the image and achieve better restoration effects. In addition, instead of only predicting the pixel values ​​of the incomplete parts of the image, the training method can predict the overall pixel values ​​of the image, which can reduce the occurrence of distortions such as ripples and distortions in the restored image. In this way, through the training method of the image restoration model provided by the embodiment of the present application, the trained image restoration model can have a better restoration effect on defective images.

[0089] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of a single-channel image restoration module provided in an embodiment of the present application. Figure 4 As shown, the single-channel image restoration module includes a feature encoder and a feature decoder.

[0090] Based on this, in a possible implementation, the embodiment of the present application provides a specific implementation of inputting multiple single-channel images into the single-channel image restoration module in S203 to obtain a single-channel prediction image corresponding to each single-channel image, including:

[0091] B1: flattening a second number of single-channel original image blocks in a target single-channel image to obtain a first image feature; the target single-channel image is each of the multiple single-channel images.

[0092] For the convenience of description, each of the multiple single-channel images is described by a target single-channel image. Figure 4 As shown, the target single-channel image is taken as a G-channel image as an example for explanation.

[0093] Based on A1-A2, it can be known that when the number of single-channel original image blocks is the second number, a flattening operation is performed on the second number of single-channel original image blocks in the target single-channel image to obtain the first image feature.

[0094] As an optional example, the flatten operation is specifically a tensor flattening Flatten operation, and the Flatten operation is used to flatten an array.

[0095] For example, when the size of each single-channel original image block is 16*16, after the single-channel original image block is flattened, a pixel value sequence of 1*256 dimension is obtained. When the second number is 128, each single-channel original image block in the second number of single-channel original image blocks is flattened in sequence, and the dimension of the first image feature finally obtained is 128*256. Among them, the dimension of each row feature of the first image feature is 1*256, and each row feature can be described by a pixel value sequence feature (referred to as a sequence feature), that is, the first image feature consists of 128 sequence features.

[0096] B2: Perform a flattening operation on a third number of single-channel defect image blocks in the target single-channel image to obtain a second image feature.

[0097] Based on A1-A2, it can be known that when the number of single-channel defective image blocks is the third number, a flattening operation is performed on the third number of single-channel defective image blocks in the target single-channel image to obtain a second image feature.

[0098] For example, when the size of each single-channel defect image block is 16*16, after flattening the single-channel defect image block, a pixel value sequence of 1*256 dimension is obtained. When the third number is 128, each single-channel original image block in the third number of single-channel defect image blocks is flattened in sequence, and the dimension of the second image feature finally obtained is 128*256. Among them, the dimension of each row feature of the second image feature is 1*256, and each row feature can be described by a sequence feature, that is, the second image feature consists of 128 sequence features.

[0099] B3: Input the first image feature into the feature encoder so that the feature encoder extracts the first image feature to obtain the third image feature.

[0100] After obtaining the first image feature, the first image feature is input into the feature encoder. The feature encoder is used to extract the input first image feature to obtain the third image feature. It should be noted that the dimension of the third image feature is the same as the dimension of the first image feature. For example, if the dimension of the first image feature is 128*256, the dimension of the third image feature is also 128*256, and the third image feature is also composed of 128 sequence features.

[0101] As an optional example, the number of feature encoders is at least one. When there are multiple feature encoders, the multiple feature encoders are connected in series. In one or more embodiments, the preferred value of the number of feature encoders is 5-6.

[0102] B4: Concatenate the second image feature and the third image feature to obtain a fourth image feature.

[0103] The third image feature output by the feature encoder is concatenated with the second image feature to obtain a fourth image feature.

[0104] As an optional example, the sequence features in the second image feature are randomly inserted into the third image feature to perform feature concatenation to obtain a fourth image feature. For example, 128 sequence features of the second image feature are randomly inserted into 128 sequence features of the third image feature, and the dimension of the fourth image feature obtained is 256*256. The fourth image feature consists of 256 sequence features.

[0105] B5: Input the fourth image feature into the feature decoder so that the feature decoder predicts each value in the fourth image feature to obtain the fifth image feature.

[0106] The obtained fourth image feature is input into the feature decoder so that the feature decoder predicts each value in the fourth image feature to obtain the fifth image feature. For example, when the dimension of the fourth image feature is 256*256, the feature decoder is used to predict the value in the 256 sequence features of the fourth image feature.

[0107] It can be understood that the fifth image feature output by the feature decoder is an image feature obtained after fully learning the image semantic information.

[0108] As an optional example, the number of feature decoders is at least one. When there are multiple feature decoders, the multiple feature decoders are connected in series. In one or more embodiments, the preferred value of the number of feature decoders is 5-6.

[0109] B6: According to the fifth image feature, obtain a single-channel predicted image corresponding to the target single-channel image.

[0110] As an optional example, the size of each sequence feature in the fifth image feature is changed to a preset size of an image block to obtain a plurality of image blocks, and then the plurality of image blocks constitute a single-channel prediction image corresponding to the target single-channel image. The size of the obtained single-channel prediction image is a standard size. The image structure of the obtained single-channel prediction image is an image structure obtained after fully learning the image semantic information.

[0111] For example, the preset size is 16*16, and the sequence feature with a dimension of 1*256 is changed to an image block of 16*16. Based on this, after the size of each sequence feature in the fifth image feature with a dimension of 256*256 is changed to the preset size of the image block, 256 image blocks can be obtained, and then, the image composed of the spliced ​​256 image blocks is the single-channel prediction image corresponding to the target single-channel image. Figure 4 As shown, through B1-B6, the G channel prediction image corresponding to the G channel image can be obtained.

[0112] Based on the contents of B1-B6, it can be seen that the single-channel image is obtained by separating the defective image, and is essentially a defective image. Through the feature encoder and feature decoder, the semantic information of the single-channel image can be fully learned, and the image structure of the single-channel image can be repaired to obtain the repaired single-channel prediction image.

[0113] As an optional example, the image restoration model also includes a smoothing processing module.

[0114] Based on this, in a possible implementation, the embodiment of the present application provides a specific implementation of merging the single-channel predicted images corresponding to the multiple single-channel images in S204 to obtain the predicted image, including:

[0115] C1: Merge the single-channel prediction images corresponding to multiple single-channel images to obtain a merged image.

[0116] The single-channel image restoration module is used to predict / restore a single-channel image to obtain a single-channel predicted image. Then, the obtained multiple restored single-channel predicted images are merged to obtain a merged image. For example, when the single-channel predicted image is an R channel predicted image, a G channel predicted image, and a B channel predicted image, the R channel predicted image, the G channel predicted image, and the B channel predicted image are merged to obtain a merged image.

[0117] C2: Input the merged image into the smoothing processing module, so that the smoothing processing module performs a smoothing operation on the merged image to obtain a predicted image.

[0118] Since when repairing a single-channel image, each image block in the single-channel image is essentially repaired, there may be small burrs between different image blocks. In order to make the repair effect of the obtained predicted image better, the merged image is input into the smoothing processing module, so that the smoothing processing module performs a smoothing operation on the merged image to obtain the predicted image.

[0119] It can be understood that the smoothing processing module can expand the receptive field of the image block, making the splicing between the image blocks more natural and smooth, and the restoration effect of the obtained predicted image is better.

[0120] As an optional example, the smoothing processing module is a gated convolutional neural network. For example, the smoothing processing module can be a 4-layer gated convolutional neural network.

[0121] It can be understood that after the smoothing module performs a smoothing operation on the merged image, the obtained predicted image is the output of the image restoration model.

[0122] Based on the contents of C1-C2, it can be known that the image restoration model also includes a smoothing processing module, so that the smoothing processing module performs smoothing on the merged image to obtain a predicted image with better restoration effect.

[0123] Based on the above content, it can be known that in one or more embodiments, the image restoration model includes a single-channel image restoration module and a smoothing processing module, and the single-channel image restoration module includes a feature encoder and a feature decoder.

[0124] Based on this, in a possible implementation, the embodiment of the present application provides a specific implementation method of training an image restoration model according to a loss value in S206, repeatedly inputting multiple single-channel images into a single-channel image restoration module, obtaining a single-channel prediction image corresponding to each single-channel image, and subsequent steps until a preset condition is met, including:

[0125] The feature encoder, feature decoder and smoothing processing module are trained according to the loss value, and multiple single-channel images are repeatedly input into the single-channel image restoration module to obtain the single-channel prediction image corresponding to each single-channel image and subsequent steps until the preset conditions are met.

[0126] Based on the obtained loss value, the feature encoder, feature decoder and smoothing processing module can be trained. Specifically, the feature encoder, feature decoder and smoothing processing module are jointly adjusted using the obtained loss value. And the above S203-S206 are repeatedly performed to train the feature encoder, feature decoder and smoothing processing module. Until the preset conditions are met, the training of the feature encoder, feature decoder and smoothing processing module is stopped to achieve the training of the image restoration model.

[0127] Based on the trained image restoration model, the embodiment of the present application also provides an image restoration method based on the image restoration model. To facilitate understanding of the present application, an image restoration method provided by the embodiment of the present application is described below in conjunction with the accompanying drawings.

[0128] See also Figure 5 , Figure 5 A flowchart of an image restoration method provided in an embodiment of the present application. Figure 5 As shown, the method may include S501-S503:

[0129] S501: Separate the image to be restored into multiple single-channel images to be restored.

[0130] The image to be repaired is a defective image, which needs to be repaired using an image repair model. First, the image to be repaired is separated into multiple single-channel images to be repaired.

[0131] In one possible implementation, the size of the image to be repaired may be first changed to a set standard size, and then the image to be repaired may be separated into a plurality of single-channel images to be repaired.

[0132] It should be noted that when training the image restoration model, if the single-channel image obtained by separating the defective image in S202 is an R channel image, a G channel image, and a B channel image, when the image restoration model is used to restore the image to be restored, the image to be restored is also separated into RGB three channels.

[0133] S502: Inputting a plurality of single-channel images to be repaired into a single-channel image repair module of an image repair model respectively to obtain a plurality of repaired single-channel images; the image repair model is trained according to the training method of the image repair model of any embodiment.

[0134] When the image restoration model includes a single-channel image restoration module, a plurality of single-channel images to be restored are respectively input into the single-channel image restoration module of the image restoration model to obtain a plurality of restored single-channel images.

[0135] S503: Merge multiple restored single-channel images to obtain a restored image.

[0136] The multiple restored single-channel images are merged to obtain a restored image. When the image to be restored is a color image, the restored image obtained after merging is also a color image.

[0137] It should be noted that the technical details of S501-S503 can be found in S201-S206 and will not be repeated here.

[0138] Based on the contents of S501-S503, when repairing the image to be repaired, the image to be repaired is first separated into multiple single-channel images to be repaired. Then, the multiple single-channel images to be repaired are respectively input into the single-channel image repair module of the trained image repair model to obtain multiple repaired single-channel images. Finally, the multiple repaired single-channel images are merged to obtain the repaired image. In this way, the repaired image has a better effect, and there are fewer distortions such as ripples and distortions in the repaired image.

[0139] It can be understood that the image to be repaired consists of a missing part image and a real part image.

[0140] As an optional example, the single-channel image restoration module includes a feature encoder and a feature decoder. Based on this, in a possible implementation, the embodiment of the present application provides a specific implementation method of inputting multiple single-channel images to be restored into the single-channel image restoration module in S502 to obtain multiple restored single-channel images, including:

[0141] D1: Flatten the real partial image in the target single-channel image to be repaired to obtain the first feature; the target single-channel image to be repaired is each of the multiple single-channel images to be repaired.

[0142] For the convenience of description, each single-channel image among the multiple single-channel images to be repaired is described using the target single-channel image to be repaired.

[0143] D2: Flatten the missing part of the target single-channel image to obtain the second feature.

[0144] As an optional example, the flatten operation is specifically a tensor flattening Flatten operation, and the Flatten operation is used to flatten an array.

[0145] D3: Input the first feature into the feature encoder so that the feature encoder extracts the first feature to obtain the third feature.

[0146] The first feature is input into a feature encoder, and the feature encoder extracts the first feature and outputs a third feature.

[0147] D4: Concatenate the second feature and the third feature to obtain the fourth feature.

[0148] After obtaining the third feature, the second feature and the third feature are concatenated to obtain the fourth feature.

[0149] It should be noted that the specific implementation method of feature splicing can be determined according to actual conditions and is not limited here.

[0150] D5: Input the fourth feature into the feature decoder so that the feature decoder predicts each value in the fourth feature to obtain the fifth feature.

[0151] The fourth feature is input into the feature decoder, and the feature decoder predicts each value in the fourth feature to obtain the fifth feature.

[0152] D6: According to the fifth feature, a restored single-channel image corresponding to the target single-channel image to be restored is obtained.

[0153] As an optional example, after obtaining the fifth feature, the size of the fifth feature is changed to the standard size of the image to obtain a single-channel predicted image corresponding to the target single-channel image.

[0154] It should be noted that the technical details of D1-D6 can be found in B1-B6 above and will not be repeated here.

[0155] In a possible implementation, the image restoration model further includes a smoothing processing module. Based on this, the present application embodiment provides a specific implementation of merging multiple restored single-channel images in S503 to obtain a restored image, including:

[0156] Merge multiple restored single-channel images to obtain a merged image to be processed;

[0157] The merged image to be processed is input into the smoothing processing module, so that the smoothing processing module performs a smoothing processing operation on the merged image to be processed to obtain a restored image.

[0158] It can be seen that the restoration effect of the image obtained after the smoothing processing of the smoothing processing module is better.

[0159] Based on the training method of an image restoration model provided by the above method embodiment, the embodiment of the present application also provides a training device for an image restoration model. The training device for the image restoration model will be described below in conjunction with the accompanying drawings.

[0160] See also Figure 6 As shown in FIG. 1 , this figure is a schematic diagram of the structure of a training device for an image restoration model provided in an embodiment of the present application. Figure 6 As shown, the training device of the image restoration model includes:

[0161] A generating unit 601 is used to perform a random masking operation on the original image to generate a defect image;

[0162] A separation unit 602, configured to separate the defect image into a plurality of single-channel images;

[0163] An input unit 603 is used to input the multiple single-channel images into the single-channel image restoration module respectively to obtain a single-channel prediction image corresponding to each single-channel image; the image restoration model includes the single-channel image restoration module;

[0164] A merging unit 604, configured to merge the single-channel prediction images respectively corresponding to the multiple single-channel images to obtain a prediction image;

[0165] An acquisition unit 605 is used to acquire a loss value of the image restoration model according to the original image and the predicted image;

[0166] The training unit 606 is used to train the image restoration model according to the loss value, repeatedly input the multiple single-channel images into the single-channel image restoration module respectively, obtain the single-channel prediction image corresponding to each single-channel image respectively, and the subsequent steps until the preset conditions are met.

[0167] In a possible implementation, the generating unit 601 includes:

[0168] A division subunit, configured to divide the original image according to a preset size to obtain a first number of original image blocks; the size of each of the original image blocks is the preset size;

[0169] A mask subunit is used to perform a random mask operation on the first number of original image blocks according to a preset mask ratio to generate a defect image; the defect image is composed of a second number of original image blocks and a third number of defect image blocks; the sum of the second number and the third number is the first number, and the third number is the product of the first number and the preset mask ratio.

[0170] In a possible implementation, the single-channel image is composed of a second number of single-channel original image blocks and a third number of single-channel defect image blocks; the single-channel image restoration module includes a feature encoder and a feature decoder, and the input unit 603 includes:

[0171] A first flattening subunit is used to perform a flattening operation on the second number of single-channel original image blocks in a target single-channel image to obtain a first image feature; the target single-channel image is each of the multiple single-channel images;

[0172] A second flattening subunit is used to perform a flattening operation on the third number of single-channel defect image blocks in the target single-channel image to obtain a second image feature;

[0173] a feature extraction subunit, configured to input the first image feature into the feature encoder, so that the feature encoder extracts the first image feature to obtain a third image feature;

[0174] a splicing subunit, configured to perform feature splicing on the second image feature and the third image feature to obtain a fourth image feature;

[0175] a prediction subunit, configured to input the fourth image feature into the feature decoder, so that the feature decoder predicts each value in the fourth image feature to obtain a fifth image feature;

[0176] The acquisition subunit is used to obtain the single-channel predicted image corresponding to the target single-channel image according to the fifth image feature.

[0177] In a possible implementation, the image restoration model further includes a smoothing processing module, and the merging unit 604 includes:

[0178] A merging subunit, used for merging the single-channel prediction images respectively corresponding to the multiple single-channel images to obtain a merged image;

[0179] The input subunit is used to input the merged image into the smoothing processing module, so that the smoothing processing module performs a smoothing operation on the merged image to obtain a predicted image.

[0180] In a possible implementation, the training unit 606 is specifically configured to:

[0181] The feature encoder, the feature decoder and the smoothing processing module are trained according to the loss value, and the steps of respectively inputting the multiple single-channel images into the single-channel image restoration module to obtain the single-channel prediction image corresponding to each single-channel image and subsequent steps are repeatedly performed until the preset conditions are met.

[0182] In a possible implementation, the smoothing processing module is a gated convolutional neural network.

[0183] In a possible implementation manner, the multiple single-channel images are an R-channel image, a G-channel image, and a B-channel image.

[0184] Based on an image restoration method provided by the above method embodiment, an embodiment of the present application further provides an image restoration device, which will be described below in conjunction with the accompanying drawings.

[0185] See also Figure 7 As shown in FIG. 1 , this figure is a schematic diagram of the structure of an image restoration device provided in an embodiment of the present application. Figure 7 As shown, the image restoration device comprises:

[0186] A separation unit 701 is used to separate the image to be restored into a plurality of single-channel images to be restored;

[0187] An input unit 702 is used to input the multiple single-channel images to be repaired into the single-channel image repair module of the image repair model respectively to obtain multiple repaired single-channel images; the image repair model is trained according to the training method of the image repair model described in any of the above embodiments;

[0188] The merging unit 703 is used to merge the multiple restored single-channel images to obtain a restored image.

[0189] In a possible implementation, the single-channel image restoration module includes a feature encoder and a feature decoder; the image to be restored is composed of a missing part image and a real part image, and the input unit 702 includes:

[0190] A first flattening subunit is used to flatten the real partial image in the target single-channel image to be repaired to obtain a first feature; the target single-channel image to be repaired is each of the multiple single-channel images to be repaired;

[0191] A second flattening subunit is used to flatten the missing part of the target single-channel image to obtain a second feature;

[0192] a feature extraction subunit, configured to input the first feature into the feature encoder, so that the feature encoder extracts the first feature to obtain a third feature;

[0193] A splicing subunit, used for splicing the second feature and the third feature to obtain a fourth feature;

[0194] a prediction subunit, configured to input the fourth feature into the feature decoder, so that the feature decoder predicts each value in the fourth feature to obtain a fifth feature;

[0195] The acquisition subunit is used to obtain the restored single-channel image corresponding to the target single-channel image to be restored according to the fifth feature.

[0196] In a possible implementation, the image restoration model further includes a smoothing processing module, and the merging unit 703 includes:

[0197] A merging subunit, used for merging the multiple restored single-channel images to obtain a merged image to be processed;

[0198] The input subunit is used to input the merged image to be processed into the smoothing processing module, so that the smoothing processing module performs a smoothing operation on the merged image to be processed to obtain a restored image.

[0199] Based on the image restoration model training method and image restoration method provided in the above method embodiments, the present application also provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the image restoration model training method described in any of the above embodiments, or the image restoration method described in any of the above embodiments.

[0200] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device 1300 suitable for implementing the embodiment of the present application. The terminal device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs (televisions), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0201] like Figure 8As shown, the electronic device 1300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage device 1306 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the electronic device 1300 are also stored. The processing device 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0202] Typically, the following devices may be connected to the I / O interface 1305: an input device 1306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1306 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1309. The communication device 1309 may allow the electronic device 1300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 1300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0203] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1309, or installed from the storage device 1306, or installed from the ROM 1302. When the computer program is executed by the processing device 1301, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0204] The electronic device provided in the embodiment of the present application and the training method of the image restoration model and the image restoration method provided in the above-mentioned embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above-mentioned embodiment, and this embodiment has the same beneficial effects as the above-mentioned embodiment.

[0205] Based on the training method of an image restoration model and an image restoration method provided in the above method embodiments, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the training method of an image restoration model as described in any of the above embodiments, or the image restoration method as described in any of the above embodiments is implemented.

[0206] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0207] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0208] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0209] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the image restoration model training method or image restoration method.

[0210] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0211] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0212] The units involved in the embodiments described in this application may be implemented by software or hardware. The name of a unit / module does not limit the unit itself in some cases. For example, a voice data acquisition module may also be described as a "data acquisition module".

[0213] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0214] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0215] According to one or more embodiments of the present application, [Example 1] provides a method for training an image restoration model, the method comprising:

[0216] Perform random masking operations on the original image to generate a defect image;

[0217] Separating the defect image into a plurality of single-channel images;

[0218] Inputting the multiple single-channel images into the single-channel image restoration module respectively to obtain the single-channel prediction image corresponding to each single-channel image; the image restoration model includes the single-channel image restoration module;

[0219] Merging the single-channel prediction images corresponding to the multiple single-channel images to obtain a prediction image;

[0220] Obtaining a loss value of the image restoration model according to the original image and the predicted image;

[0221] The image restoration model is trained according to the loss value, and the steps of respectively inputting the multiple single-channel images into the single-channel image restoration module to obtain the single-channel prediction image corresponding to each single-channel image and subsequent steps are repeatedly performed until the preset conditions are met.

[0222] According to one or more embodiments of the present application, [Example 2] provides a training method for an image restoration model, wherein a random mask operation is performed on an original image to generate a defective image, including:

[0223] Dividing the original image according to a preset size to obtain a first number of original image blocks; the size of each of the original image blocks is the preset size;

[0224] A random masking operation is performed on the first number of original image blocks according to a preset mask ratio to generate a defect image; the defect image is composed of a second number of original image blocks and a third number of defect image blocks; the sum of the second number and the third number is the first number, and the third number is the product of the first number and the preset mask ratio.

[0225] According to one or more embodiments of the present application, [Example 3] provides a training method for an image restoration model, wherein the single-channel image is composed of a second number of single-channel original image blocks and a third number of single-channel defect image blocks; the single-channel image restoration module includes a feature encoder and a feature decoder, and the multiple single-channel images are respectively input into the single-channel image restoration module to obtain a single-channel predicted image corresponding to each single-channel image, including:

[0226] Performing a flattening operation on the second number of single-channel original image blocks in a target single-channel image to obtain a first image feature; the target single-channel image is each of the multiple single-channel images;

[0227] Performing a flattening operation on the third number of single-channel defect image blocks in the target single-channel image to obtain a second image feature;

[0228] Inputting the first image feature into the feature encoder so that the feature encoder extracts the first image feature to obtain a third image feature;

[0229] Perform feature splicing on the second image feature and the third image feature to obtain a fourth image feature;

[0230] Inputting the fourth image feature into the feature decoder so that the feature decoder predicts each value in the fourth image feature to obtain a fifth image feature;

[0231] A single-channel predicted image corresponding to the target single-channel image is obtained according to the fifth image feature.

[0232] According to one or more embodiments of the present application, [Example 4] provides a training method for an image restoration model, wherein the image restoration model further includes a smoothing processing module, wherein the single-channel prediction images corresponding to the multiple single-channel images are merged to obtain a prediction image, including:

[0233] Merging the single-channel prediction images corresponding to the multiple single-channel images to obtain a merged image;

[0234] The merged image is input into the smoothing processing module, so that the smoothing processing module performs a smoothing operation on the merged image to obtain a predicted image.

[0235] According to one or more embodiments of the present application, [Example 5] provides a training method for an image restoration model, wherein the image restoration model is trained according to the loss value, and the steps of inputting the multiple single-channel images into the single-channel image restoration module are repeated to obtain the single-channel predicted images corresponding to each single-channel image and subsequent steps until a preset condition is met, including:

[0236] The feature encoder, the feature decoder and the smoothing processing module are trained according to the loss value, and the steps of respectively inputting the multiple single-channel images into the single-channel image restoration module to obtain the single-channel prediction image corresponding to each single-channel image and subsequent steps are repeatedly performed until the preset conditions are met.

[0237] According to one or more embodiments of the present application, [Example Six] provides a training method for an image restoration model, wherein the smoothing processing module is a gated convolutional neural network.

[0238] According to one or more embodiments of the present application, [Example 7] provides a training method for an image restoration model, wherein the multiple single-channel images are an R-channel image, a G-channel image, and a B-channel image.

[0239] According to one or more embodiments of the present application, [Example 8] provides an image restoration method, the method comprising:

[0240] Separate the image to be repaired into multiple single-channel images to be repaired;

[0241] Inputting the multiple single-channel images to be repaired into the single-channel image repair module of the image repair model respectively to obtain multiple repaired single-channel images; the image repair model is trained according to any one of the above-mentioned image repair model training methods;

[0242] The multiple restored single-channel images are merged to obtain a restored image.

[0243] According to one or more embodiments of the present application, [Example 9] provides an image restoration method, wherein the single-channel image restoration module includes a feature encoder and a feature decoder; the image to be restored is composed of a missing part image and a real part image, and the multiple single-channel images to be restored are respectively input into the single-channel image restoration module of the image restoration model to obtain multiple restored single-channel images, including:

[0244] Performing a flattening operation on the real partial image in the target single-channel image to be repaired to obtain a first feature; the target single-channel image to be repaired is each of the multiple single-channel images to be repaired;

[0245] Performing a flattening operation on the missing portion of the target single-channel image to obtain a second feature;

[0246] Inputting the first feature into the feature encoder so that the feature encoder extracts the first feature to obtain a third feature;

[0247] The second feature and the third feature are combined to obtain a fourth feature;

[0248] Inputting the fourth feature into the feature decoder so that the feature decoder predicts each value in the fourth feature to obtain a fifth feature;

[0249] According to the fifth feature, a restored single-channel image corresponding to the target single-channel image to be restored is obtained.

[0250] According to one or more embodiments of the present application, [Example 10] provides an image restoration method, wherein the image restoration model further includes a smoothing processing module, and the step of merging the multiple restored single-channel images to obtain a restored image includes:

[0251] Merging the multiple restored single-channel images to obtain a merged image to be processed;

[0252] The merged image to be processed is input into the smoothing processing module, so that the smoothing processing module performs a smoothing operation on the merged image to be processed to obtain a restored image.

[0253] According to one or more embodiments of the present application, [Example 11] provides a training device for an image restoration model, the device comprising:

[0254] A generating unit, used for performing a random mask operation on the original image to generate a defect image;

[0255] A separation unit, used for separating the defect image into a plurality of single-channel images;

[0256] An input unit, used to input the multiple single-channel images into the single-channel image restoration module respectively, to obtain a single-channel prediction image corresponding to each single-channel image; the image restoration model includes the single-channel image restoration module;

[0257] a merging unit, configured to merge the single-channel prediction images respectively corresponding to the multiple single-channel images to obtain a prediction image;

[0258] An acquisition unit, used for acquiring a loss value of the image restoration model according to the original image and the predicted image;

[0259] A training unit is used to train the image restoration model according to the loss value, repeatedly input the multiple single-channel images into the single-channel image restoration module respectively, obtain the single-channel prediction image corresponding to each single-channel image respectively, and subsequent steps until the preset conditions are met.

[0260] According to one or more embodiments of the present application, [Example 12] provides a generation unit, including:

[0261] A division subunit, configured to divide the original image according to a preset size to obtain a first number of original image blocks; the size of each of the original image blocks is the preset size;

[0262] A mask subunit is used to perform a random mask operation on the first number of original image blocks according to a preset mask ratio to generate a defect image; the defect image is composed of a second number of original image blocks and a third number of defect image blocks; the sum of the second number and the third number is the first number, and the third number is the product of the first number and the preset mask ratio.

[0263] According to one or more embodiments of the present application, [Example 13] provides a single-channel image consisting of a second number of single-channel original image blocks and a third number of single-channel defective image blocks; the single-channel image restoration module includes a feature encoder and a feature decoder, and the input unit includes:

[0264] A first flattening subunit is used to perform a flattening operation on the second number of single-channel original image blocks in a target single-channel image to obtain a first image feature; the target single-channel image is each of the multiple single-channel images;

[0265] A second flattening subunit is used to perform a flattening operation on the third number of single-channel defect image blocks in the target single-channel image to obtain a second image feature;

[0266] a feature extraction subunit, configured to input the first image feature into the feature encoder, so that the feature encoder extracts the first image feature to obtain a third image feature;

[0267] a splicing subunit, configured to perform feature splicing on the second image feature and the third image feature to obtain a fourth image feature;

[0268] a prediction subunit, configured to input the fourth image feature into the feature decoder, so that the feature decoder predicts each value in the fourth image feature to obtain a fifth image feature;

[0269] The acquisition subunit is used to obtain the single-channel predicted image corresponding to the target single-channel image according to the fifth image feature.

[0270] According to one or more embodiments of the present application, [Example 14] provides an image restoration model further including a smoothing processing module, and the merging unit includes:

[0271] A merging subunit, used for merging the single-channel prediction images respectively corresponding to the multiple single-channel images to obtain a merged image;

[0272] The input subunit is used to input the merged image into the smoothing processing module, so that the smoothing processing module performs a smoothing operation on the merged image to obtain a predicted image.

[0273] According to one or more embodiments of the present application, [Example 15] provides a training unit, which is specifically used for:

[0274] The feature encoder, the feature decoder and the smoothing processing module are trained according to the loss value, and the steps of respectively inputting the multiple single-channel images into the single-channel image restoration module to obtain the single-channel prediction image corresponding to each single-channel image and subsequent steps are repeatedly performed until the preset conditions are met.

[0275] According to one or more embodiments of the present application, [Example 16] provides a smoothing processing module that is a gated convolutional neural network.

[0276] According to one or more embodiments of the present application, [Example 17] provides a method in which the multiple single-channel images are an R channel image, a G channel image, and a B channel image.

[0277] According to one or more embodiments of the present application, [Example 18] provides an image restoration device, the device comprising:

[0278] A separation unit, used for separating the image to be restored into a plurality of single-channel images to be restored;

[0279] An input unit, used to input the multiple single-channel images to be repaired into the single-channel image repair module of the image repair model respectively, to obtain multiple repaired single-channel images; the image repair model is trained according to any one of the above-mentioned image repair model training methods;

[0280] The merging unit is used to merge the multiple restored single-channel images to obtain a restored image.

[0281] According to one or more embodiments of the present application, [Example 19] provides a single-channel image restoration module including a feature encoder and a feature decoder; the image to be restored consists of a missing part image and a real part image, and the input unit includes:

[0282] A first flattening subunit is used to flatten the real partial image in the target single-channel image to be repaired to obtain a first feature; the target single-channel image to be repaired is each of the multiple single-channel images to be repaired;

[0283] A second flattening subunit is used to flatten the missing part of the target single-channel image to obtain a second feature;

[0284] a feature extraction subunit, configured to input the first feature into the feature encoder, so that the feature encoder extracts the first feature to obtain a third feature;

[0285] A splicing subunit, used for splicing the second feature and the third feature to obtain a fourth feature;

[0286] a prediction subunit, configured to input the fourth feature into the feature decoder, so that the feature decoder predicts each value in the fourth feature to obtain a fifth feature;

[0287] The acquisition subunit is used to obtain the restored single-channel image corresponding to the target single-channel image to be restored according to the fifth feature.

[0288] According to one or more embodiments of the present application, [Example 20] provides an image restoration model further including a smoothing processing module, and the merging unit includes:

[0289] A merging subunit, used for merging the multiple restored single-channel images to obtain a merged image to be processed;

[0290] The input subunit is used to input the merged image to be processed into the smoothing processing module, so that the smoothing processing module performs a smoothing operation on the merged image to be processed to obtain a restored image.

[0291] According to one or more embodiments of the present application, [Example 21] provides an electronic device, including:

[0292] one or more processors;

[0293] a storage device having one or more programs stored thereon,

[0294] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-described image restoration model training methods or any of the above-described image restoration methods.

[0295] According to one or more embodiments of the present application, [Example 22] provides a computer-readable medium on which a computer program is stored, wherein when the program is executed by a processor, it implements the training method of the image restoration model as described in any of the above, or the image restoration method as described in any of the above.

[0296] According to one or more embodiments of the present application, [Example 23] provides a computer program product, which includes a computer program / instructions, and when the computer program / instructions are executed by a processor, implements the training method of the image restoration model as described in any of the above, or the image restoration method as described in any of the above.

[0297] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0298] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0299] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0300] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0301] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for an image restoration model, characterized in that: The method comprises: Perform random masking operations on the original image to generate a defect image; Separating the defect image into a plurality of single-channel images; Inputting the multiple single-channel images into the single-channel image restoration module respectively to obtain the single-channel prediction image corresponding to each single-channel image; the image restoration model includes the single-channel image restoration module; Merging the single-channel prediction images corresponding to the multiple single-channel images to obtain a prediction image; Obtaining a loss value of the image restoration model according to the original image and the predicted image; Training the image restoration model according to the loss value, repeatedly inputting the multiple single-channel images into the single-channel image restoration module, obtaining the single-channel prediction image corresponding to each single-channel image, and subsequent steps until a preset condition is met; The single-channel image is composed of a second number of single-channel original image blocks and a third number of single-channel defect image blocks; the single-channel image restoration module includes a feature encoder and a feature decoder, and the multiple single-channel images are respectively input into the single-channel image restoration module to obtain a single-channel predicted image corresponding to each single-channel image, including: Performing a flattening operation on the second number of single-channel original image blocks in a target single-channel image to obtain a first image feature; the target single-channel image is each of the multiple single-channel images; Performing a flattening operation on the third number of single-channel defect image blocks in the target single-channel image to obtain a second image feature; Inputting the first image feature into the feature encoder so that the feature encoder extracts the first image feature to obtain a third image feature; Perform feature splicing on the second image feature and the third image feature to obtain a fourth image feature; Inputting the fourth image feature into the feature decoder so that the feature decoder predicts each value in the fourth image feature to obtain a fifth image feature; A single-channel predicted image corresponding to the target single-channel image is obtained according to the fifth image feature.

2. The method according to claim 1, characterized in that: The step of performing a random mask operation on the original image to generate a defect image includes: Dividing the original image according to a preset size to obtain a first number of original image blocks; the size of each of the original image blocks is the preset size; A random masking operation is performed on the first number of original image blocks according to a preset mask ratio to generate a defect image; the defect image is composed of a second number of original image blocks and a third number of defect image blocks; the sum of the second number and the third number is the first number, and the third number is the product of the first number and the preset mask ratio.

3. The method according to claim 1, characterized in that: The image restoration model further includes a smoothing processing module, wherein the single-channel prediction images corresponding to the multiple single-channel images are merged to obtain a prediction image, including: Merging the single-channel prediction images corresponding to the multiple single-channel images to obtain a merged image; The merged image is input into the smoothing processing module, so that the smoothing processing module performs a smoothing operation on the merged image to obtain a predicted image.

4. The method according to claim 3, characterized in that The step of training the image restoration model according to the loss value, repeatedly inputting the multiple single-channel images into the single-channel image restoration module, obtaining the single-channel prediction image corresponding to each single-channel image, and subsequent steps until the preset conditions are met, includes: The feature encoder, the feature decoder and the smoothing processing module are trained according to the loss value, and the steps of respectively inputting the multiple single-channel images into the single-channel image restoration module to obtain the single-channel prediction image corresponding to each single-channel image and subsequent steps are repeatedly performed until the preset conditions are met.

5. The method according to claim 3, characterized in that: The smoothing processing module is a gated convolutional neural network.

6. The method according to claim 1, characterized in that The multiple single-channel images are an R-channel image, a G-channel image, and a B-channel image.

7. An image restoration method, characterized in that: The method comprises: Separate the image to be repaired into multiple single-channel images to be repaired; Inputting the multiple single-channel images to be repaired into the single-channel image repair module of the image repair model respectively to obtain multiple repaired single-channel images; the image repair model is trained according to the training method of the image repair model according to any one of claims 1 to 6; Merging the multiple restored single-channel images to obtain a restored image; The single-channel image restoration module includes a feature encoder and a feature decoder; the image to be restored is composed of a missing part image and a real part image, and the multiple single-channel images to be restored are respectively input into the single-channel image restoration module of the image restoration model to obtain multiple restored single-channel images, including: Performing a flattening operation on the real partial image in the target single-channel image to be repaired to obtain a first feature; the target single-channel image to be repaired is each of the multiple single-channel images to be repaired; Performing a flattening operation on the missing portion of the target single-channel image to obtain a second feature; Inputting the first feature into the feature encoder so that the feature encoder extracts the first feature to obtain a third feature; The second feature and the third feature are combined to obtain a fourth feature; Inputting the fourth feature into the feature decoder so that the feature decoder predicts each value in the fourth feature to obtain a fifth feature; According to the fifth feature, a restored single-channel image corresponding to the target single-channel image to be restored is obtained.

8. The method according to claim 7, characterized in that The image restoration model further includes a smoothing processing module, and the step of merging the plurality of restored single-channel images to obtain a restored image includes: Merging the multiple restored single-channel images to obtain a merged image to be processed; The merged image to be processed is input into the smoothing processing module, so that the smoothing processing module performs a smoothing operation on the merged image to be processed to obtain a restored image.

9. A training device for an image restoration model, characterized in that: The device comprises: A generating unit, used for performing a random mask operation on the original image to generate a defect image; A separation unit, used for separating the defect image into a plurality of single-channel images; An input unit, used to input the multiple single-channel images into the single-channel image restoration module respectively, to obtain a single-channel prediction image corresponding to each single-channel image; the image restoration model includes the single-channel image restoration module; a merging unit, configured to merge the single-channel prediction images respectively corresponding to the multiple single-channel images to obtain a prediction image; An acquisition unit, used for acquiring a loss value of the image restoration model according to the original image and the predicted image; A training unit, used for training the image restoration model according to the loss value, repeatedly inputting the multiple single-channel images into the single-channel image restoration module, obtaining the single-channel prediction image corresponding to each single-channel image, and subsequent steps, until a preset condition is met; The single-channel image is composed of a second number of single-channel original image blocks and a third number of single-channel defect image blocks; the single-channel image restoration module includes a feature encoder and a feature decoder, and the input unit is specifically used for: Performing a flattening operation on the second number of single-channel original image blocks in a target single-channel image to obtain a first image feature; the target single-channel image is each of the multiple single-channel images; Performing a flattening operation on the third number of single-channel defect image blocks in the target single-channel image to obtain a second image feature; Inputting the first image feature into the feature encoder so that the feature encoder extracts the first image feature to obtain a third image feature; Perform feature splicing on the second image feature and the third image feature to obtain a fourth image feature; Inputting the fourth image feature into the feature decoder so that the feature decoder predicts each value in the fourth image feature to obtain a fifth image feature; A single-channel predicted image corresponding to the target single-channel image is obtained according to the fifth image feature.

10. An image restoration device, characterized in that: The device comprises: A separation unit, used for separating the image to be restored into a plurality of single-channel images to be restored; An input unit, used to input the multiple single-channel images to be repaired into the single-channel image repair module of the image repair model respectively, to obtain multiple repaired single-channel images; the image repair model is trained according to the training method of the image repair model according to any one of claims 1 to 6; A merging unit, used for merging the multiple restored single-channel images to obtain a restored image; The single-channel image restoration module includes a feature encoder and a feature decoder; the image to be restored consists of a missing part image and a real part image, and the input unit is specifically used for: Performing a flattening operation on the real partial image in the target single-channel image to be repaired to obtain a first feature; the target single-channel image to be repaired is each of the multiple single-channel images to be repaired; Performing a flattening operation on the missing portion of the target single-channel image to obtain a second feature; Inputting the first feature into the feature encoder so that the feature encoder extracts the first feature to obtain a third feature; The second feature and the third feature are combined to obtain a fourth feature; Inputting the fourth feature into the feature decoder so that the feature decoder predicts each value in the fourth feature to obtain a fifth feature; According to the fifth feature, a restored single-channel image corresponding to the target single-channel image to be restored is obtained.

11. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the image restoration model as described in any one of claims 1-6, or the image restoration method as described in any one of claims 7-8.

12. A computer-readable medium, characterized in that A computer program is stored thereon, wherein when the program is executed by a processor, it implements the training method of the image restoration model as described in any one of claims 1-6, or the image restoration method as described in any one of claims 7-8.

13. A computer program product, characterized in that The computer program product includes a computer program / instructions, which, when executed by a processor, implements the training method for an image restoration model as described in any one of claims 1 to 6, or the image restoration method as described in any one of claims 7 to 8.

Images