Training method of image inpainting model, image inpainting method, device and equipment
By training an image restoration model through the synthesis and preprocessing of Bayer images, the problem of low quality of RGB format images in traffic capture scenarios was solved, achieving high-quality image conversion and improved recognition accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO LTD
- Filing Date
- 2022-11-24
- Publication Date
- 2026-06-09
AI Technical Summary
In traffic capture scenarios, the interference of light on the car window film results in low quality of the acquired RGB format images, affecting the recognizability of faces and objects inside the vehicle. Furthermore, there is a lack of high-definition open-source datasets with rainbow patterns, and conventional methods cannot effectively recover details after processing in the YUV domain.
By acquiring Bayer images of the scene to be repaired and the type of pattern to be removed in Bayer format, performing synthesis and preprocessing, training an image restoration model, converting the Bayer images to RGB images, and using a synthetic dataset to simulate image data, the loss in YUV domain processing is avoided.
It improves the quality of RGB format images, enhances the accuracy of image recognition, realizes direct conversion from Bayer format to RGB format, avoids loss of detail, and improves recognition performance.
Smart Images
Figure CN115731131B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to a training method for an image restoration model, an image restoration method, an apparatus, and a device. Background Technology
[0002] When interference or reflection occurs on the glass surface, patterns such as colored stripes and light spots are formed on the glass surface. If an image is captured on the glass at this time, the quality of the captured glass image will be affected.
[0003] For example, in traffic capture scenarios, it is necessary to collect images of passing vehicles and identify the corresponding faces and / or objects inside the vehicles from the vehicle images. However, when light interferes with the window film and causes colored stripes, the quality of the collected RGB format images is low, which further reduces the recognizability of faces and objects inside the vehicles. Summary of the Invention
[0004] The purpose of this application is to provide a training method for an image restoration model, an image restoration method, an apparatus, and a device to improve the quality of RGB format images. The specific technical solution is as follows:
[0005] This application provides a method for training an image restoration model, the method comprising:
[0006] Obtain a first Bayer image in Bayer format of the scene to be repaired, and a second Bayer image in Bayer format of the type of pattern to be removed;
[0007] The first Bayer image and the second Bayer image are combined to obtain the third Bayer image;
[0008] The first Bayer image is preprocessed to obtain a first RGB image in RGB format;
[0009] The third Bayer image is used as the input to the image restoration model, and the first RGB image is used as the ground truth of the image restoration output. The image restoration model is then trained to obtain a trained image restoration model.
[0010] In one possible embodiment, the step of compositing the first Bayer image and the second Bayer image to obtain the third Bayer image includes:
[0011] The second Bayer image is subjected to demosaic processing, saturation adjustment, and brightness adjustment to obtain the second Bayer target image.
[0012] The first Bayer image is demosaicized to obtain the first Bayer target image;
[0013] The first Bayer target image and the second Bayer target image are superimposed to obtain the third target image;
[0014] The third target image is downsampled to obtain the third Bayer image.
[0015] In one possible embodiment, the step of using the third Bayer image as input to the image restoration model and the first RGB image as the ground truth of the image restoration output to train the image restoration model and obtain a trained image restoration model includes:
[0016] The third Bayer image is input into the image restoration model for processing to obtain the predicted RGB image;
[0017] Calculate the difference between the predicted RGB image and the first RGB image. When the difference does not meet the preset requirements, adjust the parameters of the image restoration model based on the preset loss function.
[0018] When the preset termination condition is met, the trained image restoration model is obtained.
[0019] In one possible embodiment, the image inpainting model includes an ISP algorithm module, a pattern extraction module, and a pattern removal module. The step of inputting the third Bayer image into the image inpainting model for processing to obtain a predicted RGB image includes:
[0020] The third Bayer image is input into the ISP algorithm module for processing to obtain the first feature image;
[0021] The first feature image is input into the pattern extraction module for feature extraction to obtain a pattern feature image and a scene feature image;
[0022] The pattern feature image and the scene feature image are input into the pattern removal module for pattern removal to obtain the predicted RGB image.
[0023] In one possible embodiment, the image restoration model further includes an image post-processing module; the method further includes:
[0024] When the difference meets the preset requirements, the predicted RGB image is input into the image post-processing module to obtain the target RGB image.
[0025] In one possible embodiment, the preprocessing of the first bayer image to obtain a first RGB image in RGB format includes:
[0026] Perform Bayer noise reduction on the first Bayer image to obtain the first processed image;
[0027] The first processed image is processed based on the ISP algorithm to obtain the second processed image;
[0028] The second processed image is subjected to face enhancement processing to obtain the third processed image;
[0029] The brightness, saturation, and contrast of the third processed image are adjusted to obtain a first RGB image in RGB format.
[0030] This application also provides an image restoration method, the method comprising:
[0031] Obtain the image to be processed in Bayer format;
[0032] The image to be processed is input into a pre-trained image restoration model for processing to obtain an RGB image in RGB format corresponding to the image to be processed. The pre-trained image restoration model is trained based on any of the image restoration model training methods described above.
[0033] This application embodiment also provides a training device for an image restoration model, the device comprising:
[0034] The acquisition module is used to acquire a first Bayer image in Bayer format of the scene to be repaired, and a second Bayer image in Bayer format of the type of pattern to be removed.
[0035] The compositing module is used to compose the first Bayer image and the second Bayer image to obtain a third Bayer image;
[0036] The preprocessing module is used to preprocess the first bayer image to obtain a first RGB image in RGB format;
[0037] The training module is used to train the image restoration model by taking the third Bayer image as input and the first RGB image as the ground truth of the image restoration output, so as to obtain a trained image restoration model.
[0038] In one possible embodiment, the synthesis module includes:
[0039] The first processing submodule is used to perform demosaic processing, saturation adjustment, and brightness adjustment on the second bayer image to obtain the second bayer target image.
[0040] The second processing submodule is used to perform demosaic processing on the first bayer image to obtain the first bayer target image;
[0041] The overlay submodule is used to overlay the first Bayer target image and the second Bayer target image to obtain a third target image;
[0042] The third target image is downsampled to obtain the third Bayer image.
[0043] In one possible embodiment, the training module is specifically used for:
[0044] The third Bayer image is input into the image restoration model for processing to obtain the predicted RGB image;
[0045] Calculate the difference between the predicted RGB image and the first RGB image. When the difference does not meet the preset requirements, adjust the parameters of the image restoration model based on the preset loss function.
[0046] When the preset termination condition is met, the trained image restoration model is obtained.
[0047] In one possible embodiment, the image inpainting model includes an ISP algorithm module, a pattern extraction module, and a pattern removal module, wherein the training module is specifically used for:
[0048] The third Bayer image is input into the ISP algorithm module for processing to obtain the first feature image;
[0049] The first feature image is input into the pattern extraction module for feature extraction to obtain a pattern feature image and a scene feature image;
[0050] The pattern feature image and the scene feature image are input into the pattern removal module for pattern removal to obtain the predicted RGB image.
[0051] In one possible embodiment, the image restoration model further includes an image post-processing module, and the apparatus further includes a post-processing module.
[0052] The post-processing module is specifically used to input the predicted RGB image into the image post-processing module when the difference meets the preset requirements, so as to obtain the target RGB image.
[0053] In one possible embodiment, the preprocessing module is specifically used for:
[0054] Perform Bayer noise reduction on the first Bayer image to obtain the first processed image;
[0055] The first processed image is processed based on the ISP algorithm to obtain the second processed image;
[0056] The second processed image is subjected to face enhancement processing to obtain the third processed image;
[0057] The brightness, saturation, and contrast of the third processed image are adjusted to obtain a first RGB image in RGB format.
[0058] This application embodiment also provides an image restoration apparatus, the apparatus comprising:
[0059] The acquisition module is used to acquire images in Bayer format to be processed.
[0060] The processing module is used to input the image to be processed into a pre-trained image restoration model for processing, so as to obtain an RGB image in RGB format corresponding to the image to be processed, wherein the pre-trained image restoration model is trained based on the training device of any of the above-described image restoration models.
[0061] This application also provides an electronic device, including: a processor and a memory;
[0062] Memory, used to store computer programs;
[0063] The processor, when executing a program stored in memory, implements the training method or image restoration method of any of the image restoration models described above.
[0064] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the training method or image restoration method of any of the image restoration models described above.
[0065] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the training method or image restoration method of any of the image restoration models described above.
[0066] Beneficial effects of the embodiments in this application:
[0067] This application provides a method to obtain a third Bayer image by acquiring a first Bayer image in Bayer format of the scene to be repaired and a second Bayer image in Bayer format of the pattern to be removed. The first and second Bayer images are then synthesized. The first Bayer image is then preprocessed to obtain a first RGB image in RGB format. This method can simulate image data including the pattern to be removed. In the absence of a high-definition, open-source dataset of the scene to be repaired containing the pattern to be removed, a suitable dataset can be obtained through synthesis. The third Bayer image is used as input to the image inpainting model, and the first RGB image is used as the ground truth output of the image inpainting model. The trained image inpainting model can then directly convert Bayer format to RGB format images, avoiding the irreparable loss of detail and content during YUV domain processing. This improves the quality of the generated RGB format image, and further, enhances the accuracy of image recognition when using the RGB format image for identification.
[0068] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description
[0069] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.
[0070] Figure 1 A schematic diagram of a first flowchart illustrating the training method for the image restoration model provided in this application embodiment;
[0071] Figure 2a A second flowchart illustrating the training method for the image restoration model provided in this application embodiment;
[0072] Figure 2b A schematic diagram of the third process for training the image restoration model provided in the embodiments of this application;
[0073] Figure 2c A schematic diagram of the fourth process for training the image restoration model provided in this application embodiment;
[0074] Figure 3a A fifth flowchart illustrating the training method for the image restoration model provided in this application embodiment;
[0075] Figure 3b A sixth flowchart illustrating the training method for the image restoration model provided in this application embodiment;
[0076] Figure 4 This is a schematic flowchart of an image restoration method provided in an embodiment of this application;
[0077] Figure 5 A schematic diagram of a training device for an image restoration model provided in an embodiment of this application;
[0078] Figure 6 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application;
[0079] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0080] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.
[0081] In traffic camera scenarios, when light interferes with the car window film, resulting in colored stripes, the quality of the acquired RGB format image is low. Due to the lack of high-definition open-source datasets containing rainbow stripes, conventional deep learning methods for removing rainbow stripes from car windows mainly process the rainbow stripes on top of the YUV domain image, ultimately obtaining an RGB format image with the rainbow stripes removed. However, YUV domain images already suffer some degree of accuracy loss after a series of ISP (Image Signal Processor) algorithms, and even after image enhancement, some details cannot be recovered, thus limiting the improvement in image quality. To improve the quality of RGB format images, embodiments of this application provide a training method for an image restoration model, an image restoration method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product containing instructions.
[0082] The training method for the image restoration model provided in this application embodiment will be described below. This method can be applied to any electronic device that can provide image restoration model training services, such as a network video recorder, a personal computer, or a server. The training method for the image restoration model provided in this application embodiment can be implemented in at least one of software, hardware circuits, and logic circuits within an electronic device.
[0083] like Figure 1 As shown, Figure 1 This is a schematic diagram of a first flowchart of a training method for an image restoration model provided in an embodiment of this application. The method includes:
[0084] S110, acquire the first Bayer image in Bayer format of the scene to be repaired, and the second Bayer image in Bayer format of the type pattern to be removed;
[0085] S120, The first Bayer image and the second Bayer image are combined to obtain the third Bayer image;
[0086] S130, preprocess the first bayer image to obtain the first RGB image in RGB format;
[0087] S140, the third Bayer image is used as the input to the image restoration model, and the first RGB image is used as the ground truth of the image restoration output. The image restoration model is trained to obtain a trained image restoration model.
[0088] The scene to be repaired can be any scene where information needs to be captured and the captured images analyzed, such as traffic capture scenes or access control scenes. The types of patterns to be removed can be rainbow patterns (colored stripes appearing due to interference of light shining on car window film), light spots formed after light reflection, etc.
[0089] A third Bayer image can be synthesized based on a first Bayer image of the scene to be repaired and a second Bayer image of the pattern to be removed, generating a dataset. The method for synthesizing the third Bayer image from the first Bayer image of the scene to be repaired and the second Bayer image of the pattern to be removed can be as follows: The second Bayer image is demosaiced; then, the demosaiced image is processed for saturation, brightness, etc., to obtain the second Bayer target image; then, the first Bayer image is also demosaiced to obtain the first Bayer target image; finally, the first and second Bayer target images are superimposed to obtain the third Bayer image. For clarity and layout, another embodiment is described in detail below.
[0090] Simultaneously, the first bayer image needs to be preprocessed to obtain a first RGB image in RGB format. Preprocessing methods for the first bayer image can include: noise reduction, ISP algorithm processing, image feature enhancement, brightness adjustment, saturation adjustment, and contrast adjustment. The third bayer image is used as input to the image restoration model, and the first RGB image is used as the ground truth output for image restoration. The image restoration model is then trained to obtain a trained image restoration model.
[0091] By acquiring a first Bayer image in Bayer format representing the scene to be repaired, and a second Bayer image in Bayer format representing the pattern to be removed, the first and second Bayer images are synthesized to obtain a third Bayer image. The first Bayer image is then preprocessed to obtain a first RGB image in RGB format. Image data including the pattern to be removed can be simulated through synthesis. In the absence of a high-definition, open-source dataset of the scene to be repaired containing the pattern to be removed, a suitable dataset can be obtained through synthesis. The third Bayer image is used as input to the image inpainting model, and the first RGB image is used as the ground truth output for image inpainting. The image inpainting model is trained to obtain a trained image inpainting model. This trained model can directly convert Bayer format images to RGB format images, avoiding the irreparable loss of detail and content during YUV domain processing, thus improving the quality of the generated RGB format images. Furthermore, when using RGB format images for recognition, the accuracy of image recognition can be improved.
[0092] based on Figure 1 In this embodiment, the present application provides a second flowchart illustrating a training method for an image restoration model, as shown below. Figure 2a As shown, step S120 above can be achieved through the following steps:
[0093] S121, Perform demosaic processing, saturation adjustment and brightness adjustment on the above second bayer image to obtain the second bayer target image;
[0094] S122, Perform demosaic processing on the first bayer image to obtain the first bayer target image;
[0095] S123, the first bayer target image and the second bayer target image are superimposed to obtain the third target image;
[0096] S124, downsample the aforementioned third target image to obtain the third Bayer image.
[0097] For the second Bayer image, after obtaining it, demosaic processing can be performed to obtain a de-mosaiced second Bayer image. Then, saturation and brightness adjustments can be made to the de-mosaiced second Bayer image to obtain the target second Bayer image. Saturation adjustment can be performed first, followed by brightness adjustment; see [link to documentation] for details. Figure 2b The above step S121 can be achieved through the following steps:
[0098] S1211, Perform demosaic processing on the above second bayer image to obtain a demosaic-processed second bayer image;
[0099] S1212, The saturation of the second Bayer image after the demosaic processing is adjusted to obtain the second Bayer image after saturation adjustment;
[0100] S1213, adjust the brightness of the second bayer image after the above saturation adjustment to obtain the second bayer target image.
[0101] You can also adjust the brightness first, and then adjust the saturation. The specific operation can be performed according to the actual situation, and this application does not limit it.
[0102] For the first bayer image, if it is a high-resolution image, after acquiring it, demosaic processing is performed on the first bayer image to directly obtain the first bayer target image. If the first bayer image is not a high-resolution image, after acquiring it, demosaic processing is performed on it to obtain a de-mosaiced first bayer image. Then, saturation and brightness adjustments can be made to the de-mosaiced first bayer image to obtain the first bayer target image.
[0103] After obtaining the first and second Bayer target images, the first and second Bayer target images can be superimposed to obtain the third target image. Then, the third target image is downsampled to obtain the third Bayer image.
[0104] In this application, the second bayer image can be processed first to obtain the second bayer target image, and then the first bayer image can be demosaicized to obtain the first bayer target image. Alternatively, the first bayer image can be demosaicized first to obtain the first bayer target image, and then the second bayer image can be processed to obtain the second bayer target image. Or, the first bayer image and the second bayer image can be processed simultaneously to obtain both the first bayer target image and the second bayer target image. The specific order is not limited in this application.
[0105] Image data containing patterns of the type to be removed can be simulated through synthesis. In the absence of a high-definition, open-source dataset of the scene to be repaired containing patterns of the type to be removed, the corresponding dataset can be obtained through synthesis.
[0106] In one possible embodiment, the above-mentioned third Bayer image is used as input to the image restoration model, and the first RGB image is used as the ground truth of the image restoration output to train the image restoration model, thereby obtaining a trained image restoration model, including:
[0107] The third Bayer image is input into the image inpainting model for processing to obtain the predicted RGB image.
[0108] Calculate the difference between the predicted RGB image and the first RGB image. When the difference does not meet the preset requirements, adjust the parameters of the image restoration model based on the preset loss function.
[0109] When the preset termination condition is met, the trained image restoration model is obtained.
[0110] Using the third Bayer image as input to the image inpainting model, a predicted RGB image is first obtained. Then, the difference between the predicted RGB image and the first RGB image is calculated, and it is determined whether the difference meets the preset requirements. If the difference does not meet the preset requirements, the parameters of the image inpainting model are adjusted based on the preset loss function. When the preset termination condition is reached, the trained image inpainting model is obtained. The preset requirements can be set according to the actual situation and are not limited here.
[0111] The difference between the predicted RGB image and the first RGB image can be calculated using a preset similarity calculation method. A higher similarity score indicates a smaller difference between the two images. This preset similarity calculation method can employ cosine similarity, Euclidean algorithm, or Pearson correlation coefficient, among others. For example, features are extracted from both the predicted and first RGB images to obtain feature vectors for both images. Each feature vector includes at least one of the following: image resolution, image size, image color, signal-to-noise ratio, etc. The cosine value of the two feature vectors is calculated, and the similarity is determined based on this cosine value, thus determining the difference between the predicted and first RGB images.
[0112] The preset termination condition can refer to the conditions for the termination of neural network model training in existing technologies, and will not be elaborated here.
[0113] A well-trained image inpainting model can directly convert images from Bayer format to RGB format, avoiding the irreparable loss of details and content during YUV domain processing. Furthermore, when using RGB format images for recognition, the accuracy of image recognition can be improved.
[0114] In one possible embodiment, the image inpainting model includes an ISP algorithm module, a pattern extraction module, and a pattern removal module. The process of inputting the third Bayer image into the image inpainting model to obtain a predicted RGB image includes:
[0115] The third Bayer image is input into the ISP algorithm module for processing to obtain the first feature image.
[0116] The first feature image is input into the pattern extraction module to extract features, resulting in a pattern feature image and a scene feature image.
[0117] The pattern feature image and the scene feature image are input into the pattern removal module to remove the pattern and obtain the predicted RGB image.
[0118] The ISP algorithm module mainly implements at least one of the following functions: white balance, demosaic, color correction, noise reduction, gamma transformation, automatic exposure control, automatic gain control, etc. For details, please refer to the implementation of the ISP algorithm in the existing technology.
[0119] The pattern extraction module extracts features from the first feature image to obtain a pattern feature image and a scene feature image, thereby separating the pattern from the scene.
[0120] The pattern removal module can remove patterns to obtain a predicted RGB image.
[0121] In one possible embodiment, the image restoration model further includes an image post-processing module; the method further includes:
[0122] When the above differences meet the preset requirements, the predicted RGB image is input into the image post-processing module to obtain the target RGB image.
[0123] If the difference meets the preset requirements, the predicted RGB image can be input into the image post-processing module for processing to obtain the target RGB image. The image post-processing module can include traditional image algorithms that can be used to further adjust the predicted RGB image, including brightness adjustment, contrast adjustment, and detail adjustment.
[0124] based on Figure 1 In this embodiment, the present application provides a fourth flowchart illustrating the training method for an image restoration model, as shown below. Figure 2c As shown, step S130 above can be implemented through the following steps:
[0125] S131, Perform Bayer noise reduction on the first Bayer image to obtain the first processed image;
[0126] S132, The first processed image is processed based on the ISP algorithm to obtain the second processed image;
[0127] S133, Perform face enhancement processing on the second processed image to obtain the third processed image;
[0128] S134, adjust the brightness, saturation and contrast of the third processed image to obtain the first RGB image in RGB format.
[0129] For the first Bayer image, Bayer denoising can be performed first to obtain the first processed image. Then, based on the ISP algorithm, a series of ISP processing steps such as white balance, demosaic, color correction, denoising, and gamma transformation are performed on the first processed image to obtain the second processed image. Then, face enhancement processing is performed on the second processed image to obtain the third processed image. Finally, brightness, saturation, and contrast are adjusted on the third processed image to obtain the first RGB image in RGB format.
[0130] The order in which brightness, saturation, and contrast are adjusted for the third-processed image is not limited in this application.
[0131] Based on the above embodiments, this application provides a fifth flowchart illustrating the training method for an image restoration model, as shown below. Figure 3aAs shown, after acquiring the third Bayer image and the first RGB image, the method includes:
[0132] S301, Input the third Bayer image into the ISP algorithm module above for processing to obtain the first feature image;
[0133] S302, the first feature image is input into the pattern extraction module above for feature extraction to obtain a pattern feature image and a scene feature image;
[0134] S303, the above-mentioned pattern feature image and the above-mentioned scene feature image are input into the above-mentioned pattern removal module for pattern removal to obtain a predicted RGB image;
[0135] S304, calculate the difference between the predicted RGB image and the first RGB image;
[0136] S305, determine whether the difference meets the preset requirements. If yes, proceed to step S306; otherwise, return to step S302 to continue execution.
[0137] S306, the predicted RGB image is used as the input to the image post-processing module to obtain the target RGB image.
[0138] Image data including the patterns to be removed is simulated through synthesis. In the absence of a high-resolution, open-source dataset of the scene to be restored containing these patterns, a suitable dataset can be obtained through synthesis. A third Bayer image is used as input to the image restoration model, and the first RGB image is used as the ground truth output. The image restoration model is trained to obtain a trained model. This trained model can directly convert Bayer format images to RGB format images, avoiding the irreparable loss of detail and content during YUV domain processing. Furthermore, when using RGB format images for recognition, the accuracy of image recognition can be improved.
[0139] The image restoration model can be any of a CNN (Convolutional Neural Network) or a recurrent neural network, or a non-deep learning network method. For example, the image restoration model is a CNN network, which includes an ISP algorithm module, a pattern extraction module, and a pattern removal module. The ISP algorithm module, pattern extraction module, and pattern removal module are all convolutional layers. Based on the above embodiments, this application provides a sixth flowchart illustrating the training method of the image restoration model, as shown below. Figure 3b As shown, after acquiring the third Bayer image and the first RGB image, the method includes:
[0140] The third Bayer image is input into the ISP algorithm module for processing to obtain the first feature image.
[0141] The first feature image is input into the pattern extraction module to extract features, resulting in a pattern feature image and a scene feature image.
[0142] The above-mentioned pattern feature image and the above-mentioned scene feature image are input into the above-mentioned pattern removal module for pattern removal to obtain the predicted RGB image;
[0143] Calculate the difference between the predicted RGB image and the first RGB image;
[0144] Determine whether the difference meets the preset requirements. If yes, use the predicted RGB image as the input to the image post-processing module to obtain the target RGB image. If not, return to the step of inputting the third Bayer image into the ISP algorithm module for processing to obtain the first feature image and continue execution.
[0145] like Figure 4 As shown, Figure 4 This is a schematic flowchart of an image restoration method provided in an embodiment of this application. The method includes:
[0146] S410, acquire the image to be processed in Bayer format;
[0147] S420, the image to be processed is input into a pre-trained image restoration model for processing to obtain an RGB image in RGB format corresponding to the image to be processed, wherein the pre-trained image restoration model is trained based on any of the above-mentioned image restoration model training methods.
[0148] After obtaining the Bayer format image to be processed, it is input into a pre-trained image inpainting model for processing, resulting in an RGB format RGB image corresponding to the original image. This combines a deep learning network model with traditional image processing algorithms to jointly optimize image quality. Furthermore, because traditional image processing algorithms have many adjustable parameters, this approach offers greater flexibility.
[0149] It should be noted that the two-dimensional face images in this embodiment are from a publicly available dataset.
[0150] Based on the above method embodiments, this application provides a training device for an image restoration model, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of a training device for an image restoration model provided in an embodiment of this application. The device includes:
[0151] The acquisition module 510 is used to acquire a first Bayer image in Bayer format of the scene to be repaired, and a second Bayer image in Bayer format of the type of pattern to be removed.
[0152] The compositing module 520 is used to composit the first Bayer image and the second Bayer image to obtain a third Bayer image;
[0153] Preprocessing module 530 is used to preprocess the first bayer image to obtain a first RGB image in RGB format;
[0154] The training module 540 is used to train the image restoration model by taking the third Bayer image as input and the first RGB image as the ground truth of the image restoration output, so as to obtain the trained image restoration model.
[0155] In one possible embodiment, the synthesis module 520 includes:
[0156] The first processing submodule is used to perform demosaic processing, saturation adjustment, and brightness adjustment on the second bayer image to obtain the second bayer target image.
[0157] The second processing submodule is used to perform demosaic processing on the first bayer image to obtain the first bayer target image.
[0158] The overlay submodule is used to overlay the first Bayer target image and the second Bayer target image to obtain a third target image;
[0159] The third target image is downsampled to obtain the third Bayer image.
[0160] In one possible embodiment, the training module 540 is specifically used for:
[0161] The third Bayer image is input into the image inpainting model for processing to obtain the predicted RGB image.
[0162] Calculate the difference between the predicted RGB image and the first RGB image. When the difference does not meet the preset requirements, adjust the parameters of the image restoration model based on the preset loss function.
[0163] When the preset termination condition is met, the trained image restoration model is obtained.
[0164] In one possible embodiment, the image inpainting model includes an ISP algorithm module, a pattern extraction module, and a pattern removal module, wherein the training module 540 is specifically used for:
[0165] The third Bayer image is input into the ISP algorithm module for processing to obtain the first feature image.
[0166] The first feature image is input into the pattern extraction module to extract features, resulting in a pattern feature image and a scene feature image.
[0167] The pattern feature image and the scene feature image are input into the pattern removal module to remove the pattern and obtain the predicted RGB image.
[0168] In one possible embodiment, the image restoration model further includes an image post-processing module, and the device further includes a post-processing module. Specifically, the post-processing module is used to input the predicted RGB image into the image post-processing module when the difference meets the preset requirements, so as to obtain the target RGB image.
[0169] In one possible embodiment, the preprocessing module described above is specifically used for:
[0170] The first Bayer image is subjected to Bayer noise reduction to obtain the first processed image;
[0171] The first processed image is processed based on the ISP algorithm to obtain the second processed image;
[0172] The second processed image is subjected to face enhancement processing to obtain the third processed image;
[0173] The brightness, saturation, and contrast of the third processed image are adjusted to obtain the first RGB image in RGB format.
[0174] Based on the above method embodiments, this application provides an image restoration apparatus, such as... Figure 6 As shown, Figure 6 This is a schematic diagram of an image restoration apparatus provided in an embodiment of this application. The apparatus includes:
[0175] The acquisition module 610 is used to acquire the image to be processed in Bayer format;
[0176] The processing module 620 is used to input the image to be processed into a pre-trained image restoration model for processing, and obtain an RGB image in RGB format corresponding to the image to be processed, wherein the pre-trained image restoration model is trained by any of the above-mentioned image restoration models training devices.
[0177] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0178] This application also provides an electronic device, such as... Figure 7 As shown, it includes:
[0179] Memory 701 is used to store computer programs;
[0180] When processor 702 executes a program stored in memory 701, it performs the following steps:
[0181] Obtain a first Bayer image in Bayer format of the scene to be repaired, and a second Bayer image in Bayer format of the type of pattern to be removed;
[0182] The first Bayer image and the second Bayer image are combined to obtain the third Bayer image.
[0183] The first Bayer image is preprocessed to obtain a first RGB image in RGB format;
[0184] The third Bayer image is used as the input to the image restoration model, and the first RGB image is used as the ground truth of the image restoration output. The image restoration model is then trained to obtain a trained image restoration model.
[0185] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 702, the communication interface, and the memory 701 communicating with each other via the communication bus.
[0186] This application also provides an electronic device, including:
[0187] Memory, used to store computer programs;
[0188] When a processor executes a program stored in memory, it performs the following steps:
[0189] Obtain the image to be processed in Bayer format;
[0190] The image to be processed is input into a pre-trained image restoration model for processing to obtain an RGB image in RGB format corresponding to the image to be processed. The pre-trained image restoration model is trained using any of the above-mentioned image restoration model training methods.
[0191] Furthermore, the aforementioned electronic devices may also include a communication bus and / or a communication interface, with the processor, communication interface, and memory communicating with each other via the communication bus.
[0192] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0193] The communication interface is used for communication between the aforementioned electronic devices and other devices.
[0194] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0195] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0196] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the training method for any of the above-described image restoration models.
[0197] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image restoration methods.
[0198] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the training method of any of the image restoration models in the above embodiments.
[0199] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the image restoration methods described above.
[0200] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).
[0201] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0202] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0203] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.
Claims
1. A training method for an image restoration model, characterized in that, The method includes: Acquire a first Bayer image in Bayer format of the scene to be repaired, and a second Bayer image in Bayer format of the type of pattern to be removed; the type of pattern to be removed includes rainbow patterns and light spots, the light spots being light spots formed after light is reflected, the scene to be repaired is a traffic capture scene or an access control scene, and the first Bayer image is the Bayer image before preprocessing. The first Bayer image and the second Bayer image are combined to obtain the third Bayer image; The first Bayer image is preprocessed to obtain a first RGB image in RGB format; The image restoration model includes an ISP algorithm module, a pattern extraction module, and a pattern removal module; The third Bayer image is input into the ISP algorithm module for processing to obtain the first feature image. The ISP algorithm module is used to implement at least one of the following functions: white balance, demosaic, color correction, noise reduction, gamma transformation, automatic exposure control, and automatic gain control. The first feature image is input into the pattern extraction module for feature extraction to obtain a pattern feature image and a scene feature image; The pattern feature image and the scene feature image are input into the pattern removal module for pattern removal to obtain a predicted RGB image; Calculate the difference between the predicted RGB image and the first RGB image. When the difference does not meet the preset requirements, adjust the parameters of the image restoration model based on the preset loss function. When the preset termination condition is met, the trained image restoration model is obtained.
2. The method according to claim 1, characterized in that, The step of compositing the first Bayer image and the second Bayer image to obtain the third Bayer image includes: The second Bayer image is subjected to demosaic processing, saturation adjustment, and brightness adjustment to obtain the second Bayer target image. The first Bayer image is demosaicized to obtain the first Bayer target image; The first Bayer target image and the second Bayer target image are superimposed to obtain the third target image; The third target image is downsampled to obtain the third Bayer image.
3. The method according to claim 1, characterized in that, The image restoration model further includes an image post-processing module; the method further includes: When the difference meets the preset requirements, the predicted RGB image is input into the image post-processing module to obtain the target RGB image.
4. The method according to claim 1, characterized in that, The preprocessing of the first Bayer image to obtain a first RGB image in RGB format includes: Perform Bayer noise reduction on the first Bayer image to obtain the first processed image; The first processed image is processed based on the ISP algorithm to obtain the second processed image; The second processed image is subjected to face enhancement processing to obtain the third processed image; The brightness, saturation, and contrast of the third processed image are adjusted to obtain a first RGB image in RGB format.
5. An image restoration method, characterized in that, The method includes: Obtain the image to be processed in Bayer format; The image to be processed is input into a pre-trained image restoration model for processing to obtain an RGB image in RGB format corresponding to the image to be processed. The pre-trained image restoration model is trained based on the method described in any one of claims 1 to 4.
6. A training device for an image restoration model, characterized in that, The device includes: The acquisition module is used to acquire a first Bayer image in Bayer format of the scene to be repaired, and a second Bayer image in Bayer format of the type of pattern to be removed; the type of pattern to be removed includes rainbow patterns and light spots, the light spots being light spots formed after light is reflected, the scene to be repaired is a traffic capture scene or an access control scene, and the first Bayer image is the Bayer image before preprocessing. The compositing module is used to compose the first Bayer image and the second Bayer image to obtain a third Bayer image; The preprocessing module is used to preprocess the first bayer image to obtain a first RGB image in RGB format; The image restoration model includes an ISP algorithm module, a pattern extraction module, and a pattern removal module; a training module is used to input the third Bayer image into the ISP algorithm module for processing to obtain a first feature image. The ISP algorithm module is used to implement at least one of the following functions: white balance, de-mosaic, color correction, noise reduction, gamma transformation, automatic exposure control, and automatic gain control; the first feature image is input into the pattern extraction module for feature extraction to obtain a pattern feature image and a scene feature image; the pattern feature image and the scene feature image are input into the pattern removal module for pattern removal to obtain a predicted RGB image; the difference between the predicted RGB image and the first RGB image is calculated, and when the difference does not meet the preset requirements, the parameters of the image restoration model are adjusted based on a preset loss function; when the preset termination condition is reached, the trained image restoration model is obtained.
7. An image restoration device, characterized in that, The device includes: The acquisition module is used to acquire images in Bayer format to be processed. The processing module is used to input the image to be processed into a pre-trained image restoration model for processing, so as to obtain an RGB image in RGB format corresponding to the image to be processed, wherein the pre-trained image restoration model is trained based on the device described in claim 6.
8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-5.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.