Method and apparatus for generating an image restoration model, image restoration method and apparatus

Abstract

Embodiments of the present disclosure disclose a method, apparatus, computer-readable storage medium, and electronic device for generating an image restoration model. Among them, the method includes: using a data preprocessing network included in the model to be trained to preprocess the sample original image data to obtain a sample intermediate image; using an image restoration network included in the model to be trained to restore the sample intermediate image to obtain a restored image; determining a loss value representing the error between the restored image and a preset reference image based on a preset loss function; adjusting the parameters of the model to be trained based on the loss value; and in response to the model to be trained after parameter adjustment meeting the preset training end condition, determining the model to be trained after parameter adjustment as the image restoration model. Embodiments of the present disclosure do not need to use numerous modules included in the current ISP algorithm to process the original image data, reducing the complexity of data processing and improving the efficiency of image restoration.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method and device for generating an image restoration model, an image restoration method and device, a computer-readable storage medium, and an electronic device. Background Art

[0002] In the entire process of camera imaging, ISP (Image Signal Process) is responsible for receiving the original signal data of the photosensitive sensor (Sensor). It is the first processing process in the entire camera shooting and is used to process the image signals output by the image signal sensor. The main function of ISP is to perform post-processing on the signals output by the front-end image sensor. The main functions include linear correction, noise removal, dead pixel removal, interpolation, white balance, automatic exposure control, etc.

[0003] ISP is usually composed of a series of processing modules. These processing modules reconstruct the original image data (RAW) collected by the sensor into a color image according to hyperparameters. The reconstruction effect of ISP is greatly affected by hyperparameters. Currently, there are many ISP algorithms based on CNN (Convolutional Neural Network) that are used to restore or achieve a reconstruction effect better than traditional ISP. And traditional ISP optimizes images for the human eye effect. For the field of image recognition, it is unknown which modules in traditional ISP algorithms are truly effective.

[0004] The existing ISP algorithms based on CNN mainly include the following directions: methods for end-to-end implementation of the entire ISP streamline, module-based methods, and scene-based methods. Among them, the method for end-to-end implementation of the entire ISP streamline based on CNN is to construct a large-capacity network structure and use a higher-quality picture as supervision to learn all the functions involved in ISP, so that it has achieved a great improvement in the restoration effect, even exceeding the traditional ISP algorithm itself. In addition, the module-based method is to use a CNN network to implement the functions of specific modules in ISP. Finally, the scene-based method is to use CNN to implement the ISP processing functions in specific scenarios.

[0005] The above three ISP algorithms based on CNN all have the problem of large computational complexity. Summary of the Invention

[0006] Embodiments of the present disclosure provide a method and device for generating an image restoration model, a computer-readable storage medium, and an electronic device.

[0007] Embodiments of the present disclosure provide a method for generating an image restoration model. The method includes: preprocessing sample original image data by using a data preprocessing network included in a model to be trained to obtain sample intermediate images; restoring the sample intermediate images by using an image restoration network included in the model to be trained to obtain restored images; determining a loss value representing the error between the restored images and preset reference images based on a preset loss function; adjusting the parameters of the model to be trained based on the loss value; and determining the model with adjusted parameters as the image restoration model in response to the model with adjusted parameters meeting a preset training end condition.

[0008] According to another aspect of the embodiments of the present disclosure, there is provided an image restoration method. The method includes: acquiring original image data collected by an image sensor; preprocessing the original image data by using a data preprocessing network included in a pre-trained image restoration model to obtain intermediate images; and restoring the intermediate images by using an image restoration network included in the pre-trained image restoration model to obtain restored images.

[0009] According to another aspect of the embodiments of the present disclosure, there is provided an apparatus for generating an image restoration model. The apparatus includes: a first preprocessing module configured to preprocess sample original image data by using a data preprocessing network included in a model to be trained to obtain sample intermediate images; a first restoration module configured to restore the sample intermediate images by using an image restoration network included in the model to be trained to obtain restored images; a first determination module configured to determine a loss value representing the error between the restored images and preset reference images based on a preset loss function; an adjustment module configured to adjust the parameters of the model to be trained based on the loss value; and a second determination module configured to determine the model with adjusted parameters as the image restoration model in response to the model with adjusted parameters meeting a preset training end condition.

[0010] According to another aspect of the embodiments of the present disclosure, there is provided an image restoration apparatus. The apparatus includes: an acquisition module configured to acquire original image data collected by an image sensor; a second preprocessing module configured to preprocess the original image data by using a data preprocessing network included in a pre-trained image restoration model to obtain intermediate images; and a second restoration module configured to restore the intermediate images by using an image restoration network included in the pre-trained image restoration model to obtain restored images.

[0011] According to another aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium storing a computer program for executing the above-mentioned method for generating an image restoration model or the image restoration method.

[0012] According to another aspect of the embodiments of the present disclosure, an electronic device is provided. The electronic device includes: a processor; a memory for storing executable instructions executable by the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement the above-mentioned method for generating an image restoration model or the image restoration method.

[0013] Based on the method and device for generating an image restoration model, the method and device for image restoration, the computer-readable storage medium, and the electronic device provided in the above embodiments of the present disclosure, by using the data preprocessing network included in the model to be trained to preprocess the sample original image data to obtain sample intermediate images, then using the image restoration network included in the model to be trained to restore the sample intermediate images to obtain restored images, and finally training the above-mentioned model to be trained based on the preset reference images and the restored images to obtain an image restoration model. The trained image restoration model realizes the function of ISP based on a neural network, can specifically use the preprocessing network to preprocess the original image data, and then use the image restoration network to implement image restoration, without using many modules included in the current ISP algorithm to process the original image data, reducing the complexity of data processing and improving the efficiency of image restoration. The restored images can be applied in the field of image recognition. Without using many ISP modules, the original image data can be restored to the image quality required in the field of image recognition, which helps to improve the efficiency of image recognition.

[0014] The technical solutions of the present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Description of the Drawings

[0015] By describing the embodiments of the present disclosure in more detail with reference to the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become more obvious. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation to the present disclosure. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 It is a system diagram applicable to the present disclosure.

[0017] Figure 2 It is a schematic flowchart of a method for generating an image restoration model provided by an exemplary embodiment of the present disclosure.

[0018] Figure 3 It is an exemplary structural diagram of an image restoration model provided by an embodiment of the present disclosure.

[0019] Figure 4 It is a schematic flowchart of a method for generating an image restoration model provided by another exemplary embodiment of the present disclosure.

[0020] Figure 5 It is a schematic flowchart of a method for generating an image restoration model provided by another exemplary embodiment of the present disclosure.

[0021] Figure 6 It is a schematic flowchart of a method for generating an image restoration model provided by another exemplary embodiment of the present disclosure.

[0022] Figure 7 It is a schematic flowchart of a method for generating an image restoration model provided by another exemplary embodiment of the present disclosure.

[0023] Figure 8 It is a schematic flowchart of a method for generating an image restoration model provided by another exemplary embodiment of the present disclosure.

[0024] Figure 9 It is a schematic flowchart of an image restoration method provided by an exemplary embodiment of the present disclosure.

[0025] Figure 10 It is a schematic structural diagram of a device for generating an image restoration model provided by an exemplary embodiment of the present disclosure.

[0026] Figure 11 It is a schematic structural diagram of a device for generating an image restoration model provided by another exemplary embodiment of the present disclosure.

[0027] Figure 12 It is a schematic structural diagram of an image restoration device provided by an exemplary embodiment of the present disclosure.

[0028] Figure 13 It is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. Detailed implementation manners

[0029] Next, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. It should be understood that the present disclosure is not limited by the exemplary embodiments described herein.

[0030] It should be noted that: Unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and values set forth in these embodiments do not limit the scope of the present disclosure.

[0031] Those skilled in the art can understand that terms such as "first", "second", etc. in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and do not represent any specific technical meaning nor indicate an inevitable logical order between them.

[0032] It should also be understood that in the embodiments of the present disclosure, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0033] It should also be understood that for any component, data, or structure mentioned in the embodiments of the present disclosure, in the absence of a clear limitation or a contrary indication in the context, it can generally be understood as one or more.

[0034] In addition, the term "and / or" in the present disclosure is merely a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0035] It should also be understood that the description of each embodiment in the present disclosure emphasizes the differences between the embodiments, and their similarities can be referred to each other. For the sake of brevity, they will not be elaborated one by one.

[0036] At the same time, it should be understood that for the convenience of description, the dimensions of the various parts shown in the drawings are not drawn in actual proportional relationships.

[0037] The following description of at least one exemplary embodiment is actually merely illustrative and in no way a limitation on the present disclosure and its application or use.

[0038] Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods, and devices should be regarded as part of the specification.

[0039] It should be noted that like reference numerals and letters denote like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

[0040] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, and so on.

[0041] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, target programs, components, logics, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, where tasks are executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0042] Overview of the Application

[0043] In related technologies, the method of implementing the entire ISP streamline based on CNN end-to-end aims to achieve the effect of the entire ISP streamline. It does not consider which modules are truly effective for the field of image recognition (such as object detection tasks), resulting in a huge computational cost.

[0044] In related technologies, the module-based method directly uses a CNN network to implement the functions of specific modules in the ISP, aiming to achieve a better effect compared to traditional ISPs. Similarly, it does not determine which modules are key modules for the field of image recognition. The computational cost is relatively reduced compared to implementing the entire ISP streamline, but it still consumes a large amount of computing power.

[0045] In related technologies, the scenario-based method uses CNN to implement the ISP processing functions in specific scenarios, such as converting a low-light scene to a bright-light scene. There are also problems of not exploring which ISP modules are strongly related to the field of image recognition and consuming a large amount of computational power.

[0046] Exemplary System

[0047] Figure 1 There is shown a method for generating an image restoration model or a device for generating an image restoration model to which embodiments of the present disclosure can be applied, as well as an exemplary system architecture 100 of an image restoration method or an image restoration device.

[0048] As Figure 1 shown, the system architecture 100 can include a terminal device 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0049] A user can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as an image processing application, a video surveillance application, a web browser application, etc.

[0050] The terminal device 101 can be various electronic devices, including but not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0051] The server 103 can be a server that provides various services, such as a model training server that uses the sample raw image data uploaded by the terminal device 101 for model training. The model training server can use the received sample raw image data to train an image restoration model. The server 103 can also be a server that performs image restoration operations on the raw image data uploaded by the terminal device 101. This server can set the trained image restoration model and use the image restoration model to restore the raw image data.

[0052] It should be noted that the method for generating the image restoration model or the image restoration method provided by the embodiments of the present disclosure can be executed by the server 103 or by the terminal device 101. Correspondingly, the device for generating the image restoration model or the image restoration device can be set in the server 103 or in the terminal device 101.

[0053] It should be understood that Figure 1 the numbers of the terminal devices, networks, and servers in

[0054] Exemplary Method

[0055] Figure 2 is a schematic flowchart of the method for generating an image restoration model provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to an electronic device (such as Figure 1 the terminal device 101 or the server 103 shown), as Figure 2 shown, and the method includes the following steps:

[0056] Step 201, preprocess the sample raw image data using the data preprocessing network included in the model to be trained to obtain a sample intermediate image.

[0057] In this embodiment, the electronic device may use the data preprocessing network included in the model to be trained to preprocess the original sample image data to obtain intermediate sample images. The model to be trained may be a pre-set machine learning model that has not been trained or an incompletely trained machine learning model. As Figure 3 shown, the model 301 to be trained includes a data preprocessing network 3011 and an image restoration network 3012. The data preprocessing network is used to preprocess the original sample image data to obtain an image with multiple color channels as the intermediate sample image.

[0058] Generally, the original sample image data may be data collected by a photosensitive sensor (such as a CMOS sensor) of a camera. As Figure 3 shown, the original sample image data 302 is single-channel data, and the RGB color values included therein are distributed within the single channel. The data preprocessing network 3011 may convert the original sample image data 302 into an RGB three-channel intermediate sample image 303.

[0059] Step 202: Use the image restoration network included in the model to be trained to restore the intermediate sample image to obtain a restored image.

[0060] In this embodiment, the electronic device may use the image restoration network included in the model to be trained to restore the intermediate sample image to obtain a restored image. The image restoration network may include a deep neural network. The image restoration network may extract feature images (such as texture features, line shape features, etc.) from the intermediate sample image and adjust the color values of the RGB three channels according to the image features to obtain a restored image.

[0061] As Figure 3 shown, the intermediate sample image 303 is restored through the image restoration network 3012 to obtain a restored image 304.

[0062] Step 203: Based on a preset loss function, determine a loss value representing the error between the restored image and a preset reference image.

[0063] In this embodiment, the electronic device may determine a loss value representing the error between the restored image and a preset reference image based on a preset loss function. The reference image is the reference image for image restoration, and the goal of image restoration is to minimize the error between the restored image and the reference image.

[0064] The loss function can be of various types, such as the existing L1 loss function, L2 loss function, etc. Through the loss function, a loss value can be calculated, and this loss value represents the error between the color values of each channel included in the restored image and the color values of each channel included in the reference image.

[0065] Step 204: Based on the loss value, adjust the parameters of the model to be trained.

[0066] In this embodiment, the electronic device can adjust the parameters of the model to be trained based on the loss value.

[0067] The training process of the model is a process of solving the optimal solution. Among them, the optimal solution is given by means of data annotation, which in this embodiment is the color value of each channel included in the reference image. The process of the model fitting to the optimal solution is mainly carried out iteratively by minimizing the error. For a set of input sample original image data, a loss function is set. This loss function can calculate the gap between the actual output of the model (i.e., the restored image) and the expected output (i.e., the reference image), and transmit this gap to the connections between each neuron in the neural network through the backpropagation algorithm. The difference signal transmitted to each connection represents the contribution rate of this connection to the overall error. Then, the gradient descent algorithm can be used to update and modify the original model parameters.

[0068] It should be noted that the parameters of the above data preprocessing network are set in advance. Therefore, during the model training process, only the parameters of the image restoration network need to be adjusted.

[0069] By repeatedly executing Step 201 - Step 204, that is, using multiple sets of training samples (including sample original image data and corresponding reference images) to iteratively train the model, the model after each iterative training is the model to be trained for the next training.

[0070] Step 205: In response to the model to be trained after adjusting the parameters meeting the preset training end condition, determine the model to be trained after adjusting the parameters as the image restoration model.

[0071] In this embodiment, the electronic device can, in response to the model to be trained after adjusting the parameters meeting the preset training end condition, determine the model to be trained after adjusting the parameters as the image restoration model.

[0072] Specifically, repeatedly execute Step 201 - Step 204 using multiple sets of training samples to iteratively train the model, and determine whether the current model meets the training end condition after each training. When the training end condition is met, the current model after adjusting the parameters is the trained image restoration model. Among them, the training end condition can include but is not limited to at least one of the following: the loss value of the above loss function converges, the training time exceeds the preset duration, and the number of training times exceeds the preset number.

[0073] The method provided by the above embodiments of the present disclosure preprocesses the original sample image data by using the data preprocessing network included in the model to be trained to obtain an intermediate sample image, then restores the intermediate sample image by using the image restoration network included in the model to be trained to obtain a restored image, and finally trains the model to be trained based on a preset reference image and the restored image to obtain an image restoration model. The trained image restoration model realizes that in the ISP algorithm based on a neural network, the preprocessing network is specifically used to preprocess the original image data, and then the image restoration network is used to restore the image, without using many modules included in the current ISP algorithm to process the original image data, reducing the complexity of data processing and improving the efficiency of image restoration. The restored image can be applied in the field of image recognition, and the original image data can be restored to the image quality required in the field of image recognition without using many ISP modules, which helps to improve the efficiency of image recognition.

[0074] In some alternative implementation manners, as Figure 4 shown, before step 201, the method may further include:

[0075] Step 401, select at least one target functional module from a set of functional modules that implement a complete image signal processing flow.

[0076] Among them, the set of functional modules for a complete image signal processing (ISP) flow is the set of a large number of functional modules that implement the traditional ISP method. For example, it may include the original image data conversion sub-network 30111 and the brightness correction sub-network 30121 as Figure 3 shown, and may also include a white balance module, a denoising module, a color correction module, a color space conversion module, etc.

[0077] Generally, at least one functional module combination may be preset, and each functional module combination includes at least one functional module. Therefore, the electronic device may sequentially select a functional module combination from the above at least one functional module combination, and use the functional modules included in the currently selected functional module combination as the above at least one target functional module.

[0078] Step 402, perform a performance test on the at least one target functional module based on a preset test scenario to obtain a test result.

[0079] Among them, the above test scenario can be set according to the actual image processing task. For example, for the object detection task, at least one target function module can be used to process the sample image to obtain an intermediate result image, and then the intermediate result image is used to train the object detection model to obtain the object detection model to be tested. Then, the performance of the object detection model to be tested is tested using the test data set to obtain the test result. For example, the test result can be the detection accuracy of the object detection model to be tested as described above.

[0080] Step 403, in response to determining that the test result meets the preset condition, construct a model to be trained based on at least one target function module.

[0081] Among them, the preset condition is used to indicate whether the test result reaches the performance index in the above test scenario. For example, the benchmark test result of the benchmark object detection model trained with the intermediate result image obtained by using the complete ISP process (that is, the above function module set) can be compared with the above test result. If the error between the two is less than or equal to the preset error threshold, it is determined that the preset condition is met.

[0082] As an example, at least one target function module may include a raw image data conversion module and a brightness correction module. Based on this, a model to be trained including a raw image data conversion sub-network 30111 and a brightness correction sub-network 30121 as shown in Figure 3 can be constructed. It should be noted that Figure 3 the scaling sub-network 30112 shown in is set according to the actual application scenario of the model to be trained. For example, when the model to be trained is applied to the object detection task, the scaling sub-network can make the size of the restored image meet the size requirements of the input image of the object detection model.

[0083] In this embodiment, by selecting at least one target function module from the function module set that implements the complete ISP process and constructing a model to be trained based on at least one target function module, compared with the related art that requires the complete ISP process for image restoration, this embodiment only needs to execute a part of the functions of the complete ISP process to achieve high-quality image restoration, thereby simplifying the processing steps of image restoration and improving the efficiency of image restoration.

[0084] In some alternative implementation manners, the above at least one target function module includes a raw image data conversion module. The raw image data conversion module is used to separate various color values (such as RGB values) included in the raw image data collected by the camera's photosensitive sensor and generate a multi-channel image based on the separated color values. Based on this, the above step 403 may include:

[0085] Based on the raw image data conversion module, generate a raw image data conversion sub-network included in the data preprocessing network.

[0086] Specifically, the original image data conversion module can be set in the data preprocessing network of the model to be trained, and the original image data conversion module can be used as the original image data conversion sub-network included in the model to be trained.

[0087] As Figure 5 shown, step 201 may include:

[0088] Step 2011: Use the original image data conversion sub-network to convert the sample original image data into a multi-channel original image.

[0089] Specifically, the sample original image data is the original image data collected by the photosensitive sensor of the camera, and the RGB color values included therein are mixed in one channel according to a certain rule. In this step, the GRB color values in one channel can be separated into three channels of R, G, and B. Since the size of each channel after separation is the same as the size of the original image data before separation, interpolation processing can be performed on each channel to fill the missing data in each channel.

[0090] Optionally, an existing Demosaic algorithm can be used to perform interpolation on the three channels of R, G, and B to obtain a multi-channel original image. As Figure 3 shown, the data preprocessing network may include an original image data conversion sub-network 30111, and the original image data conversion sub-network 30111 can implement the Demosaic algorithm to perform channel separation and interpolation on the sample original image data 302 to obtain a multi-channel original image 305.

[0091] Step 2012: Generate a sample intermediate image based on the multi-channel original image.

[0092] Optionally, the multi-channel original image generated after interpolation can be determined as the sample intermediate image.

[0093] In this embodiment, the sample original image data is converted into a multi-channel original image through the original image data conversion sub-network, and a sample intermediate image is generated based on the multi-channel original image, providing a high-quality sample intermediate image for subsequent image restoration, and eliminating the need to use many existing ISP modules for complex data processing, improving the efficiency of image restoration.

[0094] In some alternative implementation manners, the data preprocessing network further includes a scaling sub-network, where the scaling sub-network is used to adjust the original size of the multi-channel original image to a target size.

[0095] Step 2012 can be executed as follows:

[0096] Using a scaling sub-network, the original size of the multi-channel original image is adjusted to the target size to obtain a sample intermediate image.

[0097] Among them, the target size can be a size set based on the scenario of further processing the restored image subsequently. For example, for an object detection task, the size of the restored image can be less than or equal to a preset size. Therefore, the size of the multi-channel original image can be reduced to adapt to the object detection task.

[0098] As Figure 3 shown, the data preprocessing network may include a scaling sub-network 30112 to adjust the size of the multi-channel original image 305 to obtain a sample intermediate image 303. In this embodiment, by adjusting the size of the multi-channel original image, the size of the restored image output by the trained image restoration model can be adapted to the scenario where it is applied. At the same time, by reducing the size of the multi-channel original image, the computational complexity of the generated image restoration model can be reduced, thereby improving the overall processing speed of the model to achieve the technical effect of more efficient image restoration for detection and perception tasks.

[0099] In some optional implementation manners, the above at least one target functional module includes a brightness correction module. Among them, the brightness correction module is used to perform brightness correction on the image according to the set correction parameters.

[0100] Based on this, the above step 403 may include:

[0101] Based on the brightness correction module, a brightness correction sub-network included in the image restoration network is generated.

[0102] Specifically, the brightness correction module can be set in the image restoration network of the model to be trained, and this brightness correction module is used as the brightness correction sub-network included in the model to be trained.

[0103] As Figure 6 shown, step 202 includes:

[0104] Step 2021, using the brightness correction sub-network included in the image restoration network to perform brightness correction on the sample intermediate image to obtain a corrected image.

[0105] Specifically, the response of the camera's photosensitive sensor to light is different from that of the human eye. The human eye is more sensitive to dark details than the photosensitive sensor. The function of the brightness correction sub-network is to make the corrected image conform to the characteristics of the human eye when displayed. As Figure 3 shown, the brightness correction sub-network 30121 included in the image restoration network 3012 performs brightness correction on the sample intermediate image 303 to obtain a corrected image 306.

[0106] Optionally, the brightness correction sub-network can implement the current Gamma correction method. Gamma correction is to edit the Gamma curve of an image, detect the dark and light parts in the image signal, and increase the ratio between the two, thereby improving the image contrast effect and adding more dark color levels for non-linear tone editing of the image.

[0107] The Gamma curve is a special tone curve, and the Gamma curve is represented by the Gamma function shown in the following formula (1):

[0108] y = x 1 / γ (1)

[0109] Where, x represents the image brightness before correction, and y represents the image brightness after correction. When the γ value is equal to 1, the curve is a straight line at 45° to the coordinate axes, indicating that the input and output brightness are the same at this time. A γ value higher than 1 will cause the output to be brightened, and a γ value lower than 1 will cause the output to be darkened. The process of model training can adaptively determine a γ value, and the trained model performs brightness correction on the image according to this γ value during image restoration.

[0110] Step 2022, use the image restoration sub-network included in the image restoration network to restore the corrected image to obtain the restored image.

[0111] The image restoration sub-network can be composed of a deep neural network. The image restoration sub-network can extract the feature images from the corrected image and adjust the color values of the RGB three channels according to the image features to obtain the restored image. As Figure 3 shown, the image restoration sub-network 30122 included in the image restoration network 3012 restores the corrected image 306 to obtain the restored image 304. In this embodiment, by setting the brightness correction sub-network, the brightness correction of the sample intermediate image can be realized, and the restored image that meets image recognition can be obtained by using the image restored after brightness correction. Compared with the existing ISP algorithm, a high-quality restored image can be obtained without using complex operations of ISP modules such as white balance, denoising, color correction, and color space conversion, thereby further reducing the data processing complexity of image restoration.

[0112] In some alternative implementation manners, as Figure 7 shown, step 2022 includes the following sub-steps:

[0113] Step 20221, use the first sub-network included in the image restoration sub-network to restore the target channel included in the corrected image to obtain the first restored image.

[0114] Among them, the target channel is one of the multiple channels included in the corrected image. The target channel is usually the type of color value with the largest proportion in the original sample image data. Usually, in the original sample image data, the proportion of data belonging to the G channel is the largest (usually 50%), so the G channel can be the target channel. The image restoration sub-network includes a first sub-network and a second sub-network. First, the first sub-network restores the G channel, and then the second sub-network restores the other channels.

[0115] As Figure 3 shown, the corrected image 306 passes through the first sub-network included in the image restoration sub-network 30122 to restore the G-channel data, and the first restored image 307 including the restored G channel is obtained. It should be noted that the first restored image also includes R-channel data and B-channel data. Since the R channel and B channel have also undergone convolutional operations, compared with the R-channel data and B-channel data of the corrected image, changes have occurred, but no targeted restoration processing has been performed.

[0116] Step 20222: Use the second sub-network included in the image restoration sub-network to restore the first restored image to obtain a second restored image.

[0117] Since the amount of data in the target channel is larger than that in other channels, the target channel contains more image features. The restored target channel can provide more information when restoring other channels. For example, the restored target channel can improve the accuracy of restoring other channels due to more texture details.

[0118] As Figure 3 shown, the first restored image 307 passes through the second sub-network included in the image restoration sub-network 30122 to restore the RGB-channel data of the first restored image 307, and the obtained second restored image is the above-mentioned restored image 304.

[0119] Optionally, the above first sub-network and second sub-network can adopt existing lightweight network structures. The quantization network structure usually includes a shallow network, which can reduce the complexity of the network structure and improve the efficiency of image restoration processing. As an example, the above first sub-network and second sub-network can adopt a combination of existing depthwise convolution (DepthwiseConv) and pointwise convolution (PointwiseConv).

[0120] Step 20223: Based on the second restored image, obtain the restored image.

[0121] Optionally, the second restored image can be used as the above-mentioned restored image. Or, the second restored image can be further processed (such as noise reduction, etc.) to obtain the above-mentioned restored image.

[0122] In this embodiment, by first restoring the target channel included in the corrected image and then restoring other channels based on the obtained first restored image, the data of the target channel containing more image features can be fully utilized, making the finally obtained restored image closer to the real scene.

[0123] In some alternative implementation manners, the preset loss function includes a first loss function and a second loss function. Among them, the types of the first loss function and the second loss function can be set as needed. For example, the first loss function and the second loss function can be an L1 loss function, an L2 loss function, etc.

[0124] As Figure 8 shown, step 203 includes:

[0125] Step 2031, based on the first loss function, determine a first loss value representing the error between the target channel of the first restored image and the target channel of the reference image.

[0126] If the target channel is the G channel, the type of the first loss function can be an L1 loss function, and the first loss function can be expressed as:

[0127]

[0128] where n is the number of pixels included in the target channel of the first restored image, which is equal to the number of pixels included in the reference image, y i represents the color value of the i-th pixel of the target channel of the reference image, represents the color value of the i-th pixel of the target channel of the first restored image.

[0129] The first loss value obtained through calculation can represent the error between the G channel of the first restored image and the G channel of the reference image.

[0130] As Figure 3 shown, the G channel data 3071 of the first restored image 307 is separately extracted, and based on the above first loss function, the first loss value LossG is calculated.

[0131] Step 2032, based on the second loss function, determine a second loss value representing the error between the second restored image and the reference image.

[0132] The calculation process of the second loss value utilizes the color values of all channels of the second restored image and the reference image. Therefore, the second loss value calculated by the second loss function can be denoted as LossRGB. Similarly, by using the above formula (2), the errors between the R channel, G channel, and B channel of the second restored image and the reference image can be calculated respectively, and then the average value of the errors of the three channels is calculated as the second loss value.

[0133] As Figure 3 shown, according to the second loss function, the loss value is calculated for the RGB channel data of the second restored image (i.e., the above restored image 304) and the RGB channel data of the reference image, and the second loss value LossRGB is obtained.

[0134] Step 2033: Based on the first loss value and the second loss value, determine the loss value representing the error between the restored image and the reference image.

[0135] Optionally, the first loss value and the second loss value can be added together to obtain the loss value representing the error between the restored image and the reference image. Alternatively, based on a preset weight, the first loss value and the second loss value can be weighted and summed to obtain the loss value representing the error between the restored image and the reference image.

[0136] In this embodiment, by setting the first loss function and the second loss function, the parameters of the first restoration sub-network and the second restoration sub-network can be adjusted during the model training process. When the loss value converges, it indicates that the data restoration ability of the first restoration sub-network for the target channel reaches the best, and at the same time, the data restoration ability of the second restoration sub-network for other channels based on the restored target channel reaches the best, thereby helping to improve the accuracy of image restoration.

[0137] Figure 9 is a schematic flowchart of an image restoration method provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to an electronic device (such as Figure 1 the terminal device 101 or the server 103 shown), as Figure 9 shown, the method includes the following steps:

[0138] Step 901: Obtain the original image data collected by the image sensor.

[0139] In this embodiment, the electronic device can obtain the original image data collected by the image sensor locally or remotely. Among them, the image sensor can be the photosensitive sensor in the camera included in the above electronic device, or the photosensitive sensor in the camera connected to the above electronic device.

[0140] Step 902: Use the data preprocessing network included in the pre-trained image restoration model to preprocess the original image data to obtain an intermediate image.

[0141] In this embodiment, the electronic device can use the data preprocessing network included in the pre-trained image restoration model to preprocess the original image data to obtain an intermediate image.

[0142] Among them, the image restoration model is a model trained according to the above Figure 2 corresponding embodiment. The data preprocessing network and the image restoration network included in the image restoration model can refer to the above Figure 2 corresponding embodiment and will not be elaborated here.

[0143] Step 903: Use the image restoration network included in the pre-trained image restoration model to restore the intermediate image to obtain a restored image.

[0144] In this embodiment, the electronic device can use the image restoration network included in the pre-trained image restoration model to restore the intermediate image to obtain a restored image. Generally, the obtained restored image can be used for image recognition. For example, object detection can be performed on the restored image to determine the position and category of the target object (such as a human body, an animal, a vehicle, etc.) in the restored image.

[0145] The above Figure 9 method provided by the corresponding embodiment realizes the function of ISP based on a neural network by using a pre-trained image restoration model to perform image restoration on the original image data, without using numerous modules included in the current ISP algorithm to process the original image data, reducing the complexity of data processing and improving the efficiency of image restoration. The restored image can be applied in the field of image recognition, and the original image data can be restored to the image quality required in the field of image recognition without using numerous ISP modules, which helps to improve the efficiency of image recognition.

[0146] Exemplary Device

[0147] Figure 10 is a schematic structural diagram of a generating device of an image restoration model provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to an electronic device, such as Figure 10As shown in the figure, the generating device of the image restoration model includes: a first preprocessing module 1001, configured to preprocess the sample original image data by using the data preprocessing network included in the model to be trained, so as to obtain a sample intermediate image; a first restoration module 1002, configured to restore the sample intermediate image by using the image restoration network included in the model to be trained, so as to obtain a restored image; a first determination module 1003, configured to determine a loss value representing the error between the restored image and a preset reference image based on a preset loss function; an adjustment module 1004, configured to adjust the parameters of the model to be trained based on the loss value; a second determination module 1005, configured to, in response to the model to be trained after parameter adjustment meeting the preset training end condition, determine the model to be trained after parameter adjustment as the image restoration model.

[0148] In this embodiment, the first preprocessing module 1001 may preprocess the sample original image data by using the data preprocessing network included in the model to be trained, so as to obtain a sample intermediate image. Among them, the model to be trained may be a pre-set machine learning model that has not been trained, or a machine learning model that has not been trained completely. As Figure 3 shown in the figure, the model to be trained 301 includes a data preprocessing network 3011 and an image restoration network 3012. The data preprocessing network is configured to preprocess the sample original image data to obtain an image including multiple color channels as the sample intermediate image.

[0149] In this embodiment, the first restoration module 1002 may restore the sample intermediate image by using the image restoration network included in the model to be trained, so as to obtain a restored image. Among them, the image restoration network may include a deep neural network. The image restoration network may extract feature images (such as texture features, line shape features, etc.) from the sample intermediate image, and adjust the color values of the RGB three channels according to the image features to obtain the restored image.

[0150] In this embodiment, the first determination module 1003 may determine a loss value representing the error between the restored image and a preset reference image based on a preset loss function. Among them, the reference image is the reference image for image restoration, and the goal of image restoration is to minimize the error between the restored image and the reference image.

[0151] The loss function may be of various types, such as the existing L1 loss function, L2 loss function, etc. Through the loss function, the loss value can be calculated, and the loss value represents the error between the color values of each channel included in the restored image and the color values of each channel included in the reference image.

[0152] In this embodiment, the adjustment module 1004 may adjust the parameters of the model to be trained based on the loss value.

[0153] The training process of the model is a process of solving the optimal solution. Among them, the optimal solution is given by means of data annotation, that is, the color values of each channel included in the reference image in this embodiment. The process of the model fitting to the optimal solution is mainly carried out iteratively by the method of minimizing the error. For a set of input sample original image data, a loss function is set. This loss function can calculate the gap between the actual output of the model (i.e., the restored image) and the expected output (i.e., the reference image), and conduct this gap to the connections between each neuron in the neural network through the backpropagation algorithm. The difference signal conducted to each connection represents the contribution rate of this connection to the overall error. Then, the original model parameters can be updated and modified using the gradient descent algorithm.

[0154] In this embodiment, the second determination module 1005 may determine the to-be-trained model after adjusting the parameters as the image restoration model in response to the to-be-trained model after adjusting the parameters meeting the preset training end condition.

[0155] Specifically, the model is iteratively trained repeatedly using multiple groups of training samples, and it is determined whether the current model meets the training end condition after each training. When the training end condition is met, the current model after adjusting the parameters is the trained image restoration model. Among them, the training end condition may include but is not limited to at least one of the following: the loss value of the above loss function converges, the training time exceeds the preset duration, and the number of training times exceeds the preset number.

[0156] Refer to Figure 11 , Figure 11 FIG. is a schematic structural diagram of a generating device of an image restoration model provided by another exemplary embodiment of the present disclosure.

[0157] In some optional implementation manners, the device further includes: a selection module 1006, configured to select at least one target functional module from a set of functional modules implementing a complete image signal processing flow; a testing module 1007, configured to perform a performance test on the at least one target functional module based on a preset test scenario to obtain a test result; a construction module 1008, configured to, in response to determining that the test result meets the preset condition, construct a to-be-trained model based on the at least one target functional module.

[0158] In some optional implementation manners, the at least one target functional module includes an original image data conversion module; the construction module 1008 includes: a first construction unit 10081, configured to generate an original image data conversion sub-network included in the data preprocessing network based on the original image data conversion module; the first preprocessing module 1001 includes: a conversion unit 10011, configured to use the original image data conversion sub-network to convert the sample original image data into multi-channel original images; a generation unit 10012, configured to generate sample intermediate images based on the multi-channel original images.

[0159] In some alternative implementations, the data preprocessing network further includes a scaling sub-network; the generating unit 10012 is further configured to: use the scaling sub-network to adjust the original size of the multi-channel original image to a target size, and obtain a sample intermediate image.

[0160] In some alternative implementations, at least one target functional module includes a brightness correction module; the constructing module 1008 includes: a second constructing unit 10082, configured to generate a brightness correction sub-network included in the image restoration network based on the brightness correction module; the first restoration module 1002 includes: a correction unit 10021, configured to use the brightness correction sub-network included in the image restoration network to perform brightness correction on the sample intermediate image to obtain a corrected image; a restoration unit 10022, configured to use the image restoration sub-network included in the image restoration network to restore the corrected image to obtain a restored image.

[0161] In some alternative implementations, the restoration unit 10022 includes: a first restoration sub-unit 100221, configured to use a first sub-network included in the image restoration sub-network to restore a target channel included in the corrected image to obtain a first restored image; a second restoration sub-unit 100222, configured to use a second sub-network included in the image restoration sub-network to restore the first restored image to obtain a second restored image; a generating sub-unit 100223, configured to obtain a restored image based on the second restored image.

[0162] In some alternative implementations, the preset loss function includes a first loss function and a second loss function; the first determining module 1003 includes: a first determining unit 10031, configured to determine a first loss value representing the error between the target channel of the first restored image and the target channel of the reference image based on the first loss function; a second determining unit 10032, configured to determine a second loss value representing the error between the second restored image and the reference image based on the second loss function; a third determining unit 10033, configured to determine a loss value representing the error between the restored image and the reference image based on the first loss value and the second loss value.

[0163] The generating device of the image restoration model provided in the above embodiments of the present disclosure preprocesses the sample original image data by using the data preprocessing network included in the model to be trained to obtain the sample intermediate image, then restores the sample intermediate image by using the image restoration network included in the model to be trained to obtain the restored image, and finally trains the above model to be trained based on the preset reference image and the restored image to obtain the image restoration model. The trained image restoration model realizes that in the ISP algorithm based on a neural network, the preprocessing network is specifically used to preprocess the original image data, and then the image restoration network is used to restore the image. There is no need to use many modules included in the current ISP algorithm to process the original image data, reducing the complexity of data processing and improving the efficiency of image restoration. The restored image can be applied in the field of image recognition. Without using many ISP modules, the original image data can be restored to the image quality required in the field of image recognition, which helps to improve the efficiency of image recognition.

[0164] Figure 12 It is a schematic structural diagram of an image restoration device provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to an electronic device, such as Figure 12 As shown, the image restoration device includes: an acquisition module 1201, configured to acquire the original image data collected by an image sensor; a second preprocessing module 1202, configured to preprocess the original image data by using the data preprocessing network included in the pre-trained image restoration model to obtain an intermediate image; a second restoration module 1203, configured to restore the intermediate image by using the image restoration network included in the pre-trained image restoration model to obtain the restored image.

[0165] In this embodiment, the acquisition module 1201 can acquire the original image data collected by the image sensor locally or remotely. Among them, the image sensor can be the photosensitive sensor in the camera included in the above electronic device, or the photosensitive sensor in the camera connected to the above electronic device.

[0166] In this embodiment, the second preprocessing module 1202 can preprocess the original image data by using the data preprocessing network included in the pre-trained image restoration model to obtain an intermediate image.

[0167] Among them, the image restoration model is the model trained according to the above Figure 2 corresponding embodiment. The data preprocessing network and the image restoration network included in the image restoration model can refer to the above Figure 2 corresponding embodiment, which will not be elaborated here.

[0168] In this embodiment, the second recovery module 1203 may use the image recovery network included in the pre-trained image recovery model to recover the intermediate image, and obtain a recovered image. Generally, the obtained recovered image can be used for image recognition. For example, object detection can be performed on the recovered image to determine the position and category of the target object (such as a human body, an animal, a vehicle, etc.) in the recovered image.

[0169] The image recovery device provided in the above embodiments of the present disclosure realizes the function of ISP based on a neural network by using a pre-trained image recovery model to perform image recovery on the original image data. It does not need to use many modules included in the current ISP algorithm to process the original image data, reduces the complexity of data processing, and improves the efficiency of image recovery. The recovered image can be applied in the field of image recognition. Without using many ISP modules, the original image data can be recovered to the image quality required in the field of image recognition, which helps to improve the efficiency of image recognition.

[0170] Exemplary Electronic Device

[0171] Next, refer to Figure 13 to describe an electronic device according to an embodiment of the present disclosure. The electronic device may be any one or both of the terminal device 101 and the server 103 shown in Figure 1 or a stand-alone device independent of them. The stand-alone device can communicate with the terminal device 101 and the server 103 to receive the input signals collected from them.

[0172] Figure 13 The block diagram of the electronic device according to an embodiment of the present disclosure is shown.

[0173] As Figure 13 shown, the electronic device 1300 includes one or more processors 1201 and a memory 1202.

[0174] The processor 1301 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1300 to perform desired functions.

[0175] The memory 1302 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1301 may run the program instructions to implement the generation method of the image restoration model or the image restoration method of the various embodiments of the present disclosure above and / or other desired functions. Various contents such as original image data, restored images, etc. may also be stored in the computer-readable storage medium.

[0176] In one example, the electronic device 1300 may further include: an input device 1303 and an output device 1304, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0177] For example, when the electronic device is the terminal device 101 or the server 103, the input device 1303 may be devices such as a camera, a mouse, a keyboard, etc., for inputting contents such as original image data, various commands, etc. When the electronic device is a stand-alone device, the input device 1303 may be a communication network connector for receiving the input original image data, various commands, etc. from the terminal device 101 and the server 103.

[0178] The output device 1304 may output various information to the outside, including the restored image, etc. The output device 1304 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0179] Of course, for simplicity, Figure 13 only some of the components related to the present disclosure in the electronic device 1300 are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application scenarios, the electronic device 1300 may further include any other appropriate components.

[0180] Exemplary Computer Program Product and Computer Readable Storage Medium

[0181] In addition to the above methods and devices, the embodiments of the present disclosure may also be computer program products, which include computer program instructions that, when run by a processor, cause the processor to execute the steps in the generation method of the image restoration model or the image restoration method according to various embodiments of the present disclosure described in the "Exemplary Method" section above of this specification.

[0182] The computer program product may be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The programming code may be executed entirely on the user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0183] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are run by a processor, the processor is caused to execute the steps in the method for generating an image restoration model or the image restoration method according to various embodiments of the present disclosure described in the "Exemplary Method" section above of this specification.

[0184] The computer-readable storage medium may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0185] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, benefits, effects, etc. are essential for each embodiment of the present disclosure. In addition, the above-disclosed specific details are only for illustrative purposes and for ease of understanding, and are not limitations. The above details do not limit the present disclosure to necessarily adopt the above specific details for implementation.

[0186] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments may be referred to each other. For system embodiments, since they basically correspond to method embodiments, they are described relatively simply, and the relevant parts may refer to the partial description of the method embodiments.

[0187] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open-ended terms that mean "including but not limited to" and can be used interchangeably with each other. The word "or" and "and" used herein refer to the word "and / or" and can be used interchangeably with it, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to" and can be used interchangeably with it.

[0188] The methods and apparatuses of this disclosure can be implemented in many ways. For example, the methods and apparatuses of this disclosure can be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of the steps for the method is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the specific order described above, unless otherwise specifically stated. In addition, in some embodiments, this disclosure can also be implemented as a program recorded in a recording medium, and these programs include machine-readable instructions for implementing the methods according to this disclosure. Therefore, this disclosure also covers the recording medium storing the programs for executing the methods according to this disclosure.

[0189] It should also be noted that in the apparatuses, equipment, and methods of this disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of this disclosure.

[0190] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0191] The above description has been given for purposes of illustration and description. In addition, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions, and sub-combinations thereof.

Claims

1. A method for generating an image restoration model, comprising: Preprocessing sample original image data by using a data preprocessing network included in the model to be trained, to obtain sample intermediate images; Restoring the sample intermediate images by using an image restoration network included in the model to be trained, to obtain restored images; Based on a preset loss function, determining a loss value representing the error between the restored images and preset reference images; Adjusting parameters of the model to be trained based on the loss value; In response to the model to be trained after parameter adjustment meeting a preset training end condition, determining the model to be trained after parameter adjustment as an image restoration model; Before the step of preprocessing sample original image data by using a data preprocessing network included in the model to be trained to obtain sample intermediate images, the method further comprises: Selecting at least one target functional module from a set of functional modules implementing a complete image signal processing flow; Performing performance testing on the at least one target functional module based on a preset test scenario, to obtain test results; In response to determining that the test results meet preset conditions, constructing the model to be trained based on the at least one target functional module.

2. The method according to claim 1, wherein, The at least one target functional module includes an original image data conversion module; The constructing the model to be trained based on the at least one target functional module includes: Generating an original image data conversion sub-network included in the data preprocessing network based on the original image data conversion module; The preprocessing sample original image data by using a data preprocessing network included in the model to be trained to obtain sample intermediate images includes: Converting the sample original image data into multi-channel original images by using the original image data conversion sub-network; Generating the sample intermediate images based on the multi-channel original images.

3. The method according to claim 2, wherein, The data preprocessing network further includes a scaling sub-network; The generating the sample intermediate images based on the multi-channel original images includes: Adjusting the original size of the multi-channel original images to a target size by using the scaling sub-network, to obtain the sample intermediate images.

4. The method according to any one of claims 1 to 3, wherein, The at least one target functional module includes a brightness correction module; The constructing the model to be trained based on the at least one target functional module includes: Generating a brightness correction sub-network included in the image restoration network based on the brightness correction module; The restoring the sample intermediate images by using an image restoration network included in the model to be trained to obtain restored images includes: Performing brightness correction on the sample intermediate images by using the brightness correction sub-network included in the image restoration network, to obtain corrected images; Restoring the corrected images by using an image restoration sub-network included in the image restoration network, to obtain the restored images.

5. The method according to claim 4, wherein The restoring the corrected images by using an image restoration sub-network included in the image restoration network to obtain the restored images includes: Restoring a target channel included in the corrected images by using a first sub-network included in the image restoration sub-network, to obtain a first restored image; Using the second sub-network included in the image restoration sub-network, restore the first restored image to obtain a second restored image; Based on the second restored image, obtain the restored image.

6. The method according to claim 5, wherein, The preset loss function includes a first loss function and a second loss function; The determining, based on the preset loss function, of a loss value representing the error between the restored image and a preset reference image includes: Based on the first loss function, determining a first loss value representing the error between the target channel of the first restored image and the target channel of the reference image; Based on the second loss function, determining a second loss value representing the error between the second restored image and the reference image; Based on the first loss value and the second loss value, determining a loss value representing the error between the restored image and the reference image.

7. An image restoration method, comprising: Obtaining original image data collected by an image sensor; Using a data preprocessing network included in a pre-trained image restoration model to preprocess the original image data to obtain an intermediate image; Using an image restoration network included in the pre-trained image restoration model to restore the intermediate image to obtain a restored image; The pre-trained image restoration model is trained by the generation method of the image restoration model according to any one of claims 1-6 above.

8. An apparatus for generating an image restoration model, comprising: A first preprocessing module, configured to use a data preprocessing network included in a model to be trained to preprocess sample original image data to obtain a sample intermediate image; A first restoration module, configured to use an image restoration network included in the model to be trained to restore the sample intermediate image to obtain a restored image; A first determination module, configured to determine a loss value representing the error between the restored image and a preset reference image based on a preset loss function; An adjustment module, configured to adjust the parameters of the model to be trained based on the loss value; A second determination module, configured to, in response to the model to be trained after parameter adjustment meeting a preset training end condition, determine the model to be trained after parameter adjustment as an image restoration model; Further comprising: a selection module, configured to select at least one target functional module from a set of functional modules implementing a complete image signal processing flow; A testing module, configured to perform a performance test on the at least one target functional module based on a preset test scenario to obtain a test result; A construction module, configured to, in response to determining that the test result meets a preset condition, construct the model to be trained based on the at least one target functional module.

9. An image restoration apparatus, comprising: An acquisition module, configured to acquire original image data collected by an image sensor; A second preprocessing module, configured to use a data preprocessing network included in a pre-trained image restoration model to preprocess the original image data to obtain an intermediate image; A second restoration module, configured to use an image restoration network included in a pre-trained image restoration model to restore the intermediate image to obtain a restored image; The pre-trained image restoration model is trained by the generation method of the image restoration model according to any one of claims 1-6 above.

10. A computer-readable storage medium storing a computer program for executing the method according to any one of claims 1-7 above.

11. An electronic device, comprising: a processor; a memory for storing executable instructions of the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1-7 above.

Images