Image processing method, device and computer program product thereof

Abstract

A method, apparatus and computer program product for processing an image by a convolutional neural network (CNN) are proposed. An original image is received from an image source. The original image has a predefined size and a high resolution and is represented in a first color space supported by the image source. Then, an intermediate image is obtained by reducing the original image of the first color space and converting from the first color space to a second color space. Next, a restored image is obtained by enlarging the converted intermediate image to a predefined size of the original image. The enlargement is performed by using a CNN, with the original image and the converted intermediate image as input, and returning the restored image. The CNN is pre-trained on a set of triples, which includes a past original image, a converted past intermediate image and a past restored image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a method, device and computer program product for processing images using a convolutional neural network (CNN). Background Art

[0002] Image processing pipelines (IPP) are usually used to convert raw images generated by input optical sensors into resulting images to be rendered. High-performance IPP is crucial in consumer (especially mobile) devices, domain-specific tasks like scientific tasks (e.g., DNA sequencing), etc. Nowadays, artificial intelligence (AI) and deep learning techniques are widely used in many IPPs. More specifically, CNN is currently the core and state-of-the-art technology in most deep learning tasks related to image processing. Therefore, high-performance IPP requires CNN for proper reasoning.

[0003] One possible way to obtain high performance of CNN-based IPP is to reduce the image resolution. In short, this method involves reducing the original image, processing the reduced image in a controlled manner, and upscaling the processed reduced image to the original size by using CNN. However, the above method is domain-specific because it may negatively affect the level of detail in the original image. In particular, although this method reduces the image resolution by N times, it will reduce N 2 times computation, but it causes side effects including severe detail / quality loss. In addition, if the resolution of the original image is higher than 1080p (corresponding to 1920×1080 pixels), some existing CNN-based IPPs that use downscaled images, such as trainable guided filters and deep bilateral learning, may not provide sufficient real-time processing performance. Summary of the invention

[0004] This summary is provided to introduce some concepts in a simplified form that are further described in the following detailed description. This summary is not intended to identify key features or essential features of the invention, nor is it intended to be used to limit the scope of the invention.

[0005] An object of the present invention is to provide a technical solution for converting a raw image of an image source into an image suitable for an image renderer without causing severe detail / quality loss and without using complex computational concepts such as those used in trainable guided filters and deep bilateral learning.

[0006] The above objects are achieved by the features of the independent claims in the attached claims. Further embodiments and examples are evident from the dependent claims, the detailed description and the drawings.

[0007] According to a first aspect, an image processing method is provided, and the execution process is as follows. First, an original image is received from an image source. The original image has a predefined size and a high resolution, and the original image is represented in a first color space supported by the image source. Then, an intermediate image is obtained by reducing the original image of the first color space, and converted from the first color space to a second color space supported by an image renderer. Next, a restored image is obtained by enlarging the converted intermediate image to a predefined size of the original image while providing a similar high resolution of the restored image. The enlargement is performed by using a convolutional neural network (CNN), which is used to receive the original image and the converted intermediate image as input and return the restored image based on the input. The CNN is pre-trained based on a training set of triplets, each of which includes a past original image of the image source, and a converted past intermediate image and a past restored image that both correspond to the past original image. The method performed in this way can simplify the image processing between the image source and the image renderer, while providing a high resolution of the restored image. In addition, the method performed in this way can be applied to real-time processing of original images with a resolution higher than 1080p.

[0008] In an embodiment of the first aspect, the predefined size of the original image is represented as (w, h), where w and h are the width and height of the original image in pixels, respectively. Conversely, the reducing the original image in the first color space is performed so that the intermediate image has a size represented as (w / 4, h / 4). This can reduce the computational cost of further image processing including color space conversion.

[0009] In one embodiment of the first aspect, the training set of triplets includes past raw images that are similar to the received raw images in content, resolution, and lighting conditions. This can enable CNN to be trained more efficiently.

[0010] In one embodiment of the first aspect, the formation process of each triple is as follows. First, a past original image represented in a first color space is received, and then reduced to obtain a past intermediate image in the first color space. Next, the past intermediate image represented in the first color space is converted to a second color space. Thereafter, a past restored image is obtained by converting the past original image represented in the first color space to the second color space. Finally, the past original image is combined with the converted past intermediate image and the past restored image to form a corresponding triple. This can enable a CNN trained on a training set of triples formed in this way to operate more efficiently.

[0011] In an embodiment of the first aspect, said reducing the original image and said reducing each past original image are performed by using the same interpolation algorithm or different interpolation algorithms. This can make the method according to the first aspect more flexible in use.

[0012] In one embodiment of the first aspect, the original image includes a high-frequency component and a low-frequency component. In this embodiment, the CNN includes a first extraction block for extracting the high-frequency component from the original image, a second extraction block for extracting the low-frequency component from the converted intermediate image, and a decoding block for merging the extracted high-frequency component and the extracted low-frequency component into the restored image. This architecture of the CNN can make the CNN operate more efficiently.

[0013] In one embodiment of the first aspect, each of the first extraction block, the second extraction block and the decoding block includes a 2D convolution layer of at least one neuron. This can make the converted intermediate image more efficiently upgraded to the restored image.

[0014] In one embodiment of the first aspect, the first extraction block and the second extraction block in the convolutional neural network are used for parallel operation. This can make the CNN operation faster.

[0015] According to a second aspect, an image processing device is provided, comprising at least one processor and a memory coupled to the at least one processor. The memory stores processor executable instructions, and when the at least one processor executes the processor executable instructions, the at least one processor is caused to: receive an original image from an image source, the original image having a predefined size and a high resolution, and the original image is represented in a first color space supported by the image source; obtain an intermediate image by reducing the original image in the first color space; convert the intermediate image represented in the first color space to a second color space supported by an image renderer; obtain a restored image by enlarging the converted intermediate image to a predefined size of the original image while providing a similar high resolution of the restored image, at least one processor is used to perform the enlargement by using a convolutional neural network (CNN), and the CNN is used to receive the original image and the converted intermediate image as inputs, and return the restored image based on the inputs. At least one processor is also used to pre-train the CNN based on a training set of triplets, each triplet including a past original image of the image source, and a converted past intermediate image and a past restored image, each corresponding to the past original image. The device implemented in this way can simplify image processing between the image source and the image renderer, while providing a high resolution of the restored image. In addition, the device implemented in this way can be suitable for real-time processing of original images with a resolution higher than 1080p.

[0016] In an embodiment of the second aspect, the predefined size of the original image is represented as (w, h), where w and h are the width and height of the original image in pixels, respectively. In turn, at least one processor is configured to perform said reducing the original image in the first color space such that the intermediate image has a size represented as (w / 4, h / 4). This can reduce the computational cost of further image processing including color space conversion.

[0017] In one embodiment of the second aspect, the training set of triplets includes past raw images that are similar to the received raw images in content, resolution, and lighting conditions. This can enable CNN to be trained more efficiently.

[0018] In an embodiment of the second aspect, the at least one processor is further configured to form each triple by:

[0019] - receiving a past original image represented in a first color space;

[0020] - obtaining a past intermediate image by reducing the past original image in the first color space;

[0021] - converting a past intermediate image represented in a first color space into a second color space;

[0022] - obtaining a past restored image by converting the past original image represented by the first color space into the second color space;

[0023] - Combining the past original image with the converted past intermediate image and past restored image.

[0024] This can enable a CNN trained on a training set of triplets formed in this way to operate more efficiently.

[0025] In an embodiment of the second aspect, at least one processor is further configured to perform said reducing the original image and said reducing each past original image by using the same interpolation algorithm or different interpolation algorithms. This may make the apparatus according to the second aspect more flexible in use.

[0026] In one embodiment of the second aspect, the original image includes a high-frequency component and a low-frequency component. In this embodiment, the CNN includes a first extraction block for extracting the high-frequency component from the original image, a second extraction block for extracting the low-frequency component from the converted intermediate image, and a decoding block for merging the extracted high-frequency component and the extracted low-frequency component into the restored image. This architecture of the CNN can make the CNN operate more efficiently.

[0027] In one embodiment of the second aspect, each of the first extraction block, the second extraction block and the decoding block includes a 2D convolution layer of at least one neuron. This can make the converted intermediate image more efficiently upgraded to the restored image.

[0028] In one embodiment of the second aspect, the first extraction block and the second extraction block in the CNN are used for parallel operation. This can make the CNN operation faster.

[0029] According to a third aspect, a computer program product is provided, comprising a computer-readable storage medium storing a computer program. When at least one processor executes the computer program, at least one processor is caused to execute the method according to the first aspect. Therefore, the method according to the first aspect can be embodied in the form of a computer program, thereby providing flexibility of use.

[0030] Other features and advantages of the present invention will be apparent by reading the following detailed description and reviewing the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The essence of the present invention is described below in conjunction with the accompanying drawings.

[0032] Figure 1 An exemplary block diagram of an image processing apparatus according to an aspect of the present invention is shown.

[0033] Figure 2An exemplary block diagram of an image processing method according to another aspect of the present invention is shown.

[0034] Figure 3 Shows Figure 2 An exemplary block diagram of a convolutional neural network (CNN) used in the illustrated method.

[0035] Figure 4 Shown for training Figure 3 An exemplary block diagram of the training method of the CNN shown.

[0036] Figure 5 Shows the composition Figure 4 An exemplary flow chart of sub-steps of step S404 of the method shown.

[0037] Figure 6 and Figure 7 Shows Figure 2 Comparison results of the method shown in and two existing image processing methods (i.e., original high resolution and bilinear upsampling). DETAILED DESCRIPTION

[0038] Various embodiments of the present invention are further described in detail in conjunction with the accompanying drawings. However, the present invention can be embodied in many other forms and should not be construed as being limited to any specific structure or function disclosed in the following description. On the contrary, these embodiments are provided to make the description of the present invention detailed and complete.

[0039] According to the present invention, it is obvious to those skilled in the art that the scope of the present invention covers any embodiment disclosed herein, whether the embodiment is implemented independently or in conjunction with any other embodiment of the present invention. For example, the apparatus and method disclosed herein can be implemented by using any number of embodiments provided herein. In addition, it should be understood that any embodiment of the present invention can be implemented using one or more elements or steps set forth in the appended claims.

[0040] In the embodiments of the present invention, an image source may refer to a device for capturing an image, such as a camera, an optical sensor array, a scanner, etc., and an image renderer may refer to a device for rendering a captured image, such as a television, a computer screen, a computer printer, a display, etc. The image source and the image renderer may be integrated into one user device (e.g., a camera and a display in a mobile phone), or implemented as different user devices that are connected to each other by wire or wirelessly (e.g., a laptop camera and a printer).

[0041] In embodiments of the present invention, a color space, also referred to as a color model, may refer to a specific range of colors that a camera can see, a printer can print, or a display can display. Since the color spaces supported by an image source and an image renderer are usually different from each other, the captured image should be properly processed before being rendered by the image renderer. The processing is performed by a so-called image processing pipeline (IPP), which represents a set of intermediate steps between the image capture and rendering.

[0042] In order to convert a captured image from the color space of the image source to the color space of the image renderer, the IPP generally involves downscaling the captured image and then performing the color space conversion on the downscaled image. The reason for this is that the color space conversion only requires knowledge of the smooth color gradients present in the captured image, i.e., based on the low-frequency components of the captured image. It should be noted that the high-frequency components of the captured image are represented by sharp contrast edges that constitute the texture / details in the captured image. The high-frequency components are not required in the color space conversion and can therefore be excluded from consideration to reduce computational and time costs. The downscaling allows the isolation of the low-frequency components of the captured image for further color space conversion. Embodiments of the present invention are not limited to any particular color space of the image source and the image renderer. As an example, the color space of the image source can be represented by one of the RGB, HSL, HSV, YUV / YIQ / YCbCr, and XYZ color models, while the color space of the image renderer can be represented by another of these color models. Once the color space conversion is completed, the IPP also involves upscaling or restoring the downscaled image to its original size.

[0043] In an embodiment of the present invention, a convolutional neural network (CNN) may refer to a specialized type of neural network designed for processing image data. The main component of a CNN is a convolutional layer, which performs a convolution operation. In the context of a CNN, convolution is a linear operation that involves multiplying an array of weights (called a filter or kernel) with an array of input data. The filter is intentionally made smaller than the input data array, which allows the same filter (or weight array) to be multiplied by the input data array multiple times at different points in the input of the CNN. A CNN may include multiple convolutional layers to extract different details of the original image data.

[0044] Existing IPPs, such as those based on trainable guided filters and deep bilateral learning, use CNNs to restore the downscaled image to its original size. Although such CNN-based IPPs provide appropriate image resolution after color space conversion and restoration, they have high complexity and may not provide real-time performance when the captured images have a resolution higher than 1080p.

[0045] The present invention discussed below takes into account the above-mentioned shortcomings of the existing IPP and aims to provide a technical solution for the intermediate processing of raw images of an image source without causing severe detail / quality loss and without using complex calculation concepts.

[0046] Figure 1 FIG. 1 shows an exemplary block diagram of an image processing apparatus 100 according to an aspect of the present invention. Figure 1 As shown, the apparatus 100 includes a memory 102 and a processor 104 coupled to the memory 102. The memory 102 stores executable instructions 106 to be executed by the processor 104 to perform intermediate processing of the original image, which will be discussed in more detail later. In general, the apparatus 100 can be considered as an intermediate between an image source and an image renderer. At the same time, the apparatus 100 can be integrated into one user device together with both the image source and the image renderer, or integrated into one user device with one of the image source and the image renderer.

[0047] The memory 102 can be implemented as a volatile or non-volatile memory used in modern electronic computing machines. Examples of non-volatile memory include read-only memory (ROM), flash memory, ferroelectric random-access memory (RAM), programmable ROM (PROM), electrically erasable PROM (EEPROM), solid state drive (SSD), disk storage (such as hard disk and tape), optical disk storage (such as CD, DVD and Blu-ray disc), etc. As for volatile memory, examples include dynamic RAM, synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), static RAM, etc.

[0048] With respect to the processor 104, it can be implemented as a central processing unit (CPU), a general processor, a special processor, a microcontroller, a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a complex programmable logic device, etc. It should also be noted that the processor 104 can be implemented as any combination of one or more of the above. As an example, the processor 104 can be a combination of two or more microprocessors.

[0049] The executable instructions 106 stored in the memory 102 may be used as computer executable code for causing the processor 104 to perform various aspects of the present invention. The computer executable code for performing the operations or steps of various aspects of the present invention may be written in any combination of one or more programming languages, such as Java, C++, etc. In some examples, the computer executable code may be in the form of a high-level language or in a pre-compiled form and dynamically generated by an interpreter (also pre-stored in the memory 102).

[0050] Figure 2 1 shows an exemplary block diagram of an image processing method 200 according to another aspect of the present invention. When the processor 104 executes the instruction 106 of the memory 102 of the device 100, the processor 104 of the device 100 executes the method 200. Figure 2As shown, the method 200 starts from step S202, where the processor 104 receives an original image from an image source. The original image has a predefined size and a high resolution, and the original image is represented in a first color space supported by the image source. Then, the method 200 performs step S204, where the processor 104 obtains an intermediate image by reducing the original image of the first color space. Thereafter, in step S206, the processor 104 is instructed to convert the intermediate image from the first color space to a second color space supported by the image renderer. The method 200 ends at step S208, where the processor 104 obtains a restored image by enlarging the converted intermediate image to a predefined size of the original image while providing a similar high resolution of the restored image. In step S208, the processor 104 is instructed to perform the enlargement by using a convolutional neural network (CNN), which is used to receive the original image and the converted intermediate image as inputs and return the restored image based on the inputs. Meanwhile, the CNN used in method 200 is pre-trained based on a training set of triplets, each of which includes a past original image of an image source, and a converted past intermediate image and a past restored image, each corresponding to the past original image. In general, method 200 represents a CNN-based IPP that allows for simplified intermediate processing of the original image between the image source and the image renderer, while providing a restored image resolution that is the same or similar to the original image.

[0051] In one embodiment, if the predefined size of the original image is represented as (w, h), where w and h are the width and height of the original image in pixels, respectively, the processor 104 is used to perform, in step S204, the reduction of the original image in the first color space so that the intermediate image has a size represented as (w / 4, h / 4). Then, in step S206, the intermediate image thus reduced may be processed by using less computing resources than the amount of computing resources without the size reduction. However, the present invention is not limited to the size reduction to (w / 4, h / 4), and any other reduction ratio may be used according to certain applications.

[0052] Furthermore, the reducing the original image in step S204 may be performed by using any conventional interpolation algorithm. Some examples of interpolation algorithms include cubic, bicubic, Sinc, linear, Lanczos and nearest neighbor interpolation techniques.

[0053] Figure 3An exemplary block diagram of a CNN 300 used in step S208 of method 200 is shown. CNN 300 includes a first extraction block 302, a second extraction block 304, and a decoding block 306. The first extraction block 302 is used to receive an original image 308 from an image source and extract high-frequency components therefrom. The second extraction block 304 is used to receive a converted intermediate image 310 obtained from the original image 308 in step S206 of method 200 and extract low-frequency components therefrom. The decoding block 306 is used to receive the high-frequency components from the first extraction block 302 and the low-frequency components from the second extraction block 304, and merge the high-frequency components and the low-frequency components into a restored image 312. Each of the first extraction block 302, the second extraction block 304, and the decoding block 306 may include a 2D convolutional layer of at least one neuron. Although the design of CNN 300 is similar to an hourglass autoencoder, it is indeed different because the output of CNN 300 should essentially repeat its input. In other words, the restored image 312 generated by the CNN 300 should have a size and resolution similar to that of the original image 308. The only difference between the original image 308 and the restored image 312 is that they have different color spaces, i.e., the original image 308 is represented in one color space supported by the image source, while the restored image 312 is represented in any other color space supported by the image renderer.

[0054] In one embodiment, the first extraction block 302 and the second extraction block 304 in the CNN 300 are used for parallel operation. This is done to reduce the time required to extract high-frequency components and low-frequency components from the original image 308 and the converted intermediate image 310, respectively. At the same time, this configuration of the extraction blocks 302 and 304 should not be regarded as any limitation to the present invention, and can be replaced by any other configuration according to certain applications.

[0055] As described above, CNN 300 should be trained before being used in step S208 of method 200. The training is performed by using Figure 4 Similar to method 200, when processor 104 executes instruction 106 of memory 102 of device 100, processor 104 of device 100 executes training method 400. Figure 4 As shown, method 400 starts from step S402, where processor 104 obtains a set of past raw images generated by the image source in the past. More specifically, processor 104 selects from among all the past raw images those raw images that are similar in content, resolution, and lighting conditions to the raw images received in step S202 of method 200. This can make training more efficient. Once the set of past raw images is selected, method 400 proceeds to step S404, where processor 104 is instructed to generate a training set of triplets by using the set of past raw images. Figure 5 Step S404 is described in more detail. Thereafter, in step S406, the processor 104 is instructed to train the CNN 300 by using the training set of triples. The processor 104 repeats step S406 until the output of the CNN 300 obtains a suitable result. The processor 104 can use different traditional supervised training algorithms to perform step S406 of the method 400, for example, as a back propagation algorithm. When step S406 is completed, the CNN 300 is ready to be used in step S208 of the method 200.

[0056] Figure 5 An exemplary flowchart 500 of sub-steps constituting step S404 of method 400 is shown. In other words, the flowchart 500 of the sub-steps is used to generate each triple included in the training set of triples. Similar to methods 200 and 400, when the processor 104 executes the instructions 106 of the memory 102 of the device 100, the flowchart 500 of the sub-steps is caused to be executed by the processor 104 of the device 100. More specifically, the flowchart 500 of the sub-steps starts from sub-step S502, wherein the processor 104 receives a past raw image from a set of past raw images obtained in step S402 of method 400. As the name implies, a triple includes three elements, each of which is obtained based on the past raw image received in sub-step S502. In particular, the first element is represented by the past raw image itself. The second element is represented by a converted past intermediate image, which is obtained by reducing the past original image of the first color space in sub-step S504 and by converting the past intermediate image from the first color space to the second color space in sub-step S506. The third element is represented by a past restored image, which is obtained by converting the past original image from the first color space to the second color space in sub-step S508. In order to reduce the time required to generate the triplet, sub-steps S504, 506 and sub-step 508 can be performed in parallel. The last sub-step of flowchart 500 is sub-step S510, in which the processor 104 combines the past original image, the converted past intermediate image and the past restored image into a triplet, namely (past original image, converted past intermediate image, past restored image).

[0057] In one embodiment, the downscaling of the past original image in sub-step S504 of the flowchart 500 is performed by using the same interpolation algorithm as used in step S204 of the method 200. In another embodiment, the interpolation algorithms used in sub-step S504 and step S204 are different.

[0058] The above flowchart 500 of sub-steps S502-S510 is repeated for each past original image of the set of past original images obtained in step S402 of the method 400. Then, in step S406 of the method 400, the resulting training set of triplets is used to train the CNN 300. It should be noted that during the training of the CNN 300, the first two elements of each triplet, i.e., the past original image and the converted past intermediate image, are used as inputs, while the third element of the triplet, i.e., the past restored image, is used as output, or in other words, as a target.

[0059] Figure 6 and Figure 7 The comparison results of the method 200 disclosed in this article and two existing image processing methods (such as original high resolution (HR) and bilinear upsampling) are shown. In particular, Figure 6 and Figure 7 The restored images obtained using the original HR method, the method 200, and the bilinear upsampling method are shown from left to right, respectively. It can be seen that the original HR method and the method 200 provide restored images of similar resolution, which is much better than the resolution provided by the bilinear upsampling method. At the same time, the advantage of the method 200 over the original HR method is that the latter is quite complex and is not suitable for real-time processing of original images with a resolution higher than 1080p.

[0060] Those skilled in the art will appreciate that each box or step of methods 200, 400, or each sub-step of flowchart 500, or any combination of these boxes or steps or sub-steps, can be implemented by various means such as hardware, firmware and / or software. As an example, one or more of the boxes or steps or sub-steps described above can be embodied by computer executable instructions, data structures, program modules, and other appropriate data representations. In addition, the computer executable instructions embodying the boxes or steps or sub-steps described above can be stored on a corresponding data carrier and executed by at least one processor (such as processor 104 of device 100). The data carrier can be implemented as any computer-readable storage medium that can be read by at least one processor to execute computer executable instructions. Such computer-readable storage media may include volatile and non-volatile media, removable and non-removable media. As an example, rather than a limitation, a computer-readable medium includes a medium implemented in any method or technology suitable for storing information. In more detail, actual examples of computer readable media include, but are not limited to, information delivery media, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROMs, digital versatile discs (DVDs), holographic media or other optical disk storage, magnetic tapes, tape cassettes, disk storage, and other magnetic storage devices.

[0061] Although exemplary embodiments of the present invention are described herein, it should be noted that any various changes and modifications may be made in embodiments of the present invention without departing from the scope of legal protection defined by the appended claims. In the appended claims, the word "comprising" does not exclude other elements or steps, and the term "a" or "an" does not exclude a plurality. The listing of certain measures in mutually different dependent claims does not indicate that a combination of these measures cannot be effectively used.

Claims

1. An image processing method, characterized in that: include: Receiving an original image from an image source, the original image having a predefined size and a high resolution, and the original image being represented in a first color space supported by the image source; Obtaining an intermediate image by reducing the original image in the first color space; Convert the intermediate image represented in the first color space into a second color space supported by the image renderer to obtain a converted intermediate image; The restored image is obtained by enlarging the converted intermediate image to the predefined size of the original image while providing a similar high resolution of the restored image, wherein the enlargement is performed by using a convolutional neural network, the convolutional neural network being configured to receive the original image and the converted intermediate image as inputs and return the restored image based on the inputs, The convolutional neural network is pre-trained based on a training set of triplets, each triplet comprising a past original image of the image source, and a converted past intermediate image and a past restored image both corresponding to the past original image.

2. The method according to claim 1, characterized in that The predefined size of the original image is represented as (w, h), wherein w and h are respectively the width and height of the original image in pixels, and the downscaling of the original image in the first color space is performed so that the intermediate image has a size represented as (w / 4, h / 4).

3. The method according to claim 1 or 2, characterized in that: The training set of triplets includes the past raw images that are similar to the received raw images in content, resolution, and lighting conditions.

4. The method according to any one of claims 1 to 3, characterized in that Each triple is formed by the following steps: receiving the past original image represented in the first color space; Obtaining the past intermediate image by reducing the past original image in the first color space; converting the past intermediate image represented in the first color space to the second color space; Obtaining the past restored image by converting the past original image represented by the first color space into the second color space; The past original image is combined with the converted past intermediate image and the past restored image.

5. The method according to claim 4, characterized in that The reducing the original image and the reducing each of the past original images are performed by using the same interpolation algorithm or different interpolation algorithms.

6. The method according to any one of claims 1 to 5, characterized in that The original image includes a high-frequency component and a low-frequency component, and the convolutional neural network includes: A first extraction block, used to extract the high frequency component from the original image; A second extraction block, configured to extract the low-frequency component from the converted intermediate image; A decoding block is used to merge the extracted high-frequency component and low-frequency component into the restored image.

7. The method according to claim 6, characterized in that Each of the first extraction block, the second extraction block, and the decoding block includes a 2D convolutional layer of at least one neuron.

8. The method according to claim 6 or 7, characterized in that: The first extraction block and the second extraction block in the convolutional neural network are used for parallel operation.

9. An image processing device, characterized in that: include: at least one processor; a memory coupled to the at least one processor and storing processor-executable instructions, which, when executed by the at least one processor, cause the at least one processor to: Receiving an original image from an image source, the original image having a predefined size and a high resolution, and the original image being represented in a first color space supported by the image source; Obtaining an intermediate image by reducing the original image in the first color space; Convert the intermediate image represented in the first color space into a second color space supported by the image renderer to obtain a converted intermediate image; The restored image is obtained by enlarging the converted intermediate image to the predefined size of the original image while providing a similar high resolution of the restored image, The at least one processor is configured to perform the enlargement by using a convolutional neural network, and the convolutional neural network is configured to receive the original image and the converted intermediate image as inputs and return the restored image based on the inputs, Wherein, the at least one processor is also used to pre-train the convolutional neural network based on a training set of triplets, each triplet including a past original image of the image source, and a converted past intermediate image and a past restored image each corresponding to the past original image.

10. The device according to claim 9, characterized in that The predefined size of the original image is represented as (w, h), wherein w and h are respectively the width and height of the original image in pixels, and the at least one processor is used to perform the reduction of the original image in the first color space so that the intermediate image has a size represented as (w / 4, h / 4).

11. The device according to claim 9 or 10, characterized in that The training set of triplets includes the past raw images that are similar to the received raw images in content, resolution, and lighting conditions.

12. The device according to any one of claims 9 to 11, characterized in that The at least one processor is further configured to form each triple by: receiving the past original image represented in the first color space; Obtaining the past intermediate image by reducing the past original image in the first color space; converting the past intermediate image represented in the first color space to the second color space; Obtaining the past restored image by converting the past original image represented by the first color space into the second color space; The past original image is combined with the converted past intermediate image and the past restored image.

13. The device according to claim 12, characterized in that The at least one processor is further configured to perform the reducing the original image and the reducing each of the past original images by using the same interpolation algorithm or different interpolation algorithms.

14. The device according to any one of claims 9 to 13, characterized in that The original image includes a high-frequency component and a low-frequency component, and the convolutional neural network includes: A first extraction block, used to extract the high frequency component from the original image; A second extraction block, configured to extract the low-frequency component from the converted intermediate image; A decoding block is used to merge the extracted high-frequency component and low-frequency component into the restored image.

15. The device according to claim 14, characterized in that Each of the first extraction block, the second extraction block, and the decoding block includes a 2D convolutional layer of at least one neuron.

16. The device according to claim 14 or 15, characterized in that The first extraction block and the second extraction block in the convolutional neural network are used for parallel operation.

17. A computer program product, characterized in that A computer-readable storage medium storing a computer program, which, when at least one processor executes the computer program, causes the at least one processor to perform the method according to any one of claims 1 to 8.

Images