Image processing method and device

Abstract

The invention provides an image processing method and device, relates to the technical field of image processing, and is used for improving the quality of an image shot by an under-screen lens. The method comprises the following steps: acquiring high-frequency image information of a target image to obtain a high-frequency component image, and acquiring low-frequency image information of the target image to obtain a low-frequency component image; according to the high-frequency image information, performing restoration processing on the high-frequency component image to obtain a high-frequency restored image, and according to the low-frequency image information, performing restoration processing on the low-frequency component image to obtain a low-frequency restored image; and determining a repaired image corresponding to the target image according to the high-frequency repaired image and the low-frequency repaired image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method and device. Background Art

[0002] In recent years, full-screen mobile phones have gained the favor of many users with their high screen-to-body ratio and high-quality appearance. Usually, the front camera of a full-screen mobile phone is hidden under the display screen, which is called an under-screen camera. When users use a full-screen mobile phone to take selfies, they usually need to use the under-screen camera.

[0003] However, the under-screen lens has certain defects. For example, the diffraction and diffusion phenomenon caused by the influence of the display screen causes the images taken with the under-screen lens to have low clarity and poor image quality. Summary of the invention

[0004] The present application provides an image processing method and device for improving the quality of images captured by an under-screen camera.

[0005] In order to at least achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] In a first aspect, an image processing method is provided, the method comprising: acquiring high-frequency image information of a target image to obtain a high-frequency component image, and acquiring low-frequency image information of the target image to obtain a low-frequency component image; repairing the high-frequency component image according to the high-frequency image information to obtain a high-frequency repaired image, and repairing the low-frequency component image according to the low-frequency image information to obtain a low-frequency repaired image; and determining a repaired image corresponding to the target image according to the high-frequency repaired image and the low-frequency repaired image.

[0007] In a second aspect, an image processing device is provided, which includes an acquisition unit, a processing unit and a determination unit; the acquisition unit is used to acquire high-frequency image information of a target image to obtain a high-frequency component image, and to acquire low-frequency image information of the target image to obtain a low-frequency component image; the processing unit is used to perform repair processing on the high-frequency component image according to the high-frequency image information to obtain a high-frequency repair image, and to perform repair processing on the low-frequency component image according to the low-frequency image information to obtain a low-frequency repair image; the determination unit is used to determine a repair image corresponding to the target image according to the high-frequency repair image and the low-frequency repair image.

[0008] According to a third aspect, an electronic device is provided, comprising: a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions so that the electronic device performs the image processing method according to the first aspect.

[0009] In a fourth aspect, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the image processing method of the first aspect described above.

[0010] In an embodiment of the present application, the electronic device obtains high-frequency image information of the target image to obtain a high-frequency component image, and obtains low-frequency image information of the target image to obtain a low-frequency component image. In this way, the target image can be divided into a high-frequency part and a low-frequency part. Further, based on the high-frequency image information, the high-frequency component image is repaired to obtain a high-frequency repaired image, and based on the low-frequency image information, the low-frequency component image is repaired to obtain a low-frequency repaired image, so that the high-frequency details and low-frequency details of the target image are repaired respectively. Then, based on the high-frequency repaired image and the low-frequency repaired image, the repaired image corresponding to the target image can be determined, thereby completing the repair of the target image and improving the quality of the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in some embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only drawings of some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0012] Figure 1 Schematic diagram of scenarios provided for some embodiments of the present application;

[0013] Figure 2 A schematic diagram of the structure of an electronic device provided in some embodiments of the present application;

[0014] Figure 3 A schematic diagram of a logic module of an electronic device provided in some embodiments of the present application;

[0015] Figure 4 A schematic diagram of a flow chart of an image processing method provided in some embodiments of the present application;

[0016] Figure 5 A schematic diagram of a preprocessing module provided in some embodiments of the present application;

[0017] Figure 6 A schematic diagram of the structure of an image restoration network provided in some embodiments of the present application;

[0018] Figure 7 A schematic diagram of the structure of a high-frequency repair network provided in some embodiments of the present application;

[0019] Figure 8 A schematic diagram of the structure of a residual dense block provided in some embodiments of the present application;

[0020] Fig. 9 A schematic diagram of the structure of a low-frequency repair network provided in some embodiments of the present application;

[0021] Fig.10 A schematic diagram of the structure of the encoder layer provided in some embodiments of the present application;

[0022] Fig.11 A schematic diagram of the structure of a downsampling layer provided in some embodiments of the present application;

[0023] Fig.12 A schematic diagram of the structure of a downsampling coding layer provided in some embodiments of the present application;

[0024] Fig.13 A schematic diagram of the structure of a decoding layer provided in some embodiments of the present application;

[0025] Fig.14 A schematic diagram of the structure of an upsampling decoding layer provided in some embodiments of the present application;

[0026] Fig.15 A schematic diagram of the structure of an upsampling layer provided in some embodiments of the present application;

[0027] Fig.16 A schematic diagram of a post-processing module provided in some embodiments of the present application;

[0028] Fig.17 A schematic diagram of an image mask acquisition process provided in some embodiments of the present application;

[0029] Fig.18 A schematic diagram of the repair effect provided by some embodiments of the present application;

[0030] Fig.19 A schematic diagram of regional mask processing provided for some embodiments of the present application;

[0031] Fig. 20 A schematic diagram of the structure of an image processing device provided in some embodiments of the present application;

[0032] Fig.21 A schematic diagram of the structure of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0034] In the description of the present application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. The words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit them to be different. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0035] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0036] In the embodiment of the present application, "indication" may include direct indication and indirect indication. For example, taking the indication information in the following text as an example, the indication information may directly carry information A itself or its index to achieve the purpose of directly indicating information A. Alternatively, the indication information may also carry information B that is associated with information A, thereby achieving the purpose of indirectly indicating information A while indicating information B.

[0037] In the related art, in order to ensure the quality of the image captured by the front camera of the electronic device, the front camera can be set in the non-display area of ​​the electronic device, and a hole is opened in the non-display area, and the front camera is set below the hole. The front camera can be a camera that emits a point light source. The front camera can emit light outward during the image shooting process. The light passes through the hole to illuminate the target object and is reflected back to the front camera. The reflected light is collected by the front camera, and the front camera forms an image based on the reflected light to obtain an image of the target object. However, the non-display area allows the display screen to occupy part of the area of ​​the electronic device, and the display screen cannot be a full screen.

[0038] If the display screen is full screen, the front camera can be set / hidden under the display screen. The front camera set under the display screen emits light, and the light forms reflected light when it hits the target object. The reflected light is transmitted to the front camera through the display screen. The reflected light diffracts when passing through the display screen, which reduces the brightness of the reflected light received by the front camera and makes the reflected light diverge. The image captured by the front camera is blurred and has light spots.

[0039] In order to reduce the impact of diffraction on the image, the related technology introduces a point spread function (PSF), and performs deconvolution processing based on the PSF and the image captured by the front camera. For example, the PSF is deconvolved with the first grayscale matrix of the image to obtain the second grayscale matrix of the image. Based on the second grayscale matrix of the image, the image is obtained. The first grayscale matrix and the second grayscale matrix are matrices composed of the grayscale values ​​of each pixel in the image.

[0040] PSF can reflect the light field distribution of the output image when the input object is a point light source. PSF can be obtained through front camera shooting, optical simulation, mathematical modeling and other methods. However, due to factors such as sensor saturation corresponding to the front camera and noise of the front camera, the PSF obtained through front camera shooting has errors. The PSF bias theory obtained by optical simulation and mathematical modeling reflects a difference in light field distribution from the actual light field distribution. Therefore, the PSF obtained through front camera shooting, optical simulation, mathematical modeling and other methods has low accuracy. The grayscale value of the pixel point restored during PSF deconvolution processing is incorrect, the despeckle effect is reduced, and the image quality is reduced. In addition, it takes a long time to obtain PSF based on the above three methods, which reduces efficiency.

[0041] It can be seen that PSF has defects such as high difficulty in acquisition and use, and is not suitable for scenes where images are taken with a front camera. The above-mentioned PSF method cannot accurately and efficiently repair the images taken with the front camera.

[0042] In view of this, the image processing method provided in the embodiment of the present application divides the under-screen image into a high-frequency part and a low-frequency part. The high-frequency part is used to restore the high-frequency details of the image, and the low-frequency part is used to restore the low-frequency details of the image. Further, the two are added together to obtain the final restored image, which improves the quality of the image captured by the under-screen lens and avoids the use of PSF measurement method to repair the image.

[0043] The image processing method provided by the embodiment of the present application is described in detail below in conjunction with the accompanying drawings. It should be noted that the embodiments of the present application can refer to each other, for example, the same or similar steps, method embodiments, system embodiments and device embodiments can refer to each other without limitation.

[0044] The image processing method provided in the present application can be applied to electronic devices. In some embodiments, the electronic device can be a mobile phone, a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a smart watch, etc. The present application does not specifically limit the specific form of the electronic device.

[0045] In some scenarios, such as Figure 1 As shown, the user takes a photo through an electronic device to obtain a target image. The electronic device then performs repair processing on the captured target image to obtain a repaired image corresponding to the target image. The image processing process can refer to the image processing method described in the following method embodiment, which will not be repeated here.

[0046] like Figure 2 As shown, the electronic device may include: a processor 11, a display screen 12, and an under-screen camera 13. The processor 11 may control the under-screen camera 13 to take pictures, and display the taken images through the display screen 12.

[0047] In some embodiments, the electronic device may further include an external memory interface, an internal memory, a Universal Serial Bus (USB) interface, a charging management module, a power management module, a battery, an antenna 1, an antenna 2, a mobile communication module, a wireless communication module, a sensor module, a button, a motor, an indicator, a camera, and a Subscriber Identification Module (SIM) card interface, etc. The audio module may include, but is not limited to, one or more of the following, such as a speaker, a receiver, a microphone, an earphone interface, etc. The sensor module may include, but is not limited to, one or more sensors, such as a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0048] The processor 11 may include one or more processing units, for example: the processor may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Among them, different processing units can be independent devices or integrated into one or more processors. The processor is the nerve center and command center of the electronic device. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0049] The display screen 12 can be used to display images, videos, a series of graphical user interfaces (GUIs), etc.

[0050] The external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor through the external memory interface to implement the data storage function. For example, music, video and other files are saved in the external memory card. The internal memory can be used to store computer executable program code, and the executable program code includes instructions. The processor executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory. For example, in the present application, the processor executes the instructions stored in the internal memory so that the electronic device executes the image processing method provided in the present application.

[0051] The camera is used to capture still images or videos. The object is projected onto the photosensitive element through the lens to generate an optical image. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then passed to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard optical primary colors (RGB), color coding (YUV) and other formats.

[0052] In some embodiments, the electronic device may include 1 or N cameras, where N is a positive integer greater than 1, and m of the N cameras are arranged below the display screen, and the camera arranged below the display screen may be used as a front camera, and the camera arranged below the display screen may capture an image under the screen. Other cameras may be arranged on the back of the electronic device and used as rear cameras, where m<=N.

[0053] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0054] In addition, different logic modules are deployed on top of the above components. Figure 3 As shown, the electronic device may be deployed with a pre-processing module 21, a network repair module 22, and a post-processing module 23. The electronic device may perform pre-processing operations such as noise reduction and format conversion on the target image through the pre-processing module 21, repair the pre-processed image through the network repair module 22, and output the repaired image through the post-processing module 23.

[0055] The application scenarios of the embodiments of the present application are not limited. The above application scenarios and the structure of the electronic device described in the embodiments of the present application are for the purpose of more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It is known to those skilled in the art that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0056] The following is an introduction to the image processing method provided in the embodiments of the present application.

[0057] Figure 4 The flowchart of the image processing method provided in the embodiment of the present application may include the following steps:

[0058] S201 . Acquire high-frequency image information of a target image to obtain a high-frequency component image, and acquire low-frequency image information of a target image to obtain a low-frequency component image.

[0059] As a possible implementation manner, the electronic device may input the target image into a preset Gaussian filter to obtain low-frequency image information. Further, the electronic device may subtract the low-frequency image information from the target image to obtain high-frequency image information.

[0060] As another possible implementation manner, the electronic device may input the target image into a preset high-pass filter to obtain high-frequency image information. Further, the electronic device may subtract the high-frequency image information from the target image to obtain low-frequency image information.

[0061] It should be noted that image frequency refers to an indicator of the intensity of grayscale changes in an image. Different frequency information has different functions in an image. Low-frequency image information indicates areas where grayscale values ​​change slowly in an image, corresponding to large flat areas in the image, that is, large-scale information, often referred to as the background, which is the general outline and contour of the image and reflects the approximate information of the image. High-frequency image information indicates areas where grayscale values ​​change dramatically in an image, corresponding to the edges, noise, and details of the image, and reflects small-scale detail information.

[0062] The target image can be captured by the electronic device through a camera, or obtained by the electronic device from other devices (such as downloaded by the electronic device through the Internet or sent by other devices). The target image can be an under-screen image or a non-under-screen image (such as an image captured by the electronic device through a rear camera). The embodiment of the present application does not limit the source and type of the target image.

[0063] In some embodiments, the electronic device may include a front camera and a display screen, the front camera is arranged below the display screen, and the electronic device may shoot through the front camera to obtain an under-screen image. The electronic device may use the under-screen image shot by the front camera as the target image, and repair the under-screen image to improve the quality of the under-screen image.

[0064] S202: According to the high-frequency image information, the high-frequency component image is repaired to obtain a high-frequency repaired image, and according to the low-frequency image information, the low-frequency component image is repaired to obtain a low-frequency repaired image.

[0065] As a possible implementation, the electronic device performs a first restoration process on the high-frequency component image according to the high-frequency image information to obtain a high-frequency restoration image. The electronic device performs a second restoration process on the low-frequency component image according to the low-frequency image information to obtain a low-frequency restoration image.

[0066] It should be noted that the first restoration process may be to reconstruct the high frequency information to obtain high frequency reconstruction information. The amount of information contained in the high frequency reconstruction information is greater than the amount of information contained in the high frequency information. The electronic device may obtain a high frequency restoration image based on the high frequency reconstruction information.

[0067] In some embodiments, the electronic device may repair and / or expand the high-frequency information according to the pixels of the high-frequency component image to obtain high-frequency reconstruction information, and use the image corresponding to the high-frequency reconstruction information as the high-frequency repaired image.

[0068] Exemplarily, high-frequency information restoration may be to change pixel information in a high-frequency region through surrounding pixel information to restore high-frequency details of an image. High-frequency information expansion may be to add pixel information in a high-frequency region through surrounding pixel information to expand high-frequency details of an image.

[0069] Similarly, the second restoration process may be to reconstruct the low-frequency information to obtain low-frequency reconstruction information. The amount of information contained in the low-frequency reconstruction information is greater than the amount of information contained in the low-frequency information. The electronic device may obtain a low-frequency restoration image based on the low-frequency reconstruction information.

[0070] In some embodiments, the electronic device may repair and / or expand the low-frequency information according to the pixels of the low-frequency component image to obtain low-frequency reconstruction information, and use the image corresponding to the low-frequency reconstruction information as the low-frequency repaired image.

[0071] Exemplarily, low frequency information restoration may be to change pixel information of the low frequency region through surrounding pixel information to restore low frequency details of the image. Low frequency information expansion may be to add pixel information of the low frequency region through surrounding pixel information to expand low frequency details of the image.

[0072] S203: Determine a restoration image corresponding to the target image according to the high-frequency restoration image and the low-frequency restoration image.

[0073] As a possible implementation manner, the electronic device superimposes the high-frequency restoration image and the low-frequency restoration image to obtain the restoration image.

[0074] It can be understood that the image is essentially a matrix composed of numerical values. One element in the matrix corresponds to one pixel. Therefore, the electronic device can sum the pixels of the high-frequency repaired image with the pixels of the low-frequency repaired image to achieve the superposition of the high-frequency repaired image and the low-frequency repaired image, thereby obtaining the repaired image.

[0075] In an embodiment of the present application, the electronic device obtains high-frequency image information of the target image to obtain a high-frequency component image, and obtains low-frequency image information of the target image to obtain a low-frequency component image. In this way, the target image can be divided into a high-frequency part and a low-frequency part. Further, based on the high-frequency image information, the high-frequency component image is repaired to obtain a high-frequency repaired image, and based on the low-frequency image information, the low-frequency component image is repaired to obtain a low-frequency repaired image, so that the high-frequency details and low-frequency details of the target image are repaired respectively. Then, based on the high-frequency repaired image and the low-frequency repaired image, the repaired image corresponding to the target image can be determined, thereby completing the repair of the target image and improving the quality of the target image.

[0076] In one design, the electronic device is deployed with a trained image restoration network, wherein the image restoration network can be trained by the electronic device or by other devices and pre-stored in the electronic device by operation and maintenance personnel.

[0077] For example, the electronic device may obtain multiple sample pairs, and based on the multiple sample pairs, train the image restoration network to be trained to obtain a trained image restoration network. A sample pair includes a sample under-screen image and a sample non-under-screen image of the same photographed object.

[0078] In order to obtain a repaired image, the embodiment of the present application may also include: the electronic device inputs the target image into a trained image repair network, and outputs a repaired image corresponding to the target image.

[0079] It should be noted that when the target image is in the raw data format (RAW Image Format, RAW), the electronic device can perform image signal processing (Image Signal Processing, ISP) on the target image and then input it into the trained image restoration network. The RAW format is an image file format in which the electrical signal obtained from the image sensor during shooting is directly stored without other processing after analog-to-digital conversion (i.e. A / D conversion).

[0080] For example, the image under the screen taken by the front camera of an electronic device is a RAW image. RAW images have the characteristics of bit enhancement and losslessness, which are more conducive to image restoration tasks. However, due to the differences in brightness and color of RAW images, operations such as lens shading correction and white balance need to be performed, so that RAW images are conducive to the learning of the image restoration network and improve the ability of the image restoration network to restore the image under the screen.

[0081] In practical applications, the electronic device can pre-process the off-screen image through the pre-processing module to adjust the off-screen image in RAW format to an image that is more conducive to image restoration network processing. Figure 5As shown, the preprocessing module may include: lens shading correction, white balance, gamma transformation, etc. Lens shading correction requires the use of the lens shading parameters of the camera on the electronic device (which can be obtained through the interface of the operating system on the electronic device, for example, through the android standard interface CaptureResult.STATISTICS_LENS_SHADING_CORRECTION_MAP). After the electronic device obtains the lens shading parameters, it is necessary to multiply the off-screen image by the lens shading parameters in different channels to perform lens shading correction. Lens shading correction can reduce the brightness interference caused by lens shading. Furthermore, the electronic device uses the white balance parameters of the camera (which can be obtained through the android standard interface getWhitebalance()) to adjust the white balance of the off-screen image to ensure that there is no color cast in the off-screen image. Finally, the electronic device needs to perform display parameter (gamma) transformation on the image. For example, the range of the input off-screen image is [0,0.1], and gamma=1 / 2.4. After the gamma transformation, the range of the off-screen image becomes It can be seen that the gamma transformation can enlarge a smaller image range to a larger range, which is beneficial to network learning.

[0082] like Figure 6 As shown, the image restoration network may include a filter 31, a low-frequency restoration network 32, a high-frequency restoration network 33, and a fusion network 34. The filter 31 is used to obtain high-frequency image information or low-frequency image information of the target image; the low-frequency restoration network 32 is used to restore the low-frequency component image; the high-frequency restoration network 33 is used to restore the high-frequency component image; and the fusion network 34 is used to determine the restoration image according to the high-frequency restoration image and the low-frequency restoration image.

[0083] In some embodiments, after the electronic device inputs the image into the image restoration network, the input image is processed by a filter (such as a Gaussian filter) to obtain a low-frequency component image of the image, and the high-frequency component image is obtained by subtracting the low-frequency component image from the input image. Further, the low-frequency component image is input into the low-frequency restoration network for restoration to obtain a low-frequency restoration image. The high-frequency component image is input into the high-frequency restoration network for restoration to obtain a high-frequency restoration image. Figure 7As shown in the figure, the high-frequency repair network is composed of multiple convolutional layers and multiple residual dense blocks (RDB). The high-frequency component image passes through a convolutional layer and an activation layer with a stride of 1 to increase the number of channels of the image and increase nonlinearity. Then it passes through multiple (for example, three or more) residual dense blocks. The loss of high-frequency information caused by downsampling can be avoided by processing with residual dense blocks. At the same time, the combination of multiple residual dense blocks can improve the visual experience, and then repair the high-frequency details of the image through the surrounding pixel information. Then, the first convolutional layer result of the high-frequency repair network is summed with the results of multiple residual dense blocks through jump connections, turning the network into a regression task, which is conducive to network learning. Finally, it is input into a convolutional layer with a stride of 1 to adjust the number of channels of the network (that is, the number of channels of the image output by the high-frequency repair network) to be consistent with the output of the low-frequency repair network.

[0084] Figure 8 The structure of the residual dense block is shown in Figure 8 As shown in the figure, the residual dense block includes three 3*3 convolution and activation layers (in order to meet the real-time requirements, the ordinary convolution can be replaced by the depth-separable convolution). There is a jump connection between each convolution, which is connected by concat splicing to ensure that the shallow information is shared to the deep layer, and finally input into a 3*3 convolution layer to keep the number of input channels consistent.

[0085] like Fig. 9 As shown in , the low-frequency repair network consists of multiple convolutional layers, multiple encoder layers, and multiple decoders, such as Fig. 9 The U-shaped structure in . The low-frequency component image passes through a 3*3 convolution layer and an activation layer to increase the number of channels of the image and increase nonlinearity. Then it will pass through a structure of multiple encoder layers for encoding, and then decoded by the corresponding decoder layer. The encoding and decoding processes can refer to each other to improve the restoration effect of the low-frequency component image. Finally, a convolution layer with a step size of 1 is used to obtain a low-frequency restoration image. This structure is used to adjust the number of output network channels (that is, the number of channels of the image output by the low-frequency restoration network) to keep it consistent with the output of the high-frequency restoration network.

[0086] The above-mentioned U-shaped structure improves the visual experience of the image through continuous downsampling at the encoder layer, and fuses the downsampled information during the upsampling process to obtain higher-precision pixels, thereby restoring low-frequency information such as color and diffraction.

[0087] Fig.10 The structure of the encoder layer is shown in Figure 2. Fig.10 As shown in , the encoder layer includes a downsampling layer and a downsampling encoding layer, which are used to downsample the image and obtain the features of the feature map after downsampling. Fig.11As shown in , the low-frequency component image passes through two ordinary convolutions and activation layers with a step size of 2, and then the output results of the two activation layers are added (in order to meet the real-time requirements, the ordinary convolution can be replaced by a depth-separable convolution), and the added feature map is input into the downsampling coding layer. The downsampling coding layer is as follows Fig.12 As shown in the figure, the feature map passes through a normal convolution with a step size of 1 and an activation layer (used to increase nonlinearity), and then passes through a convolution layer. The input is added to the convolution layer result through a jump connection to increase the feature extraction capability. Finally, it passes through an activation layer to increase nonlinearity (in order to meet real-time requirements, the normal convolution can be replaced by a depth-separable convolution).

[0088] Fig.13 The structure of the decoder layer is shown in Figure 2. Fig.13 As shown in , the decoder layer includes an upsampling decoding layer and an upsampling layer, wherein the upsampling decoding layer is used for extracting deep network features, and the upsampling layer is used for upsampling feature maps. Fig.14 As shown in the figure, after a normal convolution with a step size of 1 and an activation layer (used to increase nonlinearity), another convolution layer is passed, and the input is added to the convolution layer result through a jump connection to increase the feature extraction capability, and finally an activation layer is passed to increase nonlinearity (in order to meet real-time requirements, the normal convolution can be replaced by a depth-separable convolution). The upsampling layer is shown in the figure. Fig.15 As shown in the figure, the upsampling layer undergoes a transposed convolution with a step size of 2 to obtain an upsampled feature map.

[0089] Combination Fig. 9 , between the encoder layer and the decoder layer, a skip connection layer is passed to keep the channels consistent. This layer passes through a convolution with a stride of 1 and an activation layer to obtain a skip connection result, which is used to fuse the encoder features to the decoder. For example, the output result of the first convolution layer of the low-frequency repair network is summed after a skip connection and the output of the U-shaped low-frequency repair network structure to fuse the input features. The output result of the encoding layer 2 is summed after a skip connection and the output result of the decoding layer 1 to fuse the information of the encoding layer 2. The output result of the encoding layer 1 is summed after a skip connection and the output result of the decoding layer 2 to fuse the information of the encoding layer 1. The low-frequency repair network finally passes through a convolution layer with a stride of 1 to obtain a low-frequency repair image. The convolution layer with a stride of 1 is used to adjust the number of output network channels to keep it consistent with the output of the high-frequency repair network.

[0090] In some embodiments, the output of the image restoration network can be input into Fig.16 The post-processing module shown in FIG. 1 includes inverse gamma transformation, inverse white balance, and inverse lens shading. The output of the image restoration network undergoes inverse gamma transformation, for example The input range is [0,0.383], the inverse gamma transform [0 2.4 , 0.3832.4 ]=[0,0.1], and get the inverse gamma transformation value. Then divide each channel of the inverse gamma transformation value by the corresponding white balance parameter to get the inverse white balance value. Finally, divide the inverse white balance value by the lens shading parameter of each channel to get the inverse lens shading correction value, so as to keep it consistent with the input, and then get the final repaired image.

[0091] In one design, in order to train an image restoration network, an electronic device can use a sample under-screen image as input and a target loss function as a constraint to train the image restoration network to be trained; wherein the target loss function is determined based on the sample non-under-screen image, the sample under-screen image, and the restored image output by the image restoration network.

[0092] During the training process, the electronic device can obtain paired under-screen and non-under-screen RAW images. For example, the operation and maintenance personnel place the mobile phone with the screen removed on a tripod, and the mobile phone obtains a high-definition image through High Dynamic Range Imaging (HDRI) projection. Then take multiple RAW images of the same scene with the screen removed, and superimpose and reduce noise on multiple frames of RAW images to obtain non-under-screen RAW images. Take multiple identical RAW images without removing the screen, and superimpose and reduce noise on multiple frames of RAW images to obtain under-screen RAW images.

[0093] In some embodiments, in order to maintain pixel alignment, the electronic device can perform registration processing on the under-screen and non-under-screen RAW images. Since the transmittance of each channel of the under-screen RAW image is inconsistent, the image appears color cast. When shooting a group of under-screen images, a color card can be shot at the same time, so as to perform white balance transformation on the under-screen RAW image in advance, so as to maintain color consistency with the non-under-screen RAW image. For the under-screen and non-under-screen RAW images after the white balance transformation, the ISP process (including black level compensation, lens shading correction, bad pixel correction, color interpolation (demosaic), noise removal, white balance (awb) correction, color correction (colorcorrection), gamma correction) can be executed respectively to obtain the under-screen RGB image and the non-under-screen RGB image to form an RGB data pair.

[0094] In one design, in order to determine the target loss function, for any sample pair (a sample under-screen image and a sample non-under-screen image of the same photographed object), the electronic device determines the diffraction area according to the sample non-under-screen image and the sample under-screen image in the sample pair. The electronic device determines the difference between the repaired image and the sample non-under-screen image in the diffraction area to obtain first difference information; the electronic device determines the difference between the repaired image and the sample non-under-screen image to obtain second difference information. The electronic device determines the target loss function based on the first difference information and the second difference information.

[0095] In some embodiments, the electronic device may grayscale the sample under-screen image to obtain a first grayscale image, and grayscale the sample non-under-screen image to obtain a second grayscale image. The electronic device determines the first area based on the first grayscale image and the second grayscale image. Similarly, the electronic device performs a first transformation on the second grayscale image to obtain a first image, and performs a second transformation on the first image to obtain a second image. The electronic device determines the second area based on the first image and the second image, and determines the area where the first area overlaps with the second area as the diffraction area.

[0096] The third grayscale image can be obtained by subtracting the second grayscale image from the first grayscale image. Further, the electronic device corrodes, dilates, and binarizes the third grayscale image to obtain an image corresponding to the first region. The electronic device subtracts the first image from the second image to obtain an image corresponding to the second region. The first transformation process includes binarization processing, and the second transformation process includes expansion processing.

[0097] For example, for the RAW image data pair, the electronic device performs lens shading correction on it. After the lens shading correction, the following operations can be performed: Fig.17The image mask acquisition process shown in FIG. 1 is a flowchart of an image mask acquisition process. For the R, G1, G2 and B channels of the under-screen image and the non-under-screen image, the electronic device performs the grayscale operation (R+(G1+G2) / 2+B) / 3 respectively to obtain the under-screen RAW single-channel image and the non-under-screen RAW single-channel image. The electronic device subtracts the non-under-screen RAW single-channel image from the under-screen RAW single-channel image to obtain the diffusion area mask 1, and further performs corrosion, expansion and binarization on the diffusion area mask to obtain the diffusion area mask 2. The corrosion operation refers to setting an corrosion window, such as a size of 3*3, and finding the minimum value of each (3*3) area in the image by sliding the window in the image as the value of the window. The corrosion operation can make the target area "smaller", which actually causes the boundary of the image to shrink, and can be used to eliminate small and meaningless targets. The expansion operation refers to setting an expansion window, such as a size of 3*3, and finding the maximum value of each (3*3) area in the image by sliding the window in the image as the value of the window. The dilation operation will make the target area "larger", which can be used to fill certain holes in the target area and eliminate small particle noise contained in the target area. Furthermore, the electronic device binarizes the non-off-screen RAW single-channel image (sets a threshold in advance, such as 512, and sets pixels greater than the threshold to 1, and pixels less than the threshold to 0). The binary image is dilated to obtain a dilated image. The electronic device subtracts the binary image from the dilated image and binarizes it to obtain regional mask 3, and intersects regional mask 2 and regional mask 3 to obtain the final diffusion regional mask 4.

[0098] Similarly, for the RGB image data pair, the electronic device can execute the above image mask acquisition process, and perform the (R+G+B) / 3 grayscale operation on the R, G, and B channels of the under-screen and non-under-screen images, respectively, to obtain the under-screen single-channel image and the non-under-screen single-channel image, and further perform Fig.17 The image mask acquisition process shown is used to obtain the diffusion mask 4.

[0099] The target loss function can be Mask_L1(Y,X)=2*|Y*mask-X*mask|+|YX|. Wherein, Y represents the restored image (i.e., the repaired image) generated by the network model, X represents the corresponding real image (i.e., the non-off-screen image), and mask represents the diffusion mask 4 obtained in the above steps. The electronic device takes the L1 norm of the difference between Y and X, multiplies Y and X by mask and then multiplies it by two to obtain the L1 norm, adds the two L1 norms, and uses them as the loss function of the network. Training is performed under the constraints of this loss function to obtain the parameters to be learned in the network model, such as weights, biases, etc. It should be understood that the loss function can also be replaced by other types, such as L2 loss.

[0100] It should be pointed out that the image restoration network trained by the above method has significant advantages over other image restoration networks. As shown in Table 1, the parameter quantity (Params) is in M, where M means one million; PSNR and SSIM represent the peak signal-to-noise ratio and structural similarity, respectively (the larger the value, the better). For the image restoration network of the embodiment of the present application, the PSNR and SSIM indicators are second only to those based on the U-type dynamic network (UDC-UNet), and are better than the U-type model network (Uformer) and the dynamic skip connection network (DISCNet). The parameter quantity has an order of magnitude advantage over the other three networks, and the network reasoning is faster, which is more suitable for mobile terminal deployment.

[0101] For ease of understanding, Fig.18 As shown, after the off-screen image input to the image restoration network is Gaussian filtered, the edge information is reduced to obtain a low-frequency component image. The off-screen image is subtracted from the low-frequency component image to obtain a high-frequency component image with rich edge information. The low-frequency component image is input to the low-frequency restoration network to obtain a low-frequency restoration image with color fidelity. The high-frequency component image is input to the high-frequency restoration network to obtain a high-frequency restoration image with richer details. Furthermore, the electronic device adds the high-frequency restoration image to the low-frequency restoration image to obtain a restoration image, and outputs the restoration image. Compared with the input off-screen image, the output restoration image has significantly improved clarity, reduced color cast problems, and suppressed glare.

[0102] like Fig.19 As shown, the image under the screen is grayed to obtain a grayed image, and the image under the screen is grayed to obtain a grayed image. The electronic device uses the grayed image under the screen to subtract the grayed image under the screen to obtain the region mask 1, and then corrodes, dilates, and binarizes the region mask 1 to obtain the region mask 2. The input non-under-screen image is binarized to obtain a binarized image, and then the binarized image of the non-under-screen image is dilated to obtain an expanded image. Then, the expanded image is subtracted from the non-under-screen binarized image, and then binarized to obtain the region mask 3. The electronic device intersects the region mask 2 image and the region mask 3 image to obtain the region mask 4 image.

[0103] In some embodiments, by comparing the training results using the diffraction region mask, it can be found that the training results without using the region mask are weaker than the training results using the region mask. Through testing, it can be found that the training results using the region mask have a better de-diffraction effect on the chest trademark and collar of the photographed subject.

[0104] The above-mentioned embodiment mainly introduces the scheme provided by the embodiment of the present application from the perspective of the device (equipment). It is understandable that, in order to implement the above-mentioned method, the device or equipment includes a hardware structure and / or software module corresponding to the execution of each method flow, and these hardware structures and / or software modules corresponding to the execution of each method flow can constitute an electronic device. It should be easy for those skilled in the art to realize that, in conjunction with the algorithm steps of each example described in the embodiment applied for herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but this implementation should not be considered to exceed the scope of the present application.

[0105] The embodiment of the present application can divide the functional modules of the device or equipment according to the above method example. For example, the device or equipment can divide each functional module corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0106] In the case of dividing each functional module into corresponding functional modules, Fig. 20 FIG. 3 is a schematic diagram showing the structure of an image processing device 30 of a functional module, wherein the image processing device 30 of the functional module can be used to execute the image processing method provided in the above embodiment. Fig. 20 As shown, the image processing device 30 includes an acquisition unit 301 , a processing unit 302 and a determination unit 303 .

[0107] The acquisition unit 301 is used to acquire high-frequency image information of the target image to obtain a high-frequency component image, and to acquire low-frequency image information of the target image to obtain a low-frequency component image; the processing unit 302 is used to perform repair processing on the high-frequency component image according to the high-frequency image information to obtain a high-frequency repair image, and to perform repair processing on the low-frequency component image according to the low-frequency image information to obtain a low-frequency repair image; the determination unit 303 is used to determine a repair image corresponding to the target image according to the high-frequency repair image and the low-frequency repair image.

[0108] In an exemplary embodiment, the image processing device 30 is deployed in an electronic device, which also includes a front camera and a display screen. The front camera is arranged below the display screen. The target image includes an under-screen image, which is an image captured by the front camera.

[0109] In an exemplary embodiment, the acquisition unit 301 can be used to: input the target image into a preset Gaussian filter to obtain the low-frequency image information, and subtract the low-frequency image information from the target image to obtain the high-frequency image information; or input the target image into a preset high-pass filter to obtain the high-frequency image information, and subtract the high-frequency image information from the target image to obtain the low-frequency image information.

[0110] In an exemplary implementation, the processing unit 302 may be configured to: reconstruct the high-frequency information to obtain high-frequency reconstruction information; and obtain the high-frequency restoration image based on the high-frequency reconstruction information.

[0111] In an exemplary implementation, the processing unit 302 may be configured to: repair and / or expand the high-frequency information according to pixels of the high-frequency component image to obtain the high-frequency reconstruction information.

[0112] In an exemplary implementation, the processing unit 302 may be configured to: reconstruct the low-frequency information to obtain low-frequency reconstruction information; and obtain the low-frequency restoration image based on the low-frequency reconstruction information.

[0113] In an exemplary implementation, the processing unit 302 may be configured to: repair and / or expand the low-frequency information according to pixels of the low-frequency component image to obtain the low-frequency reconstruction information.

[0114] In an exemplary implementation, the determining unit 303 may be configured to: superimpose the high-frequency restoration image and the low-frequency restoration image to obtain the restoration image.

[0115] In an exemplary embodiment, a trained image restoration network may be deployed on the electronic device, and the image restoration network includes a filter, a low-frequency restoration network, a high-frequency restoration network, and a fusion network; the filter is used to obtain high-frequency image information or low-frequency image information of the target image; the low-frequency restoration network is used to restore the low-frequency component image; the high-frequency restoration network is used to restore the high-frequency component image; the fusion network is used to determine the restoration image according to the high-frequency restoration image and the low-frequency restoration image; the processing unit 302 is also used to: input the target image into the trained image restoration network, and output the restoration image corresponding to the target image. In some exemplary embodiments, the image restoration network may also be deployed on other devices that can communicate with the electronic device. In an exemplary embodiment, the acquisition unit 301 is also used to: acquire multiple sample pairs; the processing unit 302 is also used to train the image restoration network to be trained based on the multiple sample pairs to obtain the trained image restoration network; a sample pair includes a sample screen image and a sample non-screen image of the same photographed object.

[0116] In an exemplary embodiment, the processing unit 302 can be used to: take the sample under-screen image as input and the target loss function as a constraint to train the image restoration network to be trained; the target loss function is determined based on the sample non-under-screen image, the sample under-screen image and the restored image output by the image restoration network.

[0117] In an exemplary embodiment, the determination unit 303 is also used to: for any sample pair, determine the diffraction area based on the sample non-under-screen image and the sample under-screen image in the sample pair; determine the difference between the restored image and the sample non-under-screen image in the diffraction area to obtain first difference information; determine the difference between the restored image and the sample non-under-screen image to obtain second difference information; and determine the target loss function based on the first difference information and the second difference information.

[0118] In an exemplary embodiment, the processing unit 302 may be used to: when the format of the target image is a raw data format RAW, perform image signal processing ISP on the target image and then input the processed image into the trained image restoration network.

[0119] In the case of implementing the functions of the above-mentioned integrated modules in the form of hardware, the embodiments of the present application provide a possible structure of the electronic device involved in the above-mentioned embodiments. Fig.21As shown, the electronic device 40 includes: a processor 402 and a bus 404. In an exemplary embodiment, the electronic device 40 may further include a memory 401; optionally, the electronic device 40 may further include a communication interface 403.

[0120] Processor 402 may be a processor that implements or executes various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present application. The processor 402 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the embodiments of the present application. The processor 402 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0121] The communication interface 403 is used to connect with other devices via a communication network, such as Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0122] The memory 401 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0123] As a possible implementation, the memory 401 may exist independently of the processor 402, and the memory 401 may be connected to the processor 402 via a bus 404 to store instructions or program codes. When the processor 402 calls and executes the instructions or program codes stored in the memory 401, the image processing method provided in the embodiment of the present application can be implemented.

[0124] In another possible implementation, the memory 401 may also be integrated with the processor 402 .

[0125] The bus 404 may be an extended industry standard architecture (EISA) bus, etc. The bus 404 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.21 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0126] Some embodiments of the present application provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium), which stores computer program instructions. When the computer program instructions are executed on a computer, the computer executes the image processing method described in any of the above embodiments.

[0127] Exemplarily, the above-mentioned computer-readable storage media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or magnetic tapes, etc.), optical disks (e.g., compact disks (CD), digital versatile disks (DVD), etc.), smart cards and flash memory devices (e.g., erasable programmable read-only memory (EPROM), cards, sticks or key drives, etc.). The various computer-readable storage media described in this application may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0128] An embodiment of the present application provides a computer program product comprising instructions. When the computer program product is run on a computer, the computer is enabled to execute the image processing method described in any one of the above embodiments.

[0129] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire high-frequency image information of the target image to obtain a high-frequency component image, and acquire low-frequency image information of the target image to obtain a low-frequency component image; According to the high-frequency image information, the high-frequency component image is repaired to obtain a high-frequency repaired image; and, according to the low-frequency image information, the low-frequency component image is repaired to obtain a low-frequency repaired image; A repaired image corresponding to the target image is determined according to the high-frequency repaired image and the low-frequency repaired image.

2. The method according to claim 1, characterized in that The method is applied to an electronic device, which includes a front camera and a display screen, wherein the front camera is arranged below the display screen, and the target image includes an under-screen image, which is an image captured by the front camera.

3. The method according to claim 1, characterized in that: The acquiring of high-frequency image information of the target image comprises inputting the target image into a preset high-pass filter to obtain the high-frequency image information, and the acquiring of low-frequency image information of the target image comprises subtracting the high-frequency image information from the target image to obtain the low-frequency image information; or The acquiring of low-frequency image information of the target image comprises inputting the target image into a preset Gaussian filter to obtain the low-frequency image information, and the acquiring of high-frequency image information of the target image comprises subtracting the low-frequency image information from the target image to obtain the high-frequency image information.

4. The method according to claim 1, characterized in that The step of performing restoration processing on the high-frequency component image to obtain a high-frequency restoration image according to the high-frequency image information includes: Acquire pixels of the high-frequency component image, and repair and / or expand the high-frequency information according to the pixels of the high-frequency component image to obtain the high-frequency reconstruction information; Based on the high-frequency reconstruction information, obtaining the high-frequency restoration image; The step of performing restoration processing on the low-frequency component image to obtain a low-frequency restoration image according to the low-frequency image information includes: Acquire pixels of the low-frequency component image, and repair and / or expand the low-frequency information according to the pixels of the low-frequency component image to obtain the low-frequency reconstruction information; Based on the low-frequency reconstruction information, the low-frequency restoration image is obtained.

5. The method according to claim 1, characterized in that: The determining, according to the high-frequency restoration image and the low-frequency restoration image, a restoration image corresponding to the target image comprises: The high-frequency restoration image is superimposed on the low-frequency restoration image to obtain the restoration image.

6. The method according to claim 1, characterized in that The electronic device is deployed with a trained image restoration network, wherein the image restoration network includes a filter, a low-frequency restoration network, a high-frequency restoration network, and a fusion network; the method further includes: The target image is input into the trained image restoration network, and a restoration image corresponding to the target image is output.

7. The method according to claim 6, characterized in that The high-frequency restoration network is used to restore the high-frequency component image, and the high-frequency restoration network includes a convolutional layer and at least one residual dense block; The low-frequency restoration network is used to restore the low-frequency component image, and the low-frequency restoration network includes a convolution layer, an activation layer, at least one encoder, and at least one decoder.

8. The method according to claim 6, characterized in that The method further comprises: Acquire multiple sample pairs, and based on the multiple sample pairs, train the image restoration network to be trained to obtain the trained image restoration network; a sample pair includes a sample under-screen image and a sample non-under-screen image of the same photographed object.

9. The method according to claim 8, characterized in that The training of the image restoration network to be trained based on the multiple sample pairs includes: The sample under-screen image is taken as input and the target loss function is taken as a constraint to train the image restoration network to be trained; the target loss function is determined based on the sample non-under-screen image, the sample under-screen image and the restored image output by the image restoration network.

10. The method according to claim 9, characterized in that The method further comprises: For any sample pair, determining the diffraction area according to the sample non-under-screen image and the sample under-screen image in the sample pair; Determine the difference between the repaired image and the sample non-under-screen image in the diffraction area to obtain first difference information; Determine a difference between the repaired image and the sample non-off-screen image to obtain second difference information; The target loss function is determined based on the first difference information and the second difference information.

11. The method according to claim 6, characterized in that The step of inputting the target image into the trained image restoration network and outputting a restoration image corresponding to the target image comprises: When the target image is in the raw data format RAW, the target image is subjected to image signal processing ISP and then input into the trained image restoration network.

12. An electronic device, characterized in that: include: a processor and a memory for storing instructions executable by the processor; The processor is configured to execute the instruction so that the electronic device performs the image processing method as described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is enabled to execute the image processing method according to any one of claims 1 to 11.

Images