Image processing method, image processing system, terminal and readable storage medium

Abstract

The application discloses an image processing method, an image processing device, a terminal and a computer readable storage medium. The image processing method comprises the following steps: acquiring an original image; acquiring a first initial image, a second initial image and a third initial image according to first color image pixels, second color image pixels and third color image pixels in the original image, the resolutions of the first initial image, the second initial image and the third initial image being all less than the resolution of the original image; respectively performing deconvolution processing on the first initial image, the second initial image and the third initial image according to a first point spread function, a second point spread function and a third point spread function to acquire a first intermediate image, a second intermediate image and a third intermediate image; and inputting the first intermediate image, the second intermediate image and the third intermediate image into a target network model for processing to obtain a clear target image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image technology, and in particular to an image processing method, an image processing device, a terminal and a computer readable storage medium. BACKGROUND

[0002] The imaging system of a mobile phone is composed of a lens, a filter and a sensor. The lens part is usually composed of multiple lenses combined together to achieve accurate light convergence, thereby obtaining a clear image. In the design process of a mobile phone lens, the performance of the lens at only one imaging distance is usually optimized, for example, the imaging quality at infinity is focused on in the design stage, and the imaging quality at a close distance is given a lower weight. Thus, when a close-up image is acquired, the field curvature of the image is very serious after the lens is refocused, thereby causing a large difference in the clarity of different positions in the image.

[0003] Generally, this problem can be improved by increasing the number of lenses in the imaging system to limit the angle of the chief ray angle (CRA). The full field of view CRA of a common single-lens reflex lens is almost 0 degrees, so it is easy to optimize the field curvature at different focusing distances. This problem can also be solved by a pure algorithm solution, that is, using a deconvolution algorithm to correct the aberration of the acquired image, thereby making the entire image clear.

[0004] However, if the number of lenses in the imaging system is increased to solve the field curvature problem of imaging, the volume and weight of the lens are very large. If a traditional algorithm is used to correct the aberration of the acquired image, the calculation amount is large. SUMMARY

[0005] The embodiments of the present application provide an image processing method, an image processing device, a terminal and a computer readable storage medium.

[0006] The image processing method of the embodiment of the present application comprises: acquiring an original image, the original image comprising first color image pixels, second color image pixels, and third color image pixels, the first color image pixels, the second color image pixels, and the third color image pixels having single color channel image data; acquiring a first initial image according to the first color image pixels in the original image, acquiring a second initial image according to the second color image pixels in the original image, and acquiring a third initial image according to the third color image pixels in the original image, the resolutions of the first initial image, the second initial image, and the third initial image being all less than the resolution of the original image; performing deconvolution processing on the first initial image according to a first point spread function to acquire a first intermediate image, performing deconvolution processing on the second initial image according to a second point spread function to acquire a second intermediate image, and performing deconvolution processing on the third initial image according to a third point spread function to acquire a third intermediate image, wherein the first point spread function, the second point spread function, and the third point spread function correspond to the first color channel, the second color channel, and the third color channel respectively; and inputting the first intermediate image, the second intermediate image, and the third intermediate image into a target network model for processing to obtain a clear target image.

[0007] The image processing device of the embodiment of the present application comprises an acquisition module, a first processing module, a deconvolution module, and a second processing module. The acquisition module is configured to acquire an original image, the original image comprising first color image pixels, second color image pixels, and third color image pixels, the first color image pixels, the second color image pixels, and the third color image pixels having single color channel image data. The first processing module is configured to acquire a first initial image according to the first color image pixels in the original image, acquire a second initial image according to the second color image pixels in the original image, and acquire a third initial image according to the third color image pixels, the resolutions of the first initial image, the second initial image, and the third initial image being all less than the resolution of the original image. The deconvolution module is configured to perform deconvolution processing on the first initial image according to a first point spread function to acquire a first intermediate image, perform deconvolution processing on the second initial image according to a second point spread function to acquire a second intermediate image, and perform deconvolution processing on the third initial image according to a third point spread function to acquire a third intermediate image, wherein the first point spread function, the second point spread function, and the third point spread function correspond to the first color channel, the second color channel, and the third color channel respectively. The second processing module is configured to input the first intermediate image, the second intermediate image, and the third intermediate image into a target network model for processing to obtain a clear target image.

[0008] A terminal of an embodiment of the present application includes one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the programs include a method for image processing. The method for image processing includes: obtaining an original image, the original image including first color image pixels, second color image pixels, and third color image pixels, the first color image pixels, the second color image pixels, and the third color image pixels having single color channel image data; obtaining a first initial image according to the first color image pixels in the original image, obtaining a second initial image according to the second color image pixels in the original image, and obtaining a third initial image according to the third color image pixels in the original image, the first initial image, the second initial image, and the third initial image each having a resolution smaller than that of the original image; performing deconvolution processing on the first initial image according to a first point spread function to obtain a first intermediate image, performing deconvolution processing on the second initial image according to a second point spread function to obtain a second intermediate image, and performing deconvolution processing on the third initial image according to a third point spread function to obtain a third intermediate image, wherein the first point spread function, the second point spread function, and the third point spread function correspond to the first color channel, the second color channel, and the third color channel, respectively; and inputting the first intermediate image, the second intermediate image, and the third intermediate image into a target network model for processing to obtain a clear target image.

[0009] The embodiment of the present application also provides a nonvolatile computer readable storage medium storing a computer program. When the computer program is executed by one or more processors, an image processing method is implemented. The image processing method comprises the following steps: acquiring an original image, wherein the original image comprises first color image pixels, second color image pixels and third color image pixels, and the first color image pixels, the second color image pixels and the third color image pixels have single color channel image data; acquiring a first initial image according to the first color image pixels in the original image, acquiring a second initial image according to the second color image pixels in the original image, and acquiring a third initial image according to the third color image pixels in the original image, wherein the resolutions of the first initial image, the second initial image and the third initial image are all smaller than the resolution of the original image; performing deconvolution processing on the first initial image according to a first point spread function to obtain a first intermediate image, performing deconvolution processing on the second initial image according to a second point spread function to obtain a second intermediate image, and performing deconvolution processing on the third initial image according to a third point spread function to obtain a third intermediate image, wherein the first point spread function, the second point spread function and the third point spread function correspond to the first color channel, the second color channel and the third color channel respectively; and inputting the first intermediate image, the second intermediate image and the third intermediate image into a target network model for processing to obtain a clear target image.

[0010] The image processing method, the image processing device, the terminal and the computer readable storage medium of the present application can reduce the calculation amount by performing deconvolution processing on the first initial image, the second initial image and the third initial image with smaller resolutions, thereby reducing the power consumption and improving the image processing speed. In addition, the first intermediate image, the second intermediate image and the third intermediate image processed by the tight deconvolution operation are input into the target network model for further processing, so that the artifacts and ringing effect of the target image can be reduced, thereby further improving the image quality of the finally obtained image.

[0011] Additional aspects and advantages of the embodiments of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and / or additional aspects and advantages of the present application can become apparent and be easily understood from the following description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present application.

[0013] Figure 1 FIG. 1 is a flowchart of an image processing method according to an embodiment of the present application;

[0014] Figure 2 is a structural schematic diagram of an image processing apparatus of some embodiments of the present application;

[0015] Figure 3 is a structural schematic diagram of a terminal of some embodiments of the present application;

[0016] Figure 4 is a structural schematic diagram of an imaging apparatus of some embodiments of the present application;

[0017] Figure 5 is a schematic diagram of acquiring a first initial image, a second initial image and a third initial image in an image processing method of some embodiments of the present application;

[0018] Figures 6 to 9 is a flow schematic diagram of an image processing method of some embodiments of the present application;

[0019] Figure 10 is a schematic diagram of sensitivities of three channels of an image sensor in an imaging apparatus of some embodiments of the present application;

[0020] Figure 11 is a schematic diagram of a first initial image and a first point spread function of different fields of view in an image processing method of some embodiments of the present application;

[0021] Figure 12 is a schematic diagram of acquiring a first initial image, a second initial image and a third initial image in an image processing method of some embodiments of the present application;

[0022] Figure 13 is a schematic diagram of acquiring a first intermediate image, a second intermediate image and a third intermediate image in an image processing method of some embodiments of the present application;

[0023] Figure 14 is a schematic diagram of acquiring a first initial image, a second initial image and a third initial image in an image processing method of some embodiments of the present application;

[0024] Figure 15 is a schematic diagram of acquiring a second intermediate image in an image processing method of some embodiments of the present application;

[0025] FIG. 16(a) and FIG. 16(b) are schematic diagrams of a second point spread function corresponding to a first image pixel and a second image pixel respectively under a certain field of view of some embodiments of the present application;

[0026] Figure 17 is a flow schematic diagram of an image processing method of some embodiments of the present application;

[0027] Figure 18is a schematic diagram of obtaining an output image in an image processing method of some embodiments of the present application;

[0028] Figure 19 is a schematic diagram of an image processing method of some embodiments of the present application;

[0029] Figures 20 to 22 is a schematic diagram of obtaining an output image in an image processing method of some embodiments of the present application;

[0030] Figure 23 is a schematic diagram of the interaction between a non-volatile computer readable storage medium and a processor in some embodiments of the present application. DETAILED DESCRIPTION

[0031] The embodiments of the present application are described in detail below with reference to the accompanying drawings, wherein the same or similar numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of the present application, and cannot be understood as limiting the embodiments of the present application.

[0032] Referring to Figure 1 , the embodiments of the present application provide an image processing method. The image processing method comprises:

[0033] 01: obtaining an original image, the original image comprising first color image pixels A, second color image pixels B, and third color image pixels C, the first color image pixels A, the second color image pixels B, and the third color image pixels C having single color channel image data;

[0034] 02: obtaining a first initial image according to the first color image pixels A in the original image, obtaining a second initial image according to the second color image pixels B in the original image, and obtaining a third initial image according to the third color image pixels C in the original image, the resolutions of the first initial image, the second initial image, and the third initial image being less than the resolution of the original image;

[0035] 03: performing deconvolution processing on the first initial image according to a first point spread function to obtain a first intermediate image, performing deconvolution processing on the second initial image according to a second point spread function to obtain a second intermediate image, and performing deconvolution processing on the third initial image according to a third point spread function to obtain a third intermediate image, wherein the first point spread function, the second point spread function, and the third point spread function correspond to the first color channel, the second color channel, and the third color channel, respectively;

[0036] 04: inputting the first intermediate image, the second intermediate image, and the third intermediate image into a target network model for processing to obtain a clear target image.

[0037] Please combine Figure 2 The embodiment of the present application provides an image processing device 100. The image processing device 100 comprises an acquisition module 10, a first processing module 20, a deconvolution module 30 and a second processing module 40. Wherein, the method in 01 can be realized by the acquisition module 10, the method in 02 can be realized by the first processing module 20, the method in 03 can be realized by the deconvolution module 30, and the method in 04 can be realized by the second processing module 40. That is, the acquisition module 10 is used for acquiring an original image, the original image comprising a first color image pixel A, a second color image pixel B and a third color image pixel C, the first color image pixel A, the second color image pixel B and the third color image pixel C having single color channel image data; the first processing module 20 is used for acquiring a first initial image according to the first color image pixel A in the original image, acquiring a second initial image according to the second color image pixel B in the original image, and acquiring a third initial image according to the third color image pixel C in the original image, the resolution of the first initial image, the second initial image and the third initial image being smaller than the resolution of the original image; the deconvolution module 30 is used for performing deconvolution processing on the first initial image according to a first point spread function to obtain a first intermediate image, performing deconvolution processing on the second initial image according to a second point spread function to obtain a second intermediate image, and performing deconvolution processing on the third initial image according to a third point spread function to obtain a third intermediate image, wherein the first point spread function, the second point spread function and the third point spread function correspond to the first color channel, the second color channel and the third color channel respectively; and the second processing module 40 is used for inputting the first intermediate image, the second intermediate image and the third intermediate image into a target network model for processing to obtain a clear target image.

[0038] Please combine Figure 3The embodiment of the present application also provides a terminal 1000. The terminal comprises one or more processors 200, a memory 300 and one or more programs. The one or more programs are stored in the memory 300 and the instructions of the one or more programs are executed by the one or more processors 200 to implement the image processing method of the embodiment of the present application. That is, the processor 200 can implement the method in 01, 02, 03 and 04, that is, the processor 200 is configured to acquire an original image, the original image comprising a first color image pixel A, a second color image pixel B and a third color image pixel C, the first color image pixel A, the second color image pixel B and the third color image pixel C having single color channel image data; acquire a first initial image according to the first color image pixel A in the original image, acquire a second initial image according to the second color image pixel B in the original image, and acquire a third initial image according to the third color image pixel C in the original image, the resolution of the first initial image, the second initial image and the third initial image being less than the resolution of the original image; perform deconvolution processing on the first initial image according to a first point spread function to obtain a first intermediate image, perform deconvolution processing on the second initial image according to a second point spread function to obtain a second intermediate image, and perform deconvolution processing on the third initial image according to a third point spread function to obtain a third intermediate image, wherein the first point spread function, the second point spread function and the third point spread function correspond to the first color channel, the second color channel and the third color channel respectively; and input the first intermediate image, the second intermediate image and the third intermediate image into a target network model for processing to obtain a clear target image.

[0039] The image processing method, the image processing device 100 and the terminal 1000 of the present application obtain a first initial image, a second initial image and a third initial image with a smaller resolution from an original image, and then perform deconvolution processing on the first initial image, the second initial image and the third initial image respectively according to a first point spread function, a second point spread function and a third point spread function to obtain a first intermediate image, a second intermediate image and a third intermediate image. Subsequently, the first intermediate image, the second intermediate image and the third intermediate image are input into a target network model for processing to obtain a clear target image. On the one hand, since the present application performs deconvolution processing on the first initial image, the second initial image and the third initial image with a smaller resolution, compared with performing deconvolution processing on the original image with a larger resolution, the calculation amount can be reduced to reduce power consumption and improve the speed of image processing. On the other hand, in the traditional aberration correction algorithm, the image artifacts and ringing effect after deconvolution processing are relatively serious, while in the present application, the first intermediate image, the second intermediate image and the third intermediate image after the deconvolution operation are input into the target network model for further processing, which can reduce the artifacts and ringing effect of the target image, thereby further improving the image quality of the finally obtained image.

[0040] Specifically, in some embodiments, the processor 200 (or the acquisition module 50) can control the pixel array in the image sensor 401 (as shown in Figure 4 ) of the imaging device 400 (as shown in Figure 4 ) to be exposed to obtain an original image. The original image is a RAW image. The original image includes first color image pixels A, second color image pixels B and third color image pixels C, and the first color image pixels A, the second color image pixels B and the third color image pixels C have single color channel image data. That is, the first color image pixels A have first color channel image data, the second color image pixels B have second color channel image data, and the third color image pixels C have third color image data.

[0041] It should be noted that the imaging device 400 can be arranged in the terminal 1000 or outside the terminal 1000, which is not limited herein. In addition, in some embodiments, the first color can be red, the second color can be green, and the third color can be blue; or in some embodiments, the first color can be red, the second color can be yellow, and the third color can be blue; or in some embodiments, the first color can be magenta, the second color can be cyan, and the third color can be yellow. In the embodiments of the present application, the first color is red, the second color is green, and the third color is blue.

[0042] After obtaining the original image, the processor 200 (or the first processing module 20) obtains a first initial image according to the first color image pixels A in the original image, a second initial image according to the second color image pixels B in the original image, and a third initial image according to the third color image pixels C, and the resolutions of the first initial image, the second initial image and the third initial image are all less than the resolution of the original image.

[0043] For example, the first initial image is obtained according to the first color image pixels A in the original image. Please refer to Figure 5 In some embodiments, all the first color image pixels A in the original image are extracted, and then the first initial image is obtained by rearranging the plurality of first color image pixels A according to their relative positions in the original image. For example, as Figure 5 described, the first color image pixel A arranged in the first row and the first column of the first initial image comes from the first color image pixel A arranged in the first row and the first column of the original image; the first color image pixel A arranged in the first row and the second column of the first initial image comes from the first color image pixel A arranged in the first row and the third column of the original image; and the first color image pixel A arranged in the second row and the first column of the first initial image comes from the first color image pixel A arranged in the second row and the first column of the original image.

[0044] Similarly, the specific implementation of obtaining the second initial image according to the second color image pixels B in the original image and the specific implementation of obtaining the third initial image according to the third color image pixels C in the original image are the same as the specific implementation of obtaining the first initial image according to the first color image pixels A in the original image, and will not be described here.

[0045] Please refer to Figure 6 In some embodiments, the image processing method further comprises:

[0046] 05: performing image preprocessing on the original image to obtain a preprocessed original image; the image preprocessing comprises at least one of black level correction, sensor bad point correction and lens vignetting correction.

[0047] At this time, 02: obtaining a first initial image according to the first color image pixels A in the original image, a second initial image according to the second color image pixels B in the original image, and a third initial image according to the third color image pixels C in the original image comprises:

[0048] 021: obtaining a first initial image according to the first color image pixels A in the preprocessed original image, a second initial image according to the second color image pixels B in the preprocessed original image, and a third initial image according to the third color image pixels C in the preprocessed original image.

[0049] Please combine Figure 2 In some embodiments, the image processing apparatus 100 further includes an image preprocessing module 50. The method in 05 can be implemented by the image preprocessing module 50, and the method in 021 can be implemented by the first processing module 20. That is, the image preprocessing module 50 can be used to perform image preprocessing on the original image to obtain a preprocessed original image; the image preprocessing includes at least one of black level correction, sensor bad pixel correction, and lens vignetting correction; the first processing module 20 can also be used to obtain a first initial image based on a first color image pixel A in the preprocessed original image, obtain a second initial image based on a second color image pixel B in the preprocessed original image, and obtain a third initial image based on a third color image pixel C in the preprocessed original image.

[0050] Please combine Figure 3 In some embodiments, the methods in 05 and 021 can also be implemented by the processor 200. That is, the processor 200 is further configured to perform image preprocessing on the original image to obtain a preprocessed original image; the image preprocessing includes at least one of black level correction, sensor bad pixel correction, and lens vignetting correction; and to obtain a first initial image based on a first color image pixel A in the preprocessed original image, a second initial image based on a second color image pixel B in the preprocessed original image, and a third initial image based on a third color image pixel C in the preprocessed original image.

[0051] For example, after obtaining the original image, the processor 200 (or image preprocessing module 50) can first perform image preprocessing on the original image to obtain a preprocessed original image. The image preprocessing includes at least one of black level correction, sensor dead pixel correction, and lens vignetting correction. Because the original image is preprocessed before performing other operations, compared to directly performing subsequent operations on the original image, this maximizes the preservation of the original effective information of the image, which is beneficial to improving the image quality of the final obtained image.

[0052] After obtaining the pre-processed original image, the processor 200 (or the first processing module 20) obtains a first initial image according to the first color image pixels A in the pre-processed original image, obtains a second initial image according to the second color image pixels B in the pre-processed original image, and obtains a third initial image according to the third color image pixels C in the pre-processed original image. The specific implementation is the same as that of obtaining the first initial image according to the first color image pixels A in the original image, obtaining the second initial image according to the second color image pixels B in the original image, and obtaining the third initial image according to the third color image pixels C in the original image in the above embodiment, and will not be repeated here.

[0053] After obtaining the first initial image, the second initial image, and the third initial image, the processor 200 (or the deconvolution module 30) performs deconvolution processing on the first initial image according to the first point spread function to obtain a first intermediate image, performs deconvolution processing on the second initial image according to the second point spread function to obtain a second intermediate image, and performs deconvolution processing on the third initial image according to the third point spread function to obtain a third intermediate image. The first point spread function, the second point spread function, and the third point spread function correspond to the first color channel, the second color channel, and the third color channel, respectively.

[0054] Specifically, referring to Figure 7 , the deconvolution processing on the first initial image according to the first point spread function to obtain a first intermediate image, the deconvolution processing on the second initial image according to the second point spread function to obtain a second intermediate image, and the deconvolution processing on the third initial image according to the third point spread function to obtain a third intermediate image include:

[0055] 031: obtaining the first point spread function, the second point spread function, and the third point spread function corresponding to different fields of view;

[0056] 032: dividing the first initial image into first blocks I1 corresponding to a plurality of different fields of view, and performing deconvolution processing on the first blocks I1 according to the first point spread function corresponding to the same field of view to obtain a plurality of processed first blocks I1; and splicing the plurality of processed first blocks I1 to obtain a first intermediate image;

[0057] 033: dividing the second initial image into second blocks I2 corresponding to a plurality of different fields of view, and performing deconvolution processing on the second blocks I2 according to the second point spread function corresponding to the same field of view to obtain a plurality of processed second blocks I2; and splicing the plurality of processed second blocks I2 to obtain a second intermediate image;

[0058] 034: segmenting the third initial image into third blocks I3 corresponding to the plurality of different fields of view, and deconvolving the third blocks I3 according to the third point spread functions corresponding to the same field of view to obtain a plurality of processed third blocks I3; and stitching the plurality of processed third blocks I3 to obtain the third intermediate image.

[0059] Please refer to Figure 2 In some embodiments, the methods in 031, 032, 033 and 034 can be implemented by the deconvolution module 30. That is, the deconvolution module 30 is configured to obtain the first point spread functions, the second point spread functions and the third point spread functions corresponding to different fields of view; segment the first initial image into first blocks I1 corresponding to the plurality of different fields of view, and deconvolve the first blocks I1 according to the first point spread functions corresponding to the same field of view to obtain a plurality of processed first blocks I1; and stitch the plurality of processed first blocks I1 to obtain the first intermediate image; segment the second initial image into second blocks I2 corresponding to the plurality of different fields of view, and deconvolve the second blocks I2 according to the second point spread functions corresponding to the same field of view to obtain a plurality of processed second blocks I2; and stitch the plurality of processed second blocks I2 to obtain the second intermediate image; segment the third initial image into third blocks I3 corresponding to the plurality of different fields of view, and deconvolve the third blocks I3 according to the third point spread functions corresponding to the same field of view to obtain a plurality of processed third blocks I3; and stitch the plurality of processed third blocks I3 to obtain the third intermediate image.

[0060] Please refer to Figure 3 In some embodiments, the methods in 031, 032, 033 and 034 can be implemented by the processor 200. That is, the processor 200 is configured to obtain the first point spread functions, the second point spread functions and the third point spread functions corresponding to different fields of view; segment the first initial image into first blocks I1 corresponding to the plurality of different fields of view, and deconvolve the first blocks I1 according to the first point spread functions corresponding to the same field of view to obtain a plurality of processed first blocks I1; and stitch the plurality of processed first blocks I1 to obtain the first intermediate image; segment the second initial image into second blocks I2 corresponding to the plurality of different fields of view, and deconvolve the second blocks I2 according to the second point spread functions corresponding to the same field of view to obtain a plurality of processed second blocks I2; and stitch the plurality of processed second blocks I2 to obtain the second intermediate image; segment the third initial image into third blocks I3 corresponding to the plurality of different fields of view, and deconvolve the third blocks I3 according to the third point spread functions corresponding to the same field of view to obtain a plurality of processed third blocks I3; and stitch the plurality of processed third blocks I3 to obtain the third intermediate image.

[0061] The point spread function (PSF) is an image obtained after an ideal point light source passes through the lens 402 and the image sensor 401 of the imaging device 400. For ease of understanding, the point spread function can be understood as an image obtained by the imaging device 400 acquiring an ideal point light source. It should be noted that in some embodiments, the corresponding point spread functions in the same field of view are different when the lens 402 of the imaging device 400 is in different focusing positions. Therefore, in some embodiments, the focusing position of the lens 402 of the imaging device 400 when the original image is acquired can be determined first, and then the first point spread function, the second point spread function, and the third point spread function corresponding to a plurality of different fields of view in the focusing position are acquired.

[0062] In addition, the image sensor 401 includes a pixel array (not shown in the figure), and the pixel array includes a plurality of pixels. Each pixel can convert the received light into an electrical signal to generate an original image when the pixel array is exposed. That is, each image pixel in the original image corresponds to each pixel in the pixel array. Since the first initial image, the second initial image, and the third initial image are all formed by re-arranging the original image, the actual interval between two adjacent image pixels changes. For example, assuming that the size of a pixel is 1 um, the image pixels in the original image correspond to the pixels in the pixel array, so the interval between the centers of two adjacent image pixels in the original image is 1 um. However, please refer to the following description Figure 5 The first color image pixel A arranged in the first row and the first column of the first initial image comes from the first row and the first column of the original image, and the first color image pixel A arranged in the first row and the second column of the first initial image comes from the first row and the third column of the original image. It can be understood that the interval between the centers of two adjacent image pixels in the first initial image is equivalent to the interval between the centers of the image pixel arranged in the first row and the first column of the original image and the image pixel arranged in the first row and the third column, that is, the interval between the centers of two adjacent image pixels in the first initial image is 2 um, so the first initial image should also be deconvoluted using the first point spread function with a sampling interval of 2 um, so as to obtain better image processing effect. Similarly, the sampling intervals of the second point spread function and the third point spread function also need to be the same as the actual interval between two adjacent image pixels in the second initial image and the third initial image, which will not be described here.

[0063] Please refer to Figure 8 In some embodiments, the image processing method is used for the imaging device 400. The imaging device 400 includes the lens 402 and the image sensor 401, and acquires the first point spread function, the second point spread function, and the third point spread function corresponding to different fields of view, which includes:

[0064] 0311: obtaining a plurality of original point spread functions corresponding to a plurality of different fields of view, and obtaining a plurality of first point spread functions, second point spread functions and third point spread functions corresponding to the plurality of different fields of view according to the plurality of original point spread functions; wherein the imaging device 400 obtains the original point spread functions generated by the point light source.

[0065] Please refer to Figure 2 In some embodiments, the method in 0311 can be implemented by the deconvolution module 30. That is, the deconvolution module 30 can also be used to obtain a plurality of original point spread functions corresponding to a plurality of different fields of view, and obtain a plurality of first point spread functions, second point spread functions and third point spread functions corresponding to the plurality of different fields of view according to the plurality of original point spread functions; wherein the imaging device 400 obtains the original point spread functions generated by the point light source.

[0066] Please refer to Figure 3 In some embodiments, the method in 0311 can be implemented by the processor 200. That is, the processor 200 can also be used to obtain a plurality of original point spread functions corresponding to a plurality of different fields of view, and obtain a plurality of first point spread functions, second point spread functions and third point spread functions corresponding to the plurality of different fields of view according to the plurality of original point spread functions; wherein the imaging device 400 obtains the original point spread functions generated by the point light source.

[0067] Specifically, in some embodiments, the imaging device 400 is used to directly obtain the point light source under different fields of view to obtain the original point spread functions corresponding to the different fields of view. Wherein the original point spread functions are similar to the original images mentioned above, that is, each image pixel in the original point spread function has image data corresponding to a single color channel. After obtaining the original point spread function under a certain field of view, the processor 200 (or the deconvolution module 30) arranges the points of the original point spread function to obtain the first point spread function, the second point spread function and the third point spread function corresponding to the field of view. Then, the original point spread function under the next field of view is obtained, and the above steps are repeatedly executed, so that a plurality of first point spread functions, second point spread functions and third point spread functions corresponding to different fields of view can be obtained. It should be noted that the specific implementation of obtaining the first point spread function, the second point spread function and the third point spread function from the original point spread function is the same as the specific implementation of obtaining the first initial image, the second initial image and the third initial image in the above-mentioned embodiments, and will not be repeated here.

[0068] Please refer to Figure 9 In some embodiments, obtaining a plurality of first point spread functions corresponding to a plurality of different fields of view comprises:

[0069] 0312: Based on the first parameter and the second parameter, obtain the first point spread function, the second point spread function and the third point spread function under different fields of view; wherein, the first parameter is used to characterize the optical design parameters of the lens, and the second parameter is used to characterize the sensitivity of the image sensor.

[0070] Please combine Figure 2 In some embodiments, the method in 0312 can be implemented by the deconvolution module 30. That is, the deconvolution module 30 is also used to obtain the first point spread function, the second point spread function, and the third point spread function under different fields of view based on the first parameter and the second parameter; wherein, the first parameter is used to characterize the optical design parameters of the lens, and the second parameter is used to characterize the sensitivity of the image sensor.

[0071] Please combine Figure 3 In some embodiments, the method in 0312 can be implemented by the processor 200. That is, the processor 200 is also used to obtain the first point spread function, the second point spread function, and the third point spread function under different fields of view according to the first parameter and the second parameter; wherein, the first parameter is used to characterize the optical design parameters of the lens, and the second parameter is used to characterize the sensitivity of the image sensor.

[0072] Specifically, the first parameter is used to characterize the optical design parameters of the lens 402, and the second parameter is used to characterize the sensitivity of the image sensor 401. In some embodiments, PSF data of different wavelengths under different fields of view can be directly derived based on the first parameter, i.e., the optical design parameters of the lens 402. Then, based on the second parameter, i.e., the sensitivity of the image sensor 401, the first point spread function, the second point spread function, and the third point spread function corresponding to the first color channel A, the second color channel B, and the third color channel C under different fields of view are fitted accordingly.

[0073] For example, in some embodiments, the PSF of different wavelengths under different fields of view can be directly derived by using ray tracing based on the optical design parameters of lens 402. Assume that we have 13*9 field of view positions, and we need to derive the PSF of wavelengths in the range of 400-700nm for each field of view. Among them, the range of 400-700nm is related to the cutoff wavelength of the infrared filter design in image sensor 401, and can be modified according to the actual situation, without limitation here. For each field of view position, the following processing is performed: (1) Obtain the sensitivity curves of the first color channel A, the second color channel B and the third color channel C of image sensor 401 (e.g. Figure 10 As shown, Figure 10For the three-channel sensitivity curve of the image sensor 401); (2) according to the sensitivity curve, the weight corresponding to each color channel at each wavelength position is obtained; (3) according to the weight of each color channel and the PSF of the current wavelength, the final point spread function corresponding to the color channel is obtained by weighted summation. For example, taking the acquisition of the second point spread function corresponding to the second color in a certain field of view as an example, a plurality of PSFs in the wavelength range of 400-700 nm in the field of view are derived according to the optical design parameters of the lens 402; then, according to the sensitivity curve corresponding to the second color channel B, the corresponding weight at each wavelength is obtained; the PSF and the weight corresponding to the same wavelength are multiplied, and the plurality of products are summed to obtain the second point spread function. Similarly, the above method can also be used to obtain the second point spread function corresponding to other fields of view, the first point spread function corresponding to the first color in a plurality of fields of view, and the third point spread function corresponding to the third color in a plurality of fields of view, which will not be described here.

[0074] It should be noted that in some embodiments, the first point spread function, the second point spread function and the third point spread function corresponding to a plurality of fields of view at different focusing positions can be obtained in advance before the terminal 1000 (or the image processing device 100) is shipped, and the plurality of first point spread functions, the second point spread functions and the third point spread functions are stored in the terminal 1000 (or the image processing device 100). When image processing is performed, it only needs to be directly called. Of course, in some embodiments, other ways can also be used to obtain the first point spread function, the second point spread function and the third point spread function, which are not limited here.

[0075] After obtaining a plurality of first point spread functions, second point spread functions and third point spread functions corresponding to different fields of view, the first initial image is divided into a plurality of first blocks I1 corresponding to different fields of view, and the first blocks I1 are deconvoluted according to the first point spread functions corresponding to the same field of view to obtain a plurality of processed first blocks I1; and the plurality of processed first blocks I1 are spliced to obtain a first intermediate image. The second initial image is divided into a plurality of second blocks I2 corresponding to different fields of view, and the second blocks I2 are deconvoluted according to the second point spread functions corresponding to the same field of view to obtain a plurality of processed second blocks I2; and the plurality of processed second blocks I2 are spliced to obtain a second intermediate image. The third initial image is divided into a plurality of third blocks I3 corresponding to different fields of view, and the third blocks I3 are deconvoluted according to the third point spread functions corresponding to the same field of view to obtain a plurality of processed third blocks I3; and the plurality of processed third blocks I3 are spliced to obtain a third intermediate image.

[0076] For example, taking the acquisition of the first intermediate image as an example, the first initial image is divided into first blocks I1 corresponding to a plurality of different fields of view, and the first blocks I1 are deconvoluted according to the first point spread functions corresponding to the same field of view to obtain a plurality of processed first blocks I1. As shown in Figure 11 Figure 11 The left side of the middle is a first point spread function diagram of different fields of view, Figure 11 The right side of the middle is a schematic diagram of the first initial image. Each cell in the first point spread function diagram of different fields of view corresponds to a field of view. Assuming that there are 13*9 fields of view, the processor 200 (or the deconvolution module 30) first divides the first initial image into 13*9 first blocks I1, which are divided into 13 blocks horizontally and 9 blocks vertically, and each first block I1 corresponds to a field of view. The first blocks I1 are deconvoluted according to the first point spread functions corresponding to the same field of view. For example, the field of view corresponding to the block I arranged in the first row and the first column of the first initial image corresponds to the field of view corresponding to the first point spread function arranged in the first row and the first column of the first point spread function diagram. The first block I1 arranged in the first row and the first column of the luminance image is deconvoluted according to the first point spread function arranged in the first row and the first column of the first point spread function diagram to obtain the processed first block I1. It should be noted that, in some embodiments, the deconvolution algorithm can use Wiener filtering, or the deconvolution algorithm can use Richard-lucy deconvolution, which is not limited herein. After deconvolution of the first blocks I1 corresponding to all fields of view, the first intermediate image can be obtained by splicing all the processed first blocks I1. Similarly, the second intermediate image and the third intermediate image can also be obtained by using the same method, which is not limited herein.

[0077] Please refer to Figure 12 In some embodiments, the original image includes a plurality of subunits U, each subunit U includes a first color image pixel A, a second color image pixel B and a third color image pixel C, and in the same subunit U, the number of first color image pixels A is the same as the number of third color image pixels C, and the number of second color image pixels B is greater than the number of first color image pixels A. At this time, the second initial image is acquired according to the second color image pixels B in the original image, including: extracting the second color image pixels B located at the same position in the plurality of subunits to arrange to obtain a plurality of frames of second initial images corresponding to different positions in the subunits. The second intermediate image is obtained by deconvoluting the second initial image according to the second point spread function, including: deconvoluting the plurality of frames of second initial images according to the second point spread function to obtain a plurality of frames of second intermediate images.

[0078] Please refer to Figure 2 ​In some embodiments, the first processing module 20 can also be configured to extract second color image pixels B located at the same position in the plurality of sub-units to arrange to obtain a plurality of frames of second initial images corresponding to different positions in the sub-units; and the deconvolution module 30 can also be configured to respectively deconvolve the plurality of frames of second initial images according to second point spread functions to obtain a plurality of frames of second intermediate images.

[0079] Please refer to Figure 3 In some embodiments, the processor 200 can also be configured to extract second color image pixels B located at the same position in the plurality of sub-units to arrange to obtain a plurality of frames of second initial images corresponding to different positions in the sub-units; and respectively deconvolve the plurality of frames of second initial images according to second point spread functions to obtain a plurality of frames of second intermediate images.

[0080] Since the number of first color image pixels A is the same as the number of third color image pixels C, and the number of second color image pixels B is greater than the number of first color image pixels A in the same sub-unit U, as shown in Figure 5 If all second color image pixels B are directly extracted to rearrange to generate a frame of second initial images, the resolution of the frame of second initial images will necessarily be different from the resolutions of the first initial images and the third initial images. However, as shown in Figure 12 In the present embodiment, when obtaining the second initial images according to the second color image pixels B in the original images, the second color image pixels B located at the same position in the plurality of sub-units are extracted to arrange to obtain a plurality of frames of second initial images corresponding to different positions in the sub-units, so that the resolutions of the plurality of frames of second initial images are the same as the resolutions of the first initial images and the third initial images, which can make the resolutions of the first intermediate images, the second intermediate images and the third intermediate images obtained after deconvolution the same, thereby facilitating subsequent other operations on the first intermediate images, the second intermediate images and the third intermediate images.

[0081] For example, please refer to Figure 12, the original image includes a plurality of sub-units U, each of the sub-units U includes one first color image pixel A, two second color image pixels B and one third color image pixel C. Wherein, in each of the sub-units U, one of the second color image pixels B is located at the upper right corner of the sub-unit U, and the other of the second color image pixels B is located at the lower left corner of the sub-unit U. When the second initial image is acquired according to the second color image pixels B in the original image, first, all of the second color image pixels B arranged at the upper right corner of the sub-unit U are extracted from the original image, and then the plurality of the second color image pixels B extracted are rearranged according to their relative positions in the original image, so that one of the second initial images E1 is obtained. Subsequently, all of the second color image pixels B arranged at the lower left corner of the sub-unit U are extracted from the original image, and then the plurality of the second color image pixels B extracted are rearranged according to their relative positions in the original image, so that another of the second initial images E2 is obtained. For example, as shown in Figure 12 , the second color image pixel B arranged at the first row and the first column of the second initial image E1 is from the second color image pixel B arranged at the first row and the second column of the original image; the second color image pixel B arranged at the first row and the second column of the second initial image E1 is from the second color image pixel B arranged at the first row and the fourth column of the original image; the second color image pixel B arranged at the second row and the first column of the second initial image E1 is from the second color image pixel B arranged at the third row and the second column of the original image. The second color image pixel B arranged at the first row and the first column of the second initial image E2 is from the second color image pixel B arranged at the second row and the first column of the original image; the second color image pixel B arranged at the first row and the second column of the second initial image E2 is from the second color image pixel B arranged at the second row and the third column of the original image; the second color image pixel B arranged at the second row and the first column of the second initial image E2 is from the second color image pixel B arranged at the fourth row and the first column of the original image.

[0082] When there are multiple second initial images, in some embodiments, the processor 200 (or the deconvolution module 30) respectively deconvolves the multiple second initial images according to the second point spread function to obtain multiple second intermediate images. As shown in Figure 13 , the second initial image E1 is deconvolved according to the second point spread function to obtain the second intermediate image M1; the second initial image E2 is deconvolved according to the second point spread function to obtain the second intermediate image M2.

[0083] Please refer to Figure 14 and Figure 15In some embodiments, the second color image includes the first image pixel B1 and the second image pixel B2 of the same color, the original image includes a plurality of subunits U, each subunit U includes a first color image pixel A, a second color image pixel B and a third color image pixel C. Each subunit U includes the first image pixel B1, the second image pixel B2, the first color image pixel A and the third color image pixel C. At this time, the second initial image is obtained according to the second color image pixel B in the original image, including: extracting all first image pixels B1 in the original image for arrangement to obtain a second initial image corresponding to the first image pixel B1, and extracting all second image pixels B2 in the original image for arrangement to obtain a second initial image corresponding to the second image pixel B2. The second intermediate image is obtained by deconvolution processing on the second initial image according to the second point spread function, including: deconvolution processing on the second initial image corresponding to the first image pixel B1 according to the second point spread function corresponding to the first image pixel B1 to obtain a second intermediate image corresponding to the first image pixel B1; and deconvolution processing on the second initial image corresponding to the second image pixel B2 according to the second point spread function corresponding to the second image pixel B2 to obtain a second intermediate image corresponding to the second image pixel B2; wherein the displacement between the center coordinates of the second point spread functions corresponding to the first image pixel B1 and the second image pixel B2 respectively is related to the displacement between the first image pixel B1 and the second image pixel B2 in the same subunit.

[0084] Please combine Figure 2 In some embodiments, the first processing module 20 can also be used to extract all first image pixels B1 in the original image for arrangement to obtain a second initial image corresponding to the first image pixel B1, and extract all second image pixels B2 in the original image for arrangement to obtain a second initial image corresponding to the second image pixel B2; the deconvolution module 30 can also be used to deconvolution processing on the second initial image corresponding to the first image pixel B1 according to the second point spread function corresponding to the first image pixel B1 to obtain a second intermediate image corresponding to the first image pixel B1; and deconvolution processing on the second initial image corresponding to the second image pixel B2 according to the second point spread function corresponding to the second image pixel B2 to obtain a second intermediate image corresponding to the second image pixel B2; wherein the displacement between the center coordinates of the second point spread functions corresponding to the first image pixel B1 and the second image pixel B2 respectively is related to the displacement between the first image pixel B1 and the second image pixel B2 in the same subunit.

[0085] Please combine Figure 3In some embodiments, the processor 200 can also be configured to extract all first image pixels B1 in the original image and arrange them to obtain a second initial image corresponding to the first image pixels B1, extract all second image pixels B2 in the original image and arrange them to obtain a second initial image corresponding to the second image pixels B2, perform deconvolution processing on the second initial image corresponding to the first image pixels B1 according to the second point spread function corresponding to the first image pixels B1 to obtain a second intermediate image corresponding to the first image pixels B1, and perform deconvolution processing on the second initial image corresponding to the second image pixels B2 according to the second point spread function corresponding to the second image pixels B2 to obtain a second intermediate image corresponding to the second image pixels B2, wherein the displacement between the center coordinates of the second point spread functions corresponding to the first image pixels B1 and the second image pixels B2 respectively is related to the displacement between the first image pixels B1 and the second image pixels B2 in the same subunit.

[0086] Although the first image pixels B1 and the second image pixels B2 both correspond to the second color, the positions of the first image pixels B1 and the second image pixels B2 in the subunit are different. When performing deconvolution operation on the second initial image, the second initial image corresponding to the first image pixels B1 is deconvolved using the second point spread function corresponding to the first image pixels B1, and the second initial image corresponding to the second image pixels B2 is deconvolved using the second point spread function corresponding to the second image pixels B2. Compared with using the same second point spread function to deconvolve the second initial image corresponding to the first image pixels B1 and the second initial image corresponding to the second image pixels B2, the PSF information of different regions of the image can be accurately obtained, and the deconvolution can be performed in detail according to the accurate PSF, so that the image quality of the finally obtained image can be further improved.

[0087] It should be noted that the displacement between the center coordinates of the second point spread functions corresponding to the first image pixels B1 and the second image pixels B2 respectively is related to the displacement between the first image pixels B1 and the second image pixels B2 in the same subunit. As shown in FIG. 16, FIG. 16(a) is a schematic diagram of the second point spread function corresponding to the first image pixels B1 in a certain field of view, and FIG. 16(b) is a schematic diagram of the second point spread function corresponding to the second image pixels B2 in the field of view. The shapes of the PSFs in FIG. 16(a) and FIG. 16(b) are exactly the same, and the center of the PSF in FIG. 16(a) is located at the upper right of the center of FIG. 16(b). In the same subunit of the original image, the first image pixels B1 are also located at the upper right of the second image pixels B2.

[0088] If the method described in 0311 above is used to directly obtain the second point diffusion function based on the origin diffusion function, in some embodiments, the sub-units of the origin diffusion function also include the first image pixel B1 and the second image pixel B2. After obtaining the origin diffusion function, the first image pixel B1 in the origin diffusion function can be directly sampled and arranged to obtain the second point diffusion function corresponding to the first image pixel B1; the second image pixel B2 in the origin diffusion function can be sampled and arranged to obtain the second point diffusion function corresponding to the second image pixel B2.

[0089] If the method described in 0312 above is used to obtain the second point diffusion function based on the first and second parameters, in some embodiments, the second point diffusion function obtained by the method described in 0312 can be used as the second point diffusion function corresponding to the first image pixel B1. Then, based on the distribution between the first image pixel B1 and the second image pixel B2 in the same sub-unit of the original image, the second point diffusion function corresponding to the first image pixel B1 is moved to obtain the second point diffusion function corresponding to the second image pixel B2. For example, as... Figure 14 As shown, within the same sub-unit of the original image, if the first image pixel B1 is moved down one image pixel and then left one image pixel, the position of the second image pixel B2 can be reached. Then, after obtaining the second point spread function corresponding to the first image pixel B1, this second point spread function can be moved down one image pixel and then left one image pixel to obtain the second point spread function corresponding to the second image pixel B2. Alternatively, the second point spread function obtained as described in 0312 can be used as the second point spread function corresponding to the second image pixel B2, and then the second point spread function corresponding to the second image pixel B2 can be moved according to the distribution between the first image pixel B1 and the second image pixel B2 within the same sub-unit of the original image to obtain the second point spread function corresponding to the first image pixel B1. No limitation is imposed here.

[0090] After obtaining the first intermediate image, the second intermediate image, and the third intermediate image, the processor 200 (or the second processing module 40) inputs the first intermediate image, the second intermediate image, and the third intermediate image into the target network model for processing to obtain a clear target image. It should be noted that if the number of second intermediate images includes multiple frames, then all second intermediate images, as well as the first and third intermediate images, need to be input into the target network model for processing.

[0091] Since the deconvolution is performed in blocks, when the multiple blocks (including the first block I1, the second block I2 and the third block I3, the same below, not repeated here) are spliced together, the edges of different blocks may have boundaries, and ringing effect may occur. Therefore, in the embodiment, the first intermediate image, the second intermediate image and the third intermediate image are input into the target network model for processing to obtain a clear target image, which can remove the artifacts at the edges of different blocks and avoid the ringing effect.

[0092] For example, referring to Figure 17 In some embodiments, the first intermediate image, the second intermediate image and the third intermediate image are input into the target network model for processing to obtain a clear target image, including:

[0093] 041: inputting the first intermediate image, the second intermediate image and the third intermediate image into the first target network model for processing to obtain an output image, each image pixel in the output image corresponding to a single color channel; and performing demosaicing on the output image to obtain a target image.

[0094] For example, referring to Figure 2 In some embodiments, the method in 041 can be implemented by the second processing module 40. That is, the second processing module 40 is configured to input the first intermediate image, the second intermediate image and the third intermediate image into the first target network model for processing to obtain an output image, each image pixel in the output image corresponding to a single color channel; and perform demosaicing on the output image to obtain a target image.

[0095] For example, referring to Figure 3 In some embodiments, the method in 041 can also be implemented by the processor 200. That is, the processor 200 is configured to input the first intermediate image, the second intermediate image and the third intermediate image into the first target network model for processing to obtain an output image, each image pixel in the output image corresponding to a single color channel; and perform demosaicing on the output image to obtain a target image.

[0096] For example, in some embodiments, the network model is trained in advance, so that only the first intermediate image, the second intermediate image and the third intermediate image need to be input into the trained first target network model. After the first intermediate image, the second intermediate image and the third intermediate image are subjected to pixel rearrangement, multiple convolution, pooling and deconvolution in the first target network model, a clear output image can be output, and each image pixel in the output image corresponds to a single color channel.

[0097] For example, as Figure 18 shown, Figure 18Fig. 1 is a schematic diagram of processing of the input of four frames of 256*256*1 intermediate images by the first target network model. In the figure, Pixel shuffle is pixel rearrangement, ResBlock is a residual block in a ResNet network (also known as a residual network), and conv is a vector convolution operation. It can be understood that the first, second, and third intermediate images input into the first target network model first reach the Pixel shuffle layer for pixel rearrangement to become a RAW image arranged according to a Bayer array, and then are processed by the ResNet network to obtain a clear output image. Of course, the first target network model can also have other structures, which are not limited here.

[0098] After obtaining the output image, the output image is processed to obtain a target image. Each image pixel in the target image has image data of a first color channel, a second color channel, and a third color channel.

[0099] Referring to Figure 19 In some embodiments, the first, second, and third intermediate images are input into the target network model for processing to obtain a clear target image, including:

[0100] 042: inputting the first, second, and third intermediate images into a second target network model for processing to obtain a target image, each image pixel in the target image having image data of a first color channel, a second color channel, and a third color channel; the second target network model being different from the first target network model.

[0101] Referring to Figure 2 In some embodiments, the method in 042 can be implemented by the second processing module 40. That is, the second processing module 40 can also be used to input the first, second, and third intermediate images into a second target network model for processing to obtain a target image, each image pixel in the target image having image data of a first color channel, a second color channel, and a third color channel; the second target network model being different from the first target network model.

[0102] Referring to Figure 3 In some embodiments, the method in 042 can also be implemented by the processor 200. The processor 200 can also be used to input the first, second, and third intermediate images into a second target network model for processing to obtain a target image, each image pixel in the target image having image data of a first color channel, a second color channel, and a third color channel; the second target network model being different from the first target network model.

[0103] For example, in some embodiments, the network model can also be designed to realize the function of image super-resolution, that is, after the first intermediate image, the second intermediate image and the third intermediate image are input into the second target network model, the image output by the network model does not need to be demosaiced, and the target image can be directly obtained.

[0104] For example, as shown in FIG. 6, Figure 20 Figure 20 FIG. 6 is a schematic diagram of the second target network model processing the input of four frames of 256*256*1 intermediate images. Among them, Pixel shuffle is a pixel rearrangement, ResBlock is a residual block in the ResNet network (also known as residual network), and conv is a vector convolution operation. It can be understood that the first intermediate image, the second intermediate image and the third intermediate image input into the second target network model are first processed by the ResNet network, then subjected to the Pixel shuffle layer for Pixel shuffle operation, and finally subjected to the conv layer for vector convolution operation to obtain a clear target image (it should be noted that Figure 20 A+B+C in the target image only represents that the image data of the image pixel has the first color channel A, the second color channel B and the third color channel C, and does not represent that the image data of the image pixel is A+B+C, and the same applies below, which will not be repeated here).

[0105] For example, as shown in FIG. 6, Figure 21 Figure 21 FIG. 6 is a schematic diagram of the second target network model processing the input of four frames of 256*256*1 intermediate images. Among them, ResBlock is a residual block in the ResNet network (also known as residual network), transconv is a deconvolution operation, and conv is a vector convolution operation. The first intermediate image, the second intermediate image and the third intermediate image input into the second target network model are first processed by the ResNet network, then subjected to the transconv layer for deconvolution, and finally subjected to the conv layer for vector convolution operation to obtain a clear target image.

[0106] For example, as shown in FIG. 6, Figure 22 Figure 22 ​​​A schematic diagram of processing of the input of the second target network model is a four-frame 256*256*1 intermediate image. Among them, ResBlock is a residual block in the ResNet network (also known as residual network), upsample is upsample, and conv is vector convolution operation. The first intermediate image, the second intermediate image, and the third intermediate image input into the second target network model are first processed by the ResNet network, then upsampled by the upsample layer, and finally vector-convoluted by the conv layer to obtain a clear target image. Of course, the second target network model can also be of other structures, which are not limited here.

[0107] Referring to Figure 23 The embodiments of the present application also provide a non-volatile computer readable storage medium 500 containing a computer program 501. When the computer program 501 is executed by one or more processors 200, the processor 200 executes the image processing method described in any one of the above embodiments. For example, when the computer program 501 is executed by one or more processors 200, the processor 200 executes

[0108] 01, 02, 03, 04, 05, 021, 031, 032, 033, 034, 0311, 0312, 041, 042, the image processing method.

[0109] For example, please refer to Figure 1 When the computer program 501 is executed by one or more processors 200, the processor 200 executes the following method:

[0110] 01: Obtain an original image, the original image including a first color image pixel A, a second color image pixel B, and a third color image pixel C, the first color image pixel A, the second color image pixel B, and the third color image pixel C having single color channel image data;

[0111] 02: Obtain a first initial image according to the first color image pixel A in the original image, a second initial image according to the second color image pixel B in the original image, and a third initial image according to the third color image pixel C in the original image, the resolution of the first initial image, the second initial image, and the third initial image being less than the resolution of the original image;

[0112] 03: deconvolution processing is performed on the first initial image according to a first point spread function to obtain a first intermediate image, deconvolution processing is performed on the second initial image according to a second point spread function to obtain a second intermediate image, and deconvolution processing is performed on the third initial image according to a third point spread function to obtain a third intermediate image, wherein the first point spread function, the second point spread function and the third point spread function correspond to the first color channel, the second color channel and the third color channel respectively;

[0113] 04: the first intermediate image, the second intermediate image and the third intermediate image are input into a target network model for processing to obtain a clear target image.

[0114] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the exemplary description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0115] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for performing specific logic functions or steps in the process. The various embodiments of the application can include additional or fewer steps or processes in addition to or other than those of the process described in this specification as a representative embodiment. The various embodiments of the application can include additional or fewer steps or processes in addition to or other than those of the process described in this specification as a representative embodiment.

[0116] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. An image processing method, characterized in that, include: Acquire an original image, the original image including a first color image pixel, a second color image pixel, and a third color image pixel, wherein the first color image pixel, the second color image pixel, and the third color image pixel have single color channel image data; A first initial image is obtained based on the first color image pixels in the original image, a second initial image is obtained based on the second color image pixels in the original image, and a third initial image is obtained based on the third color image pixels in the original image. The resolutions of the first initial image, the second initial image, and the third initial image are all smaller than the resolution of the original image. The first initial image is deconvolved according to the first point spread function to obtain the first intermediate image; the second initial image is deconvolved according to the second point spread function to obtain the second intermediate image; and the third initial image is deconvolved according to the third point spread function to obtain the third intermediate image, wherein the first point spread function, the second point spread function, and the third point spread function correspond to the first color channel, the second color channel, and the third color channel, respectively. The first intermediate image, the second intermediate image, and the third intermediate image are input into the target network model for processing to obtain a clear target image.

2. The image processing method according to claim 1, characterized in that, The original image comprises multiple sub-units, each sub-unit including pixels of the first color image, pixels of the second color image, and pixels of the third color image. Within the same sub-unit, the number of pixels of the first color image and the number of pixels of the third color image are the same, and the number of pixels of the second color image is greater than the number of pixels of the first color image. The step of obtaining the second initial image based on the second color image pixels in the original image includes: The second color image pixels located at the same position in multiple sub-units are extracted and arranged to obtain multiple frames of the second initial image corresponding to different positions in the sub-units; The step of deconvolving the second initial image according to the second point spread function to obtain the second intermediate image includes: The second initial image is deconvolved on multiple frames according to the second point diffusion function to obtain multiple intermediate frames of the second image.

3. The image processing method according to claim 1, characterized in that, The second color image pixel includes a first image pixel and a second image pixel of the same color. The original image includes multiple sub-units, each sub-unit including the first image pixel, the second image pixel, the first color image pixel, and the third color image pixel. The step of obtaining the second initial image based on the second color image pixels in the original image includes: All first image pixels in the original image are extracted and arranged to obtain a second initial image corresponding to the first image pixels; all second image pixels in the original image are extracted and arranged to obtain a second initial image corresponding to the second image pixels. The step of deconvolving the second initial image according to the second point spread function to obtain multiple frames of the second intermediate image includes: According to the second point spread function corresponding to the first image pixel, the second initial image corresponding to the first image pixel is deconvolved to obtain the second intermediate image corresponding to the first image pixel; and according to the second point spread function corresponding to the second image pixel, the second initial image corresponding to the second image pixel is deconvolved to obtain the second intermediate image corresponding to the second image pixel. The displacement between the center coordinates of the second point diffusion function corresponding to the first image pixel and the second image pixel is related to the displacement between the first image pixel and the second image pixel in the same sub-unit.

4. The image processing method according to claim 1, characterized in that, The process includes deconvolving the first initial image according to a first point spread function to obtain a first intermediate image, deconvolving the second initial image according to a second point spread function to obtain a second intermediate image, and deconvolving the third initial image according to a third point spread function to obtain a third intermediate image, comprising: Obtain the first point diffusion function, the second point diffusion function, and the third point diffusion function corresponding to different fields of view; The first initial image is segmented into first blocks corresponding to multiple different fields of view, and the first blocks are deconvolved according to the first point spread function corresponding to the same field of view to obtain multiple processed first blocks; and the multiple processed first blocks are stitched together to obtain a first intermediate image. The second initial image is segmented into second blocks corresponding to multiple different fields of view, and the second blocks are deconvolved according to the second point spread function corresponding to the same field of view to obtain multiple processed second blocks; and the multiple processed second blocks are stitched together to obtain a second intermediate image; The third initial image is segmented into third blocks corresponding to multiple different fields of view, and the third blocks are deconvolved according to the third point spread function corresponding to the same field of view to obtain multiple processed third blocks; and the multiple processed third blocks are stitched together to obtain a third intermediate image.

5. The image processing method according to claim 4, characterized in that, An imaging device, comprising a lens and an image sensor, wherein acquiring the first point spread function, the second point spread function, and the third point spread function corresponding to different fields of view includes: Obtain multiple original point spread functions corresponding to different fields of view, and obtain a first point spread function, a second point spread function, and a third point spread function based on the original point spread functions; wherein the imaging device generates the original point spread functions by capturing images of a point light source; or Based on the first parameter and the second parameter, the first point spread function, the second point spread function, and the third point spread function under different fields of view are obtained; wherein, the first parameter is used to characterize the optical design parameters of the lens, and the second parameter is used to characterize the sensitivity of the image sensor.

6. The image processing method according to claim 1, characterized in that, The step of inputting the first intermediate image, the second intermediate image, and the third intermediate image into the target network model for processing to obtain a clear target image includes: The first intermediate image, the second intermediate image, and the third intermediate image are input into a first target network model for processing to obtain an output image, wherein each pixel in the output image corresponds to a single color channel; and the output image is subjected to de-mosaic processing to obtain the target image; or The first intermediate image, the second intermediate image, and the third intermediate image are input into the second target network model for processing to obtain the target image. Each image pixel in the target image has image data of a first color channel, a second color channel, and a third color channel. The second target network model is different from the first target network model.

7. The image processing method according to claim 1, characterized in that, The image processing method further includes: The original image is preprocessed to obtain the preprocessed original image; the image preprocessing includes at least one of black level correction, sensor bad pixel correction and lens vignetting correction. The step of obtaining a first initial image based on the first color image pixels in the original image, obtaining a second initial image based on the second color image pixels in the original image, and obtaining a third initial image based on the third color image pixels in the original image includes: A first initial image is obtained based on the first color image pixels in the preprocessed original image, a second initial image is obtained based on the second color image pixels in the preprocessed original image, and a third initial image is obtained based on the third color image pixels in the preprocessed original image.

8. An image processing system, characterized in that, include: The acquisition module is used to acquire an original image, which includes a first color image pixel, a second color image pixel, and a third color image pixel, wherein the first color image pixel, the second color image pixel, and the third color image pixel have single color channel image data; A first processing module is configured to obtain a first initial image based on the first color image pixels in the original image, obtain a second initial image based on the second color image pixels in the original image, and obtain a third initial image based on the third color image pixels, wherein the resolution of the first initial image, the second initial image, and the third initial image is all smaller than the resolution of the original image; The deconvolution module is used to deconvolve the first initial image according to a first point spread function to obtain a first intermediate image, to deconvolve the second initial image according to a second point spread function to obtain a second intermediate image, and to deconvolve the third initial image according to a third point spread function to obtain a third intermediate image, wherein the first point spread function, the second point spread function, and the third point spread function correspond to the first color channel, the second color channel, and the third color channel, respectively. The second processing module is used to input the first intermediate image, the second intermediate image, and the third intermediate image into the target network model for processing to obtain a clear target image.

9. A terminal, characterized in that, include: Memory; One or more processors; and One or more programs, wherein the one or more programs are stored in the memory and executed by one or more processors, the programs comprising performing the image processing method according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium storing a computer program that, when executed by one or more processors, implements the image processing method according to any one of claims 1 to 7.

Images