Image processing method, image processing device, terminal and readable storage medium

Abstract

The application discloses an image processing method, an image processing device, a terminal and a readable storage medium. The image processing method comprises the following steps: performing chroma conversion on an original image to obtain a YUV initial image, each image pixel in the original image has image data of a first color channel, a second color channel and a third color channel; each image pixel in the YUV initial image has image data of a luminance channel, a first chroma channel and a second chroma channel; obtaining a luminance image, a first chroma image and a second chroma image corresponding to the luminance channel, the first chroma channel and the second chroma channel respectively according to the YUV initial image; performing deconvolution processing on the luminance image according to a first point spread function to obtain an intermediate image, the first point spread function corresponding to the luminance channel; inputting the intermediate image, the first chroma image and the second chroma image into a target neural network model for processing to obtain a clear YUV target image.

Description

Technical Field

[0001] This application relates to the field of imaging technology, and in particular to an image processing method, an image processing apparatus, a terminal, and a computer-readable storage medium. Background Technology

[0002] A mobile phone's imaging system consists of lenses, filters, and sensors. The lens component typically comprises multiple lenses combined to accurately converge light, thereby capturing a clear image. However, mobile phone lenses are often designed to optimize performance only at a single imaging distance. For example, the design phase might prioritize image quality at infinity, while neglecting image quality optimization at close range. Consequently, when capturing close-up scenes, the field curvature becomes severe after the lens refocuses, resulting in significant differences in sharpness across different parts of the image.

[0003] This problem can usually be improved by increasing the number of lenses in the imaging system to limit the Chief Ray Angle (CRA). The CRA of a common SLR lens is almost 0 degrees across the entire field of view, so it is easy to optimize the field curvature at different focusing distances. Alternatively, this problem can be solved by a pure algorithm, that is, by using a deconvolution algorithm to correct aberrations in the acquired image, thereby making the entire image clearer.

[0004] However, if the field curvature problem in imaging is solved by increasing the number of lenses in the imaging system, the size and weight of the lens will be very large; if traditional algorithms are used to correct aberrations in the acquired image, the computational load will be large. Summary of the Invention

[0005] This application provides an image processing method, an image processing apparatus, a terminal, and a computer-readable storage medium.

[0006] The image processing method of this application includes: performing chroma conversion on an original image to obtain a YUV initial image, wherein each image pixel in the original image has image data of a first color channel, a second color channel, and a third color channel; each image pixel in the YUV initial image has image data of a luminance channel, a first chroma channel, and a second chroma channel; obtaining a luminance image, a first chroma image, and a second chroma image corresponding to the luminance channel, the first chroma channel, and the second chroma channel, respectively, based on the YUV initial image; performing deconvolution processing on the luminance image according to a first point spread function to obtain an intermediate image, wherein the first point spread function corresponds to the luminance channel; and inputting the intermediate image, the first chroma image, and the second chroma image into a target neural network model for processing to obtain a clear YUV target image.

[0007] The image processing apparatus of this application includes a chroma conversion module, a first processing module, a deconvolution module, and a second processing module. The chroma conversion module performs chroma conversion on an original image to obtain a YUV initial image, wherein each pixel in the original image has image data for a first color channel, a second color channel, and a third color channel; each pixel in the YUV image has image data for a luminance channel, a first chroma channel, and a second chroma channel. The first processing module obtains a luminance image, a first chroma image, and a second chroma image corresponding to the luminance channel, the first chroma channel, and the second chroma channel, respectively, based on the YUV image. The deconvolution module performs deconvolution processing on the luminance image according to a first point spread function to obtain an intermediate image, wherein the first point spread function corresponds to the luminance channel. The second processing module inputs the intermediate image, the first chroma image, and the second chroma image into a target neural network model for processing to obtain a clear YUV target image.

[0008] The terminal of this application includes one or more processors, a memory, and one or more programs, wherein one or more programs are stored in the memory and executed by one or more processors, and the programs include methods for performing image processing. The image processing method includes: performing chroma conversion on an original image to obtain a YUV initial image, wherein each image pixel in the original image has image data of a first color channel, a second color channel, and a third color channel; each image pixel in the YUV initial image has image data of a luminance channel, a first chroma channel, and a second chroma channel; obtaining a luminance image, a first chroma image, and a second chroma image corresponding to the luminance channel, the first chroma channel, and the second chroma channel, respectively, based on the YUV initial image; performing deconvolution processing on the luminance image according to a first point spread function, wherein the first point spread function corresponds to the luminance channel; and inputting the intermediate image, the first chroma image, and the second chroma image into a target neural network model for processing to obtain a clear YUV target image.

[0009] This application also provides a non-volatile computer-readable storage medium storing a computer program. When the computer program is executed by one or more processors, it implements an image processing method. The image processing method includes: performing chroma conversion on an original image to obtain a YUV initial image, wherein each image pixel in the original image has image data of a first color channel, a second color channel, and a third color channel; each image pixel in the YUV initial image has image data of a luminance channel, a first chroma channel, and a second chroma channel; obtaining a luminance image, a first chroma image, and a second chroma image corresponding to the luminance channel, the first chroma channel, and the second chroma channel, respectively, based on the YUV initial image; performing deconvolution processing on the luminance image according to a first point spread function, wherein the first point spread function corresponds to the luminance channel; and inputting the intermediate image, the first chroma image, and the second chroma image into a target neural network model for processing to obtain a clear YUV target image.

[0010] The image processing method, image processing apparatus, terminal, and computer-readable storage medium of this application, on the one hand, reduce the computational load and increase the computational speed by performing deconvolution processing only on the luminance image corresponding to the luminance channel Y, compared to the traditional aberration correction algorithm which requires multiple deconvolution operations for different color channels, thereby improving the speed of image processing; on the other hand, by inputting the first chromaticity image, the second chromaticity image, and the intermediate image obtained after deconvolution processing into the target neural network model for further processing, artifacts and ringing effects of the target image can be reduced, thereby further improving the image quality of the final acquired image.

[0011] Additional aspects and advantages of the embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0012] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0013] Figure 1 This is a schematic flowchart of an image processing method according to certain embodiments of this application;

[0014] Figure 2 This is a schematic diagram of the structure of an image processing apparatus according to certain embodiments of this application;

[0015] Figure 3 This is a schematic diagram of the terminal structure according to some embodiments of this application;

[0016] Figure 4This is a schematic diagram of the imaging device according to some embodiments of this application;

[0017] Figure 5 This is a schematic diagram illustrating the principle of obtaining the original image in the image processing method of certain embodiments of this application;

[0018] Figure 6 This is a schematic diagram of the original image and the initial YUV image in the image processing method of certain embodiments of this application;

[0019] Figure 7 This is a schematic diagram illustrating the acquisition of a luminance image, a first chromaticity image, and a second chromaticity image from an initial YUV image in an image processing method according to certain embodiments of this application.

[0020] Figures 8 to 10 This is a schematic flowchart of an image processing method according to certain embodiments of this application;

[0021] Figure 11 This is a schematic diagram of the sensitivity of the three channels of the image sensor in an imaging apparatus according to certain embodiments of this application;

[0022] Figure 12 This is a schematic diagram of the brightness image and the first point spread function of different fields of view in the image processing method of certain embodiments of this application;

[0023] Figure 13 This is a schematic diagram illustrating the principle of acquiring YUV target images in the image processing method of certain embodiments of this application;

[0024] Figures 14 to 17 This is a schematic flowchart of an image processing method according to certain embodiments of this application;

[0025] Figure 18 This is a schematic diagram illustrating the image processing method of certain embodiments of this application for obtaining a first training image, a second training image, and a third training image from a training image;

[0026] Figure 19 This is a schematic diagram illustrating the image processing method of certain embodiments of this application for obtaining a first comparison image, a second comparison image, and a third comparison image based on a second sample image;

[0027] Figure 20 This is a schematic flowchart of an image processing method according to certain embodiments of this application;

[0028] Figure 21 This is a schematic diagram illustrating the interaction between a non-volatile computer-readable storage medium and a processor in certain embodiments of this application. Detailed Implementation

[0029] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0030] Please see Figure 1 This application provides an image processing method. The image processing method includes:

[0031] 01: Perform chroma conversion on the original image to obtain the initial YUV image, wherein each image pixel in the original image has image data of the first color channel A, the second color channel B, and the third color channel C; each image pixel in the initial YUV image has image data of the luminance channel Y, the first chroma channel U, and the second chroma channel V;

[0032] 02: Based on the initial YUV image, obtain the luminance image, the first chrominance image, and the second chrominance image corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively;

[0033] 03: Deconvolve the brightness image using the first-point spread function to obtain the intermediate image; the first-point spread function corresponds to the brightness channel Y; and

[0034] 04: Input the intermediate image, the first chroma image, and the second chroma image into the target neural network model for processing to obtain a clear YUV target image.

[0035] Please combine Figure 2This application provides an image processing apparatus 100. The image processing apparatus 100 includes a chroma conversion module 10, a first processing module 20, a deconvolution module 30, and a second processing module 40. The method described in 01 can be implemented by the chroma conversion module 10, the method described in 02 can be implemented by the first processing module 20, the method described in 03 can be implemented by the deconvolution module 30, and the method described in 04 can be implemented by the second processing module 40. Specifically, the chroma conversion module 10 performs chroma conversion on the original image to obtain a YUV initial image, wherein each image pixel in the original image has image data of a first color channel A, a second color channel B, and a third color channel C; and each image pixel in the YUV initial image has image data of a luminance channel Y, a first chroma channel U, and a second chroma channel V. The first processing module 20 obtains a luminance image, a first chroma image, and a second chroma image corresponding to the luminance channel Y, the first chroma channel U, and the second chroma channel V, respectively, based on the YUV initial image. The deconvolution module 30 is used to deconvolve the luminance image according to the first point spread function to obtain an intermediate image. The first point spread function corresponds to the luminance channel Y. The second processing module 40 is used to input the intermediate image, the first chroma image, and the second chroma image into the target neural network model for processing to obtain a clear YUV target image.

[0036] Please combine Figure 3 This application also provides a terminal 1000. The terminal 1000 includes one or more processors 200, a memory 300, and one or more programs. The one or more programs are stored in the memory 300 and are executed by the one or more processors 200 of the instructions for the image processing method of this application. In other words, processor 200 can implement the methods described in 01, 02, 03, and 04 above. Specifically, processor 200 performs chroma conversion on the original image to obtain a YUV initial image, wherein each image pixel in the original image has image data of a first color channel A, a second color channel B, and a third color channel C; each image pixel in the YUV initial image has image data of a luminance channel Y, a first chroma channel U, and a second chroma channel V; obtains a luminance image, a first chroma image, and a second chroma image corresponding to the luminance channel Y, the first chroma channel U, and the second chroma channel V, respectively, based on the YUV initial image; performs deconvolution processing on the luminance image according to a first point spread function to obtain an intermediate image, the first point spread function corresponding to the luminance channel Y; and inputs the intermediate image, the first chroma image, and the second chroma image into a target neural network model for processing to obtain a clear YUV target image.

[0037] The image processing method, image processing apparatus 100, and terminal 1000 of this application obtain a YUV initial image by performing chromaticity conversion on the original image, and obtain a luminance image, a first chromaticity image, and a second chromaticity image corresponding to the luminance channel Y, the first chromaticity channel U, and the second chromaticity channel V, respectively, based on the YUV initial image; subsequently, deconvolve the luminance image corresponding to the luminance channel Y and the first point spread function to obtain an intermediate image, and input the intermediate image, the first chromaticity image, and the second chromaticity image into a target neural network model for processing to obtain a clear YUV target image. On the one hand, since this application only performs deconvolution processing on the luminance image corresponding to the luminance channel Y, compared with the traditional aberration correction algorithm which requires multiple deconvolution operations for different color channels, it can reduce the amount of computation and increase the computation speed, thereby improving the speed of image processing. On the other hand, in the traditional aberration correction algorithm, the image obtained after deconvolution processing has serious artifacts and ringing effects. However, in this application, since the first chromaticity image, the second chromaticity image, and the intermediate image obtained after deconvolution processing are input into the target neural network model for further processing, the artifacts and ringing effects of the target image can be reduced, thereby further improving the image quality of the final acquired image.

[0038] Specifically, in some embodiments, the image processing method further includes acquiring the original image. See also... Figure 2 The image processing device 100 also includes an acquisition module 50, which can also be used to acquire the original image. Similarly, the processor 200 of the terminal 1000 can also be used to acquire the original image. For example, in some embodiments, the processor 200 (or the acquisition module 50) can control the imaging device 400 (e.g., Figure 4 Image sensor 401 (as shown) Figure 4 The pixel array shown is exposed to obtain a RAW image, which is then de-mosaiced to obtain the original image. In the RAW image, each pixel has only image data corresponding to a single color channel, while each pixel in the original image has image data for multiple color channels. It should be noted that the imaging device 400 can be located inside or outside the terminal 1000; this is not limited here.

[0039] For example, such as Figure 5As shown, the pixel array in the image sensor 401 includes multiple minimal repeating units, each of which includes a first color pixel, a second color pixel, and a third color pixel. Each pixel in the pixel array can only receive light of the wavelength corresponding to its color and converts the received light into an electrical signal. That is, the first color pixel can only receive light of the wavelength corresponding to the first color, the second color pixel can only receive light of the wavelength corresponding to the second color, and the third color pixel can only receive light of the wavelength corresponding to the third color. During pixel array exposure, each pixel in the pixel array converts the received light into an electrical signal to generate a RAW image. In the RAW image, the image pixel corresponding to the first color pixel of the pixel array has image data of the first color channel A; the image pixel corresponding to the second color pixel of the pixel array has image data of the second color channel B; and the image pixel corresponding to the third color pixel of the pixel array has image data of the third color channel C. For example, assuming the pixel arranged in the first row and first column of the pixel array is the first color pixel, then the image pixel arranged in the first column of the RAW image has image data of the first color channel A.

[0040] like Figure 5 As shown, after obtaining the RAW image, the processor 200 (or acquisition module 50) performs de-mosaic processing on the RAW image to obtain the original image. At this time, each image pixel in the original image has image data of the first color channel A, the second color channel B, and the third color channel C (it should be noted that...). Figure 5 In the original image, A+B+C only indicates that the image pixel simultaneously possesses the first color channel A, the second color channel B, and the third color channel C; it does not mean that the image pixel's image data is A+B+C. The same interpretation applies to Y+U+V in other images (and will not be elaborated further). Methods such as bilinear interpolation, gradient-based methods, and adaptive methods can be used to de-mosaic the RAW image; no restrictions are placed here.

[0041] It should be noted that in some embodiments, the first color can be red, the second color can be green, and the third color can be blue; or, in some embodiments, the first color can be red, the second color can be yellow, and the third color can be blue; or, in some embodiments, the first color can be magenta, the second color can be cyan, and the third color can be yellow. In the embodiments of this application, the first color is red, the second color is green, and the third color is blue as an example for illustration.

[0042] In some embodiments, before demosaicing the RAW image, image preprocessing may be performed on the RAW image, including at least one of black level correction, lens shading correction, and bad pixel compensation. This allows the original effective information of the RAW image to be included, thereby improving the quality of the final image.

[0043] After obtaining the original image, the processor 200 (or the chroma conversion module 10) performs chroma conversion on the original image to obtain the initial YUV image. For example, Figure 6 As shown, each image pixel in the initial YUV image has image data corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V.

[0044] Specifically, taking red as the first color, green as the second color, and blue as the third color as an example, the steps of the color conversion process can be to convert the image data of the red channel R, green channel G, and blue channel Bu of all image pixels in the image into the image data of the luminance channel Y, the first chromaticity channel U, and the second chromaticity channel V using the following formulas: (1) Y = 0.257*R + 0.504*G + 0.098*Bu + 16; (2) U = -0.148*R - 0.291*G + 0.439*Bu + 128; (3) V = 0.439*R - 0.368*G - 0.071*Bu + 128; thereby converting the original image belonging to the RGB domain into the initial YUV image belonging to the YUV domain.

[0045] Please refer to Figure 7 After obtaining the initial YUV image, the processor 200 (or the first processing module 20) acquires the luminance image, the first chrominance image, and the second chrominance image, respectively, corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V. It should be noted that all image pixels in the luminance image only have image data for the luminance channel Y, all image pixels in the first chrominance image only have image data for the first chrominance channel U, and all image pixels in the second chrominance image only have image data for the second chrominance channel V.

[0046] Specifically, taking the acquisition of a brightness image as an example, please refer to [link to relevant documentation]. Figure 7 In some embodiments, an image pixel is arbitrarily extracted from the initial YUV image, and the image data of the luminance channel Y of that image pixel is obtained. This image data is then used as the image data of the image pixels arranged at the same position in the luminance image. Subsequently, another image pixel is extracted from the initial YUV image, and the above steps are repeated until all image pixels in the initial YUV image have been extracted, thus obtaining the luminance image. Of course, in some embodiments, image pixels can also be extracted from the initial YUV image according to a certain rule, and this is not limited here.

[0047] Similarly, the same method is used to obtain the first chromaticity image and the second chromaticity image. For example, the image data of the image pixels arranged in the first row and first column of the luminance image is the same as the image data of the luminance channel Y of the image pixels arranged in the first row and first column of the initial YUV image; the image data of the image pixels arranged in the first row and first column of the first chromaticity image is the same as the image data of the first chromaticity channel U of the image pixels arranged in the first row and first column of the initial YUV image; and the image data of the image pixels arranged in the first row and first column of the second chromaticity image is the same as the image data of the second chromaticity channel V of the image pixels arranged in the first row and first column of the initial YUV image.

[0048] After acquiring the luminance image, the first chroma image, and the second chroma image, the processor 200 (or the deconvolution module 30) performs deconvolution processing on the luminance image according to the first point spread function to obtain an intermediate image, wherein the first point spread function corresponds to the luminance channel Y.

[0049] When the lens 402 of the imaging device 400 is focused at infinity, the acquired image is clear and does not require algorithmic recovery. However, when the lens 402 of the imaging device 400 focuses on a close-up scene, severe edge curvature results in a blurry image. Since the image data corresponding to the luminance channel Y mainly consists of luminance information, the blurring caused by field curvature primarily affects the high-frequency components of the luminance channel Y. In this embodiment, deconvolution is performed on the luminance image corresponding to the luminance channel Y. Compared to performing multiple deconvolution operations on the images corresponding to all channels, this approach not only solves the field curvature problem but also reduces computational complexity and increases computational speed.

[0050] Please see Figure 8 In some embodiments, the brightness image is deconvolved according to a first point spread function to obtain an intermediate image, wherein the first point spread function corresponds to the brightness channel Y, and the method further includes:

[0051] 031: Obtain multiple first-point spread functions corresponding to different fields of view;

[0052] 032: The brightness image is segmented into blocks I corresponding to multiple different fields of view, and each block I under the corresponding field of view is deconvolved according to multiple first-point spread functions to obtain multiple processed blocks I; and

[0053] 033: Stitch together multiple processed blocks I to obtain an intermediate image.

[0054] Please combine Figure 2In some embodiments, the methods in 031, 032, and 033 can be implemented by the deconvolution module 30. That is, the deconvolution module 30 is also used to obtain multiple first point spread functions corresponding to different fields of view; to segment the brightness image into blocks I corresponding to multiple different fields of view, and to perform deconvolution processing on the blocks I under the corresponding fields of view according to the multiple first point spread functions to obtain multiple processed blocks I; and to stitch the multiple processed blocks I together to obtain an intermediate image.

[0055] Please combine Figure 3 In some embodiments, the methods in 031, 032, and 033 can also be implemented by the processor 200. That is, the processor 200 is further configured to obtain multiple first point spread functions corresponding to different fields of view; segment the brightness image into blocks I corresponding to multiple different fields of view, and perform deconvolution processing on the blocks I under the corresponding fields of view according to the multiple first point spread functions to obtain multiple processed blocks I; and stitch the multiple processed blocks I together to obtain an intermediate image.

[0056] Field curvature introduces a non-uniform blur, where the blurriness gradually increases from the center to the edges of the image. Traditional deconvolution algorithms (e.g., Wiener deconvolution) struggle to handle images with non-uniform blur. However, in this embodiment, the brightness image is segmented into blocks I corresponding to multiple different fields of view, and deconvolution is performed on each block I within its corresponding field of view using multiple first-point spread functions. This effectively solves the problem of inconsistent blur levels across the entire image caused by field curvature.

[0057] The point spread function (PSF) is the image obtained after an ideal point light source passes through the lens 402 and image sensor 401 of the imaging device 400. For ease of understanding, the PSF can be understood as a frame of the image obtained by the imaging device 400 from an ideal point light source. It should be noted that in some embodiments, the PSF for the same field of view differs depending on the focus position of the lens 402 of the imaging device 400. Therefore, in some embodiments, the focus position of the lens 402 of the imaging device 400 when acquiring the original image can be determined first, and then the first PSF corresponding to multiple different fields of view at that focus position can be obtained.

[0058] Specifically, please refer to Figure 9 In some embodiments, the image processing method is used in the imaging device 400. The imaging device 400 includes a lens 402 and an image sensor 401, acquiring multiple first point spread functions corresponding to different fields of view, including:

[0059] 0311: Obtain multiple original point spread functions corresponding to different fields of view, and obtain multiple first point spread functions corresponding to different fields of view based on the multiple original point spread functions; wherein, the imaging device 400 obtains the original point spread function generated by the point light source.

[0060] Please combine Figure 2 In some embodiments, the method in 0311 can be implemented by the deconvolution module 30. That is, the deconvolution module 30 can also be used to obtain multiple original point spread functions corresponding to different fields of view, and to obtain multiple first point spread functions corresponding to different fields of view based on the multiple original point spread functions; wherein, the imaging device 400 obtains the original point spread function generated by the point light source.

[0061] Please combine Figure 3 In some embodiments, the method in 0311 can also be implemented by the processor 200. That is, the processor 200 can also be used to obtain multiple original point spread functions corresponding to different fields of view, and to obtain multiple first point spread functions corresponding to different fields of view based on the multiple original point spread functions; wherein, the imaging device 400 obtains the original point spread function generated by the point light source.

[0062] Specifically, in some embodiments, the imaging device 400 directly acquires point light sources under different fields of view to obtain the original point spread function corresponding to each field of view. The original point spread function is similar to the RAW image mentioned above; that is, multiple image pixels in the original point spread function each have image data corresponding to a single color channel. After obtaining the original point spread function for a certain field of view, the processor 200 (or the deconvolution module 30) performs de-mosaicing and chroma conversion processing on the original point spread function to obtain the first point spread function corresponding to that field of view. Subsequently, the original point spread function for the next field of view is acquired, and the above steps are repeated, thus obtaining multiple first point spread functions corresponding to different fields of view. It should be noted that the specific implementation method for obtaining the first point spread function based on the original point spread function is the same as the specific implementation method for obtaining the brightness image from the RAW image in the above embodiments, and will not be described again here.

[0063] Please see Figure 10 In some embodiments, multiple first point spread functions corresponding to different fields of view are obtained, including:

[0064] 0312: Based on the first parameter and the second parameter, obtain the second point spread function, the third point spread function and the fourth point spread function corresponding to the first color channel A, the second color channel B and the third color channel C respectively under different fields of view, and obtain the first point spread function under the corresponding field of view based on the second point spread function, the third point spread function and the fourth point spread function under the same field of view; wherein, the first parameter is used to characterize the optical design parameters of the lens 402, and the second parameter is used to characterize the sensitivity of the image sensor 401.

[0065] Please combine Figure 2 In some embodiments, the method in 0312 can be implemented by the deconvolution module 30. That is, the deconvolution module 30 is also used to obtain the second point spread function, the third point spread function, and the fourth point spread function corresponding to the first color channel A, the second color channel B, and the third color channel C respectively under different fields of view according to the first parameter and the second parameter, and to obtain the first point spread function under the corresponding field of view according to the second point spread function, the third point spread function, and the fourth point spread function under the same field of view; wherein, the first parameter is used to characterize the optical design parameters of the lens 402, and the second parameter is used to characterize the sensitivity of the image sensor 401.

[0066] Please combine Figure 3 In some embodiments, the method in 0312 can also be implemented by the processor 200. That is, the processor 200 is also used to obtain, according to the first parameter and the second parameter, the second point spread function, the third point spread function and the fourth point spread function corresponding to the first color channel A, the second color channel B and the third color channel C respectively under different fields of view, and to obtain the first point spread function under the corresponding field of view according to the second point spread function, the third point spread function and the fourth point spread function under the same field of view; wherein, the first parameter is used to characterize the optical design parameters of the lens 402, and the second parameter is used to characterize the sensitivity of the image sensor 401.

[0067] Specifically, the first parameter is used to characterize the optical design parameters of the lens 402, and the second parameter is used to characterize the sensitivity of the image sensor 401. In some embodiments, PSF data of different wavelengths under different fields of view can be directly derived based on the first parameter, i.e., the optical design parameters of the lens 402. Then, based on the second parameter, i.e., the sensitivity of the image sensor 401, the second point spread function, the third point spread function, and the fourth point spread function corresponding to the first color channel A, the second color channel B, and the third color channel C under different fields of view are fitted accordingly.

[0068] For example, in some embodiments, the PSF of different wavelengths under different fields of view can be directly derived by using ray tracing based on the optical design parameters of lens 402. Assume that we have 13*9 field of view positions, and we need to derive the PSF of wavelengths in the range of 400-700nm for each field of view. Among them, the range of 400-700nm is related to the cutoff wavelength of the infrared filter design in image sensor 401, and can be modified according to the actual situation, without limitation here. For each field of view position, the following processing is performed: (1) Obtain the sensitivity curves of the first color channel A, the second color channel B and the third color channel C of image sensor 401 (e.g. Figure 11 As shown, Figure 11 (1) Sensitivity curves of the three channels of image sensor 401; (2) Obtain the weights of the three color channels at each wavelength position according to the sensitivity curves; (3) Obtain the final point spread function corresponding to the color channel by weighted summation according to the weight of each color channel and the PSF of the current wavelength. For example, to obtain the third point spread function corresponding to the second color in a certain field of view, multiple PSFs with wavelength ranges between 400-700nm are derived by ray tracing according to the optical design parameters of lens 402; then, according to the sensitivity curve corresponding to the second color channel B, the weights corresponding to multiple wavelengths are obtained; the PSFs and weights corresponding to the same wavelength are multiplied, and the multiple products are summed to obtain the third point spread function. Similarly, the above method can also be used to obtain the third point spread function corresponding to other fields of view, the second point spread function corresponding to the first color in multiple fields of view, and the fourth point spread function corresponding to the third color in multiple fields of view, which will not be elaborated here.

[0069] After obtaining the second, third, and fourth point spread functions, the processor 200 (or deconvolution module 30) can obtain the first point spread function for the corresponding field of view based on the second, third, and fourth point spread functions for the same field of view. For example, in some embodiments, when the first color is red, the second color is green, and the third color is blue, the first point spread function can be calculated using the formula Y. psf =0.257*R psf +0.504*G psf +0.098*B psf +16 is obtained, where Y psf In the first point diffusion function, R psf Represents the second-point diffusion function, G psf Represents the third-point diffusion function, B psf The fourth point shows the diffusion function.

[0070] It should be noted that, in some embodiments, the first point spread function corresponding to multiple fields of view at different focus positions can be pre-acquired before the terminal 1000 (or image processing device 100) leaves the factory, and the multiple first point spread functions can be stored in the terminal 1000 (or image processing device 100). During image processing, they can simply be directly called. Of course, in some embodiments, other methods can also be used to obtain the first point spread function corresponding to the brightness channel Y, and this is not limited here.

[0071] After obtaining multiple first point spread functions corresponding to different fields of view, the processor 200 (or deconvolution module 30) segments the brightness image into blocks I corresponding to multiple different fields of view, and performs deconvolution processing on the blocks I under the corresponding fields of view according to the multiple first point spread functions to obtain multiple processed blocks I.

[0072] For example, such as Figure 12 As shown, Figure 12 The left-hand image shows a schematic diagram of the first-point spread function in different fields of view. Figure 12 The right-hand side of the diagram shows a schematic of the brightness image. Each cell in the first point spread function (FFD) diagram for different fields of view corresponds to a field of view. Assuming there are 13*9 fields of view, the processor 200 (or deconvolution module 30) first divides the brightness image into 13*9 blocks I, 13 horizontally and 9 vertically, with each block I corresponding to a field of view. Then, it processes each block I under its corresponding field of view according to multiple first point spread functions. For example, the field of view corresponding to block I in the first row and first column of the brightness image corresponds to the field of view corresponding to the first point spread function in the first row and first column of the FFD diagram. Based on the first point spread function in the first row and first column of the FFD diagram, the block I in the first row and first column of the brightness image is deconvolved to obtain the processed block I. It should be noted that in some embodiments, the deconvolution algorithm can use Wiener filtering; or, the deconvolution algorithm can also use Richard-Lucy deconvolution, which is not limited here.

[0073] Please see Figure 13 After deconvolution processing of all blocks I corresponding to all fields of view, the intermediate image is obtained by stitching all processed blocks I together. After obtaining the intermediate image, the processor 200 (or the second processing module 40) inputs the intermediate image, the first chroma image, and the second chroma image into the target neural network model for processing to obtain a clear YUV target image.

[0074] Because block-based deconvolution is used, when multiple blocks I are stitched together, there may be boundaries at the edges of the stitched blocks I, and ringing effects may occur. Therefore, in this embodiment, the intermediate image, the first chroma image, and the second chroma image are input into the target neural network model for processing to obtain a clear YUV target image, which can remove artifacts at the edges of different blocks I and avoid ringing effects.

[0075] It should be noted that in some embodiments, the neural network model is pre-trained so that only an intermediate image, a first chroma image, and a second chroma image are input into the trained neural network model, i.e., the target neural network model. The intermediate image, the first chroma image, and the second chroma image undergo multiple convolution, pooling, and deconvolution operations in the target neural network model, which can output a clear YUV target image. Each pixel in the YUV target image has image data for the luminance channel Y, the first chroma channel U, and the second chroma channel V.

[0076] After obtaining the YUV target image, it can be processed again to further improve the quality of the final image. For an example, please refer to [link to example image]. Figure 14 In some embodiments, the image processing method further includes:

[0077] 05: Perform image post-processing on the YUV target image to obtain the processed YUV target image. Image post-processing includes at least one of noise reduction processing and sharpening processing.

[0078] Please combine Figure 2 In some embodiments, the image processing apparatus 100 further includes an image post-processing module 60, and the method described in 05 can be implemented by the image post-processing module 60. That is, the image post-processing module 60 can also be used to perform image post-processing on the YUV target image to obtain a processed YUV target image. The image post-processing includes at least one of noise reduction processing and sharpening processing.

[0079] Please combine Figure 3 In some embodiments, the method in 05 can also be implemented by the processor 200. That is, the processor 200 can also be used to perform image post-processing on the YUV target image to obtain a processed YUV target image, and the image post-processing includes at least one of noise reduction processing and sharpening processing.

[0080] After obtaining the YUV target image, the processor 200 (or image post-processing module 60) performs image post-processing on the YUV target image to obtain a processed YUV target image. The image post-processing includes at least one of noise reduction and sharpening. Since noise reduction or sharpening requires non-linear operations on the image, this is very detrimental to the restoration of blurred images. However, in this embodiment, the blurred original image is restored first, and then the clear YUV target image is post-processed. This not only restores the clarity of the blurred image but also further improves the image quality of the final obtained image.

[0081] In some embodiments, to ensure that the image output by the target neural network model achieves the expected effect, an initial neural network model needs to be set up and trained with a large amount of data to obtain the target neural network model. In this embodiment, the neural network model can use a Pix2Pix model or a Unet network structure to build the neural network module. Of course, other models and network structures can also be used to build the neural network, and no limitation is made here. In one example, the training of the neural network model can be performed by the user while using the image processing device 100 or terminal 1000; in this way, the neural network model used is highly adapted to the current usage scenario. In another example, the training of the neural network model can be completed before the user uses the image processing device 100 or terminal 1000, and the obtained target neural network model can be pre-stored in the image processing device 100 or terminal 1000.

[0082] Specifically, please refer to Figure 15 In some embodiments, the image processing method further includes:

[0083] 06: Obtain a sample image set, which includes multiple sample image groups. Each sample image group includes a first sample image and a second sample image corresponding to the same scene. The clarity of the second sample image is greater than that of the first sample image, and each image pixel in the first and second sample images has image data of the luminance channel Y, the first chrominance channel U, and the second chrominance channel V.

[0084] 07: Obtain the luminance sample image, the first chrominance sample image, and the second chrominance sample image corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively, based on the first sample image, and perform deconvolution processing on the luminance sample image according to the first point spread function to obtain the intermediate sample image;

[0085] 08: Input the intermediate sample image, the first chroma sample image, and the second chroma sample image into the initial neural network model to obtain the training image;

[0086] 09: Calculate the total loss value of the initial neural network model based on the training images and the second sample images; and

[0087] 010: Iteratively train the initial neural network model based on the total loss value to obtain the target neural network model.

[0088] Please combine Figure 2 In some embodiments, the image processing apparatus 100 further includes a training module 70, and the methods in 06, 07, 08, 09 and 010 can be implemented by the training module 70. That is, the training module 70 can be used to acquire a sample image set, which includes multiple sample image groups. Each sample image group includes a first sample image and a second sample image corresponding to the same scene. The second sample image has a higher resolution than the first sample image, and each pixel in the first and second sample images has image data of a luminance channel Y, a first chrominance channel U, and a second chrominance channel V. Based on the first sample image, luminance sample images, first chrominance sample images, and second chrominance sample images corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V are acquired respectively. The luminance sample images are deconvolved according to the first point spread function to obtain intermediate sample images. The intermediate sample images, the first chrominance sample images, and the second chrominance sample images are input into the initial neural network model to obtain training images. The total loss value of the initial neural network model is calculated based on the training images and the second sample images. The initial neural network model is iteratively trained based on the total loss value to obtain the target neural network model.

[0089] Please combine Figure 3 In some embodiments, the methods in 06, 07, 08, 09, and 010 can also be implemented by the processor 200. That is, the processor 200 can also be used to acquire a sample image set, which includes multiple sample image groups. Each sample image group includes a first sample image and a second sample image corresponding to the same scene. The clarity of the second sample image is greater than that of the first sample image, and each image pixel in the first and second sample images has image data of a luminance channel Y, a first chrominance channel U, and a second chrominance channel V. Based on the first sample image, a luminance sample image, a first chrominance sample image, and a second chrominance sample image corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V are acquired respectively. The luminance sample image is deconvolved according to a first point spread function to obtain an intermediate sample image. The intermediate sample image, the first chrominance sample image, and the second chrominance sample image are input into an initial neural network model to obtain a training image. The total loss value of the initial neural network model is calculated based on the training image and the second sample image. The initial neural network model is iteratively trained based on the total loss value to obtain a target neural network model.

[0090] For example, processor 200 (or training module 70) acquires a sample image set, which includes multiple sample image groups. Each sample image group includes a first sample image and a second sample image corresponding to the same scene. The second sample image has higher sharpness than the first sample image, and each pixel in both the first and second sample images has image data for a luminance channel Y, a first chrominance channel U, and a second chrominance channel V. That is, the first and second sample images are images acquired from the same scene; the first sample image is relatively blurry and has field curvature, while the second sample image is relatively sharp. In other words, the first sample image is the image that needs image processing, and the second sample image is the corrected image.

[0091] The processor 200 (or training module 70) acquires luminance sample images, first chrominance sample images, and second chrominance sample images corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively, based on the first sample image. It then performs deconvolution processing on the luminance sample images according to a first point spread function to obtain intermediate sample images. The specific implementation of acquiring the luminance sample images, first chrominance sample images, and second chrominance sample images corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V from the first sample image is the same as the specific implementation of acquiring the luminance image, first chrominance image, and second chrominance image from the initial YUV image in the above embodiment. Similarly, the specific implementation of deconvolving the luminance sample images according to the first point spread function to obtain intermediate sample images is the same as the specific implementation of deconvolving the luminance image and the first point spread function to obtain intermediate images in the above embodiment, and will not be described in detail here.

[0092] Of course, in some embodiments, the first sample image can also be input into the first processing module 20, and the first processing module 20 can obtain a luminance sample image, a first chrominance sample image, and a second chrominance sample image corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively, based on the first sample image; the luminance sample image can be input into the deconvolution module 30 and deconvolved with the first point spread function to obtain an intermediate sample image.

[0093] After obtaining the intermediate sample image, the first chroma sample image, and the second chroma sample image, these images are input into the initial neural network model. The intermediate sample image, the first chroma sample image, and the second chroma sample image undergo a series of convolution, pooling, and deconvolution operations within the initial neural network model, enabling the model to output training images. After obtaining the training images, the processor 200 (or training module 70) calculates the total loss value of the initial neural network model based on the training images and the second sample image.

[0094] Please see Figure 16 In some embodiments, the total loss value of the initial neural network model is calculated based on the training images and the second sample images, including:

[0095] 091: Based on the training image and the second sample image, obtain the first loss value corresponding to the luminance channel Y, the second loss value corresponding to the first chrominance channel U, and the third loss value corresponding to the second chrominance channel V; and

[0096] 092: Obtain the total loss value based on the first loss value, the second loss value, the third loss value, and the preset first weight, second weight, and third weight, wherein the first weight, the second weight, and the third weight correspond to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively.

[0097] Please combine Figure 2 In some embodiments, the methods in 091 and 092 can be implemented by the training module 70. That is, the training module 70 can also be used to obtain, based on the training image and the second sample image, a first loss value corresponding to the luminance channel Y, a second loss value corresponding to the first chrominance channel U, and a third loss value corresponding to the second chrominance channel V; and to obtain a total loss value based on the first loss value, the second loss value, the third loss value, and preset first weights, second weights, and third weights, wherein the first weights, second weights, and third weights correspond to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively.

[0098] Please combine Figure 3 In some embodiments, the methods in 091 and 092 can be implemented by the processor 200. That is, the processor 200 can also be used to obtain, based on the training image and the second sample image, a first loss value corresponding to the luminance channel Y, a second loss value corresponding to the first chrominance channel U, and a third loss value corresponding to the second chrominance channel V; and to obtain a total loss value based on the first loss value, the second loss value, the third loss value, and preset first weights, second weights, and third weights, wherein the first weights, second weights, and third weights correspond to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively.

[0099] Specifically, please refer to Figure 17 In some embodiments, based on the training image and the second sample image, a first loss value corresponding to the luminance channel Y, a second loss value corresponding to the first chroma channel U, and a third loss value corresponding to the second chroma channel V are obtained, including:

[0100] 0911: Based on the training images, obtain the first training image, the second training image, and the third training image corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively; and based on the second sample image, obtain the first comparison image, the second comparison image, and the third comparison image corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively.

[0101] 0912: When the chromatic aberration dimension of the lens 402 used to acquire the original image is greater than a preset value, calculate the first loss value based on the first training image, the first comparison image, and the first loss function; calculate the second loss value based on the second training image, the second comparison image, and the second loss function; calculate the third loss value based on the third training image, the third comparison image, and the second loss function.

[0102] 0913: When the chromatic aberration dimension of the lens 402 used to acquire the original image is less than a preset value, calculate the first loss value based on the first training image, the first comparison image and the first loss function; calculate the second loss value based on the second training image, the second comparison image and the third loss function; calculate the third loss value based on the third training image, the third comparison image and the third loss function.

[0103] Please combine Figure 2In some embodiments, 0911, 0912 and 0913 can be implemented by the training module 70. That is, the training module 70 can also be used to obtain a first training image, a second training image, and a third training image corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively, based on the training images, and to obtain a first comparison image, a second comparison image, and a third comparison image corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively, based on the second sample image; when the chrominance difference size of the lens 402 used to acquire the original image is greater than a preset value, a first loss value is calculated based on the first training image, the first comparison image, and the first loss function; a second loss value is calculated based on the second training image, the second comparison image, and the second loss function; a third loss value is calculated based on the third training image, the third comparison image, and the second loss function; and when the chrominance difference size of the lens 402 used to acquire the original image is less than a preset value, a first loss value is calculated based on the first training image, the first comparison image, and the first loss function; a second loss value is calculated based on the second training image, the second comparison image, and the third loss function; and a third loss value is calculated based on the third training image, the third comparison image, and the third loss function.

[0104] Please combine Figure 3 In some embodiments, 0911, 0912 and 0913 can be implemented by processor 200. That is, the processor 200 can also be used to obtain a first training image, a second training image, and a third training image corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively, based on the training images, and to obtain a first comparison image, a second comparison image, and a third comparison image corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively, based on the second sample image; when the chrominance difference size of the lens 402 used to acquire the original image is greater than a preset value, a first loss value is calculated based on the first training image, the first comparison image, and the first loss function; a second loss value is calculated based on the second training image, the second comparison image, and the second loss function; a third loss value is calculated based on the third training image, the third comparison image, and the second loss function; and when the chrominance difference size of the lens 402 used to acquire the original image is less than a preset value, a first loss value is calculated based on the first training image, the first comparison image, and the first loss function; a second loss value is calculated based on the second training image, the second comparison image, and the third loss function; and a third loss value is calculated based on the third training image, the third comparison image, and the third loss function.

[0105] For example, please refer to Figure 18 and Figure 19In some embodiments, a first training image, a second training image, and a third training image, corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, are first obtained from the training images; a first comparison image, a second comparison image, and a third comparison image, corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, are then obtained from the second sample image. The image pixels in the first training image and the first comparison image have image data corresponding to the luminance channel Y; the image pixels in the second training image and the second comparison image have image data corresponding to the first chrominance channel U; and the image pixels in the third training image and the third comparison image have image data corresponding to the second chrominance channel V. It should be noted that the specific implementation method for obtaining the luminance image, the first chrominance image, and the second chrominance image from the training images and the second sample image is the same as the specific implementation method for obtaining the luminance image, the first chrominance image, and the second chrominance image from the initial YUV image in the above embodiments, and will not be repeated here.

[0106] After obtaining the first training image, the second training image, the third training image, the first comparison image, the second comparison image, and the third comparison image, the processor 200 (or the training module 70) can select different loss functions to calculate the loss value based on the quality of chromatic aberration control of the lens 402 used to acquire the original image. In some embodiments, the quality of chromatic aberration control of the lens 402 used to acquire the original image can be judged based on the chromatic aberration size. For example, if the chromatic aberration size is less than a preset value, the chromatic aberration control of the lens 402 can be considered relatively good; if the chromatic aberration size is greater than the preset value, the chromatic aberration control of the lens 402 can be considered relatively poor.

[0107] It should be noted that chromatic aberration can be transverse chromatic aberration, axial chromatic aberration, etc. The size of chromatic aberration can be simply understood as follows: when a beam of light passes through a lens, due to the difference in optical path difference between different wavelengths, it cannot converge to a single point on the image plane. The range of diffusion is the size of the chromatic aberration.

[0108] If the chromatic aberration size of lens 402 used to acquire the original image is greater than a preset value, it can be considered that the chromatic aberration control of lens 402 is relatively poor. In this case, a first loss function is selected to calculate the first loss value corresponding to the luminance channel Y, and a second loss function is selected to calculate the second and third loss values ​​corresponding to the first chroma channel U and the second chroma channel V, respectively. That is, the first loss value is calculated based on the first training image, the first comparison image, and the first loss function; the second loss value is calculated based on the second training image, the second comparison image, and the second loss function; and the third loss value is calculated based on the third training image, the third comparison image, and the second loss function. If the chromatic aberration size of lens 402 used to acquire the original image is less than a preset value, it can be considered that the chromatic aberration control of lens 402 is relatively good. In this case, the first loss function is selected to calculate the first loss value corresponding to the luminance channel Y, and a third loss function is selected to calculate the second and third loss values ​​corresponding to the first chroma channel U and the second chroma channel V, respectively. That is, a first loss value is calculated based on the first training image, the first comparison image, and the first loss function; a second loss value is calculated based on the second training image, the second comparison image, and the third loss function; and a third loss value is calculated based on the third training image, the third comparison image, and the third loss function.

[0109] It should be noted that when the chromatic aberration size of the lens 402 used to acquire the original image is equal to the preset value, either the second loss function can be selected to calculate the second loss value and the third loss value corresponding to the first chromatic channel U and the second chromatic channel V respectively; or the third loss function can be selected to calculate the second loss value and the third loss value corresponding to the first chromatic channel U and the second chromatic channel V respectively. No restriction is imposed here.

[0110] Taking the calculation of the first loss value based on the first training image, the first comparison image, and the first loss function as an example, in some embodiments, an image pixel can be arbitrarily extracted from the first training image, and then an image pixel located at the same position as the first comparison image can be selected. The image data corresponding to these two image pixels and the brightness channel Y are obtained respectively, and the difference between the two image data is calculated. Subsequently, the next image pixel is extracted from the first training image, and the above steps are repeated until all image pixels in the first training image have been extracted, thus obtaining multiple differences. Substituting the multiple differences into the first loss function for calculation, the first loss value can be obtained. The methods for calculating the second and third loss values ​​are similar and will not be described in detail here.

[0111] It is worth noting that the second loss function differs from the third loss function. Different loss functions determine the training direction of the model. Training with the third loss function will make the recovered image smoother, while training with the second loss function will make the recovered image sharper. In this embodiment, on the one hand, when the chromatic aberration control of lens 402 is relatively poor, that is, when the chromatic aberration size is greater than a preset value, the chromaticity channels (including the first chromaticity channel U and the second chromaticity channel V, the same below will not be elaborated) are trained using the second loss function. This can make the chromaticity of the finally recovered image sharper, thereby improving the effect of the chromaticity channels and compensating for the chromatic aberration problem caused by the design of lens 402. On the other hand, when the chromatic aberration control of lens 402 is relatively good, that is, when the chromatic aberration size is less than a preset value, the third loss function is used for training. This can make the chromaticity of the finally recovered image smoother, which is beneficial to improving the image quality of the final obtained image.

[0112] It should be noted that, in some embodiments, the second loss function can be the sum of the absolute values ​​of the differences, and the third loss function can be the sum of the squares of the differences. Furthermore, the first loss function can be the same as the second loss function, the same as the third loss function, or different from both the second and third loss functions; no restrictions are placed here.

[0113] After obtaining the first loss value, the second loss value, and the third loss value, the processor 200 (or the training module 70) obtains the total loss value based on the first loss value, the second loss value, the third loss value, and the preset first weight, second weight, and third weight, wherein the first weight, the second weight, and the third weight correspond to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively.

[0114] For example, in some embodiments, the total loss value can be calculated according to the formula: L 总 =α*L y +β*L U +γ*L v Calculated, where L 总 For total loss value, L y For the first loss value, L U For the second loss value, L v α is the third loss value, β is the first weight, β is the second weight, and γ is the third weight.

[0115] In some embodiments, different second and third weights can be selected to calculate the total loss value based on the quality of chromatic aberration control of the lens 402 used to acquire the original image. For details, please refer to... Figure 20 In some embodiments, obtaining the total loss value based on the first loss value, the second loss value, the third loss value, and preset first weights, second weights, and third weights further includes:

[0116] 0921: When the chromatic aberration dimension of the lens 402 used to acquire the original image is greater than a preset value, the total loss value is calculated based on the first loss value, the second loss value, the third loss value, the first weight, the first value, and the third value.

[0117] 0922: When the chromatic aberration dimension of the lens 402 used to acquire the original image is less than a preset value, the total loss value is calculated based on the first loss value, the second loss value, the third loss value, the first weight, the second value, and the fourth value.

[0118] Please combine Figure 2 In some embodiments, the methods in 0921 and 0922 can be implemented by the training module 70. That is, the training module 70 can also be used to: calculate a total loss value based on a first loss value, a second loss value, a third loss value, a first weight, a first value, and a third value when the chromatic difference size of the lens 402 used to acquire the original image is greater than a preset value; and calculate a total loss value based on a first loss value, a second loss value, a third loss value, a first weight, a second value, and a fourth value when the chromatic difference size of the lens 402 used to acquire the original image is less than a preset value.

[0119] Please combine Figure 3 In some embodiments, the methods in 0921 and 0922 can be implemented by the processor 200. That is, the processor 200 can also be used to: calculate a total loss value based on a first loss value, a second loss value, a third loss value, a first weight, a first value, and a third value when the chromatic difference size of the lens 402 used to acquire the original image is greater than a preset value; and calculate a total loss value based on a first loss value, a second loss value, a third loss value, a first weight, a second value, and a fourth value when the chromatic difference size of the lens 402 used to acquire the original image is less than a preset value.

[0120] Specifically, if the chromatic aberration dimension of the lens 402 used to acquire the original image is greater than a preset value, it can be considered that the chromatic aberration control of the lens 402 is relatively poor, and the total loss value can be calculated using the formula: L 总 =α*L y +β1*L U +γ1*L v Calculated, where L 总 For total loss value, L y For the first loss value, L U For the second loss value, L vLet α be the third loss value, β1 be the first weight, and γ1 be the third value. If the chromatic aberration dimension of the lens 402 used to acquire the original image is less than a preset value, it can be considered that the chromatic aberration control of the lens 402 is relatively good. The total loss value can be calculated using the formula: L 总 =α*L y +β2*L U +γ2*L v Calculated, where L 总 For total loss value, L y For the first loss value, L U For the second loss value, L v Let α be the third loss value, β2 be the first weight, β2 be the second value, and γ2 be the fourth value. In this embodiment, when the chromatic aberration control of lens 402 is relatively poor, the larger first value among the second weights and the larger third value among the third weights are selected to calculate the total loss value. That is, the proportion of the chromaticity channel is increased, and the recovery weight of the neural network model is concentrated on the image chromaticity. When the chromatic aberration control of lens 402 is relatively good, the smaller second value among the second weights and the smaller fourth value among the third weights are selected to calculate the total loss value. That is, the recovery weight of the neural network model is concentrated on the image blur.

[0121] After obtaining the total loss value of the initial neural network model, the processor 200 (or training module 70) iteratively trains the initial neural network model based on the total loss value to obtain the target neural network model. In some embodiments, the Adam optimizer can be used to iteratively train the initial neural network model based on the total loss value until the total loss value of the output of the initial neural network model converges, and the model at this point is saved as the target neural network model. The Adam optimizer combines the advantages of AdaGra (Adaptive Gradient) and RMSProp optimization algorithms, comprehensively considering the first moment estimation (mean of the gradient) and the second moment estimation (uncentered variance of the gradient) to calculate the update step size.

[0122] It should be noted that the termination conditions for iterative training can include: the number of iterations reaches the target number; or the total loss value of the initial neural network model's output meets the set convergence condition. In one example, meeting the set convergence condition for the total loss value can include: the total loss value is less than a set threshold. Of course, the specific set conditions are not necessarily limited.

[0123] After obtaining the target neural network model, in some embodiments, the target neural network model can be stored locally on the terminal 10001 (or image processing device 100). Alternatively, the target neural network model can be stored on a server communicatively connected to the terminal 1000 (or image processing device 100). Storing the target neural network model on a server can reduce the storage space occupied by the terminal 1000 (or image processing device 100) and improve the operating efficiency of the terminal 1000 (or image processing device 100). Of course, in some embodiments, the target neural network model can also periodically or irregularly acquire new training data for training and updating.

[0124] Please see Figure 21 This application also provides a non-volatile computer-readable storage medium 500 containing a computer program 501. When the computer program 501 is executed by one or more processors 200, the processors 200 perform the image processing method described in any of the above embodiments. For example, when the computer program 501 is executed by one or more processors 200, the processors 200 perform...

[0125] Image processing methods in 01, 02, 03, 04, 031, 032, 033, 0311, 0312, 05, 06, 07, 08, 09, 010, 091, 092, 0911, 0912, 0913, 0921, and 0922.

[0126] For example, please combine Figure 1 When computer program 501 is executed by one or more processors 200, the processors 200 perform the following methods:

[0127] 01: Perform chroma conversion on the original image to obtain the initial YUV image, wherein each image pixel in the original image has image data of the first color channel A, the second color channel B, and the third color channel C; each image pixel in the initial YUV image has image data of the luminance channel Y, the first chroma channel U, and the second chroma channel V;

[0128] 02: Based on the initial YUV image, obtain the luminance image, the first chrominance image, and the second chrominance image corresponding to the luminance channel Y, the first chrominance channel U, and the second chrominance channel V, respectively;

[0129] 03: Deconvolve the brightness image using the first-point spread function to obtain the intermediate image; the first-point spread function corresponds to the brightness channel Y; and

[0130] 04: Input the intermediate image, the first chroma image, and the second chroma image into the target neural network model for processing to obtain a clear YUV target image.

[0131] In the description of this specification, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with the described embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0132] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0133] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. An image processing method, characterized in that, include: The original image is chroma-converted to obtain a YUV initial image, wherein each image pixel in the original image has image data of a first color channel, a second color channel, and a third color channel; and each image pixel in the YUV initial image has image data of a luminance channel, a first chroma channel, and a second chroma channel. Based on the initial YUV image, a luminance image, a first chrominance image, and a second chrominance image are obtained, corresponding to the luminance channel, the first chrominance channel, and the second chrominance channel, respectively. The brightness image is deconvolved in blocks according to the first point spread function to obtain an intermediate image. The first point spread function is a point spread function corresponding to the brightness channel for different fields of view. and The intermediate image, the first chroma image, and the second chroma image are input into a target neural network model for processing to obtain a clear YUV target image. The target neural network model is obtained by training an initial neural network model based on a total loss value. The total loss value is obtained based on a first loss value corresponding to the luminance channel, a second loss value corresponding to the first chroma channel, and a third loss value corresponding to the second chroma channel. Specifically, when the chroma difference size of the lens used to acquire the original image is greater than a preset value, the second loss value and the third loss value are calculated according to a second loss function; when the chroma difference size of the lens used to acquire the original image is less than a preset value, the second loss value and the third loss value are calculated according to a third loss function. The second loss function differs from the third loss function; training with the third loss function makes the recovered image smoother, while training with the second loss function makes the recovered image sharper.

2. The image processing method according to claim 1, characterized in that, The intermediate image is obtained by deconvolving the brightness image in blocks according to the first point spread function. The first point spread function is a point spread function corresponding to the brightness channel for different fields of view, including: Obtain multiple first-point diffusion functions corresponding to different fields of view; The brightness image is segmented into blocks corresponding to multiple different fields of view, and each block under the corresponding field of view is deconvolved according to multiple first point spread functions to obtain multiple processed blocks; and The processed blocks are stitched together to obtain an intermediate image.

3. The image processing method according to claim 2, characterized in that, For an imaging device, the imaging device includes a lens and an image sensor, wherein acquiring multiple first point spread functions corresponding to different fields of view includes: Obtain multiple original point spread functions corresponding to different fields of view, and obtain the first point spread function based on the original point spread functions; wherein the imaging device generates the original point spread functions by capturing a point light source; or Based on the first parameter and the second parameter, the second point spread function, the third point spread function, and the fourth point spread function corresponding to the first color channel, the second color channel, and the third color channel under different fields of view are obtained respectively. Based on the second point spread function, the third point spread function, and the fourth point spread function under the same field of view, the first point spread function under the corresponding field of view is obtained. The first parameter is used to characterize the optical design parameters of the lens, and the second parameter is used to characterize the sensitivity of the image sensor.

4. The image processing method according to claim 1, characterized in that, The image processing method further includes: The YUV target image is post-processed to obtain the processed YUV target image. The post-processing includes at least one of noise reduction and sharpening.

5. The image processing method according to claim 1, characterized in that, The image processing method further includes: A sample image set is obtained, which includes multiple sample image groups. Each sample image group includes a first sample image and a second sample image corresponding to the same scene. The clarity of the second sample image is greater than that of the first sample image, and each image pixel in the first sample image and the second sample image has image data of a luminance channel, a first chroma channel and a second chroma channel. Based on the first sample image, obtain the luminance sample image, the first chrominance sample image, and the second chrominance sample image corresponding to the luminance channel, the first chrominance channel, and the second chrominance channel, respectively, and perform deconvolution processing on the luminance sample image according to the first point spread function to obtain the intermediate sample image; The intermediate sample image, the first chroma sample image, and the second chroma sample image are input into the initial neural network model to obtain the training image; Calculate the total loss value of the initial neural network model based on the training images and the second sample images; and The initial neural network model is iteratively trained based on the total loss value to obtain the target neural network model.

6. The image processing method according to claim 5, characterized in that, The step of calculating the total loss value of the initial neural network model based on the training image and the second sample image includes: Based on the training image and the second sample image, a first loss value corresponding to the luminance channel, a second loss value corresponding to the first chroma channel, and a third loss value corresponding to the second chroma channel are obtained respectively. The total loss value is obtained based on the first loss value, the second loss value, the third loss value, and the preset first weight, second weight, and third weight, wherein the first weight, the second weight, and the third weight correspond to the luminance channel, the first chroma channel, and the second chroma channel, respectively.

7. The image processing method according to claim 6, characterized in that, The step of obtaining a first loss value corresponding to the luminance channel, a second loss value corresponding to the first chroma channel, and a third loss value corresponding to the second chroma channel based on the training image and the second sample image includes: Based on the training images, a first training image, a second training image, and a third training image are obtained, respectively corresponding to the luminance channel, the first chroma channel, and the second chroma channel. Based on the second sample image, a first contrast image, a second contrast image, and a third contrast image are obtained, respectively corresponding to the luminance channel, the first chroma channel, and the second chroma channel. If the chromatic aberration dimension of the lens used to acquire the original image is greater than a preset value, the first loss value is calculated based on the first training image, the first comparison image, and the first loss function; the second loss value is calculated based on the second training image, the second comparison image, and the second loss function; and the third loss value is calculated based on the third training image, the third comparison image, and the second loss function. If the chromatic aberration dimension of the lens used to acquire the original image is less than a preset value, the first loss value is calculated based on the first training image, the first comparison image, and the first loss function; the second loss value is calculated based on the second training image, the second comparison image, and the third loss function; and the third loss value is calculated based on the third training image, the third comparison image, and the third loss function.

8. The image processing method according to claim 6, characterized in that, The second weight includes a first value and a second value, and the third weight includes a third value and a fourth value. The first value is greater than the second value, and the third value is greater than the fourth value. The step of obtaining the total loss value based on the first loss value, the second loss value, the third loss value, and the preset first weight, second weight, and third weight includes: If the chromatic aberration dimension of the lens used to acquire the original image is greater than a preset value, the total loss value is calculated based on the first loss value, the second loss value, the third loss value, the first weight, the first value, and the third value. If the chromatic aberration dimension of the lens used to acquire the original image is less than a preset value, the total loss value is calculated based on the first loss value, the second loss value, the third loss value, the first weight, the second value, and the fourth value.

9. An image processing system, characterized in that, include: A chroma conversion module is used to perform chroma conversion on the original image to obtain a YUV initial image, wherein each image pixel in the original image has image data of a first color channel, a second color channel, and a third color channel; and each image pixel in the YUV initial image has image data of a luminance channel, a first chroma channel, and a second chroma channel. The first processing module is used to obtain a luminance image, a first chromaticity image, and a second chromaticity image corresponding to the luminance channel, the first chromaticity channel, and the second chromaticity channel, respectively, based on the initial YUV image; The deconvolution module is used to perform block-based deconvolution processing on the brightness image according to the first point spread function to obtain an intermediate image. The first point spread function is a point spread function corresponding to the brightness channel for different fields of view. and The second processing module is used to input the intermediate image, the first chroma image, and the second chroma image into a target neural network model for processing to obtain a clear YUV target image. The target neural network model is obtained by training an initial neural network model based on a total loss value. The total loss value is obtained based on a first loss value corresponding to the luminance channel, a second loss value corresponding to the first chroma channel, and a third loss value corresponding to the second chroma channel. Specifically, when the chroma difference size of the lens used to acquire the original image is greater than a preset value, the second loss value and the third loss value are calculated according to a second loss function; when the chroma difference size of the lens used to acquire the original image is less than a preset value, the second loss value and the third loss value are calculated according to a third loss function. The second loss function is different from the third loss function; training with the third loss function makes the recovered image smoother, while training with the second loss function makes the recovered image sharper.

10. A terminal, characterized in that, include: Memory; One or more processors; and One or more programs, wherein the one or more programs are stored in the memory and executed by one or more processors, the programs comprising performing the image processing method according to any one of claims 1 to 8.

11. A non-volatile computer-readable storage medium storing a computer program that, when executed by one or more processors, implements the image processing method according to any one of claims 1 to 8.

Images