Image processing method, electronic equipment, storage medium and chip

Abstract

The invention provides an image processing method, electronic equipment, a storage medium and a chip. According to the method, image quality enhancement can be carried out on an image with the characteristic of a non-uniform out-of-focus image, and the characteristic of the non-uniform out-of-focus image is that one part of the image is clear, the other part of the image is blurred, and the blurring degrees of the blurred part are different. Specifically, an image enhancement model can be trained based on training data comprising a clear image and a non-uniform out-of-focus image, and the training data comprise the non-uniform out-of-focus image, so that the training data can reflect the characteristics of the non-uniform out-of-focus image; therefore, in the process of inputting the new non-uniform out-of-focus image in the trained image enhancement model for processing, the characteristics of the non-uniform out-of-focus image are considered, and compared with a related scheme without considering the characteristics of the non-uniform out-of-focus image, the image quality of the non-uniform out-of-focus image can be better enhanced, and the visual experience of a user is improved.

Description

Technical Field

[0001] The present application relates to the field of images, and in particular to an image processing method, electronic device, storage medium, and chip. Background Art

[0002] Electronic devices can capture images. However, due to factors such as the aperture factor or inaccurate focus adjustment, the captured images may be non-uniformly out of focus. Non-uniformly out of focus images are characterized by having portions that are sharp and portions that are blurred, with varying degrees of blur in the blurred portions.

[0003] When users preview or view non-uniformly defocused images on electronic devices, the human eye perceives the image as partially sharp and partially blurred, with varying degrees of blurriness in the blurred parts. As users' expectations for a superior visual experience in previewing or viewing images grow, non-uniformly defocused images may impact their visual experience.

[0004] It can be seen that the degree of blur of each part in the non-uniform defocused image is different. How to better enhance the image quality of the non-uniform defocused image based on different blur degrees to meet the user's requirements for visual experience is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The present application provides an image processing method, electronic device, storage medium and chip, which can better enhance the image quality of non-uniform out-of-focus images and improve the user's visual experience.

[0006] In a first aspect, an image processing method is provided, which is applied to an electronic device, and the method includes:

[0007] In response to a first operation, a first image is acquired, the first operation being an operation of performing image quality enhancement on the first image, the first image including a first area and a second area, the second area consisting of a plurality of second sub-areas, a first blur degree of the image in the first area being less than a minimum value of a second blur degree of the image in the plurality of second sub-areas, and at least some of the second blur degrees in the plurality of second blur degrees are different; image quality enhancement processing is performed on the first image based on an image enhancement model to obtain a second image, the blur degree of the second image being less than a minimum value of the plurality of second blur degrees, the image enhancement model being obtained by training at least one set of training data, each set of training data including a clear image and a non-uniform defocused image obtained based on the clear image, the non-uniform defocused image including a third area and a fourth area, the fourth area consisting of a plurality of fourth sub-areas, a third blur degree of the image in the third area being less than a minimum value of a fourth blur degree of the image in the plurality of fourth sub-areas, at least some of the fourth blur degrees in the plurality of fourth blur degrees are different, and the blur degree of the clear image is less than a minimum value of the plurality of fourth blur degrees.

[0008] It should be understood that the first operation refers to the user operation 1 in the following embodiment. The first image refers to the non-uniform defocused image 1 mentioned in the following embodiment, and the first image has the characteristics of a non-uniform defocused image. The second area in the first image can refer to Figure 2 The first area is shown in area A. Figure 2 The region other than region A in the image shown. Region A may also include multiple second sub-regions ( Figure 2 The blur degree of the image within the first region is referred to as a first blur degree, and the blur degree of the image within each second sub-region is referred to as a second blur degree.

[0009] The fact that the first blur level is less than the minimum of the multiple second blur levels can be understood as indicating that the image within the first area is clearer than the image within any second sub-area. The fact that at least some of the multiple second blur levels are different can be understood as indicating that the second blur levels of all second sub-areas within the second area are different. It can also be understood as indicating that the second blur levels of some second sub-areas within the second area are the same, while the second blur levels of other second sub-areas are different. These characteristics can indicate that the first image exhibits non-uniform out-of-focus characteristics.

[0010] The blur degree of the second image is less than the minimum value of multiple second blur degrees, which can be understood as: the second image is clearer than the image in any second sub-area in the first image, the second image has the same clarity as the first area in the first image, or the clarity of the second image is higher than the clarity of the first area in the first image.

[0011] It should also be understood that the clear images and the non-uniformly defocused images obtained based on the clear images included in each set of training data may refer to the data included in the training data of step S71 in the embodiments below. Terms related to the non-uniformly defocused images, such as the third area, fourth area, fourth sub-area, third blurriness, and fourth blurriness, can be referred to the above explanation of the first area, second area, second sub-area, first blurriness, and fourth blurriness in the first image, and will not be repeated here.

[0012] Among them, the third blur degree of the image in the third area is less than the minimum value of the fourth blur degrees of the images in multiple fourth sub-areas, and at least some of the multiple fourth blur degrees are different. These features can illustrate that the training data can reflect the characteristics of non-uniform defocused images.

[0013] The fact that the blur level of the clear image in the training data is less than the minimum of the plurality of fourth blur levels can be understood as meaning that the clear image is clearer than the image within any fourth subregion in the non-uniform defocused image. The clear image has the same clarity as the third region in the non-uniform defocused image, or the clear image has a higher clarity than the third region.

[0014] Regarding the implementation method of obtaining a first image in response to a first operation, and performing image quality enhancement processing on the first image based on an image enhancement model to obtain a second image, you can refer to the implementation method of obtaining a non-uniform defocused image 1 in response to user operation 1 in the following embodiment, and processing the non-uniform defocused image 1 through a trained image enhancement model to obtain a clear image 1, which will not be repeated here.

[0015] A related approach uses a neural network model to enhance the image quality of a first image with non-uniform defocus characteristics. The training data used by this model includes pairs of sharp and blurred images, where the blurred image in the pair is obtained by uniformly degrading the sharp image. This uniform degradation method results in essentially the same degree of blur across all parts of the blurred image, failing to reflect the characteristics of non-uniform defocus images. Consequently, the related approach's use of a neural network model to enhance the image quality of the first image with non-uniform defocus characteristics is less effective.

[0016] The training data of the image enhancement model in the embodiment of the present application includes non-uniform defocused images, and the non-uniform defocused images are obtained based on clear images. That is to say, the training data in the embodiment of the present application can reflect the characteristics of non-uniform defocused images. Since the training data can reflect the characteristics of non-uniform defocused images, the image enhancement model trained based on the training data can better learn the mapping relationship between non-uniform defocused images and clear images. After that, the first image is input into the trained image enhancement model. The image enhancement model can perform image quality enhancement processing on the first image based on the above mapping relationship, thereby improving the effect of image quality enhancement. Compared with related solutions, the image quality of non-uniform defocused images can be better enhanced, thereby improving the user's visual experience.

[0017] In conjunction with the first aspect, in a possible implementation of the first aspect, a process of obtaining a non-uniform defocused image based on a clear image includes:

[0018] Based on the depth map of the clear image, an image mask corresponding to the depth map is obtained, where the clear image includes M×N first pixel values, the depth map includes M×N depth data, and the image mask includes M×N blur parameters. The blur parameters are used to characterize the degree of blur of an image formed in an electronic device by an object located on the depth data in space. The values of at least some of the M×N blur parameters are different, and M and N are integers greater than 1. Based on the image mask, the clear image and the uniformly defocused image obtained based on the clear image are fused to obtain a non-uniform defocused image. The degree of blur of the uniformly defocused image is greater than that of the clear image. The uniformly defocused image includes M×N second pixel values, and the non-uniform defocused image includes M×N third pixel values.

[0019] The schematic diagrams of the clear image, depth map, image mask, uniform defocus image and non-uniform defocus image in the embodiments of the present application can be referred to. Figure 8 The first pixel value refers to the pixel value 1 mentioned in the embodiment below, the second pixel value refers to the pixel value 2 mentioned in the embodiment below, and the third pixel value is the pixel value 3 mentioned in the embodiment below. Figure 9 The data included in the clear image shown. M×N depth data can be referred to Figure 9 The data included in the depth map shown. The M×N blur parameters can be referred to Figure 9 The data in the image mask shown. The M×N second pixel values can be referred to Figure 9 The data included in the uniformly out-of-focus image shown. The M×N third pixel values can be referred to Figure 9 The non-uniform defocused image shown includes the data.

[0020] The fact that at least some of the M×N fuzzy parameters have different values can be understood as meaning that all of the M×N fuzzy parameters have different values. Alternatively, it can be understood as meaning that some of the M×N fuzzy parameters have the same values, while others have different values. For an explanation of the meaning of the fuzzy parameters, please refer to the following examples and will not be elaborated upon here.

[0021] In implementation, when calculating the image mask, the electronic device may obtain M×N diameters of the circle of confusion based on the M×N depth data included in the depth map, and then use the normalized results of the M×N diameters of the circle of confusion as M×N blur parameters to obtain the image mask. Alternatively, the electronic device may obtain M×N diameters of the circle of confusion based on the M×N depth data included in the depth map, obtain M×N ratios of the diameters of the circle of confusion to the pixel size based on the M×N diameters of the circle of confusion and the pixel size, and then use the normalized results of the ratios of the M×N diameters of the circle of confusion to the pixel size as M×N blur parameters to obtain the image mask.

[0022] When determining a non-uniform defocused image, the electronic device may assign different weight coefficients to the clear image and the uniform defocused image based on the M×N blur parameters included in the image mask. For example, the larger the blur parameter, the greater the weight coefficient assigned by the electronic device to the second pixel value in the uniform defocused image, and the smaller the weight coefficient assigned to the first pixel value in the clear image. For another example, the smaller the blur parameter, the greater the weight coefficient assigned by the electronic device to the second pixel value in the uniform defocused image, and the smaller the weight coefficient assigned to the first pixel value in the clear image. The clear image and the uniform defocused image are then fused using the above-mentioned weight coefficients to obtain a non-uniform defocused image.

[0023] In an embodiment of the present application, the electronic device can obtain an image mask including M×N blur parameters based on the depth map of the clear image. Since the blur parameters in the image mask can characterize the degree of blur of the image formed in the electronic device by an object located on the depth data in space, and the values of at least some of the M×N blur parameters are different, the clear image and the uniformly defocused image obtained based on the clear image are fused based on the image mask, and an image that can accurately reflect the characteristics of the non-uniform defocused image can be obtained.

[0024] In conjunction with the first aspect, in a possible implementation of the first aspect, obtaining an image mask corresponding to the depth map based on the depth map of the clear image includes:

[0025] The electronic device determines the diameters of M×N confusion circles corresponding to the M×N depth data based on the in-focus object distance and M×N-1 out-of-focus object distances; obtains M×N blur parameters based on the diameters of the M×N confusion circles to obtain an image mask corresponding to the depth map; wherein one depth data corresponds to the diameter of one confusion circle, the in-focus object distance is any depth data randomly selected from the M×N depth data, and the M×N-1 out-of-focus object distances are the depth data other than the in-focus object distance in the M×N depth data.

[0026] For an explanation of the meanings of the in-focus object distance, the M×N-1 out-of-focus object distances, and the diameters of the M×N circles of confusion, please refer to the following embodiments and will not be repeated here. Any depth data randomly selected from the M×N depth data may refer to the depth data 1 mentioned in the following embodiments.

[0027] During implementation, the electronic device may substitute the focused object distance and M×N-1 out-of-focus object distances into Formula 6 mentioned in the following embodiment to determine the diameters of the M×N circles of confusion corresponding to the M×N depth data.

[0028] When calculating an image mask, the electronic device may use the normalized result of the M×N circle of confusion diameters as M×N blur parameters to obtain the image mask. Alternatively, after obtaining the M×N circle of confusion diameters, the electronic device may calculate the ratios of the M×N circle of confusion diameters to the pixel size based on the M×N circle of confusion diameters and the pixel size, and then use the normalized result of the ratios of the M×N circle of confusion diameters to the pixel size as the M×N blur parameters to obtain the image mask.

[0029] In an embodiment of the present application, the electronic device can randomly select any one depth data from M×N depth data as the in-focus object distance, and then determine the diameter of the confusion circle based on the randomly selected in-focus object distance and the out-of-focus object distance to obtain an image mask. Thereafter, the non-uniform out-of-focus image in the training data can be obtained based on the image mask. That is, the non-uniform out-of-focus image in the training data is obtained based on the randomly selected in-focus object distance. Through this random selection method, the training data can be enriched. In this way, the image enhancement model can be trained with the training data to improve the generalization of the image enhancement model.

[0030] In conjunction with the first aspect, in a possible implementation of the first aspect, the electronic device is configured with a photosensitive element; and based on the diameters of the M×N circles of confusion, M×N blur parameters are obtained to obtain an image mask corresponding to the depth map, including:

[0031] Based on the diameters of the M×N circles of confusion and the pixel size of the photosensitive element, the ratios of the M×N circles of confusion to the pixel size are determined; the ratios of the M×N circles of confusion to the pixel size are normalized to obtain M×N blur parameters to obtain an image mask corresponding to the depth map.

[0032] It should be understood that pixel size refers to the size of a pixel unit on a photosensitive element (such as an image sensor) of an electronic device. Generally, the pixel size of an electronic device is a fixed value after it leaves the factory.

[0033] It should also be understood that the ratio of the circle of confusion to the pixel can reflect the clarity of the image. For example, a larger ratio indicates a blurrier image, while a smaller ratio indicates a clearer image. Therefore, in embodiments of the present application, the ratio of the circle of confusion to the pixel can represent the degree of blur.

[0034] In implementation, the electronic device can determine the ratios of M×N circles of confusion to the pixel size through Formula 7 mentioned in the following embodiment, and normalize the ratios of M×N circles of confusion to the pixel size through Formula 8 to obtain M×N blur parameters.

[0035] It should also be understood that in some embodiments, in order to avoid some normalized results not being within the range of 0 to 1, the electronic device can also truncate the normalized results through Formula 9 mentioned in the following embodiment so that the values of the M×N fuzzy parameters are within the range of 0 to 1.

[0036] In an embodiment of the present application, since the ratio of the circle of confusion to the pixel can reflect the clarity of the image, the electronic device can determine the ratios of M×N circles of confusion to the pixel size based on the diameters of the M×N circles of confusion and the pixel size of the photosensitive element; the ratios of the M×N circles of confusion to the pixel size are normalized to obtain M×N blur parameters, so that the blur parameters can more accurately reflect the degree of blur of the image formed in the electronic device by an object located on the depth data in space.

[0037] In conjunction with the first aspect, in a possible implementation of the first aspect, based on the image mask, a clear image and a uniformly defocused image obtained based on the clear image are fused to obtain a non-uniformly defocused image, including:

[0038] A matrix operation is performed on the M×N first pixel values and the M×N complementary parameters corresponding to the M×N blur parameters to obtain a first operation result, where one blur parameter corresponds to one complementary parameter, and the sum of one blur parameter and the corresponding complementary parameter is 1; a matrix operation is performed on the M×N second pixel values and the M×N blur parameters to obtain a second operation result; and M×N third pixel values are obtained based on the sum of the first operation result and the second operation result to obtain a non-uniform defocused image.

[0039] It should be understood that the M×N complementary parameters corresponding to the M×N fuzzy parameters may refer to (1-mask) shown in Formula 10. For example, the data in the M×N complementary parameters may refer to Figure 11The data in the matrix corresponding to (1-mask) corresponds one to one with the data in the matrix corresponding to mask. For example, 1-M1 corresponds to M1, and the sum of 1-M1 and M1 is 1, 1-M2 corresponds to M2, and the sum of 1-M2 and M2 is 1, etc.

[0040] The first operation result can be referred to Figure 11 The result of matrix operation between the matrix corresponding to HQ and the matrix corresponding to (1-mask) shown in the figure. The second operation result can be referred to Figure 11 The result of matrix operation between the matrix corresponding to Blur and the matrix corresponding to mask is shown. The M×N third pixel values can be referred to Figure 11 The data in the matrix corresponding to LQ is shown. For more information about this implementation, please refer to Figure 11 The process of matrix operation shown is not repeated here.

[0041] In the embodiment of the present application, since the blur parameter can reflect the degree of blur, the larger the value of the blur parameter, the higher the degree of blur, and the smaller the value of the blur parameter, the lower the degree of blur. The M×N first pixel values in the embodiment of the present application refer to the pixel values in the clear image, and the clear image can be considered as an image with a lower degree of blur. The M×N second pixel values refer to the pixel values in the uniformly defocused image, and the uniformly defocused image can be considered as an image with a higher degree of blur. In addition, since the embodiment of the present application can perform a matrix operation on the M×N first pixel values and the M×N complementary parameters corresponding to the M×N blur parameters to obtain a first operation result; perform a matrix operation on the M×N second pixel values and the M×N blur parameters to obtain a second operation result, and fuse the clear image and the uniformly defocused image based on the sum of the first operation result and the second operation result to obtain a fused image (non-uniform defocused image). Therefore, when the blur parameter is large, the complementary parameter is relatively small, which results in the uniformly defocused image with a higher degree of blur having a greater impact on the degree of blur of the fused image, and the clear image with a lower degree of blur having a smaller impact on the degree of blur of the fused image, thereby making the degree of blur of the fused image relatively large. When the blur parameter is small, the complementary parameter is relatively large. This results in the uniformly defocused image with a higher degree of blur having a smaller impact on the blur level of the fused image, while the clear image with a lower degree of blur has a greater impact on the blur level of the fused image, resulting in a relatively smaller blur level in the fused image. In other words, the fused image obtained by the method provided in the embodiments of the present application can accurately reflect the different blur levels of the blurred parts in the non-uniformly defocused image.

[0042] Furthermore, in the embodiment of the present application, the electronic device can obtain M×N third pixel values through matrix operations, which can improve data processing efficiency.

[0043] In a second aspect, an image processing method is provided, the method comprising:

[0044] Acquire at least one set of training data, each set of training data includes a clear image and a non-uniformly defocused image obtained based on the clear image, the non-uniformly defocused image includes a third area and a fourth area, the fourth area is composed of multiple fourth sub-areas, a third blur degree of the image in the third area is less than a minimum value of a fourth blur degree of the image in the multiple fourth sub-areas, at least some of the multiple fourth blur degrees are different, and the blur degree of the clear image is less than a minimum value of the multiple fourth blur degrees; train an image enhancement model based on the at least one set of training data until the image enhancement model converges.

[0045] For the relevant meanings of the terms in this embodiment, reference may be made to other embodiments, which will not be described in detail here. For the specific implementation of this embodiment, reference may be made to other embodiments, which will not be described in detail here.

[0046] The training data for the image enhancement model in the embodiments of the present application includes non-uniform defocused images. These non-uniform defocused images are derived from clear images. In other words, the training data in the embodiments of the present application can reflect the characteristics of non-uniform defocused images. Because the training data can reflect the characteristics of non-uniform defocused images, the image enhancement model trained based on this training data can effectively learn the mapping relationship between non-uniform defocused images and clear images.

[0047] In conjunction with the second aspect, in a possible implementation of the second aspect, a process of obtaining a non-uniform defocused image based on a clear image includes:

[0048] Based on the depth map of the clear image, an image mask corresponding to the depth map is obtained, where the clear image includes M×N first pixel values, the depth map includes M×N depth data, and the image mask includes M×N blur parameters. The blur parameters are used to characterize the degree of blur of an image formed in an electronic device by an object located on the depth data in space. The values of at least some of the M×N blur parameters are different, and M and N are integers greater than 1. Based on the image mask, the clear image and the uniformly defocused image obtained based on the clear image are fused to obtain a non-uniform defocused image. The degree of blur of the uniformly defocused image is greater than that of the clear image. The uniformly defocused image includes M×N second pixel values, and the non-uniform defocused image includes M×N third pixel values.

[0049] For the relevant meanings of the terms in this embodiment, reference may be made to other embodiments, which will not be described in detail here. For the specific implementation of this embodiment, reference may be made to other embodiments, which will not be described in detail here.

[0050] In an embodiment of the present application, the electronic device can obtain an image mask including M×N blur parameters based on the depth map of the clear image. Since the blur parameters in the image mask can characterize the degree of blur of the image formed in the electronic device by an object located on the depth data in space, and the values of at least some of the M×N blur parameters are different, the clear image and the uniformly defocused image obtained based on the clear image are fused based on the image mask, and an image that can accurately reflect the characteristics of the non-uniform defocused image can be obtained.

[0051] In a third aspect, an electronic device is provided, wherein the electronic device is configured to execute the method provided in the first aspect. Specifically, the electronic device may include a processing unit configured to execute any possible implementation of the first or second aspect.

[0052] In a fourth aspect, an electronic device is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs comprising instructions, which, when executed by the one or more processors, enable the electronic device to execute a method in any possible implementation of the first or second aspect above.

[0053] In a fifth aspect, a server is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs comprising instructions, which, when executed by the one or more processors, enable the electronic device to execute a method in any possible implementation of the above-mentioned second aspect.

[0054] In a sixth aspect, a computer-readable storage medium is provided, comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in the first or second aspect above.

[0055] In the seventh aspect, a chip is provided, comprising a memory for storing instructions; and a processor for calling and executing instructions from the memory, so that an electronic device equipped with the chip executes the method described in the first or second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic diagram of an imaging device is provided.

[0057] Figure 2 Schematic diagram of a non-uniform out-of-focus image is provided.

[0058] Figure 3 This is a structural diagram of an electronic device provided in an embodiment of the present application.

[0059] Figure 4 Schematic diagram of the software system of the electronic device according to an embodiment of the present application.

[0060] Figure 5 This is a schematic diagram of an application scenario provided by an embodiment of the present application.

[0061] Figure 6 This is a schematic diagram of another application scenario provided by an embodiment of the present application.

[0062] Figure 7 This is a schematic diagram of the training process of an image enhancement model provided in an embodiment of the present application.

[0063] Figure 8 This is a schematic diagram of obtaining a non-uniform defocused image provided in an embodiment of the present application.

[0064] Figure 9 This is a schematic diagram of data processing provided in an embodiment of the present application.

[0065] Figure 10 This is an imaging schematic diagram provided in an embodiment of the present application.

[0066] Figure 11 This is a schematic diagram of a matrix operation provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] Before introducing the technical solutions in the embodiments of the present application, the relevant terms involved in the technical solutions are explained.

[0068] Lens modules are used by electronic devices to form images. These modules include a lens assembly and a photosensitive element. The lens assembly refers to the lens portion of the lens module, typically consisting of multiple lens elements that focus light. The aperture of the lens assembly is called the aperture, and its size can be adjusted. A photosensitive element, such as an image sensor or film, is a component that captures light and converts it into an electrical signal.

[0069] The focal point is the point where light rays passing through a lens assembly converge. The focal length is the distance between the center of the lens assembly and the sharp image formed on the sensor. In other words, it's the distance between the center of the lens assembly and the focal point.

[0070] The focal plane is the area on the imaging plane where a sharp image is formed after adjusting the lens focal length and aperture. Ideally, the focal plane is a flat surface parallel to the imaging plane. Only objects within the focal plane are imaged sharply; objects elsewhere appear blurred.

[0071] The image plane refers to the plane where the image is formed in an electronic device, such as the plane where the photosensitive element is located.

[0072] Aberration refers to the deviation between the image formed in an actual optical system and that formed in an ideal optical system. In an ideal optical system, light rays parallel to the principal optical axis should converge precisely at a single point after passing through a lens assembly, forming a sharp focus. However, in actual optical systems, due to factors such as the shape, material, and design of the lenses in the lens assembly, light rays do not converge exactly as theoretically predicted, resulting in image deviation. Aberrations include spherical aberration, coma, and other types.

[0073] The circle of confusion, also known as the diffuse spot, refers to the situation in which, during the imaging process, the imaging light beam cannot converge at one point due to aberration, forming a diffuse circular projection on the image plane.

[0074] Focus usually refers to a clear imaging result, that is, the image of the object forms a clear point on the imaging plane.

[0075] Defocus usually refers to unclear imaging results, that is, the image of the object does not form a clear point on the imaging plane.

[0076] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0077] Electronic devices can capture images through lens modules. When light from a point light source passes through the lens assembly of an electronic device, the ideal image should be that the light is focused on a point on the photosensitive element, forming a clear image. For example, please refer to Figure 1 , Figure 1 A schematic diagram of an imaging device is provided. Figure 1 In the figure, light from point source A1 focuses on a single point on the photosensitive element, forming a clear image A2. Typically, the photosensitive element in an electronic device is located on the focal plane of the lens module. Therefore, the clear image formed by light focused on a single point on the photosensitive element is called the focused image, and the distance between the point source and the lens module is called the focused object distance.

[0078] But in reality, due to the existence of aberration, the light emitted by a point light source begins to gather and diffuse before and after the focus, making the image of the point blurred, forming a diffusion circle. For example, Figure 1 In the image, due to the presence of aberration, the light from the point light source B1 begins to converge at point B2 before the photosensitive element, and then the light diffuses onto the photosensitive element to form a circle of confusion. The diameter of the circle of confusion is δ B The light forms a diffusion circle on the photosensitive element, and the absence of a clear image can be called a defocused image. The distance between the point light source corresponding to the light and the lens module is called the defocused object distance.

[0079] Normally, when the diameter of the circle of confusion is small enough that the human eye cannot discern it, the human eye perceives the image as clear. The circle of confusion at this critical point is called the permissible circle of confusion. If the diameter of the circle of confusion exceeds this permissible range, the human eye will be able to recognize that the image is blurred, and thus perceive the image as unclear. Therefore, the diameter of the circle of confusion can characterize the degree of blur of the image, that is, the larger the diameter of the circle of confusion, the blurrier the image, and the smaller the diameter of the circle of confusion, the clearer the image. It can also be understood that the degree of blur of the image is different for circles of confusion of different diameters.

[0080] As we all know, the diameter of the circle of confusion is related to the distance between the point light source and the lens module (i.e., the object distance) and the lens assembly's aperture (i.e., the aperture size). Therefore, when an electronic device is shooting, after the aperture size is adjusted to a fixed value, the diameter of the circle of confusion formed by point light sources at different object distances will be different. Due to the different diameters of the circle of confusion, the degree of image blur is different, and therefore, the degree of blur of the point image formed by point light sources at different object distances is different.

[0081] The image captured by the electronic device can be regarded as a composition of multiple point images formed on the photosensitive element after the light emitted by multiple point light sources passes through the lens assembly. From the above analysis, it can be seen that if the diameter of the confusion circle corresponding to a part of the multiple point images does not exceed the allowable range of the permissible confusion circle, then the human eye feels that the part of the point image is clear. If the diameter of the confusion circle corresponding to another part of the multiple point images exceeds the allowable range of the permissible confusion circle, then the human eye feels that the part of the point image is blurred, and the object distance of the multiple point light sources corresponding to the other part of the point image may be different, so the blur degree of each point image in the other part of the point image may be different. For the sake of convenience of description, the embodiment of the present application refers to the image in which some point images are clear, some point images are blurred, and the blur degree of each point image in the blurred point images is different as a non-uniform defocused image. For example, please refer to Figure 2 , Figure 2 A schematic diagram of a non-uniform out-of-focus image is provided.

[0082] Figure 2 In the image shown, the image in the eye area A at the upper left corner of the person is blurred, and the blurring degree of each part in the area A is different. The images in other areas are clear.

[0083] As can be seen from the above, when a user previews or views a non-uniformly defocused image on an electronic device, the human eye perceives the image as partially sharp and partially blurred, with varying degrees of blurriness in the blurred areas. As users' expectations for a superior visual experience in previewing or viewing images grow, non-uniformly defocused images may impact their visual experience.

[0084] In order to improve the overall clarity of non-uniform defocused images and enhance the user's visual experience, a relevant solution is to enhance the image quality of non-uniform defocused images through a neural network model to improve the overall clarity of non-uniform defocused images.

[0085] The neural network model in the related solution is trained using an image pair of a clear image and a blurred image. The blurred image in the image pair is obtained by uniformly degrading the clear image. This uniform degradation method ensures that the degree of blurring of each part of the blurred image is substantially the same. For ease of description, the blurred image obtained by the uniform degradation method is referred to as a uniformly out-of-focus image in the embodiments of the present application.

[0086] In other words, after training on pairs of sharp and uniformly out-of-focus images, the neural network model in the related solution can learn the mapping relationship between uniformly out-of-focus images and sharp images. When the uniformly out-of-focus image is input again into the neural network model, the neural network model can enhance the uniformly out-of-focus image based on the mapping relationship to obtain a sharp image.

[0087] However, as can be seen from the above, the characteristics of non-uniform defocused images are: part of the image is clear, part is blurred, and the degree of blur of the blurred part is different. Uniform defocused images cannot reflect this characteristic. Since uniform defocused images cannot reflect the characteristics of non-uniform defocused images, the mapping relationship between uniform defocused images and clear images learned by the neural network model trained on image pairs of uniform defocused images and clear images is different from the mapping relationship between non-uniform defocused images and clear images. Therefore, the effect of enhancing the image quality of non-uniform defocused images through the mapping relationship in the neural network model of the relevant scheme is poor.

[0088] In order to solve the above technical problems, an embodiment of the present application provides an image processing method, which can train an image enhancement model through image pairs of clear images and non-uniform defocused images to obtain a trained image enhancement model. During the training process, the image enhancement model can learn the mapping relationship between non-uniform defocused images and clear images. Thereafter, the non-uniform defocused images are input into the trained image enhancement model, and the image enhancement model can enhance the image quality of the non-uniform defocused images based on the mapping relationship between the non-uniform defocused images and the clear images. Since the image enhancement model provided by the embodiment of the present application is obtained by training based on non-uniform defocused images, the characteristics of non-uniform defocused images are taken into account. Compared with related solutions, the image quality enhancement effect can be improved, thereby improving the user experience.

[0089] The image processing method provided in the embodiment of the present application involves the training of an image enhancement model. In the embodiment of the present application, the image enhancement model can be trained by a server or an electronic device before the electronic device leaves the factory, and then the trained image enhancement model is deployed in the electronic device. After the electronic device leaves the factory, the image quality of the non-uniform defocused image can be enhanced based on the trained image enhancement model. Among them, the server may refer to a cloud server, a server cluster, etc., and the embodiment of the present application does not limit the type of server. It should be noted that whether the server trains the image enhancement model required for this embodiment or the electronic device trains the image enhancement model required for this embodiment, the training logic is the same, and the specific training process will be described in detail in the following embodiments, which will not be repeated here.

[0090] The image processing method provided in the embodiment of the present application is applied to an electronic device with a shooting function. The electronic device may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The electronic device may be a straight screen mobile phone, a folding screen mobile phone, a camera, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control (industrial control), a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid (smart grid), a wireless terminal in transportation safety (transportation safety), a wireless terminal in a smart city (smart city), a wireless terminal in a smart home (smart home), etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the electronic device.

[0091] In order to better understand the embodiments of the present application, the structure of the electronic device according to the embodiments of the present application is introduced below.

[0092] Figure 3 1 is a schematic diagram of the structure of an electronic device 100 (taking a mobile phone as an example) provided in an embodiment of the present application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, an antenna 1, an antenna 2, a mobile communication module 140, a wireless communication module 150, a sensor module 160, a pressure sensor 160A, a touch sensor 160B, a display screen 170, a camera 180, and the like.

[0093] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0094] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (App), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0095] In an embodiment of the present application, when an image is captured by an electronic device, the processor can enhance the image quality of the captured image based on an image enhancement model to obtain a clear image.

[0096] The processor 110 may also include a memory for storing instructions and data.

[0097] The wireless communication module 150 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication function of the electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 140, wireless communication module 150, modulation and demodulation processor and baseband processor. Specifically in the embodiment of the present application, the electronic device 100 can be connected to the Bluetooth device through antenna 1, antenna 2, mobile communication module 140, wireless communication module 150, etc., so that the electronic device 100 and the Bluetooth device can realize wireless communication functions.

[0098] The display screen 170 refers to a foldable screen, which is a branch of the flexible screen. The display screen 170 is used to display images, videos, etc.

[0099] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100.

[0100] The internal memory 121 may be used to store computer executable program codes, where the executable program codes include instructions.

[0101] The pressure sensor 160A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 160A can be disposed on the display screen 170. There are many types of pressure sensors 160A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0102] The touch sensor 160B, also known as a "touch panel," can be disposed on the display screen 170. The touch sensor 160B and the display screen 170 form a touch screen, also known as a "touch screen." The touch sensor 160B is used to detect touch operations applied to or near the touch sensor 160B.

[0103] The electronic device 100 can implement a shooting function through an ISP, a camera 293, a video codec, a GPU, a display screen 294, and an application processor.

[0104] The camera 293 may also be referred to as a lens module, which includes a lens assembly and a photosensitive element, etc. For an explanation of the lens module, please refer to the above embodiment, which will not be repeated here.

[0105] This concludes the introduction to the hardware structure of the electronic device 100. It is understood that Figure 3 The components included in the hardware structure shown do not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits. For example, the low-power wake-up receiving module 180 can be integrated in a Bluetooth module, and the Bluetooth module can be integrated in the processor 110. For another example, the Bluetooth module can be integrated in the processor 110, and the low-power wake-up receiving module 180 and the processor 110 are two independent components.

[0106] Furthermore, operating systems run on the above components, such as the iOS operating system developed by Apple, the Android open-source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on these operating systems.

[0107] The operating system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the operating system of the electronic device 100.

[0108] The sub-device is a schematic diagram of the software system of the electronic device 100 for example. The software system includes several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. In some embodiments, Figure 4 As shown, the Android system can include five layers, which are application layer, application framework layer, hardware abstraction layer, driver layer and hardware layer from top to bottom.

[0109] The application layer may include a camera application, and may also include gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications, which are not limited in this embodiment of the present application.

[0110] The application framework layer provides an application programming interface (API) and a programming framework for the application programs in the application layer; the application framework layer may include some predefined functions.

[0111] For example, the application framework layer may include a camera access interface, which may include camera management and camera devices. Camera management may be used to provide an access interface for managing the camera, while camera devices may be used to provide an interface for accessing the camera.

[0112] The hardware abstraction layer (HAL) abstracts the hardware. For example, the HAL can include a camera abstraction layer and other hardware device abstraction layers; the camera HAL can call camera algorithms.

[0113] For example, the hardware abstraction layer includes camera algorithm modules, etc.

[0114] For example, the algorithm in the camera algorithm module may refer to an algorithm that does not rely on specific hardware implementation; for example, a code that can generally be run in a CPU, etc.

[0115] The image enhancement model mentioned above can be deployed in the camera application of the application layer, can be deployed as a service in the application framework layer, or can be deployed in the camera algorithm module of the hardware abstraction layer, which is not limited in this embodiment. In some embodiments, the camera algorithm module can also train the image enhancement model.

[0116] The driver layer is used to provide drivers for different hardware devices. For example, the driver layer may include a camera device driver.

[0117] The hardware layer can include camera devices and other hardware devices.

[0118] It is understandable that Figure 4 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer layers than shown, and each layer may include more or fewer components. For example, the software structure may also include system libraries, etc., and this application does not limit this.

[0119] It is understandable that, in order to implement the image processing method in the embodiment of the present application, the electronic device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments.

[0120] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0121] The execution subject of the image processing method provided in the embodiment of the present application can be the above-mentioned electronic device, or it can be a functional module and / or functional entity in the electronic device that can implement the image processing method, and the present application solution can be implemented through hardware and / or software. The specific implementation can be determined according to actual usage requirements and is not limited by the embodiment of the present application.

[0122] The above embodiments introduce the hardware structure and software system of the electronic device. The following embodiments introduce the application scenarios of the image processing method provided in the embodiments of the present application.

[0123] For example, please refer to Figure 5 , Figure 5 Schematic diagram of an application scenario provided by an embodiment of the present application, illustrating the process of processing a non-uniformly out-of-focus image to obtain a clear image during image capture by a mobile phone.

[0124] Figure 5 The user interface 510 shown in (a) includes a "camera" icon 511, and the user can click the icon 511. In response to the user clicking the icon 511, the mobile phone displays Figure 5The interface 520 shown in (b) of FIG. 5 is a diagram showing an interface 520. The user can perform a shooting operation in the interface 520. For example, the user can click on the control 521 in the interface 520. In response to the user clicking on the control 521, the mobile phone first captures the shooting object 523 in the viewfinder 522. If the captured image is a non-uniformly defocused image, the non-uniformly defocused image is processed by the image processing method provided in the embodiment of the present application to obtain a clear image. Finally, the mobile phone generates a thumbnail of the clear image and displays it. Figure 5 The interface 530 shown in (c) includes a thumbnail 531. The user can also click on the thumbnail 531, and the mobile phone displays a Figure 5 The interface 540 shown in (d) includes a clear image 541.

[0125] For example, please refer to Figure 6 , Figure 6 Schematic diagram of another application scenario provided by an embodiment of the present application, illustrating a process in which a mobile phone processes a non-uniformly out-of-focus image to obtain a clear image after capturing the non-uniformly out-of-focus image.

[0126] For example, when a user clicks the Figure 5 After the control 521 in the interface 520 is pressed, the mobile phone first captures the subject 523 in the viewfinder 522, captures a non-uniform defocused image, and then generates a thumbnail of the non-uniform defocused image and displays it. Figure 5 The interface 530 shown in (c) includes a thumbnail 531. The user can click on the thumbnail 531, and the mobile phone displays the thumbnail 531 in response to the user clicking on the thumbnail 531. Figure 6 The interface 610 shown in (a) of FIG. 6 includes a non-uniform defocused image 611 and an image quality enhancement control 6122. The user can click on the image quality enhancement control 612. In response to the user clicking on the control 612, the mobile phone processes the non-uniform defocused image using the image processing method provided in the embodiment of the present application to obtain a clear image and displays it. Figure 6 6 (b) shows interface 620, which includes a clear image 621. Of course, in other embodiments, the image quality enhancement control can also be placed in other applications, such as a gallery, image editing software, etc. The interface elements of the image quality enhancement control can also be in other styles, which are not limited in this embodiment of the application.

[0127] above Figure 5 and Figure 6In the image capture, the mobile phone can capture non-uniformly out-of-focus images through the viewfinder. It should be understood that the reasons why the images captured by the mobile phone are non-uniformly out-of-focus images include the following: the aperture (diaphragm) of the mobile phone's lens assembly is large, the focal length of the mobile phone's lens assembly is long, the mobile phone is far away from the photographed object, the user's hand is shaky when shooting or the focus is inaccurate when shooting, and the portrait of the user is not parallel to the imaging plane of the lens module when shooting a portrait.

[0128] The above embodiments introduce the application scenarios of the image processing method provided by the embodiments of the present application. The following embodiments are divided into two parts to introduce the implementation process of the image processing method provided by the embodiments of the present application. The first part introduces the training process of the image enhancement model, and the second part introduces the processing process of enhancing the image quality of non-uniform defocused images based on the trained image enhancement model.

[0129] Part 1: Training Process of Image Enhancement Model

[0130] For example, please refer to Figure 7 , Figure 7 This is a schematic diagram of the training process of an image enhancement model provided in an embodiment of the present application. This process can be performed by an electronic device or server, or by a processor in the electronic device or server, or by a chip in the electronic device or server, and is not limited in any way by the embodiments of the present application. For ease of description, the process is described in detail using an electronic device as an example.

[0131] S71, the electronic device obtains at least one set of training data, each set of training data includes a clear image and a non-uniform defocused image obtained based on the clear image.

[0132] S72: The electronic device trains the image enhancement model based on at least one set of training data until the image enhancement model converges.

[0133] In step S71, the clear image in each set of training data can be obtained by:

[0134] For example, when acquiring clear images, a camera with a high frame rate (e.g., 240 frames per second (240 fps)) can be used to acquire images. Images acquired by a high frame rate camera can ensure that the images are clear.

[0135] In step S71, the non-uniform defocused image in each set of training data can be obtained by:

[0136] The following example illustrates a non-uniform defocused image obtained based on a clear image in a set of training data. The method for obtaining the non-uniform defocused image in each set of training data can refer to the method for obtaining the non-uniform defocused image in the set of training data, which will not be repeated here.

[0137] First, based on the clear image, a depth map corresponding to the clear image is obtained, where the clear image includes M×N pixel values 1, and the depth map includes M×N depth data, where M and N are integers greater than 1.

[0138] As can be seen from the above embodiments, the characteristics of a non-uniformly defocused image are: part of the image is sharp, part is blurred, and the blurred parts have different degrees of blur. The degree of image blur is related to the diameter of the circle of confusion, which is related to the distance between the point light source and the lens module (i.e., the object distance). In other words, the degree of image blur is related to the object distance. Therefore, electronic devices can determine the degree of image blur based on the object distance.

[0139] Since electronic devices need to determine the degree of image blur based on the object distance, and the single pixel value in the depth map is the vertical distance from a point on an object in space to the lens module, that is, the depth data in the depth map can be used to represent the distance between the object and the lens module (i.e., the object distance). Therefore, when the electronic device obtains a non-uniform defocused image based on a clear image, it can first obtain the depth map corresponding to the clear image based on the clear image.

[0140] During implementation, the electronic device can process the clear image through a deep learning model to obtain a depth map, or it can obtain the depth map through a coordinate measuring machine method or a moiré fringe method. This embodiment of the present application is not limited to this.

[0141] For example, the depth map obtained by the electronic device based on the clear image can refer to Figure 8 , Figure 8 This is a schematic diagram of obtaining a non-uniform defocused image provided in an embodiment of the present application.

[0142] It should be understood that the electronic device processes the clear image in fact to process the data included in the clear image. For example, the clear image includes M×N pixel values 1 (for example, the pixel value 1 can be a combination of the intensities of three colors: red, green, and blue). After processing, M×N depth data can be obtained. For example, please refer to Figure 9 , Figure 9 This is a schematic diagram of data processing provided in an embodiment of the present application. Figure 9 An example is given where M×N=3×3. Figure 9 The medium-definition image includes 3×3 pixel values 1, totaling 9 pixel values 1, including H1 to H9. The data structure of these 9 pixel values 1 is a matrix. The electronic device can process the matrix to obtain a depth map, which includes 9 depth data from D1 to D9.

[0143] Secondly, based on the depth map, the electronic device obtains an image mask corresponding to the depth map. The image mask includes M×N blur parameters. Each blur parameter is used to represent the degree of blur of the image formed by the light emitted by the object on the depth data in the depth map in space through the lens module. The blur parameters corresponding to depth data of different values are different.

[0144] In implementation, the electronic device processes the depth map to obtain an image mask, including steps A to C below.

[0145] Step A: randomly select a depth data 1 from the M×N depth data included in the depth map as the focus object distance (the focus object distance may not be limited to the value on the depth map, and generally should not deviate too much from the point on the depth map).

[0146] It should be understood that the focus object distance refers to the distance between the object and the lens module that can be focused on a point on the photosensitive element to form a clear image.

[0147] For example, the electronic device Figure 9 A depth data item, such as D1, is randomly selected from the nine depth data items included in the depth map as the focus object distance.

[0148] For example, the focus object distance can also be data within the preset numerical range of the depth data 1. The preset numerical range of the depth data 1 can be a numerical range greater than the depth data 1 and less than or equal to a first numerical value, or a numerical range less than the depth data 1 and greater than or equal to a second numerical value. The first numerical value can refer to data greater than the preset numerical value of the depth data 1, and the second numerical value can refer to data less than the preset numerical value of the depth data 1. For example, if M×N depth data include four depth data, namely 1m, 4m, 7m, and 9m, a depth data 1 can be randomly selected, such as 4m. Any value within the preset numerical range of the depth data 1 is used as the focus distance. The preset numerical range of the depth data 1 can be generally understood as the numerical range of nearby values of the depth data 1. For example, the preset numerical range of 4m can be (4m, 5m] or [3m, 4m), where 5m can be considered the first numerical value and 3m can be considered the second numerical value. Therefore, the preset numerical range of 4m can be generally understood as the numerical range of nearby values of 1m. The electronic device can use any value within the range, such as 5m, as the focus distance. The embodiment of the present application does not limit the preset value range of the depth data 1.

[0149] In step B, the electronic device uses the depth data except the depth data 1 among the M×N depth data as the out-of-focus object distances to obtain M×N-1 out-of-focus object distances.

[0150] It should be understood that the out-of-focus object distance refers to the distance between the object that does not form a clear image on the photosensitive element and the lens module.

[0151] For example, the electronic device Figure 9 In the illustrated depth map, one depth data item, such as D1, is randomly selected from the nine depth data items included in the depth map as the in-focus object distance, and the remaining eight depth data items (D2 to D9) are used as the out-of-focus object distances.

[0152] It should also be noted that, when the electronic device uses data within a preset numerical range of the depth data 1 as the in-focus object distance, it may also use M×N depth data as the out-of-focus object distance.

[0153] In step C, the electronic device obtains M×N blur parameters based on the M×N depth data.

[0154] The M×N depth data may refer to the depth data corresponding to the 1 in-focus object distance and the depth data corresponding to the M×N-1 out-of-focus object distances, or may refer to the depth data corresponding to the M×N out-of-focus object distances.

[0155] In implementation, step C includes the following steps C1 to C3.

[0156] In step C1 , the electronic device determines the diameter of the circle of confusion corresponding to each depth data to obtain the diameters of M×N circles of confusion.

[0157] It should be understood that the calculation formula of the diameter of the circle of confusion can be obtained by the following method:

[0158] For example, please refer to Figure 10 , Figure 10 This is an imaging schematic diagram provided in an embodiment of the present application. Figure 10 In the figure, point C1 can be considered as an object in space located at the focused object distance in the depth map mentioned in the above embodiment, and the distance between point C1 and the lens assembly is the focused object distance d0. The light emitted from point C1 passes through the lens assembly and focuses on a point on the photosensitive element to form a clear point image C2. The distance between point C2 and the lens assembly (i.e., the image distance) is v0. Point E1 can be considered as an object in space located at any of the out-of-focus object distances in the depth map mentioned in the above embodiment, and the light emitted from point E1 begins to converge at point E2 before the photosensitive element, and then the light diffuses onto the photosensitive element to form a circle of confusion. The diameter of the formed circle of confusion is δ i The distance between point E2 and the lens assembly (i.e., image distance) is Vi. Figure 10 The aperture of the middle lens assembly is D.

[0159] Based on the Gaussian imaging formula, we know that the imaging formula of point C1 is:

[0160]

[0161] Where d0 refers to the object distance at point C1, v0 refers to the image distance at point C1, and f refers to the focal length of the lens assembly.

[0162] The imaging formula of point E1 is:

[0163]

[0164] Among them, d i refers to the object distance at point E1, v i is the object distance at point E1, and f is the focal length of the lens assembly.

[0165] based on Figure 10 The geometric relationship shown and the properties of similar triangles show that:

[0166]

[0167] Combining the above formulas 1, 2 and 3, we can get:

[0168]

[0169] As we all know, the relationship between the aperture factor and the aperture of the lens assembly is:

[0170]

[0171] Among them, F refers to the aperture factor, D refers to the aperture of the lens assembly, and f refers to the focal length of the lens assembly.

[0172] Combining the above formulas 4 and 5, we can know that:

[0173]

[0174] Among them, δ i It refers to the diameter of the circle of confusion formed by an object located on any one of the M×N depth data in space, d0 refers to the depth data 1 corresponding to the focus object distance mentioned in the above embodiment, and d i It refers to any one of the M×N depth data, f refers to the focal length of the lens assembly, and F refers to the aperture coefficient.

[0175] During implementation, the electronic device may substitute each depth data item among the M×N depth data items, the depth data item 1 corresponding to the focus object distance, and the focal length and aperture coefficient of the lens assembly into the above formula 6 to obtain the diameter of the circle of confusion corresponding to each depth data item, thereby obtaining the diameters of the M×N circles of confusion. It should be understood that, typically, when capturing an image, the focal length and aperture coefficient of the lens assembly are adjusted to fixed values. Therefore, when calculating the diameter of the circle of confusion using formula 6, f and F in formula 6 can be considered to be constants.

[0176] In step C2, the electronic device determines the ratio of each circle of confusion to the pixel based on the diameters of the M×N circles of confusion and the pixel size, and obtains the ratios of the M×N circles of confusion to the pixel.

[0177] It should be understood that pixel size refers to the size of a pixel unit on the photosensitive element of an electronic device (such as an image sensor). Pixel size is usually measured in microns (μm). Common pixel sizes include 0.8μm, 1.0μm, 1.6μm, 1.22μm,

[0178] 2.24 μm, etc. The embodiment of the present application does not limit the value of the pixel size. Generally, after an electronic device leaves the factory, the pixel size thereof is a fixed value.

[0179] It should also be understood that the ratio of the circle of confusion to the pixel can reflect the clarity of the image. For example, a larger ratio indicates a blurrier image, while a smaller ratio indicates a clearer image. Therefore, in the embodiments of the present application, the ratio of the circle of confusion to the pixel can represent the degree of blur mentioned in the above embodiments.

[0180] During implementation, the electronic device may substitute the diameters of the M×N circles of confusion and the pixel size into the following formula to obtain the ratio of each circle of confusion to the pixel, thereby obtaining the ratios of the M×N circles of confusion to the pixel.

[0181]

[0182] Among them, δ i It refers to the diameter of any one of the M×N circles of confusion, d p refers to the pixel size, p i It refers to the ratio of any diffusion circle to the pixel.

[0183] In step C3, the electronic device normalizes the ratios of the M×N circles of confusion to the pixels to obtain M×N fuzzy parameters.

[0184] In implementation, electronic devices can be normalized using the following formula:

[0185]

[0186] Among them, pnor m It is in p i A value is calculated according to a certain rule (such as the value at 95% of the statistical value) or selected as the normalized value (usually p i The larger value is limited to a certain range to avoid abnormally small or large values), def i It refers to the normalized result of the ratio of a diffusion circle to a pixel.

[0187] It should be understood that the value of the normalized result of the ratio of the circle of confusion to the pixel should be within the range of 0 to 1. In some embodiments, to avoid some normalized results not being within the range of 0 to 1, the electronic device may further truncate the normalized result obtained by Formula 8 using the following formula:

[0188] mask i =clip(def i , 0,1). Formula 9

[0189] Among them, mask i Refers to any fuzzy parameter among the M×N fuzzy parameters.

[0190] For example, Figure 9 The depth map shown includes 9 depth data from D1 to D9. The data structure of these 9 depth data is a matrix. The electronic device can process the matrix based on the above formulas 6 to 9 to obtain an image mask. The image mask includes 9 blur parameters from M1 to M9. The schematic diagram of the image mask can be referred to Figure 8 The image mask is shown in .

[0191] Then, the electronic device performs uniform degradation processing on the clear image to obtain a uniformly defocused image, which includes M×N pixel values 2.

[0192] It should be understood that a uniformly defocused image is more blurred than a clear image. In some embodiments, the degree of blur between a uniformly defocused image and a clear image can be measured using parameters, such as the variance of pixel values. In a clear image, the numerical variation of multiple pixel values 1 is relatively small, and thus the variance of multiple pixel values 1 is relatively small. In a uniformly defocused image, the numerical variation of multiple pixel values 2 is relatively large, and thus the variance of multiple pixel values 1 is relatively large.

[0193] In implementation, the electronic device can uniformly degrade the clear image by using methods such as Gaussian filtering and mean filtering to obtain a uniformly defocused image, which is not limited in the present embodiment. Figure 9 The clear image shown includes 9 pixel values 1 from H1 to H9. The data structure of these 9 pixel values 1 is a matrix. The electronic device can uniformly degrade the matrix to obtain a uniformly defocused image. The uniformly defocused image includes 9 pixel values 2 from B1 to B9. The uniformly defocused image can be referred to Figure 8 A uniformly out-of-focus image is shown.

[0194] Finally, the electronic device can fuse the clear image and the uniformly defocused image based on the image mask to obtain a non-uniformly defocused image.

[0195] In implementation, the electronic device can perform fusion processing using the following formula to obtain a non-uniform out-of-focus image:

[0196] LQ=HQ×(1-mask)+Blur×mask. Formula 10

[0197] Among them, HQ refers to the clear image, HQ can be considered as a matrix including M×N pixel values 1. Mask refers to the image mask, mask can be considered as a matrix including M×N blur parameters. Blur refers to the uniform defocused image, Blur can be considered as a matrix including M×N pixel values 2. LQ refers to the non-uniform defocused image (this image can be referred to Figure 8 The non-uniform defocused image shown), LQ can be considered as a matrix including M×N pixel values 3 (the matrix can be referred to Figure 9 Matrix of non-uniform out-of-focus images shown).

[0198] As can be seen from the above embodiments, the blur parameter in the mask can reflect the degree of blur. The larger the value of the blur parameter, the higher the degree of blur, and the smaller the value of the blur parameter, the lower the degree of blur. HQ can be considered as an image with a lower degree of blur, and Blur can be considered as an image with a higher degree of blur. By fusing HQ and Blur according to Formula 10, a fused image (non-uniform defocused image) can be obtained. When the blur parameter in the mask is large, the image with a higher degree of blur, Blur, has a greater impact on the degree of blur of the fused image, while the image with a lower degree of blur, HQ, has a lower impact on the degree of blur of the fused image, so the degree of blur of the fused image is relatively large. When the blur parameter in the mask is small, the image with a higher degree of blur, Blur, has a smaller impact on the degree of blur of the fused image, while the image with a lower degree of blur, HQ, has a greater impact on the degree of blur of the fused image, so the degree of blur of the fused image is relatively low. Since the mask includes multiple blur parameters, the values of the multiple blur parameters may be different. Therefore, the fused image, i.e., the non-uniform defocused image, can reflect different degrees of blur due to the different blur parameters.

[0199] In fact, electronic devices obtain non-uniform out-of-focus images by performing matrix operations. For example, please refer to Figure 11 , Figure 11 is a schematic diagram of a matrix operation provided in an embodiment of the present application. The electronic device performs a matrix operation on the matrix corresponding to HQ and the matrix corresponding to (1-mask), performs a matrix operation on the matrix corresponding to Blur and the matrix corresponding to mask, and then performs an addition operation on the results of the above two matrix operations to obtain the matrix corresponding to LQ. The matrix operation can be a matrix multiplication operation or an element-by-element multiplication operation, etc., which is not limited in the embodiment of the present application.

[0200] The electronic device can obtain at least one set of training data through the method shown in the above step S71, each set of training data includes a clear image and a non-uniform defocused image obtained based on the clear image, the clear image includes M×N pixel values 1, and the non-uniform defocused image includes M×N pixel values 3.

[0201] In step S72, the image enhancement model may refer to a feedforward neural network model, a convolutional neural network model, a recurrent neural network model, etc. The embodiment of the present application does not limit the type of model.

[0202] When training the image enhancement model, first, at least one set of training data is input into the initial image enhancement model to obtain the training analysis results of the initial image enhancement model.

[0203] Because the image enhancement model is not yet fully trained, there will be some deviation or error between the training analysis results and the standard analysis results. It should be understood that the clear images in a set of training data are the standard analysis results of the non-uniformly out-of-focus images in that set of training data during training.

[0204] Secondly, the global error of this round of training is calculated based on the training analysis results and the standard analysis results.

[0205] It should be understood that after obtaining each training analysis result, the global error of the current round of training can be calculated based on each training analysis result and the corresponding standard analysis result, and a determination can be made as to whether the global error meets a preset condition, such as whether the global error is less than 5%. Here, the preset condition can be determined during the training of the image enhancement model. For example, the preset condition can be set as the global error being less than a specific threshold, which can be a percentage value. The smaller the specific threshold, the more stable the image enhancement model obtained after the final training is, and the higher the accuracy of the predicted working condition will be.

[0206] In the embodiment of the present application, the global error refers to the loss function, which may include mean square error loss, mean absolute error loss, cross entropy loss function, etc. The embodiment of the present application does not limit the type of loss function.

[0207] Then, if the global error does not meet the preset conditions, the model parameters of the image enhancement model are adjusted, and the image enhancement model after the model parameters are adjusted is determined as the initial image enhancement model.

[0208] It should be understood that when the global error of this round of training does not meet the preset conditions, for example, when the global error of this round of training is 10%, the model parameters of the image enhancement model can be adjusted, and the image enhancement model after the model parameters are adjusted is determined as the initial image enhancement model, and then re-trained with the training data to repeatedly adjust the model parameters of the image enhancement model so that the global error subsequently calculated based on the training analysis results and the corresponding standard analysis results is minimized until the final global error meets the preset conditions.

[0209] Finally, if the global error meets the preset conditions, it is determined that the image enhancement model has converged.

[0210] It should be understood that when the global error of this round of training meets a preset condition, for example, when the global error of this round of training is less than 5%, it can be determined that the image enhancement model has converged.

[0211] In an embodiment of the present application, the training data of the image enhancement model includes non-uniform defocused images. In the process of obtaining a non-uniform defocused image based on a clear image, an image mask containing M×N blur parameters can be obtained based on the depth map of the clear image. Since the blur parameters in the image mask are obtained based on the ratio of the circle of confusion to the pixel, the ratio of the circle of confusion to the pixel can reflect the degree of blur of the image. Therefore, the blur parameters in the image mask can reflect the degree of blur of the image formed by the light emitted by the object on the depth data in the depth map in space through the lens module, and the blur parameters corresponding to depth data of different values are different. Since the blur parameters in the image mask can reflect the degree of blur, the clear image and the uniform defocused image are fused based on the image mask to obtain an image reflecting the characteristics of the non-uniform defocused image mentioned in the above embodiment. In other words, the training data provided by the embodiment of the present application can reflect the characteristics of the non-uniform defocused image mentioned in the above embodiment.

[0212] Since the training data can reflect the characteristics of the non-uniform defocused images mentioned in the above embodiments, the image enhancement model is trained based on the training data. The image enhancement model can better learn the mapping relationship between non-uniform defocused images and clear images. Thereafter, new data (for example, new non-uniform defocused images) is input into the trained image enhancement model. The image enhancement model can perform image quality enhancement processing on the new data based on the above mapping relationship, thereby improving the image quality enhancement effect.

[0213] Furthermore, in an embodiment of the present application, a depth data 1 is randomly selected from the depth map of the clear image in each set of training data as the focus object distance, and then an image mask of each set of training data is obtained based on the randomly selected focus object distance, and then a non-uniform defocused image is obtained based on the image mask. In other words, the non-uniform defocused image in each set of training data is obtained based on the randomly selected focus object distance. This random selection method can enrich the training data, and thus, the generalization of the image enhancement model can be improved by training the image enhancement model with this training data.

[0214] The generalization of a neural network model refers to its ability to perform well on unseen data, that is, its ability to recognize and process new input data. A neural network model with good generalization is able to learn universal patterns from the training data and apply these patterns to new and different data, rather than just memorizing the training data.

[0215] Part 2: The process of enhancing the image quality of non-uniform defocused images based on the trained image enhancement model

[0216] This process can be executed by an electronic device, a processor in the electronic device, or a chip in the electronic device, and the present application does not impose any limitation thereto. For ease of description, the process will be described in detail using an electronic device as an example.

[0217] The process may include the following steps:

[0218] First, the electronic device acquires a non-uniform defocused image 1 in response to a user operation 1 .

[0219] It should be understood that user operation 1 refers to an operation of enhancing the image quality. For example, user operation 1 may refer to the user performing an operation of enhancing the image quality. Figure 5 The shooting operation performed in the interface 520 shown in FIG. Figure 6 The operation of clicking the image quality enhancement control 612 in the illustrated interface 610 may also refer to other operations, such as the operation of a user previewing an image in a gallery, etc. This embodiment of the present application does not limit this.

[0220] The method of obtaining the non-uniform defocused image 1 may be: for example, the electronic device responds to the user clicking Figure 5 By operating the control 521 in the interface 520 shown, the subject 523 in the viewfinder 522 is captured, and a non-uniform defocused image 1 can be obtained. The non-uniform defocused image 1 can be referred to as Figure 8 The non-uniform defocused image shown is not limited in this embodiment of the present application.

[0221] For example, the electronic device responds to the user clicking Figure 6By clicking the image quality enhancement control 612 in the illustrated interface 610 , the non-uniform defocused image 1 captured by the electronic device through the viewfinder 522 can be obtained.

[0222] Secondly, the electronic device processes the non-uniform defocused image 1 through the trained image enhancement model to obtain a clear image 1.

[0223] It should be understood that the clear image 1 can refer to Figure 8 Clear image shown.

[0224] In implementation, the trained image enhancement model can process the non-uniform defocused image 1 based on the learned mapping relationship between the non-uniform defocused image and the clear image to obtain the clear image 1.

[0225] In the embodiment of the present application, since the training data can reflect the characteristics of the non-uniform defocused images mentioned in the above embodiment, the image enhancement model is trained based on the training data. The image enhancement model can better learn the mapping relationship between the non-uniform defocused images and the clear images. Thereafter, new data (for example, new non-uniform defocused images) is input into the trained image enhancement model. The image enhancement model can perform image quality enhancement processing on the new data based on the above mapping relationship, thereby improving the image quality enhancement effect.

[0226] Moreover, when the embodiment of the present application processes the non-uniform defocused image based on the image enhancement model, the parts of the non-uniform defocused image with a higher degree of blur can be processed with higher computing resources, and the parts of the non-uniform defocused image with a lower degree of blur can be processed with lower computing resources. In this way, the computing resources of the electronic device can be reasonably allocated, avoiding waste of computing resources.

[0227] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0228] The present application provides a computer program product that, when executed on an electronic device, enables the electronic device to execute the technical solution in the above embodiment. The implementation principle and technical effects are similar to those of the above method-related embodiments and will not be described in detail here.

[0229] The embodiment of the present application provides a readable storage medium, which contains instructions. When the instructions are executed on an electronic device, the electronic device executes the technical solution of the above embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0230] The present application provides a chip for executing instructions. When the chip is running, the technical solution of the above embodiment is executed. The implementation principle and technical effect are similar and will not be described here.

[0231] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0232] It should be understood that the “embodiment” mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the various embodiments in the entire specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0233] Those skilled in the art will understand that the various numerical numbers such as first and second involved in this application are only for the convenience of description and are not used to limit the scope of the embodiments of this application, and also indicate the order of precedence.

[0234] In this application, elements expressed in the singular are intended to mean "one or more" rather than "one and only one" unless otherwise specified. In this application, unless otherwise specified, "at least one" is intended to mean "one or more" and "a plurality" is intended to mean "two or more."

[0235] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. A can be singular or plural, and B can be singular or plural.

[0236] The term "at least one of..." in this document refers to all or any combination of the listed items. For example, "at least one of A, B and C" can mean: A exists alone, B exists alone, C exists alone, A and B exist at the same time, B and C exist at the same time, and A, B and C exist at the same time. A can be singular or plural, B can be singular or plural, and C can be singular or plural.

[0237] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0238] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0239] The same or similar parts between the various embodiments in this application can refer to each other. In the various embodiments in this application, and the various implementation methods / implementation methods / implementation methods in each embodiment, if there is no special explanation and logical conflict, the terms and / or descriptions between different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment are consistent and can be referenced to each other. The technical features in different embodiments and the various implementation methods / implementation methods / implementation methods in each embodiment can be combined to form new embodiments, implementation methods, implementation methods, or implementation methods according to their inherent logical relationships. The above-described implementation methods of this application do not constitute a limitation on the scope of protection of this application.

[0240] The above is only a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims. In short, the above is only a preferred embodiment of the technical solution of the present application, and is not used to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method includes: In response to a first operation, a first image is acquired, where the first operation is an operation of enhancing image quality of the first image, the first image includes a first region and a second region, the second region is composed of a plurality of second sub-regions, a first blur degree of the image in the first region is less than a minimum value of second blur degrees of the images in the plurality of second sub-regions, and at least some of the plurality of second blur degrees are different; The first image is enhanced based on an image enhancement model to obtain a second image, wherein the blur degree of the second image is less than a minimum value of multiple second blur degrees, and the image enhancement model is obtained by training at least one set of training data, each set of the training data includes a clear image and a non-uniform defocused image obtained based on the clear image, the non-uniform defocused image includes a third area and a fourth area, the fourth area is composed of multiple fourth sub-areas, the third blur degree of the image in the third area is less than a minimum value of the fourth blur degrees of the images in the multiple fourth sub-areas, at least some of the multiple fourth blur degrees are different, and the blur degree of the clear image is less than the minimum value of the multiple fourth blur degrees.

2. The method according to claim 1, characterized in that The process of obtaining the non-uniform defocused image based on the clear image includes: Obtaining, based on a depth map of the clear image, an image mask corresponding to the depth map, wherein the clear image includes M×N first pixel values, the depth map includes M×N depth data, and the image mask includes M×N blur parameters, the blur parameters being used to characterize a degree of blur of an image formed by the electronic device of an object located in space on the depth data, at least some of the M×N blur parameters having different values, where M and N are integers greater than 1; Based on the image mask, the clear image and the uniformly defocused image obtained based on the clear image are fused to obtain the non-uniformly defocused image, the blurring degree of the uniformly defocused image is greater than the blurring degree of the clear image, the uniformly defocused image includes M×N second pixel values, and the non-uniformly defocused image includes M×N third pixel values.

3. The method according to claim 2, characterized in that The obtaining, based on the depth map of the clear image, an image mask corresponding to the depth map, comprises: Determining, based on the in-focus object distance and the M×N-1 out-of-focus object distances, the diameters of the M×N circles of confusion corresponding to the M×N depth data; Based on the diameters of the M×N circles of confusion, M×N blur parameters are obtained to obtain an image mask corresponding to the depth map; Among them, one depth data corresponds to the diameter of the confusion circle, the focused object distance is any depth data randomly selected from the M×N depth data, and the M×N-1 out-of-focus object distances are the depth data in the M×N depth data except the focused object distance.

4. The method according to claim 3, characterized in that The electronic device is configured with a photosensitive element; and the obtaining of M×N blur parameters based on the diameters of the M×N circles of confusion to obtain an image mask corresponding to the depth map includes: Determining a ratio of the M×N circles of confusion to the pixel size based on the diameters of the M×N circles of confusion and the pixel size of the photosensitive element; Normalizing the ratios of the M×N circles of confusion to the pixel size to obtain M×N blur parameters, so as to obtain an image mask corresponding to the depth map.

5. The method according to any one of claims 2 to 4, characterized in that The step of fusing the clear image and the uniformly defocused image obtained based on the clear image based on the image mask to obtain the non-uniformly defocused image includes: performing a matrix operation on the M×N first pixel values and the M×N complementary parameters corresponding to the M×N blur parameters to obtain a first operation result, wherein one blur parameter corresponds to one complementary parameter, and a sum of one blur parameter and the corresponding complementary parameter is 1; Performing a matrix operation on the M×N second pixel values and the M×N blur parameters to obtain a second operation result; Based on the sum of the first operation result and the second operation result, M×N third pixel values are obtained to obtain the non-uniform defocus image.

6. An image processing method, characterized in that: include: Acquiring at least one set of training data, each set of training data comprising a clear image and a non-uniformly defocused image obtained based on the clear image, the non-uniformly defocused image comprising a third region and a fourth region, the fourth region being composed of a plurality of fourth subregions, a third blur degree of the image within the third region being less than a minimum value of fourth blur degrees of the images within the plurality of fourth subregions, at least some of the plurality of fourth blur degrees being different, and a blur degree of the clear image being less than a minimum value of the plurality of fourth blur degrees; The image enhancement model is trained based on at least one set of the training data until the image enhancement model converges.

7. The method according to claim 6, characterized in that The process of obtaining the non-uniform defocused image based on the clear image includes: Obtaining, based on a depth map of the clear image, an image mask corresponding to the depth map, wherein the clear image includes M×N first pixel values, the depth map includes M×N depth data, and the image mask includes M×N blur parameters, the blur parameters being used to characterize a degree of blur of an image formed in the electronic device of an object located in space on the depth data, at least some of the M×N blur parameters having different values, where M and N are integers greater than 1; Based on the image mask, the clear image and the uniformly defocused image obtained based on the clear image are fused to obtain the non-uniformly defocused image, the blurring degree of the uniformly defocused image is greater than the blurring degree of the clear image, the uniformly defocused image includes M×N second pixel values, and the non-uniformly defocused image includes M×N third pixel values.

8. An electronic device, characterized in that: include: one or more processors; one or more memories; The one or more memories store one or more computer programs, and the one or more computer programs include instructions. When the instructions are executed by the one or more processors, the electronic device performs the method as described in any one of claims 1 to 5 or claims 6 to 7.

9. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 5 or claims 6 to 7.

10. A chip, characterized in that: The chip includes: a memory for storing instructions; A processor is configured to call and execute the instructions from the memory, so that an electronic device equipped with the chip executes the method according to any one of claims 1 to 5 or claims 6 to 7.

Images