An image processing method, an electronic device, a storage medium and a chip

Abstract

The application provides an image processing method, an electronic device, a storage medium and a chip. The method can enhance the quality of an image with non-uniform defocus image characteristics. The characteristics of the non-uniform defocus image are that part of the image is clear and part of the image is blurred, and the blurred part has different degrees of blurring. Specifically, the image enhancement model can be trained based on training data including clear images and non-uniform defocus images. Since the training data includes non-uniform defocus images, the training data can reflect the characteristics of the non-uniform defocus images. Therefore, when a new non-uniform defocus image is input into the trained image enhancement model for processing, the characteristics of the non-uniform defocus image are considered. Compared with related schemes that do not consider the characteristics of the non-uniform defocus image, the quality of the non-uniform defocus image can be better enhanced, and the user's visual experience is improved.

Description

Technical Field

[0001] This application relates to the field of images, and more particularly to an image processing method, electronic device, storage medium, and chip. Background Technology

[0002] Electronic devices can capture images, but due to inaccurate aperture or focus adjustments, the captured images may be non-uniformly out of focus. Non-uniformly out-of-focus images are characterized by the following: parts of the image are sharp, while other parts are blurry, and the degree of blurriness varies in the blurred parts.

[0003] When users preview or view non-uniformly focused images on electronic devices, the human eye perceives parts of the image as sharp and parts as blurry, with varying degrees of blurriness in the blurred areas. As users' demands for a superior visual experience when previewing or viewing images increase, non-uniformly focused images may negatively impact their visual experience.

[0004] It can be seen that the degree of blur in different parts of a non-uniformly defocused image is different. How to better enhance the image quality of non-uniformly defocused images based on different degrees of blur in order to meet the user's requirements for visual experience is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides an image processing method, electronic device, storage medium, and chip that can better enhance the image quality of non-uniformly out-of-focus images and improve the user's visual experience.

[0006] Firstly, an image processing method is provided for use in an electronic device, the method comprising:

[0007] In response to a first operation, a first image is acquired. The first operation is an operation to enhance the image quality of the first image. The first image includes a first region and a second region. The second region is composed of multiple second sub-regions. The first blur degree of the image in the first region is less than the minimum value of the second blur degree of the images in the multiple second sub-regions. At least some of the second blur degrees are different. The first image is then enhanced based on an image enhancement model to obtain a second image. The blur degree of the second image is less than the minimum value of the multiple second blur degrees. The image enhancement model is obtained by training on at least one set of training data. Each set of training data includes a clear image and a non-uniformly defocused image obtained based on the clear image. The non-uniformly defocused image includes a third region and a fourth region. The fourth region is composed of multiple fourth sub-regions. The third blur degree of the image in the third region is less than the minimum value of the fourth blur degree of the images in the multiple fourth sub-regions. At least some of the fourth blur degrees are different. The blur degree of the clear image is less than the minimum value of the multiple fourth blur degrees.

[0008] It should be understood that the first operation refers to user operation 1 in the embodiments below. The first image refers to the non-uniformly defocused image 1 mentioned in the embodiments below, and the first image has the characteristics of a non-uniformly defocused image. The second region in the first image can be referred to Figure 2 The first region, A, is shown in the diagram. Figure 2 The image shown includes areas other than region A. Region A may also include multiple second sub-regions. Figure 2 (Not shown in the image). The degree of blurring of the image in the first region is called the first degree of blurring, and the degree of blurring of the image in each second sub-region is called the second degree of blurring.

[0009] The first blur level being less than the minimum of multiple second blur levels can be understood as the image within the first region being clearer than the image within any of the second sub-regions. The fact that at least some of the second blur levels are different can be understood as all the second blur levels within the second region being different. Alternatively, it can be understood as some second sub-regions within the second region having the same second blur level, while others have different second blur levels. These characteristics reflect that the first image has the properties of a non-uniformly defocused image.

[0010] The fact that the blur level of the second image is less than the minimum of multiple second blur levels can be understood as: the second image is clearer than the image in any second sub-region of the first image, the second image has the same clarity as the first region in the first image, or the clarity of the second image is higher than the clarity of the first region in the first image.

[0011] It should also be understood that the sharp image and the non-uniformly out-of-focus image obtained based on the sharp image included in each set of training data can refer to the data included in the training data of step S71 in the embodiments below. Related terms for non-uniformly out-of-focus images, such as the third region, fourth region, fourth sub-region, third blur, and fourth blur, can be found in the above explanations of the first region, second region, second sub-region, first blur, and fourth blur in the first image, and will not be repeated here.

[0012] Among them, the third blur degree of the image in the third region is less than the minimum value of the fourth blur degree of the images in multiple fourth sub-regions, and at least some of the fourth blur degrees are different. These features can illustrate that the training data can reflect the characteristics of non-uniform defocused images.

[0013] The fact that the blurriness of a sharp image in the training data is less than the minimum of multiple fourth blurriness levels can be understood as: the sharp image is sharper than the image within any fourth sub-region of the non-uniformly defocused image. The sharp image has the same sharpness as the third region in the non-uniformly defocused image, or the sharp image has a higher sharpness than the third region.

[0014] Regarding the implementation method of obtaining a first image in response to the first operation, and performing image quality enhancement processing on the first image based on the image enhancement model to obtain a second image, you can refer to the implementation method of obtaining a non-uniformly defocused image 1 in response to user operation 1 in the following embodiment, and processing the non-uniformly defocused image 1 through the trained image enhancement model to obtain a clear image 1, which will not be elaborated here.

[0015] The proposed solution uses a neural network model to enhance the image quality of a first image exhibiting non-uniform defocus characteristics. The training data used by this model consists of image pairs containing sharp and blurred images, where the blurred image is obtained from the sharp image using a uniform degradation method. This uniform degradation method results in a roughly uniform degree of blurring across different parts of the blurred image, failing to reflect the characteristics of a non-uniform defocus image. Consequently, the proposed solution using a neural network model to enhance the image quality of the first image exhibiting non-uniform defocus characteristics is ineffective.

[0016] The training data for the image enhancement model in this embodiment includes non-uniformly defocused images, which are obtained based on sharp images. In other words, the training data in this embodiment can reflect the characteristics of non-uniformly defocused images. Since the training data can reflect the characteristics of non-uniformly defocused images, the image enhancement model trained based on this data can better learn the mapping relationship between non-uniformly defocused images and sharp images. Then, by inputting a first image into the trained image enhancement model, the model can perform image quality enhancement processing on the first image based on the aforementioned mapping relationship, improving the image quality enhancement effect. Compared with related solutions, this method can better enhance the image quality of non-uniformly defocused images, improving the user's visual experience.

[0017] In conjunction with the first aspect, in one possible implementation of the first aspect, the process of obtaining a non-uniformly defocused image based on a sharp image includes:

[0018] Based on the depth map of the clear image, an image mask corresponding to the depth map is obtained. The clear image includes M×N first pixel values, and the depth map includes M×N depth data. The image mask includes M×N blur parameters, which are used to characterize the degree of blur of the image formed by an object located on the depth data in the electronic device. At least some of the blur parameters in the M×N blur parameters have different values, and M and N are integers greater than 1. Based on the image mask, the clear image and the uniformly defocused image obtained based on the clear image are fused to obtain a non-uniformly defocused image. The degree of blur of the uniformly defocused image is greater than that of the clear image. The uniformly defocused image includes M×N second pixel values, and the non-uniformly defocused image includes M×N third pixel values.

[0019] The schematic diagrams of sharp images, depth maps, image masks, uniformly out-of-focus images, and non-uniformly out-of-focus images in the embodiments of this application can be referred to. Figure 8 The first pixel value refers to pixel value 1 mentioned in the following embodiments, the third pixel value refers to pixel value 2 mentioned in the following embodiments, and the fourth pixel value refers to pixel value 3 mentioned in the following embodiments. M×N first pixel values ​​can be referenced. Figure 9 The data included in the clear image shown. M×N depth data can be referenced. Figure 9 The depth map shown includes data. The M×N blur parameters can be referenced. Figure 9 The data shown is from the image mask. The M×N second pixel values ​​can be referenced. Figure 9 The data included in the uniformly out-of-focus image shown. The M×N third pixel values ​​can be referenced. Figure 9 The data included in the non-uniformly out-of-focus image shown.

[0020] The statement that at least some of the fuzzy parameters in M×N fuzzy parameters have different values ​​can be understood as: all the fuzzy parameters in M×N fuzzy parameters have different values. Alternatively, it can be understood as: some of the fuzzy parameters have the same value, while the rest are different. For an explanation of the meaning of fuzzy parameters, please refer to the examples below; they will not be repeated here.

[0021] In implementation, when calculating the image mask, the electronic device can either obtain M×N blur circle diameters based on M×N depth data included in the depth map, and then use the normalized results of these M×N blur circle diameters as M×N blur parameters to obtain the image mask; or, based on the M×N depth data included in the depth map, obtain M×N blur circle diameters, and then, based on these M×N blur circle diameters and the pixel size, obtain M×N ratios of the blur circle diameters to the pixel size, and then use the normalized results of these ratios as M×N blur parameters to obtain the image mask.

[0022] When determining a non-uniformly defocused image, the electronic device can assign different weight coefficients to the sharp image and the uniformly defocused image based on M×N blur parameters included in the image mask. For example, the larger the blur parameter, the larger the weight coefficient assigned to the second pixel value in the uniformly defocused image, and the smaller the weight coefficient assigned to the first pixel value in the sharp image. Conversely, the smaller the blur parameter, the larger the weight coefficient assigned to the second pixel value in the uniformly defocused image, and the smaller the weight coefficient assigned to the first pixel value in the sharp image. Then, the sharp image and the uniformly defocused image are fused using these weight coefficients to obtain the non-uniformly defocused image.

[0023] In this embodiment, an electronic device can obtain an image mask containing M×N blur parameters based on the depth map of a clear image. Since the blur parameters in the image mask can characterize the degree of blur of the image formed by an object located on the depth data in space in the electronic device, and at least some of the blur parameters have different values, by fusing the clear image and the uniformly defocused image obtained based on the image mask, an image that can accurately reflect the characteristics of the non-uniformly defocused image can be obtained.

[0024] In conjunction with the first aspect, in one possible implementation of the first aspect, based on the depth map of the clear image, an image mask corresponding to the depth map is obtained, including:

[0025] The electronic device determines the diameters of M×N circles of confusion corresponding to M×N depth data based on the focusing distance and M×N-1 defocusing distances; based on the diameters of the M×N circles of confusion, it obtains M×N blur parameters to obtain the image mask corresponding to the depth map; wherein, one depth data corresponds to the diameter of one circle of confusion, the focusing distance is any depth data randomly selected from the M×N depth data, and the M×N-1 defocusing distances are the depth data other than the focusing distance from the M×N depth data.

[0026] The meanings of the focusing distance, the M×N-1 defocusing distances, and the diameters of the M×N circles of confusion can be explained in the following embodiments, and will not be repeated here. Any randomly selected depth data from the M×N depth data can refer to depth data 1 mentioned in the following embodiments.

[0027] In implementation, the electronic device can substitute the focusing distance and M×N-1 defocusing distances into Formula 6 mentioned in the following embodiments to determine the diameters of the M×N blur circles corresponding to the M×N depth data.

[0028] When calculating an image mask, electronic devices can use the normalized results of M×N circles of confusion diameters as M×N blur parameters to obtain the image mask. Alternatively, after obtaining the M×N circles of confusion diameters, the ratio of the M×N circles of confusion diameters to the pixel size can be obtained, and then the normalized results of these ratios can be used as the M×N blur parameters to obtain the image mask.

[0029] In this embodiment, the electronic device can randomly select any one of the M×N depth data as the focusing distance, and then determine the diameter of the circle of confusion based on the randomly selected focusing distance and the defocusing distance to obtain an image mask. After that, the non-uniform defocused image in the training data can be obtained based on the image mask. That is to say, the non-uniform defocused image in the training data is obtained based on the randomly selected focusing distance. By using this random selection method, the training data can be enriched. In this way, the generalization of the image enhancement model can be improved by training the image enhancement model with the training data.

[0030] In conjunction with the first aspect, in one possible implementation of the first aspect, the electronic device is equipped with a photosensitive element; and, based on the diameters of M×N blur circles, M×N blur parameters are obtained to obtain an image mask corresponding to the depth map, including:

[0031] Based on the diameters of M×N blur circles and the pixel size of the photosensitive element, the ratio of the M×N blur circles to the pixel size is determined; the ratio of the M×N blur circles to the pixel size is normalized to obtain M×N blur parameters, so as to obtain the image mask corresponding to the depth map.

[0032] It should be understood that pixel size refers to the size of a single pixel unit on the photosensitive element (such as an image sensor) of an electronic device. Typically, the pixel size of an electronic device is a fixed value after it leaves the factory.

[0033] It should also be understood that the ratio of the circle of confusion to the number of pixels can reflect the sharpness of the image. For example, the larger the ratio, the more blurred the image; the smaller the ratio, the sharper the image. Therefore, embodiments of this application can use the ratio of the circle of confusion to the number of pixels to represent the degree of blur.

[0034] In implementation, the electronic device can determine the ratio of M×N blur circles to pixel size using Formula 7 mentioned in the following embodiments, and normalize the ratio of M×N blur circles to pixel size using Formula 8 to obtain M×N blur parameters.

[0035] It should also be understood that, in some embodiments, in order to avoid some normalization results not being in the range of 0 to 1, the electronic device may also truncate the normalization results using Formula 9 mentioned in the embodiments below, so that the values ​​of the M×N fuzzy parameters are in the range of 0 to 1.

[0036] In this embodiment, since the ratio of the circle of confusion to the pixel size can reflect the sharpness of the image, the electronic device can determine the ratio of the M×N circles of confusion to the pixel size based on the diameter of the M×N circles of confusion and the pixel size of the photosensitive element; the ratio of the M×N circles of confusion to the pixel size is normalized to obtain M×N blur parameters. This allows the blur parameters to more accurately reflect the degree of blur of the image formed by the object located on the depth data in space in the electronic device.

[0037] In conjunction with the first aspect, in one possible implementation of the first aspect, based on an image mask, a sharp image and a uniformly defocused image obtained from the sharp image are fused to obtain a non-uniformly defocused image, including:

[0038] Perform matrix operations on M×N first pixel values ​​and M×N complementary parameters corresponding to M×N blur parameters to obtain a first operation result. One blur parameter corresponds to one complementary parameter, and the sum of one blur parameter and its corresponding complementary parameter is 1. Perform matrix operations on M×N second pixel values ​​and M×N blur parameters to obtain a second operation result. Based on the sum of the first and second operation results, obtain M×N third pixel values ​​to obtain a non-uniform defocused image.

[0039] It should be understood that the M×N complementary parameters corresponding to the M×N fuzzy parameters can refer to (1-mask) shown in Formula 10. For example, the data in the M×N complementary parameters can be referenced... Figure 11The data in the matrix corresponding to (1-mask) are in one-to-one correspondence with the data in the matrix corresponding to mask. For example, 1-M1 corresponds to M1 and the sum of 1-M1 and M1 is 1, 1-M2 corresponds to M2 and the sum of 1-M2 and M2 is 1, and so on.

[0040] The first calculation result can be used as a reference. Figure 11 The result of matrix operations between the matrix corresponding to HQ and the matrix corresponding to (1-mask) is shown. The second result can be referenced... Figure 11 The result of matrix operations between the matrix corresponding to Blur and the matrix corresponding to mask is shown. The M×N third pixel values ​​can be referenced. Figure 11 The data within the matrix corresponding to LQ is shown. For more information on this implementation, please refer to [reference needed]. Figure 11 The matrix operation process shown is not described in detail here.

[0041] In this embodiment, since the blur parameter reflects the degree of blur, the larger the value of the blur parameter, the higher the degree of blur, and the smaller the value of the blur parameter, the lower the degree of blur. In this embodiment, the M×N first pixel values ​​refer to the pixel values ​​in the clear image, which can be considered as an image with a low degree of blur. The M×N second pixel values ​​refer to the pixel values ​​in the uniformly defocused image, which can be considered as an image with a high degree of blur. Furthermore, since this embodiment can perform matrix operations on the M×N first pixel values ​​and the M×N complementary parameters corresponding to the M×N blur parameters to obtain a first operation result; and perform matrix operations on the M×N second pixel values ​​and the M×N blur parameters to obtain a second operation result, and fuse the clear image and the uniformly defocused image based on the sum of the first and second operation results to obtain a fused image (non-uniformly defocused image), when the blur parameter is large, the complementary parameter is relatively small. This results in the uniformly defocused image with a high degree of blur having a greater impact on the blur degree of the fused image, while the clear image with a low degree of blur has a smaller impact on the blur degree of the fused image, thus making the fused image relatively large in blur. When the blur parameter is small, the complementary parameter is relatively large. This results in a smaller impact of the uniformly defocused image with a higher degree of blur on the blur degree of the fused image, while the sharp image with a lower degree of blur has a larger impact on the blur degree of the fused image, thus making the blur degree of the fused image relatively small. In other words, the fused image obtained by the method provided in this application embodiment can accurately reflect the different degrees of blur in the blurred parts of the non-uniformly defocused image.

[0042] Furthermore, in this embodiment of the application, the electronic device can obtain M×N third pixel values ​​through matrix operations, which can improve data processing efficiency.

[0043] Secondly, an image processing method is provided, the method comprising:

[0044] Obtain at least one set of training data, each set of training data including a clear image and a non-uniformly defocused image obtained based on the clear image. The non-uniformly defocused image includes a third region and a fourth region. The fourth region is composed of multiple fourth sub-regions. The third blur degree of the image in the third region is less than the minimum value of the fourth blur degree of the images in the multiple fourth sub-regions. At least some of the fourth blur degrees are different. The blur degree of the clear image is less than the minimum value of the multiple fourth blur degrees. Train the image enhancement model based on at least one set of training data until the image enhancement model converges.

[0045] The meanings of the terms used in this embodiment can be found in other embodiments, and will not be repeated here. The specific implementation of this embodiment can also be found in other embodiments, and will not be repeated here.

[0046] The training data for the image enhancement model in this embodiment includes non-uniformly defocused images, which are obtained based on sharp images. In other words, the training data in this embodiment can reflect the characteristics of non-uniformly defocused images. Because the training data can reflect the characteristics of non-uniformly defocused images, the image enhancement model trained based on this training data can better learn the mapping relationship between non-uniformly defocused images and sharp images.

[0047] In conjunction with the second aspect, in one possible implementation of the second aspect, the process of obtaining a non-uniformly defocused image based on a sharp image includes:

[0048] Based on the depth map of the clear image, an image mask corresponding to the depth map is obtained. The clear image includes M×N first pixel values, and the depth map includes M×N depth data. The image mask includes M×N blur parameters, which are used to characterize the degree of blur of the image formed by an object located on the depth data in the electronic device. At least some of the blur parameters in the M×N blur parameters have different values, and M and N are integers greater than 1. Based on the image mask, the clear image and the uniformly defocused image obtained based on the clear image are fused to obtain a non-uniformly defocused image. The degree of blur of the uniformly defocused image is greater than that of the clear image. The uniformly defocused image includes M×N second pixel values, and the non-uniformly defocused image includes M×N third pixel values.

[0049] The meanings of the terms used in this embodiment can be found in other embodiments, and will not be repeated here. The specific implementation of this embodiment can also be found in other embodiments, and will not be repeated here.

[0050] In this embodiment, an electronic device can obtain an image mask containing M×N blur parameters based on the depth map of a clear image. Since the blur parameters in the image mask can characterize the degree of blur of the image formed by an object located on the depth data in space in the electronic device, and at least some of the blur parameters have different values, by fusing the clear image and the uniformly defocused image obtained based on the image mask, an image that can accurately reflect the characteristics of the non-uniformly defocused image can be obtained.

[0051] Thirdly, an electronic device is provided for performing the method provided in the first aspect. Specifically, the electronic device may include a processing unit for performing any possible implementation of the first or second aspect.

[0052] Fourthly, an electronic device is provided, comprising: one or more processors; one or more memories; wherein the one or more memories store one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method in any possible implementation of the first or second aspect described above.

[0053] Fifthly, a server is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform any possible implementation of the method in the second aspect above.

[0054] A sixth aspect provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first or second aspect.

[0055] In a seventh aspect, a chip is provided, including a memory for storing instructions; and a processor for retrieving and executing the instructions from the memory, causing an electronic device on which the chip is mounted to perform the method described in the first or second aspect above. Attached Figure Description

[0056] Figure 1 This is an imaging schematic diagram of an electronic device.

[0057] Figure 2 This is a schematic diagram of a non-uniformly defocused image.

[0058] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0059] Figure 4 This is a schematic diagram of the software system of an electronic device according to an embodiment of this application.

[0060] Figure 5 This is a schematic diagram of an application scenario provided in an embodiment of this application.

[0061] Figure 6 This is a schematic diagram of another application scenario provided by the embodiments of this application.

[0062] Figure 7 This is a schematic diagram illustrating the training process of an image enhancement model provided in an embodiment of this application.

[0063] Figure 8 This is a schematic diagram of obtaining a non-uniformly defocused image provided in an embodiment of this application.

[0064] Figure 9 This is a schematic diagram of a data processing method provided in an embodiment of this application.

[0065] Figure 10 This is an imaging schematic diagram provided in an embodiment of this application.

[0066] Figure 11 This is a schematic diagram of a matrix operation provided in an embodiment of this application. Detailed Implementation

[0067] Before introducing the technical solutions in the embodiments of this application, the relevant terms involved in the technical solutions will be explained.

[0068] A lens module is used by electronic devices to create images. A lens module includes a lens assembly and a photosensitive element. The lens assembly refers to the lens portion that makes up the lens module, typically composed of multiple lens elements, used to focus light. The aperture of the lens assembly is called the aperture, and its size can be adjusted. The photosensitive element is the component used to capture light and convert it into electrical signals, such as an image sensor or film.

[0069] The focal point is the point at which light passes through the lens assembly and focuses. The focal length is the distance between the center point of the lens assembly and the point on the image sensor where a clear image is formed; that is, the distance between the center of the lens assembly and the focal point.

[0070] The focal plane is the area on the image plane where a sharp image is formed after adjusting the lens's focal length and aperture. Ideally, the focal plane is a flat surface parallel to the image plane. Only objects on the focal plane will be in sharp focus; objects in other locations will appear blurry.

[0071] The image plane refers to the plane on which an image is formed in an electronic device. For example, the plane on which the photosensitive element in an electronic device is located.

[0072] Aberration refers to the deviation between the image formed in an actual optical system and the image formed in an ideal optical system. In an ideal optical system, light rays parallel to the principal axis should converge precisely to a single point after passing through the lens assembly, forming a sharp focus. However, in actual optical systems, due to factors such as the shape, material, and design of the lenses within the lens assembly, light rays do not converge exactly as theoretically predicted, resulting in image deviation. Aberrations include spherical aberration, coma, and other types.

[0073] A circle of confusion, also known as a blur spot, refers to a diffuse circular projection on the image plane caused by aberrations during the imaging process, where the imaging beam cannot converge to a single point.

[0074] Focusing usually refers to the clarity of the image, that is, the object's image forms a clear point on the imaging plane.

[0075] Out of focus usually refers to an unclear image, that is, the image of the object does not form a clear point on the imaging plane.

[0076] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0077] Electronic devices can capture images through a lens module. When light emitted from a point light source passes through the lens assembly of the electronic device, ideally, the light should be focused onto a single point on the image sensor, forming a clear image. For an example, please refer to... Figure 1 , Figure 1 This is an imaging schematic diagram of an electronic device. Figure 1 In this image, the light emitted from point light source A1 is focused on a single point on the photosensitive element, forming a clear image A2. Typically, the photosensitive element in an electronic device is located on the focal plane of the lens module. Therefore, the clear image formed by light rays focusing on a single point on the photosensitive element can be called a focused image, and the distance between the point light source corresponding to this light ray and the lens module is called the focusing distance.

[0078] However, in reality, due to aberrations, the light emitted from a point light source converges and diffuses before and after the focal point, causing the image of the point to become blurry, forming a circle of confusion. For example, Figure 1 In the image, due to aberrations, the light emitted from point light source B1 initially converges at point B2 before reaching the photosensitive element. The light then diffuses onto the photosensitive element, forming a circle of confusion with a diameter of δ. B When light rays form a blurred circle on the image sensor, failing to create a clear image, it can be called an out-of-focus image. The distance between the point light source corresponding to this light ray and the lens module is called the out-of-focus object distance.

[0079] Generally, when the diameter of the circle of confusion is small enough that it is imperceptible to the human eye, the image is perceived as sharp. This critical point is called the permissible circle of confusion. If the diameter of the circle of confusion exceeds this permissible range, the human eye can perceive the image as blurry, thus perceiving it as unclear. Therefore, the diameter of the circle of confusion characterizes the degree of blurriness of an image; that is, the larger the diameter of the circle of confusion, the blurrier the image, and the smaller the diameter, the sharper the image. In other words, circles of confusion of different diameters result in different degrees of blurriness.

[0080] As is well known, the diameter of the circle of confusion is related to the distance between the point light source and the lens module (i.e., the object distance) and the aperture of the lens assembly (i.e., the aperture size). Therefore, when an electronic device is taking a picture, after the aperture size is adjusted to a fixed value, the diameter of the circle of confusion formed by a point light source at different object distances will be different. Since the degree of blurring of the image is different for circles of confusion of different diameters, the degree of blurring of the point image formed by a point light source at different object distances will also be different.

[0081] Images captured by electronic devices can be viewed as multiple point images formed on a photosensitive element by light emitted from multiple point light sources passing through a lens assembly. As analyzed above, if the diameter of the circle of confusion corresponding to a portion of the point images does not exceed the permissible range of the circle of confusion, then the human eye perceives that portion of the point image as sharp. If the diameter of the circle of confusion corresponding to another portion of the point images exceeds the permissible range, then the human eye perceives that portion of the point image as blurry. Furthermore, the object distances of the multiple point light sources corresponding to this other portion of the point images may be different, thus the degree of blurriness of each point image in this other portion of the point image may be different. For ease of description, in this application, an image in which some point images are sharp, some point images are blurry, and the degree of blurriness of each point image in the blurry image is different is referred to as a non-uniformly defocused image. For example, please refer to... Figure 2 , Figure 2 This is a schematic diagram of a non-uniformly defocused image.

[0082] Figure 2 In the image shown, the image within region A, the upper left eye area of ​​the figure, is blurry, and the degree of blurriness varies in different parts of region A, while the images in other regions are clear.

[0083] As mentioned above, when users preview or view non-uniformly defocused images on electronic devices, the human eye perceives a portion of the image as clear and another portion as blurry, with varying degrees of blurriness in the blurred areas. As users' demands for a superior visual experience when previewing or viewing images increase, non-uniformly defocused images may negatively impact their visual experience.

[0084] To improve the overall sharpness of non-uniformly out-of-focus images and enhance the user's visual experience, the proposed solution is to use a neural network model to enhance the image quality of non-uniformly out-of-focus images, thereby improving their overall sharpness.

[0085] The neural network model in the relevant scheme is trained using image pairs of sharp and blurred images. The blurred image in the image pair is obtained from the sharp image using a uniform degradation method. This uniform degradation method ensures that the blurriness of different parts of the blurred image is essentially the same. For ease of description, the blurred image obtained by the uniform degradation method in this application is referred to as a uniformly defocused image.

[0086] In other words, the neural network model in the relevant scheme, after being trained on image pairs of sharp and uniformly out-of-focus images, can learn the mapping relationship between the uniformly out-of-focus and sharp images. When the uniformly out-of-focus image is input again into the neural network model, the model can enhance the image quality of the uniformly out-of-focus image according to the above mapping relationship, thus obtaining a sharp image.

[0087] However, as mentioned above, the characteristics of non-uniformly focused images are that part of the image is sharp and part is blurred, and the degree of blurriness varies in the blurred parts. Uniformly focused images cannot reflect this characteristic. Because uniformly focused images cannot reflect the characteristics of non-uniformly focused images, the mapping relationship learned by the neural network model trained on image pairs of uniformly focused and sharp images is different from the mapping relationship between non-uniformly focused and sharp images. Therefore, the effect of enhancing the image quality of non-uniformly focused images through the mapping relationship in the neural network model of related schemes is poor.

[0088] To address the aforementioned technical problems, this application provides an image processing method that trains an image enhancement model using image pairs of sharp and non-uniformly out-of-focus images, resulting in a trained image enhancement model. During training, the image enhancement model learns the mapping relationship between the non-uniformly out-of-focus image and the sharp image. Then, the non-uniformly out-of-focus image is input into the trained image enhancement model, which enhances the image quality of the non-uniformly out-of-focus image based on the mapping relationship between the two images. Since the image enhancement model provided in this application is trained on non-uniformly out-of-focus images, it takes into account the characteristics of non-uniformly out-of-focus images, thus improving the image quality enhancement effect and enhancing the user experience compared to related solutions.

[0089] The image processing method provided in this application involves training an image enhancement model. This application allows the image enhancement model to be trained via a server or electronic device before the electronic device leaves the factory, and then the trained image enhancement model is deployed in the electronic device. After leaving the factory, the electronic device can enhance the image quality of non-uniformly out-of-focus images based on the trained image enhancement model. The server can refer to a cloud server, server cluster, etc., and this application does not limit the type of server. It should be noted that whether the image enhancement model required for this embodiment is trained by a server or by an electronic device, the training logic is the same. The specific training process will be described in detail in the following embodiments and will not be repeated here.

[0090] The image processing method provided in this application is applied to electronic devices with shooting capabilities. Electronic devices can also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Electronic devices can be candybar phones, foldable phones, cameras, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on. The embodiments of this application do not limit the specific technologies or device forms used in the electronic devices.

[0091] To better understand the embodiments of this application, the structure of the electronic device of the embodiments of this application is described below.

[0092] Figure 3 This is a schematic diagram of the structure of an electronic device 100 (taking a mobile phone as an example) provided in an embodiment of this application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, an antenna 1, an antenna 2, a mobile communication module 140, a wireless communication module 150, a sensor module 160, a pressure sensor 160A, a touch sensor 160B, a display screen 170, a camera 180, etc.

[0093] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0094] Processor 110 may include one or more processing units, such as: application processor (App), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0095] In this embodiment of the application, when an electronic device captures an image, the processor can enhance the image quality of the captured image based on an image enhancement model to obtain a clear image.

[0096] The processor 110 may also include a memory for storing instructions and data.

[0097] The wireless communication module 150 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication function of the electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 140, wireless communication module 150, modem processor, and baseband processor. Specifically, in this embodiment, the electronic device 100 can connect to a Bluetooth device through antenna 1, antenna 2, mobile communication module 140, and wireless communication module 150 to enable wireless communication between the electronic device 100 and the Bluetooth device.

[0098] Display screen 170 refers to a foldable screen, which is a type of flexible screen. Display screen 170 is used to display images, videos, etc.

[0099] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the electronic device 100.

[0100] Internal memory 121 can be used to store computer executable program code, which includes instructions.

[0101] Pressure sensor 160A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 160A may be disposed on display screen 170. There are many types of pressure sensors 160A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0102] Touch sensor 160B, also known as a "touch panel," can be located on display screen 170. The touch sensor 160B and display screen 170 together form a touchscreen, also known as a "touch screen." Touch sensor 160B is used to detect touch operations applied to or near it.

[0103] Electronic device 100 can perform shooting functions through ISP, camera 293, video codec, GPU, display 294 and application processor.

[0104] The camera 293 can also be called a lens module, which includes a lens assembly and a photosensitive element. For an explanation of the lens module, please refer to the above embodiment; it will not be repeated here.

[0105] This concludes the introduction to the hardware structure of electronic device 100. It is understandable that... Figure 3 The components included in the hardware structure shown do not constitute a specific limitation on the electronic device 100. The electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have different component configurations. The various components shown in the figures can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits (ASICs). For example, a low-power wake-up receiver module 180 can be integrated into a Bluetooth module, which in turn can be integrated into a processor 110. Alternatively, the Bluetooth module can be integrated into the processor 110, and the low-power wake-up receiver module 180 and the processor 110 can be two separate components.

[0106] In addition, an operating system runs on top of these components. Examples include Apple's iOS, Google's Android, and Microsoft's Windows. Applications can be installed and run on this operating system.

[0107] The operating system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the operating system of electronic device 100.

[0108] The sub-device is a schematic diagram of the software system of the electronic device 100 in the example. The software system includes several layers, each with a clear role and division of labor, and the layers communicate with each other through a software interface. In some embodiments, such as Figure 4 As shown, the Android system can include five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer, driver layer, and hardware layer.

[0109] The application layer may include camera applications, and may also include applications such as gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS. This application embodiment does not limit this.

[0110] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer; the application framework layer may include some predefined functions.

[0111] For example, the application framework layer may include a camera access interface; the camera access interface may include camera management and camera devices. Specifically, camera management can provide an access interface for managing cameras; camera devices can provide an interface for accessing cameras.

[0112] A hardware abstraction layer is used to abstract hardware. For example, a hardware abstraction layer can include a camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can call camera algorithms.

[0113] For example, the hardware abstraction layer includes camera algorithm modules, etc.

[0114] For example, the algorithm in the camera algorithm module can refer to a code that does not depend on specific hardware implementation; for example, code that can typically run on a CPU.

[0115] The image enhancement model mentioned above can be deployed in the camera application at the application layer, or as a service in the application framework layer, or in the camera algorithm module at the hardware abstraction layer. This embodiment does not limit this. In some embodiments, the camera algorithm module can also train the image enhancement model.

[0116] The driver layer is used to provide drivers for different hardware devices. For example, the driver layer may include camera device drivers.

[0117] The hardware layer can include camera devices and other hardware devices.

[0118] Understandable, Figure 4 The layers in the illustrated software architecture and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer layers than illustrated, and each layer may include more or fewer components. For example, the software architecture may also include system libraries, etc., which are not limited in this application.

[0119] It is understood that, in order to implement the image processing methods in the embodiments of this application, electronic devices include hardware and / or software modules that perform various functions. Based on the algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments.

[0120] It should be noted that although the embodiments of this application are described using the Android system as an example, the basic principles are also applicable to electronic devices based on operating systems such as iOS or Windows.

[0121] The execution subject of the image processing method provided in this application embodiment can be the aforementioned electronic device, or a functional module and / or functional entity in the electronic device that can implement the image processing method. Furthermore, the solution of this application can be implemented by hardware and / or software, and the specific implementation can be determined according to actual usage requirements. This application embodiment does not impose any limitations.

[0122] The above embodiments describe the hardware structure and software system of the electronic device. The following embodiments describe the application scenarios of the image processing method provided in this application.

[0123] For example, please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. It shows the process by which a mobile phone processes a non-uniformly out-of-focus image during image capture to obtain a clear image.

[0124] Figure 5 The user interface 510 shown in (a) includes a "camera" icon 511, which the user can tap. In response to the user tapping icon 511, the phone displays... Figure 5Interface 520 is shown in (b) above. Users can perform shooting operations on interface 520; for example, a user can click on control 521 on interface 520. In response to the user clicking control 521, the mobile phone first captures the subject 523 in the viewfinder 522. If the captured image is a non-uniformly out-of-focus image, the image processing method provided in this embodiment is used to process the non-uniformly out-of-focus image to obtain a clear image. Finally, the mobile phone generates a thumbnail of the clear image and displays it. Figure 5 The interface 530 shown in (c) includes a thumbnail 531. The user can also click on the thumbnail 531, and the phone responds to this click by displaying... Figure 5 Interface 540 is shown in (d) and includes a clear image 541.

[0125] For example, please refer to Figure 6 , Figure 6 This is a schematic diagram illustrating another application scenario provided by an embodiment of this application. It shows the process by which a mobile phone processes a non-uniformly out-of-focus image to obtain a clear image after capturing the image.

[0126] For example, when a user clicks the above Figure 5 After accessing control 521 in interface 520, the phone first captures the subject 523 in viewfinder 522, capturing a non-uniformly out-of-focus image, then generates a thumbnail of the non-uniformly out-of-focus image and displays it. Figure 5 The interface 530 shown in (c) includes a thumbnail 531. The user can click on the thumbnail 531, and the phone responds to this action by displaying... Figure 6 The interface 610 shown in (a) includes a non-uniformly out-of-focus image 611 and an image quality enhancement control 6122. The user can click the image quality enhancement control 612. In response to the user's click, the mobile phone processes the non-uniformly out-of-focus image using the image processing method provided in this embodiment to obtain a clear image, which is then displayed. Figure 6 The interface 620 shown in (b) includes a clear image 621. Of course, in other embodiments, the image enhancement control can also be placed in other applications, such as gallery apps, image editing software, etc. The interface elements of the image enhancement control can also have other styles, which are not limited in this embodiment.

[0127] The above Figure 5 and Figure 6In practice, mobile phones can capture non-uniformly out-of-focus images through the viewfinder. It should be understood that the reasons for non-uniformly out-of-focus images captured by mobile phones include the following: the aperture of the phone's lens assembly is large; the focal length of the phone's lens assembly is long; the phone is far from the subject; the user's hand is shaky or the focus is inaccurate during shooting; and the image of the subject is not parallel to the imaging plane of the lens module when shooting portraits.

[0128] The above embodiments illustrate the application scenarios of the image processing method provided in this application. The following embodiments are divided into two parts to describe the implementation process of the image processing method provided in this application. The first part describes the training process of the image enhancement model, and the second part describes the image quality enhancement process of non-uniformly defocused images based on the trained image enhancement model.

[0129] Part 1: The Training Process of Image Augmentation Models

[0130] For example, please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating the training process of an image enhancement model provided in an embodiment of this application. This process can be executed by an electronic device or server, by a processor within the electronic device or server, or by a chip within the electronic device or server; this embodiment of the application does not impose any limitations. For ease of description, an electronic device will be used as an example to illustrate the process in detail.

[0131] S71, the electronic device acquires at least one set of training data, each set of training data including a sharp image and a non-uniformly defocused image obtained based on the sharp image.

[0132] S72, the electronic device trains the image enhancement model based on at least one set of training data until the image enhancement model converges.

[0133] In step S71, the clear images in each set of training data can be obtained in the following way:

[0134] For example, when capturing clear images, a high frame rate (e.g., 240 frames per second) camera can be used. Images captured by a high frame rate camera are guaranteed to be clear.

[0135] In step S71, the non-uniformly out-of-focus images in each set of training data can be obtained in the following way:

[0136] The following example illustrates how a non-uniformly out-of-focus image is obtained from a clear image in a set of training data. The method for obtaining non-uniformly out-of-focus images in each set of training data can be referred to the method for obtaining non-uniformly out-of-focus images in that set of training data, and will not be repeated here.

[0137] First, based on the clear image, the corresponding depth map is obtained. The clear image includes M×N pixel values ​​of 1, and the depth map includes M×N depth data, where M and N are integers greater than 1.

[0138] As described in the examples above, the characteristics of a non-uniformly defocused image are: part of the image is clear, and part is blurred, and the degree of blurriness varies in the blurred parts. The degree of blurriness is related to the diameter of the circle of confusion, which in turn is related to the distance between the point light source and the lens module (i.e., the object distance). In other words, the degree of blurriness is related to the object distance. Therefore, electronic devices can determine the degree of blurriness based on the object distance.

[0139] Since electronic devices need to determine the degree of blur in an image based on the object distance, and the individual pixel value in the depth map is the vertical distance from a point on an object in space to the lens module, the depth data in the depth map can be used to represent the distance between the object and the lens module (i.e., the object distance). Therefore, in the process of obtaining a non-uniformly out-of-focus image based on a sharp image, the electronic device can first obtain the depth map corresponding to the sharp image based on the sharp image.

[0140] In practice, electronic devices can process clear images using deep learning models to obtain depth maps, or they can obtain depth maps using coordinate measuring machine methods or moiré fringe methods. This application does not limit the specific methods used.

[0141] For example, the depth map obtained by the electronic device based on the clear image can refer to Figure 8 , Figure 8 This is a schematic diagram of obtaining a non-uniformly defocused image provided in an embodiment of this application.

[0142] It should be understood that processing a sharp image by an electronic device actually involves processing the data contained within the sharp image. For example, a sharp image may contain M×N pixel values ​​of 1 (for instance, these pixel values ​​could be a combination of the intensities of red, green, and blue colors). After processing, M×N depth data can be obtained. For an example, please refer to... Figure 9 , Figure 9 This is a schematic diagram of a data processing method provided in an embodiment of this application. Figure 9 The example is M×N=3×3. Figure 9 The clear image consists of 9 pixel values ​​(3×3). These 9 pixel values ​​are H1 to H9. The data structure of these 9 pixel values ​​is a matrix. The electronic device can process this matrix to obtain a depth map, which includes 9 depth data points (D1 to D9).

[0143] Secondly, the electronic device obtains an image mask corresponding to the depth map based on the depth map. The image mask includes M×N blur parameters. Each blur parameter is used to represent the degree of blur of the image formed by the light emitted by the object located on the depth data in the depth map through the lens module. The blur parameters are different for different depth data values.

[0144] In practice, the process of the electronic device processing the depth map to obtain the image mask includes the following steps A to C.

[0145] Step A: Randomly select a depth data point 1 from the M×N depth data points included in the depth map as the focus distance (the focus distance is not limited to the value on the depth map, but it should generally not deviate too much from the point on the depth map).

[0146] It should be understood that focusing distance refers to the distance between the object that can be focused on a point on the image sensor to form a clear image and the lens module.

[0147] For example, electronic devices in Figure 9 The depth data shown is randomly selected from the nine depth data points, such as D1, as the focusing distance.

[0148] For example, the focusing distance can also be a value within the preset range of depth data 1. The preset range of depth data 1 can be: a range of values ​​greater than depth data 1 and less than or equal to a first value, or a range of values ​​less than depth data 1 and greater than or equal to a second value. Here, the first value can refer to data greater than the preset value of depth data 1, and the second value can refer to data less than the preset value of depth data 1. For example, M×N depth data include 4 depth data points: 1m, 4m, 7m, and 9m. One depth data point, such as 4m, can be randomly selected. Any value within the preset range of this depth data 1 can be used as the focusing distance. The preset range of depth data 1 can be simply understood as the range of values ​​near depth data 1. For instance, the preset range of 4m can be (4m, 5m) or [3m, 4m]. 5m can be considered the first value mentioned above, and 3m can be considered the second value mentioned above. Therefore, the preset range of 4m can be simply understood as the range of values ​​1m away from 4m. The electronic device can use any value within this range, such as 5m, as the focusing distance. This application embodiment does not limit the preset value range of depth data 1.

[0149] Step B: The electronic device uses the depth data other than depth data 1 from the M×N depth data as the defocus distance, and obtains M×N-1 defocus distances.

[0150] It should be understood that out-of-focus distance refers to the distance between the object that does not form a clear image on the image sensor and the lens module.

[0151] For example, electronic devices in Figure 9 The depth data shown is randomly selected from the nine depth data points, such as D1, as the focusing distance. The remaining eight depth data points (D2 to D9) are then used as the out-of-focus distances.

[0152] It should also be noted that when the electronic device uses data within the preset range of depth data 1 as the focusing distance, M×N depth data can also be used as the out-of-focus distance.

[0153] Step C: The electronic device obtains M×N fuzzy parameters based on M×N depth data.

[0154] M×N depth data can refer to the depth data corresponding to the above-mentioned 1 focusing object distance and the depth data corresponding to M×N-1 out-of-focus object distances, or it can refer to the depth data corresponding to the above-mentioned M×N out-of-focus object distances.

[0155] In implementation, step C includes the following steps C1 to C3.

[0156] In step C1, the electronic device determines the diameter of the blur circle corresponding to each depth data to obtain the diameters of M×N blur circles.

[0157] It should be understood that the formula for calculating the diameter of the circle of confusion can be obtained through the following method:

[0158] For example, please refer to Figure 10 , Figure 10 This is an imaging schematic diagram provided in an embodiment of this application. Figure 10 In this context, point C1 can be considered as an object in space located at the focal distance in the depth map mentioned in the above embodiments. The distance between point C1 and the lens assembly is the focal distance d0. The light emitted from point C1 passes through the lens assembly and focuses on a point on the image sensor, forming a clear point image C2. The distance (i.e., image distance) between point C2 and the lens assembly is v0. Point E1 can be considered as an object in space located at any of the defocus distances in the depth map mentioned in the above embodiments. The light emitted from point E1 begins to converge at point E2 in front of the image sensor, and then the light diffuses onto the image sensor, forming a circle of confusion with a diameter δ. i The distance between point E2 and the lens assembly (i.e., the image distance) is Vi. Figure 10 The aperture of the lens assembly is D.

[0159] Based on the Gaussian imaging formula, the imaging formula for point C1 is:

[0160]

[0161] Where d0 refers to the object distance at point C1, v0 refers to the image distance at point C1, and f refers to the focal length of the lens assembly.

[0162] The imaging formula for point E1 is:

[0163]

[0164] Where, d i This refers to the object distance at point E1, v i E1 refers to the object distance, and f refers to the focal length of the lens assembly.

[0165] based on Figure 10 From the geometric relationships shown and the properties of similar triangles, we can conclude that:

[0166]

[0167] Combining formulas 1, 2, and 3 above, we can obtain:

[0168]

[0169] As is well known, the relationship between aperture value and lens aperture is as follows:

[0170]

[0171] Where F refers to the aperture value, D refers to the aperture of the lens assembly, and f refers to the focal length of the lens assembly.

[0172] Combining formulas four and five above, we can see that:

[0173]

[0174] Where, δ i d0 refers to the diameter of the circle of confusion formed by an object located at any one of the M×N depth data points in space, where d0 refers to the depth data 1 corresponding to the focusing object distance mentioned in the above embodiment. i It refers to any one of the M×N depth data, f refers to the focal length of the lens assembly, and F refers to the aperture coefficient.

[0175] In implementation, the electronic device can substitute each depth data point from the M×N depth data points, the depth data point 1 corresponding to the focusing object distance, the focal length of the lens assembly, and the aperture coefficient into Formula 6 above to obtain the diameter of the circle of confusion corresponding to each depth data point, thus obtaining the diameters of the M×N circles of confusion. It should be understood that, typically, the focal length and aperture coefficient of the lens assembly are adjusted to fixed values ​​during image capture; therefore, when calculating the diameter of the circle of confusion using Formula 6, f and F in Formula 6 can be considered constants.

[0176] In step C2, the electronic device determines the ratio of each blur circle to a pixel based on the diameter and pixel size of the M×N blur circles, thus obtaining the ratios of the M×N blur circles to pixels.

[0177] It should be understood that pixel size refers to the size of a single pixel unit on the photosensitive element of an electronic device (such as an image sensor). Pixel size is usually measured in micrometers (μm), and common pixel sizes include 0.8μm, 1.0μm, 1.6μm, and 1.22μm.

[0178] 2.24μm, etc., the embodiments of this application do not limit the numerical value of the pixel size. Usually, the pixel size of an electronic device is a fixed value after it leaves the factory.

[0179] It should also be understood that the ratio of the circle of confusion to the number of pixels can reflect the sharpness of the image. For example, the larger the ratio, the more blurred the image; the smaller the ratio, the sharper the image. Therefore, in the embodiments of this application, the ratio of the circle of confusion to the number of pixels can represent the degree of blur mentioned in the above embodiments.

[0180] In practice, the electronic device can substitute the diameters and pixel sizes of M×N blur circles into the following formula to obtain the ratio of each blur circle to a pixel, thus obtaining the ratio of M×N blur circles to pixels.

[0181]

[0182] Where, δ i It refers to the diameter of any one of the M×N circles of confusion, d p This refers to the pixel size, p i It refers to the ratio of any given circle of confusion to a single pixel.

[0183] In step C3, the electronic device normalizes the ratios of the M×N blur circles to the pixels to obtain M×N blur parameters.

[0184] In practice, electronic devices can be normalized using the following formula:

[0185]

[0186] Among them, pnor m It is in p i The normalized value is calculated according to certain rules (such as the value at the 95th percentile of the statistical value) or a value is selected as the normalized value (usually p). i (and limit the value to a certain range to avoid being abnormally small or large), def i This refers to the normalized result of the ratio of a diffusion circle to a pixel.

[0187] It should be understood that the normalized result of the ratio of the circle of confusion to the pixel should be within the range of 0 to 1. In some embodiments, to avoid some normalization results not falling within the range of 0 to 1, the electronic device may also truncate the normalization result obtained by Formula 8 using the following formula:

[0188] mask i =clip(def) i Formula Nine (0, 1)

[0189] Among them, mask i It refers to any one of the M×N fuzzy parameters.

[0190] For example, Figure 9 The depth map shown includes nine depth data points, D1 to D9. These nine depth data points are structured as a matrix. The electronic device can process this matrix using formulas six through nine to obtain an image mask. The image mask includes nine blur parameters, M1 to M9. A schematic diagram of the image mask can be found in [reference needed]. Figure 8 The image mask shown in the image.

[0191] Then, the electronic device performs uniform degradation processing on the clear image to obtain a uniformly defocused image, which includes M×N pixel values.

[0192] It should be understood that a uniformly out-of-focus image is a more blurred image than a sharp image. In some embodiments, the blurriness of a uniformly out-of-focus image and a sharp image can be measured by some parameters, such as the variance of pixel values. In a sharp image, the values ​​of multiple pixel values ​​1 vary relatively little, and therefore the variance of multiple pixel values ​​1 is small. In a uniformly out-of-focus image, the values ​​of multiple pixel values ​​2 vary relatively much, and therefore the variance of multiple pixel values ​​1 is large.

[0193] In implementation, electronic devices can uniformly degrade clear images using methods such as Gaussian filtering and mean filtering to obtain uniformly out-of-focus images. This application does not limit this approach. For example, Figure 9 The clear image shown includes nine pixel values ​​(H1 to H9) of value 1. These nine pixel values ​​(H1 to H9) form a matrix. An electronic device can perform uniform degradation processing on this matrix to obtain a uniformly defocused image. The uniformly defocused image includes nine pixel values ​​(B1 to B9) of value 2. A uniformly defocused image can be referenced. Figure 8 The image shown is uniformly out of focus.

[0194] Finally, the electronic device can fuse a sharp image and a uniformly out-of-focus image based on an image mask to obtain a non-uniformly out-of-focus image.

[0195] In practice, electronic devices can perform fusion processing using the following formula to obtain a non-uniformly defocused image:

[0196] LQ = HQ × (1 - mask) + Blur × mask. Formula 10

[0197] Here, HQ refers to a sharp image, which can be considered as an M×N matrix containing M×N pixels with a value of 1. mask refers to an image mask, which can be considered as an M×N matrix containing blur parameters. Blur refers to a uniformly out-of-focus image, which can be considered as an M×N matrix containing M×N pixels with a value of 2. LQ refers to a non-uniformly out-of-focus image (this image can be found in [reference]). Figure 8 The non-uniformly out-of-focus image shown can be considered as an M×N matrix of pixel values ​​(this matrix can be referenced). Figure 9 The matrix of the non-uniformly out-of-focus image is shown.

[0198] As shown in the above examples, the blur parameters in the mask reflect the degree of blur. A larger value indicates a higher degree of blur, while a smaller value indicates a lower degree of blur. HQ can be considered an image with a lower degree of blur, and Blur can be considered an image with a higher degree of blur. By fusing HQ and Blur using Formula 10, a fused image (non-uniformly out of focus image) can be obtained. When the blur parameters in the mask are large, the image with a higher degree of blur, Blur, has a greater impact on the blur degree of the fused image, while the image with a lower degree of blur, HQ, has a smaller impact. Therefore, the fused image has a relatively high degree of blur. When the blur parameters in the mask are small, the image with a higher degree of blur, Blur, has a smaller impact on the blur degree of the fused image, while the image with a lower degree of blur, HQ, has a larger impact. Therefore, the fused image has a relatively low degree of blur. Since the mask includes multiple blur parameters, and the values ​​of these parameters may be different, the fused image, i.e., the non-uniformly out of focus image, can reflect different degrees of blur due to the different blur parameters.

[0199] In fact, electronic devices obtain non-uniformly defocused images by performing matrix operations. For example, please refer to... Figure 11 , Figure 11 This is a schematic diagram of a matrix operation provided in an embodiment of this application. The electronic device performs matrix operations on the matrix corresponding to HQ and the matrix corresponding to (1-mask), performs matrix operations on the matrix corresponding to Blur and the matrix corresponding to mask, and then performs an addition operation on the results of the above two matrix operations to obtain the matrix corresponding to LQ. The matrix operations can be matrix multiplication, element-wise multiplication, etc., and this embodiment of the application does not limit the specific operations.

[0200] The electronic device can obtain at least one set of training data through the method shown in step S71 above. Each set of training data includes a clear image and a non-uniformly out-of-focus image obtained based on the clear image. The clear image includes M×N pixel values ​​1, and the non-uniformly out-of-focus image includes M×N pixel values ​​3.

[0201] In step S72, the image enhancement model can refer to a feedforward neural network model, a convolutional neural network model, a recurrent neural network model, etc. The embodiments of this application do not limit the type of model.

[0202] When training an image enhancement model, firstly, at least one set of training data is input into the initial image enhancement model to obtain the training analysis results of the initial image enhancement model.

[0203] Since the image enhancement model has not yet been fully trained initially, there will be some deviation and error between the output training analysis results and the standard analysis results. It should be understood that the sharp images in a set of training data represent the standard analysis results of the non-uniformly out-of-focus images in that set of training data during training.

[0204] Secondly, the global error of this round of training is calculated based on the training analysis results and the standard analysis results.

[0205] It should be understood that after obtaining the training analysis results, the global error of this round of training can be calculated based on the training analysis results and the corresponding standard analysis results, and it can be determined whether the global error meets the preset conditions, such as whether the global error is less than 5%. Here, the preset conditions can be determined when training the image enhancement model. For example, the preset condition can be set as the global error being less than a specific threshold. This specific threshold can be a percentage value. The smaller the specific threshold, the more stable the image enhancement model obtained after training, and the higher the accuracy of the predicted working conditions.

[0206] In this embodiment, global error refers to the loss function, which may include mean squared error loss, mean absolute error loss, cross-entropy loss function, etc. This embodiment does not limit the type of loss function.

[0207] Then, if the global error does not meet the preset conditions, the model parameters of the image enhancement model are adjusted, and the image enhancement model with the adjusted model parameters is determined as the initial image enhancement model.

[0208] It should be understood that when the global error of this round of training does not meet the preset conditions, for example, when the global error of this round of training is 10%, the model parameters of the image enhancement model can be adjusted, and the image enhancement model with the adjusted model parameters can be determined as the initial image enhancement model. Then, the model can be retrained with the training data to repeatedly adjust the model parameters of the image enhancement model so that the global error calculated based on the training analysis results and the corresponding standard analysis results is minimized until the final global error meets the preset conditions.

[0209] Finally, if the global error meets the preset conditions, the image enhancement model is determined to have converged.

[0210] It should be understood that when the global error of this round of training meets the preset conditions, for example, when the global error of this round of training is less than 5%, it can be determined that the image enhancement model has converged.

[0211] In this embodiment, the training data for the image enhancement model includes non-uniformly defocused images. During the process of obtaining a non-uniformly defocused image based on a sharp image, an image mask containing M×N blur parameters can be obtained based on the depth map of the sharp image. Since the blur parameters in the image mask are based on the ratio of the circle of confusion to the number of pixels, and this ratio reflects the degree of blur in the image, the blur parameters in the image mask can reflect the degree of blur in the image formed by light emitted from objects located at depth data in the depth map passing through the lens module. Furthermore, different depth data values ​​correspond to different blur parameters. Because the blur parameters in the image mask reflect the degree of blur, fusing the sharp image and the uniformly defocused image based on the image mask can yield an image reflecting the characteristics of the non-uniformly defocused image mentioned in the above embodiments. In other words, the training data provided in this embodiment can reflect the characteristics of the non-uniformly defocused image mentioned in the above embodiments.

[0212] Since the training data can reflect the characteristics of the non-uniformly out-of-focus image mentioned in the above embodiments, the image enhancement model can be trained based on the training data. The image enhancement model can learn the mapping relationship between the non-uniformly out-of-focus image and the sharp image. Then, new data (such as a new non-uniformly out-of-focus image) is input into the trained image enhancement model. The image enhancement model can perform image quality enhancement processing on the new data based on the above mapping relationship, thereby improving the image quality enhancement effect.

[0213] Furthermore, in this embodiment, a depth data point 1 is randomly selected from the depth map of the sharp image in each training data set as the focus distance. Then, an image mask for each training data set is obtained based on this randomly selected focus distance. Subsequently, a non-uniformly defocused image is obtained based on the image mask. In other words, the non-uniformly defocused image in each training data set is obtained based on a randomly selected focus distance. This random selection method enriches the training data, thereby improving the generalization ability of the image enhancement model when trained using this training data.

[0214] The generalization ability of a neural network model refers to its performance on unseen data, that is, its ability to recognize and process new input data. A neural network model with good generalization ability can learn general rules from the training data and apply these rules to new and different data, rather than simply memorizing the training data.

[0215] Part Two: The Process of Image Enhancement for Non-Uniformly Defocused Images Based on the Trained Image Enhancement Model

[0216] This process can be performed by an electronic device, by a processor within the electronic device, or by a chip in the electronic device; the embodiments in this application do not impose any limitations. For ease of description, an electronic device will be used as an example to illustrate the process in detail.

[0217] This process may include the following steps:

[0218] First, the electronic device responds to user operation 1 by acquiring a non-uniformly defocused image 1.

[0219] It should be understood that user action 1 refers to an operation that enhances the image quality. For example, user action 1 could mean that the user... Figure 5 The shooting operation performed in the interface 520 shown. It can also refer to the user's actions in... Figure 6 The operation of clicking the image enhancement control 612 in the interface 610 shown can also refer to other operations, such as a user previewing an image in the gallery. This application embodiment does not limit this to such operations.

[0220] One way to acquire a non-uniformly defocused image 1 is, for example, by an electronic device responding to a user's click. Figure 5 The operation of the control 521 in the interface 520 shown captures the subject 523 in the viewfinder 522, and can obtain a non-uniformly defocused image 1. The non-uniformly defocused image 1 can be referenced. Figure 8 The image shown is a non-uniformly out-of-focus image. This application does not limit the scope of the embodiments shown.

[0221] For example, electronic devices respond to user clicks. Figure 6By clicking the image enhancement control 612 in the interface 610 shown, the non-uniformly out-of-focus image 1 captured by the electronic device through the viewfinder 522 can be obtained.

[0222] Secondly, the electronic device processes the non-uniformly defocused image 1 using the trained image enhancement model to obtain a clear image 1.

[0223] It should be understood that clear image 1 can be used as a reference. Figure 8 The image shown is clear.

[0224] In practice, the trained image enhancement model can process the non-uniformly defocused image 1 based on the learned mapping relationship between the non-uniformly defocused image and the sharp image to obtain the sharp image 1.

[0225] In this embodiment, since the training data can reflect the characteristics of the non-uniformly out-of-focus image mentioned in the above embodiments, the image enhancement model is trained based on the training data. The image enhancement model can learn the mapping relationship between the non-uniformly out-of-focus image and the sharp image well. Then, new data (such as a new non-uniformly out-of-focus image) is input into the trained image enhancement model, and the image enhancement model can perform image quality enhancement processing on the new data based on the above mapping relationship, thereby improving the image quality enhancement effect.

[0226] Furthermore, when processing non-uniformly defocused images based on the image enhancement model in this application embodiment, higher computing resources can be used to process the parts of the non-uniformly defocused image with higher blur levels, while lower computing resources can be used to process the parts of the non-uniformly defocused image with lower blur levels. In this way, the computing resources of electronic devices can be allocated reasonably, avoiding the waste of computing resources.

[0227] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0228] This application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to those of the related embodiments described above, and will not be repeated here.

[0229] This application provides a readable storage medium containing instructions that, when executed by an electronic device, cause the electronic device to perform the technical solution described in the above embodiments. The implementation principle and technical effects are similar and will not be repeated here.

[0230] This application provides a chip for executing instructions. When the chip is running, it executes the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0231] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0232] It should be understood that the term "embodiment" used throughout the specification means that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, various embodiments throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0233] Those skilled in the art will understand that the various numerical designations such as "first," "second," etc., involved in this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application, nor do they indicate the order of sequence.

[0234] In this application, the use of singular pronouns to denote "one or more" rather than "one and only one," unless otherwise specified. In this application, unless otherwise specified, "at least one" is intended to mean "one or more," and "more than" is intended to mean "two or more."

[0235] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. Here, A can be singular or plural, and B can be singular or plural.

[0236] In this document, the term "at least one of..." means all or any combination of the listed items. For example, "at least one of A, B and C" can mean: A exists alone, B exists alone, C exists alone, A and B exist simultaneously, B and C exist simultaneously, and A, B and C exist simultaneously. A can be singular or plural, B can be singular or plural, and C can be singular or plural.

[0237] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0238] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0239] The same or similar parts between the various embodiments in this application can be referred to mutually. In the various embodiments of this application, and in the various implementation methods / methods / implementations within each embodiment, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments and between the various implementation methods / methods / implementations within each embodiment are consistent and can be mutually referenced. The technical features in different embodiments and the various implementation methods / methods / implementations within each embodiment can be combined according to their inherent logical relationships to form new embodiments, implementation methods, methods, or implementation approaches. The above-described embodiments of this application do not constitute a limitation on the scope of protection of this application.

[0240] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the claims. In conclusion, the above description is merely a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, the method includes: In response to a first operation, a first image is acquired. The first operation is an operation to enhance the image quality of the first image. The first image includes a first region and a second region. The second region is composed of a plurality of second sub-regions. The first blur degree of the image in the first region is less than the minimum value of the second blur degree of the images in the plurality of second sub-regions. At least some of the second blur degrees are different. The first image is enhanced using an image enhancement model to obtain a second image. The blur level of the second image is less than the minimum value of a plurality of second blur levels. The image enhancement model is obtained by training on at least one set of training data. Each set of training data includes a clear image and a non-uniform defocused image obtained based on the clear image. The non-uniform defocused image includes a third region and a fourth region. The fourth region is composed of a plurality of fourth sub-regions. The third blur level of the image in the third region is less than the minimum value of the fourth blur level of the images in the plurality of fourth sub-regions. At least some of the fourth blur levels are different. The blur level of the clear image is less than the minimum value of the plurality of fourth blur levels. The process of obtaining the non-uniformly out-of-focus image based on the clear image includes: Based on the clear image, a depth map corresponding to the clear image is obtained. The clear image includes M×N first pixel values, and the depth map includes M×N depth data, where M and N are integers greater than 1. Based on the focusing distance and M×N-1 defocusing distances in the depth map, the diameters of M×N circles of confusion corresponding to the M×N depth data are determined, with one depth data corresponding to one diameter of the circle of confusion. The focusing distance is any depth data randomly selected from the M×N depth data, and the M×N-1 defocusing distances are depth data other than the focusing distance from the M×N depth data. Based on the diameters of the M×N blur circles and the pixel size of the photosensitive element, determine the ratio of the M×N blur circles to the pixel size; The ratios of the M×N blur circles to the pixel size are normalized to obtain M×N blur parameters, which are used to obtain an image mask corresponding to the depth map. The image mask includes the M×N blur parameters, which are used to characterize the degree of blur of the image formed by an object located on the depth data in the electronic device. At least some of the blur parameters have different values. Based on the image mask, the clear image and the uniformly out-of-focus image obtained based on the clear image are fused to obtain the non-uniformly out-of-focus image. The blur degree of the uniformly out-of-focus image is greater than that of the clear image. The uniformly out-of-focus image includes M×N second pixel values, and the non-uniformly out-of-focus image includes M×N third pixel values.

2. The method according to claim 1, characterized in that, The process of fusing the sharp image and the uniformly out-of-focus image obtained based on the sharp image, using the image mask, to obtain the non-uniformly out-of-focus image includes: Perform matrix operations on M×N first pixel values ​​and M×N complementary parameters corresponding to the M×N blur parameters to obtain a first operation result, wherein one blur parameter corresponds to one complementary parameter, and the sum of one blur parameter and its corresponding complementary parameter is 1; Perform matrix operations on the M×N second pixel values ​​and the M×N blur parameters to obtain the second operation result; Based on the sum of the first and second calculation results, M×N third pixel values ​​are obtained to obtain the non-uniform defocused image.

3. An image processing method, characterized in that, include: At least one set of training data is acquired, each set of training data including a clear image and a non-uniformly defocused image obtained based on the clear image, the non-uniformly defocused image including a third region and a fourth region, the fourth region being composed of a plurality of fourth sub-regions, the third blur degree of the image in the third region being less than the minimum value of the fourth blur degree of the images in the plurality of fourth sub-regions, at least some of the fourth blur degrees being different, and the blur degree of the clear image being less than the minimum value of the plurality of fourth blur degrees; The image enhancement model is trained based on at least one set of the training data until the image enhancement model converges; The process of obtaining the non-uniformly out-of-focus image based on the sharp image includes: Based on the clear image, a depth map corresponding to the clear image is obtained. The clear image includes M×N first pixel values, and the depth map includes M×N depth data, where M and N are integers greater than 1. Based on the focusing distance and M×N-1 defocusing distances in the depth map, the diameters of M×N circles of confusion corresponding to the M×N depth data are determined, with one depth data corresponding to one diameter of the circle of confusion. The focusing distance is any depth data randomly selected from the M×N depth data, and the M×N-1 defocusing distances are depth data other than the focusing distance from the M×N depth data. Based on the diameters of the M×N blur circles and the pixel size of the photosensitive element, determine the ratio of the M×N blur circles to the pixel size; The ratios of the M×N blur circles to the pixel size are normalized to obtain M×N blur parameters, which are used to obtain an image mask corresponding to the depth map. The image mask includes the M×N blur parameters, which are used to characterize the degree of blur of the image formed by an object located on the depth data in the electronic device. At least some of the blur parameters have different values. Based on the image mask, the clear image and the uniformly out-of-focus image obtained based on the clear image are fused to obtain the non-uniformly out-of-focus image. The blur degree of the uniformly out-of-focus image is greater than that of the clear image. The uniformly out-of-focus image includes M×N second pixel values, and the non-uniformly out-of-focus image includes M×N third pixel values.

4. An electronic device, characterized in that, include: One or more processors; One or more memory units; The one or more memories store one or more computer programs, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as claimed in claims 1 to 2 or any one of claims 3.

5. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as claimed in claims 1 to 2, or any one of claims 3.

6. A chip, characterized in that, The chip includes: Memory, used to store instructions; A processor for retrieving and executing the instructions from the memory, causing an electronic device on which the chip is mounted to perform the method as claimed in claims 1 to 2, or any one of claims 3.

Images