Image processing method, apparatus and electronic device

Abstract

This application discloses an image processing method, apparatus, and electronic device, belonging to the field of image processing technology. The image processing method includes: obtaining a point spread function (PSF) corresponding to a first image; wherein the PSF is determined based on the PSF corresponding to a captured image, and the PSF corresponding to the captured image is determined based on the diffuse light spots formed by point light sources in the captured scene in the captured image; the similarity between the capture parameters of the first image and the capture parameters of the captured image is greater than or equal to a similarity threshold; and performing deconvolution processing on the first image based on the PSF to obtain a second image.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, specifically relating to an image processing method, apparatus, and electronic device. Background Technology

[0002] The point spread function (PSF) is a function that describes the distribution of an imaging device's response to an ideal point light source, reflecting the optical transmission characteristics of the imaging device. In practical applications of imaging devices, there are several reasons for image blurring, and PSF is one of the key factors affecting image sharpness.

[0003] Currently, PSF is mainly determined through theoretical estimation, which results in low accuracy of PSF. Consequently, when performing image processing based on PSF, problems such as ringing and artifacts are easily caused in the processed image, affecting the image clarity. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, apparatus, and electronic device that can accurately determine the PSF corresponding to the imaging sensor and improve image clarity.

[0005] In a first aspect, embodiments of this application provide an image processing method, including: Obtain the point spread function (PSF) corresponding to the first image; wherein, the PSF is determined based on the PSF corresponding to the captured image, and the PSF corresponding to the captured image is determined based on the diffuse light spots formed by point light sources in the captured scene in the captured image; the similarity between the capture parameters of the first image and the capture parameters of the captured image is greater than or equal to the similarity threshold. The first image is deconvolved using PSF to obtain the second image.

[0006] Secondly, embodiments of this application provide an image processing apparatus, including: The acquisition module is used to acquire the point spread function (PSF) corresponding to the first image; wherein, the PSF is determined based on the PSF corresponding to the captured image, and the PSF corresponding to the captured image is determined based on the diffuse light spot formed by the point light source in the shooting scene in the captured image; the similarity between the shooting parameters of the first image and the shooting parameters of the captured image is greater than or equal to the similarity threshold. The processing module is used to perform deconvolution processing on the first image based on the PSF to obtain the second image.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run programs or instructions to implement the steps of the method as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which, when executed by at least one processor, implements the steps of the method described in the first aspect.

[0011] In this embodiment, the PSF corresponding to the captured image is determined based on the diffuse light spot formed by point light sources in the shooting scene. Since this diffuse light spot can truly reflect the optical characteristics of the imaging sensor, the PSF determined based on the diffuse light spot in the captured image can accurately characterize the optical characteristics of the imaging sensor, effectively avoiding deviations caused by theoretical estimation and improving the accuracy of the PSF. Furthermore, since the shooting parameters of the first image and the captured image are similar, the PSF corresponding to the captured image can be adapted to the optical state corresponding to the first image. Therefore, using this PSF as the PSF corresponding to the first image has high reliability. Thus, performing deconvolution processing on the first image based on this PSF can effectively suppress or eliminate problems such as ringing and artifacts in the first image, improving image clarity and realism, and enhancing image quality. Attached Figure Description

[0012] Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this application; Figure 2 A flowchart illustrating another image processing method provided in this application embodiment; Figure 3 A flowchart illustrating another image processing method provided in this application embodiment; Figure 4 This application provides a schematic diagram of an image processing flow in a static scene. Figure 5 This application provides a schematic diagram of an image processing flow in a vehicle-mounted scenario. Figure 6 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 8This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0015] As mentioned above, the photosensitive surface area (PSF) is one of the key factors affecting image sharpness. Furthermore, the PSF's radius of diffusion, energy distribution, and shape directly reflect the optical distortion, imaging accuracy, and optical path stability of the imaging sensor, serving as an important reference for determining the sensor's quality and the presence of optical defects. In addition, the PSF can be used for sample construction in super-resolution reconstruction. Moreover, the PSF can be applied to industrial point inspection, drone aerial photography, and astronomical telescope imaging. Therefore, accurately determining the PSF is of great significance.

[0016] Currently, the PSF is mainly determined through theoretical estimation, which results in a significant deviation between the estimated PSF and the actual PSF corresponding to the imaging sensor. Consequently, when processing images captured by the imaging sensor based on the theoretical PSF, problems such as ringing and artifacts can easily occur in the processed images, affecting image sharpness.

[0017] Therefore, embodiments of this application provide an image processing method, apparatus, and electronic device that can accurately determine the PSF corresponding to the imaging sensor, effectively solve problems such as ringing and artifacts in images, and improve image clarity.

[0018] The image processing method, apparatus, and electronic device provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0019] It should be noted that the image processing method provided in this application can be executed by an electronic device with image processing capabilities, such as a mobile phone, tablet computer, laptop computer, or PDA. In some embodiments of this application, the electronic device may also support telephoto shooting. Some embodiments of this application use an electronic device as the execution subject to illustrate the image processing method provided in this application.

[0020] The image processing method provided in this application can be applied to scenarios where shooting is done using a telephoto lens. This scenario can be a static scene, such as when an electronic device equipped with an imaging sensor is stationary. Alternatively, the scenario can be a dynamic scene, such as when an electronic device equipped with an imaging sensor is in motion. Whether the scene is static or dynamic, the solution provided in this application can accurately determine the corresponding PSF (Photogram Filter), effectively solving problems such as ringing and artifacts in the image and improving image clarity.

[0021] Figure 1 This is a flowchart illustrating an image processing method provided in an embodiment of this application. This image processing method can be applied to electronic devices that support telephoto shooting, such as telephoto cameras and mobile phones. Figure 1 As shown, the image processing method may include the following steps 110-120.

[0022] Step 110: Obtain the point spread function (PSF) corresponding to the first image; wherein, the PSF is determined based on the PSF corresponding to the captured image, and the PSF corresponding to the captured image is determined based on the diffuse light spots formed by point light sources in the captured scene in the captured image; the similarity between the capture parameters of the first image and the capture parameters of the captured image is greater than or equal to the similarity threshold.

[0023] The first image can be an image obtained by telephoto lens. For example, a user can activate the telephoto lens function of an electronic device to capture the scene and obtain the first image. For example, a user can activate the telephoto lens function of an electronic device via voice, touch, or other means. The first image can include one frame or multiple frames. Each frame has corresponding shooting parameters, which may include, for example, the focal length, aperture, exposure time, shooting temperature, focus position, model, sensor number, driving conditions, and ambient light level of the imaging sensor. Driving conditions may include, for example, vehicle speed and vehicle vibration amplitude. Different image frames can have different shooting parameters, corresponding to different PSFs (Pressure Sensing Frames).

[0024] The PSF corresponding to the first image is the PSF corresponding to the shooting parameters of the first image. This PSF can truly reflect the optical characteristics of the imaging sensor and has higher accuracy.

[0025] In this embodiment, the PSF can be determined based on the PSF corresponding to the captured image. The PSF corresponding to the captured image can be the PSF corresponding to the capture parameters of the captured image. In this embodiment, the similarity between the capture parameters of the first image and the capture parameters of the captured image is greater than or equal to a similarity threshold. In some embodiments, the first image and the captured image can be the same image. When the first image is the captured image, the PSF can be determined first based on the first image, and then deconvolution processing can be performed on the first image based on the PSF.

[0026] When the first image differs from the captured image, the corresponding PSF can be pre-determined and stored using the captured image. For example, a PSF library can be pre-built, and subsequent PSFs obtained from each capture can be stored in this library. This library can store multiple PSFs, each corresponding to a unique capture parameter. The PSF library can be continuously updated, for example, by adding samples as captures are taken to adapt to lens aging, temperature drift, assembly stress changes, etc., becoming more accurate with use. Subsequently, the corresponding PSF can be directly obtained from the PSF library by matching the capture parameters.

[0027] Each of the above methods determines the PSF based on the diffuse spot formed by point light sources in the shooting scene in the captured image. This diffuse spot is closely related to the optical characteristics of the imaging sensor. Therefore, the PSF obtained based on this diffuse spot can truly reflect the optical characteristics of the imaging sensor and has higher accuracy. Moreover, the above methods do not require additional hardware calibration equipment, thereby reducing the cost and operational complexity of PSF determination.

[0028] The point light source here can be a naturally occurring point light source in the shooting scene. For example, it can include bright spots from lights, stars in the night sky, metallic reflections, and eye-like reflections. It can also include bright spots from traffic lights, oncoming vehicle lights, road markings, and metal guardrails. It can also include light sources in the infrared and ultraviolet bands, or laser points and reflected light sources.

[0029] A diffuse spot, also known as a point spread spot, refers to a spatially distributed light spot image formed on an imaging sensor after light emitted from an ideal point light source in the shooting scene, such as stars in the night sky, far-field lights, or specular reflections, passes through the optical lens, aperture, and other components of the imaging sensor. The grayscale distribution of this spot directly reflects the PSF (Power-Side Filter) characteristics of the imaging sensor under that optical condition; that is, the diffusion radius, energy concentration, and symmetry of the spot characterize the imaging sensor's response capability to point light sources. In this embodiment, the diffuse spot is an object identified and extracted from the captured image, and its quality directly affects the accuracy of the PSF. The closer the spot is to the response of an ideal point light source, the more accurately the obtained PSF reflects the optical transmission characteristics of the imaging sensor.

[0030] In some embodiments, after receiving the first image, the electronic device can obtain the PSF corresponding to the shooting parameters of the first image from the PSF library, providing a reliable basis for subsequent image processing.

[0031] Step 120: Perform deconvolution on the first image based on the PSF to obtain the second image.

[0032] For example, the PSF obtained above can be used to perform non-blind deconvolution processing on the first image to obtain the second image. The second image is the first image after eliminating ringing and artifacts, and the clarity of the second image is higher than that of the first image.

[0033] For example, the first image can be considered as the convolution result of the original sharp image and the PSF obtained above, and the noise effect can be superimposed to construct an image degradation model. This model can be represented by the following formula (1): (1) in, This represents the original blurred image, which in this embodiment corresponds to the first image. This represents a clear image; in this embodiment, it corresponds to the second image. Represents convolution operation. This is noise. For example, based on the deconvolution algorithm and combined with the above formula, iterative back-reasoning can be performed to finally obtain the second image. During the iterative back-reasoning process, , and It is known.

[0034] Deconvolution algorithms, such as the Richardson-Lucy (RL) algorithm, are iterative methods based on maximum likelihood estimation. The RL algorithm fully utilizes prior information from the known PSF (Predicted Segmentation File) to progressively approximate a clear, real image. In each iteration, the RL algorithm compares the convolution result of the currently estimated image and the PSF with the first image, calculates corrections, and updates the image estimate. In some embodiments, the regularization parameter of the RL algorithm can be set to 0.001-0.005 to avoid noise amplification and image distortion during iteration. In some embodiments, gradient smoothing constraints can also be added to the RL algorithm to further suppress noise amplification, ringing, and artifacts.

[0035] In some embodiments, the number of iterations of the RL algorithm can be set to 15-25 times. The specific value can be adaptively adjusted according to the blurriness of the first image. The higher the blurriness, the more iterations are needed, with a maximum of 30 iterations. To avoid noise amplification and image distortion during the iteration process, an adaptive threshold filtering process can be performed on the estimated image after each iteration. The filtering threshold can be dynamically adjusted according to the current iteration number to balance the deblurring effect and noise suppression. The termination condition of the iteration process can be set to the variance of the gray values ​​of the images obtained from two adjacent iterations being less than or equal to a preset threshold. The size of the preset threshold can be set according to actual needs. For example, in some embodiments, the preset threshold can be set to 0.001. That is, the iteration terminates when the variance of the gray values ​​of the images obtained from two adjacent iterations is less than or equal to 0.001. The image obtained at this time is the second image, which has effectively solved the blurring problem caused by PSF.

[0036] This embodiment uses non-blind deconvolution processing based on a real PSF, which can effectively restore image details and improve image clarity.

[0037] Considering that the second image may also contain blurring issues caused by motion, noise, etc., and that this blurring is not caused by the PSF, in some embodiments, after deconvolving the first image according to the PSF to obtain the second image, the image processing method may further include the following steps: The second image is input into the image quality reconstruction network, which performs super-resolution reconstruction on the second image to obtain the third image.

[0038] The image reconstruction network here can employ a CNN super-resolution network; this embodiment does not limit the specific structure of the CNN super-resolution network. Super-resolution reconstruction can improve the resolution and sharpness of an image. The sharpness of the third image is higher than that of the second image, and the resolution of the third image is greater than that of the second image.

[0039] Since the blurring issue caused by PSF has been effectively resolved previously, the image reconstruction network can focus on the super-resolution reconstruction task, reducing the network's burden. Based on this, the network structure can be simplified, the training cycle shortened, and the image reconstruction efficiency improved.

[0040] Taking an in-vehicle scenario as an example, to meet the real-time processing requirements of such scenarios, a lightweight image reconstruction network can be used, such as a pruned CNN super-resolution reconstruction network. For instance, redundant neurons or connections can be removed from a traditional CNN super-resolution reconstruction network to reduce the number of parameters and computational load. During pruning, the image reconstruction accuracy should be maintained as much as possible. This approach can effectively utilize the limited computing resources of in-vehicle embedded hardware while ensuring super-resolution performance, reducing memory usage and inference latency, and meeting the real-time image processing requirements during vehicle operation.

[0041] In practical applications, the CNN super-resolution reconstruction network can be optimized or adaptively adjusted according to different driving conditions such as daytime, nighttime, highways, and urban roads to ensure the consistency of reconstruction results in multiple scenarios.

[0042] This embodiment constructs a two-level path for image preprocessing and super-resolution reconstruction based on PSF. First, preprocessing operations such as non-blind deconvolution are performed using the corresponding PSF to obtain the preprocessed image. Then, the preprocessed image is input into the image quality reconstruction network for super-resolution reconstruction, which effectively solves the image blurring problem and improves the image resolution.

[0043] Figure 2 A flowchart illustrating another image processing method provided in this application embodiment. Figure 2 and Figure 1 The difference is that, Figure 2 It also includes steps 210-230.

[0044] Step 210: Identify the candidate diffuse light spots corresponding to the candidate point light sources from the captured images.

[0045] The shooting scene may include one or more point light sources, that is, the captured image may include one or more candidate point light sources, and correspondingly, there may be one or more candidate diffuse light spots.

[0046] For example, a point light source recognition model can be used to extract features and perform semantic analysis on the captured image to identify candidate diffuse light spots corresponding to candidate point light sources. Here, a candidate point light source is a point light source corresponding to a candidate diffuse light spot, and one candidate diffuse light spot corresponds to one candidate point light source.

[0047] This point light source recognition model is pre-trained and can identify various point light sources in the shooting scene, including but not limited to: bright light spots (such as streetlights, car lights, and indoor lighting), bright spots of stars in the night sky, reflective points of metal (such as reflections from chrome-plated car parts and metal guardrails), reflective points of the eyeball, bright spots of traffic lights, oncoming vehicle lights, and reflective road markings. This point light source recognition model identifies objectively existing point light sources in the shooting scene, also known as natural point light sources.

[0048] Considering the different scenarios involved in practical applications, such as vehicle-mounted scenarios (e.g., traffic light highlights, oncoming vehicle lights, road reflective markings, etc.) and non-vehicle-mounted scenarios (e.g., light highlights, starlight highlights, metal reflective points, etc.), in order to accurately identify various point light sources, for example, different scenario datasets can be used to train the point light source recognition model.

[0049] In practical applications, this point light source recognition model can be integrated into the processor of an imaging device. For example, after the telephoto shooting function of an electronic device is activated, the device can load this point light source recognition model for subsequent PSF determination. This embodiment does not limit the specific structure of the point light source recognition model; for example, a convolutional neural network can be used.

[0050] For example, when the point light source recognition model detects a point light source in an image, its corresponding image in the image is marked as a candidate diffuse light spot. These light spots appear as bright areas with a certain grayscale distribution in the image.

[0051] For example, when there are multiple frames of captured images, the candidate point light source and candidate diffuse spot contained in each frame can be identified separately, and the diffuse spot corresponding to the same physical point light source can be matched between multiple frames, providing a basis for subsequent screening of target diffuse spots.

[0052] This embodiment utilizes a point light source recognition model to automatically identify candidate diffuse spots from everyday captured images, providing a basis for determining the PSF (Power Source Filter). This requires no manual intervention or additional hardware equipment, thereby reducing the requirements for users and the cost of determining the PSF.

[0053] Step 220: Determine the target diffuse spot that meets the preset conditions from the candidate diffuse spots; the preset conditions include the first preset conditions, which include at least one of the following: spatial size conditions, brightness distribution conditions, signal-to-noise ratio conditions, shape and symmetry conditions, and light source attribute conditions.

[0054] The quality of the diffuse spot directly affects the accuracy of the PSF. The closer the spot is to the response of an ideal point light source, the more accurately the PSF reflects the optical transmission characteristics of the imaging sensor. To accurately determine the diffuse spot, candidate diffuse spots can be screened, and those that meet preset conditions can be selected as the target diffuse spot for determining the PSF.

[0055] The preset conditions here may include a first preset condition, which may include at least one of spatial size conditions, brightness distribution conditions, signal-to-noise ratio conditions, shape and symmetry conditions, and light source attribute conditions. In some embodiments, the first preset condition may include spatial size conditions, brightness distribution conditions, signal-to-noise ratio conditions, shape and symmetry conditions, and light source attribute conditions. Through multi-dimensional screening, the accuracy of the target diffuse spot can be improved, thereby improving the accuracy of the PSF.

[0056] The spatial size condition limits the pixel range occupied by the candidate diffuse spot in the image, ensuring it is small enough to approximate the response of an ideal point light source. The brightness distribution condition evaluates whether the energy distribution pattern of the candidate diffuse spot conforms to the typical characteristics of a PSF. The signal-to-noise ratio condition evaluates the degree to which the candidate diffuse spot is affected by image noise. The shape and symmetry condition evaluates whether the geometry of the candidate diffuse spot conforms to the isotropic response characteristics of the imaging sensor. The light source attribute condition limits the properties that the point light source corresponding to the candidate diffuse spot should possess.

[0057] In some embodiments of this application, the spatial size condition includes the ratio of the area of ​​the candidate diffuse spot to the area of ​​the captured image being less than or equal to a first threshold. For example, when the ratio of the area of ​​the candidate diffuse spot to the area of ​​the captured image is greater than the first threshold, it indicates that the corresponding point light source is large in physical space or too close to the imaging sensor, and cannot be equivalent to an ideal point light source. Its diffusion pattern may include interference from the shape of the light source itself, easily leading to distortion of the PSF diffusion pattern. Therefore, by limiting the ratio to be less than or equal to the first threshold, spots with acceptable sizes and close to geometric points can be screened, providing a basis for accurate PSF extraction. For example, in some embodiments, the pixel area of ​​the candidate diffuse spot should not exceed 9*9.

[0058] In some embodiments of this application, the brightness distribution conditions may include a decrease in the grayscale value of the candidate diffuse spot from the center to the edge, and a difference between the peak brightness value of the candidate diffuse spot and the background brightness value being less than or equal to a second threshold; wherein, the background brightness value is the pixel brightness within the region associated with the candidate diffuse spot. For example, the decrease in the grayscale value of the candidate diffuse spot from the center to the edge indicates that the brightness of the candidate diffuse spot follows a distribution pattern of high brightness in the center and decreasing brightness around the edges. This distribution pattern is a typical diffusion characteristic formed by an ideal point light source after passing through an imaging sensor, reflecting the continuity and monotonicity of the PSF energy distribution. Therefore, locally overexposed, dark areas, broken, or multi-peak spots can be filtered out, improving the reliability of the spots.

[0059] The background brightness value here can be the pixel brightness within the region associated with the candidate diffuse spot. The region associated with the candidate diffuse spot can be a ring-shaped area of ​​predetermined width surrounding the spot, centered on it. The background brightness value can be the average of the grayscale values ​​of all pixels within this ring-shaped area. This average value accurately reflects the true brightness level of the background. The larger the difference between the peak brightness of the candidate diffuse spot and the background brightness value, the clearer the spot's outline and the less it is affected by background noise. In some embodiments, the second threshold can be set to 30 dB.

[0060] In some embodiments of this application, the signal-to-noise ratio (SNR) condition may include the SNR of the image region where the candidate diffuse spot is located being greater than or equal to a third threshold. A higher SNR in the image region where the candidate diffuse spot is located indicates a stronger signal intensity in that region, and less noise in the image background region (such as Gaussian noise, thermal noise, salt-and-pepper noise, etc.) is likely to alter the core contour and energy distribution of the candidate diffuse spot. In some embodiments, the third threshold may be set to 25 dB.

[0061] In some embodiments of this application, shape and symmetry conditions may include a candidate diffuse spot with a circularity greater than or equal to a fourth threshold, and a coincidence degree between the energy center and geometric center of the candidate diffuse spot greater than or equal to a fifth threshold. The closer the candidate diffuse spot is to a circle, the more accurately it reflects the optical characteristics of the imaging sensor. For example, the circularity can be determined based on the area and perimeter of the candidate diffuse spot; for instance, circularity = 4π × area / perimeter². A larger circularity indicates a closer approximation to a circle or near-circular shape with no significant distortion. The value of circularity ranges from 0 to 1, with 1 representing a standard circle. In some embodiments, the fourth threshold may be set to 0.85.

[0062] The energy center is the weighted center of the grayscale of the candidate diffuse spot, and the geometric center is the shape center of the candidate diffuse spot. The higher the overlap between the energy center and the geometric center of the candidate diffuse spot, the more symmetrical the energy distribution of the candidate diffuse spot. In some embodiments, the fifth threshold can be set to 95%. This can eliminate candidate diffuse spots with large energy shifts, providing morphological assurance for accurate PSF extraction.

[0063] In some embodiments of this application, the light source attribute condition may include candidate diffuse spots formed by a point light source in the shooting scene after passing through an imaging sensor, which is used to acquire the captured image. Natural point light sources, which can be equivalent to ideal point light sources, are typically small in size in physical space or far from the imaging sensor. The light emitted by these sources can be approximated as parallel incident light from a point light source. The diffuse spot formed after passing through the imaging system can directly characterize the PSF characteristics of the imaging sensor under the current optical conditions. This condition can effectively eliminate interference sources unsuitable for PSF calibration, such as surface light sources, large near-field light sources, and strong glare.

[0064] This embodiment quantifies the abstract spot selection criteria into specific technical indicators, making the selection process operable and repeatable. This ensures that the electronic device can automatically and accurately identify diffuse spots that meet the requirements. Through multi-dimensional selection, it can be ensured that the selected diffuse spots conform as closely as possible to the optical characteristics of the imaging sensor, thereby improving the accuracy of the PSF.

[0065] Step 230: Determine the PSF corresponding to the captured image based on the target diffuse spot.

[0066] Once the target diffuse spot is determined, the PSF can be determined based on the target diffuse spot. This PSF can be adaptively determined based on any imaging sensor, effectively solving the problem of poor image processing performance caused by using the same PSF for a class of sensors. It can adapt to the individual differences of a single sensor and improve robustness.

[0067] For example, the PSF can be determined in the following way: Cropping the image region containing the target diffuse spot; Energy normalization is performed on the image region where the target diffuse spot is located to obtain the corresponding PSF.

[0068] For example, an image region of a predetermined size can be cropped with the center point of the target diffuse spot as the center. The cropped image region needs to include the energy distribution range of the target diffuse spot, while also minimizing interference from the background region. In some embodiments, the size of this image region is a 9*9 pixel window.

[0069] For example, the captured image region can be subjected to energy normalization processing to eliminate the influence of light intensity differences under different shooting scenarios on the PSF. For example, energy normalization can be performed using the following formula (2): (2) in, This represents the normalized pixel grayscale value, also known as PSF. pixels within the image region grayscale value, This represents the sum of grayscale values ​​of all pixels within the image region. This PSF accurately reflects the transmission characteristics of the imaging sensor under the current conditions.

[0070] In some embodiments, before performing energy normalization processing on the image region where the target diffuse spot is located, the image region can be denoised first to eliminate the influence of image noise on the true energy distribution of the target diffuse spot. For example, a non-local means (NLM) algorithm can be used to denoise the image region. This algorithm can effectively suppress noise while preserving image edge and texture details. For example, a search window of 9*9 pixels and a similarity window of 3*3 pixels can be preset. The search window defines the range for finding similar pixel blocks, and the similarity window is used to calculate the similarity between pixel blocks. These parameters can be adaptively adjusted according to the noise intensity of the current image. For example, when the noise is strong, the search window or the size of the similarity window can be appropriately increased to enhance the denoising effect; when the noise is weak, smaller parameters are maintained to preserve the original information of the image to the maximum extent. Through the above denoising processing, Gaussian noise, thermal noise, salt-and-pepper noise, etc., in the image can be effectively suppressed while preserving the true energy distribution contour of the PSF.

[0071] The resulting PSF can be represented as a matrix, where the value of each element reflects the intensity distribution of the imaging sensor's response to an ideal point light source.

[0072] This embodiment helps to focus on the target diffuse spot by cropping the image area where the target diffuse spot is located, reducing interference from the background area. Moreover, through energy normalization processing, it can effectively eliminate the influence of different brightness scenes on the PSF amplitude, improving the robustness and accuracy of the PSF.

[0073] This embodiment first identifies candidate diffuse spots in the captured image, and then filters the candidate diffuse spots based on conditions such as spatial size and brightness distribution. This helps to ensure the quality of the filtered diffuse spots and improve the accuracy and reliability of the PSF.

[0074] Taking a static shooting scenario as an example, this static shooting scenario can be an electronic device equipped with an imaging sensor in a relatively stationary state. In this case, the multiple frames of images captured by the electronic device are relatively stable. In this scenario, the captured images can include multiple consecutive image frames. The preset conditions may also include a second preset condition, which may include a stability condition. The stability condition is used to evaluate the degree of change of the candidate diffuse spot between multiple image frames; only candidate diffuse spots that remain stable during the shooting process can be used as target diffuse spots.

[0075] Figure 3 A flowchart illustrating another image processing method provided in this application embodiment. Figure 3 and Figure 2 The difference is that, Figure 2 Step 220 in the middle can be refined as follows: Figure 3 Steps 310-320 in the process.

[0076] Step 310: Determine the first candidate diffuse spot that meets the first preset condition from the candidate diffuse spots.

[0077] For example, a candidate diffuse spot that meets the first preset condition can be determined as the first candidate diffuse spot. There can be one or more first candidate diffuse spots.

[0078] Step 320: For the same first candidate diffuse spot in multiple image frames, determine the stability parameters of the first candidate diffuse spot across multiple image frames. The stability parameters include at least one of the following: centroid offset, shape similarity, and peak brightness change rate.

[0079] For the first candidate diffuse spot identified in the initial screening, it can be further determined whether it meets the stability condition. For example, when multiple frames of images are captured, the electronic device, after identifying the candidate diffuse spots in each image frame, has already matched the candidate diffuse spots in each image frame, obtaining candidate diffuse spots formed by the same physical point light source in multiple image frames. Therefore, in a static scene, the same first candidate diffuse spot in multiple image frames can be analyzed to determine the stability of the first candidate diffuse spot across multiple image frames.

[0080] In static scenes, point light sources are typically bright spots of light, bright spots of stars in the night sky, or reflective points of metal. For example, stability parameters can be determined for the same first candidate diffuse spot across multiple image frames. These stability parameters may include one or more of the following: centroid offset, shape similarity, and peak brightness change rate.

[0081] The center of gravity offset is the pixel-level shift of the grayscale center of gravity of the same first candidate diffuse spot in multiple consecutive image frames. It reflects the magnitude of the relative motion between the device and the light source during the shooting process. The smaller the center of gravity offset, the more stable the position of the first candidate diffuse spot.

[0082] Shape similarity characterizes the degree of morphological consistency of the same first candidate diffuse spot across multiple image frames, reflecting the morphological stability of the first candidate diffuse spot. Higher shape similarity indicates more similar grayscale distribution patterns of the first candidate diffuse spot across multiple image frames. Shape similarity can effectively filter out candidate diffuse spots deformed by factors such as flicker, partial occlusion, or atmospheric disturbance.

[0083] The peak brightness variation rate is the degree of fluctuation in the maximum gray value of the same first candidate diffuse spot across multiple image frames, reflecting the luminous stability of the light source itself. The smaller the peak brightness variation rate, the more stable the brightness of the light source, and the absence of obvious flickering or jumping phenomena.

[0084] Taking stability parameters including centroid offset, shape similarity, and peak brightness change rate as examples, in some embodiments, step 310 above may include the following steps: Based on the centroid coordinates of the first candidate diffuse spot in each image frame, determine the centroid offset of the first candidate diffuse spot between multiple image frames; The first candidate diffuse spot is determined in the image region of each image frame. Based on the average gray value of the image region and the gray value of each pixel in the image region, the shape similarity of the first candidate diffuse spot among multiple image frames is determined. Based on the grayscale peak value of the first candidate diffuse spot in each image frame, determine the peak brightness change rate of the first candidate diffuse spot across multiple image frames.

[0085] For example, the centroid coordinates of the first candidate diffuse spot in each image frame can be calculated using the following formula (3): (3) in, The pixel coordinates of the centroid. This represents the image region centered on the first candidate diffuse spot. The pixel coordinates represent the pixels within this image region. Representing pixels The grayscale value, which is also the pixel value. The brightness value.

[0086] For example, the centroid coordinates of the first candidate diffuse spot in each image frame can be calculated according to the above formula (3). Based on the centroid coordinates of the first candidate diffuse spot in each image frame, the centroid offset of the first candidate diffuse spot between two adjacent image frames can be obtained.

[0087] For example, the shape similarity of the same first candidate diffuse spot across multiple image frames can be calculated using the following formula (4): (4) in, Represents shape similarity, , These are the pixels within the image regions where the first candidate diffuse spot is located in the i-th and j-th image frames, respectively. grayscale value, , The number of image frames. , , respectively, are the average gray values ​​of all pixels in the image region where the first candidate diffuse spot is located in the i-th and j-th image frames.

[0088] For example, based on the grayscale peak value of the first candidate diffuse spot in each image frame, the maximum and minimum grayscale values ​​of the first candidate diffuse spot in all image frames can be determined. Based on the maximum and minimum grayscale values, and the grayscale peak value of the first candidate diffuse spot in each image frame, the peak brightness change rate of the first candidate diffuse spot across multiple image frames can be determined. For example, the peak brightness change rate can be calculated using the following formula (5): (5) in, Represents the rate of change of peak brightness. This represents the maximum grayscale value of the first candidate diffuse spot across all image frames, for example, , The grayscale peak value of the first candidate diffuse spot in the i-th image frame. This represents the minimum grayscale value of the first candidate diffuse spot across all image frames, for example, .

[0089] In this embodiment, in a static scene, based on the first preset condition, the stability of the candidate diffuse spot across multiple image frames is further analyzed from multiple dimensions. This helps to filter out interference such as device motion and light source flicker, improve the quality of the target candidate spot, and thus improve the robustness and accuracy of the PSF.

[0090] Step 330: The first candidate diffuse spot whose stability parameter meets the corresponding threshold is determined as the target diffuse spot that meets the stability condition.

[0091] For example, if the stability parameter of a first candidate diffuse spot meets the corresponding threshold, the first candidate diffuse spot is considered to meet the stability condition.

[0092] For example, when the stability parameter includes the center of gravity offset, the stability parameter can satisfy the corresponding threshold if the center of gravity offset is less than or equal to the center of gravity offset threshold. In some embodiments, the center of gravity offset threshold can be set to 1 pixel.

[0093] For example, when the stability parameter includes shape similarity, the stability parameter satisfying the corresponding threshold can be that the shape similarity is greater than or equal to the similarity threshold. In some embodiments, the similarity threshold can be set to 0.9.

[0094] For example, when the stability parameter includes the peak brightness change rate, the stability parameter satisfying the corresponding threshold can be that the peak brightness change rate is less than or equal to the change rate threshold. In some embodiments, the change rate threshold can be set to 5%.

[0095] In this embodiment, stability conditions are further introduced in static scenes. By combining multiple image frames to determine the stability of candidate diffuse spots, dynamic interference such as device movement and light source flicker can be effectively filtered out, ensuring that the selected spot is stable in the time dimension. This can further improve the quality of the selected spot and thus improve the accuracy of PSF.

[0096] In some embodiments of this application, target diffuse spots that meet the conditions of spatial size, brightness distribution, signal-to-noise ratio, shape and symmetry, light source properties, and stability can be determined from candidate diffuse spots. Through multi-dimensional screening, it helps to ensure that the screened diffuse spots truly and accurately characterize the optical transmission characteristics of the imaging sensor and guarantee the accuracy of the PSF.

[0097] Taking a shooting scene including a dynamic shooting scene as an example, this dynamic shooting scene can be a scene where the electronic device equipped with the imaging sensor is in motion. For example, the dynamic shooting scene can be a vehicle scene, that is, a vehicle equipped with the imaging sensor is in motion. For example, during vehicle movement, a clear image can be obtained through the solution provided in the embodiments of this application, providing assistance for safe driving.

[0098] In dynamic scenarios, due to the significant differences between adjacent image frames, this implementation can determine the PSF based on a single image frame. That is, the captured image can include a single image frame. In this scenario, the preset conditions can also include a third preset condition, which includes the imaging sensor's acceleration being less than or equal to an acceleration threshold. The imaging sensor can be a vehicle-mounted camera that acquires the captured image, and this vehicle-mounted camera can be a camera that supports telephoto shooting. This telephoto camera can capture images of vehicles ahead, road signs, traffic lights, etc., from a distance.

[0099] For dynamic scenes, in order to effectively filter out light spot distortion caused by device jitter, the third preset condition may, for example, include the acceleration of the imaging sensor being less than or equal to an acceleration threshold, that is, the vehicle's acceleration being less than or equal to an acceleration threshold. In some embodiments, the acceleration threshold may be set to... In this way, the light spot distortion caused by equipment jitter can be effectively filtered out, and a stable diffuse light spot can still be obtained under single-frame conditions, resulting in an accurate PSF and meeting the real-time requirements of the vehicle.

[0100] In some embodiments of this application, in dynamic scenes, preset conditions may include spatial size conditions, brightness distribution conditions, signal-to-noise ratio conditions, shape and symmetry conditions, light source attribute conditions, and acceleration conditions. The acceleration condition refers to the acceleration of the imaging sensor being less than or equal to an acceleration threshold. The resulting target diffuse spot approximates an ideal point light source with a high signal-to-noise ratio, effectively eliminating interfering targets such as strong glare, random road surface reflections, and large-area spots, thus improving the reliability of the target diffuse spot.

[0101] In some embodiments of this application, the PSF corresponding to the captured image includes the PSF corresponding to the capturing parameters of the captured image. After step 230, the image processing method may further include the following steps: The quality of the second image is evaluated to obtain the quality evaluation index of the second image; The confidence level of the PSF is determined based on the quality assessment indicators; If the confidence level is greater than or equal to the confidence threshold, obtain the shooting parameters corresponding to the captured image; Associate and store the shooting parameters and their corresponding PSF.

[0102] For example, a referenceless image quality evaluation algorithm can be used, that is, without referencing the original sharp image, the quality of the second image is evaluated based solely on the features of the second image itself, resulting in quality evaluation metrics. Quality evaluation metrics may include, but are not limited to, ringing artifacts, edge distortion, sharpness consistency, and noise level. Ringing artifacts detect edge oscillations in the second image caused by excessive deconvolution or PSF mismatch. Artifacts detect the presence of unrealistic textures, halos, or structures in the second image. Edge distortion evaluates the sharpness and fidelity of the edges in the second image, determining whether there is distortion or blurring. Sharpness consistency assesses the uniformity of sharpness across different regions of the second image, identifying areas that are overly sharp or soft. Noise level detects the intensity of residual noise in the second image, determining whether it has been excessively amplified during the deconvolution process.

[0103] By evaluating the quality of the second image, index values ​​in different dimensions can be obtained. The confidence level of the PSF can be determined based on the index values ​​of each index. For example, each index value can be compared with a reference range to obtain a score for each index. The scores of each index can be weighted and summed to obtain a total score, which serves as the confidence level of the PSF. A higher total score indicates a higher confidence level. Of course, the confidence level of the PSF can also be calculated in other ways; this embodiment does not limit this method.

[0104] For example, when the confidence level of a PSF is greater than or equal to a confidence threshold, it indicates that the currently obtained PSF is reliable; otherwise, the PSF is considered abnormal and is deleted. When the confidence level of a PSF is greater than or equal to the confidence threshold, the PSF can be associated with the corresponding shooting parameters and stored in the PSF library for later use. When the confidence level of a PSF is less than the confidence threshold, it can switch to the best PSF used in the past. The PSF can then be recalibrated after the shooting function is enabled. For example, in a vehicle-mounted scenario, when the confidence level of a PSF is less than the confidence threshold, it can switch to the best PSF used in the past to ensure the current image quality output does not affect the vehicle's driving safety decisions. The PSF can be recalibrated when the vehicle reaches a stable road section, such as a smooth road or a uniformly lit area.

[0105] In some embodiments, in a vehicle-mounted scenario, the shooting parameters may also include driving conditions such as vehicle speed and vehicle vibration amplitude.

[0106] In some embodiments of this application, after the electronic device's shooting function is activated, the electronic device can obtain the shooting parameters corresponding to the current shooting scene and match these shooting parameters with shooting parameters in the PSF library. If there is no PSF in the PSF library corresponding to the current shooting parameters, a prompt message can be output, prompting the user that there is no PSF in the PSF library corresponding to the current shooting parameters and asking whether it is necessary to determine the PSF based on the captured image. After receiving confirmation input from the user, the electronic device can first determine the PSF according to the scheme of the above embodiments, and then perform image processing based on the PSF. For example, if no confirmation input from the user is received, the PSF closest to the current shooting parameters can be selected from the PSF library.

[0107] In this embodiment, after obtaining the second image, a quality assessment is performed on the second image, and the reliability of the PSF is verified based on the quality assessment results. Abnormal PSFs are automatically filtered out, which can effectively prevent abnormal PSFs from polluting the PSF library and ensure the reliability of PSFs in the PSF library.

[0108] In some embodiments of this application, step 110 may include the following steps: Obtain the mapping relationship between shooting parameters and PSF, as well as the first shooting parameters corresponding to the first image; Based on the first shooting parameters, determine the PSF corresponding to the first image from the mapping relationship.

[0109] For example, the mapping relationship between shooting parameters and PSF can be obtained from the PSF library, and the first shooting parameters corresponding to the first image can be matched with each shooting parameter in the mapping relationship. The shooting parameter with the highest similarity to the first shooting parameter can be selected, and the PSF corresponding to the shooting parameter with the highest similarity can be used as the PSF corresponding to the first image.

[0110] For example, if the shooting parameters in the PSF library all deviate significantly from the first shooting parameter, the initial PSF preset at the factory can be used.

[0111] This embodiment can pre-establish a mapping relationship between shooting parameters and PSF, and then directly obtain the corresponding PSF based on the shooting parameters, thereby improving the image processing efficiency.

[0112] The following is combined Figure 4 The image processing procedure for static scenes is explained.

[0113] Step 410: Device initialization.

[0114] Once the shooting function is enabled, the electronic device can load a point light source recognition model and a PSF library. The PSF library stores multiple PSFs and their corresponding shooting parameters. This point light source recognition model is pre-trained to adapt to common point light sources such as bright lights, starlight, metallic reflections, and specular reflections from the eye. The electronic device can also simultaneously load an initial PSF preset at the factory, which can be used when the user takes their first shot after the device leaves the factory.

[0115] Step 420: Acquire multiple frames of images.

[0116] Users can activate the telephoto camera's shooting function to continuously capture multiple frames of images. The electronic device can record shooting parameters for each frame, such as focal length, aperture, exposure time, shooting temperature, and focus position.

[0117] Step 430: Select the target diffuse spot from multiple image frames.

[0118] The target diffuse spot needs to meet at least one of the following conditions: spatial size, brightness distribution, signal-to-noise ratio, shape and symmetry, light source properties, and stability.

[0119] Step 440: Determine PSF based on the target diffuse spot.

[0120] For example, noise reduction and energy normalization can be performed on the image region where the target diffuse spot is located to obtain the corresponding PSF without the need for additional hardware devices.

[0121] Step 450: Perform non-blind deconvolution processing on the first image obtained from the daily shooting scene based on PSF to obtain the second image.

[0122] Step 460: Input the second image into the image quality reconstruction network to perform super-resolution reconstruction and obtain the third image.

[0123] Step 470: Perform quality assessment on the third image to obtain quality assessment indicators, and determine the confidence level of PSF based on the quality assessment indicators.

[0124] This section describes the quality assessment of the third image. In practical applications, the quality assessment can also be performed on the second image.

[0125] Step 480: Is the confidence level of PSF greater than or equal to the confidence threshold? If yes, proceed to step 490; otherwise, proceed to step 4100.

[0126] Step 490: Bind the PSF with the corresponding shooting parameters and save it to the PSF library.

[0127] Step 4100: Delete PSF.

[0128] This embodiment is adaptable to everyday shooting scenarios with telephoto cameras, requiring no specialized calibration equipment or operators. It leverages the diffused light spots formed by point light sources in typical shooting scenarios to achieve accurate PSF measurement and calibration, reducing calibration costs and user requirements. Furthermore, by screening point light sources across multiple dimensions, the stability and reliability of the light sources used for PSF calibration are ensured, thereby improving PSF accuracy.

[0129] The following is combined Figure 5 The image processing procedure in dynamic scenes is explained. A vehicle-mounted scene is used as an example of a dynamic scene.

[0130] Step 510: Device initialization.

[0131] Once the shooting function is enabled, the vehicle can load a point light source recognition model and a PSF library. The PSF library stores multiple PSFs and their corresponding shooting parameters. This point light source recognition model is pre-trained to adapt to point light sources in vehicle scenarios such as traffic light highlights, oncoming vehicle lights, road reflective markings, and reflective points on metal guardrails. The vehicle can also simultaneously load an initial PSF preset at the factory, which can be used when the user takes the first shot after the vehicle leaves the factory.

[0132] Step 520: Acquire a single frame image.

[0133] Users can activate the shooting function of the vehicle's telephoto camera to capture a single frame of the current road scene. The vehicle can record the shooting parameters of that frame, such as focal length, aperture, exposure time, shooting temperature, focus position, and corresponding driving condition data, such as vehicle speed, vehicle vibration amplitude, and ambient light intensity.

[0134] Step 530: Filter the target diffuse spot from a single image frame.

[0135] The target diffuse spot needs to meet at least one of the following conditions: spatial size, brightness distribution, signal-to-noise ratio, shape and symmetry, light source properties, and vehicle acceleration. The vehicle acceleration condition may include, for example, a vehicle acceleration less than or equal to an acceleration threshold.

[0136] Step 540: Determine PSF based on the target diffuse spot.

[0137] For example, noise reduction and energy normalization can be performed on the image region where the target diffuse spot is located to obtain the corresponding PSF without the need for additional hardware devices.

[0138] Step 550: Perform non-blind deconvolution processing on the first image obtained from the vehicle scene captured by PSF to obtain the fourth image.

[0139] Step 560: Input the fourth image into the image quality reconstruction network for super-resolution reconstruction to obtain the fifth image.

[0140] Step 570: Perform quality assessment on the fifth image to obtain quality assessment indicators, and determine the confidence level of PSF based on the quality assessment indicators.

[0141] This section describes the quality assessment of the fifth image; the quality assessment of the fourth image can also be performed here.

[0142] Step 580: Is the confidence level of PSF greater than or equal to the confidence threshold? If yes, proceed to step 590; otherwise, proceed to step 5100.

[0143] Step 590: Bind the PSF with the corresponding shooting parameters and save it to the PSF library.

[0144] Step 5100: Delete PSF.

[0145] For example, if the current PSF is unreliable, it can be deleted, and a PSF closest to the current shooting parameters can be selected from the PSF library to ensure the current image quality output. Once the vehicle has traveled to a stable road section, such as a smooth, straight road with uniform lighting, the PSF calibration process can be restarted.

[0146] This embodiment is adaptable to in-vehicle scenarios, meeting the real-time imaging needs of dynamic vehicle movement. It achieves accurate PSF measurement without requiring multi-frame acquisition, improving image processing efficiency and meeting the requirements of real-time in-vehicle imaging. It eliminates the need for specialized calibration equipment and operators, relying solely on single-frame images captured during vehicle movement to complete accurate PSF measurement and calibration, reducing calibration costs and user requirements. This solution dynamically adapts to changes in the optical state of in-vehicle lenses and different driving conditions, exhibiting good robustness. It effectively removes image blurring caused by vehicle vibration, speed changes, and other interference, significantly improving the detail clarity of targets such as forward vehicle recognition and road sign monitoring, while balancing real-time performance and safety. With no additional hardware costs, it is easy to promote and apply to mass-produced in-vehicle devices, providing reliable image data support for intelligent driving assistance decision-making.

[0147] In addition to the aforementioned everyday shooting scenarios and vehicle-mounted scenarios, in some embodiments, this image processing method can also be applied to fixed-point detection scenarios in the industrial field. For example, by optimizing the point light source recognition model, the ability to filter point light sources under dust and stray light interference can be enhanced. Combined with industrial sensor data such as vibration, temperature, and dust concentration, the filtering of PSF calibration areas can be optimized to meet the needs of precision component defect detection, production line monitoring, and other requirements.

[0148] In some embodiments, the image processing method can also be applied to long-range aerial photography scenarios using drones. For example, it can combine the advantages of the static and dynamic scenarios mentioned above to achieve real-time calibration of the PSF during dynamic flight, adapting to scenarios such as high-altitude long-range photography and terrain monitoring.

[0149] In some embodiments, this image processing method can also be applied to astronomical telescope imaging scenarios, optimize point light source recognition models in low-light environments, extract stars in the starry sky as point light sources, combine multiple image frames, improve the accuracy of PSF, and assist in the reconstruction of high-definition astronomical images.

[0150] In some embodiments, transfer learning can be used to quickly adapt a point light source recognition model trained in one scenario, such as an in-vehicle scenario, to a new scenario, such as an industrial scenario, reducing the data requirements and costs of retraining.

[0151] In some embodiments, techniques such as quantization compression and layer pruning can be used to further reduce the computational power of the point light source recognition model, thereby making it compatible with low-end embedded devices, such as smart home cameras and wearable devices, and improving the robustness and universality of the solution.

[0152] It should be noted that the image processing method provided in this application embodiment can be executed by an image processing device or a processing module within that image processing device for executing the image processing method. This application embodiment uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application embodiment.

[0153] Figure 6 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application.

[0154] like Figure 6 As shown, the image processing apparatus 600 may include: The acquisition module 601 is used to acquire the point spread function (PSF) corresponding to the first image; wherein, the PSF is determined based on the PSF corresponding to the captured image, and the PSF corresponding to the captured image is determined based on the diffuse light spot formed by the point light source in the captured scene in the captured image; the similarity between the capture parameters of the first image and the capture parameters of the captured image is greater than or equal to the similarity threshold. The processing module 602 is used to perform deconvolution processing on the first image based on the PSF to obtain the second image.

[0155] In this embodiment, the PSF corresponding to the captured image is determined based on the diffuse light spot formed by point light sources in the shooting scene. Since this diffuse light spot can truly reflect the optical characteristics of the imaging sensor, the PSF determined based on the diffuse light spot in the captured image can accurately characterize the optical characteristics of the imaging sensor, effectively avoiding deviations caused by theoretical estimation and improving the accuracy of the PSF. Furthermore, since the shooting parameters of the first image and the captured image are similar, the PSF corresponding to the captured image can be adapted to the optical state corresponding to the first image. Therefore, using this PSF as the PSF corresponding to the first image has high reliability. Thus, performing deconvolution processing on the first image based on this PSF can effectively suppress or eliminate problems such as ringing and artifacts in the first image, improving image clarity and realism, and enhancing image quality.

[0156] In some possible implementations of the embodiments of this application, the image processing apparatus 600 may further include: The recognition module is used to identify candidate diffuse light spots corresponding to candidate point light sources from the captured image; the candidate point light sources include at least one. The determination module is used to determine the target diffuse spot that meets the preset conditions from the candidate diffuse spots; the preset conditions include a first preset condition, which includes at least one of the following: spatial size condition, brightness distribution condition, signal-to-noise ratio condition, shape and symmetry condition, and light source attribute condition; and to determine the PSF corresponding to the captured image based on the target diffuse spot.

[0157] In some possible implementations of the embodiments of this application, the shooting scene includes a static shooting scene, and the captured image includes multiple consecutive image frames; the preset conditions also include a second preset condition, which includes a stability condition; The module is specifically used for: From the candidate diffuse spots, determine the first candidate diffuse spot that meets the first preset condition; For the same first candidate diffuse spot in multiple image frames, determine the stability parameters of the first candidate diffuse spot across multiple image frames. The stability parameters include at least one of the following: centroid offset, shape similarity, and peak brightness change rate. The first candidate diffuse spot whose stability parameter meets the corresponding threshold is determined as the target diffuse spot that meets the stability condition.

[0158] In some possible implementations of the embodiments of this application, the stability parameters include centroid offset, shape similarity, and peak brightness change rate; The module is specifically used for: Based on the centroid coordinates of the first candidate diffuse spot in each image frame, determine the centroid offset of the first candidate diffuse spot between multiple image frames; Determine the image region of the first candidate diffuse spot in each image frame; determine the shape similarity of the first candidate diffuse spot across multiple image frames based on the average gray value of the image region and the gray value of each pixel in the image region. Based on the grayscale peak value of the first candidate diffuse spot in each image frame, determine the peak brightness change rate of the first candidate diffuse spot across multiple image frames.

[0159] In some possible implementations of the embodiments of this application, the shooting scene includes a dynamic shooting scene, and the captured image includes a single image frame; the preset conditions also include a third preset condition, which includes that the acceleration of the imaging sensor is less than or equal to an acceleration threshold, and the imaging sensor is used to acquire the captured image.

[0160] In some possible implementations of the embodiments of this application, the spatial size condition includes the ratio of the area of ​​the candidate diffuse spot to the area of ​​the captured image being less than or equal to a first threshold. The brightness distribution conditions include that the gray values ​​of the candidate diffuse spot decrease from the center to the edge, and the difference between the peak brightness of the candidate diffuse spot and the background brightness value is less than or equal to a second threshold; wherein, the background brightness value is the pixel brightness in the region associated with the candidate diffuse spot. The signal-to-noise ratio (SNR) condition includes that the SNR of the image region where the candidate diffuse spot is located is greater than or equal to a third threshold; The shape and symmetry conditions include that the circularity of the candidate diffuse spot is greater than or equal to the fourth threshold, and that the coincidence between the energy center and the geometric center of the candidate diffuse spot is greater than or equal to the fifth threshold. The light source attribute conditions include that the candidate diffuse light spot is formed by the point light source in the shooting scene after passing through the imaging sensor, which is used to acquire the shooting image.

[0161] In some possible implementations of the embodiments of this application, the determining module is specifically used for: Cropping the image region containing the target diffuse spot; Energy normalization is performed on the image region where the target diffuse spot is located to obtain the corresponding PSF.

[0162] In some possible implementations of the embodiments of this application, the PSF corresponding to the captured image includes the PSF corresponding to the shooting parameters of the captured image; The image processing apparatus 600 may further include: The evaluation module is used to evaluate the quality of the second image after the determination module determines the PSF corresponding to the captured image based on the target diffuse spot, and obtain the quality evaluation index of the second image. The determination module is also used to determine the confidence level of the PSF based on quality assessment indicators; The acquisition module 601 is also used to acquire the shooting parameters corresponding to the captured image when the confidence level is greater than or equal to the confidence level threshold; The storage module is used to associate and store shooting parameters and their corresponding PSF.

[0163] The image processing device in this application embodiment can be a device or a component in an electronic device, such as an integrated circuit or a chip. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0164] The electronic device in this application embodiment can be a terminal with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0165] The image processing apparatus provided in this application embodiment can achieve... Figures 1 to 5 The various processes in the image processing method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0166] like Figure 7 As shown, this application embodiment also provides an electronic device 700, including a processor 701 and a memory 702. The memory 702 stores programs or instructions that can run on the processor 701. When the program or instructions are executed by the processor 701, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0167] It should be noted that the electronic devices in the embodiments of this application include the mobile terminals and non-mobile terminals mentioned above.

[0168] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0169] The electronic device 800 includes, but is not limited to, components such as: radio frequency unit 801, network module 802, audio output unit 803, input unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, and processor 810.

[0170] Those skilled in the art will understand that the electronic device 800 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 810 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The structure of the electronic device 800 shown does not constitute a limitation on the electronic device 800. The electronic device 800 may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be described in detail here.

[0171] The processor 810 is used to obtain the point spread function (PSF) corresponding to the first image; wherein the PSF is determined based on the PSF corresponding to the captured image, and the PSF corresponding to the captured image is determined based on the diffuse light spots formed by point light sources in the captured scene in the captured image; the similarity between the capture parameters of the first image and the capture parameters of the captured image is greater than or equal to a similarity threshold; the first image is deconvolved based on the PSF to obtain the second image.

[0172] In this embodiment, the PSF corresponding to the captured image is determined based on the diffuse light spot formed by point light sources in the shooting scene. Since this diffuse light spot can truly reflect the optical characteristics of the imaging sensor, the PSF determined based on the diffuse light spot in the captured image can accurately characterize the optical characteristics of the imaging sensor, effectively avoiding deviations caused by theoretical estimation and improving the accuracy of the PSF. Furthermore, since the shooting parameters of the first image and the captured image are similar, the PSF corresponding to the captured image can be adapted to the optical state corresponding to the first image. Therefore, using this PSF as the PSF corresponding to the first image has high reliability. Thus, performing deconvolution processing on the first image based on this PSF can effectively suppress or eliminate problems such as ringing and artifacts in the first image, improving image clarity and realism, and enhancing image quality.

[0173] In some possible implementations of embodiments of this application, the processor 810 is specifically used for: Identify candidate diffuse light spots corresponding to candidate point light sources from the captured images; the candidate point light sources include at least one. The target diffuse spot that meets the preset conditions is determined from the candidate diffuse spots; the preset conditions include a first preset condition, which includes at least one of the following: spatial size condition, brightness distribution condition, signal-to-noise ratio condition, shape and symmetry condition, and light source attribute condition; Based on the target diffuse spot, determine the PSF corresponding to the captured image.

[0174] In some possible implementations of the embodiments of this application, the shooting scene includes a static shooting scene, and the captured image includes multiple consecutive image frames; the preset conditions also include a second preset condition, which includes a stability condition; The 810 processor is specifically used for: From the candidate diffuse spots, determine the first candidate diffuse spot that meets the first preset condition; For the same first candidate diffuse spot in multiple image frames, determine the stability parameters of the first candidate diffuse spot across multiple image frames. The stability parameters include at least one of the following: centroid offset, shape similarity, and peak brightness change rate. The first candidate diffuse spot whose stability parameter meets the corresponding threshold is determined as the target diffuse spot that meets the stability condition.

[0175] In some possible implementations of the embodiments of this application, the stability parameters include centroid offset, shape similarity, and peak brightness change rate; The 810 processor is specifically used for: Based on the centroid coordinates of the first candidate diffuse spot in each image frame, determine the centroid offset of the first candidate diffuse spot between multiple image frames; Determine the image region of the first candidate diffuse spot in each image frame; determine the shape similarity of the first candidate diffuse spot across multiple image frames based on the average gray value of the image region and the gray value of each pixel in the image region. Based on the grayscale peak value of the first candidate diffuse spot in each image frame, determine the peak brightness change rate of the first candidate diffuse spot across multiple image frames.

[0176] In some possible implementations of the embodiments of this application, the shooting scene includes a dynamic shooting scene, and the captured image includes a single image frame; the preset conditions also include a third preset condition, which includes that the acceleration of the imaging sensor is less than or equal to an acceleration threshold, and the imaging sensor is used to acquire the captured image.

[0177] In some possible implementations of the embodiments of this application, the spatial size condition includes the ratio of the area of ​​the candidate diffuse spot to the area of ​​the captured image being less than or equal to a first threshold. The brightness distribution conditions include that the gray values ​​of the candidate diffuse spot decrease from the center to the edge, and the difference between the peak brightness of the candidate diffuse spot and the background brightness value is less than or equal to a second threshold; wherein, the background brightness value is the pixel brightness in the region associated with the candidate diffuse spot. The signal-to-noise ratio (SNR) condition includes that the SNR of the image region where the candidate diffuse spot is located is greater than or equal to a third threshold; The shape and symmetry conditions include that the circularity of the candidate diffuse spot is greater than or equal to the fourth threshold, and that the coincidence between the energy center and the geometric center of the candidate diffuse spot is greater than or equal to the fifth threshold. The light source attribute conditions include that the candidate diffuse light spot is formed by the point light source in the shooting scene after passing through the imaging sensor, which is used to acquire the shooting image.

[0178] In some possible implementations of embodiments of this application, the processor 810 is specifically used for: Cropping the image region containing the target diffuse spot; Energy normalization is performed on the image region where the target diffuse spot is located to obtain the corresponding PSF.

[0179] In some possible implementations of the embodiments of this application, the PSF corresponding to the captured image includes the PSF corresponding to the shooting parameters of the captured image; The 810 processor is specifically used for: After determining the PSF corresponding to the captured image based on the target diffuse spot, the quality of the second image is evaluated to obtain the quality evaluation index of the second image. The confidence level of the PSF is determined based on the quality assessment indicators; If the confidence level is greater than or equal to the confidence threshold, obtain the shooting parameters corresponding to the captured image; Associate and store the shooting parameters and their corresponding PSF.

[0180] It should be understood that, in this embodiment, the input unit 804 may include a graphics processing unit (GPU) 8041 and a microphone 8042. The GPU 8041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 806 may include a display panel 8061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0181] The memory 809 can be used to store software programs and various data. The memory 809 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 809 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 809 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0182] Processor 810 may include one or more processing units; optionally, processor 810 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 810.

[0183] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0184] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0185] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0186] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0187] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0188] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0189] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the related technology, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0190] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, include: Obtain the point spread function (PSF) corresponding to the first image; wherein the PSF is determined based on the PSF corresponding to the captured image, and the PSF corresponding to the captured image is determined based on the diffuse light spots formed by point light sources in the captured scene in the captured image; the similarity between the capture parameters of the first image and the capture parameters of the captured image is greater than or equal to a similarity threshold. The first image is deconvolved based on the PSF to obtain the second image.

2. The method according to claim 1, characterized in that, The method further includes: Identify candidate diffuse light spots corresponding to candidate point light sources from the captured images; the candidate point light sources include at least one; From the candidate diffuse spots, a target diffuse spot that meets the preset conditions is determined; the preset conditions include a first preset condition, which includes at least one of the following: spatial size condition, brightness distribution condition, signal-to-noise ratio condition, shape and symmetry condition, and light source attribute condition; Based on the target diffuse spot, determine the PSF corresponding to the captured image.

3. The method according to claim 2, characterized in that, The shooting scene includes a static shooting scene, and the captured image includes multiple consecutive image frames; the preset conditions also include a second preset condition, which includes a stability condition. The step of determining the target diffuse spot that meets the preset conditions from the candidate diffuse spots includes: From the candidate diffuse spots, determine the first candidate diffuse spot that satisfies the first preset condition; For the same first candidate diffuse spot in multiple image frames, determine the stability parameter of the first candidate diffuse spot among multiple image frames. The stability parameter includes at least one of the following: centroid offset, shape similarity, and peak brightness change rate. The first candidate diffuse spot whose stability parameter satisfies the corresponding threshold is determined as the target diffuse spot that satisfies the stability condition.

4. The method according to claim 3, characterized in that, The stability parameters include centroid offset, shape similarity, and peak brightness change rate; For the same first candidate diffuse spot in multiple image frames, determining the stability parameter of the first candidate diffuse spot across multiple image frames includes: Based on the centroid coordinates of the first candidate diffuse spot in each image frame, determine the centroid offset of the first candidate diffuse spot between multiple image frames; The image region of the first candidate diffuse spot is determined in each image frame; the shape similarity of the first candidate diffuse spot among multiple image frames is determined based on the average gray value of the image region and the gray value of each pixel in the image region. Based on the grayscale peak value of the first candidate diffuse spot in each image frame, the peak brightness change rate of the first candidate diffuse spot among multiple image frames is determined.

5. The method according to claim 2, characterized in that, The shooting scene includes a dynamic shooting scene, and the captured image includes a single image frame; the preset conditions also include a third preset condition, which includes that the acceleration of the imaging sensor is less than or equal to an acceleration threshold, and the imaging sensor is used to acquire the captured image.

6. The method according to any one of claims 2-5, characterized in that, The spatial size condition includes the ratio of the area of ​​the candidate diffuse spot to the area of ​​the captured image being less than or equal to a first threshold. The brightness distribution conditions include that the gray value of the candidate diffuse spot decreases from the center to the edge, and the difference between the peak brightness value of the candidate diffuse spot and the background brightness value is less than or equal to a second threshold; wherein, the background brightness value is the pixel brightness in the region associated with the candidate diffuse spot; The signal-to-noise ratio condition includes that the signal-to-noise ratio of the image region where the candidate diffuse spot is located is greater than or equal to a third threshold. The shape and symmetry conditions include that the circularity of the candidate diffuse spot is greater than or equal to a fourth threshold, and that the coincidence between the energy center and the geometric center of the candidate diffuse spot is greater than or equal to a fifth threshold. The light source attribute conditions include that the candidate diffuse light spot is formed by a point light source in the shooting scene passing through an imaging sensor, and the imaging sensor is used to acquire the shooting image.

7. The method according to any one of claims 2-5, characterized in that, The step of determining the PSF corresponding to the captured image based on the target diffuse light spot includes: Cropping the image region containing the target diffuse spot; The image region containing the target diffuse spot is subjected to energy normalization processing to obtain the corresponding PSF.

8. The method according to any one of claims 2-5, characterized in that, The PSF corresponding to the captured image includes the PSF corresponding to the shooting parameters of the captured image; After determining the PSF corresponding to the captured image based on the target diffuse spot, the method further includes: The quality of the second image is evaluated to obtain the quality evaluation index of the second image; The confidence level of the PSF is determined based on the aforementioned quality assessment indicators; If the confidence level is greater than or equal to the confidence threshold, the shooting parameters corresponding to the captured image are obtained; The shooting parameters and their corresponding PSF are stored together.

9. An image processing apparatus, characterized in that, include: An acquisition module is used to acquire the point spread function (PSF) corresponding to the first image; wherein the PSF is determined based on the PSF corresponding to the captured image, and the PSF corresponding to the captured image is determined based on the diffuse light spots formed by point light sources in the shooting scene in the captured image; the similarity between the shooting parameters of the first image and the shooting parameters of the captured image is greater than or equal to a similarity threshold. The processing module is used to perform deconvolution processing on the first image based on the PSF to obtain the second image.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1-8.

Images