Image processing method, electronic device, storage medium and computer program product

Abstract

The present disclosure relates to an image processing method, an electronic device, a storage medium and a computer program product. The image processing method comprises: obtaining phase difference information of each image block in a current frame image; the phase difference information indicating an offset between a current focusing position and an expected focusing position; determining a moire region of the current frame image based on the phase difference information of each image block; and performing moire removal processing on the moire region to obtain a moire-removed image. The embodiments of the present disclosure can improve the moire processing rate.

Description

Technical Field

[0001] This disclosure relates to image processing technology, and more particularly to an image processing method, electronic device, storage medium, and computer program product. Background Technology

[0002] With the rapid development of mobile phone cameras, digital cameras, and surveillance cameras, capturing high-quality images has become the core of market competition. Despite continuous innovation in camera lenses, image sensors, and hardware architecture driven by modern technology, moiré patterns remain a persistent problem in photography. Moiré patterns arise from the interference between the pixel arrangement frequency of the image sensor and the frequency of repeating patterns in the scene, and are particularly pronounced in high-resolution imaging.

[0003] In related technologies, either the entire RAW image is demoired or the detected high-frequency regions are demoired, which has the problems of high computing power and inaccurate detection. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an image processing method, electronic device, storage medium, and computer program product that can improve the moiré pattern processing speed.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, including...

[0006] Obtain the phase difference information of each image block in the current frame image; the phase difference information indicates the offset between the current focus position and the desired focus position;

[0007] Based on the phase difference information of each of the image blocks, the moiré region of the current frame image is determined;

[0008] The moiré pattern area is then subjected to demoiré processing to obtain the demoiré processed image.

[0009] In some embodiments, determining the moiré region of the current frame image based on the phase difference information of each of the image blocks includes:

[0010] Based on the phase difference information of each image block, determine whether each image block has moiré patterns;

[0011] Based on the phase difference information of each image block, the difference between the phase difference between the current frame image and the previous frame image at the same image block position is determined;

[0012] If the absolute value of the difference is greater than a first preset threshold and the number of image blocks with moiré patterns in the current frame image is greater than a second preset threshold, the moiré pattern region is determined based on the image blocks with moiré patterns.

[0013] In some embodiments, determining whether each image block has moiré patterns based on the phase difference information of each image block includes:

[0014] The phase difference information of each image block is processed for similarity to obtain the similarity curve of each image block;

[0015] The image patch corresponding to the similarity curve with multiple peak points is identified as the image patch with moiré pattern.

[0016] In some embodiments, determining whether each image block has moiré patterns based on the phase difference information of each image block includes:

[0017] Obtain the current focus distance and the expected focus distance of each image block, determined based on the phase difference information of each image block;

[0018] The image block corresponding to the sum of the current focus distance and the desired focus distance being greater than the preset maximum travel distance is identified as the image block with moiré pattern.

[0019] In some embodiments, obtaining the phase difference information of each image block in the current frame image includes:

[0020] When the current frame image is acquired using phase focusing, the phase difference information of each image block output during the phase focusing process is obtained.

[0021] In some embodiments, the current frame image includes a RAW image; the step of performing demoiring processing on the moiré region to obtain a demoired image includes:

[0022] The moiré region of the RAW image is subjected to demoiré processing to obtain a demoiré processed RGB image.

[0023] In some embodiments, determining the moiré region of the current frame image based on the phase difference information of each of the image blocks includes:

[0024] After performing brightness and color correction on the current frame image to correct lens shadows, the moiré region of the current frame image is determined based on the phase difference information of each image block.

[0025] In some embodiments, performing demoiring processing on the moiré region to obtain a demoired image includes:

[0026] Interpolate and restore the information of the green channel in the moiré region to obtain the restored green channel result;

[0027] Based on the recovery result of the green channel, the information of the red channel and the information of the blue channel in the moiré region are interpolated and recovered respectively to obtain the image after demoiré processing.

[0028] According to a second aspect of the present disclosure, an electronic device is provided, comprising:

[0029] The phase difference acquisition unit is configured to acquire phase difference information of each image block in the current frame image; the phase difference information indicates the offset between the current focus position and the desired focus position.

[0030] The moiré pattern determination unit is configured to determine the moiré pattern region of the current frame image based on the phase difference information of each of the image blocks;

[0031] The moiré processing unit is configured to perform demoiré processing on the moiré region to obtain a demoiré processed image.

[0032] According to a third aspect of the present disclosure, an electronic device is provided, including a processor; a memory for storing computer programs or instructions; wherein the processor executes the computer programs or instructions to implement the steps of the method described in the first aspect above.

[0033] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, the storage medium storing a computer program or instructions that, when executed by a processor, implement the steps of the method described in the first aspect above.

[0034] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0035] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0036] In this embodiment, the phase difference information indicates the offset between the current focus position and the desired focus position; the moiré region of the current frame image is determined based on the phase difference information of each image block. Therefore, this embodiment utilizes the phase difference information during the focusing process to determine the moiré region, eliminating the need for additional phase difference calculations, thus saving computational power and improving the moiré processing speed.

[0037] Furthermore, compared to the existing technology where the identification of high-frequency regions is prone to misjudgment due to the inability to accurately distinguish moiré areas from other textures, the embodiments of this disclosure utilize phase difference information to obtain more accurate moiré regions, thereby reducing the damage to the details of the image contained in non-moiré regions, preserving the original details and clarity of the current frame image to the greatest extent, and thus improving the overall image quality.

[0038] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0040] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0041] Figure 2 This is a schematic diagram of an existing moiré pattern image according to an exemplary embodiment.

[0042] Figure 3 This is a schematic diagram of L and R images output according to an exemplary embodiment.

[0043] Figure 4 This is a schematic diagram illustrating the process of image processing in different image domains, as shown in an exemplary embodiment.

[0044] Figure 5 This is a schematic diagram of the similarity curve of an image patch without moiré patterns, as illustrated in an exemplary embodiment.

[0045] Figure 6 This is a schematic diagram of the similarity curve of an image patch with moiré patterns, as illustrated in an exemplary embodiment.

[0046] Figure 7 This is a schematic diagram of a reference pixel grid showing the gradient values ​​in each direction at position B(i,j) in an exemplary embodiment.

[0047] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0048] Figure 9 This is a structural diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0050] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 1 As shown, the image processing method is applied to an electronic device, and the electronic device performs the image processing method mainly by the following steps:

[0051] In step S11, the phase difference information of each image block in the current frame image is obtained; the phase difference information indicates the offset between the current focus position and the desired focus position.

[0052] In step S12, the moiré region of the current frame image is determined based on the phase difference information of each image block;

[0053] In step S13, the moiré pattern area is subjected to demoiré processing to obtain the demoiré processed image.

[0054] In this embodiment of the disclosure, the image processing method is applied to a moiré pattern scene. For example, a moiré pattern image may be generated when a mobile phone camera uses autofocus for selfies, video calls, or when a digital camera captures images, thus placing the user in a moiré pattern scene.

[0055] Figure 2 This is a schematic diagram of an existing moiré image according to an exemplary embodiment. Figure 2 As shown, objects containing repeating patterns, such as textiles, plaid shirts, or certain types of building materials, are prone to generating moiré patterns. For this moiré pattern scenario, the image processing method of this disclosure embodiment can be used to reduce moiré interference and improve image quality.

[0056] It should be noted that the image processing method is applied to electronic devices, including electronic devices with cameras, such as mobile phones, tablets, laptops, wearable devices, digital cameras, or various monitoring devices, etc. This disclosure does not limit the scope of the application.

[0057] In step S101, the current frame image can be divided into multiple image blocks. For example, the current frame image can be divided into M*N image blocks. Here, M and N are both positive integers greater than 1, and this embodiment of the disclosure does not impose any limitations on this.

[0058] It should be noted that the smaller the image blocks in the current frame, the higher the accuracy, but the greater the computational cost. Here, the length and width of the image blocks are no less than 3% of the size of the current frame image. This ensures that each image block retains sufficient detail information, thereby enabling a better determination of the phase difference information of each image block.

[0059] In this embodiment of the disclosure, the electronic device using the image processing method can establish a communication connection with other devices. Here, obtaining the phase difference information of multiple image blocks in the current frame image may include: receiving phase difference information sent by other devices with which a communication connection has been established.

[0060] For example, an electronic device that applies this image processing method includes a monitoring device, and other devices include a mobile phone, which can receive phase difference information sent by the mobile phone.

[0061] It should be noted that the electronic device can also determine phase difference information for use in phase focusing during the acquisition of the current frame image using phase focusing. In some embodiments, obtaining the phase difference information of each image block in the current frame image may further include: when acquiring the current frame image using phase focusing, obtaining the phase difference information of each image block output during the phase focusing process.

[0062] In other words, the embodiments of this disclosure can directly obtain the phase difference information output during the phase focusing process and locate the moiré pattern region based on the phase difference information, without needing to calculate the phase difference information.

[0063] Here, the electronic device may have a phase detection autofocus module, in which each pixel on the sensor can be specifically divided into two parts. When light passes through the lens and is projected onto these pixels, due to the pixel division, each part receives different light information, thus forming two different images. The phase difference information can then be obtained by calculating the phase difference between the two different images using the phase detection autofocus module.

[0064] For example, Figure 3 This is a schematic diagram of L and R images output according to an exemplary embodiment. Figure 3 As shown, pixels can be divided into R pixels and L pixels. R pixels collect light rays from the right, and L pixels collect light rays from the left. All L pixels in each image block output as an L image, and all R pixels in each image block output as an R image, thus enabling the generation of L and R images. Here, the phase difference between the L and R images is calculated using a phase detection autofocus module, allowing the output of phase difference information for different image blocks.

[0065] In this embodiment of the disclosure, the phase detection autofocus module can be a standalone module in the electronic device, or it can be integrated into the phase detection autofocus module. This embodiment of the disclosure does not limit this.

[0066] Understandably, compared to existing technologies that process the entire image, especially high-resolution images or high-frequency regions, which involve large computational loads and long processing times, this disclosure proposes to directly utilize the phase difference information output during its own phase focusing process to locate moiré patterns. This eliminates the need for additional phase difference calculations, thus saving computational resources, reducing computational load, and improving the processing efficiency of moiré patterns. Furthermore, unlike high-frequency region detection which may fail to accurately distinguish moiré patterns from other textures and lead to misjudgments, this disclosure determines moiré patterns based on the phase difference output by the phase detection module. This improves the accuracy of moiré pattern region determination, thereby better reducing moiré interference and improving the overall image quality.

[0067] In step S102, after determining the phase difference information of each image block, the moiré region of the current image can be determined based on the phase difference information of each image block.

[0068] It should be noted that, in the embodiments of this disclosure, it can first determine whether it is necessary to perform demoiring processing on the current frame image based on the phase difference information of each image block, and if it is determined that it is necessary to perform demoiring processing on the current frame image, the moiré region of the current frame image is determined.

[0069] Here, in some embodiments, the image blocks that have moiré patterns are determined based on the phase difference information of each image block, and the moiré pattern region of the current image is determined if it is determined that at least one image block has moiré patterns.

[0070] In other embodiments, the difference between the phase difference between the previous frame image and the current frame image blocks at the same image block position can be used to determine whether each image block has a moiré pattern, and if it is determined that at least one image block has a moiré pattern, the moiré pattern region of the current image can be determined.

[0071] In other embodiments, the moiré region of the current image can also be determined based on the number of image blocks with moiré patterns and the difference between them.

[0072] In this embodiment of the disclosure, the phase difference information indicates the offset between the current focus position and the desired focus position. Here, autofocus can be performed based on the phase difference information, specifically by determining the direction and distance the lens needs to move based on the phase difference information, so as to adjust the current focus position to the desired focus position and complete autofocus.

[0073] In other words, the embodiments of this disclosure no longer use phase difference information for autofocus, but use phase difference information to determine the moiré pattern area. That is, the embodiments of this disclosure reuse the phase difference information in the phase focusing process in the process of determining the moiré pattern area, and therefore do not require additional computing power to calculate.

[0074] In step S301, after determining the moiré pattern region, the moiré pattern region can be demoiré processed to obtain the demoiré processed image.

[0075] It should be noted that directly performing demoiring processing on the current frame image or on the detected high-frequency regions may result in blurring or loss of image details in non-moiring areas. Therefore, embodiments of this disclosure can first determine the moiring regions and then perform demoiring processing on them, thereby reducing the damage to image details contained in non-moiring areas.

[0076] In this embodiment of the disclosure, images after demoiring processing can be obtained based on phase difference information in different image domains. In some embodiments, the current frame image includes a RAW image; the step of performing demoiring processing on the moiré region to obtain the demoired image includes: performing demoiring processing on the moiré region of the RAW image to obtain a demoired RGB image.

[0077] In other words, embodiments of this disclosure can determine moiré regions in the RAW domain and perform demoiré processing in the RGB domain. For example, the demosaic module included in the electronic device can be used to perform demoiré processing on the moiré regions of the RAW image to obtain a demoiré processed RGB image. Specifically, the position information of the moiré regions determined in the RAW domain can be sent to the demosaic module, enabling the demosaic module to perform demoiré processing on the moiré regions of the RAW image.

[0078] The location information of the moiré pattern region includes the coordinate information of the moiré pattern region.

[0079] In this embodiment of the disclosure, the determination of the moiré region is performed in the RAW domain, and the demoiré processing is performed in the RGB domain, both of which are performed before the image processing in the YUV domain. Therefore, the positioning of the moiré region and the demoiré processing are not affected by the image processing in the YUV domain, thus enabling better determination of the moiré region and demoiré processing of the moiré region based on the phase difference information.

[0080] In some embodiments, determining the moiré region of the current frame image based on the phase difference information of each of the image blocks includes:

[0081] After performing brightness and color correction on the current frame image to correct lens shadows, the moiré region of the current frame image is determined based on the phase difference information of each image block.

[0082] In this embodiment of the disclosure, brightness and color correction is performed on the current frame image to correct lens shadows, which can reduce the inconsistency in color and brightness between the center and the periphery of the current frame image caused by uneven optical refraction of the lens.

[0083] Understandably, after performing brightness and color correction on the current frame image to correct lens shadows, detecting the moiré region of the current frame image based on the phase difference information of each image block can reduce the impact of inconsistent brightness between the center and the surrounding area on the detection of the moiré region, thus achieving better detection results.

[0084] For example, Figure 4 This is a schematic diagram illustrating the image processing flow in different image domains, as shown in an exemplary embodiment. Figure 4 As shown, the current frame image may include a RAW image.

[0085] In the RAW domain, the bad pixel correction module is used to repair bad pixels or damaged pixels in the RAW image; the black level correction module is used to correct the black level value in the RAW image; the digital gain module is used to adjust the brightness or contrast of the RAW image; the lens shading correction module is used to perform brightness or color correction on the RAW image; and the white balance module is used to perform white balance processing on the RAW image.

[0086] In the RGB domain, the demosaic module is used to remove moiré patterns from the RAW image, resulting in a demoiré-processed RGB image. The Color Correction Matrix (CCM) module corrects color deviations in the RGB image by adjusting the channel values ​​of the red, green, and blue channels; the Gamma correction module adjusts the Gamma value of the RGB image to change the brightness distribution; and the color space conversion module converts the RGB color space to the YUV color space.

[0087] In the YUV domain, the noise reduction module is used to reduce noise and interference; the tone mapping module is used to adjust the tone distribution; the color processing module is used to adjust the color distribution, enhance color contrast and saturation, etc.; the sharpening module is used to enhance edge and detail information; the encoding module is used to encode to obtain an encoded image; and the format conversion module is used to convert the encoded image into an image in the Joint Photographic Experts Group (JPEG) format.

[0088] It should be noted that, as Figure 4As shown, in this embodiment, after the lens shadow correction module, based on the phase difference information moiré region, a demoiselle module is used to process the moiré region to obtain a demoiselle-processed RGB image. Therefore, the image processing method of this embodiment is not affected by modules such as the color processing module or sharpening module in the YUV domain.

[0089] In this embodiment of the disclosure, the moiré pattern region can be filtered or interpolated to obtain the processed image, and this embodiment of the disclosure does not limit this.

[0090] It is understood that in this embodiment, the phase difference information indicates the offset between the current focus position and the desired focus position; the moiré region of the current frame image is determined based on the phase difference information of each image block. Therefore, this embodiment utilizes the phase difference information during the focusing process to determine the moiré region, eliminating the need for additional phase difference calculations, thus saving computational power and improving the moiré processing speed.

[0091] Furthermore, compared to the existing technology where the identification of high-frequency regions is prone to misjudgment due to the inability to accurately distinguish moiré areas from other textures, the embodiments of this disclosure utilize phase difference information to obtain more accurate moiré regions, thereby reducing the damage to the details of the image contained in non-moiré regions, preserving the original details and clarity of the current frame image to the greatest extent, and thus improving the overall image quality.

[0092] In some embodiments, determining the moiré region of the current frame image based on the phase difference information of each of the image blocks includes:

[0093] Based on the phase difference information of each image block, determine whether each image block has moiré patterns;

[0094] Based on the phase difference information of each image block, the difference between the phase difference between the current frame image and the previous frame image at the same image block position is determined;

[0095] If the absolute value of the difference is greater than a first preset threshold and the number of image blocks with moiré patterns in the current frame image is greater than a second preset threshold, the moiré pattern region is determined based on multiple image blocks with moiré patterns.

[0096] In this embodiment of the disclosure, when encountering moiré interference, the phase difference between two consecutive frames changes significantly, resulting in low similarity between the two frames. Therefore, the presence of moiré patterns in an image block can be determined based on the phase difference between the two consecutive frames at the same image block position, i.e., the difference in phase difference between the two consecutive frames at the same position, thereby determining whether the image block is in a moiré scene.

[0097] Here, a first preset threshold can be set according to the actual situation. If the absolute value of the difference is greater than the first preset threshold, the image block is determined to have moiré patterns, thus indicating that it is in a moiré scene and requires demoiré processing. If the absolute value of the difference is less than or equal to the first preset threshold, the image block is determined to not have moiré patterns, thus indicating that it is in a non-moiré scene and does not require demoiré processing.

[0098] For example, the phase difference between the current frame image and the previous frame image at the same image block position can be represented as PD_value(n) and PD_value(n-1), respectively, and the first preset threshold can be represented as PD_TH. Here, when |PD_value(n-1)-PD_value(n)|>PD_TH, it is determined that the corresponding image block has moiré patterns.

[0099] In this embodiment of the disclosure, the number of image blocks with moiré patterns can be obtained by counting the number of such image blocks. It should be noted that the second preset threshold can be set based on the false detection rate. In this way, by comparing the number of image blocks with moiré patterns with the second preset threshold, it is possible to better determine whether a scene is in a moiré pattern.

[0100] Here, if the number of image patches with moiré patterns is less than or equal to the second preset threshold, it indicates that there is false detection of moiré patterns or that the image patches with moiré patterns have little impact. Therefore, if the number of image patches with moiré patterns is less than or equal to the second preset threshold, it can be determined that the scene is not a moiré pattern scene and no moiré pattern removal processing is required.

[0101] If the number of image blocks with moiré patterns is greater than the second preset threshold, that is, if the number of abnormal image blocks is greater than the second preset threshold, it can be determined that the scene is a moiré pattern scene and moiré removal processing is required.

[0102] In this embodiment of the disclosure, if the absolute value of the phase difference is greater than the first preset threshold and the number of image blocks with moiré patterns in the current frame image is greater than the second preset threshold, it indicates that the scene is a moiré pattern scene and moiré pattern removal processing is required. That is, it is necessary to determine the moiré pattern region based on the image blocks with moiré patterns and perform moiré pattern removal processing on the moiré pattern region.

[0103] The above-described method of determining the moiré region based on image blocks with moiré patterns may include: using the image regions of the image blocks with moiré patterns as the moiré region. When there are multiple image blocks with moiré patterns, the sum of the regions of the multiple image blocks with moiré patterns is used as the moiré region.

[0104] It is understandable that determining whether a scene is a moiré pattern based on the absolute value of the phase difference being greater than the first preset threshold and the number of image blocks with moiré patterns in the current frame being greater than the second preset threshold, i.e., determining whether to perform demoiré processing, can make the determination of moiré pattern scenes more reliable, and thus make the moiré pattern regions determined by image blocks with moiré patterns more reliable.

[0105] In some embodiments, determining whether each image block has moiré patterns based on the phase difference information of each image block includes:

[0106] The phase difference information of each image block is processed for similarity to obtain the similarity curve of each image block;

[0107] The image patch corresponding to the similarity curve with multiple peak points is identified as the image patch with moiré pattern.

[0108] In this embodiment of the disclosure, the Sum of Absolute Differences (SAD) algorithm is used to perform similarity processing on the phase difference information of each image block to obtain the similarity curve of each image block.

[0109] It should be noted that the similarity curve of the image patch can be obtained using formula (1). Where, I L and I R Let L and R be the image and ...

[0110]

[0111] Here, let the similarity cost = -costSAD, then the similarity curve of block(m,n) is as follows: Figure 5 As shown. Figure 5 This is a schematic diagram of the similarity curve of an image patch without moiré patterns, as illustrated in an exemplary embodiment. Figure 5 The horizontal axis represents the phase difference, or offset, of image patches, while the vertical axis represents the similarity. Figure 5 As can be seen, the location of the peak point can be found through the SAD algorithm, and the horizontal coordinate of the peak point can correspond to the desired focus position.

[0112] It should be noted that, Figure 5 The similarity curve in the image is a curve with a single peak. If moiré patterns are encountered, the similarity curve will exhibit multiple peaks.

[0113] For example, Figure 6 This is a schematic diagram illustrating the similarity curves of image patches with moiré patterns, as shown in an exemplary embodiment. Figure 6 As shown, when an image patch has moiré patterns, the similarity curve of the image patch has multiple peak points. Therefore, whether an image patch has moiré patterns can be represented by whether the similarity curve of the image patch has multiple peak points.

[0114] In this embodiment of the disclosure, after determining the similarity curves of each image block, peak point detection can be performed on the similarity curves to determine whether they have a single peak point or multiple peak points. If the similarity curve has multiple peak points, then the image block corresponding to the similarity curve has moiré patterns. That is, based on the characteristics of the similarity curves, it is possible to determine whether the image block has moiré patterns.

[0115] It should be noted that peak point detection of the similarity curve may include: obtaining the similarity h[i] of the current point in the similarity curve; when the similarity h[i] of the current point is greater than the similarity h[i-1] of the previous point and the similarity h[i] of the current point is greater than the similarity h[i+1] of the next point, a peak point is determined to be detected, that is, i is the peak point.

[0116] It is understood that the embodiments of this disclosure can determine image blocks with moiré patterns by peak point detection, that is, determine that the image block has moiré patterns when multiple peak points are detected, and determine that the image block does not have moiré patterns when a single peak point is detected.

[0117] In some embodiments, determining whether each image block has moiré patterns based on the phase difference information of each image block includes:

[0118] Obtain the current focus distance and the expected focus distance of each image block, determined based on the phase difference information of each image block;

[0119] The image block corresponding to the sum of the current focus distance and the desired focus distance being greater than the preset maximum travel distance is identified as the image block with moiré pattern.

[0120] In this embodiment of the present disclosure, when the electronic device uses phase focusing to acquire the current frame image, since the current focus distance and the desired focus distance are determined during phase focusing to achieve autofocus, this embodiment of the present disclosure can directly obtain the current focus distance and the desired focus distance, thus reducing additional computing power.

[0121] Of course, it is also possible to obtain phase difference information and determine the current focusing distance and the desired focusing distance based on the phase difference. The embodiments of the present disclosure do not limit this.

[0122] In the embodiments of the present disclosure, it is possible to determine whether a moiré pattern exists in an image block based on the sum of the current focusing distance and the desired focusing distance of each image block corresponding to the same image block. That is, an image block corresponding to the sum of the current focusing distance and the desired focusing distance outside the preset stroke range can be determined as an image block with a moiré pattern. An image block corresponding to the sum of the current focusing distance and the desired focusing distance within the preset stroke range can be determined as an image block without a moiré pattern.

[0123] Among them, the preset stroke range corresponds to a preset maximum stroke distance and a preset minimum stroke distance.

[0124] For example, when the sum of the current focusing distance and the desired focusing distance of the same image block is greater than the preset maximum stroke distance and less than the preset minimum stroke distance, it is determined that the same image block has a moiré pattern; when the sum of the current focusing distance and the desired focusing distance of the same image block is between the preset maximum stroke distance and the preset minimum stroke distance, that is, within the preset stroke range, it is determined that the same image block does not have a moiré pattern.

[0125] Exemplarily, the current focusing distance can be Current_pos, and the desired focusing distance can be PD_value1 (this desired focusing distance can be Figure 5 the abscissa corresponding to the peak point in

[0126] Here, for Current_pos and PD_value1 with the same unit, when OTP MACRO < Current_pos + PD_value1, or Current_pos + PD_value1 < OTP INF, it can be determined that the corresponding image block is the image block with a moiré pattern.

[0127] It should be noted that for Current_pos and PD_value1 with different units, when OTP MACRO < Current_pos + PD_value1 * DCC, or Current_pos + PD_value1 * DCC < OTP INF, it can be determined that the corresponding image block is the image block with a moiré pattern. Among them, DCC is unit conversion.

[0128] In some embodiments, performing demoiring processing on the moiré region to obtain a demoired image includes:

[0129] Interpolate and restore the information of the green channel in the moiré region to obtain the restored green channel result;

[0130] Based on the recovery result of the green channel, the information of the red channel and the information of the blue channel in the moiré region are interpolated and recovered respectively to obtain the image after demoiré processing.

[0131] In this embodiment, considering that the sampling frequency of the green channel information is higher than that of the red and blue channels, the green channel information retains more high-frequency information during the interpolation restoration process, resulting in weaker aliasing compared to the red and blue channels. Furthermore, the color classifications of red (R), green (G), and blue (B) have a high correlation. Therefore, this embodiment proposes to first perform interpolation restoration on the green channel information, and then perform interpolation restoration on the red and blue channel information based on the restoration result of the green channel. This approach yields a better image after moiré removal processing.

[0132] In this embodiment of the disclosure, the green channel information can be interpolated and restored using formula (2) to obtain the restored green channel result. in, and G1, G2, G3, and G4 are the inverse gradient weights in the North (N), South (S), West (W), and East (E) directions, respectively, and G1, G2, G3, and G4 are the estimated values ​​in each direction.

[0133] NSWE

[0134]

[0135] The inverse gradient weights for each direction can be calculated from the corresponding directional gradient values. The reference pixel grid used to calculate the gradient values ​​for each direction at a given position B(i,j) is as follows: Figure 7 As shown.

[0136] Let P N and P W Let P be the gradient values ​​in the north and west directions at (i, j), respectively. P can be determined using formulas (3) and (4). N and P W .

[0137]

[0138] Among them, the gradient value P in the south direction S Gradient value P in the east direction E The calculation of P Nand P W Similar. P S With P N In the expression, each corresponding element is symmetrical about the horizontal direction of (i, j), P E With P W The corresponding elements in the expression are symmetrical about the vertical direction of (i, j).

[0139] After obtaining the gradient values ​​in each direction, the inverse gradient weights in each direction can be obtained using formulas (5), (6), (7), and (8).

[0140]

[0141] Here, the estimated values ​​of each direction in the G direction can be approximately determined by using second-order Newton interpolation with formulas (9) to (12).

[0142] G N =G i+1,j -f[(i+1,j),(i-2,j)[-2f[(i+1,j),(i-2,j),(i-4,j)[(9)

[0143] G S =G i-1,j -f[(i-1,j),(i+2,j)]-2f[(i-1,j),(i+2,j),(i+4,j)](10)

[0144] G W =G i,j+1 -f[(i,j+1),(i,j-2)]+2f[(i,j+1),(i,j-2),(i,j-4)](11)

[0145] G E =G i,j-1 +f[(i,j-1),(i,j+2)]-2f[(i,j-1),(i,j+2),(i,j+4)](12)

[0146] Here, the first-order difference quotient f[(i+1,j),(i-2,j)] and the second-order difference quotient f[(i+1,j),(i-2,j),(i-4,j)] in the N direction can be calculated using formulas (13) and (14).

[0147]

[0148] It should be noted that the estimated values ​​of the S, W, and E directions can be obtained using the same method.

[0149] In this embodiment of the disclosure, after obtaining the recovery result of the green channel, the information of the red channel and the information of the blue channel can be interpolated and recovered respectively.

[0150] Here, a weighted average of the chromatic aberration between the R and G or B and G channels along the focusing direction is used, i.e., northwest (NW), southwest (SW), northeast (NE), and southeast (SE). The chromatic aberration at position B is used for reconstruction. For example, here, we can use formula (15) to obtain... Formula (16) can be used to obtain in, and These are the inverse gradient weights in the NW, SW, NE, and SE directions, respectively. It is C NW C SW C NE and C SE The weighted average.

[0151]

[0152] Here, the gradient value P in the northwest direction can be obtained using formulas (17) to (20). NW The gradient value P in the southwest direction SW The gradient value P in the northeast direction NE The gradient value P in the southeast direction SE .

[0153]

[0154] Here, the inverse gradient weights in the northwest direction can be obtained using formulas (21) to (24). Inverse gradient weights in the southwest direction Inverse gradient weights in the northeast direction Inverse gradient weights in the southeast direction

[0155]

[0156] Here, the color difference C in the northwest direction can be obtained using formulas (25) to (29). NW Color difference C in the southwest direction SW Color difference C in the northeast direction NE Color difference C in the southeast direction SE .

[0157]

[0158]

[0159] Here, formulas (29) and (30) can be used to obtain the information of the red channel at position G. Information from the Blue Channel Thus, through the above calculation process, the image after demoiring and restoration of complete color is finally recovered.

[0160]

[0161] In this embodiment of the disclosure, during the process of obtaining the demoired image by performing the above-described interpolation algorithm on the moiré region, a conventional interpolation algorithm can be used on the non-moiré region to obtain a conventionally interpolated image. The conventionally interpolated image and the demoired image are then stitched together to obtain the final image. This approach preserves image information in the non-moiré region while effectively suppressing moiré patterns in the moiré region, thus improving the quality and clarity of the final image.

[0162] It should be noted that the final image may include an RGB image. The non-moiré and moiré regions together constitute the entire image region of the current image frame. The aforementioned traditional interpolation algorithms include nearest neighbor interpolation or bilinear interpolation, etc., and this disclosure does not limit these methods.

[0163] This disclosure also proposes an electronic device. Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Figure 8 As shown, the electronic device mainly includes:

[0164] The phase difference acquisition unit 201 is configured to acquire phase difference information of each image block in the current frame image; the phase difference information indicates the offset between the current focus position and the desired focus position.

[0165] The moiré pattern determination unit 202 is configured to determine the moiré pattern region of the current frame image based on the phase difference information of each of the image blocks;

[0166] The moiré processing unit 203 is configured to perform moiré removal processing on the moiré region to obtain a moiré-removed image.

[0167] In some embodiments, the moiré pattern determination unit 202 is further configured to determine whether each of the image blocks has moiré patterns based on the phase difference information of each of the image blocks; determine the difference between the phase difference between the current frame image and the previous frame image at the same image block position based on the phase difference information of each of the image blocks; and determine the moiré pattern region based on the image blocks with moiré patterns when the absolute value of the difference is greater than a first preset threshold and the number of image blocks with moiré patterns in the current frame image is greater than a second preset threshold.

[0168] In some embodiments, the moiré pattern determination unit 202 is further configured to perform similarity processing on the phase difference information of each of the image blocks to obtain a similarity curve for each of the image blocks; and to determine the image block corresponding to the similarity curve having multiple peak points as the image block having moiré patterns.

[0169] In some embodiments, the moiré pattern determination unit 202 is further configured to obtain the current focus distance and the expected focus distance of each image block determined based on the phase difference information of each image block; and to determine the image block with moiré pattern as the image block whose sum of the current focus distance and the expected focus distance is greater than a preset maximum travel distance.

[0170] In some embodiments, the phase difference acquisition unit 201 is further configured to acquire phase difference information of each image block output during the phase focusing process when the current frame image is acquired using phase focusing.

[0171] In some embodiments, the moiré processing unit 203 is further configured such that the current frame image includes a RAW image; and performs demoiré processing on the moiré region of the RAW image to obtain a demoiré processed RGB image.

[0172] In some embodiments, the moiré pattern determination unit 202 is further configured to determine the moiré pattern region of the current frame image based on the phase difference information of each of the image blocks after performing brightness and color correction on the current frame image to correct lens shadows.

[0173] In some embodiments, the moiré processing unit 203 is further configured to interpolate and restore the information of the green channel in the moiré region to obtain the restoration result of the green channel; based on the restoration result of the green channel, interpolate and restore the information of the red channel and the information of the blue channel in the moiré region respectively to obtain the image after demoiré processing.

[0174] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0175] Figure 9 This is a structural diagram of an electronic device according to an exemplary embodiment. For example, the electronic device may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0176] Reference Figure 9The electronic device may include one or more of the following components: processing component 902, memory 904, power supply component 906, multimedia component 908, audio component 910, input / output (I / O) interface 912, sensor component 914, and communication component 916.

[0177] Processing component 902 typically controls the overall operation of an electronic device, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 902 may include one or more processors 920 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 902 may include one or more modules to facilitate interaction between processing component 902 and other components. For example, processing component 902 may include a multimedia module to facilitate interaction between multimedia component 908 and processing component 902.

[0178] Memory 904 is configured to store various types of data to support operation on the electronic device. Examples of such data include at least one of the following: instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, and videos. Memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0179] Power supply component 906 provides power to various components of an electronic device. Power supply component 906 may include at least one of the following: a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0180] Multimedia component 908 includes a screen that provides an output interface between the electronic device and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 908 includes a front-facing camera and / or a rear-facing camera. When the electronic device is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0181] Audio component 910 is configured to output and / or input audio signals. For example, audio component 910 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 904 or transmitted via communication component 916. In some embodiments, audio component 910 also includes a speaker for outputting audio signals.

[0182] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0183] Sensor assembly 914 includes one or more sensors for providing state assessments of various aspects of the electronic device. For example, sensor assembly 914 may detect the on / off state of the electronic device, the relative positioning of components such as the display and keypad of the electronic device, changes in the position of the electronic device or a component within the electronic device, the presence or absence of user contact with the electronic device, the orientation or acceleration / deceleration of the electronic device, and temperature changes of the electronic device. Sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 914 may also include an optical sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 914 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0184] Communication component 916 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic device can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 916 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 916 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0185] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0186] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including executable instructions or a computer program, which can be executed by a processor 920 of an electronic device to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0187] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform any of the image processing methods described above in the embodiments of this disclosure. For example, the method includes:

[0188] Obtain the phase difference information of each image block in the current frame image; the phase difference information indicates the offset between the current focus position and the desired focus position;

[0189] Based on the phase difference information of each of the image blocks, the moiré region of the current frame image is determined;

[0190] The moiré pattern area is then subjected to demoiré processing to obtain the demoiré processed image.

[0191] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the image processing methods described above in this disclosure.

[0192] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0193] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: Obtain the phase difference information of each image block in the current frame image; The phase difference information indicates the offset between the current focus position and the desired focus position; Based on the phase difference information of each of the image blocks, the moiré region of the current frame image is determined; The moiré pattern area is then subjected to demoiré processing to obtain the demoiré processed image.

2. The method according to claim 1, characterized in that, Determining the moiré region of the current frame image based on the phase difference information of each of the image blocks includes: Based on the phase difference information of each image block, determine whether each image block has moiré patterns; Based on the phase difference information of each image block, the difference between the phase difference between the current frame image and the previous frame image at the same image block position is determined; If the absolute value of the difference is greater than a first preset threshold and the number of image blocks with moiré patterns in the current frame image is greater than a second preset threshold, the moiré pattern region is determined based on the image blocks with moiré patterns.

3. The method according to claim 2, characterized in that, The step of determining whether each image block has moiré patterns based on the phase difference information of each image block includes: The phase difference information of each image block is processed for similarity to obtain the similarity curve of each image block; The image patch corresponding to the similarity curve with multiple peak points is identified as the image patch with moiré pattern.

4. The method according to claim 2, characterized in that, The step of determining whether each image block has moiré patterns based on the phase difference information of each image block includes: Obtain the current focus distance and the expected focus distance of each image block, determined based on the phase difference information of each image block; The image block corresponding to the sum of the current focus distance and the desired focus distance being greater than the preset maximum travel distance is identified as the image block with moiré pattern.

5. The method according to any one of claims 1 to 4, characterized in that, The step of obtaining the phase difference information of each image block in the current frame image includes: When the current frame image is acquired using phase focusing, the phase difference information of each image block output during the phase focusing process is obtained.

6. The method according to any one of claims 1 to 4, characterized in that, The current frame image includes a RAW image; the process of performing demoiring processing on the moiré region to obtain a demoired image includes: The moiré region of the RAW image is subjected to demoiré processing to obtain a demoiré processed RGB image.

7. The method according to any one of claims 1 to 4, characterized in that, Determining the moiré region of the current frame image based on the phase difference information of each of the image blocks includes: After performing brightness and color correction on the current frame image to correct lens shadows, the moiré region of the current frame image is determined based on the phase difference information of each image block.

8. The method according to any one of claims 1 to 4, characterized in that, The step of performing moiré removal processing on the moiré region to obtain a moiré-removed image includes: Interpolate and restore the information of the green channel in the moiré region to obtain the restored green channel result; Based on the recovery result of the green channel, the information of the red channel and the information of the blue channel in the moiré region are interpolated and recovered respectively to obtain the image after demoiré processing.

9. An electronic device, characterized in that, include: The phase difference acquisition unit is configured to acquire the phase difference information of each image block in the current frame image; The phase difference information indicates the offset between the current focus position and the desired focus position; The moiré pattern determination unit is configured to determine the moiré pattern region of the current frame image based on the phase difference information of each of the image blocks; The moiré processing unit is configured to perform demoiré processing on the moiré region to obtain a demoiré processed image.

10. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images