Image reconstruction methods, image sensors and terminal devices

Abstract

This invention provides an image reconstruction method, an image sensor, and a terminal device. The method includes: acquiring a color filter array image collected by the image sensor; acquiring a high-frequency image and a low-frequency image based on the color filter array image; for each pixel in the high-frequency image, acquiring the region feature fusion weight and distance feature fusion weight corresponding to that pixel; for each pixel in the high-frequency image, acquiring the mixing weight corresponding to that pixel based on the region feature fusion weight and distance feature fusion weight; and fusing the high-frequency image and the low-frequency image based on the mixing weight corresponding to each pixel in the high-frequency image to generate a reconstructed image. This invention can improve the image quality noise and false colors in the edge areas while ensuring the clarity of details in the central region of the image, thereby significantly improving the overall image reconstruction quality.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image reconstruction method, an image sensor, and a terminal device. Background Technology

[0002] To achieve higher resolution and better low-light performance, existing image sensors typically employ a 2×2 quad Bayer pattern or a higher-density 3×3 or 4×4 color filter array (CFA). This design effectively increases the photosensitive area and signal-to-noise ratio of the image sensor by integrating more photosensitive units on a single image sensor and using techniques such as pixel binning in the physical structure, especially enabling the capture of more details in low-light environments.

[0003] However, the introduction of this high-density color filter array structure also brings new challenges to image reconstruction algorithms. In traditional image processing workflows, color reconstruction mainly relies on demosaicing algorithms for standard Bayer images, such as nearest neighbor interpolation, bilinear interpolation, or adaptive gradient methods. However, in quad Bayer or more complex arrangements, the correlation between color difference and brightness information between adjacent pixels is stronger, and the structure is more complex. Using traditional interpolation methods can easily introduce problems such as artifacts, blurred details, or color difference distortion, making it difficult to fully utilize the advantages of high-resolution image sensors.

[0004] It should be noted that the information disclosed in the background section of this invention is intended only to enhance the understanding of the general background of this invention, and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide an image reconstruction method, an image sensor, and a terminal device that can improve the image quality, noise, and false colors in the edge areas while ensuring the clarity of details in the central area of ​​the image, thereby significantly improving the overall reconstruction quality of the image.

[0006] To achieve the above objectives, the present invention provides an image reconstruction method, comprising:

[0007] Acquire the color filter array image captured by the image sensor;

[0008] High-frequency and low-frequency images are obtained respectively based on the color filter array image;

[0009] For each pixel in the high-frequency image, the region feature fusion weight corresponding to the pixel is obtained based on the image feature information of the region where the pixel is located, and the distance feature fusion weight corresponding to the pixel is obtained based on the distance information between the pixel and the optical center.

[0010] For each pixel in the high-frequency image, the fusion weight corresponding to the pixel is obtained based on the region feature fusion weight and the distance feature fusion weight corresponding to the pixel;

[0011] The high-frequency image and the low-frequency image are fused according to the mixing weight corresponding to each pixel in the high-frequency image to generate a reconstructed image.

[0012] Optionally, the high-frequency image is obtained through the following steps:

[0013] The color filter array image is subjected to rearranged mosaic processing to obtain a first standard Bayer image;

[0014] The first standard Bayer image is subjected to high-frequency filtering to obtain the high-frequency image.

[0015] Optionally, the low-frequency image can be obtained through the following steps:

[0016] The color filter array image is subjected to pixel merging processing to obtain a second standard Bayer image;

[0017] The second standard Bayer image is upsampled to obtain the third standard Bayer image;

[0018] The third standard Bayer image is subjected to low-pass filtering to obtain the low-frequency image.

[0019] Optionally, obtaining the region feature fusion weight corresponding to the pixel based on the image feature information of the region where the pixel is located includes:

[0020] Based on the variance information of the region where the pixel is located, obtain the region feature fusion weight corresponding to the pixel.

[0021] Optionally, obtaining the region feature fusion weight corresponding to the pixel based on the variance information of the region where the pixel is located includes:

[0022] Based on the variance information of the region where the pixel is located and the mapping relationship between the pre-obtained regional feature fusion weights and variance, the regional feature fusion weights corresponding to the pixel are obtained.

[0023] Optionally, obtaining the region feature fusion weight corresponding to the pixel based on the variance information of the region where the pixel is located and the pre-obtained mapping relationship between the region feature fusion weight and the variance includes:

[0024] If the variance of the region where the pixel is located is less than or equal to the first variance threshold, then the feature fusion weight of the region corresponding to the pixel is the preset minimum weight.

[0025] If the variance of the region where the pixel is located is greater than or equal to the second variance threshold, then the feature fusion weight of the region corresponding to the pixel is the preset maximum weight.

[0026] If the variance of the region where the pixel is located is greater than the first variance threshold and less than the second variance threshold, then the feature fusion weight of the region corresponding to the pixel is greater than the preset minimum weight and less than the preset maximum weight. Furthermore, the greater the variance of the region where the pixel is located, the greater the feature fusion weight of the region corresponding to the pixel.

[0027] Optionally, for each pixel in the high-frequency image, the variance information of the region where the pixel is located is obtained through the following steps:

[0028] All green pixels other than the pixel within a preset neighborhood window centered on the pixel are taken as the first target neighborhood pixels corresponding to the pixel.

[0029] Sort all the first target neighbor pixels corresponding to the pixel in ascending or descending order of pixel value;

[0030] Based on the sorting result of the first target neighbor pixels corresponding to the pixel, the first target neighbor pixels with the largest pixel value and the first target neighbor pixels with the smallest pixel value are removed to filter out the second target neighbor pixels corresponding to the pixel.

[0031] Based on the pixel values ​​of all the second target neighboring pixels corresponding to the pixel, obtain the variance information of the region where the pixel is located.

[0032] Optionally, obtaining the distance feature fusion weight corresponding to the pixel based on the distance information between the pixel and the optical center includes:

[0033] Based on the distance information between the pixel and the optical center, and the mapping relationship between the pre-acquired distance feature fusion weight and the distance, the distance feature fusion weight corresponding to the pixel is obtained.

[0034] Optionally, obtaining the distance feature fusion weight corresponding to the pixel based on the distance information between the pixel and the optical center and the pre-acquired mapping relationship between the distance feature fusion weight and the distance includes:

[0035] If the distance between the pixel and the optical center is less than or equal to the first distance threshold, then the distance feature fusion weight corresponding to the pixel is the preset maximum weight;

[0036] If the distance between the pixel and the optical center is greater than or equal to the second distance threshold, then the distance feature fusion weight corresponding to the pixel is the preset minimum weight;

[0037] If the distance between the pixel and the optical center is greater than the first distance threshold and less than the second distance threshold, then the distance feature fusion weight corresponding to the pixel is less than the preset maximum weight and greater than the preset minimum weight. Furthermore, the greater the distance between the pixel and the optical center, the smaller the distance feature fusion weight corresponding to the pixel.

[0038] To achieve the above objectives, the present invention also provides an image sensor, including an image signal processor configured to implement the image reconstruction method described in any of the preceding claims.

[0039] To achieve the above objectives, the present invention also provides a terminal device, the terminal device including a processor and a memory, the memory storing a computer program, and when the computer program is executed by the processor, implementing the image reconstruction method described in any of the preceding claims.

[0040] Compared with existing technologies, the image reconstruction method, image sensor, and terminal device provided by this invention have the following advantages:

[0041] The image reconstruction method provided by this invention first acquires a color filter array image collected by an image sensor; then, it acquires high-frequency and low-frequency images based on the color filter array image; next, for each pixel in the high-frequency image, it acquires the corresponding region feature fusion weight based on the image feature information of the region where the pixel is located, and the corresponding distance feature fusion weight based on the distance information between the pixel and the optical center; then, for each pixel in the high-frequency image, it acquires the corresponding mixing weight based on the corresponding region feature fusion weight and distance feature fusion weight; finally, it fuses the high-frequency image and the low-frequency image based on the mixing weight corresponding to each pixel in the high-frequency image to generate a reconstructed image. Therefore, the image reconstruction method provided by this invention can reconstruct edge and texture details using the high-frequency image; the low-frequency image provides a high signal-to-noise ratio substrate, effectively suppressing noise, thus enabling the reconstructed image to maintain sharpness in textured areas and present a clean appearance in flat areas, thereby achieving an optimal balance between detail and noise. Furthermore, by obtaining the corresponding regional feature fusion weight for each pixel in the high-frequency image based on the image feature information of the region where the pixel is located, this invention can avoid excessive sharpening in flat areas leading to noise amplification or excessive noise reduction in textured areas leading to detail blurring. By obtaining the corresponding distance feature fusion weight for each pixel in the high-frequency image based on the distance information between the pixel and the optical center, it can compensate for the inherent edge image quality degradation of the lens, significantly improving the visual consistency of the image from the center to the edge. In summary, by employing the image reconstruction method provided by this invention, it is possible to improve image quality noise and false colors in edge areas while ensuring the clarity of details in the central region of the image, thereby significantly improving the overall image reconstruction quality.

[0042] Since the image sensor and terminal device provided by this invention belong to the same inventive concept as the image reconstruction method provided by this invention, the image sensor and terminal device provided by this invention have at least all the beneficial effects of the image reconstruction method provided by this invention. For details, please refer to the relevant descriptions of the beneficial effects of the image reconstruction method provided by this invention above. Therefore, the beneficial effects of the image sensor and terminal device provided by this invention will not be elaborated here. Attached Figure Description

[0043] Figure 1 A flowchart of an image reconstruction method provided according to an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of a four-Bayer array image.

[0045] Figure 3This is a schematic diagram of a 3×3 CFA image.

[0046] Figure 4 This is a schematic diagram of a 4×4 CFA image.

[0047] Figure 5 A schematic diagram for generating the first standard Bayer image.

[0048] Figure 6 A schematic diagram for generating a second standard Bayer image.

[0049] Figure 7 This is a distribution map of the first target neighborhood pixels of the green pixels in a high-frequency image.

[0050] Figure 8 This is a distribution map of the first target neighborhood pixels of the blue pixels in a high-frequency image.

[0051] Figure 9 This is a schematic diagram illustrating the mapping relationship between regional feature fusion weights and variance provided in one embodiment of the present invention.

[0052] Figure 10 This is a schematic diagram showing the distance between a pixel in a high-frequency image and the optical center.

[0053] Figure 11 This is a schematic diagram illustrating the mapping relationship between distance feature fusion weights and distance, provided in one embodiment of the present invention.

[0054] Figure 12 This is a block diagram of an image sensor provided according to an embodiment of the present invention.

[0055] Figure 13 This is an image processing flowchart of an image signal processor provided according to an embodiment of the present invention.

[0056] Figure 14 This is a block diagram of a terminal device provided according to an embodiment of the present invention.

[0057] Figure 15 This is a flowchart illustrating the image processing of a terminal device according to an embodiment of the present invention.

[0058] The reference numerals in the attached figures are explained as follows: Image sensor - 100; Image signal processor - 110; Pixel array - 120; Readout circuit - 130; Timing and system control module - 140; Output interface - 150; Terminal device - 200; Processor - 210; Communication interface - 220; Memory - 230; Communication bus - 240. Detailed Implementation

[0059] The image reconstruction method, image sensor, and terminal device proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent. It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Moreover, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0060] Furthermore, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0061] The core idea of ​​this invention is to provide an image reconstruction method, an image sensor, and a terminal device that can improve the image quality, noise, and false colors in the edge areas while ensuring the clarity of details in the central area of ​​the image, thereby significantly improving the overall reconstruction quality of the image.

[0062] It should be noted that the image reconstruction method provided by the present invention can be applied to the image sensor and terminal device provided by the present invention. The terminal device can be, but is not limited to, a camera, a video camera, a mobile phone, a tablet computer, a learning machine, and a medical imaging device.

[0063] To achieve the above-mentioned goals, this invention provides an image reconstruction method, please refer to... Figure 1 This is a flowchart of an image reconstruction method provided by an embodiment of the present invention. Figure 1 As shown, the image reconstruction method provided by this invention includes the following steps:

[0064] Step S100: Acquire the color filter array image captured by the image sensor;

[0065] Step S200: Obtain high-frequency images and low-frequency images respectively based on the color filter array image;

[0066] Step S300: For each pixel in the high-frequency image, obtain the region feature fusion weight corresponding to the pixel based on the image feature information of the region where the pixel is located, and obtain the distance feature fusion weight corresponding to the pixel based on the distance information between the pixel and the optical center.

[0067] Step S400: For each pixel in the high-frequency image, obtain the mixing weight corresponding to the pixel based on the region feature fusion weight and distance feature fusion weight corresponding to the pixel;

[0068] Step S500: Based on the mixing weights corresponding to each pixel in the high-frequency image, the high-frequency image and the low-frequency image are fused to generate a reconstructed image.

[0069] Therefore, the image reconstruction method provided by this invention can reconstruct detailed information such as edges and textures using the high-frequency image; the low-frequency image provides a high signal-to-noise ratio substrate, effectively suppressing noise. This allows the reconstructed image to maintain sharpness in textured areas and present a clean appearance in flat areas, achieving an optimal balance between detail and noise. Furthermore, by obtaining the corresponding regional feature fusion weight for each pixel in the high-frequency image based on the image feature information of the region where the pixel is located, this invention avoids over-sharpening in flat areas leading to noise amplification or over-denoising in textured areas leading to detail blurring. By obtaining the corresponding distance feature fusion weight for each pixel in the high-frequency image based on the distance information between the pixel and the optical center, this invention can compensate for the inherent edge image quality degradation of the lens, significantly improving the visual consistency of the image from the center to the edge. In summary, by employing the image reconstruction method provided by this invention, it is possible to improve image quality noise and false colors in edge areas while ensuring the clarity of details in the central area of ​​the image, thereby significantly improving the overall image reconstruction quality.

[0070] It should be noted that, as those skilled in the art will understand, the color filter array image acquired by the image sensor can be a 2×2 quad Bayer array image (e.g., Figure 2 As shown, this is a schematic diagram of a four-Bayer array image), or a higher 3×3 CFA (color filter array) image (such as...). Figure 3As shown, it is a schematic diagram of a 3×3 CFA image, and a 4×4 CFA (Color Filter Array) image (such as...). Figure 4 As shown, it is a schematic diagram of a 4×4 CFA image.

[0071] In some exemplary embodiments, the high-frequency image is acquired through the following steps:

[0072] The color filter array image is subjected to rearranged mosaic processing to obtain a first standard Bayer image;

[0073] The first standard Bayer image is subjected to high-frequency filtering to obtain the high-frequency image.

[0074] Since standard Bayer images are the foundation of traditional image processing algorithms and have a regular pixel arrangement, remosaic processing of the color filter array image can convert non-standard Bayer format color filter array images (such as quad Bayer images, 3×3 or 4×4 CFA images) into a first standard Bayer image in standard Bayer format. This simplifies subsequent processing and improves compatibility. Furthermore, by performing high-frequency filtering on the first standard Bayer image, high-frequency components (such as edge and texture details) can be extracted more accurately, ensuring that more subtle features are preserved in the reconstructed image.

[0075] Please continue to refer to this. Figure 5 This is a schematic diagram illustrating the generation of the first standard Bayer image. For example... Figure 5 As shown, a 2×2 quad Bayer image can be converted into a first standard Bayer image by rearranging the mosaic. Specifically, the details of how to perform the rearrangement mosaic processing on the color filter array image can be adapted for understanding by referring to relevant materials known to those skilled in the art, and will not be elaborated upon here.

[0076] In some exemplary embodiments, the low-frequency image is acquired through the following steps:

[0077] The color filter array image is subjected to pixel merging processing to obtain a second standard Bayer image;

[0078] The second standard Bayer image is upsampled to obtain the third standard Bayer image;

[0079] The third standard Bayer image is subjected to low-pass filtering to obtain the low-frequency image.

[0080] Therefore, by performing pixel merging processing on the color filter array image (merging pixel blocks of the same color, for example, merging a 2×2 four-Bayer pixel block of the same color into one superpixel), multiple small pixels can be merged into one large pixel, significantly suppressing random noise. Since the resolution of the merged second standard Bayer image is reduced (e.g., ... Figure 6 As shown in the diagram (which illustrates the generation of the second standard Bayer image), by upsampling the second standard Bayer image, the low-resolution second standard Bayer image can be enlarged to the same resolution as the original image (color filter array image) to obtain the third standard Bayer image. This ensures that the low-frequency image obtained through low-pass filtering is size-aligned with the high-frequency image, facilitating subsequent pixel-level fusion. By performing low-pass filtering on the size-aligned third standard Bayer image, subtle high-frequency noise or artifacts that may have been introduced during the upsampling process can be further filtered out, thereby extracting pure low-frequency components and ensuring that the final fused reconstructed image is very pure in flat areas.

[0081] It should be noted that, as those skilled in the art will understand, the second standard Bayer image can be upsampled using an interpolation algorithm (e.g., bilinear interpolation). For details on the interpolation algorithm, please refer to relevant content known to those skilled in the art for an adaptive understanding, and it will not be elaborated here.

[0082] In some exemplary embodiments, obtaining the region feature fusion weight corresponding to the pixel based on the image feature information of the region where the pixel is located includes:

[0083] Based on the variance information of the region where the pixel is located, obtain the region feature fusion weight corresponding to the pixel.

[0084] Since pixel values ​​vary drastically in regions with high variance, it indicates that the region may contain rich texture, while pixel values ​​are uniform in regions with low variance, indicating that the region may be flat. Therefore, by using the variance information of the region where a pixel is located, it is possible to accurately determine whether the region where the pixel is located is a flat region or a textured region. This allows for the accurate assignment of higher weights to pixels in textured regions of high-frequency images and lower weights to pixels in flat regions of high-frequency images, thereby ensuring the reconstruction of high-quality images with rich details, controllable noise, and uniform image quality.

[0085] It should be noted that, as those skilled in the art will understand, for each pixel in the high-frequency image, in addition to obtaining the region feature fusion weight corresponding to the pixel based on the variance information of the region where the pixel is located, the region feature fusion weight corresponding to the pixel can also be obtained based on the gradient information or frequency domain energy information of the region where the pixel is located.

[0086] In some exemplary embodiments, for each pixel in the high-frequency image, the variance information of the region where the pixel is located is obtained through the following steps:

[0087] All green pixels other than the pixel within a preset neighborhood window centered on the pixel are taken as the first target neighborhood pixels corresponding to the pixel.

[0088] Sort all the first target neighboring pixels corresponding to the pixel in ascending or descending order of pixel value;

[0089] Based on the sorting result of the first target neighbor pixels corresponding to the pixel, the first target neighbor pixels with the largest pixel value and the first target neighbor pixels with the smallest pixel value are removed to filter out the second target neighbor pixels corresponding to the pixel.

[0090] Based on the pixel values ​​of all the second target neighboring pixels corresponding to the pixel, obtain the variance information of the region where the pixel is located.

[0091] Therefore, for each pixel in the high-frequency image, all first target neighbor pixels corresponding to that pixel are firstly found within a preset neighborhood window centered on that pixel. Then, the first target neighbor pixels with the largest and smallest pixel values ​​are removed to filter out all second target neighbor pixels corresponding to that pixel. Finally, the variance of the region where the pixel is located is calculated based on the pixel values ​​of all the second target neighbor pixels. This ensures that the calculated variance can truly reflect the structural texture changes of the region where the pixel is located, thus avoiding misclassifying flat but noisy regions as textured regions and preventing the injection of too much high-frequency noise into flat regions during subsequent fusion. It also provides a more reliable basis for determining the fusion weights of regional features based on variance. In addition, since the number of green pixels in most Bayer arrays is twice the number of red or blue pixels, the green channel can provide more sampling points under the same window size. Therefore, by selecting green pixels as target neighbor pixels, not only can the validity of the calculated variance be guaranteed, but the calculated variance can also better reflect the details and structural changes that the human eye is sensitive to.

[0092] Please continue to refer to this. Figure 7 This is a distribution map of the first target neighborhood pixels of green pixels in a high-frequency image. For example... Figure 7 As shown in the image, the green pixels indicated by the white dots are all green pixels P. G The first target neighboring pixel, it can be seen, within the 5×5 preset neighborhood window, the green pixel P GThere are 12 pixels in the first target neighborhood. These 12 pixels P can be sorted by pixel value from smallest to largest or from largest to smallest. i Sort: P i ’ =sort(P i ), i∈1~12, and then remove P according to the sorting result. i ’ The maximum and minimum values ​​in (P1) ’ P 12 ’ ), to filter out green pixels P G The second target neighbor pixels are then used to calculate the average value (avg) for all second target neighbor pixels: avg = sum(P i ’ ) / 10, i∈2~11, then calculate the green pixel P according to the following formula. G The variance var of the region:

[0093] Var=sum(abs(P i ’ -avg)) / 10, i∈2~11

[0094] Here, abs represents taking the absolute value.

[0095] Please continue to refer to this. Figure 8 This is a distribution map of the first target neighborhood pixels of the blue pixels in a high-frequency image. For example... Figure 8 As shown in the image, the green pixels indicated by the white dots are all blue pixels P. B The first target neighboring pixel, it can be seen, within the 5×5 preset neighborhood window, the green pixel P B The first target neighborhood has a total of 12 pixels. Regarding how to base it on the blue pixel P... B The blue pixel P is calculated based on the first target neighboring pixel. B For information on the variance of the region, please refer to the section above on how to calculate the green pixel P. G The relevant content regarding the variance of the region should be understood adaptively, and will not be elaborated upon here.

[0096] It should be noted that, although Figure 7 and Figure 8 The example given uses a preset neighborhood window size of 5×5 pixels. However, as those skilled in the art will understand, this does not constitute a limitation of the invention, and the specific size of the preset neighborhood window can be set according to actual needs. Furthermore, it should be noted that for information on how to calculate the variance of the region containing a red pixel in a high-frequency image, please refer to the section above on how to calculate the variance of a green pixel P.G The variance of the region is understood adaptively and will not be elaborated further here. Additionally, it should be noted that for pixels located at the edges of the high-frequency image (e.g., pixels in the first row of the high-frequency image), the preset neighborhood window may exceed the effective range of the image. In this case, the area exceeding the range can be filled, and then the first target neighboring pixel of that pixel can be found.

[0097] In some exemplary embodiments, obtaining the region feature fusion weight corresponding to the pixel based on the variance information of the region where the pixel is located includes:

[0098] Based on the variance information of the region where the pixel is located and the mapping relationship between the pre-obtained regional feature fusion weights and variance, the regional feature fusion weights corresponding to the pixel are obtained.

[0099] Therefore, for each pixel in the high-frequency image, the region feature fusion weight corresponding to the pixel can be quickly and accurately determined by using the variance information of the region where the pixel is located and the mapping relationship between the pre-obtained region feature fusion weight and variance.

[0100] In some exemplary embodiments, obtaining the region feature fusion weight corresponding to the pixel based on the variance information of the region where the pixel is located and the pre-acquired mapping relationship between the region feature fusion weight and the variance includes:

[0101] If the variance of the region where the pixel is located is less than or equal to the first variance threshold, then the feature fusion weight of the region corresponding to the pixel is the preset minimum weight.

[0102] If the variance of the region where the pixel is located is greater than or equal to the second variance threshold, then the feature fusion weight of the region corresponding to the pixel is the preset maximum weight.

[0103] If the variance of the region where the pixel is located is greater than the first variance threshold and less than the second variance threshold, then the feature fusion weight of the region corresponding to the pixel is greater than the preset minimum weight and less than the preset maximum weight. Furthermore, the greater the variance of the region where the pixel is located, the greater the feature fusion weight of the region corresponding to the pixel.

[0104] Therefore, by uniformly assigning the minimum regional feature fusion weight (i.e., the preset minimum weight) to pixels in the flattest region of the high-frequency image, noise can be effectively suppressed; by uniformly assigning the maximum regional feature fusion weight (i.e., the preset maximum weight) to pixels in the most complex texture region of the high-frequency image, details can be effectively preserved; by setting regional feature fusion weights that are positively correlated with the variance of the region to pixels in the high-frequency image located between the flattest and most complex texture regions, a smooth transition of regional feature fusion weights can be achieved, thereby effectively avoiding artifacts.

[0105] For details, please refer to Figure 9 This is a schematic diagram illustrating the mapping relationship between regional feature fusion weights and variance provided in one embodiment of the present invention. For example... Figure 9 As shown, for each pixel in the high-frequency image, if the variance var of the region where the pixel is located is less than or equal to the first variance threshold var th1 Then the preset minimum weight β is used as the region feature fusion weight w for that pixel. area If the var of the region where the pixel is located is greater than or equal to the second variance threshold var th2 Then the preset maximum weight α is used as the region feature fusion weight w for that pixel. area If the variance var of the region where the pixel is located is greater than the first variance threshold var th1 And less than the second variance threshold var th2 Then the region feature fusion weight w of that pixel area The region feature fusion weight w has a positive linear relationship with the variance var of the region where the pixel is located (the larger the variance var, the greater the weight w). area The larger (the larger).

[0106] It should be noted that, as those skilled in the art will understand, the present invention addresses the first variance threshold var. th1 and the second variance threshold var th2 The specific value is not limited, and the first variance threshold var th1 and the second variance threshold var th2 The specific values ​​can be set according to actual needs. It should also be noted that, as those skilled in the art will understand, the present invention does not limit the specific values ​​of the preset minimum weight β and the preset maximum weight α. The specific value of the preset minimum weight β can be 0~0.2, and the specific value of the preset maximum weight α can be 0.8~1.

[0107] In some exemplary embodiments, obtaining the distance feature fusion weight corresponding to the pixel based on the distance information between the pixel and the optical center includes:

[0108] Based on the distance information between the pixel and the optical center, and the mapping relationship between the pre-acquired distance feature fusion weight and the distance, the distance feature fusion weight corresponding to the pixel is obtained.

[0109] Therefore, for each pixel in the high-frequency image, the distance feature fusion weight corresponding to the pixel can be quickly and accurately determined by using the distance information between the pixel and the optical center and the pre-acquired mapping relationship between the distance feature fusion weight and the distance.

[0110] For details, please refer to Figure 10 This is a schematic diagram illustrating the distance between a pixel in a high-frequency image and the optical center. For example... Figure 10 As shown, the optical center is the center point of the high-frequency image (the red dot in the figure). Assuming the coordinates of the optical center are (x0, y0), the coordinates in the high-frequency image are (x...). i ,y i The pixel at which the distance from the optical center is dis = abs(x) i -x0)+abs(y i -y0), where abs represents taking the absolute value.

[0111] In some exemplary embodiments, obtaining the distance feature fusion weight corresponding to the pixel based on the distance information between the pixel and the optical center and the pre-acquired mapping relationship between the distance feature fusion weight and the distance includes:

[0112] If the distance between the pixel and the optical center is less than or equal to the first distance threshold, then the distance feature fusion weight corresponding to the pixel is the preset maximum weight;

[0113] If the distance between the pixel and the optical center is greater than or equal to the second distance threshold, then the distance feature fusion weight corresponding to the pixel is the preset minimum weight;

[0114] If the distance between the pixel and the optical center is greater than the first distance threshold and less than the second distance threshold, then the distance feature fusion weight corresponding to the pixel is less than the preset maximum weight and greater than the preset minimum weight. Furthermore, the greater the distance between the pixel and the optical center, the smaller the distance feature fusion weight corresponding to the pixel.

[0115] Due to the characteristics of the lens, the image resolution is better in the optical center area and worse in the edge area. Therefore, by uniformly assigning the maximum distance feature fusion weight (preset maximum weight) to pixels in the high-frequency image that are close to the optical center (distance less than or equal to the first distance threshold), more high-frequency information (detail information) can be extracted. By uniformly assigning the minimum distance feature fusion weight (preset minimum weight) to pixels in the high-frequency image that are far from the optical center (distance greater than or equal to the second distance threshold), the signal-to-noise ratio can be prioritized in this area, and noise and dispersion can be suppressed to the maximum extent. By setting a distance feature fusion weight that is negatively correlated with the corresponding distance for pixels in the high-frequency image located in the transition area (distance greater than the first distance threshold and less than the second distance threshold), a smooth transition of the distance feature fusion weight can be achieved, effectively avoiding artifacts and significantly improving the visual consistency of the image from the center to the edge.

[0116] For details, please refer to Figure 11 This is a schematic diagram illustrating the mapping relationship between distance feature fusion weights and distance provided in one embodiment of the present invention. For example... Figure 11 As shown, for each pixel in the high-frequency image, if the distance dis between the pixel and the optical center is less than or equal to a first distance threshold dis... th1 Then the preset maximum weight α is used as the distance feature fusion weight w for that pixel. dis If the distance dis between the pixel and the optical center is greater than or equal to the second distance threshold dis th2 Then the preset minimum weight β is used as the distance feature fusion weight w for that pixel. dis If the distance dis between the pixel and the optical center is greater than the first distance threshold dis th1 And less than the second distance threshold dis th2 Then the distance feature fusion weight w of the pixel dis The distance *dis* between the pixel and the optical center has a negative linear relationship (the larger the distance *dis*, the greater the distance feature fusion weight *w*). dis The smaller).

[0117] It should be noted that, as those skilled in the art will understand, the present invention addresses the first distance threshold dis. th1 The second distance threshold dis th2 The specific value is not limited, and the first distance threshold dis th1 and the second distance threshold dis th2 The specific value can be set according to actual needs.

[0118] In some exemplary embodiments, obtaining the mixing weight corresponding to each pixel in the high-frequency image based on the region feature fusion weight and distance feature fusion weight corresponding to that pixel includes:

[0119] For each pixel in the high-frequency image, determine whether the sum of the region feature fusion weight and the distance feature fusion weight corresponding to the pixel is greater than 1. If so, use 1 as the mixing weight corresponding to the pixel. If not, use the sum of the region feature fusion weight and the distance feature fusion weight corresponding to the pixel as the mixing weight corresponding to the pixel.

[0120] Therefore, by using this setting, the mixing weight of each pixel in the high-frequency image can be limited to between 0 and 1 (inclusive), thereby ensuring the stability of the subsequent fusion process and preventing excessive enhancement of high-frequency information.

[0121] Specifically, for each pixel in the high-frequency image, the mixing weight w of that pixel can be calculated according to the following formula. h :

[0122] w h =Clip(w area +w dis ,0,1)

[0123] Clip indicates a clipping operation.

[0124] Therefore, after determining the mixing weight of each pixel in the high-frequency image, the high-frequency image and the low-frequency image can be fused according to the following formula:

[0125] Value final =HF·w h +LF·(1-w h )

[0126] Where, Value final HF is the pixel value of a certain pixel in the high-frequency image and LF is the pixel value of the corresponding pixel in the low-frequency image after fusion.

[0127] Based on the same inventive concept, the present invention also provides an image sensor, please refer to... Figure 12 This is a block diagram of the image sensor provided in one embodiment of the present invention. Figure 12As shown, the image sensor 100 provided by the present invention includes an image signal processor 110, which is configured to implement the image reconstruction method described in any of the preceding claims. Since the image sensor 100 and the image reconstruction method provided by the present invention belong to the same inventive concept, the image sensor 100 provided by the present invention possesses at least all the beneficial effects of the image reconstruction method provided by the present invention. Therefore, the beneficial effects of the image sensor 100 provided by the present invention can be referred to the relevant descriptions of the beneficial effects of the image reconstruction method provided by the present invention above, and will not be repeated here.

[0128] Please continue to refer to this. Figure 12 ,like Figure 12 As shown, the image sensor 100 provided by the present invention further includes a pixel array 120, a readout circuit 130, a timing and system control module 140, and an output interface 150. Under the coordinated control of the timing and system control module 140, the pixel array 120 converts the acquired raw optical signal into an electrical signal. The readout circuit 130 performs analog-to-digital conversion on the electrical signal to obtain raw image data (color filter array image), and sends the converted raw image data (color filter array image) to the image signal processor 110. Please continue to refer to... Figure 13 This is an image processing flowchart of an image signal processor 110 provided in one embodiment of the present invention. Figure 13 As shown, the image signal processor 110 first performs preprocessing on the original image data (color filter array image) such as black level correction (BLC) and bad pixel correction (DPC), and then uses the image reconstruction method provided by this invention to reconstruct the preprocessed color filter array image to generate a reconstructed image in standard Bayer format. Afterwards, the reconstructed image is sent to the terminal device 200 through the output interface 150. The terminal device 200 can perform demosaicing on the reconstructed image to output an RGB image.

[0129] Based on the same inventive concept, the present invention also provides a terminal device, please refer to [reference needed]. Figure 14 This is a block structure diagram of a terminal device provided in one embodiment of the present invention. Figure 14As shown, the terminal device 200 includes a processor 210 and a memory 230. The memory 230 stores a computer program, which, when executed by the processor 210, implements the image reconstruction method described above. Since the terminal device 200 provided by this invention and the image reconstruction method provided by this invention belong to the same inventive concept, the terminal device 200 provided by this invention possesses at least all the beneficial effects of the image reconstruction method provided by this invention. Therefore, the beneficial effects of the terminal device 200 provided by this invention can be referred to the relevant descriptions of the beneficial effects of the image reconstruction method provided by this invention above, and will not be repeated here.

[0130] Furthermore, when the computer program is executed by the processor, it also performs the following steps:

[0131] The reconstructed image is then de-mosaiced to generate an RGB image.

[0132] Specifically, regarding how to perform demosaicing on the reconstructed image, please refer to the content known to those skilled in the art for an adaptive understanding, and will not be elaborated here.

[0133] Please continue to refer to this. Figure 15 This is a flowchart illustrating the image processing of a terminal device 200 according to an embodiment of the present invention. Figure 15 As shown, after receiving the color filter array image transmitted by the image sensor 100, the terminal device 200 performs bad pixel correction (DPC), black level correction (BLC), lens shading correction (LSC), noise reduction (NR), automatic white balance (AWB), image reconstruction, and demosaic on the color filter array image in sequence, and finally outputs an RGB image.

[0134] Please continue to refer to this. Figure 14 ,like Figure 14 As shown, the terminal device 200 also includes a communication interface 220 and a communication bus 240. The processor 210, the communication interface 220, and the memory 230 communicate with each other via the communication bus 240. The communication bus 240 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 240 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 220 is used for communication between the terminal device 200 and other devices.

[0135] It should be noted that the processor 210 referred to in this invention can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 210 is the control center of the terminal device 200, connecting all parts of the terminal device 200 via various interfaces and lines.

[0136] It should also be noted that the memory 230 can be used to store the computer program, and the processor 210 implements various functions of the terminal device 200 by running or executing the computer program stored in the memory 230 and calling the data stored in the memory 230. The memory 230 may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable memory (PROM), electrically programmable memory (EPROM), electrically erasable programmable memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, random access memory is available in a variety of forms, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous random access memory (SDRAM), dual data rate synchronous random access memory (DDRSDRAM), enhanced synchronous random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), memory bus direct random access memory (RDRAM), direct memory bus dynamic random access memory (DRDRAM), and memory bus dynamic random access memory (RDRAM), etc.

[0137] In summary, compared with the prior art, the image reconstruction method, image sensor, and terminal device provided by the present invention have the following beneficial effects:

[0138] This invention reconstructs detailed information such as edges and textures using high-frequency images and provides a high signal-to-noise ratio substrate using low-frequency images, effectively suppressing noise. This allows the reconstructed image to maintain sharpness in textured areas while maintaining a clean appearance in flat areas, achieving an optimal balance between detail and noise. Furthermore, by obtaining regional feature fusion weights for each pixel in the high-frequency image based on the image feature information of its location, this invention avoids over-sharpening in flat areas leading to noise amplification or over-denoising in textured areas leading to blurred details. By obtaining distance feature fusion weights for each pixel in the high-frequency image based on its distance from the optical center, this invention compensates for the inherent edge image quality degradation of the lens, significantly improving the visual consistency of the image from the center to the edge. In summary, this invention can improve image quality, noise, and false colors in edge areas while maintaining the clarity of details in the central image area, thus significantly improving the overall image reconstruction quality.

[0139] It should be noted that the above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. An image reconstruction method, characterized in that, include: Acquire the color filter array image captured by the image sensor; High-frequency and low-frequency images are obtained respectively based on the color filter array image; For each pixel in the high-frequency image, the region feature fusion weight corresponding to the pixel is obtained based on the image feature information of the region where the pixel is located, and the distance feature fusion weight corresponding to the pixel is obtained based on the distance information between the pixel and the optical center. For each pixel in the high-frequency image, the fusion weight corresponding to the pixel is obtained based on the region feature fusion weight and the distance feature fusion weight corresponding to the pixel; Based on the mixing weights corresponding to each pixel in the high-frequency image, the high-frequency image and the low-frequency image are fused to generate a reconstructed image; The step of obtaining the region feature fusion weight corresponding to the pixel based on the image feature information of the region where the pixel is located includes: If the variance of the region where the pixel is located is less than or equal to the first variance threshold, then the feature fusion weight of the region corresponding to the pixel is the preset minimum weight. If the variance of the region where the pixel is located is greater than or equal to the second variance threshold, then the feature fusion weight of the region corresponding to the pixel is the preset maximum weight. If the variance of the region where the pixel is located is greater than the first variance threshold and less than the second variance threshold, then the feature fusion weight of the region corresponding to the pixel is greater than the preset minimum weight and less than the preset maximum weight. Furthermore, the greater the variance of the region where the pixel is located, the greater the feature fusion weight of the region corresponding to the pixel. The step of obtaining the distance feature fusion weight corresponding to the pixel based on the distance information between the pixel and the optical center includes: If the distance between the pixel and the optical center is less than or equal to the first distance threshold, then the distance feature fusion weight corresponding to the pixel is the preset maximum weight; If the distance between the pixel and the optical center is greater than or equal to the second distance threshold, then the distance feature fusion weight corresponding to the pixel is the preset minimum weight; If the distance between the pixel and the optical center is greater than the first distance threshold and less than the second distance threshold, then the distance feature fusion weight corresponding to the pixel is less than the preset maximum weight and greater than the preset minimum weight. Furthermore, the greater the distance between the pixel and the optical center, the smaller the distance feature fusion weight corresponding to the pixel.

2. The image reconstruction method according to claim 1, characterized in that, The high-frequency image is obtained through the following steps: The color filter array image is subjected to rearranged mosaic processing to obtain a first standard Bayer image; The first standard Bayer image is subjected to high-frequency filtering to obtain the high-frequency image.

3. The image reconstruction method according to claim 1, characterized in that, The low-frequency image is obtained through the following steps: The color filter array image is subjected to pixel merging processing to obtain a second standard Bayer image; The second standard Bayer image is upsampled to obtain the third standard Bayer image; The third standard Bayer image is subjected to low-pass filtering to obtain the low-frequency image.

4. The image reconstruction method according to claim 1, characterized in that, For each pixel in the high-frequency image, the variance information of the region where the pixel is located is obtained through the following steps: All green pixels other than the pixel within a preset neighborhood window centered on the pixel are taken as the first target neighborhood pixels corresponding to the pixel. Sort all the first target neighbor pixels corresponding to the pixel in ascending or descending order of pixel value; Based on the sorting result of the first target neighbor pixels corresponding to the pixel, the first target neighbor pixels with the largest pixel value and the first target neighbor pixels with the smallest pixel value are removed to filter out the second target neighbor pixels corresponding to the pixel. Based on the pixel values ​​of all the second target neighboring pixels corresponding to the pixel, obtain the variance information of the region where the pixel is located.

5. An image sensor, characterized in that, The image signal processor is configured to implement the image reconstruction method according to any one of claims 1 to 4.

6. A terminal device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, implements the image reconstruction method according to any one of claims 1 to 4.

Images