Image post-processing method, image sensor, application processor and electronic device

Abstract

This invention provides an image post-processing method, an image sensor, an application processor, and an electronic device. The method includes: acquiring an initial Bayer image after hardware rearrangement mosaic processing; determining key and non-key regions in the initial Bayer image and extracting key and non-key region images; using a pre-trained lightweight deep learning model to post-process the key region images to repair false colors and artifacts generated after hardware rearrangement mosaic processing, thereby obtaining a processed key region image; and using a preset smoothing fusion algorithm to fuse the processed key and non-key region images to obtain a repaired Bayer image. This invention can effectively solve the artifact and false color problems left after hardware rearrangement mosaic processing without significantly increasing computational cost and power consumption.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image post-processing method, an image sensor, an application processor, and an electronic device. Background Technology

[0002] With technological advancements, current image sensors generally employ Q×Q color filter array (CFA) patterns (such as...). Figures 1 to 3 As shown, where, Figure 1 It is a 2×2 color filter array pattern. Figure 2 It is a 3×3 color filter array pattern. Figure 3 (This refers to a 4×4 color filter array pattern). Such Q×Q color filter array patterns typically require remosaic processing, first converting to a Bayer image, and then the image signal processor in the application processor (AP) generates an RGB image from the Bayer image. However, during the remosaic process, due to the sparse spatial sampling rate of the Q×Q color filter array pattern, artifacts and false colors are easily generated.

[0003] To address this technical challenge, existing technologies employ end-to-end deep neural network models to directly reconstruct high-quality Bayer or RGB images from Q×Q color filter array images. However, due to the large number of parameters and extremely high computational complexity of end-to-end deep neural network models, deploying them on image sensors faces stringent limitations in chip area and power consumption. Conversely, deploying them on application processors presents processing speed bottlenecks, making it difficult to meet the demands of real-time preview or high frame rate video.

[0004] It should be noted that the information disclosed in the background section of this invention is intended only to enhance the understanding of the general background of this invention, and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide an image post-processing method, an image sensor, an application processor, and an electronic device that can effectively solve the artifacts and false colors left over after hardware rearrangement mosaic processing without significantly increasing computing costs and power consumption.

[0006] To achieve the above objectives, the present invention provides an image post-processing method, comprising: acquiring an initial Bayer image after hardware rearrangement mosaic processing; determining key regions and non-key regions in the initial Bayer image, and extracting key region images and non-key region images; using a pre-trained lightweight deep learning model to post-process the key region images to repair false colors and artifacts generated after hardware rearrangement mosaic processing, thereby obtaining processed key region images; and using a preset smoothing fusion algorithm to fuse the processed key region images and the non-key region images to obtain a repaired Bayer image.

[0007] Optionally, the lightweight deep learning model includes a pixel rearrangement layer, multiple cascaded residual modules, a pixel inverse rearrangement layer, and a global skip connection layer connected in sequence. The pixel rearrangement layer is used to perform pixel rearrangement on the key region image to obtain a four-channel first feature map, where the four channels of the first feature map correspond one-to-one with the four color components of the key region image. The multiple cascaded residual modules are used to extract high-frequency residual information to generate a second feature map. The pixel inverse rearrangement layer is used to perform pixel inverse rearrangement on the second feature map output by the last residual module to obtain a high-frequency residual image with the same size as the key region image. The global skip connection layer is used to add the key region image and the high-frequency residual image pixel by pixel to obtain the processed key region image.

[0008] Optionally, at least two of the cascaded residual modules have different convolution kernel sizes.

[0009] Optionally, each residual module includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence. The output of the first activation layer and the input of the first convolutional layer are added together to serve as the input of the second convolutional layer, and the output of the second activation layer and the input of the first convolutional layer are added together to serve as the output of the residual module.

[0010] Optionally, determining the key and non-key regions in the initial Bayer image includes: using a rectangular region with a preset ratio in the middle of the initial Bayer image as the key region, or using a target detection model deployed on an application processor to detect the initial Bayer image to identify the key region in the initial Bayer image; and using other regions of the initial Bayer image located outside the key region as the non-key regions.

[0011] Optionally, the key area includes at least one of an autofocus area, a face area, and a salient target area.

[0012] Optionally, the step of fusing the processed key region image and the non-key region image using a preset smoothing fusion algorithm includes: for pixels located within the key region, setting the fusion weight of the pixel value in the processed key region image to 1, and setting the fusion weight of the pixel value in the non-key region image to 0; for pixels located within the non-key region, setting the fusion weight of the pixel value in the processed key region image to 0, and setting the fusion weight of the pixel value in the non-key region image to 1; for pixels located in the transition zone between the key region and the non-key region, setting the fusion weight of the pixel value in the processed key region image to between 0 and 1, and the closer the pixel is to the key region, the greater the fusion weight of the pixel value in the processed key region image.

[0013] To achieve the above objectives, the present invention also provides an image sensor, the image sensor including an image signal processor configured to implement the image post-processing method described above.

[0014] To achieve the above objectives, the present invention also provides an application processor, which stores a computer program. When the computer program is executed by the processor, it implements the image post-processing method described above.

[0015] To achieve the above objectives, the present invention also provides an electronic device, which includes the image sensor or the application processor described above.

[0016] Compared with existing technologies, the image post-processing method, image sensor, application processor, and electronic device provided by this invention have the following advantages:

[0017] Because the lightweight deep learning model in the image post-processing method provided by this invention only undertakes the task of repairing high-frequency defects such as artifacts and false colors left over from hardware rearrangement mosaic processing, without changing the low-frequency basic information of the image, it can not only effectively eliminate artifacts and false colors and significantly improve image quality, but also retain the efficiency of hardware rearrangement mosaic processing. Furthermore, since the image post-processing method provided by this invention only uses a lightweight deep learning model to repair false colors and artifacts in key areas of the initial Bayer image after hardware rearrangement mosaic processing, it can effectively reduce computational overhead and memory bandwidth usage while improving the image quality of key areas, thus achieving an optimal balance between processing speed, power consumption, and image quality. In addition, by employing a preset smooth fusion algorithm to fuse the processed key area image and non-key area image, this invention can eliminate abrupt changes or artifacts at the boundary between key and non-key areas, allowing for a pixel-level natural transition between key and non-key areas at the boundary, thereby ensuring the overall visual consistency of the fused image. Meanwhile, since the network structure of the lightweight deep learning model used in this invention is shallow, the amount of computation and parameters is extremely small, and the inference speed is extremely fast. It can even meet the real-time processing requirements of video stream frame rate. Furthermore, since the lightweight deep learning model has low hardware resource requirements, the image post-processing method provided by this invention can not only be deployed in the neural network processor of the application processor, but also integrated into the image signal processor inside the image sensor, thus adapting to different hardware platforms.

[0018] Since the image sensor, application processor, and electronic device provided by this invention belong to the same inventive concept as the image post-processing method provided by this invention, the image sensor, application processor, and electronic device provided by this invention have at least all the beneficial effects of the image post-processing method provided by this invention. For details, please refer to the relevant descriptions above, which will not be repeated here. Attached Figure Description

[0019] Figure 1 It is a 2×2 color filter array pattern.

[0020] Figure 2 It is a 3×3 color filter array pattern.

[0021] Figure 3 It is a 4×4 color filter array pattern.

[0022] Figure 4 A flowchart of an image post-processing method provided according to an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of the image post-processing method provided by the present invention deployed on the application processor.

[0024] Figure 6 This is a schematic diagram of the image post-processing method provided by the present invention deployed on an image sensor.

[0025] Figure 7 This is a schematic diagram illustrating the determination of critical and non-critical regions according to an embodiment of the present invention.

[0026] Figure 8 A schematic diagram illustrating the determination of critical and non-critical regions, provided for another embodiment of the present invention.

[0027] Figure 9 This is a network structure diagram of a lightweight deep learning model provided in one embodiment of the present invention.

[0028] Figure 10 This is a schematic diagram of the structure of a residual module provided in one embodiment of the present invention.

[0029] Figure 11 This is a block diagram of an image sensor provided according to an embodiment of the present invention.

[0030] The reference numerals in the attached figures are explained as follows:

[0031] Application processor-100; Neural network processor-110; Image sensor-200; Image signal processor-210; Pixel array-220; Readout circuit-230; Timing and system control module-240; Output interface-250; Critical region-310; Non-critical region-320; Pixel rearrangement layer-410; Residual module-420; First convolutional layer-421; First activation layer-422; Second convolutional layer-423; Second activation layer-424; Pixel inverse rearrangement layer-430; Global skip connection layer-440. Detailed Implementation

[0032] The image post-processing method, image sensor, application processor, and electronic device proposed in this invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Please refer to the accompanying drawings to make the objectives, features, and advantages of this invention more apparent and understandable. It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings are only for illustrative purposes and to enable those skilled in the art to understand and read the invention, and are not intended to limit the implementation conditions of the invention. Any modifications to the structure, changes in proportions, or adjustments to the size, provided that the effects and objectives achieved by this invention are the same or similar, should still fall within the scope of the technical content disclosed in this invention.

[0033] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. The singular forms "a," "an," and "the" include plural objects; the term "or" is generally used to include "and / or"; the term "several" is generally used to include "at least one"; and the term "at least two" is generally used to include "two or more".

[0034] Furthermore, in the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0035] The core idea of ​​this invention is to provide an image post-processing method, an image sensor, an application processor, and an electronic device that can effectively solve the artifacts and false colors left over after hardware rearrangement mosaic processing without significantly increasing computing costs and power consumption.

[0036] To achieve the above-mentioned goals, this invention provides an image post-processing method, please refer to [the following text is missing]. Figure 4 This is a flowchart of an image post-processing method provided by an embodiment of the present invention. For example... Figure 4 As shown, the image post-processing method provided by the present invention includes the following steps:

[0037] Step S100: Obtain the initial Bayer image after hardware rearrangement mosaic processing.

[0038] Step S200: Determine the key regions and non-key regions in the initial Bayer image, and extract the key region images and non-key region images.

[0039] Step S300: The key region image is post-processed using a pre-trained lightweight deep learning model to repair the false colors and artifacts generated after hardware rearrangement mosaic processing, thereby obtaining the processed key region image.

[0040] Step S400: The processed key region image and the non-key region image are fused using a preset smoothing fusion algorithm to obtain the repaired Bayer image.

[0041] Because the lightweight deep learning model in the image post-processing method provided by this invention only undertakes the task of repairing high-frequency defects such as artifacts and false colors left over from hardware rearrangement mosaic processing, without changing the low-frequency basic information of the image, it can not only effectively eliminate artifacts and false colors and significantly improve image quality, but also retain the efficiency of hardware rearrangement mosaic processing. Furthermore, since the image post-processing method provided by this invention only uses a lightweight deep learning model to repair false colors and artifacts in key areas of the initial Bayer image after hardware rearrangement mosaic processing, it can effectively reduce computational overhead and memory bandwidth usage while improving the image quality of key areas, thus achieving an optimal balance between processing speed, power consumption, and image quality. In addition, by employing a preset smooth fusion algorithm to fuse the processed key area image and non-key area image, this invention can eliminate abrupt changes or artifacts at the boundary between key and non-key areas, allowing for a pixel-level natural transition between key and non-key areas at the boundary, thereby ensuring the overall visual consistency of the fused image. Meanwhile, since the network structure of the lightweight deep learning model used in this invention is shallow, the amount of computation and parameters is extremely small, and the inference speed is extremely fast. It can even meet the real-time processing requirements of video stream frame rate. Furthermore, since the lightweight deep learning model has low hardware resource requirements, the image post-processing method provided by this invention can not only be deployed in the neural network processor of the application processor, but also integrated into the image signal processor inside the image sensor, thus adapting to different hardware platforms.

[0042] Please continue to refer to this. Figure 5 This is a schematic diagram illustrating the deployment of the image post-processing method provided by the present invention on the application processor. Figure 5 As shown, for an application processor 100 with a neural network processor 110, the image post-processing method provided by the present invention can be deployed on the application processor, and the neural network processor 110 of the application processor 100 executes the image post-processing method provided by the present invention. Further, as... Figure 5 As shown, the original color filter array image is processed by the image signal processor 210 of the image sensor 200 through hardware rearrangement mosaic processing, and then input to the neural network processor 110 of the application processor 100 for post-processing.

[0043] Please continue to refer to this. Figure 6 This is a schematic diagram illustrating the deployment of the image post-processing method provided by the present invention on an image sensor. For example... Figure 6 As shown, for application processors 100 that do not have neural network processors, the image post-processing method provided by this invention can be deployed on the image sensor. Further, as... Figure 6 As shown, after the original color filter array image undergoes hardware rearrangement mosaic processing by the image signal processor 210 of the image sensor 200, the image signal processor 210 continues to use the image post-processing method provided by the present invention to post-process the initial Bayer image obtained after hardware rearrangement mosaic processing. The repaired Bayer image obtained after post-processing is then transmitted to the application processor 100 for subsequent processing.

[0044] It should be noted that for information on how to train lightweight deep learning models, please refer to relevant content in the field of deep learning that is known to those skilled in the art for an adaptive understanding, and will not be elaborated upon here.

[0045] In some exemplary embodiments, the critical region 310 (e.g. Figure 7 (As shown) includes at least one of the following: an autofocus area, a face area, and a salient object area. This setting allows deep learning post-processing to precisely target the image subject that the photographer is most interested in and that the human eye is most sensitive to, thus concentrating limited computational resources precisely on the areas that most significantly improve image quality. Furthermore, by limiting the key area 310 to subject areas such as autofocus, faces, and salient objects (e.g., food, flowers, animals), non-critical background areas can be completely preserved without requiring deep learning post-processing. This not only further reduces computational overhead but also ensures that the original style of the background area remains unchanged.

[0046] In some exemplary embodiments, determining the critical and non-critical regions in the initial Bayer image includes: designating a rectangular region at a predetermined central ratio of the initial Bayer image as the critical region; and designating other regions of the initial Bayer image located outside the critical region as the non-critical regions.

[0047] Therefore, by using a rectangular area with a preset central ratio from the initial Bayer image as the key region, the entire post-processing workflow can run quickly, effectively reducing power consumption and latency. Furthermore, since photographers typically place the main subject (such as people, plants, or animals) in the center of the frame in most everyday photography scenarios, which highly overlaps with the preset central ratio rectangular area, using this key region from the initial Bayer image allows for coverage of most scenes at minimal cost.

[0048] Please refer to Figure 7 This is a schematic diagram illustrating the determination of critical and non-critical regions according to an embodiment of the present invention. Figure 7 As shown, since the central area of ​​an image is usually the main content and focus point, a rectangular area with a preset proportion in the center of the initial Bayer image can be selected as the key area 310. It should be noted that the present invention does not limit the specific value of the preset proportion. The preset proportion can be, but is not limited to, 1 / 2 (that is, half of the central area of ​​the initial Bayer image can be used as the key area 310, that is, the key area 310 is located in the central area of ​​the initial Bayer image, and the width of the key area 310 is 1 / 2 of the width of the entire initial Bayer image, and the height of the key area 310 is 1 / 2 of the height of the entire initial Bayer image).

[0049] In some other exemplary embodiments, determining the critical regions 310 and non-critical regions 320 in the initial Bayer image includes: using a target detection model deployed on the application processor 100 to detect the initial Bayer image to identify the critical regions 310 in the initial Bayer image; and designating other regions of the initial Bayer image located outside the critical regions 310 as the non-critical regions 320.

[0050] Therefore, by employing the target detection model deployed on the application processor 100, key regions 310 such as faces, salient targets, and focus points (autofocus areas) in the initial Bayer image can be accurately identified, avoiding misprocessing of non-subject areas (non-key areas 320) and ensuring that the original noise distribution and style of the background area are fully preserved. Furthermore, since the target detection model is a lightweight model already deployed on the application processor 100 for implementing functions such as face recognition, scene recognition, and autofocus, there is no need to train or deploy new dedicated detection models, thus avoiding additional model storage and training overhead and further reducing costs.

[0051] Please continue to refer to this. Figure 8 This is a schematic diagram illustrating the determination of critical regions 310 and non-critical regions 320 according to another embodiment of the present invention. Figure 8 As shown, the object detection model deployed on the application processor 100 can understand the image content from a semantic level and accurately locate the main subject area (key area 310) that the user is truly concerned about. This allows for the accurate delineation of irregular shapes such as facial contours and foreground object boundaries, enabling subsequent deep learning post-processing to act on the key area 310 that truly needs repair pixel by pixel. This further allows limited computing resources to be precisely invested in the areas most critical to improving subjective image quality.

[0052] Please continue to refer to this. Figure 9This is a network structure diagram of a lightweight deep learning model provided in one embodiment of the present invention. For example... Figure 9 As shown, in some exemplary embodiments, the lightweight deep learning model includes a pixel rearrangement layer 410, multiple cascaded residual modules 420, a pixel inverse rearrangement layer 430, and a global skip connection layer 440 connected in sequence. The pixel rearrangement layer 410 is used to perform pixel rearrangement on the key region image to obtain a four-channel first feature map, wherein the four channels of the first feature map correspond one-to-one with the four color components of the key region image. The multiple cascaded residual modules 420 are used to extract high-frequency residual information to generate a second feature map. The pixel inverse rearrangement layer 430 is used to perform pixel inverse rearrangement on the second feature map output by the last residual module 420 to obtain a high-frequency residual image with the same size as the key region image. The global skip connection layer 440 is used to add the key region image and the high-frequency residual image pixel by pixel to obtain the processed key region image.

[0053] Therefore, the pixel rearrangement layer 410 can rearrange the key region image with height H and width W into a first feature map with height H / 2, width W / 2, and 4 channels. In other words, the pixel rearrangement layer 410 can rearrange spatially adjacent pixels in the key region image according to their color components into a four-channel first feature map. The four channels of the first feature map correspond to the four color components R, Gr, Gb, and B in the key region image, respectively. This allows each convolution kernel channel of the subsequent residual module 420 to independently operate on its corresponding color component during convolution operations, specifically extracting high-frequency artifacts and false color features related to each color component. This avoids crosstalk between different color components and significantly improves the accuracy and effect of false color and artifact restoration. Furthermore, through the collaborative work of multiple cascaded residual modules 420, high-frequency artifacts and false color components of different scales and types in the image can be gradually separated from shallow to deep and from coarse to fine, thus facilitating the acquisition of more complete and comprehensive high-frequency residual information and ensuring that the final processed key region image has superior visual quality. Furthermore, by employing a skip connection layer to directly pass the unprocessed key region image to the summation node at the output of the lightweight deep learning model, and adding it pixel-by-pixel with the high-frequency residual image, the low-frequency basic information of the key region image can be transmitted without loss. This allows the deep learning model to focus on learning how to repair high-frequency defects such as artifacts and false colors left over from hardware rearrangement mosaic processing, without needing to learn how to reconstruct or preserve these low-frequency contents. This effectively reduces the learning burden of the deep learning model, enabling it to complete complex tasks with a very shallow structure. Simultaneously, since the lightweight deep learning model does not include any downsampling or upsampling operations throughout the entire processing, from the input of the pixel rearrangement layer 410 to the output of the pixel inverse rearrangement layer 430, it can maintain lightweight computation while maximizing the accuracy of high-frequency details.

[0054] It should be noted that the present invention does not limit the number of residual modules 420 in the lightweight deep learning model. The number of residual modules 420 can be set according to actual needs. For example, the number of residual modules 420 can be 2, 3, 4, 5 or more (i.e. N=2,3,4,5,...).

[0055] In some exemplary embodiments, at least two of the cascaded residual modules 420 have convolutional kernel sizes that differ from each other. Since convolutional kernels of different sizes have different receptive fields—for example, smaller kernels (e.g., 3×3 kernels) can focus on small local neighborhoods, accurately capturing pixel-level artifacts and point-like false colors, while larger kernels (e.g., 5×5 or 7×7 kernels) can cover a wider spatial range, effectively identifying artifacts and false colors distributed in stripes or blocks—multi-scale high-frequency residual information can be extracted by configuring convolutional kernels of different sizes at different levels. This enhances the adaptability of lightweight deep learning models to different texture complexities.

[0056] Please continue to refer to this. Figure 10 This is a schematic diagram of the residual module 420 provided in one embodiment of the present invention. Figure 10 As shown, in some exemplary embodiments, each residual module 420 includes a first convolutional layer 421, a first activation layer 422, a second convolutional layer 423, and a second activation layer 424 connected in sequence. The output of the first activation layer 422 and the input of the first convolutional layer 421 are added together to serve as the input of the second convolutional layer 423, and the output of the second activation layer 424 and the input of the first convolutional layer 421 are added together to serve as the output of the residual module 420.

[0057] Since each residual module 420 contains two cascaded convolution-activation units, and each stage has a jump connection from the input end to the output end of that stage (the first-stage jump connection adds the output of the first activation layer 422 to the input of the first convolution layer 421 and sends it to the second convolution layer 423, and the second-stage jump connection adds the output of the second activation layer 424 to the input of the first convolution layer 421 and uses it as the output of the residual module 420), this design can form a multi-stage residual learning mechanism within the residual module 420, enabling lightweight deep learning models to gradually extract and refine high-frequency residual information with finer granularity, thereby facilitating more refined and thorough separation of high-frequency artifacts and false colors. Furthermore, by setting two skip connection paths in each residual module 420, the gradient signal during backpropagation can be directly fed back from the output of the residual module 420 to the input of the first convolutional layer 421, avoiding the attenuation or vanishing of gradients that may occur when passing through multiple convolutional layers and activation functions. This makes the training process of the lightweight deep learning model more stable and efficient.

[0058] In some exemplary embodiments, the step of fusing the processed key region image and the non-key region image using a preset smoothing fusion algorithm includes: for pixels located within the key region 310, setting the fusion weight of the pixel value in the processed key region image to 1, and setting the fusion weight of the pixel value in the non-key region image to 0; for pixels located within the non-key region 320, setting the fusion weight of the pixel value in the processed key region image to 0, and setting the fusion weight of the pixel value in the non-key region image to 1; for pixels located in the transition zone between the key region 310 and the non-key region 320, setting the fusion weight of the pixel value in the processed key region image to between 0 and 1, and if the pixel is closer to the key region 310 (i.e., the smaller the distance between it and the boundary of the key region 310), the greater the fusion weight of the pixel value in the processed key region image.

[0059] Therefore, by setting a transition zone at the boundary between the critical region 310 and the non-critical region 320, and assigning a gradual blending weight between 0 and 1 to the pixels within the transition zone, abrupt brightness jumps, color banding, or sharpness differences can be avoided at the boundary between the processed critical region image and the unprocessed non-critical region image, thus ensuring the overall visual consistency of the final output image. Furthermore, for the core region pixels located within the critical region 310 (outside the non-critical region 320 and the transition zone), setting the blending weight of their pixel values ​​in the processed critical region image to 1 (i.e., the pixel value is entirely adopted from the processed critical region image) ensures that the artifact and false color restoration effects of deep learning post-processing are preserved to the maximum extent in the critical region 310, thereby effectively guaranteeing image quality improvement. For background pixels located within the non-critical region 320 (outside the transition zone and critical region 310), setting the fusion weight of their pixel values ​​in the processed critical region image to 0 (i.e., the pixel values ​​are entirely adopted from the unprocessed non-critical region image) ensures that the original texture, noise distribution, and color style of the non-critical region 320 (background region) are completely unaffected by deep learning post-processing.

[0060] It should be noted that the width of the transition band can be flexibly configured according to the actual application scenario. For example, in the performance-priority preview mode, a narrower transition band can be set to reduce the amount of mixing calculations; in the image quality-priority snapshot mode, a wider transition band can be set to obtain a smooth and delicate fusion effect.

[0061] Based on the same inventive concept, the present invention also provides an image sensor 200, please refer to... Figure 11This is a block diagram of the image sensor 200 provided in one embodiment of the present invention. Figure 11 As shown, the image sensor 200 provided by the present invention includes an image signal processor 210, which is configured to implement the image post-processing method described above. Since the image sensor 200 and the image post-processing method provided by the present invention belong to the same inventive concept, the image sensor 200 provided by the present invention has at least all the beneficial effects of the image post-processing method provided by the present invention. For details, please refer to the relevant descriptions above; further elaboration will not be repeated here.

[0062] Please continue to refer to this. Figure 11 ,like Figure 11 As shown, the image sensor 200 provided by the present invention further includes a pixel array 220, a readout circuit 230, a timing and system control module 240, and an output interface 250. Under the coordinated control of the timing and system control module 240, the pixel array 220 converts the acquired raw optical signal into an electrical signal. The readout circuit 230 performs analog-to-digital conversion on the electrical signal to obtain raw image data (a Q×Q color filter array image), and sends the converted raw image data (a Q×Q color filter array image) to an image signal processor 210. The image signal processor 210 performs hardware rearrangement mosaic processing on the Q×Q color filter array image to obtain an initial Bayer image. Then, the image post-processing method described above is used to post-process the initial Bayer image to obtain a repaired Bayer image. The repaired Bayer image is transmitted to the application processor 100 through the output interface 250.

[0063] Based on the same inventive concept, the present invention also provides an application processor 100, which includes a neural network processor 110 configured to implement the image post-processing method described above. Since the application processor 100 provided by the present invention and the image post-processing method provided by the present invention belong to the same inventive concept, the application processor 100 provided by the present invention possesses at least all the beneficial effects of the image post-processing method provided by the present invention. Therefore, the beneficial effects of the application processor 100 provided by the present invention can be referred to the relevant descriptions of the beneficial effects of the image post-processing method provided by the present invention above, and will not be repeated here.

[0064] Based on the same inventive concept, the present invention also provides an electronic device, which includes the image sensor 200 and / or the application processor 100 described above. Since the electronic device provided by the present invention includes the image sensor 200 or application processor 100 provided by the present invention, it also possesses at least all the beneficial effects of the image post-processing method provided by the present invention. Specific details can be found in the relevant descriptions above, and will not be repeated here. It should be noted that the electronic device provided by the present invention can be, but is limited to, a camera, video camera, tablet computer, learning machine, or medical imaging device.

[0065] In summary, compared with the prior art, the image post-processing method, image sensor, application processor, and electronic device provided by the present invention have the following beneficial effects:

[0066] In this invention, the lightweight deep learning model only handles the repair of high-frequency defects such as artifacts and false colors left over from hardware-based mosaic re-processing, without altering the low-frequency basic information of the image. This not only effectively eliminates artifacts and false colors, significantly improving image quality, but also preserves the efficiency of hardware-based mosaic re-processing. Furthermore, since this invention only uses a lightweight deep learning model to repair false colors and artifacts in key regions of the initial Bayer image after hardware-based mosaic re-processing, it can effectively reduce computational overhead and memory bandwidth usage while improving the image quality of key regions, thus achieving an optimal balance between processing speed, power consumption, and image quality. Additionally, by employing a preset smoothing fusion algorithm to fuse the processed key region and non-key region images, this invention can eliminate abrupt changes or artifacts at the boundaries between key and non-key regions, allowing for a pixel-level natural transition between the two regions, thereby ensuring the overall visual consistency of the fused image. Meanwhile, since the network structure of the lightweight deep learning model used in this invention is shallow, the amount of computation and parameters is extremely small, and the inference speed is extremely fast. It can even meet the real-time processing requirements of video stream frame rate. Furthermore, since the lightweight deep learning model has low hardware resource requirements, the image post-processing method provided by this invention can not only be deployed in the neural network processor of the application processor, but also integrated into the image signal processor inside the image sensor, thus adapting to different hardware platforms.

[0067] It should be noted that the apparatus and methods disclosed in the embodiments herein can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments herein. In this regard, each block in a flowchart or block diagram may represent a module, program, or part of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system to perform the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of this article can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0068] It should also be noted that the above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure are within the protection scope of the present invention. Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the present invention and its equivalents, the present invention also intends to include these modifications and variations.

Claims

1. An image post-processing method, characterized in that, include: Obtain the initial Bayer image after hardware rearrangement mosaic processing; Identify the key and non-key regions in the initial Bayer image, and extract the key region images and non-key region images; A pre-trained lightweight deep learning model is used to post-process the key region image to repair the false colors and artifacts generated after hardware rearrangement mosaic processing, thereby obtaining the processed key region image. A preset smoothing fusion algorithm is used to fuse the processed key region image and the non-key region image to obtain the repaired Bayer image.

2. The image post-processing method according to claim 1, characterized in that, The lightweight deep learning model includes a pixel rearrangement layer, multiple cascaded residual modules, a pixel inverse rearrangement layer, and a global skip connection layer connected in sequence. The pixel rearrangement layer is used to perform pixel rearrangement operation on the key region image to obtain a four-channel first feature map, and the four channels of the first feature map correspond one-to-one with the four color components of the key region image. The multiple cascaded residual modules are used to extract high-frequency residual information to generate a second feature map; The pixel inverse rearrangement layer is used to perform pixel inverse rearrangement operation on the second feature map output by the last stage residual module to obtain a high-frequency residual image with the same size as the key region image. The global skip connection layer is used to add the key region image to the high-frequency residual image pixel by pixel to obtain the processed key region image.

3. The image post-processing method according to claim 2, characterized in that, At least two of the cascaded residual modules have different kernel sizes.

4. The image post-processing method according to claim 2, characterized in that, Each residual module includes a first convolutional layer, a first activation layer, a second convolutional layer, and a second activation layer connected in sequence. The output of the first activation layer and the input of the first convolutional layer are added together to serve as the input of the second convolutional layer, and the output of the second activation layer and the input of the first convolutional layer are added together to serve as the output of the residual module.

5. The image post-processing method according to claim 1, characterized in that, The determination of key and non-key regions in the initial Bayer image includes: The key region is a rectangular area with a preset ratio in the middle of the initial Bayer image, or the initial Bayer image is detected by a target detection model deployed on the application processor to identify the key region in the initial Bayer image. The regions in the initial Bayer image that are outside the critical regions are designated as the non-critical regions.

6. The image post-processing method according to claim 1, characterized in that, The key area includes at least one of the following: the autofocus area, the face area, and the salient target area.

7. The image post-processing method according to claim 1, characterized in that, The step of fusing the processed key region image and the non-key region image using a preset smoothing fusion algorithm includes: For a pixel located within the critical region, the fusion weight of the pixel value in the processed critical region image is set to 1, and the fusion weight of the pixel value in the non-critical region image is set to 0. For a pixel located in the non-critical area, the fusion weight of the pixel value in the processed critical area image is set to 0, and the fusion weight of the pixel value in the non-critical area image is set to 1. For pixels located in the transition zone between the critical region and the non-critical region, the fusion weight of the pixel value in the processed critical region image is set to between 0 and 1. The closer the pixel is to the critical region, the greater the fusion weight of the pixel value in the processed critical region image.

8. An image sensor, characterized in that, It includes an image signal processor configured to implement the image post-processing method as described in any one of claims 1 to 7.

9. An application processor, characterized in that, The method includes a neural network processor configured to implement the image post-processing method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, This includes the image sensor as described in claim 8 or the application processor as described in claim 9.

Images