Image demosaicing method, device and equipment

Abstract

The invention provides an image demosaicing method, device and equipment, and is applied to the technical field of image processing. The method comprises the following steps: for a to-be-processed image containing a mosaic area, determining an area type of each pixel in a mosaic image based on the mosaic image in the to-be-processed image; and matching a corresponding color interpolation strategy for each pixel according to the region type. Color interpolation is carried out on all pixels based on a color interpolation strategy, and a preliminary demosaiced image is generated; the problems that a traditional image demosaicing method is poor in adaptability, low in processing efficiency and rough in processing granularity can be solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image de-mosaic method, apparatus and device. Background Technology

[0002] With the widespread application of digital imaging technology, image sensors play a core role in smartphones, digital cameras, surveillance equipment, and various embedded vision systems. Currently, mainstream image sensors, such as charge-coupled devices (CCDs) and complementary metal-oxide-semiconductor (CMOS) sensors, generally employ Bayer color filter arrays (CFAs) to capture color images. In this array layout, each pixel location is covered by only one of the three primary colors: red, green, and blue. Therefore, each pixel can only sense the intensity information of a single color. To obtain a complete full-color image, the two missing color channels must be reconstructed; this process is called demosaicking.

[0003] Traditional demosaic techniques have evolved into various algorithmic approaches, including classical interpolation-based methods, optimization-based reconstruction methods, and deep learning-based methods. These methods typically begin by analyzing the color distribution characteristics of local pixels and then selecting an appropriate interpolation strategy to recover the missing color information. However, in practical applications, image content is highly diverse, containing various types such as smooth regions, sharp edges, fine textures, and complex structures. Because traditional demosaic techniques often rely on preset fixed interpolation patterns, they lack the ability to dynamically recognize and respond to the characteristics of different image regions, leading to distortions such as color aliasing and jaggies in edge or texture-dense areas. Summary of the Invention

[0004] The purpose of this application is to provide an image demosaic method, apparatus, and device to solve the problems of poor adaptability, low processing efficiency, and coarse processing granularity in traditional image demosaic methods.

[0005] In a first aspect, embodiments of this application provide an image de-mosaic method. This method includes, for an image to be processed containing mosaic regions, determining the region type of each pixel in the mosaic image based on the mosaic image in the image to be processed; matching a corresponding color interpolation strategy for each pixel according to the region type; and performing color interpolation on all pixels based on the color interpolation strategy to generate a preliminary de-mosaic image.

[0006] The image demosaicing method provided in this application first performs semantic-aware analysis on the input mosaic image, dividing each pixel region into smooth regions, edge regions, or high-frequency texture regions. Then, it dynamically selects the corresponding interpolation strategy based on the region type: bilinear interpolation is used for smooth regions for fast processing, edge-guided interpolation is used for edge regions to maintain edge sharpness, and texture-optimized interpolation based on channel ratio consistency is used for texture regions to effectively suppress false color and jagged edges. Finally, by performing adaptive interpolation and optional post-processing optimization on the entire image, a high-quality full-color image is generated. This application not only overcomes the shortcomings of traditional algorithms in texture processing but also avoids the high computational burden of deep learning methods, achieving a balance between image quality and processing efficiency. It is particularly suitable for real-time image processing scenarios in embedded image signal processors and mobile devices.

[0007] One possible implementation involves determining the region type of each pixel in a mosaic image within the image to be processed. This includes: for any target pixel among multiple pixels in the mosaic image, calculating the gray-level variance of the local mosaic image within a preset sampling window. The preset window is a sampling window of a preset size centered on the target pixel. If the gray-level variance is less than or equal to a gray-level variance threshold, the region type of the target pixel is determined to be a smooth region. If the gray-level variance is greater than the gray-level variance threshold, the gradient magnitude and direction of the preset sampling window are calculated. If the gradient magnitude is greater than a magnitude threshold and the directionality is significant, the region type of the target pixel is determined to be an edge region. If the gradient magnitude is greater than a magnitude threshold and the directionality is disordered, the region type of the target pixel is determined to be a textured region.

[0008] One possible implementation involves determining the region type of each pixel in a mosaic image within the image to be processed. This includes: inputting the mosaic image into a pre-trained convolutional neural network to obtain the classification probability of each pixel in the mosaic image; and determining the region type of the corresponding pixel based on the classification probability.

[0009] One possible implementation involves performing color interpolation on all pixels using a color interpolation strategy to generate a preliminary demosaic image. This includes: for any target pixel among multiple pixels in the mosaic image, reconstructing the missing color channel of the target pixel based on the corresponding color interpolation strategy; determining the complete RGB color channel values ​​of the target pixel based on its original color channel and the reconstructed missing color channel; and generating a preliminary demosaic image based on the complete RGB color channel values ​​of all pixels in the mosaic image.

[0010] One possible implementation involves, when the pixel's region type is an edge region, reconstructing the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel. This includes: calculating the gradient values ​​of the target pixel in multiple preset directions; determining the direction with the minimum gradient value as the edge direction of the target pixel; performing interpolation along the edge direction of the target pixel to reconstruct the green channel of the target pixel; and based on the reconstructed green channel, performing interpolation along the edge direction of the target pixel to obtain the red and blue channels of the target pixel.

[0011] One possible implementation, when the pixel's region type is a texture region, involves reconstructing the missing color channel of the target pixel based on a color interpolation strategy corresponding to the target pixel. This includes: determining the initial value of the target pixel's green channel based on the same-color pixels existing within the target pixel's neighborhood window; calculating a first average scaling factor between the known red channel values ​​and the known green channel values ​​within the neighborhood window, and a second average scaling factor between the known blue channel values ​​and the known green channel values; multiplying the initial green channel value by the first average scaling factor to reconstruct the target pixel's red channel; and multiplying the initial green channel value by the second average scaling factor to reconstruct the target pixel's blue channel.

[0012] One possible implementation involves, when the pixel's region type is a smooth region, reconstructing the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel. This includes: within the neighborhood of the target pixel, acquiring pixel values ​​that are adjacent to the target pixel and have the same color as the missing color channel, based on the target pixel's missing color channel; and averaging all acquired pixel values ​​to obtain the missing color channel.

[0013] One possible implementation of the image demosaic method provided in this application embodiment further includes: performing pseudo-color suppression processing on the preliminary demosaic image to obtain a full-color demosaic image. And / or, performing edge enhancement processing on the preliminary demosaic image to obtain a full-color demosaic image.

[0014] Secondly, embodiments of this application provide an image de-mosaic device, which includes: a determining module, a matching module, and a generating module.

[0015] The determination module is used to determine the region type of each pixel in the mosaic image based on the mosaic image in the image to be processed, which contains a mosaic region.

[0016] The matching module is used to match the corresponding color interpolation strategy for each pixel based on the region type.

[0017] The generation module is used to perform color interpolation on all pixels based on a color interpolation strategy to generate a preliminary de-mosaic image.

[0018] One possible implementation involves a determining module that, when determining the region type of each pixel in a mosaic image based on the mosaic image in the image to be processed, specifically performs the following: For any target pixel among multiple pixels in the mosaic image, calculates the gray-level variance of the local mosaic image within a preset sampling window. The preset window is a sampling window of a preset size centered on the target pixel. If the gray-level variance is less than or equal to a gray-level variance threshold, the region type of the target pixel is determined to be a smooth region. If the gray-level variance is greater than the gray-level variance threshold, calculates the gradient magnitude and direction of the preset sampling window. If the gradient magnitude is greater than a magnitude threshold and the directionality is significant, the region type of the target pixel is determined to be an edge region. If the gradient magnitude is greater than a magnitude threshold and the directionality is disordered, the region type of the target pixel is determined to be a textured region.

[0019] One possible implementation involves a determination module that, when determining the region type of each pixel in a mosaic image based on the mosaic image in the image to be processed, specifically: inputting the mosaic image into a pre-trained convolutional neural network to obtain the classification probability of each pixel in the mosaic image; and determining the region type of the corresponding pixel based on the classification probability.

[0020] One possible implementation involves a generation module that, when performing color interpolation on all pixels based on a color interpolation strategy to generate a preliminary demosaic image, specifically performs the following: For any target pixel among multiple pixels in the mosaic image, reconstructs the missing color channel of the target pixel based on the corresponding color interpolation strategy. Based on the original color channel and the reconstructed missing color channel of the target pixel, determines the complete RGB color channel values ​​of the target pixel. Finally, based on the complete RGB color channel values ​​of all pixels in the mosaic image, generates a preliminary demosaic image.

[0021] One possible implementation involves the generation module, when the pixel's region type is an edge region, specifically performing the following steps when reconstructing the missing color channel of the target pixel based on its corresponding color interpolation strategy: Calculating the gradient values ​​of the target pixel in multiple preset directions; determining the direction with the minimum gradient value as the edge direction of the target pixel; performing interpolation along the edge direction of the target pixel to reconstruct its green channel; and performing further interpolation along the edge direction of the target pixel based on the reconstructed green channel to obtain its red and blue channels.

[0022] One possible implementation, when the pixel's region type is a texture region, involves the generation module, specifically when reconstructing the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel, performing the following steps: Determining the initial value of the target pixel's green channel based on the same-color pixels existing within the target pixel's neighborhood window. Calculating a first average scaling factor between the known red channel values ​​and the known green channel values ​​within the neighborhood window, and a second average scaling factor between the known blue channel values ​​and the known green channel values. Multiplying the initial green channel value by the first average scaling factor to reconstruct the target pixel's red channel. Multiplying the initial green channel value by the second average scaling factor to reconstruct the target pixel's blue channel.

[0023] One possible implementation, when the pixel region type is a smooth region, involves the generation module, when reconstructing the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel, specifically: within the neighborhood of the target pixel, based on the missing color channel of the target pixel, acquiring pixel values ​​that are adjacent to the target pixel and have the same color as the missing color channel. The missing color channel is then obtained by averaging all acquired pixel values.

[0024] In one possible implementation, the image demosaic apparatus provided in this application embodiment is further configured to: perform pseudo-color suppression processing on the preliminary demosaic image to obtain a full-color demosaic image; and / or, perform edge enhancement processing on the preliminary demosaic image to obtain a full-color demosaic image.

[0025] Thirdly, embodiments of this application provide an image demosaic device that has the function of implementing the image demosaic method of the first aspect or any possible implementation thereof. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the image demosaic method described in the first aspect or any possible implementation thereof.

[0027] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, enable the computer to execute the image demosaic method described in the first aspect or any possible implementation thereof.

[0028] The technical effects of any of the design methods in aspects two through five can be found in aspect one or in different possible implementations of aspect one, and will not be repeated here. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 A system structure diagram of an image demosaic system provided in this application embodiment; Figure 2 A flowchart of an image demosaic method provided in this application embodiment; Figure 3 A schematic diagram of an image demosaic device provided in this application embodiment; Figure 4 Another system architecture diagram of an image demosaic system provided for embodiments of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0032] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0033] Due to the high diversity of image content, including smooth regions, sharp edges, fine textures, and complex structures, related technologies have revealed several common problems when handling such varied scenes: Traditional algorithms often rely on preset fixed interpolation patterns, lacking the ability to dynamically recognize and respond to the characteristics of different image regions, resulting in distortions such as color aliasing and jaggies in edge or texture-dense areas; At the same time, although some advanced algorithms have improved reconstruction quality by introducing complex models or network structures, their computational overhead has increased significantly, making it difficult to meet the real-time and low-power requirements of resource-constrained devices; In addition, most current region segmentation mechanisms roughly divide images into flat and edge regions, failing to fully recognize high-frequency textures as a special and crucial visual component, and lacking refined processing mechanisms for such regions, thus limiting further improvement in overall image quality.

[0034] Based on this, embodiments of this application provide an image de-mosaic method, apparatus, and device. The method includes, for an image to be processed containing mosaic regions, determining the region type of each pixel in the mosaic image based on the mosaic image in the image to be processed; matching a corresponding color interpolation strategy for each pixel according to the region type; and performing color interpolation on all pixels based on the color interpolation strategy to generate a preliminary de-mosaic image.

[0035] The image demosaicing method provided in this application first performs semantic-aware analysis on the input mosaic image, dividing each pixel region into smooth regions, edge regions, or high-frequency texture regions. Then, it dynamically selects the corresponding interpolation strategy based on the region type: bilinear interpolation is used for smooth regions for fast processing, edge-guided interpolation is used for edge regions to maintain edge sharpness, and texture-optimized interpolation based on channel ratio consistency is used for texture regions to effectively suppress false color and jagged edges. Finally, by performing adaptive interpolation and optional post-processing optimization on the entire image, a high-quality full-color image is generated. This application not only overcomes the shortcomings of traditional algorithms in texture processing but also avoids the high computational burden of deep learning methods, achieving a balance between image quality and processing efficiency. It is particularly suitable for real-time image processing scenarios in embedded image signal processors and mobile devices.

[0036] The methods provided in the embodiments of this application will now be described in conjunction with the specific accompanying drawings.

[0037] On one hand, embodiments of this application provide an image de-mosaic system. For example... Figure 1 As shown, the image demosaic system 100 may include: a sensor 101, a processor 102, a memory 103, and a display 104.

[0038] Sensor 101 contains raw image data of the Bayer array mosaic. For example, sensor 101 can be a complementary metal-oxide-semiconductor image sensor or a charge-coupled device image sensor. In practical applications, sensor 101 may also include drive circuitry for controlling parameters such as exposure and gain, and an ADC unit for analog-to-digital conversion to output digitized Bayer array image data.

[0039] The processor 102 is used to receive and process the original image data, and to process the original image data using the image de-mosaic method provided in this application embodiment to generate a de-mosaic image. For example, the processor 102 can be an image signal processor, a general-purpose processor, a graphics processor, or an application-specific integrated circuit.

[0040] Specifically, the processor 102 first performs semantic awareness on each pixel or pixel block using built-in region analysis logic (such as performing lightweight CNN inference or traditional gradient calculation) to identify it as a smooth region, edge region, or high-frequency texture region. Then, based on the recognition results, it selects a corresponding strategy from multiple built-in interpolators for color reconstruction: low-complexity bilinear interpolation is used for smooth regions, gradient-guided edge interpolation is used for edge regions to maintain sharpness, and optimized interpolation based on local channel ratio consistency is used for texture regions to suppress false color. After completing color reconstruction of all pixels to obtain a preliminary RGB image, the processor 102 can further perform various image post-processing operations, including but not limited to white balance correction, color correction, gamma correction, noise reduction, and edge enhancement, to generate a final high-quality full-color image.

[0041] The memory 103 can be used to store the raw image data acquired by the sensor 101, intermediate data generated during the processing of the processor 102 (such as region classification maps and preliminary interpolation results), and the final generated full-color image data. For example, the memory 103 may include various types such as cache, static random access memory, or dynamic random access memory.

[0042] Display 104 can be used to display the final full-color image output by processor 102. For example, display 104 can be a mobile phone screen, a car center console screen, a monitoring display, a liquid crystal display, an organic light-emitting diode display, or a micro-display. When the image processed by the system of this application is displayed on display 104, it can effectively reduce false color and jagged distortion at the edges of objects and complex textures (such as clothing fibers, vegetation in natural scenes), thereby providing a clear, natural, and detailed visual experience.

[0043] It should be noted that the above Figure 1The image de-mosaic system 100 shown is merely an example illustrating the application scenario of the solution in this application, and is not intended to limit the application scenario of the solution in this application.

[0044] On the one hand, embodiments of this application provide an image demosaic method, which can be performed by... Figure 1 The image depixelation system 100 is executed as shown. Figure 2 As shown, the method may include the following steps.

[0045] S201, for the image to be processed containing a mosaic region, determine the region type of each pixel in the mosaic image based on the mosaic image in the image to be processed.

[0046] The image to be processed can be the raw image data of the Bayer array directly output by the image sensor without any de-mosaic processing. The mosaic image can be the image to be processed itself, or a specific area in the image to be mosaicked.

[0047] Specifically, the process involves obtaining the image to be processed that contains the mosaic region, and then labeling the region type of each pixel in the mosaic image based on the mosaic image in the image to be processed.

[0048] Region types can include smooth regions, edge regions, and textured regions. Smooth regions refer to areas with gradual pixel value changes and a lack of high-frequency details, such as the sky, walls, or evenly lit surfaces. Edge regions refer to areas where pixel values ​​change drastically along a certain direction, forming clear boundaries, such as the outline of an object or the edges of text. Textured regions refer to areas with frequent pixel value changes, full of high-frequency details but lacking a single dominant direction, and with complex structures, such as grass, fabric, leaves, and animal fur.

[0049] One possible implementation involves determining the region type of each pixel in a mosaic image within the image to be processed. This includes: for any target pixel among multiple pixels in the mosaic image, calculating the gray-level variance of the local mosaic image within a preset sampling window. The preset window is a sampling window of a preset size centered on the target pixel. If the gray-level variance is less than or equal to a gray-level variance threshold, the region type of the target pixel is determined to be a smooth region. If the gray-level variance is greater than the gray-level variance threshold, the gradient magnitude and direction of the preset sampling window are calculated. If the gradient magnitude is greater than a magnitude threshold and the directionality is significant, the region type of the target pixel is determined to be an edge region. If the gradient magnitude is greater than a magnitude threshold and the directionality is disordered, the region type of the target pixel is determined to be a textured region.

[0050] For example, for any target pixel among multiple pixels in a mosaic image, a sampling window of a preset size (e.g., 5×5, 7×7) is defined centered on that pixel. Since each pixel in a mosaic image contains only one color channel information, it must first be converted to grayscale representation, or the single-channel pixel value can be directly used for calculation. First, the grayscale variance of all pixels within this window is calculated. Variance is an effective indicator of the drastic change in brightness in a local area.

[0051] If the grayscale variance is less than or equal to a preset grayscale variance threshold, the region type of the target pixel is determined to be a smooth region. This indicates that the pixel values ​​within this window change gradually, lacking significant edges or texture structures, and belong to a flat or slowly changing region in the image.

[0052] If the grayscale variance is greater than a grayscale variance threshold, it indicates a significant change in the region, requiring further differentiation between directional edges and non-directional complex textures. In this case, the gradient magnitude and direction of pixels within the preset sampling window are calculated, for example using classic edge detection operators such as the Sobel operator or the Prewitt operator. The gradient magnitude reflects the intensity of the change, while the gradient direction indicates the primary orientation of the change.

[0053] If the gradient magnitude is higher than a preset magnitude threshold and the directionality is significant (e.g., by calculating a histogram of local gradient directions, if a dominant direction exists), the region type of the target pixel is determined to be an edge region. This indicates that the region has a clear boundary, and the pixel value changes abruptly along a specific direction.

[0054] When the gradient magnitude is determined to be above a threshold, but the directionality is chaotic (e.g., the local gradient direction is uniformly distributed without a clear dominant direction), the region type of the target pixel is determined to be a texture region. This indicates that the region is full of high-frequency details, such as grass, fabric, hair, etc., whose pixel values ​​change frequently but lack a consistent global direction.

[0055] Another possible implementation involves determining the region type of each pixel in the mosaic image within the image to be processed. This includes: inputting the mosaic image into a pre-trained convolutional neural network to obtain the classification probability of each pixel in the mosaic image; and determining the region type of the corresponding pixel based on the classification probability.

[0056] For example, the entire mosaic image is fed into a pre-trained lightweight convolutional neural network (CNN) model, with each pixel-centered image patch (e.g., 9×9 or 11×11 Bayer pattern patches) as input. This model is specifically designed with a shallow layer count and few parameters (e.g., only 2-4 convolutional, pooling, and fully connected layers) to ensure real-time operation on resource-constrained embedded ISPs or mobile processors.

[0057] The model's forward propagation outputs a classification probability vector for each pixel in the mosaic image, indicating whether it belongs to one of three categories: smooth, edge, or texture. Then, the region type of the corresponding pixel is determined based on this probability vector. A maximum probability decision rule can be used, selecting the category with the highest probability value as the final region type for that pixel. For example, if a pixel's output probability is [smooth: 0.1, edge: 0.15, texture: 0.75], then it is determined to be a texture region.

[0058] In this application, a CNN model is trained offline using a large number of Bayer images containing various scenes (smooth skies, sharp building edges, complex natural textures, etc.) and corresponding manually annotated region maps. After training, its weight parameters can be stored in the device's memory or hardware circuitry. Compared to traditional methods, the CNN-based perception module can better understand the semantic content of images, reducing misjudgments caused by local noise or complex patterns, thus laying a more reliable foundation for subsequent adaptive interpolation.

[0059] S202, based on the region type, matches the corresponding color interpolation strategy for each pixel.

[0060] Specifically, based on pre-defined strategy mapping rules, a color interpolation strategy suitable for the region type of that pixel is matched for each pixel.

[0061] One possible implementation is to match a bilinear interpolation strategy to pixels in regions of type smooth.

[0062] Because color changes are slow and pixel value gradients are small within smooth regions, this process does not place high demands on the edge-preserving and high-frequency detail recovery capabilities of the interpolation algorithm. Bilinear interpolation calculates the missing color by simply averaging the values ​​of neighboring pixels of the same color, offering advantages such as extremely low computational complexity and ease of implementation. Using this strategy in such regions can maximize computational resource savings and improve overall processing efficiency with almost no introduction of visual distortion.

[0063] Another possible implementation is to match a bilinear interpolation strategy to pixels of region type edge region.

[0064] Because edge regions are characterized by clear, directional brightness abrupt changes, traditional fixed-direction interpolation averages across the edges, leading to blurred edges and jagged effects. The method provided in this application matches a bilinear interpolation strategy to the edge regions. It first detects the direction of the local edge (horizontal, vertical, or diagonal), and then strictly performs interpolation along that direction. This ensures that the interpolation operation does not affect the color information on both sides of the edge, thereby maximizing edge sharpness and avoiding false colors and jagged imperfections.

[0065] Another possible implementation is to match pixels of texture type with a bilinear interpolation strategy.

[0066] Because textured regions consist of numerous dense, irregular high-frequency details with chaotic directionality, edge-guided interpolation is prone to failure due to directional misjudgment, resulting in severe color noise (pseudo-color). The method provided in this application matches a bilinear interpolation strategy to pixels in the textured region, assuming that the proportional relationship between the red, green, and blue color channels is relatively stable and consistent within a small local texture block. Therefore, the green channel is first estimated, and then the red and blue channels are extrapolated using the calculated local R / G and B / G average scaling factors. This effectively suppresses the human-eye-sensitive pseudo-color spots caused by color channel mismatch in complex textured regions, and is key to improving the overall naturalness and detail realism of the image.

[0067] S203, perform color interpolation on all pixels based on the color interpolation strategy to generate a preliminary de-mosaic image.

[0068] Specifically, for any target pixel among multiple pixels in the mosaic image, the missing color channel of the target pixel is reconstructed based on the color interpolation strategy corresponding to the target pixel. Based on the original color channel and the reconstructed missing color channel of the target pixel, the complete RGB color channel value of the target pixel is determined. Based on the complete RGB color channel values ​​of all pixels in the mosaic image, a preliminary de-mosaic image is generated.

[0069] For example, in a Bayer array, each pixel location records only one of the three colors: R, G, and B; that is, the recorded color is the pixel's original color channel. This process performs color interpolation on each pixel in the input mosaic image, sequentially applying a color interpolation strategy corresponding to the pixel's region type, according to the scanning order. By utilizing the existing original pixel values ​​of different colors in the pixel's surrounding neighborhood, a specific interpolation algorithm is used to estimate the values ​​of the two missing color channels of the pixel. Then, the pixel's original color channel values ​​are combined with the missing color channels reconstructed through the interpolation algorithm to form a pixel value containing all three color components: R, G, and B. After all pixels in the mosaic image have undergone this reconstruction and combination process, the preliminary demosaic image is obtained. This preliminary demosaic image is a color image with the same resolution as the original image, and each pixel contains complete RGB information.

[0070] One possible implementation involves, when the pixel's region type is an edge region, reconstructing the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel. This includes: calculating the gradient values ​​of the target pixel in multiple preset directions; determining the direction with the minimum gradient value as the edge direction of the target pixel; performing interpolation along the edge direction of the target pixel to reconstruct the green channel of the target pixel; and based on the reconstructed green channel, performing interpolation along the edge direction of the target pixel to obtain the red and blue channels of the target pixel.

[0071] For example, the gradient value or color difference value of the target pixel in its surrounding neighborhood (e.g., using a pattern formed by its nearest neighbor pixels of the same color) is first calculated in several preset directions (typically horizontal, vertical, and two diagonal directions). The direction with the smallest gradient value has the gentlest color change, i.e., it is most likely along the edge of the object.

[0072] After determining the edge direction, select the two nearest green (G) pixel values ​​along that edge direction (green pixels are the most densely packed in the Bayer array and are key to reconstruction accuracy), and perform a simple average or weighted average on them as the reconstructed green channel value of the target pixel.

[0073] After reconstructing the G channel, the red channel value of the target pixel is calculated by applying the relationship between the known red (R) pixel value and the reconstructed neighboring green (G) pixel value (such as the color difference RG) in the same edge direction to the target pixel location. Simultaneously, the blue channel value of the target pixel is calculated by applying the relationship between the known blue (B) pixel value and the reconstructed neighboring green (G) pixel value (such as the color difference BG) in the same edge direction to the target pixel location.

[0074] Another possible implementation, when the pixel region type is a texture region, reconstructs the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel. This includes: determining the initial value of the green channel of the target pixel based on the same-color pixels existing in the neighborhood window of the target pixel; calculating a first average scaling factor of the known red channel value and the known green channel value in the neighborhood window, and a second average scaling factor of the known blue channel value and the known green channel value; multiplying the initial green channel value by the first average scaling factor to reconstruct the red channel of the target pixel; and multiplying the initial green channel value by the second average scaling factor to reconstruct the blue channel of the target pixel.

[0075] For example, within a preset neighborhood window (e.g., 7x7), the target pixel location itself may not have any green pixels. In this case, using all the existing green pixels within the window, the initial value G_initial of the green channel at the target pixel location is estimated using an interpolation method (e.g., a weighted average considering spatial distance and local color similarity).

[0076] Next, within the same neighborhood window, identify all known pixel locations. For a known red pixel, its R value is known, and its G value is also obtained through neighboring green pixels or interpolation (if the pixel itself is not a green pixel), thus allowing the calculation of a local R / G ratio. Calculate the R / G ratio for all such known points and obtain the first average scaling factor R / G_avg. Similarly, calculate the B / G ratio for all known blue pixels and their corresponding green pixels, and obtain the second average scaling factor B / G_avg.

[0077] Finally, the obtained color scaling factor and the initial value of the green channel of the target pixel are input into the following formula to reconstruct its missing color channel.

[0078] The formula for reconstructing the red channel is: R_interp = G_initial * (R / G_avg) The formula for reconstructing the blue channel is: B_interp = G_initial * (B / G_avg) By smoothing the R / G and B / G ratios within a local area, this effectively avoids generating inconsistent color noise points in texture details, thereby suppressing the generation of false color phenomena.

[0079] Another possible implementation involves, when the pixel's region type is a smooth region, reconstructing the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel. This includes: within the neighborhood of the target pixel, acquiring pixel values ​​that are adjacent to the target pixel and have the same color as the missing color channel, based on the target pixel's missing color channel; and averaging all acquired pixel values ​​to obtain the missing color channel.

[0080] For example, based on the nearest neighbor of the target pixel (usually taking its 4 or 8 nearest neighboring pixels), for a missing color channel of the target pixel (e.g., missing B channel), find all pixels in the aforementioned neighborhood that are the original color channels (i.e., B channel).

[0081] Then, the original values ​​of these found pixels of the same color are directly averaged to obtain the average value as the interpolation result of the missing color channel of the target pixel.

[0082] Since the pixel values ​​in the smooth region change little during this process, the error introduced by this simple averaging operation can be ignored, thereby improving the calculation speed.

[0083] Furthermore, after generating the initial demosaic image, the following post-processing operations can be selectively performed to further improve the subjective visual quality of the image.

[0084] One possible implementation is to perform pseudo-color suppression processing on the initial demosaic image to obtain a full-color demosaic image.

[0085] For example, a small (e.g., 3x3 or 5x5) median filter or an edge-preserving filter (e.g., a bilateral filter) can be used to filter each color channel in the initial demosaic image to obtain a full-color demosaic image.

[0086] This application effectively removes isolated noise points by employing a median filter, and uses an edge-preserving filter to suppress false color while preserving important edge and texture details. This results in a more natural and smooth color transition in the final full-color demosaic image.

[0087] Another possible approach is to perform edge enhancement processing on the initial demosaic image to obtain a full-color demosaic image.

[0088] For example, a Gaussian blur is applied to the initial image to obtain a blurred version. Then, this blurred version is subtracted from the original initial image to obtain a mask containing high-frequency edge information. Finally, this mask is added back to the original initial image with a controllable intensity (gain coefficient) to enhance edge contrast.

[0089] It should be noted that the two post-processing methods described above can be used individually or in combination, and this application does not limit this. For example, in actual system implementation, pseudo-color suppression processing can be performed on the initially de-mosaiced image to remove chromatic noise; then, edge enhancement processing can be performed on the image after pseudo-color suppression to restore and strengthen the edge sharpness that may have been slightly softened. This combined processing strategy can synergistically improve the final visual quality of the image without introducing significant side effects, achieving better performance in both color purity and structural clarity.

[0090] The above primarily describes the solutions provided in the embodiments of this application from the perspective of the working principle of the device. It is understood that, in order to achieve the above functions, the image de-mosaic device includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art should readily recognize that, in conjunction with the algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0091] This application embodiment can divide the image demosaicing device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module.

[0092] It should be noted that the module division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. When dividing functional modules according to their respective functions, Figure 3 A schematic diagram of a possible composition of the image demosaic device involved in the above and embodiment is shown. For example... Figure 3 As shown, the image demosaic device 300 may include: a determining module 301, a matching module 302, and a generating module 303.

[0093] The determining module 301 is used to support the execution of the image demosaic device 300. Figure 2 The illustrated image de-mosaic method, S201.

[0094] Matching module 302 is used to support the execution of image demosaic device 300. Figure 2 The illustrated image de-mosaic method, S202.

[0095] Generation module 303 is used to support the execution of image demosaic device 300. Figure 2 The illustrated image de-mosaic method, S203.

[0096] One possible implementation involves a determining module that, when determining the region type of each pixel in a mosaic image based on the mosaic image in the image to be processed, specifically performs the following: For any target pixel among multiple pixels in the mosaic image, calculates the gray-level variance of the local mosaic image within a preset sampling window. The preset window is a sampling window of a preset size centered on the target pixel. If the gray-level variance is less than or equal to a gray-level variance threshold, the region type of the target pixel is determined to be a smooth region. If the gray-level variance is greater than the gray-level variance threshold, calculates the gradient magnitude and direction of the preset sampling window. If the gradient magnitude is greater than a magnitude threshold and the directionality is significant, the region type of the target pixel is determined to be an edge region. If the gradient magnitude is greater than a magnitude threshold and the directionality is disordered, the region type of the target pixel is determined to be a textured region.

[0097] One possible implementation involves a determination module that, when determining the region type of each pixel in a mosaic image based on the mosaic image in the image to be processed, specifically: inputting the mosaic image into a pre-trained convolutional neural network to obtain the classification probability of each pixel in the mosaic image; and determining the region type of the corresponding pixel based on the classification probability.

[0098] One possible implementation involves a generation module that, when performing color interpolation on all pixels based on a color interpolation strategy to generate a preliminary demosaic image, specifically performs the following: For any target pixel among multiple pixels in the mosaic image, reconstructs the missing color channel of the target pixel based on the corresponding color interpolation strategy. Based on the original color channel and the reconstructed missing color channel of the target pixel, determines the complete RGB color channel values ​​of the target pixel. Finally, based on the complete RGB color channel values ​​of all pixels in the mosaic image, generates a preliminary demosaic image.

[0099] One possible implementation involves the generation module, when the pixel's region type is an edge region, specifically performing the following steps when reconstructing the missing color channel of the target pixel based on its corresponding color interpolation strategy: Calculating the gradient values ​​of the target pixel in multiple preset directions; determining the direction with the minimum gradient value as the edge direction of the target pixel; performing interpolation along the edge direction of the target pixel to reconstruct its green channel; and performing further interpolation along the edge direction of the target pixel based on the reconstructed green channel to obtain its red and blue channels.

[0100] One possible implementation, when the pixel's region type is a texture region, involves the generation module, specifically when reconstructing the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel, performing the following steps: Determining the initial value of the target pixel's green channel based on the same-color pixels existing within the target pixel's neighborhood window. Calculating a first average scaling factor between the known red channel values ​​and the known green channel values ​​within the neighborhood window, and a second average scaling factor between the known blue channel values ​​and the known green channel values. Multiplying the initial green channel value by the first average scaling factor to reconstruct the target pixel's red channel. Multiplying the initial green channel value by the second average scaling factor to reconstruct the target pixel's blue channel.

[0101] One possible implementation, when the pixel region type is a smooth region, involves the generation module, when reconstructing the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel, specifically: within the neighborhood of the target pixel, based on the missing color channel of the target pixel, acquiring pixel values ​​that are adjacent to the target pixel and have the same color as the missing color channel. The missing color channel is then obtained by averaging all acquired pixel values.

[0102] In one possible implementation, the image demosaic apparatus provided in this application embodiment is further configured to: perform pseudo-color suppression processing on the preliminary demosaic image to obtain a full-color demosaic image; and / or, perform edge enhancement processing on the preliminary demosaic image to obtain a full-color demosaic image.

[0103] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0104] The image demosaic device 300 provided in this application embodiment is used to perform the above-mentioned... Figure 2 The image depixelation method shown can achieve the same effect as the image depixelation method described above.

[0105] This application also provides an image de-mosaic device that can perform the image de-mosaic method and related steps described in the above method embodiments.

[0106] This application also provides a computer-readable storage medium storing instructions that, when executed, perform the image demosaic method and related steps described in the above method embodiments.

[0107] This application also provides a computer program product that, when run on a computer, causes the computer to execute the image de-mosaic method and related steps described in the above method embodiments.

[0108] In some embodiments, the methods shown in this application can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0109] This application also provides an image demosaic system 100, such as... Figure 4 As shown, the image demosaic system 100 includes at least one processor 401 and at least one interface circuit 402.

[0110] As an example, when the image demosaic system 100 includes a processor and an interface circuit, the processor can be... Figure 4 The processor 401 shown in the solid box (or the processor 401 shown in the dashed box) can be an interface circuit. Figure 4 The interface circuit 402 is shown in the solid box (or the dashed box). When the image demosaic system 100 includes two processors and two interface circuits, the two processors include... Figure 4 The processor 401 shown in the solid box and the processor 401 shown in the dashed box, these two interface circuits include Figure 4 Interface circuit 402 is shown in both solid and dashed boxes. No limitations are imposed on this.

[0111] Processor 401 and interface circuit 402 can be interconnected via a line. For example, interface circuit 402 can be used to receive signals. Alternatively, interface circuit 402 can be used to send signals to other devices (e.g., processor 401). For instance, interface circuit 402 can read computer instructions stored in memory and send those instructions to processor 401. Processor 401 executes the instructions and, in conjunction with input / output devices, implements the various steps in the above embodiments, such as implementing... Figure 2 The methods illustrated are the steps performed in the embodiments shown. Of course, this image demosaic system may also include other discrete components, and this application embodiment does not specifically limit this.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0114] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for depigmenting images, characterized in that, The method includes: For an image to be processed containing a mosaic region, the region type of each pixel in the mosaic image is determined based on the mosaic image in the image to be processed; Based on the region type, a corresponding color interpolation strategy is matched for each pixel; Based on the color interpolation strategy, color interpolation is performed on all pixels to generate a preliminary de-mosaic image.

2. The method according to claim 1, characterized in that, The step of determining the region type of each pixel in the mosaic image based on the mosaic image in the image to be processed includes: For any target pixel among multiple pixels in the mosaic image, calculate the grayscale variance of the local mosaic image of a preset sampling window; the preset window is a sampling window of a preset size centered on the target pixel; If the grayscale variance is determined to be less than or equal to the grayscale variance threshold, the region type of the target pixel is determined to be a smooth region; If the grayscale variance is greater than the grayscale variance threshold, the gradient magnitude and direction of the preset sampling window are calculated; if the gradient magnitude is higher than the magnitude threshold and the directionality is significant, the region type of the target pixel is determined to be an edge region; if the gradient magnitude is higher than the magnitude threshold and the directionality is disordered, the region type of the target pixel is determined to be a texture region.

3. The method according to claim 1, characterized in that, Based on the mosaic image in the image to be processed, determine the region type of each pixel in the mosaic image, including: The mosaic image is input into a pre-trained convolutional neural network to obtain the classification probability of each pixel in the mosaic image; Based on the classification probability, the region type of the corresponding pixel is determined.

4. The method according to claim 1, characterized in that, The step of performing color interpolation on all pixels based on the color interpolation strategy to generate a preliminary de-mosaic image includes: For any target pixel among multiple pixels in the mosaic image, the missing color channel of the target pixel is reconstructed based on the color interpolation strategy corresponding to the target pixel; Based on the original color channels and the reconstructed missing color channels of the target pixel, determine the complete RGB color channel values ​​of the target pixel; The preliminary de-mosaic image is generated based on the complete RGB color channel values ​​of all pixels in the mosaic image.

5. The method according to claim 4, characterized in that, When the region type of a pixel is an edge region, the step of reconstructing the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel includes: Calculate the gradient values ​​of the target pixel in multiple preset directions; The direction of minimum gradient value is determined as the edge direction of the target pixel; Interpolation is performed along the edge direction of the target pixel to reconstruct the green channel of the target pixel; Based on the reconstructed green channel, interpolation is performed along the edge direction of the target pixel to obtain the red and blue channels of the target pixel.

6. The method according to claim 4, characterized in that, When the region type of a pixel is a texture region, the step of reconstructing the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel includes: Based on the same-color pixels existing in the neighborhood window of the target pixel, determine the initial value of the green channel of the target pixel; Calculate the first average scaling factor of the known red channel value and the known green channel value within the neighborhood window, and the second average scaling factor of the known blue channel value and the known green channel value; The red channel of the target pixel is reconstructed by multiplying the initial value of the green channel by the first average scaling factor. The blue channel of the target pixel is reconstructed by multiplying the initial value of the green channel by the second average scaling factor.

7. The method according to claim 4, characterized in that, When the region type of a pixel is a smooth region, the step of reconstructing the missing color channel of the target pixel based on the color interpolation strategy corresponding to the target pixel includes: Within the neighborhood of the target pixel, based on the missing color channel of the target pixel, obtain the pixel value that is adjacent to the target pixel and has the same color as the missing color channel; The missing color channel is obtained by averaging all the acquired pixel values.

8. The method according to claim 1, characterized in that, The method further includes: The initial demosaic image is subjected to pseudo-color suppression processing to obtain a full-color demosaic image; And / or, perform edge enhancement processing on the initial demosaic image to obtain the full-color demosaic image.

9. An image de-mosaic device, characterized in that, The device includes: The determination module is used to determine the region type of each pixel in the mosaic image based on the mosaic image in the image to be processed, for an image containing a mosaic region; The matching module is used to match a corresponding color interpolation strategy for each pixel according to the region type; The generation module is used to perform color interpolation on all pixels based on the color interpolation strategy to generate a preliminary de-mosaic image.

10. An image de-mosaic device, characterized in that, The image demosaic device includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the image demosaic method according to any one of claims 1 to 8.

Images