Image downsampling method and apparatus based on bayer format

Abstract

The application discloses a Bayer format-based image downsampling method and device. The method comprises the following steps: receiving a Raw image and a downsampling size of the Raw image; reading Raw domain data of the Raw image, wherein the Raw domain data comprises Bayer format data and an input size; selecting a target downsampling mode from multiple levels of downsampling modes according to an image scaling ratio of the input size and the downsampling size; performing the target downsampling mode on the Bayer format data in the Raw domain to obtain a downsampling result; and outputting the downsampling result after encapsulating the downsampling result into a Raw format file. In this way, the downsampling mode can be selected according to the image scaling ratio, and then the Bayer format data is downsampled in the Raw domain, so that the calculation amount is reduced and the operation efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image downsampling method and apparatus based on the Bayer format. Background Technology

[0002] Downsampling of a raw image refers to reducing the spatial resolution in the raw data domain (before it is fully converted to a standard RGB image). Traditional mainstream methods involve processing in two stages.

[0003] First, inverse Bayer interpolation (Demosaic) is performed on the raw image using an interpolation algorithm to estimate the missing R / G / B channel values ​​for each pixel, outputting a full-resolution three-channel RGB image. Then, image scaling (Resize) is performed, downsampling each channel of the three-channel RGB image using a scaling algorithm to obtain a scaled-down image at the target resolution.

[0004] However, the process of performing inverse Bayer interpolation has drawbacks such as computational redundancy, color channel crosstalk, loss of Bayer structure, and insufficient anti-aliasing filtering. Summary of the Invention

[0005] The purpose of this application is to provide an image downsampling method and apparatus based on the Bayer format, which can select the downsampling mode according to the image scaling ratio, and then downsample the Bayer format data in the Raw domain to reduce the amount of computation and improve the computational efficiency.

[0006] To achieve the above objectives, this application provides an image downsampling method based on the Bayer format, the method comprising:

[0007] Receive the raw image and its downsampled size;

[0008] Read the Raw domain data of the Raw image, the Raw domain data including Bayer format data and input dimensions;

[0009] Based on the input size and downsampling size, the image scaling ratio is obtained, and based on the image scaling ratio, the target downsampling mode is selected from the multi-level downsampling modes;

[0010] The target downsampling mode is applied to the Bayer format data in the Raw domain to obtain the downsampling result;

[0011] The downsampling results are then encapsulated into a Raw format file and output.

[0012] Furthermore, in the Bayer-based image downsampling method, the target downsampling mode includes a fast downsampling mode and a core downsampling mode. The target downsampling mode also includes a hierarchical downsampling mode, which is a cascaded combination of the fast downsampling mode and the core downsampling mode.

[0013] The fast downsampling is used to scale down the image by an integer multiple; the core downsampling is used to scale down the image by an integer multiple or a non-integer multiple.

[0014] Furthermore, in the Bayer-based image downsampling method, the fast downsampling method employs at least one optional integer multiple.

[0015] Furthermore, in the Bayer-based image downsampling method, the step of selecting a target downsampling mode from multiple downsampling modes according to the image scaling ratio includes:

[0016] When the image scaling ratio is 1 / n, where n is at least one optional integer multiple, a fast downsampling method is selected, and at least one optional integer multiple scaling downsampling is performed.

[0017] Furthermore, in the Bayer-based image downsampling method, the step of selecting the target downsampling mode from the multi-level downsampling modes also includes:

[0018] When the image scaling ratio is less than a preset threshold, a hierarchical downsampling method is selected, and a multi-level cascade of fast downsampling and core downsampling is performed, or a multi-level cascade of fast downsampling and fast downsampling is performed, or a multi-level cascade of core downsampling and core downsampling is performed.

[0019] When the image scaling ratio is greater than a preset threshold, core downsampling is selected and core downsampling is performed.

[0020] Furthermore, in the Bayer-based image downsampling method, the step of performing the fast downsampling includes:

[0021] Based on the channel arrangement pattern of Bayer format data, pixels in the same channel are filtered and merged.

[0022] Furthermore, in the Bayer-based image downsampling method, the step of performing the core downsampling includes:

[0023] Perform support domain calculations to obtain support domain data;

[0024] Perform anti-aliasing kernel calculations to obtain the anti-aliasing kernel function coefficients;

[0025] Perform channel isolation calculation to obtain the channel isolation coefficient;

[0026] The downsampling result is obtained by convolving the support domain data, anti-aliasing kernel coefficients, and channel isolation coefficients.

[0027] Furthermore, in the Bayer-based image downsampling method, the step of performing support domain calculation to obtain support domain data includes:

[0028] The position index and phase index are obtained by performing index calculations in the vertical and horizontal directions respectively through reverse mapping.

[0029] Select the supporting field data in the location index along the preset window.

[0030] Furthermore, in the Bayer-based image downsampling method, the step of performing anti-aliasing kernel calculation to obtain the anti-aliasing kernel function coefficients includes:

[0031] Using the phase index, the filter coefficients in the horizontal and vertical directions are calculated by looking up the table from the reference position phase index and using the anti-aliasing kernel function.

[0032] The anti-aliasing kernel function coefficients are obtained based on the filtering coefficients in the horizontal and vertical directions.

[0033] Furthermore, in the Bayer-based image downsampling method, the step of performing channel isolation calculation to obtain the channel isolation coefficients includes:

[0034] Determine the Bayer channel type for the current downsampled output position and the reference position in the support domain;

[0035] The channel isolation coefficient corresponding to the channel coupling factor is calculated based on the Bayer channel type.

[0036] To achieve the above objectives, this application also provides a downsampling device for Bayer domain images, including a data receiving module, a data reading module, a mode selection module, a downsampling execution module, and a data output module, wherein...

[0037] The data receiving module is used to receive raw images and their downsampling dimensions;

[0038] The data reading module is used to read the Raw domain data of the Raw image, which includes Bayer format data and input dimensions;

[0039] The mode selection module is used to select a target downsampling mode from multiple downsampling modes based on the image scaling ratio of the input size and the downsampling size;

[0040] The downsampling execution module is used to perform the target downsampling mode on the Bayer format data in the Raw domain to obtain the downsampling result;

[0041] The data output module is used to encapsulate the downsampling results into a Raw format file and then output it.

[0042] Furthermore, in the Bayer-based image downsampling device, the downsampling execution module includes a fast downsampling unit and a core downsampling unit, and at least one of the fast downsampling unit and the core downsampling unit is invoked according to the selected target downsampling mode.

[0043] Furthermore, in the Bayer-based image downsampling device, the core downsampling unit includes a support domain calculator, an anti-aliasing kernel calculator, a channel isolator, and a convolutional unit. The support domain calculator is used to perform support domain calculations; the anti-aliasing kernel calculator is used to perform anti-aliasing kernel calculations; the channel isolator is used to perform channel isolation calculations; and the convolutional unit is used to perform convolution calculations.

[0044] The image downsampling method and apparatus based on Bayer format provided in this application include receiving a Raw image and its downsampling size, then reading the Raw domain data of the Raw image, the Raw domain data including Bayer format data and input size, then selecting a target downsampling mode from multi-level downsampling modes according to the image scaling ratio of the input size and the downsampling size, then performing the target downsampling mode on the Bayer format data in the Raw domain to obtain a downsampling result, and finally encapsulating the downsampling result into a Raw format file and outputting it. This design, by selecting the downsampling mode based on the image scaling ratio and then downsampling the Bayer format data in the Raw domain, reduces computational load while improving operational efficiency; it achieves downsampling at any ratio in the Bayer format while ensuring image clarity and detail restoration without introducing moiré patterns and false colors; selecting the target downsampling mode based on the image scaling ratio avoids the kernel function expansion problem in high-magnification single-level downsampling, reducing the caching cost required for the kernel function support domain, eliminating the need for extensive redundant calculations, and allowing insertion and use at any position in the ISP Bayer domain, thus having wide applicability. Attached Figure Description

[0045] Those skilled in the art will understand that the accompanying drawings are provided to better understand this application and do not constitute any limitation on the scope of this application.

[0046] Figure 1 This is a flowchart illustrating the image downsampling method based on the Bayer format according to an embodiment of this application;

[0047] Figure 2 This is a schematic diagram illustrating the effect of Bayer format data pixel merging according to an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of the layout of a support domain and a reference position according to an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the layout of a support domain with Gr as the reference position according to an embodiment of this application;

[0050] Figure 5 This is a schematic diagram of the structure of an image downsampling device based on the Bayer format provided in an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the structure of a core downsampling unit provided in an embodiment of the present invention. Detailed Implementation

[0053] Interpolation algorithms used for performing demosaic interpolation can include bilinear interpolation, gradient-based interpolation, and AMaZE (Adaptive Homogeneity-Directed). Scaling algorithms used for image resizing can include bilinear interpolation, bicubic interpolation, Lanczos, and box filtering.

[0054] In inverse Bayer interpolation, each pixel of the raw image is expanded into three RGB values, thus increasing the computational load to three times the original. However, in the subsequent downsampling operation, approximately 70% to 80% of the interpolated pixels are discarded. This "expand first, discard later" approach results in significant computational waste and a large amount of redundant computation in high-ratio downsampling scenarios.

[0055] Meanwhile, the three channels of the output RGB image have different spatial frequency characteristics: the G channel, with the highest sampling rate (covering 50% of the pixels), contains the richest high-frequency luminance information; while the R and B channels have the same sampling rate, each covering only 25%. However, during the scaling stage, the system usually uses a uniform kernel function to process the three channels in the same way. This causes the high-frequency aliasing of the G channel to spread to the R / B channels, especially in the high spatial frequency region, which is prone to producing color moiré patterns and causing color crosstalk between channels.

[0056] Furthermore, the Bayer CFA structure is completely destroyed during this processing. If subsequent ISP processes or archiving formats require the output to remain in Raw format, an additional "re-Bayerization" step must be performed, i.e., generating the Bayer CFA from the RGB image. This process not only introduces new interpolation errors but also fails to guarantee that the output arrangement is completely consistent with the original sensor structure.

[0057] Currently, related technologies have proposed solutions for direct integer-multiple binning in the raw domain (such as the hardware binning mode of Sony IMX series sensors), which directly averages adjacent pixels of the same color. However, this method has obvious limitations: it only supports integer multiple ratios (such as 2×, 3×, 4×) and cannot handle arbitrary non-integer scaling; it also lacks explicit anti-aliasing filtering, and moiré patterns still occur when downsampling at large ratios; for non-integer ratios, only approximate ratio processing can be used (such as approximating 2.1× as 2×), resulting in a discrepancy between the actual output resolution and the target resolution.

[0058] Based on this, this application provides an image downsampling method and apparatus based on the Bayer format. The Bayer format-based image downsampling method includes receiving a raw image and its downsampling size, then reading the raw domain data of the raw image, the raw domain data including Bayer format data and an input size, then selecting a target downsampling mode from multiple downsampling modes according to the image scaling ratio of the input size and the downsampling size, then performing the target downsampling mode on the Bayer format data in the raw domain to obtain a downsampling result, and finally encapsulating the downsampling result into a raw format file and outputting it. This design allows for direct downsampling of the raw image within the raw domain using Bayer format data, leveraging the characteristics of the Bayer format to ensure independent anti-crosstalk for each channel and avoid color artifacts caused by cross-channel interference; the direct downsampling mode avoids computational redundancy caused by demosaic interpolation, improving computational efficiency.

[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0060] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0061] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, this information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, can be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups.

[0062] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0063] It should be noted that step designations such as S101 and S102 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the protection scope of this application.

[0064] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0065] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0066] In one embodiment, such as Figure 1 The diagram illustrates a flowchart of an image downsampling method based on the Bayer format, which includes the following steps:

[0067] Step S1: Raw image and its downsampling size.

[0068] The Raw image represents the raw data output by the image sensor, including a Raw file containing metadata or a headerless raw data stream. Pixels in the Raw image are arranged in a Bayer array, for example, periodically arranged in RGGB or GRBG. Parsing and reading the Raw image yields the Bayer format data and the input dimensions. The pixel values ​​of the Raw image are directly obtained from the sensor's photoelectric acquisition. The downsampling size represents the target output size of the image after downsampling processing.

[0069] For example, a Raw image is 12-bit image data.

[0070] Step S2: Read the Raw domain data of the Raw image, which includes Bayer format data and input dimensions.

[0071] Specifically, by reading the Raw domain data of a Raw image, Bayer format data can be obtained, along with the input size of the Raw image. Raw domain data refers to the raw pixel data directly output by the image sensor without any image signal processing (ISP). Its core form is Bayer format—each pixel position records only one of the three color components: red, green, and blue, arranged in a checkerboard pattern according to a specific color arrangement rule (such as RGGB), and each pixel value is usually stored in grayscale form as an 8- to 16-bit integer.

[0072] Input dimensions include input height Hin and input width WIn. Further, the input height Hin and input width WIn of the raw image. Bayer format data refers to the raw image data output by the image sensor, recording a single color component at each pixel location, with colors arranged in a specific checkerboard pattern.

[0073] Step S3: Select the target downsampling mode from the multi-level downsampling modes based on the image scaling ratio of the input size and the downsampling size.

[0074] As an example, the image scaling ratio is the ratio of the downsampled size to the input size. The downsampling operation in this embodiment includes width downsampling and height downsampling. Specifically, width and height are downsampled simultaneously. Accordingly, the image scaling ratio can include a height scaling ratio and a width scaling ratio. Specific definitions:

[0075] Height scaling ...(1), HOut represents the height dimension of the output image, HIn represents the height dimension of the Raw image; width scaling ratio ...(2), WOut represents the width of the output image, and WIn represents the width of the Raw image.

[0076] As an example, the target downsampling mode includes a fast downsampling method and a core downsampling method. The target downsampling mode also includes a hierarchical downsampling method, which is a cascaded combination of the fast downsampling method and the core downsampling method. The fast downsampling method is used to scale and downsample the image by an integer multiple; the core downsampling method is used to scale and downsample the image by an integer multiple or a non-integer multiple. The scaled-down image in the fast downsampling method and the core downsampling method can be the original Bayer format data or downsampled Bayer format data.

[0077] Furthermore, the fast downsampling method employs at least one optional integer multiple. The optional integer multiple can be any integer. Considering the hardware ISP computing power, image size constraints, and limited input image size, the integer multiple can, for example, be an integer multiple of 2, 3, 4, etc.

[0078] As an example, the step of selecting a target downsampling mode from multiple downsampling modes based on the image scaling ratio of the input size and the downsampling size includes:

[0079] When the image scaling ratio is 1 / n, where n is at least one optional integer multiple, a fast downsampling method is selected, and at least one optional integer multiple scaling downsampling is performed.

[0080] Furthermore, the step of selecting the target downsampling mode from the multi-level downsampling modes also includes: when the image scaling ratio is less than a preset threshold, selecting a hierarchical downsampling method, performing a multi-level cascade of fast downsampling and core downsampling, or performing a multi-level cascade of fast downsampling with fast downsampling, or performing a multi-level cascade of core downsampling with core downsampling. Further explanation: the fast downsampling in the hierarchical downsampling method can use at least one selectable integer multiple. For example, the preset threshold is 0.5. When the image scaling ratio is greater than the preset threshold, selecting the core downsampling method and performing core downsampling.

[0081] This application presets multiple downsampling methods in a multi-level downsampling mode according to the image scaling ratio. Different downsampling methods have different computational loads. One downsampling method is selected according to the image scaling ratio (i.e., the target downsampling mode is selected), thus reducing the computational load and avoiding computational redundancy caused by the inverse Bayer interpolation design.

[0082] For example, selectable integer multiples are 2, 4, and 8, with a preset threshold of 0.5 and a threshold range of 0.5 to 1. When the image scaling ratio is 1 / 2, a 2x fast downsampling is performed on the Bayer format data. When the image scaling ratio is 1 / 64, and the selectable integer scaling factors include 1 / 2, 1 / 4, and 1 / 8, 1 / 2 and 1 / 4 can be selected as integer scaling factors to perform two 2x and two 4x fast downsampling operations on the Bayer format data. When the image scaling ratio is less than 0.5, for example, when the image scaling ratio is 0.45, its height scaling ratio is less than 0.5, and its width scaling ratio is less than 0.5, a hierarchical downsampling method is selected, initiating two levels of scaling. First, an integer scaling factor of 0.5 is selected to perform a 2x fast downsampling on the Bayer format data to obtain an intermediate image; then, a non-integer scaling factor of 0.9 is used to perform core downsampling on the intermediate image to obtain the target image. The product of the integer scaling factor and the non-integer scaling factor equals the image scaling ratio between the downsampled image and the input image. When the image scaling ratio is greater than 0.5, such as 0.9, you can directly select the core downsampling method and perform core downsampling directly.

[0083] Thus, when the image scaling ratio falls within the threshold range, core downsampling is directly selected. When the image scaling ratio is less than the minimum value of the threshold range, the image scaling ratio can be split, and then a tiered downsampling method can be selected for both integer-multiple fast downsampling and non-integer-multiple core downsampling. This design, using inverse Bayer interpolation (Demosaic), ensures that at high scaling ratios, no additional false colors, jagged edges, or moiré patterns are introduced, while maintaining a consistent overall sharpness.

[0084] Step S4: Perform the target downsampling mode on the Bayer format data in the Raw domain to obtain the downsampling result.

[0085] In this embodiment, performing the target downsampling mode on the Bayer format data in the Raw domain means performing corresponding downsampling processing on the Bayer format data in the Raw domain according to the target downsampling mode. In one embodiment, the target downsampling mode includes a fast downsampling mode, a core downsampling mode, and a hierarchical downsampling mode.

[0086] As an example, the steps of performing the fast downsampling include: filtering pixels in the same channel and merging pixels according to the channel arrangement rules of the Bayer format data.

[0087] To further explain, the core characteristics of Bayer format data include: regular color arrangement, color determined by parity position, checkerboard distribution of G channels, and a pixel interval of 2 for the same color. Regular color arrangement refers to repeating the color in one of four patterns: RGGB, GRBG, GBRG, or BGGR, with a 2×2 periodic unit. Parity position determines color because the parity of the row and column indices jointly determines the color component of a pixel. The checkerboard distribution of G channels means that G pixels occupy half of all rows, exhibiting an alternating distribution. A pixel interval of 2 for the same color means that the minimum sampling interval for the same color channel is 2 pixels.

[0088] By utilizing the parity of the pixel coordinate rows and columns in the Bayer format data, the channel arrangement pattern is determined, thereby determining the pixel positions in the same channel. Then, pixels in the same channel are filtered and pixel merging is performed.

[0089] like Figure 2 The diagram illustrates a pixel merging process for Bayer format data. Specifically, a horizontal 2×vertical 2×pixel merging (Binning) downsampling is performed on the Bayer format data, meaning that pixel merging is performed twice along both the horizontal and vertical directions.

[0090] Continue to refer to the appendix Figure 2 The calculation process for filtering pixels in the same channel and merging pixels includes:

[0091] ...(3), Binning[i] represents the pixel position after the current downsampling, and Input[i] represents the pixel position before the current downsampling.

[0092] For example, ; , as well as Calculation and Similarly. ; ; .

[0093] As an example, the steps for performing the core downsampling include:

[0094] Perform support domain calculation to obtain support domain data; perform anti-aliasing kernel calculation to obtain anti-aliasing kernel function coefficients; perform channel isolation calculation to obtain channel isolation coefficients; and use the support domain data, anti-aliasing kernel function coefficients, and channel isolation coefficients for convolution to obtain downsampling results.

[0095] The core downsampling method in this embodiment is to perform downsampling operations based on the Bayer format. In this embodiment, each output pixel is reverse-mapped to the coordinates of the input data, a weighted sampling window is constructed with these coordinates as the center, and then the data is summed according to channel constraints within the window.

[0096] Specifically, assuming the input image is in Bayer format under core downsampling, let InW be the width and InH be the height of the input image, and the output image is also in Bayer format, let OutW be the width and OutH be the height. For the output pixel at position (i, j), the reference position index of its interpolation window is denoted as (vpo, hpo), and the phase index is denoted as (vph, hph).

[0097] In one embodiment, the step of performing support domain calculation to obtain support domain data includes:

[0098] The position index and phase index are obtained by performing index calculations in the vertical and horizontal directions respectively through reverse mapping; the support domain data in the position index is selected along the preset window.

[0099] To further explain, during the support domain calculation process, the reference position and phase of the downsampled support domain are calculated to obtain the support domain data. The reference position refers to the origin of the support domain, which is the calculation point obtained through reverse mapping. The reference position of the support domain is the origin of the support domain obtained by reverse mapping the downsampled output position to the Bayer format data coordinate system. The reference position itself does not necessarily correspond to the actual pixel point in the Bayer format data. The support domain is delineated based on this reference position, and the phase index and the Bayer channel type corresponding to the reference position are obtained.

[0100] For example, the support domain data is obtained in two steps.

[0101] In the first step, the phase indices are calculated separately in the vertical and horizontal directions using reverse mapping. That is, the phase indices in the horizontal and vertical directions are calculated independently.

[0102] Taking the horizontal direction as an example, the horizontal phase index hph∈[0, 63]. hph represents the sub-pixel offset of the output pixel relative to the reference integer pixel hpo, and 0~63 represents the 64 equal parts of one pixel period.

[0103] ...(4),

[0104] j represents the x-coordinate index of the output position; hds_step represents the horizontal downsampling step size, hds_step= rH = WIn / WOut, stored in U0.20 fixed-point format (i.e., 20 decimal places, no integer places), so 1 is represented as 1 << 20. (hds_step >> 1): half a step size, used to align the sampling point to the pixel center (rather than the edge) to avoid offset. -(1 << 19): i.e. -0.5 (in U0.20). Haddr(j) ​​represents the corresponding position of the output pixel j on the input image (in fixed-point number with 20 decimal places), and is centered and fixed offset.

[0105] ...(5), Formula (5) means shifting 20 bits to the right, removing the decimal part, to obtain the integer pixel index on the input image. Formula (5) represents the reference position of the interpolation window. That is, hpo means extracting integer coordinates.

[0106] ...(6), This indicates a right shift operation of n bits. `(hpo << 20)` means converting the integer coordinates back to fixed-point numbers to obtain the fixed-point representation of the integer. `Hadr -(hpo << 20)` means obtaining the decimal part (range 0 ~ 2^n). 20 -1). Right shift by 14 bits: Compresses 20 decimal bits of precision to 6 phase bits of precision (because 20-14 = 6, so hph range is 0~63).

[0107] The vertical integer position index vpo and the phase index vph can be obtained similarly. For example,

[0108] Horizontal phase index vph∈[0, 63], ; ; .

[0109] In the second step, as Figure 3 As shown, assuming the window size of the support domain is 4 pixels * 6 pixels, the data to be filtered can be obtained by moving up 1, down 2, left 2, and right 3 according to the reference position A. That is, supporting domain data .

[0110] In one embodiment, the step of performing anti-aliasing kernel calculation to obtain anti-aliasing kernel function coefficients includes: using the phase index, looking up the table from the reference position phase index, and calculating the filtering coefficients in the horizontal and vertical directions respectively using the anti-aliasing kernel function; and obtaining the anti-aliasing kernel function coefficients based on the filtering coefficients in the horizontal and vertical directions.

[0111] During the system design phase, based on the characteristic of denser color alternation in the horizontal direction of Bayer format data, a 6-tap Lanczos3 kernel was selected as the horizontal anti-aliasing kernel function, and a 4-tap Lanczos2 kernel function was selected in the vertical direction. For each phase index from 0 to 63, the above kernel functions were sampled to generate a horizontal coefficient table LUTtap6 and a vertical coefficient table LUTtap4. In real-time processing, the corresponding filter coefficients can be directly obtained by looking up the tables using the phase index, and then used for subsequent interpolation calculations.

[0112] For example, based on the horizontal phase index hph, six coefficients are read from the 6-tap horizontal coefficient table LUTtap6 to form the horizontal filter coefficients. .

[0113] ...(7)

[0114] Next, based on the phase index vph in the vertical direction, four coefficients are read from the 4-tap vertical coefficient table LUTtap4 to form the vertical filter coefficients. .

[0115] ...(8)

[0116] Then, the vertical filter coefficients and the transposes of the horizontal filter coefficients are multiplied by a matrix to obtain the final two-dimensional anti-aliasing kernel function coefficients. .

[0117] ...(9)

[0118] Two-dimensional anti-aliasing kernel function coefficients Used to perform convolution operations on image pixels.

[0119] In one embodiment, the step of performing channel isolation calculation to obtain the channel isolation coefficient includes: determining the Bayer channel type of the current downsampling output position and the reference position in the support domain; and calculating the channel isolation coefficient of the corresponding channel coupling factor based on the Bayer channel type.

[0120] In this Bayer format data, each pixel location has a Bayer channel type (e.g., Bayer channel attributes such as R, Gb, Gr, B), and each pixel location belongs to a different color channel. The downsampled output location corresponds to a set of support domains (convolution windows), and the reference location of the support domain also corresponds to a color channel. Based on this, during the channel isolation calculation in this embodiment, the Bayer channel type of the current downsampled output location and the Bayer channel type of the reference location of the support domain are determined in order to select appropriate filtering parameters and avoid cross-channel erroneous calculations.

[0121] Next, the channel isolation coefficient of each channel coupling factor is calculated based on this. For example... Figure 4 As shown, assume the baseline channel of the support domain is Gr. If the current channel... Is it Gb or Gr?

[0122] So, the channel isolation coefficient ...(10);

[0123] If the current channel It's R.

[0124] So, the channel isolation coefficient (11);

[0125] If the current channel It's B.

[0126] So, the channel isolation coefficient (12);

[0127] If the reference channel of the supporting domain is of another type, the method is similar.

[0128] If the current channel When it is R or B, in order to adequately guarantee the resolution of RB, the coupling coefficient of the G channel can be adjusted according to the configuration.

[0129] Specifically, if the current channel If it is R, then

[0130] ...(13); If the current channel It's B, then

[0131] ...(14).

[0132] As an example, .

[0133] In one embodiment, the process of obtaining the downsampling result by convolving the support domain data, anti-aliasing kernel coefficients, and channel isolation coefficients includes:

[0134] The anti-aliasing kernel function coefficients are then modified to obtain the modified anti-aliasing kernel function coefficients. , ...(15);

[0135] Next, the support domain data and the corrected anti-aliasing kernel function coefficients are compared. Perform convolution to obtain the convolution result. ...(16);

[0136] Finally, normalize the convolution result to obtain the downsampling result. ...(17). Among them, This means a divided by b. This represents the convolution operation. This represents the dot product operation.

[0137] Step S6: The downsampling result is encapsulated into a Raw format file and then output. That is, the output of step S4... .

[0138] For example, the downsampling result is a Bayer-formatted matrix Output(i, j), where i is the row index and j is the column index, and the matrix contains OutH rows and OutW columns of pixel sample values. The pixel sample values ​​in Output(i, j) are read sequentially from i=0 to i=OutH-1 and j=0 to j=OutW-1, scanned row by row. Each pixel sample value is written continuously to the data area of ​​the Raw file according to a predetermined byte order. Metadata including the image width OutW, image height OutH, and the arrangement order of the Bayer color filter array is recorded before the data area of ​​the Raw file or in an external file. The metadata and pixel data areas are merged, and the output is a Raw format file, thus completing the Raw format encapsulation of the downsampling result.

[0139] In the aforementioned Bayer format-based image downsampling method, a raw image and its downsampling size are received. Then, the raw domain data of the raw image, including Bayer format data and the input size, is read. Next, a target downsampling mode is selected from multiple downsampling modes based on the image scaling ratio of the input size and the downsampling size. The target downsampling mode is then applied to the Bayer format data to obtain the downsampling result. Finally, the downsampling result is encapsulated into a raw format file and output. This design achieves downsampling at any ratio in the Bayer format while ensuring image clarity and detail reproduction without introducing moiré patterns or false colors. Selecting the target downsampling mode based on the image scaling ratio avoids kernel function expansion issues in high-magnification single-level downsampling, reduces the caching cost required for the kernel function support domain, eliminates the need for extensive redundant calculations, and can be inserted and used at any position in the ISP Bayer domain, making it widely applicable.

[0140] The technical solution provided in this application specifically optimizes downsampling at high magnification (e.g., r < 0.5, especially typical high-magnification scenarios with r ≤ 0.35, such as 50MP → 12MP, 50MP → 3MP, 100MP → 4K, etc.). In such scenarios, compared to the traditional two-stage process where approximately 89% to 99% of the intermediate results of demosaic interpolation are discarded during image resizing, resulting in significant computational waste, this application completes downsampling in one step within the Raw domain data, eliminating this redundancy and reducing computation by approximately 60% to 80%. During Raw domain downsampling, the integrity of the Bayer CFA four-color checkerboard structure is maintained, ensuring that the output image remains in valid Bayer Raw format without requiring a reconstruction step. An independent channel-constrained weighted sampling mechanism is designed for each color channel in the Bayer format data, achieving accurate downsampling of arbitrary scaling factors (including non-integer scaling factors) under the premise of channel isolation. While processing each channel independently, a configurable anti-aliasing filter bank effectively suppresses high-frequency aliasing within the channel, reducing color moiré patterns.

[0141] like Figure 5 As shown in the figure, this application embodiment also provides an image downsampling device based on the Bayer format, including: a data receiving module 100, a data reading module 200, a mode selection module 300, a downsampling execution module 400, and a data output module 500, wherein,

[0142] The data receiving module 100 is used to receive raw images and their downsampling dimensions;

[0143] The data reading module 200 is used to read the Raw domain data of the Raw image, the Raw domain data including Bayer format data and input size;

[0144] The mode selection module 300 is used to select a target downsampling mode from multiple downsampling modes based on the image scaling ratio of the input size and the downsampling size;

[0145] The downsampling execution module 400 is used to perform the target downsampling mode on the Bayer format data to obtain the downsampling result;

[0146] The data output module 500 is used to encapsulate the downsampling results into a Raw format file and then output it.

[0147] To further explain, the downsampling execution module 400 includes a fast downsampling unit 410 and a core downsampling unit 420, and calls at least one of the fast downsampling unit 410 and the core downsampling unit 420 according to the target downsampling mode selected by the general selection.

[0148] In one embodiment, when the fast downsampling mode is selected, the fast downsampling unit 410 is invoked; when the core downsampling mode is selected, the core downsampling unit 420 is invoked; when the hierarchical downsampling mode is selected, the fast downsampling unit 410 and the core downsampling unit 420 are invoked sequentially.

[0149] Furthermore, such as Figure 6 As shown, the core downsampling unit 420 includes a support domain calculator 421, an anti-aliasing kernel calculator 422, a channel isolator 423, and a convolutional unit 424. The support domain calculator 421 is used to perform support domain calculation; the anti-aliasing kernel calculator 422 is used to perform anti-aliasing kernel calculation; the channel isolator 423 is used to perform channel isolation calculation; and the convolutional unit 424 is used to perform convolution calculation.

[0150] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention provides an electronic device, which includes: a processor and a memory storing a computer program; when the processor runs the computer program, the method applied to the above-described device is implemented.

[0151] The memory can be volatile or non-volatile, or a combination of both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0152] The memory in this embodiment of the invention is used to store various types of data to support the operation of the device. Examples of this data include: any computer programs used to operate on the device, such as operating systems and applications; contact data; phonebook data; messages; pictures; videos, etc. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications, such as media players, browsers, etc., used to implement various application services. Here, the program implementing the method of this embodiment of the invention can be included in the application.

[0153] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a storage medium storing a computer program. The computer-readable storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc. When the computer program stored in the computer-readable storage medium is run by a processor, it implements the above method. For the specific steps implemented when the computer program is executed by the processor, please refer to [link to relevant documentation]. Figure 1 The description of the illustrated embodiments will not be repeated here.

[0154] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0156] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An image downsampling method based on the Bayer format, characterized in that, The method includes: Receive the raw image and its downsampled size; Read the Raw domain data of the Raw image, the Raw domain data including Bayer format data and input dimensions; Based on the image scaling ratio of the input size and the downsampling size, a target downsampling mode is selected from the multi-level downsampling modes; The target downsampling mode is applied to the Bayer format data in the Raw domain to obtain the downsampling result; The downsampling results are then encapsulated into a Raw format file and output.

2. The image downsampling method based on Bayer format according to claim 1, characterized in that, The target downsampling mode includes a fast downsampling method and a core downsampling method. The target downsampling mode also includes a hierarchical downsampling method, which is a cascaded combination of the fast downsampling method and the core downsampling method. The fast downsampling is used to scale down the image by an integer multiple; the core downsampling is used to scale down the image by an integer multiple or a non-integer multiple.

3. The image downsampling method based on Bayer format according to claim 2, characterized in that, The fast downsampling method uses at least one selectable integer multiple.

4. The image downsampling method based on the Bayer format according to claim 2, characterized in that, The step of selecting a target downsampling mode from multiple downsampling modes based on the image scaling ratio of the input size and the downsampling size includes: When the image scaling ratio is 1 / n, where n is at least one optional integer multiple, a fast downsampling method is selected, and at least one optional integer multiple scaling downsampling is performed.

5. The image downsampling method based on the Bayer format according to claim 4, characterized in that, The steps for selecting a target downsampling mode from a multi-level downsampling mode also include: When the image scaling ratio is less than a preset threshold, a hierarchical downsampling method is selected, and a multi-level cascade of fast downsampling and core downsampling is performed, or a multi-level cascade of fast downsampling and fast downsampling is performed, or a multi-level cascade of core downsampling and core downsampling is performed. When the image scaling ratio is greater than a preset threshold, core downsampling is selected and core downsampling is performed.

6. The image downsampling method based on the Bayer format according to any one of claims 2 to 5, characterized in that, The steps for performing the fast downsampling include: Based on the channel arrangement pattern of Bayer format data, pixels in the same channel are filtered and merged.

7. The image downsampling method based on the Bayer format according to any one of claims 2 to 5, characterized in that, The steps for performing the core downsampling include: Perform support domain calculations to obtain support domain data; Perform anti-aliasing kernel calculations to obtain the anti-aliasing kernel function coefficients; Perform channel isolation calculation to obtain the channel isolation coefficient; The downsampling result is obtained by convolving the support domain data, anti-aliasing kernel coefficients, and channel isolation coefficients.

8. The image downsampling method based on Bayer format according to claim 7, characterized in that, The steps to perform support domain calculations and obtain support domain data include: The position index and phase index are obtained by performing index calculations in the vertical and horizontal directions respectively through reverse mapping. Select the supporting field data in the location index along the preset window.

9. The image downsampling method based on the Bayer format according to claim 8, characterized in that, The steps for performing anti-aliasing kernel calculations and obtaining the anti-aliasing kernel function coefficients include: Using the phase index, the filter coefficients in the horizontal and vertical directions are calculated by looking up the table from the reference position phase index and using the anti-aliasing kernel function. The anti-aliasing kernel function coefficients are obtained based on the filtering coefficients in the horizontal and vertical directions.

10. The image downsampling method based on the Bayer format according to claim 9, characterized in that, The steps to perform channel isolation calculation and obtain the channel isolation coefficient include: Determine the Bayer channel type for the current downsampled output position and the reference position in the support domain; The channel isolation coefficient corresponding to the channel coupling factor is calculated based on the Bayer channel type.

11. An image downsampling device based on the Bayer format, characterized in that, include: The data receiving module is used to receive raw images and their downsampling dimensions; The data reading module is used to read the Raw domain data of the Raw image, the Raw domain data including Bayer format data and input dimensions; The mode selection module is used to select a target downsampling mode from multiple downsampling modes based on the image scaling ratio of the input size and the downsampling size; The downsampling execution module is used to execute the target downsampling mode on the Bayer format data in the Raw domain to obtain the downsampling result; The data output module is used to encapsulate the downsampling results into a Raw format file and then output it.

12. The image downsampling device based on the Bayer format according to claim 11, characterized in that, The downsampling execution module includes a fast downsampling unit and a core downsampling unit, and calls at least one of the fast downsampling unit and the core downsampling unit according to the selected target downsampling mode.

13. The image downsampling device based on the Bayer format according to claim 11, characterized in that, The core downsampling unit includes a support domain calculator, an anti-aliasing kernel calculator, a channel isolator, and a convolutional unit. The support domain calculator is used to perform support domain calculations; the anti-aliasing kernel calculator is used to perform anti-aliasing kernel calculations; the channel isolator is used to perform channel isolation calculations; and the convolutional unit is used to perform convolution calculations.

Images