A raw domain epipolar correction and demosaicing combined processing system and method

Abstract

The application relates to the technical field of image processing, and discloses a RAW domain epipolar correction and demosaicing combined processing system and method, the method comprising the following steps: collecting an original RAW format image pair acquired by a binocular camera, and generating left-eye RAW format image data and right-eye RAW format image data; constructing a RAW domain ring buffer with adaptive depth; generating left-eye RAW buffer data and right-eye RAW buffer data; in a single processing pipeline, extracting local RAW feature window data in parallel based on original image integer coordinate data; and performing color restoration in combination with demosaicing interpolation weight data generated by edge perception decision-making; the joint calculation of edge perception demosaicing processing and geometric resampling can be directly completed in the RAW domain, and the color artifacts and neighborhood information mismatch problems caused by twice independent resampling of RAW data and RGB images in the traditional process are reduced.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a RAW domain epipolar correction and demosaicing joint processing system and method. Background Technology

[0002] RAW contains data processed from the image sensors of digital cameras, scanners, and film scanners. Demosaic is a digital image processing algorithm that aims to reconstruct a full-color image from incomplete color samples output by a photosensitive element covered with a color filter array.

[0003] Currently, in image preprocessing for binocular stereo vision systems, traditional serial processing systems are commonly used, namely cascaded schemes of "de-mosaicing followed by epipolar correction" and "epidural correction followed by de-mosaicing." These two schemes have the following inherent drawbacks:

[0004] Cumulative error and information loss caused by two independent resampling operations: Regardless of the cascading order, image data needs to be resampled twice: once for color interpolation using the demosaicing algorithm and once for pixel position interpolation for geometric correction. On hardware platforms such as FPGAs, each resampling operation introduces an approximation error determined by the interpolation algorithm. The two cascaded resampling operations are serial rather than merged, leading to error accumulation and compromising the integrity of the original sensor data. In particular, when geometric correction is performed on RAW data first, the special arrangement of the Bayer array makes the interpolation process extremely complex and severely introduces color artifacts. If demosaicing is performed first to generate a complete RGB image and then geometric correction is performed, it will result in a second resampling of the already interpolated RGB image containing errors, exacerbating the blurring of high-frequency textures and loss of detail.

[0005] High bandwidth and high storage resource consumption: In the process of "de-mosaicing followed by correction", the complete 24-bit RGB image frame obtained after de-mosaicing needs to be stored in external DDR and large-scale on-chip cache for subsequent geometric correction module to read. This increases the pressure on off-chip storage bandwidth and the consumption of on-chip storage resources, which poses a challenge to high-resolution, high-frame-rate real-time processing systems.

[0006] Therefore, a combined RAW domain epipolar correction and demosaic processing system and method are proposed to solve the above problems. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a RAW domain epipolar correction and demosaic joint processing system and method, which solves the problem mentioned in the background art of failing to effectively preserve the edge and color information of the original sensor during geometric transformation.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a RAW domain epipolar correction and demosaicing joint processing system and method, the method comprising the following steps:

[0009] S1. Acquire raw RAW format image pairs from the binocular camera and generate left-eye RAW format image data and right-eye RAW format image data;

[0010] S2. Construct an adaptive depth RAW domain circular cache to cache the left-eye RAW format image data and the right-eye RAW format image data, generating left-eye RAW cache data and right-eye RAW cache data.

[0011] S3. Based on the pre-generated binocular camera epipolar correction inverse mapping table, read the original image coordinates corresponding to the target image pixels one by one and generate the original image floating-point coordinate data.

[0012] S4. The original image floating-point coordinate data is split into integer and fractional parts to generate original image integer coordinate data and coordinate interpolation weight component data.

[0013] S5. Based on the original image integer coordinate data, extract local pixel windows in parallel from the corresponding left-eye RAW cache data and right-eye RAW cache data to generate local RAW feature window data; S6. Perform edge perception decision based on the local RAW feature window data to generate de-mosaic interpolation weight data.

[0014] S7. Combine the local RAW feature window data with the demosaic interpolation weight data to perform demosaic processing and generate RGB pixel data with integer coordinate points.

[0015] S8. Using the integer coordinate point RGB pixel data and the coordinate interpolation weight component data, perform geometric correction interpolation calculation to generate target corrected RGB image data.

[0016] Preferably, the generation of left-eye RAW format image data and right-eye RAW format image data in step S1 includes the following steps:

[0017] S11. Simultaneously acquire the raw image streams in Bayer array format output by the left and right cameras through the binocular camera image acquisition interface deployed on the heterogeneous computing platform.

[0018] S12. The original image stream output by the left eye camera is parsed into continuous left eye RAW format image data, and the original image stream output by the right eye camera is parsed into continuous right eye RAW format image data.

[0019] Preferably, the generation of left-eye RAW cache data and right-eye RAW cache data in S2 includes the following steps:

[0020] S21. Calculate the maximum positive and maximum negative offsets of the original image when performing epipolar correction remapping in the vertical direction based on the camera calibration parameters.

[0021] S22. Based on the maximum positive offset, the maximum negative offset, and the neighborhood window size required by the demosaic algorithm, calculate the minimum row cache depth required for the adaptive depth RAW domain ring cache.

[0022] S23. In the on-chip programmable logic, instantiate two RAW domain ring caches with the adaptive depth respectively;

[0023] S24. Stream the left-eye RAW format image data into one of the adaptive depth RAW domain circular buffers to generate left-eye RAW buffer data; stream the right-eye RAW format image data into another adaptive depth RAW domain circular buffer to generate right-eye RAW buffer data.

[0024] Preferably, generating the original image floating-point coordinate data in S3 includes the following steps: S31, in the processing system of the heterogeneous computing platform, the intrinsic and extrinsic parameters and distortion coefficients of the binocular camera are obtained through a camera calibration tool;

[0025] S32. Based on the intrinsic and extrinsic parameters and distortion coefficients of the binocular camera, the epipolar correction inverse mapping table of the binocular camera is calculated using a stereo correction algorithm, and the epipolar correction inverse mapping table of the binocular camera is stored in an external memory.

[0026] S33. Driven by the pixel clock of the target correction image, the binocular camera epipolar correction inverse mapping table is sequentially read from the external memory through the high-speed data transmission interface to obtain the original image floating-point coordinate data corresponding to the current target pixel in the original image.

[0027] Preferably, the step S4 of generating the original image integer coordinate data and coordinate interpolation weight component data includes the following steps:

[0028] S41. Receive the original image floating-point coordinate data;

[0029] S42. Extract the integer part of the original image floating-point coordinate data through the hardware rounding logic circuit to generate the original image integer coordinate data;

[0030] S43. Using hardware subtractors and multipliers, calculate the fractional difference between the original image floating-point coordinate data and its corresponding integer part to generate the coordinate interpolation weight component data.

[0031] Preferably, generating local RAW feature window data in S5 includes the following steps:

[0032] S51. Receive the original image integer coordinate data;

[0033] S52. Based on the pixel position indicated by the integer coordinate data of the original image, calculate the read address of all pixels in the RAW domain circular buffer of the adaptive depth within the preset neighborhood of the pixel.

[0034] S53. Within the same clock cycle, all pixel values ​​are read in parallel from the left-eye RAW cache data and the right-eye RAW cache data according to the read address to generate the local RAW feature window data.

[0035] Preferably, generating the demosaic interpolation weight data in step S6 includes the following steps:

[0036] S61. Receive the local RAW feature window data;

[0037] S62. For pixels with different color components in the local RAW feature window data, calculate the gradient in the horizontal, vertical and diagonal directions respectively.

[0038] S63. Compare the gradient values ​​in each direction, and determine the main direction of the image edge through the comparator and selector logic circuit;

[0039] S64. Based on the main direction of the image edge, assign interpolation weights in different directions to generate the demosaic interpolation weight data.

[0040] Preferably, generating integer coordinate point RGB pixel data in step S7 includes the following steps:

[0041] S71. Receive the local RAW feature window data and the demosaic interpolation weight data;

[0042] S72. Identify the color channel of each pixel in the local RAW feature window data according to the Bayer array pattern;

[0043] S73. For the reference pixel and its neighboring pixels corresponding to the integer coordinate data of the original image, the missing color channel values ​​are weighted and interpolated using the demosaic interpolation weight data.

[0044] S74. For the reference pixel and each pixel at an integer coordinate position in its neighborhood, calculate the complete red, green, and blue three-channel color values ​​respectively, and generate the RGB pixel data of the integer coordinate point.

[0045] Preferably, generating the target corrected RGB image data in step S8 includes the following steps:

[0046] S81, Receive the RGB pixel data of the integer coordinate point and the coordinate interpolation weight component data;

[0047] S82. Construct a bilinear interpolation calculation array, and input the RGB values ​​of the four corner points interpolated from the integer coordinate point RGB pixel data, as well as the horizontal and vertical weight components in the coordinate interpolation weight component data, into the bilinear interpolation calculation array.

[0048] S83. In the bilinear interpolation calculation array, two-level interpolation calculation is performed through a multiplier and adder tree. First, horizontal interpolation is performed based on the horizontal weight component, and then vertical interpolation is performed on the horizontal interpolation result based on the vertical weight component.

[0049] S84. Output the red, green, and blue channel color values ​​of the current target pixel obtained by final interpolation, as the target corrected RGB image data.

[0050] Preferably, the system includes:

[0051] The RAW data acquisition and caching module acquires binocular raw RAW image streams through the image interface receiving unit, generates single-channel RAW data through the RAW data parsing unit, and streams and caches the RAW line cache data in on-chip memory through the adaptive ring buffer unit, outputting RAW line cache data.

[0052] The coordinate mapping and scheduling module receives a pre-stored camera inverse mapping table, sequentially reads floating-point coordinates under the pixel clock drive through the coordinate reading unit, and deconstructs the floating-point coordinates into integer coordinates and fractional weights using the coordinate splitting unit, outputting the original image integer coordinate data and coordinate interpolation weight component data.

[0053] The feature extraction and calculation module receives the RAW row cache data and the original image integer coordinate data, calculates the pixel neighborhood address through the feature window address generation unit, extracts the local RAW pixel window through the parallel reading unit, and uses the edge-aware weight calculation unit to analyze the image direction to generate interpolation weights, and outputs the local RAW feature window data and de-mosaic interpolation weight data.

[0054] The joint processing and output module receives the local RAW feature window data, the demosaic interpolation weight data, the RGB pixel data of the integer coordinate points, and the coordinate interpolation weight component data. It generates RGB values ​​at the integer coordinate points through the demosaic color restoration unit, performs bilinear fusion calculation through the geometric correction interpolation unit, and streams the corrected image data through the RGB data output unit.

[0055] Compared with the prior art, the present invention provides a joint processing system and method for RAW domain epipolar correction and demosaicing, which has the following beneficial effects:

[0056] 1. In this invention, during binocular vision image preprocessing, an adaptive depth RAW domain circular buffer is constructed, and a coordinate mapping mechanism driven by an epipolar correction inverse mapping table is adopted. This ensures the traceability and efficient access of the original RAW data during the geometric correction process. Simultaneously, within a single processing pipeline, local RAW feature window data is extracted in parallel based on the integer coordinate data of the original image, and color restoration is performed by combining the demosaic interpolation weight data generated by edge-aware decision. This allows for the joint calculation of edge-aware demosaic processing and geometric resampling directly in the RAW domain, reducing color artifacts and neighborhood information mismatch problems introduced by two independent resampling operations on RAW data and RGB images in traditional processes, thus ensuring the accuracy and reliability of the corrected image. This invention improves color consistency and geometric accuracy, thereby enhancing the accuracy of subsequent stereo matching and 3D reconstruction. The joint processing method proposed in this invention is not mathematically equivalent to two independent cascaded processes. Traditional cascaded processing is a series of two independent function transformations. However, this invention integrates the color interpolation for de-mosaicing and the position interpolation for geometric correction into a unified resampling process driven by the core algorithm. This process calculates the final corrected RGB pixel values ​​in one go based on the local neighborhood and geometric mapping weights of the original RAW data. This not only reduces the information isolation and error propagation between two independent interpolation operations in the algorithm, but also reduces the two resampling error sources to one in the hardware implementation, directly improving geometric accuracy and color fidelity from both mathematical and physical perspectives.

[0057] 2. In this invention, during binocular vision image preprocessing, the RGB pixel data of integer coordinate points generated in the demosaicing step are directly geometrically corrected and interpolated with the coordinate interpolation weight component data generated in the coordinate mapping step. This ensures that the entire transformation process from RAW data to the final corrected RGB image involves only one high-quality resampling driven by the core algorithm for each output pixel. This method reduces the high-frequency texture blurring and detail loss caused by secondary geometric interpolation of the complete RGB image in the traditional "de-mosaicing followed by correction" process. It allows the output corrected RGB image to retain the true edge and detail information of the original sensor, thereby improving the clarity of the corrected image and the efficiency of the stereo matching algorithm in utilizing texture features.

[0058] 3. In this invention, during binocular vision image preprocessing, image data is streamed by constructing an adaptive RAW domain row buffer only on-chip. Only a pre-calculated, minimal-data-volume binocular camera epipolar correction inverse mapping table needs to be read from external memory. The entire processing flow does not require caching complete intermediate RGB image frames to external high-speed memory. This system reduces dependence on off-chip storage bandwidth and access latency, while saving on-chip storage resources. The pipelined design of the joint processing ensures high throughput and low latency from RAW data input to corrected RGB data output, enabling the system to process high-resolution, high-frame-rate binocular video streams in real time with high energy efficiency. This is suitable for applications with low power consumption. For embedded stereo vision systems with stringent requirements for real-time performance and integration, this invention constructs an adaptive depth RAW domain circular buffer. The entire process eliminates the need to generate and cache complete intermediate RGB image frames. RAW data is single-channel per pixel, reducing the data volume by two-thirds compared to 24-bit RGB images. After demosaicing a pixel, the joint processing pipeline immediately uses the RGB result as the geometric interpolation calculation for that pixel and outputs it without writing back to the intermediate frame buffer. This solves the problem of insufficient storage space for 24-bit RGB images, reduces dependence on off-chip memory bandwidth and on-chip memory resource consumption, and makes high-resolution real-time processing possible on resource-constrained embedded FPGA platforms. Attached Figure Description

[0059] Figure 1 This is a flowchart of a method for combined RAW domain epipolar correction and demosaic processing according to the present invention;

[0060] Figure 2 This is a schematic diagram of the architecture of a RAW domain epipolar correction and demosaicing joint processing system according to the present invention. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Please see Figure 1 - Figure 2 This invention relates to a combined RAW domain epipolar correction and demosaic processing system and method, the method comprising the following steps:

[0063] S1. Acquire raw RAW format image pairs from the binocular camera and generate left-eye RAW format image data and right-eye RAW format image data;

[0064] S2. Construct an adaptive depth RAW domain circular buffer to cache the left-eye RAW format image data and the right-eye RAW format image data, generating left-eye RAW cache data and right-eye RAW cache data.

[0065] S3. Based on the pre-generated binocular camera epipolar correction inverse mapping table, read the original image coordinates corresponding to the target image pixels one by one and generate the original image floating-point coordinate data.

[0066] S4. Split the original image floating-point coordinate data into integer and fractional parts to generate original image integer coordinate data and coordinate interpolation weight component data.

[0067] S5. Based on the original image integer coordinate data, extract local pixel windows in parallel from the corresponding left and right RAW cache data to generate local RAW feature window data.

[0068] S6. Perform edge-aware decision based on local RAW feature window data and generate de-mosaic interpolation weight data;

[0069] S7. Combine local RAW feature window data with demosaic interpolation weight data to perform demosaic processing and generate RGB pixel data with integer coordinates.

[0070] S8. Perform geometric correction interpolation calculations using integer coordinate point RGB pixel data and coordinate interpolation weight component data to generate target corrected RGB image data.

[0071] The steps involved in generating left-eye RAW format image data and right-eye RAW format image data in S1 are as follows:

[0072] S11. Simultaneously acquire the raw image streams in Bayer array format output by the left and right cameras through the binocular camera image acquisition interface deployed on the heterogeneous computing platform.

[0073] S12. Parse the raw image stream output by the left eye camera into continuous left eye RAW format image data, and parse the raw image stream output by the right eye camera into continuous right eye RAW format image data.

[0074] The steps involved in generating left-eye RAW cache data and right-eye RAW cache data in S2 are as follows:

[0075] S21. Calculate the maximum positive and maximum negative offsets of the original image during epipolar correction remapping in the vertical direction based on camera calibration parameters, using the following formula:

[0076] For each pixel coordinate in the original image The rotation matrix of the right camera relative to the left camera, obtained from the binocular camera calibration. Translation vector and camera intrinsic parameter matrix Calculate its corresponding vertical coordinates in the ideally corrected image. This calculation can be achieved by constructing a remapping relationship, the general functional relationship of which is expressed as: ;

[0077] in, This represents the coordinate mapping function determined by the calibration parameters;

[0078] Calculate the remapping offset of the pixel in the vertical direction. : ;

[0079] In the formula, Indicates a positive offset. Indicates a negative offset;

[0080] Iterate through all valid pixels of the original image and find them one by one. The maximum positive and minimum negative values ​​are used as the maximum positive offset when performing epipolar correction remapping in the vertical direction. offset in the maximum negative direction :

[0081] ;

[0082] ;

[0083] S22. Based on the maximum positive offset, the maximum negative offset, and the neighborhood window size required by the demosaic algorithm, calculate the minimum line cache depth required for the adaptive depth RAW domain ring cache. The specific calculation formula is as follows: ;

[0084] in, and These represent the maximum positive offset and the maximum negative offset calculated in step S21, respectively. This represents the neighborhood window size required in the vertical direction for the demosaic algorithm;

[0085] S23. In the on-chip programmable logic, instantiate two adaptive depth RAW domain ring caches respectively;

[0086] S24. Stream the left-eye RAW format image data into one of the adaptive depth RAW domain circular buffers to generate left-eye RAW buffer data; stream the right-eye RAW format image data into another adaptive depth RAW domain circular buffer to generate right-eye RAW buffer data.

[0087] Generating raw image floating-point coordinate data in S3 includes the following steps:

[0088] S31. In the processing system of a heterogeneous computing platform, the intrinsic and extrinsic parameters and distortion coefficients of the binocular camera are obtained through camera calibration tools.

[0089] S32. Based on the intrinsic and extrinsic parameters and distortion coefficients of the stereo camera, the inverse epipolar correction mapping table of the stereo camera is calculated using a stereo correction algorithm, and then stored in external memory. The specific calculation process is as follows:

[0090] For any integer pixel coordinate in the target image The coordinates of each pixel in the original image are calculated using the following formula. This allows us to construct an inverse mapping table.

[0091] ;

[0092] ;

[0093] in, Correct the integer pixel coordinates of the image for the target. This is the intrinsic parameter matrix of the virtual camera after stereo calibration. These are the normalized planar coordinates in the calibrated virtual camera coordinate system. For rotation matrix, The coordinates of a 3D point in the original camera coordinate system. and For the original camera's normalized planar coordinates, For distortion correction function, Let be the distortion coefficient vector of the camera. This is the intrinsic parameter matrix of the original camera. For each pixel coordinate in the original image;

[0094] S33. Driven by the pixel clock of the target correction image, the binocular camera epipolar correction inverse mapping table is sequentially read from the external memory through the high-speed data transmission interface to obtain the original image floating-point coordinate data corresponding to the current target pixel in the original image.

[0095] The steps involved in generating the original image integer coordinate data and coordinate interpolation weighted component data in S4 are as follows:

[0096] S41. Receive the raw image floating-point coordinate data;

[0097] S42. Extract the integer part of the original image floating-point coordinate data through the hardware rounding logic circuit to generate the original image integer coordinate data;

[0098] S43. Using hardware subtractors and multipliers, calculate the fractional difference between the original image's floating-point coordinate data and its corresponding integer part to generate coordinate interpolation weight component data. The specific calculation formula is as follows:

[0099] ;

[0100] ;

[0101] in, The original image's floating-point coordinate data. Represents floating point, The original image's integer coordinate data. Represents integers, and These are the original coordinate differences in the horizontal and vertical directions, respectively. and That is, the difference , The coordinate interpolation weighted component data obtained by direct assignment Indicates the horizontal direction. Indicates the vertical direction.

[0102] Generating local RAW feature window data in S5 includes the following steps:

[0103] S51, Receive the original image integer coordinate data;

[0104] S52. Using the pixel position indicated by the integer coordinate data of the original image as a reference, calculate the read address of all pixels in the RAW domain circular buffer with adaptive depth within the preset neighborhood. The specific calculation formula and process are as follows:

[0105] Define the neighborhood window: Let the preset neighborhood be a rectangular window with a radius of . using the original image integer coordinate data The coordinates of any pixel in its neighborhood, centered at . Must meet: ;

[0106] This formula defines the... Centered on, with side length as The rectangular neighborhood;

[0107] Calculate cache row index and column address: Assume the total depth of the adaptive depth RAW field ring cache is... Rows, each row is 1 pixel wide ;

[0108] Row index calculation: Due to the characteristics of the circular cache, for the y-axis... Its row index in the cache By measuring cache depth The modulo operation yields: ;

[0109] This operation enables automatic wrapping of line addresses, ensuring that within a limited... Access data in any row within the row depth;

[0110] Column address calculation: The address of a pixel within a cache row is determined by its x-coordinate. Decide;

[0111] Final read address: Combining row index and column address, pixels Read address in adaptive depth RAW domain ring cache It can be calculated as: ;

[0112] S53. Within the same clock cycle, read all pixel values ​​in parallel from the left and right RAW buffer data according to the read address to generate local RAW feature window data.

[0113] Generating de-mosaic interpolation weight data in S6 includes the following steps:

[0114] S61. Receive local RAW feature window data;

[0115] S62. For pixels of different color components in the local RAW feature window data, calculate the gradient in the horizontal, vertical, and diagonal directions respectively. The specific calculation formula and process are as follows:

[0116] In local RAW feature window data, around the center pixel For the pixels in the neighborhood of a given color channel, the gradient is calculated using the following formula:

[0117] Horizontal gradient: ;

[0118] in, Represents the gradient in the horizontal direction. and For indexing;

[0119] This formula calculates the absolute difference between two pixels of the same color on the same scan line as the center pixel, which are one pixel apart.

[0120] Vertical gradient: ;

[0121] in, Represents the gradient in the vertical direction. and For indexing;

[0122] This formula calculates the absolute difference between two pixels above and below the center pixel that are the same color and are one pixel apart.

[0123] Diagonal gradient:

[0124] The gradients are typically calculated along the two main diagonal directions as follows:

[0125] ;

[0126] ;

[0127] in, and This represents the gradient in the two main diagonal directions. and For indexing;

[0128] These formulas calculate the top left to bottom right respectively. and top right - bottom left The absolute difference between two pixel pairs on the diagonal that have the same color as the center pixel and are separated by two pixels.

[0129] S63. Compare the gradient values ​​in each direction, and determine the main direction of the image edge through the comparator and selector logic circuit;

[0130] S64. Based on the main direction of the image edges, assign interpolation weights in different directions to generate de-mosaic interpolation weight data.

[0131] Generating RGB pixel data with integer coordinates in S7 includes the following steps:

[0132] S71, Receive local RAW feature window data and demosaic interpolation weight data;

[0133] S72. Identify the color channel of each pixel in the local RAW feature window data according to the Bayer array pattern;

[0134] S73. For the reference pixel and its neighboring pixels corresponding to the integer coordinate data of the original image, the missing color channel values ​​are weighted and interpolated using the demosaic interpolation weight data. The specific calculation formula is as follows:

[0135] For the base pixel The missing color channel value Through its neighborhood The weighted interpolation is obtained by taking the values ​​of pixels with known values ​​for that channel: ;

[0136] In the formula, For pixels The known color channel values, To assign to pixels De-mosaic interpolation weights, For the number of pixels, For indexing;

[0137] For any pixel in the neighborhood ;

[0138] In the formula, For pixels Relative to the reference pixel Spatial distance, For the weight function, Representation and pixel Related edge-aware decision factors For indexing;

[0139] S74. For the reference pixel and each pixel at integer coordinates in its neighborhood, calculate the complete red, green, and blue channel color values ​​respectively, generating RGB pixel data at integer coordinates. The specific calculation formula and process are as follows:

[0140] For each integer coordinate position of the reference pixel and its neighboring window Its set is denoted as By using a weighted interpolation method, the two missing color channel values ​​are calculated independently. Combined with the one color channel value known in the RAW data of the pixel, the complete RGB three-channel color value at that location is constituted. The mathematical expression is as follows:

[0141] for Its RGB pixel data It is a three-dimensional vector: ;

[0142] The calculation of each channel value follows these rules:

[0143] Assign values ​​directly to known channel values: when the position The corresponding color component in the original Bayer array image is Then the channel value The original brightness value at this location is directly taken from the local RAW feature function data. : ;

[0144] Weighted interpolation of missing channel values: for the other two missing color channels and Its value is calculated using the formula in S73: ;

[0145] Here, Indicates missing interpolation channels The set of pixels in the neighborhood that have a known value for that channel. and Corresponding to the set of Known channels of pixels The brightness value and its corresponding demosaic interpolation weights, For indexing.

[0146] Generating target-corrected RGB image data in S8 includes the following steps:

[0147] S81, Receive integer coordinate point RGB pixel data and coordinate interpolation weight component data;

[0148] S82. Construct a bilinear interpolation calculation array, and input the RGB values ​​of the four corner points of the interpolation in the integer coordinate point RGB pixel data, as well as the horizontal and vertical weight components in the coordinate interpolation weight component data, into the bilinear interpolation calculation array.

[0149] For a specific target pixel in an RGB image Assume it is located at a corner point with four integer coordinates. , Within the rectangular region formed, the horizontal weight components of the target point relative to its corner points are: Vertical weight components are ,satisfy , ,in Then the target point The component interpolation formula is: ;

[0150] Similarly, Components and The interpolation formula for the components and The components are in completely identical form, only the subscripts are different. Replace with and ;

[0151] ;

[0152] in, , , , and , The subscripts correspond to the channel pixel values ​​of the four corner points;

[0153] S83. In the bilinear interpolation calculation array, two-level interpolation calculation is performed through a multiplier and adder tree. First, horizontal interpolation is performed based on the horizontal weight component, and then vertical interpolation is performed on the horizontal interpolation result based on the vertical weight component.

[0154] Let the coordinates of the target interpolation point be... The four neighboring pixels of its integer coordinate grid are: top left corner Top right corner bottom left corner bottom right corner ;

[0155] S831, Horizontal interpolation: ;

[0156] Character meaning: After interpolation in the horizontal direction, The middle pixel value of the row, These are the weighted components in the horizontal direction. These are complementary weighted components in the horizontal direction. The original pixel value in the horizontal direction;

[0157] S832, Vertical Interpolation: ;

[0158] Character meaning: target coordinates Interpolated pixel value at the location, These are the weighted components in the vertical direction. These are complementary weight components in the vertical direction. The intermediate value for vertical interpolation;

[0159] S84. Output the red, green, and blue channel color values ​​of the current target pixel obtained by final interpolation, as the target RGB image data for correction.

[0160] The system includes:

[0161] The RAW data acquisition and caching module acquires binocular raw RAW image streams through the image interface receiving unit, generates single-channel RAW data through the RAW data parsing unit, and streams and caches the RAW line cache data in on-chip memory through the adaptive ring buffer unit, outputting RAW line cache data.

[0162] The coordinate mapping and scheduling module receives a pre-stored camera inverse mapping table, sequentially reads floating-point coordinates through the coordinate reading unit driven by the pixel clock, and deconstructs the floating-point coordinates into integer coordinates and fractional weights using the coordinate splitting unit, outputting the original image integer coordinate data and coordinate interpolation weight component data.

[0163] The feature extraction and calculation module receives RAW line buffer data and original image integer coordinate data, calculates pixel neighborhood addresses through the feature window address generation unit, extracts local RAW pixel windows through the parallel reading unit, and uses the edge-aware weight calculation unit to analyze the image direction to generate interpolation weights, and outputs local RAW feature window data and de-mosaic interpolation weight data.

[0164] The joint processing and output module receives local RAW feature window data, demosaic interpolation weight data, RGB pixel data at integer coordinate points, and coordinate interpolation weight component data. It generates RGB values ​​at integer coordinate points through the demosaic color restoration unit, performs bilinear fusion calculation through the geometric correction interpolation unit, and streams the corrected image data through the RGB data output unit.

[0165] The operation steps of a RAW domain epipolar correction and demosaic joint processing system and method are as follows:

[0166] Step 1: Acquiring and caching raw data:

[0167] First, the system synchronously acquires the raw RAW image streams in Bayer array format from the left and right cameras via the image acquisition interface of the binocular cameras. Then, the system calculates the maximum vertical position offset of the image during epipolar correction based on the camera calibration parameters, and dynamically determines a minimum row buffer depth by combining the neighborhood window size required for the demosaicing algorithm. Based on this depth, two adaptive RAW domain circular buffers are instantiated in the on-chip programmable logic for the left and right eye images respectively, and the raw RAW image data is streamed into them for caching. This step constrains the massive amount of image data into an efficient on-chip buffer, laying the foundation for subsequent real-time pipelined processing.

[0168] Step 2: Coordinate Mapping and Weight Generation

[0169] Based on the "epochal correction inverse mapping table" pre-calculated and stored using a stereo correction algorithm, the system finds and maps back to the corresponding sub-pixel level accurate position in the original image for each target pixel in the final corrected image, i.e., the original image floating-point coordinate data. Then, the hardware circuit decomposes the floating-point coordinates into an integer part and a fractional part: the integer part becomes the "original image integer coordinate data" for locating the original image pixel, while the fractional part is directly converted into the "coordinate interpolation weight component data" for subsequent geometric interpolation, which represents the offset ratio of the target pixel relative to the original integer coordinate point.

[0170] Step 3: Local Feature Extraction and Edge Analysis

[0171] Based on the integer coordinates obtained in the previous step, the system reads all pixel values ​​within a preset neighborhood window around the corresponding RAW circular buffer in parallel, forming "local RAW feature window data". Subsequently, the system immediately analyzes this RAW feature window data, calculates the gradients in the horizontal, vertical, and diagonal directions, determines the main direction of the edge of the local region of the image, and dynamically allocates interpolation weights in different directions according to the determination result of this edge direction, generating "de-mosaic interpolation weight data", thereby ensuring that the subsequent color restoration can follow the image texture and reduce artifacts caused by interpolation across the edge.

[0172] Step 4: Combined Demosaic and Geometric Correction Interpolation:

[0173] This is the key step in achieving "joint processing" in this method. The system utilizes the extracted local RAW feature window data and the calculated demosaic interpolation weights to first perform a high-quality edge-aware demosaic algorithm at integer coordinate points in the original image, recovering complete RGB color values ​​for these coordinate points and generating "integer coordinate point RGB pixel data". Immediately afterwards, instead of writing these RGB data back to the frame buffer, they are immediately fed together with the "coordinate interpolation weight component data" generated in step two into a bilinear interpolation calculation array. In this array, through a single multiply-accumulate calculation, weight-based geometric resampling is simultaneously completed, directly outputting the final corrected RGB value of the target pixel. This process merges the traditional two independent resampling operations into a single algorithm-driven joint calculation.

[0174] Step 5: Pipeline Output and System Collaboration

[0175] All the above steps are executed continuously in a high-efficiency hardware pipeline. The entire system consists of four modules working together: the "RAW data acquisition and caching module" is responsible for data access and caching; the "coordinate mapping and scheduling module" is responsible for pixel-level coordinate mapping and scheduling; the "feature extraction and calculation module" is responsible for local feature analysis and weight calculation; and the "joint processing and output module" is responsible for the joint calculation of demosaicing and geometric correction, and streams the corrected final image. Through this system, high throughput and low latency processing is achieved from the input of raw RAW data to the output of the corrected RGB image.

[0176] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0177] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for combined RAW domain epipolar correction and demosaic processing, characterized in that, The method includes the following steps: S1. Acquire raw RAW format image pairs from the binocular camera and generate left-eye RAW format image data and right-eye RAW format image data; S2. Construct an adaptive depth RAW domain circular cache to cache the left-eye RAW format image data and the right-eye RAW format image data, generating left-eye RAW cache data and right-eye RAW cache data. S3. Based on the pre-generated binocular camera epipolar correction inverse mapping table, read the original image coordinates corresponding to the target image pixels one by one and generate the original image floating-point coordinate data. S4. The original image floating-point coordinate data is split into integer and fractional parts to generate original image integer coordinate data and coordinate interpolation weight component data. S5. Based on the original image integer coordinate data, extract local pixel windows in parallel from the corresponding left-eye RAW cache data and right-eye RAW cache data to generate local RAW feature window data. S6. Perform edge-aware decision based on the local RAW feature window data and generate de-mosaic interpolation weight data; S7. Combine the local RAW feature window data with the demosaic interpolation weight data to perform demosaic processing and generate RGB pixel data with integer coordinate points. S8. Perform geometric correction interpolation calculations using the integer coordinate point RGB pixel data and the coordinate interpolation weight component data to generate target corrected RGB image data; The generation of left-eye RAW cache data and right-eye RAW cache data in S2 includes the following steps: S21. Calculate the maximum positive and maximum negative offsets of the original image when performing epipolar correction remapping in the vertical direction based on the camera calibration parameters. S22. Based on the maximum positive offset, the maximum negative offset, and the neighborhood window size required by the demosaic algorithm, calculate the minimum row cache depth required for the adaptive depth RAW domain ring cache. S23. In the on-chip programmable logic, instantiate two RAW domain ring caches with the adaptive depth respectively; S24. Stream the left-eye RAW format image data into one of the adaptive depth RAW domain circular buffers to generate left-eye RAW buffer data; stream the right-eye RAW format image data into another adaptive depth RAW domain circular buffer to generate right-eye RAW buffer data. The process of generating de-mosaic interpolation weight data in S6 includes the following steps: S61. Receive the local RAW feature window data; S62. For pixels with different color components in the local RAW feature window data, calculate the gradient in the horizontal, vertical and diagonal directions respectively. S63. Compare the gradient values ​​in each direction, and determine the main direction of the image edge through the comparator and selector logic circuit; S64. Based on the main direction of the image edge, assign interpolation weights in different directions to generate the demosaic interpolation weight data.

2. The method for combined RAW domain epipolar correction and demosaicing according to claim 1, characterized in that, The step of generating left-eye RAW format image data and right-eye RAW format image data in S1 includes the following steps: S11. Simultaneously acquire the raw image streams in Bayer array format output by the left and right cameras through the binocular camera image acquisition interface deployed on the heterogeneous computing platform. S12. The original image stream output by the left eye camera is parsed into continuous left eye RAW format image data, and the original image stream output by the right eye camera is parsed into continuous right eye RAW format image data.

3. The method for combined RAW domain epipolar correction and demosaicing according to claim 1, characterized in that, The process of generating the original image floating-point coordinate data in S3 includes the following steps: S31. In the processing system of the heterogeneous computing platform, the intrinsic and extrinsic parameters and distortion coefficients of the binocular camera are obtained through a camera calibration tool; S32. Based on the intrinsic and extrinsic parameters and distortion coefficients of the binocular camera, the epipolar correction inverse mapping table of the binocular camera is calculated using a stereo correction algorithm, and the epipolar correction inverse mapping table of the binocular camera is stored in an external memory. S33. Driven by the pixel clock of the target correction image, the binocular camera epipolar correction inverse mapping table is sequentially read from the external memory through the high-speed data transmission interface to obtain the original image floating-point coordinate data corresponding to the current target pixel in the original image.

4. The method for combined RAW domain epipolar correction and demosaicing according to claim 3, characterized in that, The step of generating the original image integer coordinate data and coordinate interpolation weight component data in S4 includes the following steps: S41. Receive the original image floating-point coordinate data; S42. Extract the integer part of the original image floating-point coordinate data through the hardware rounding logic circuit to generate the original image integer coordinate data; S43. Using hardware subtractors and multipliers, calculate the fractional difference between the original image floating-point coordinate data and its corresponding integer part to generate the coordinate interpolation weight component data.

5. The method for combined RAW domain epipolar correction and demosaicing according to claim 4, characterized in that, The process of generating local RAW feature window data in S5 includes the following steps: S51. Receive the original image integer coordinate data; S52. Based on the pixel position indicated by the integer coordinate data of the original image, calculate the read address of all pixels in the RAW domain circular buffer of the adaptive depth within the preset neighborhood of the pixel. S53. Within the same clock cycle, all pixel values ​​are read in parallel from the left-eye RAW cache data and the right-eye RAW cache data according to the read address to generate the local RAW feature window data.

6. The method for combined RAW domain epipolar correction and demosaicing according to claim 1, characterized in that, The step of generating integer coordinate point RGB pixel data in S7 includes the following steps: S71. Receive the local RAW feature window data and the demosaic interpolation weight data; S72. Identify the color channel of each pixel in the local RAW feature window data according to the Bayer array pattern; S73. For the reference pixel and its neighboring pixels corresponding to the integer coordinate data of the original image, the missing color channel values ​​are weighted and interpolated using the demosaic interpolation weight data. S74. For the reference pixel and each pixel at an integer coordinate position in its neighborhood, calculate the complete red, green, and blue three-channel color values ​​respectively, and generate the RGB pixel data of the integer coordinate point.

7. The method for combined RAW domain epipolar correction and demosaicing according to claim 6, characterized in that, The step of generating the target corrected RGB image data in S8 includes the following steps: S81, Receive the RGB pixel data of the integer coordinate point and the coordinate interpolation weight component data; S82. Construct a bilinear interpolation calculation array, and input the RGB values ​​of the four corner points interpolated from the integer coordinate point RGB pixel data, as well as the horizontal and vertical weight components in the coordinate interpolation weight component data, into the bilinear interpolation calculation array. S83. In the bilinear interpolation calculation array, two-level interpolation calculation is performed through a multiplier and adder tree. First, horizontal interpolation is performed based on the horizontal weight component, and then vertical interpolation is performed on the horizontal interpolation result based on the vertical weight component. S84. Output the red, green, and blue channel color values ​​of the current target pixel obtained by final interpolation, as the target corrected RGB image data.

8. A RAW domain epipolar correction and demosaicing joint processing system, used to implement the RAW domain epipolar correction and demosaicing joint processing method according to any one of claims 1-7, characterized in that, The system includes: The RAW data acquisition and caching module acquires binocular raw RAW image streams through the image interface receiving unit, generates single-channel RAW data through the RAW data parsing unit, and streams and caches the RAW line cache data in on-chip memory through the adaptive ring buffer unit, outputting RAW line cache data. The coordinate mapping and scheduling module receives a pre-stored camera inverse mapping table, sequentially reads floating-point coordinates under the pixel clock drive through the coordinate reading unit, and deconstructs the floating-point coordinates into integer coordinates and fractional weights using the coordinate splitting unit, outputting the original image integer coordinate data and coordinate interpolation weight component data. The feature extraction and calculation module receives the RAW row cache data and the original image integer coordinate data, calculates the pixel neighborhood address through the feature window address generation unit, extracts the local RAW pixel window through the parallel reading unit, and uses the edge-aware weight calculation unit to analyze the image direction to generate interpolation weights, and outputs the local RAW feature window data and de-mosaic interpolation weight data. The joint processing and output module receives the local RAW feature window data, the demosaic interpolation weight data, the RGB pixel data of the integer coordinate points, and the coordinate interpolation weight component data. It generates RGB values ​​at the integer coordinate points through the demosaic color restoration unit, performs bilinear fusion calculation through the geometric correction interpolation unit, and streams the corrected image data through the RGB data output unit.

Images