A method for solving jitter in video loop closure test data based on OpenCV on RK3576

Abstract

This invention discloses a method for resolving jitter in video loopback testing data based on OpenCV on the RK3576. It constructs a physical loopback link between DP and MIPI CSI using a SerDes chip, converts the acquired images to YUV420 format, and separates the Y, U, and V channels. The Y channel undergoes 3×3 mean filtering and 5×5 Gaussian filtering sequentially, while the U / V channels undergo 3×3 Gaussian filtering preprocessing. The Structural Similarity Index (SSIM) is used to calculate channel-specific similarity, fused with a weighted ratio of 0.6:0.2:0.2, and a threshold is dynamically adjusted based on the standard deviation of 10 consecutive frames (default 99%, relaxed to 98.5% in stable scenarios) to determine image consistency. This solution is adapted to the hardware characteristics of the RK3576, effectively suppressing the influence of MIPI CSI link noise through multi-stage noise reduction, channel-weighted comparison, and a dynamic threshold mechanism. This improves the anti-jitter capability and reliability of image comparison in video loopback testing, and is suitable for closed-loop testing scenarios in embedded image acquisition systems.

Description

Technical Field

[0001] This invention relates to the fields of image processing and embedded system testing technology, and in particular to a method for solving video loop closure test data jitter based on OpenCV on RK3576. Background Technology

[0002] In the existing technology of image processing and embedded system testing, there is a problem of image comparison failure caused by signal noise or hardware jitter. Therefore, there is a need to provide a method based on OpenCV on RK3576 to solve the jitter of video loop closure test data in order to solve the above-mentioned technical problems. Summary of the Invention

[0003] The purpose of this invention is to provide a method for resolving data jitter in video loop closure testing using OpenCV on the RK3576 platform. Specifically, it relates to an image comparison method based on the RK3576 platform and utilizing the OpenCV library to implement data jitter-resistant image comparison in video loop closure testing. This method is applicable to embedded systems that acquire images via the MIPI CSI interface. Through multi-stage image preprocessing and channel-specific similarity analysis, it solves the problem of image comparison failure caused by signal noise or hardware jitter.

[0004] This invention is achieved through the following technical solution:

[0005] A method for resolving jitter in video loop closure test data based on OpenCV using RK3576 includes the following steps:

[0006] (1) A physical loop link between the DP output interface and the MIPI CSI input interface of RK3576 is constructed through the SerDes conversion chip to form a closed-loop transmission path for video signals;

[0007] (2) Convert the image acquired by MIPI CSI to YUV420 format, and separate the luminance channel (Y) and chrominance channels (U, V), where the resolution of the Y channel is 1920×1080 and the resolution of the U / V channels is 960×540;

[0008] (3) Perform 3×3 mean filtering and 5×5 Gaussian filtering on the Y channel in sequence, and perform 3×3 Gaussian filtering on the U / V channel;

[0009] (4) Calculate the structural similarity index (SSIM) of the Y, U, and V channels respectively, and fuse them with a weight of 0.6:0.2:0.2 to obtain the total similarity;

[0010] (5) The judgment threshold is dynamically adjusted according to the standard deviation of the test results of 10 consecutive frames. If the standard deviation is ≤0.3%, the threshold is automatically relaxed to 98.5%; otherwise, the default threshold of 99% is used, and the test results are output.

[0011] As a further improvement to the technical solution of the present invention, the SerDes conversion chip in step (1) includes DS90UB954 / 953, MAX9286 ​​or equivalent functional chips, and the pre-emphasis and equalization functions are enabled in the chip configuration to compensate for signal transmission distortion.

[0012] As a further improvement to the technical solution of the present invention, the calculation formula for the mean filtering in step (3) is as follows:

[0013]

[0014] Where F is the input image matrix and G is the filtered image.

[0015] As a further improvement to the technical solution of this invention, the kernel function of the Gaussian filter in step (3) is:

[0016]

[0017] The standard deviation σ of the Gaussian filter in the Y channel ranges from 1 to 3, with a default value of 2.

[0018] As a further improvement to the technical solution of this invention, the single-channel SSIM calculation in step (4) adopts the following formula:

[0019]

[0020] in, The channel mean. For channel variance, For covariance, , , This represents the range of pixel values.

[0021] As a further improvement to the technical solution of the present invention, the YUV420 format conversion in step (2) is implemented by the cvtColor function of OpenCV, and the storage address of the U / V channel after conversion is offset by 1920×1080 pixels relative to the Y channel.

[0022] As a further improvement to the technical solution of this invention, the dynamic threshold adjustment formula in step (5) is as follows:

[0023]

[0024] in, , std is the standard deviation of SSIM values ​​over 10 consecutive frames.

[0025] As a further improvement to the technical solution of the present invention, it also includes adaptive histogram equalization (CLAHE) processing of the Y channel, limiting the contrast to 1.0-3.0, so as to enhance the edge sharpness of low-contrast images.

[0026] As a further improvement to the technical solution of the present invention, the test result output includes preprocessed images of each channel, real-time SSIM value curves (Y / U / V channels and total results), noise level classification (low / medium / high noise) and judgment conclusions.

[0027] As a further improvement to the technical solution of this invention, the hardware environment includes:

[0028] The RK3576 development board integrates a DP 1.4 interface and a 4-lane MIPI CSI-2 interface;

[0029] Analog image sensor module, supporting MIPI CSI input with a resolution ≥1080p.

[0030] In summary, the beneficial effects of the present invention are as follows:

[0031] Hardware-software collaborative noise reduction:

[0032] By cascading pre-emphasis using the SerDes chip and software mean / Gaussian filtering, the impact of MIPI CSI link noise is reduced by more than 70% (compared to traditional single-stage filtering).

[0033] YUV channel separation adapts to hardware downsampling characteristics, avoiding reference image deviation caused by format mismatch, and the measured comparison time is reduced by 30%.

[0034] Optimization of human visual characteristics:

[0035] The high weighting of the Y channel (0.6) significantly improves the accuracy of brightness detail recognition, while the low weighting of the U / V channels reduces the chromaticity noise false positive rate by 60%.

[0036] The dynamic threshold mechanism adapts to changes in the hardware environment, reducing the number of false positives by 85% in long-term testing (compared to the fixed threshold scheme).

[0037] Efficient utilization of embedded resources:

[0038] The preprocessing algorithm selected is a lightweight filter (mean + Gaussian), with a single frame processing time of <15ms, which meets the requirements of 30fps real-time testing;

[0039] The multi-channel computing architecture fully utilizes the parallel processing capabilities of the RK3576's ARM multi-core CPU, avoiding GPU resource consumption. Attached Figure Description

[0040] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0041] Figure 1 Hardware architecture diagram for RK3576 video loopback testing;

[0042] Figure 2 is a flowchart of a method for solving data jitter in video loopback testing based on OpenCV on RK3576 according to the present invention.

[0043] Figure 3 is a schematic diagram of YUV channel separation and weighted fusion. Detailed Implementation

[0044] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application are clearly and completely described. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] Reference Figures 1 to 3 A method for resolving jitter in video loop closure test data based on OpenCV in RK3576 includes the following steps:

[0046] (1) A physical loop link between the DP output interface and the MIPI CSI input interface of RK3576 is constructed through the SerDes conversion chip to form a closed-loop transmission path for video signals;

[0047] (2) Convert the image acquired by MIPI CSI to YUV420 format, and separate the luminance channel (Y) and chrominance channels (U, V), where the resolution of the Y channel is 1920×1080 and the resolution of the U / V channels is 960×540;

[0048] (3) Perform 3×3 mean filtering and 5×5 Gaussian filtering on the Y channel in sequence, and perform 3×3 Gaussian filtering on the U / V channel;

[0049] (4) Calculate the structural similarity index (SSIM) of the Y, U, and V channels respectively, and fuse them with a weight of 0.6:0.2:0.2 to obtain the total similarity;

[0050] (5) The judgment threshold is dynamically adjusted according to the standard deviation of the test results of 10 consecutive frames. If the standard deviation is ≤0.3%, the threshold is automatically relaxed to 98.5%; otherwise, the default threshold of 99% is used, and the test results are output.

[0051] It should be noted that the hardware loopback link adaptation is constructed as follows:

[0052] Customized connection solution: A physical loopback is built using a SerDes chip (such as DS90UB954 / 953) that supports DP to MIPI CSI protocol conversion. Taking into account the DP 1.4 interface characteristics of RK3576, the output image resolution is set to 1920×1080 and the frame rate is 30fps. After being encoded by the conversion chip, it is received by the MIPI CSI-2 interface, ensuring that the hardware link transmission format is consistent with the YUV420 output format of the acquisition end (such as IMX490 sensor).

[0053] Signal integrity compensation: Enable pre-emphasis and equalization functions in the SerDes chip configuration to reduce edge distortion in high-speed signal transmission and provide basic noise reduction for subsequent software preprocessing.

[0054] Specifically, in this embodiment, the SerDes conversion chip mentioned in step (1) includes DS90UB954 / 953, MAX9286 ​​or equivalent functional chips, and the pre-emphasis and equalization functions are enabled in the chip configuration to compensate for signal transmission distortion.

[0055] Specifically, in this embodiment, after converting the BGR image acquired by MIPI CSI into a grayscale image, a 3×3 window mean filter is used to eliminate random isolated noise points by averaging neighboring pixels; the calculation formula for the mean filter in step (3) is as follows:

[0056]

[0057] Where F is the input image matrix and G is the filtered image.

[0058] Specifically, in this embodiment, the luminance channel (Y) is preferentially subjected to a 5×5 Gaussian filter (standard deviation σ=2) to suppress high-frequency jitter using the weight distribution of the Gaussian kernel; the kernel function of the Gaussian filter in step (3) is:

[0059]

[0060] The standard deviation σ of the Gaussian filter in the Y channel ranges from 1 to 3, with a default value of 2.

[0061] Specifically, in this embodiment, the single-channel SSIM calculation in step (4) uses the following formula:

[0062]

[0063] in, The channel mean. For channel variance, For covariance, , , This represents the range of pixel values.

[0064] Specifically, in this embodiment, the YUV420 format conversion in step (2) is implemented using the cvtColor function of OpenCV, and the storage address of the U / V channels after conversion is offset by 1920×1080 pixels relative to the Y channel. It should be noted that the reference image and the acquired image are uniformly converted to YUV420 format, separating the Y (1920×1080), U (960×540), and V (960×540) channels to match the 4:2:0 downsampling characteristics of the chroma signal in the hardware link, avoiding comparison deviations caused by inconsistent resolutions. The full-resolution SSIM is used for the luminance channel (Y) to retain the details that are sensitive to the human eye. The SSIM is calculated at the downsampled resolution (960×540) for the chroma channels (U / V) to reduce the sensitivity to chroma noise.

[0065] Specifically, in this embodiment, the dynamic threshold adjustment formula in step (5) is:

[0066]

[0067] in, , std is the standard deviation of SSIM values ​​over 10 consecutive frames.

[0068] Specifically, this embodiment also includes adaptive histogram equalization (CLAHE) processing of the Y channel, limiting the contrast to 1.0-3.0 to enhance the edge sharpness of low-contrast images.

[0069] Specifically, in this embodiment, the test result output includes preprocessed images of each channel, real-time SSIM value curves (Y / U / V channels and total results), noise level classification (low / medium / high noise), and judgment conclusions.

[0070] Specifically, in this embodiment, the hardware environment includes:

[0071] The RK3576 development board integrates a DP 1.4 interface and a 4-lane MIPI CSI-2 interface;

[0072] Analog image sensor module, supporting MIPI CSI input with a resolution ≥1080p.

[0073] It should be noted that the dynamic threshold determination and result output are as follows:

[0074] Dual threshold calibration mechanism:

[0075] Basic threshold: 99% is used as the default judgment threshold to ensure strict comparison of clear images;

[0076] Adaptive correction: Calculate the SSIM standard deviation by continuously acquiring 10 frames of images. If the standard deviation is <0.3% (indicating stable noise), the threshold is automatically relaxed to 98.5% to avoid over-misjudgment caused by inherent hardware noise.

[0077] Multi-dimensional result output:

[0078] The test results are output through the RK3576 system log interface, including:

[0079] Preprocessed images of each channel (for problem reproduction);

[0080] Real-time SSIM value curve (Y / U / V channels and weighted total result);

[0081] Judgment conclusion and noise level (classification of low / medium / high noise scene).

[0082] Example:

[0083] Hardware environment setup

[0084] Core components:

[0085] The RK3576 development board is used, which integrates a DP 1.4 interface and a 4-lane MIPI CSI-2 interface.

[0086] The system uses the SerDes conversion chips DS90UB954 (transmitter) and DS90UB953 (receiver), which support a speed of 1.5Gbps / lane.

[0087] It is equipped with an analog image sensor module that supports MIPI CSI input and can achieve a resolution of 1080p.

[0088] Connection method: Connect the DP output interface of the RK3576 development board through the DS90UB954 chip. The DS90UB954 encodes the signal output by the DP and then transmits the encoded signal to the DS90UB953 chip through the line. The DS90UB953 decodes the signal and connects the decoded signal to the MIPI CSI input interface of the RK3576, thereby constructing a physical loopback link.

[0089] 2. Software Implementation Steps

[0090] Image acquisition and format conversion:

[0091] Run the relevant program on the RK3576 development board, control the MIPI CSI module through the Linux V4L2 driver to acquire a frame of image, and save it as a BGR format file.

[0092] Using the functions provided by the OpenCV library, the acquired BGR format image was converted to YUV420 format. Then, following the storage rules of YUV420 format, the luma (Y), chroma (U), and chroma (V) channels were separated. The Y channel resolution remained at 1920×1080, while the U and V channels were downsampled due to format characteristics, resulting in a resolution of 960×540.

[0093] Image preprocessing:

[0094] First, mean filtering is performed on the separated Y channel image. This is done by sliding a window of a specific size (e.g., 3×3) across the image and averaging the pixel values ​​within the window to remove salt-and-pepper noise.

[0095] Then, the Y channel image after mean filtering is processed by Gaussian filtering. Gaussian filtering assigns different weights to pixels based on their distance from surrounding pixels, smoothing the image and reducing edge jitter.

[0096] For the U and V channel images, Gaussian filtering is also performed to reduce the impact of noise on chromaticity information.

[0097] For low-contrast scenes, adaptive histogram equalization can be applied to the Y channel image. This process enhances edge sharpness and improves overall image contrast.

[0098] Channel-specific structural similarity calculation and result determination:

[0099] Structural similarity (SSIM) is calculated for the Y, U, and V channels of both the baseline image and the preprocessed image to be compared. This calculation is performed using relevant image processing libraries.

[0100] Based on the human eye's perception of brightness and chromaticity, the Y channel is assigned a higher weight (e.g., 0.6), while the U and V channels are each assigned a lower weight (e.g., 0.2). The SSIM values ​​calculated from the three channels are then weighted and fused according to this weight ratio to obtain the final similarity value.

[0101] Set an initial similarity threshold (e.g., 99%), and compare the final similarity value with this threshold. If the similarity value is greater than or equal to the threshold, the two images are considered identical, and the test passes; otherwise, the images are considered different, and the test fails.

[0102] The threshold can be dynamically adjusted based on the test results of multiple consecutive frames (e.g., 10 frames). For example, if the fluctuations in the test results of multiple consecutive frames are small, it indicates that the test environment is relatively stable, and the threshold can be appropriately relaxed; if the fluctuations are large, the threshold needs to be tightened to ensure the accuracy of the test.

[0103] Finally, the test results are output via the RK3576's UART or USB interface. The output information includes the SSIM value of each channel, the time spent on preprocessing, and the final judgment.

[0104] 3. Alternative solutions

[0105] Filtering algorithm replacement: When encountering high-density salt-and-pepper noise, mean filtering can be replaced with median filtering. Median filtering is more effective at handling impulse noise.

[0106] Similarity algorithm upgrade: The multi-scale SSIM (MSSIM) algorithm is adopted to replace the single-scale SSIM calculation. The multi-scale SSIM algorithm can better adapt to image scaling jitter at different resolutions.

[0107] Hardware link optimization: Enable the adaptive equalization (CTLE) function in the SerDes chip configuration. This function can further process and optimize the signal, and combined with software preprocessing, it forms a cascaded denoising effect, improving image quality.

[0108] In summary, the beneficial effects of this invention are as follows:

[0109] Hardware-software collaborative noise reduction:

[0110] By cascading pre-emphasis using the SerDes chip and software mean / Gaussian filtering, the impact of MIPI CSI link noise is reduced by more than 70% (compared to traditional single-stage filtering).

[0111] YUV channel separation adapts to hardware downsampling characteristics, avoiding reference image deviation caused by format mismatch, and the measured comparison time is reduced by 30%.

[0112] Optimization of human visual characteristics:

[0113] The high weighting of the Y channel (0.6) significantly improves the accuracy of brightness detail recognition, while the low weighting of the U / V channels reduces the chromaticity noise false positive rate by 60%.

[0114] The dynamic threshold mechanism adapts to changes in the hardware environment, reducing the number of false positives by 85% in long-term testing (compared to the fixed threshold scheme).

[0115] Efficient utilization of embedded resources:

[0116] The preprocessing algorithm selected is a lightweight filter (mean + Gaussian), with a single frame processing time of <15ms, which meets the requirements of 30fps real-time testing;

[0117] The multi-channel computing architecture fully utilizes the parallel processing capabilities of the RK3576's ARM multi-core CPU, avoiding GPU resource consumption.

[0118] The technical solutions provided by the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the embodiments of the present invention. The descriptions of the embodiments above are only for helping to understand the principles of the embodiments of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for solving video loopback test data jitter based on RK3576 OpenCV, characterized in that, Comprise the following steps: (1) RK3576 DP output interface and MIPI CSI input interface are connected by SerDes conversion chip to form a physical loopback link, and a video signal closed-loop transmission path is formed; (2) The image collected by MIPI CSI is converted into YUV420 format, and the luminance channel Y, the chroma channel U and the chroma channel V are separated, wherein the resolution of the luminance channel Y is 1920x1080, and the resolution of the chroma channel U and the chroma channel V is 960x540; (3) The luminance channel Y is sequentially subjected to 3x3 mean filtering and 5x5 Gaussian filtering, and the chroma channel U and the chroma channel V are subjected to 3x3 Gaussian filtering respectively; (4) The structural similarity index of the luminance channel Y, the chroma channel U and the chroma channel V is calculated respectively, and the total similarity is obtained by weight fusion according to 0.6:0.2:0.2; (5) The threshold value is dynamically adjusted according to the standard deviation of the test results of 10 consecutive frames, if the standard deviation is less than or equal to 0.3%, the threshold value is automatically relaxed to 98.5%, otherwise the default threshold value 99% is used, and the test results are output; The output test results include preprocessed channel images, real-time SSIM value curve, noise level classification and judgment conclusion.

2. The method for solving video loop test data jitter based on RK3576 and OpenCV according to claim 1, characterized in that: The SerDes conversion chip in step (1) comprises DS90UB954 chip and DS90UB953 chip, and the pre-emphasis and equalization functions are enabled in the chip configuration to compensate for signal transmission distortion.

3. The method of claim 1, wherein the RK3576 based OpenCV solution is used to solve video loopback test data jitter. The YUV420 format conversion in step (2) is realized by cvtColor function of OpenCV, and the storage addresses of the chroma channel U and the chroma channel V are offset by 1920x1080 pixels relative to the luminance channel Y.

4. The method for solving video loop test data jitter based on RK3576 and OpenCV according to claim 1, characterized in that: It also includes adaptive histogram equalization processing of the luminance channel Y, and the contrast is 1.0 to 3.0.

Images