Video frame component oscillogram real-time generation and superposition display method and system

Abstract

The invention relates to a video frame component oscillogram real-time generation and superposition display method and system. The method comprises the steps that video signal input and component extraction are carried out, double frames are alternately cached to a PSRAM, inter-row burst reading is initiated in the column direction, and N columns of data in one row are read in each burst mode; a plurality of block random access memories in the FPGA chip are utilized to construct a multi-column parallel statistical unit, a histogram array is constructed, and parallel statistics is carried out on M columns of data at the same time; after the statistics is completed, alternately writing the statistical results of each column into two special statistical result buffer areas in the external PSRAM; according to a user output display demand, suddenly reading a statistical value of a corresponding column from a statistical result buffer area of an external PSRAM according to the demand, generating an oscillogram in the FPGA chip, superposing the oscillogram and an original video signal, and outputting the superposed oscillogram and the original video signal; compared with the prior art, the method has the core advantages of efficient storage access, high parallel processing, low delay superposition, resource optimization and utilization and the like.

Description

Technical Field

[0001] This invention relates to the field of digital video image processing technology, and more specifically, to a method and system for real-time generation and overlay display of video frame component waveforms. Background Technology

[0002] In the field of video processing and analysis, visualizing and statistically analyzing key parameters such as brightness distribution and color component intensity of image frames (e.g., using waveform graphs or vector oscilloscopes) is a core method for evaluating image quality, adjusting exposure, and correcting color balance. These visualization tools are widely used in broadcasting, film post-production, video surveillance, and real-time streaming media processing, providing technicians with intuitive and quantitative data for image quality assessment.

[0003] Traditional video statistical visualization solutions primarily rely on high-performance central processing units (CPUs) or graphics processing units (GPUs) for post-processing. The basic process involves storing multiple frames of video stream data into system memory, then performing centralized computation on the CPU / GPU to generate statistical histograms or waveform data, which are finally overlaid on the video feed. While software solutions offer high flexibility, they face significant performance bottlenecks when processing high-resolution (e.g., 1080p, 4K, and even 8K) and high-frame-rate real-time video streams. First, the transport and centralized processing of video data consumes substantial memory bandwidth and computing resources, resulting in high system latency, making it difficult to meet the low-latency requirements of real-time monitoring or on-site production. Second, the general-purpose computing architecture of CPUs / GPUs is inefficient in handling pixel-by-pixel statistical tasks, especially in scenarios requiring full-resolution statistics frame-by-frame, where resource consumption remains high, making it difficult to complete complex statistical calculations while ensuring smooth video output.

[0004] To overcome the latency and efficiency issues of traditional software solutions, FPGA chips, due to their parallel processing capabilities and customizable hardware pipeline architecture, have gradually become an important platform for real-time video processing. Existing FPGA solutions mostly employ a pixel-by-pixel pipelined processing architecture, embedding statistical operations (such as luminance extraction, color component separation, and accumulation) into the processing chain of the video pixel stream to achieve low-latency real-time analysis. However, when performing vertical statistical analysis on high-resolution video streams (e.g., summing the luminance values ​​of all pixels in each column or creating a distribution histogram), existing FPGA solutions still face significant challenges. The core bottleneck lies in the limitations of on-chip storage resources and the efficiency of external memory access. 1. On-chip storage resource bottleneck: Vertical statistics require accumulating pixel values ​​column by column, which means that an independent accumulator or histogram buffer needs to be maintained for each column. For high-resolution videos (such as 4K resolution, with 3840 horizontal pixel columns), the number of accumulator or histogram entries required is huge. Although the Block RAM (BRAM) resources on the FPGA chip have fast access speed, their capacity is limited and it is difficult to directly store complete statistical information of all columns of the entire frame. Especially when multiple components (such as R, G, B channels) or high-precision histograms need to be statistically analyzed at the same time, the BRAM resources will be quickly exhausted.

[0005] 2. External memory bandwidth and access efficiency bottleneck: When the on-chip BRAM is insufficient to hold the intermediate vertical statistical data for the entire frame, existing solutions typically require external memory, such as PSRAM or Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), to store and update the intermediate results of column-by-column statistics. However, the data access pattern for vertical statistics is highly discontinuous: the input of each row of pixels is sequential (row scanning), but column-by-column accumulation requires random access to the accumulated value of the corresponding column in external memory. This frequent, fine-grained random access puts enormous pressure on the bandwidth of external memory and causes the processing pipeline to stall due to access latency, making it difficult to complete the full frame statistics within a single frame. In addition, high-frequency random access also leads to low efficiency of the memory controller, further exacerbating the bandwidth bottleneck.

[0006] 3. Difficulty in real-time overlay display and latency control: The results of vertical statistics usually need to be overlaid synchronously with the video stream for users to observe in real time; however, due to the aforementioned statistical processing delay, the generation of statistical results often lags behind the current display frame, resulting in asynchronous overlay display; existing technologies lack an effective mechanism to coordinate the timing relationship between statistical calculation and video output, making it difficult to achieve low-latency overlay of statistical results while ensuring high throughput and uninterrupted output of the video stream.

[0007] In summary, existing technologies lack a complete hardware architecture that can efficiently and collaboratively utilize FPGA internal BRAM resources, optimize external memory access patterns, and combine intelligent buffering strategies to achieve high-throughput vertical statistics and low-latency overlay display when dealing with high-resolution video streams. This has become a key technical obstacle restricting the performance improvement of real-time video quality analysis tools. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a method for real-time generation and overlay display of video frame component waveforms, and a system for real-time generation and overlay display of video frame component waveforms, which addresses the problems of slow statistical speed, inability to meet high frame rate real-time processing, and low hardware resource utilization in the prior art.

[0009] The technical solution adopted by this invention to solve its technical problem is: A method for real-time generation and overlay display of video frame component waveforms is constructed, wherein the method includes: Video signal input and component extraction: Receive video signal, parse synchronization signal through the internal logic module of FPGA chip, and extract the luminance component, RGB component or YUV component of each pixel in real time according to user configuration to form component data stream; Dual-frame alternating buffering to PSRAM: The extracted whole-frame component data is alternately written to two independent frame buffers in the external PSRAM, line by line, to achieve a double buffering mechanism; Cross-row burst read and multi-column parallel statistics: For the frame to be counted in the external PSRAM, cross-row burst read is initiated in the column direction, and N columns of data in one row are read in each burst; multi-column parallel statistics unit is constructed using multiple block random access memories inside the FPGA chip to construct a histogram array, and parallel statistics are performed on M columns of data at the same time; Double-buffered write-back of statistical results: After the statistics are completed, the statistical results of each column are alternately written to two dedicated statistical result buffers in the external PSRAM; Statistical image generation and video overlay output: Based on the user's output display requirements, the statistical values ​​of the corresponding columns are read in bursts as needed from the statistical result buffer of the external PSRAM, a waveform is generated inside the FPGA chip, and the waveform is overlaid with the original video signal and output.

[0010] The method for real-time generation and overlay display of video frame component waveforms according to the present invention wherein the depth of the histogram array is matched with the height of the video frame.

[0011] The video frame component waveform real-time generation and overlay display method of the present invention, wherein the two independent frame buffers alternately written to the external PSRAM include: When the current frame data is written to the first frame buffer, the previous frame data is read from the second frame buffer for statistical analysis.

[0012] The method for real-time generation and overlay display of video frame component waveforms according to the present invention further includes, after alternately writing the statistical results of each column into two dedicated statistical result buffers in an external PSRAM: Clear the block random access memory used for statistics inside the FPGA chip.

[0013] The video frame component waveform real-time generation and overlay display method of the present invention, wherein the number of columns N for cross-row burst reading and the number of columns M for parallel statistics are dynamically configured according to the number of Block RAM resources and PSRAM bandwidth inside the FPGA.

[0014] The method for real-time generation and overlay display of video frame component waveforms according to the present invention, wherein N≥8 and M≥8.

[0015] The video frame component waveform real-time generation and overlay display method of the present invention, wherein the horizontal axis of the waveform graph represents the image position and the vertical axis represents each component; the image brightness represents the quantity of each component at a specified position in the image, and the brighter the brightness, the larger the proportion of this component in this column.

[0016] The method for real-time generation and overlay display of video frame component waveforms according to the present invention includes statistical types such as: brightness value, and the percentage of each RGB component and YUV component in the vertical direction of the image.

[0017] The method for real-time generation and overlay display of video frame component waveforms according to the present invention includes video signals such as HDMI, SDI, or parallel RGB / YUV.

[0018] A system for real-time generation and overlay display of video frame component waveforms is provided to implement the above-mentioned method for real-time generation and overlay display of video frame component waveforms. The system includes: a component extraction module, a PSRAM module, a multi-column parallel statistics module, and a chart overlay module. The component extraction module receives the video signal and parses the synchronization signal. According to the user configuration, it extracts the luminance component, RGB component or YUV component of each pixel in real time to form a component data stream. The extracted whole frame component data is written alternately to the two independent frame buffers of the PSRAM module in rows. The multi-column parallel statistics module initiates cross-row burst reads in the column direction for the frame to be counted in the external PSRAM, and reads N columns of data in one row in each burst; constructs a histogram array and performs parallel statistics on M columns of data at the same time; after the statistics are completed, the statistical results of each column are alternately written to two dedicated statistical result buffers in the external PSRAM. The chart overlay module reads the statistical values ​​of the corresponding columns from the statistical result buffer of the external PSRAM as needed according to the user's output display requirements, generates a waveform graph inside the FPGA chip, and outputs the waveform graph after overlaying it with the original video signal.

[0019] The beneficial effects of this invention are as follows: 1. High-efficiency storage access: By using the "double frame buffering" and "double buffering of statistical results" mechanisms, combined with the cross-row burst read of PSRAM, the external storage bandwidth is maximized, solving the problem of low random access efficiency in vertical statistics.

[0020] 2. High parallel processing: By utilizing the BRAM resources of the FPGA to construct multi-column parallel statistical units, synchronous processing of multiple columns of data is achieved, which greatly improves the throughput of statistical calculations and meets the real-time requirements of high frame rate and high resolution video.

[0021] 3. Low-latency overlay: Through pipeline optimization and timely resource clearing, the rapid generation of statistical results and strict synchronization of video output are ensured, realizing low-latency overlay display of waveform graphs.

[0022] 4. Optimized resource utilization: It cleverly balances the use of the FPGA's internal BRAM (for high-speed parallel computing) and external PSRAM (for large-capacity data caching), achieving high-performance vertical statistics functions with limited hardware resources. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a flowchart of a preferred embodiment of the method for real-time generation and overlay display of video frame component waveforms; Figure 2 This is a schematic diagram of the PSRAM storage layout of the video frame component waveform real-time generation and overlay display method according to a preferred embodiment of the present invention; Figure 3 This is a diagram illustrating the brightness waveform overlay display effect of the video frame component waveform real-time generation and overlay display method according to a preferred embodiment of the present invention. Figure 4 This is a preferred embodiment of the video frame component waveform real-time generation and overlay display method of the present invention, showing the RGB waveform overlay display effect; Figure 5 This is a YUV waveform overlay display effect diagram of the video frame component waveform real-time generation and overlay display method of the preferred embodiment of the present invention; Figure 6 This is a block diagram illustrating the principle of a video frame component waveform real-time generation and overlay display system according to a preferred embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0025] The preferred embodiment of the present invention provides a method for real-time generation and overlay display of video frame component waveforms, such as... Figure 1 As shown, see also Figures 2-5 ,include: S01: Video signal input and component extraction: It receives standard video signals (such as HDMI, SDI or parallel RGB / YUV), parses the synchronization signal (HSYNC / VSYNC) through the internal logic module of the FPGA, and extracts the luminance component (Y), RGB component or YUV component of each pixel in real time according to the user configuration to form a component data stream; S02: Alternately buffer two frames to PSRAM: The extracted whole-frame component data is written alternately, line by line, to two independent frame buffers (Frame Buffer A and Frame Buffer B) in the external PSRAM to implement a double-buffering mechanism. When the current frame is written to Buffer A, the data of the previous frame can be read from Buffer B for statistics, avoiding read-write conflicts; S03: Cross-row burst reads and multi-column parallel statistics: For frames to be counted in PSRAM, a cross-row burst read is initiated in the column direction, and N (N≥8) columns of data in one row are read in each burst. Multiple block RAMs inside the FPGA are used to build a multi-column parallel statistical unit to construct a histogram array. It supports parallel statistics on M (M≥8) columns at the same time, which greatly improves throughput efficiency. S04: Statistical results are double-buffered and written back: After the statistics are completed, the statistical results of each column (statistical values ​​of the quantity of each component) are alternately written to two dedicated statistical result buffers (Stat Buffer X / Y) in PSRAM; at the same time, all Block RAM used for statistics is cleared to prepare for the next frame of statistics and ensure that there is no state residue. S05: Statistical Image Generation and Video Overlay Output: Based on the output display requirements (such as waveform height and position), the statistical values ​​of the corresponding columns are burst-read from the statistical result buffer of PSRAM as needed; a waveform graph is generated inside the FPGA (such as the horizontal axis representing the image position, the vertical axis representing each component; the image brightness represents the quantity of each component at a specified position in the image, the brighter the brightness, the larger the proportion of this component in this column); the generated waveform graph is alpha-blended or overlaid with the original video signal in a specified area (such as the right side or bottom of the screen); the final synthesized video signal is output. It should be noted that, as a preferred option, the number of columns N for burst reading and the number of columns M for statistical parallel processing can be dynamically configured based on the PSRAM bandwidth, FPGA resources, and resolution; the statistical types include: brightness value, and the percentage statistics of each RGB component and YUV component in the vertical direction of the image; the overlay area supports user-configurable position, transparency, color mapping, and coordinate axis customization; the PSRAM interface adopts a dedicated high-speed interface, supporting high-bandwidth burst transmission.

[0026] This invention has at least the following advantages: 1. High-efficiency storage access: By using the "double frame buffering" and "double buffering of statistical results" mechanisms, combined with the cross-row burst read of PSRAM, the external storage bandwidth is maximized, solving the problem of low random access efficiency in vertical statistics.

[0027] 2. High parallel processing: By utilizing the BRAM resources of the FPGA to construct multi-column parallel statistical units, synchronous processing of multiple columns of data is achieved, which greatly improves the throughput of statistical calculations and meets the real-time requirements of high frame rate and high resolution video.

[0028] 3. Low-latency overlay: Through pipeline optimization and timely resource clearing, the rapid generation of statistical results and strict synchronization of video output are ensured, realizing low-latency overlay display of waveform graphs.

[0029] 4. Optimized resource utilization: It cleverly balances the use of the FPGA's internal BRAM (for high-speed parallel computing) and external PSRAM (for large-capacity data caching), achieving high-performance vertical statistics functions with limited hardware resources.

[0030] Taking 1080p@60Hz video input as an example, the explanation is as follows: The video is decoded via HDMI to obtain 24-bit RGB data, and the user selects the "luminance component"; The FPGA has a built-in Y = 0.299R + 0.587G + 0.114B calculation module, which outputs an 8-bit Y value; For each frame of 1920×1080 pixel Y data, the arithmetic mean of every N horizontally adjacent Y data is taken as the height of a point on the waveform. N can be customized according to requirements; here, N is set to 4, meaning 480 points are sampled. Video frame component data is alternately written to two non-overlapping frame buffers in PSRAM: Buffer A starts at address 0x000000, and Buffer B starts at address 0x3FF800. Each buffer is 1920×1080 bytes in size (corresponding to a single component data of one frame of 1080p video). The aforementioned percentage statistics refer to: for each image column, counting the frequency of each component value (e.g., Y=0~255), and mapping it to the display height. Statistics are started after the falling edge of VSYNC: Step 1: The controller initiates a burst read on Buffer B, reading 128 columns (128 bytes in total) from one row each time. Step 2: Parallel accumulation and statistics are started across 8 blocks of RAM, with each block allocated 16 columns (16 bits of data). Each column is an 8-bit wide, 256-bit deep histogram. Each block of RAM has a bit width of 32 bits and a depth of 1024, allowing for the statistics of 16 columns, thus maximizing the utilization of the block RAM.

[0031] Step 3: After completing steps 1 and 2, perform statistics on 128 columns in each row of data. Repeat this process 1080 times to complete the statistics on 128 columns in 1080 rows. The completed data is also written in a burst across rows. Because the blanking time for each row in a video is generally short, a continuous burst read operation is used to read the statistics for an entire row. Here, each burst write operation involves a burst of 8*128 bits, or 128 columns, and this is repeated 256 times (the data bit width is 8 bits, and the maximum resolution is 256; the data bit width can be changed according to actual needs) to complete the storage of the 128 columns of statistics. The starting address is 0x3FFC00 (this address is 23 bits wide, with the high 12 bits being the row address and the low 11 bits being the column address), and the buffer size is 480×256 bytes. While reading data from block RAM and writing it to PSRAM, the other end is cleared to prepare for the next statistics.

[0032] Step 4: Running Step 3 4 times will complete the statistics for 1080 rows × 512 columns, but the goal is to only run up to 480 columns. At the same time, the address of Stat Buffer Y can be changed according to the actual situation.

[0033] During video output, during the blanking period after each row of valid pixels, the statistical values ​​of the corresponding row are read from Stat Buffer X into the FPGA's Block RAM. Then, based on the frequency of each component (e.g., brightness values ​​0–255) in each column, visual pixels whose brightness is proportional to their frequency are generated, and different colors are used to distinguish the statistical results from different components. Finally, the video pixels are output and displayed using alpha blending. The vertical axis from bottom to top represents the scale of each component from 0 to 255, and the horizontal axis from left to right represents the position of the image from left to right. Simultaneously, the brightness of a specified background area can be reduced before alpha blending to highlight the graph; the final HDMI output includes a waveform overlaid on the original image.

[0034] A system for real-time generation and overlay display of video frame component waveforms is provided to implement the aforementioned method for real-time generation and overlay display of video frame component waveforms, such as... Figure 6 As shown, the system includes: a component extraction module 100, a PSRAM module 101, a multi-column parallel statistics module 102, and a chart overlay module 103; The component extraction module 100 receives the video signal and parses the synchronization signal. According to the user configuration, it extracts the luminance component, RGB component or YUV component of each pixel in real time to form a component data stream. The extracted whole frame component data is written alternately to the two independent frame buffers of the PSRAM module 101 in rows. The multi-column parallel statistics module 102 initiates cross-row burst reads in the column direction for the frame to be counted in the external PSRAM, and reads N columns of data in one row in each burst; constructs a histogram array and performs parallel statistics on M columns of data at the same time; after the statistics are completed, the statistical results of each column are alternately written to two dedicated statistical result buffers in the external PSRAM. The chart overlay module 103 reads the statistical values ​​of the corresponding column from the statistical result buffer of the external PSRAM as needed according to the user's output display requirements, generates a waveform graph inside the FPGA chip, and outputs the waveform graph after overlaying it with the original video signal. This invention has at least the following advantages: High real-time performance: Through cross-row burst reading + multi-column parallel statistics, the statistical throughput is significantly improved, ensuring that the vertical statistics of the entire frame are completed within a single frame time, meeting the real-time processing requirements of 60fps and above; Resource efficiency: Utilizing parallel processing of multiple block RAMs within the FPGA avoids bottlenecks caused by frequent access to external storage, reduces the number of row crossings, and lowers PSRAM bandwidth pressure; Low-latency overlay: A double-buffering mechanism is adopted (both frame data and statistical results are double-buffered) to achieve "statistics and display on the same side", with an overlay latency of ≤2 frames; Highly flexible: Supports multiple component selection including brightness, RGB, and YUV; statistical types and display styles are configurable. Highly practical for engineering applications: It only requires FPGA + PSRAM / DDR, without the need for an additional processor, making it suitable for embedded video analysis devices.

[0035] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for real-time generation and overlay display of video frame component waveforms, characterized in that the method... include: Video signal input and component extraction: Receive video signal, parse synchronization signal through the internal logic module of FPGA chip, and extract the luminance component, RGB component or YUV component of each pixel in real time according to user configuration to form component data stream; Dual-frame alternating buffering to PSRAM: The extracted whole-frame component data is alternately written to two independent frame buffers in the external PSRAM, line by line, to achieve a double buffering mechanism; Cross-row burst read and multi-column parallel statistics: For the frame to be counted in the external PSRAM, cross-row burst read is initiated in the column direction, and N columns of data in one row are read in each burst; multi-column parallel statistics unit is constructed using multiple block random access memories inside the FPGA chip to construct a histogram array, and parallel statistics are performed on M columns of data at the same time; Double-buffered write-back of statistical results: After the statistics are completed, the statistical results of each column are alternately written to two dedicated statistical result buffers in the external PSRAM; Statistical image generation and video overlay output: Based on the user's output display requirements, the statistical values ​​of the corresponding columns are read in bursts as needed from the statistical result buffer of the external PSRAM, a waveform is generated inside the FPGA chip, and the waveform is overlaid with the original video signal and output.

2. The method for real-time generation and overlay display of video frame component waveforms according to claim 1, characterized in that, The depth of the histogram array is matched to the height of the video frame.

3. The method for real-time generation and overlay display of video frame component waveforms according to claim 1, characterized in that, The two independent frame buffers that are alternately written to the external PSRAM include: When the current frame data is written to the first frame buffer, the previous frame data is read from the second frame buffer for statistical analysis.

4. The method for real-time generation and overlay display of video frame component waveforms according to claim 1, characterized in that, The process of alternately writing the statistical results of each column to two dedicated statistical result buffers in the external PSRAM also includes: Clear the block random access memory used for statistics inside the FPGA chip.

5. The method for real-time generation and overlay display of video frame component waveforms according to claim 1, characterized in that, The number of columns N for cross-row burst reads and the number of columns M for parallel statistics are dynamically configured based on the amount of Block RAM resources and PSRAM bandwidth within the FPGA.

6. The method for real-time generation and overlay display of video frame component waveforms according to claim 1, characterized in that, The N≥8, M≥8.

7. The method for real-time generation and overlay display of video frame component waveforms according to claim 1, characterized in that, The horizontal axis of the waveform graph represents the image position, and the vertical axis represents each component; the image brightness represents the quantity of each component at a specified position in the image, and the brighter the brightness, the greater the proportion of that component in this column.

8. The method for real-time generation and overlay display of video frame component waveforms according to claim 1, characterized in that, The statistical types include: brightness values, and the percentage of each RGB and YUV component in the vertical direction of the image.

9. The method for real-time generation and overlay display of video frame component waveforms according to claim 1, characterized in that, The video signal includes HDMI, SDI, or parallel RGB / YUV.

10. A system for real-time generation and overlay display of video frame component waveforms, used to implement the method for real-time generation and overlay display of video frame component waveforms as described in any one of claims 1-9, characterized in that, The system includes: a component extraction module, a PSRAM module, a multi-column parallel statistics module, and a chart overlay module; The component extraction module receives the video signal and parses the synchronization signal. According to the user configuration, it extracts the luminance component, RGB component or YUV component of each pixel in real time to form a component data stream. The extracted whole frame component data is written alternately to the two independent frame buffers of the PSRAM module in rows. The multi-column parallel statistics module initiates cross-row burst reads in the column direction for the frame to be counted in the external PSRAM, and reads N columns of data in one row in each burst; constructs a histogram array and performs parallel statistics on M columns of data at the same time; after the statistics are completed, the statistical results of each column are alternately written to two dedicated statistical result buffers in the external PSRAM. The chart overlay module reads the statistical values ​​of the corresponding columns from the statistical result buffer of the external PSRAM as needed according to the user's output display requirements, generates a waveform graph inside the FPGA chip, and outputs the waveform graph after overlaying it with the original video signal.

Images