A method, system, device, medium, and product for multi-video frame fusion display.
By dynamically adjusting the read threshold based on real-time statistics of write progress and frame rate ratio in the FPGA chip, the latency problem of multi-channel asynchronous video layer fusion was solved, achieving low-latency video display, improving user experience and reducing chip cost.
Patent Information
- Application Number
- CN202611228108.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-13
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies cause a delay of at least one frame in the time required for fusion of multiple asynchronous video layers, which affects the user experience, especially in applications with high real-time requirements, resulting in asynchrony.
By dynamically determining the reading threshold through real-time statistics of the writing progress and combining the input and output frame rates, video data can be read and used for fusion without waiting for the complete frame buffer. The reading coefficients, which are linear combinations of integers, are used in the FPGA chip to achieve low-latency reading, ensuring the continuity and integrity of the picture.
It achieves a latency of less than one frame from input to fused output for multiple asynchronous video streams, ensuring the continuity and integrity of the images in each layer, improving the user experience, and reducing chip costs.
Smart Images

Figure CN123123520A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video processing technology, and in particular to a method, system, device, medium, and product for multi-video frame fusion display. Background Technology
[0002] When performing multi-layer fusion display, video processing equipment needs to merge multiple input video sources into a single video feed before outputting it to the display device. With the increasing demands for real-time performance in applications such as video conferencing, real-time monitoring, and in-vehicle displays, the end-to-end latency of video from input to display has become a key indicator for measuring device performance.
[0003] Currently, when performing multi-channel asynchronous video layer fusion, existing technologies typically employ the method of first completely caching at least one frame from each input video source, then reading the video data from each source from the cache according to the local output timing, performing layer fusion processing, and finally outputting the fused image to ensure the integrity and continuity of the fused image.
[0004] However, since the timing of each video source and the local output is asynchronous, the existing technology requires that each video source must have a complete frame of data in its corresponding buffer before it can be read and participate in the fusion. Therefore, the delay from video input to layer fusion output is at least one frame. In some application scenarios with high real-time requirements, this delay can cause a perceptible asynchrony between the local video output and the remote audio or control commands, affecting the user experience. Summary of the Invention
[0005] This invention provides a method, system, device, medium, and product for multi-video frame fusion display, which can solve the problem of how to reduce the delay from video input to layer fusion output.
[0006] This invention provides a method for multi-video frame fusion display, comprising: Acquire at least two video data signals, including a first data signal and a second data signal; For the first data signal, the first data signal is cached sequentially into different cache areas of the memory in units of frames; Get the writing progress of the current frame in the current buffer area, and get the input frame rate of the first data signal and the output frame rate of the video output device; determine the reading threshold based on the input frame rate and the output frame rate, wherein the reading threshold is less than the data volume of a complete video frame; At the beginning of the current output frame, the write progress is compared with the read threshold, and the first read data of the current output frame is determined in the memory based on the comparison result. The first read data and the second read data read from the second data signal are fused to obtain fused video data, and the fused video data is output to the video display device for display through the video output device.
[0007] This invention, through real-time statistics of the writing progress and dynamic determination of the reading threshold in conjunction with the input and output frame rates, enables video data to be read and participate in fusion without waiting for a complete frame buffer. This achieves a delay of less than one frame from input to fused output for multiple asynchronous videos, while ensuring the continuity and integrity of the images in each layer after fusion. Furthermore, it supports different input and output frame rates for each video channel, thereby improving the user experience.
[0008] Furthermore, based on the comparison result, the first read data for the current output frame is determined in the memory, specifically as follows: If the comparison result shows that the write progress is greater than or equal to the read threshold, then the first read data is obtained from the current cache area; Alternatively, if the comparison result shows that the write progress is less than the read threshold, then the first read data is obtained from the previous cache area.
[0009] By comparing the write progress with the read threshold, the current write frame is read when the write progress reaches the threshold to achieve low-latency reading, and the previous complete frame is read when the write progress has not reached the threshold to ensure that the image is not torn. This reduces latency while ensuring the integrity of each layer of the image without cross-frame tearing.
[0010] Furthermore, based on the input frame rate and the output frame rate, a reading threshold is determined, specifically as follows: Calculate the ratio of the input frame rate to the output frame rate to obtain the frame rate ratio; The reading coefficient is determined based on the frame rate ratio and the preset mapping table; The read threshold is determined based on the read coefficient and the effective image pixels of the current write frame.
[0011] By calculating the input-output frame rate ratio and then querying a preset mapping table based on the frame rate ratio to determine the reading coefficient, and finally combining this with the effective image pixels to determine the reading threshold, the threshold can be dynamically and adaptively adjusted according to the difference between the input and output frame rates, thus matching the low-latency reading requirements under different frame rate ratio conditions.
[0012] Furthermore, the reading coefficient is A linear combination of integers, where N is a positive integer.
[0013] This is achieved by limiting the reading coefficient to... The method involves linear combinations of integers. When applied to binary chips such as FPGA (Field-Programmable Gate Array) chips, the threshold calculation can be performed directly through shift and addition operations without the need for multipliers or dividers, saving multiplier and logic resources in the chip and reducing chip cost.
[0014] Furthermore, the effective image pixels are calculated based on the first horizontal resolution and the first vertical resolution, wherein the first horizontal resolution and the first vertical resolution are video control signals of the first data signal.
[0015] This method calculates effective image pixels by directly parsing the horizontal and vertical resolutions from the video control signal of the first data signal, ensuring that the threshold calculation is precisely matched with the input image size and guaranteeing the accuracy of the reading and judgment.
[0016] Further, the input frame rate of the first data signal is obtained, specifically as follows: Obtain the first clock frequency of the first data signal; The input frame rate is obtained by calculating the ratio of the first clock frequency to the effective image pixels.
[0017] By directly obtaining the clock frequency from the video control signal and calculating the input frame rate, there is no need to periodically monitor the number of times the input frames are transmitted within one second. The frame rate ratio can be obtained without waiting for one second, which improves the real-time performance of threshold adjustment when the frame rate changes.
[0018] Another embodiment of the present invention provides a multi-video frame fusion display system, including: a signal acquisition module, a buffer module, a threshold determination module, a decision module, and a fusion output module; The signal acquisition module is used to acquire at least two video data signals, including a first data signal and a second data signal. The caching module is used to cache the first data signal in frames sequentially to different cache areas of the memory. The threshold determination module is used to obtain the writing progress of the current frame in the current buffer area, and to obtain the input frame rate of the first data signal and the output frame rate of the video output device; and to determine the reading threshold based on the input frame rate and the output frame rate, wherein the reading threshold is less than the data volume of a complete video frame; The decision module is used to compare the write progress with the read threshold at the beginning of the current output frame, and determine the first read data of the current output frame in the memory based on the comparison result. The fusion output module is used to fuse the first read data and the second read data read from the second data signal to obtain fused video data, and then output the fused video data to the video display device for display through the video output device.
[0019] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the steps of the multi-video frame fusion display method of the present invention.
[0020] Another embodiment of the present invention provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform the steps of the multi-video frame fusion display method of the present invention.
[0021] Another embodiment of the present invention also provides a computer program product, which is stored in a storage medium and is executed by at least one processor using the steps of the multi-video frame fusion display method of the present invention. Attached Figure Description
[0022] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0023] Figure 1 This is a flowchart illustrating a multi-video frame fusion display method provided in an embodiment of the present invention; Figure 2 This is a device connection diagram provided in an embodiment of the present invention; Figure 3 This is a block diagram of a multi-layer fusion processing method provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a multi-video frame fusion display system provided in an embodiment of the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.
[0026] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.
[0027] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0028] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.
[0029] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).
[0030] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.
[0031] See Figure 1To address the issue of reducing latency in video processing from input to layer fusion output, this invention provides a multi-video frame fusion display method. To further illustrate the application scenario, this embodiment provides a method such as... Figure 2 The device connection diagram shown is from Figure 2 As can be seen: Video transmitting device 1 receives HDMI (High-Definition Multimedia Interface) video data from the video source input, and then sends the HDMI video data to the video receiving and processing device via Ethernet / fiber optic cable. Video transmitting device 2 does the same. The video receiving and processing device receives the video streams transmitted by the two video transmitting devices via two Ethernet / fiber optic cables. After unpacking the two Ethernet / fiber optic video streams, it obtains two independent video data streams, which are defined as the first data signal and the second data signal, respectively. The first data signal and the second data signal are independently buffered into ddr (Double Data Rate) memory chips (i.e., the memory referred to in this invention). Then, the FPGA generates the clock and timing required for video output to the display through the local crystal oscillator and internal PLL (Phase-Locked Loop). The first data signal and the second data signal in the ddr are read out and layered. After the fusion processing, they are sent to the display device via the HDMI interface.
[0032] This invention is implemented in the FPGA chip of a video receiving and processing device, and specifically includes the following steps: Step 101: Acquire at least two video data signals, including a first data signal and a second data signal.
[0033] In this step, the FPGA receives the video stream data transmitted via Ethernet / fiber optic cable, obtaining Video 1 and Video 2. Video 1 and Video 2 are two independent video streams. However, the low-latency processing of Video 2 in this invention is consistent with that of Video 1. Taking the processing of Video 1 as an example, the video control signal and video data signal are decompressed from Video 1. The video control signal includes the clock frequency, horizontal resolution, and vertical resolution of the video source, which are represented by src_freq, src_hactive, and src_vactive, respectively.
[0034] Step 102: For the first data signal, the first data signal is cached sequentially into different cache areas of the memory in units of frames.
[0035] In this step, the FPGA unpacks the first data signal and writes it sequentially into multiple buffer areas of the DDR, frame by frame, according to the frame number. Each buffer area stores one complete frame of video data. During the writing process, each video frame is written cyclically between the multiple buffer areas according to the frame number; that is, after the current frame is written, the next frame is written to the next buffer area. When all buffer areas have been written, the process returns to the first buffer area to continue writing.
[0036] Preferably, the DDR can cache a maximum of 4 frames. For example, video frame number 1 (represented by frame1) is cached in DDR space 1 (represented by ddr_page1), video frame number 2 (represented by frame2) is cached in DDR space 2 (represented by ddr_page2), video frame number 3 (represented by frame3) is cached in DDR space 3 (represented by ddr_page3), video frame number 4 (represented by frame4) is cached in DDR space 4 (represented by ddr_page4), and video frame number 5 (represented by frame5) is cached back in DDR space 1 (represented by ddr_page1), and so on.
[0037] Step 103: Obtain the writing progress of the current frame in the current buffer area, and obtain the input frame rate of the first data signal and the output frame rate of the video output device; determine the reading threshold based on the input frame rate and the output frame rate, wherein the reading threshold is less than the data volume of a complete video frame.
[0038] In this step, a register wr_cnt is defined to represent the count of the amount of data written to the ddr for the current input video frame number. The FPGA counts the amount of data written to the current buffer area for the current frame in real time to obtain the writing progress.
[0039] The FPGA also acquires the input frame rate of the first data signal and the output frame rate of the video output device to the display device. Based on the relative magnitude of the input and output frame rates, a read threshold smaller than the amount of data in a complete video frame is determined. Specifically, the faster the input frame rate is relative to the output frame rate, the smaller the read threshold is set; conversely, the slower the input frame rate is relative to the output frame rate, the larger the read threshold is set. A register `rd_threshold` is defined to represent the read threshold, used to determine whether to read the current written frame from the DDR for output display or to read the previous written frame for output display.
[0040] Step 104: At the beginning of the current output frame, compare the write progress with the read threshold, and determine the first read data of the current output frame in the memory based on the comparison result.
[0041] In this step, whenever the video output timing is at the start of a frame (such as the falling edge of VSYNC), the FPGA compares the writing progress of the current writing frame with the reading threshold determined in step 103, that is, judges the magnitude relationship between the data quantity wr_cnt and the reading threshold rd_threshold.
[0042] When wr_cnt>=rd_threshold, the DDR read space number is controlled to be consistent with the current DDR write space number. For example, if the current video frame write space is ddr_page3, the DDR read space is also controlled to be ddr_page3. At this time, it can be ensured that the video output delay is less than one frame delay compared with the input source, and no cross-frame tearing occurs when reading the video frame.
[0043] When wr_cnt<rd_threshold, the DDR read space number is controlled to be the previous space number of the current DDR write space number. For example, if the current video frame write space is ddr_page3, the DDR read space is controlled to be ddr_page2. At this time, it can be ensured that no cross-frame tearing occurs in the read video frame.
[0044] Step 105: Fuse the first read data and the second read data read from the second data signal to obtain fused video data, and output the fused video data to a video display device for display through a video output device.
[0045] In this step, as Figure 3 shown in the multi-layer fusion processing block diagram, when performing layer fusion on the first read data and the second read data, the pixel data of video 1 after being read from the DDR is defined as pix1, and the pixel data of video 2 after being read from the DDR is defined as pix2. Two layer transparency registers alpha1 and alpha2 are defined, which respectively represent the transparency of layer 1 and layer 2 during fusion. The smaller the value, the more transparent the layer, and alpha1+alpha2<=256. The pixel after layer fusion is defined as pix_o, and the layer fusion processing formula is pix_o=(pix1*alpha1+pix2*alpha2) / 256. The image pixel data pix_o obtained after fusing the layers of video 1 and video 2 is output to the display device.
[0046] It should be noted that the phase, frame rate ratio and reading threshold of the two paths of videos are not necessarily the same. However, since the reading operation is driven by the same clock source, when the output timing scans to the Y-th column of the X-th row of the screen, the FPGA will simultaneously take out the pixel at the Y-th column of the X-th row from the buffer area of video 1 and the buffer area of video 2. Even if the video input frame rate (DDR writing) is four times larger than the video output frame rate (DDR reading), the frame being read by the read pointer will never be overwritten when writing a new frame, thus ensuring that the output picture is always complete and tear-free.
[0047] As an example of an embodiment of the present invention, the effective image pixels are calculated based on the first horizontal resolution and the first vertical resolution, wherein the first horizontal resolution and the first vertical resolution are video control signals of the first data signal.
[0048] In this embodiment, the FPGA parses the first horizontal resolution `src_hactive` and the first vertical resolution `src_vactive` from the video control signal of the first data signal. A register `src_total` is defined to represent the number of effective pixels in a frame of the input source image, `src_total = src_hactive * src_vactive`, thus obtaining the total number of effective pixels in a frame of the image, which is used for subsequent reading threshold calculation. Since the horizontal and vertical resolutions are directly derived from the video control signal, the effective image pixels precisely match the input image size, requiring no additional measurement or estimation. This ensures that the threshold calculation is consistent with the actual size of the input image, avoiding reading judgment deviations caused by pixel mismatch.
[0049] As an example of an embodiment of the present invention, the input frame rate of the first data signal is obtained as follows: Obtain the first clock frequency of the first data signal; The input frame rate is obtained by calculating the ratio of the first clock frequency to the effective image pixels.
[0050] In this embodiment, the FPGA obtains the first clock frequency src_freq from the video control signal of the first data signal. Since the video frame rate is equal to the clock frequency divided by the product of the horizontal resolution and the vertical resolution, the FPGA calculates the input frame rate based on the first clock frequency src_freq and the effective image pixels src_total, i.e., input frame rate = src_freq / src_total = src_freq / (src_hactive × src_vactive). Through this method, the FPGA directly parses parameters from the video control signal and calculates the input frame rate, without needing to periodically monitor the cumulative number of input frame transmissions within one second, and without waiting for one second to obtain the input frame rate.
[0051] Accordingly, the FPGA reads the output clock frequency dst_freq, output horizontal resolution dst_hactive, and output vertical resolution dst_vactive of the video output device from the local configuration register, and calculates the output frame rate = dst_freq / (dst_hactive × dst_vactive). The output frame rate is configured by the user through registers and is independent of the input frame rate.
[0052] As an example of an embodiment of the present invention, the reading coefficient is... is an integer linear combination, where N is a positive integer.
[0053] In this embodiment, as described above, all read coefficients are integer linear combinations, including 6.25%, 25%, 50%, and 75%, corresponding to 1 / 16, 1 / 4, 1 / 2, and 3 / 4 respectively. Therefore, when calculating using these parameters, it is not necessary to use the multiplier or divider of the FPGA, and the processing can be performed directly using shifts and additions, which saves the multiplier resources of the FPGA and a large amount of logic resources required for using the divider (the multiplier resources and logic resources of the FPGA are limited, and reducing various resources is equivalent to lowering the chip cost).
[0054] As an example of the embodiment of the present invention, determining the read threshold according to the input frame rate and the output frame rate is specifically as follows: Calculate the ratio of the input frame rate to the output frame rate to obtain the frame rate ratio; Determine the read coefficient according to the frame rate ratio and a preset mapping table; Determine the read threshold according to the read coefficient and the valid image pixels of the currently written frame.
[0055] In this embodiment, the FPGA calculates the frame rate ratio according to the input frame rate and the output frame rate. Here, a register ratio is defined to represent the frame rate ratio between the input source and the output, so ratio=(src_freq / (src_hactive*src_vactive)) / (dst_freq / (dst_hactive*dst_vactive))=(src_freq*(dst_hactive*dst_vactive)) / (dst_freq*(src_hactive*src_vactive)).
[0056] The FPGA queries the preset mapping table according to the calculated frame rate ratio, and determines the read coefficient corresponding to the frame rate ratio. In the preset mapping table, the correspondence between frame rate ratios and read coefficients is as follows: When ratio>1, it means that the input frame rate is greater than the output frame rate. At this time, the video output data rate is slower than the video source input data rate, that is, the DDR read rate is slower than the DDR write rate. At this time, the read coefficient is determined to be 6.25%, and the read threshold rd_threshold=src_total*6.25% (representing the data amount of 1 / 16 frame).
[0057] When 0.75<ratio<=1, it means that the input frame rate is less than the output frame rate. At this time, the read coefficient is determined to be 25%, and the read threshold rd_threshold=src_total*25%; When 0.5<ratio<=0.75, it means that the input frame rate is lower than the output frame rate. At this time, the read coefficient is determined to be 50%, and the read threshold rd_threshold=src_total*50%; When 0.25<ratio<=0.5, it means that the input frame rate is lower than the output frame rate. At this time, the read coefficient is determined to be 75%, and the read threshold rd_threshold=src_total*75%; When ratio<=0.25, it means that the input frame rate is lower than the output frame rate. At this time, the read coefficient is determined to be 100%, and the read threshold rd_threshold=src_total*100%.
[0058] It should be noted that the above read coefficients 6.25%, 25%, 50%, and 75% are all integer linear combinations of 1 / 2^N, and the FPGA implements the above multiplication calculation by shift operation: src_total*6.25%=src_total*(1 / 16). In FPGA logic, it is only necessary to shift src_total right by 4 bits, which is expressed as src_total>>4; src_total*25%=src_total*(1 / 4). In FPGA logic, it is only necessary to shift src_total right by 2 bits, which is expressed as src_total>>2; src_total*50%=src_total*(1 / 2). In FPGA logic, it is only necessary to shift src_total right by 1 bit, which is expressed as src_total>>1; src_total*75%=src_total*(1 / 2)+src_total*(1 / 4). In FPGA logic, it is only necessary to shift src_total right by 1 bit and then add the result of shifting src_total right by 2 bits, which is expressed as (src_total>>1+src_total>>2).
[0059] As an example of the embodiment of the present invention, determining the first read data serving as the current output frame in the memory according to the comparison result is specifically: If the comparison result indicates that the writing progress is greater than or equal to the read threshold, the first read data is obtained by reading from the current buffer area; Alternatively, if the comparison result indicates that the writing progress is less than the read threshold, the first read data is obtained by reading from the previous buffer area.
[0060] In this embodiment, 0.5<ratio<=0.75 is taken as an example: ratio represents the frame rate ratio between the input source and the output. If the output frame rate is 100hz, the frame rate range of the input source is 50hz~75hz, that is, the time for the output to scan one frame is 1 / 100(hz)=10ms, and the time for the input source to scan one frame is 1 / 50~1 / 75(hz)=13ms~20ms, which means the rate at which the output end outputs pixels (DDR reading) is twice the rate at which the input source inputs pixels (DDR writing). At this time, rd_threshold=src_total*50% is set, that is, the threshold is set to half of the input resolution. At the start of each output frame (VSYNC falling edge), that is, before reading the first pixel data of each frame from the DDR, wr_cnt (the amount of data that has been written to the DDR in the current input frame) is judged.
[0061] If wr_cnt>=rd_threshold, that is, the DDR write amount of the current input frame has exceeded half a frame, even if the current DDR read rate is twice the DDR write rate (the output scanning speed is twice the input scanning speed), it can be ensured that when the DDR read scans to the data of a certain row and a certain column, the data of that row and that column at the input end has definitely been written into the DDR. Therefore, reading the current input frame can also ensure that a complete video frame will not have cross-frame tearing, and the minimum delay in this case is 50% of the frame delay time.
[0062] If wr_cnt<rd_threshold, that is, the DDR write amount of the current input frame is less than the data amount of half a frame, since the current DDR read rate is twice the DDR write rate, when the DDR read scans to the data of a certain row and a certain column, the data of that row and that column at the input end may not have been scanned yet (that is, not written into the DDR). Therefore, at this time, only the previous frame of the current input frame can be read. Since the previous input frame has been completely written into the DDR, only reading the previous frame can ensure that the read frame is a complete frame without tearing.
[0063] As Figure 4 shown, on the basis of the above method embodiment, an embodiment of the present invention provides a multi-video frame fusion display system 400, comprising: a signal acquisition module 401, a cache module 402, a threshold determination module 403, a decision module 404, and a fusion output module 405; The signal acquisition module 401 is configured to acquire at least two video data signals, where the video data signals include a first data signal and a second data signal; The cache module 402 is configured to, for the first data signal, sequentially cache the first data signal in different cache areas of the memory on a frame basis; The threshold determination module 403 is used to obtain the writing progress of the current frame in the current buffer area, and to obtain the input frame rate of the first data signal and the output frame rate of the video output device; and to determine the reading threshold based on the input frame rate and the output frame rate, wherein the reading threshold is less than the data volume of the complete video frame; The decision module 404 is used to compare the write progress with the read threshold at the beginning of the current output frame, and determine the first read data of the current output frame in the memory based on the comparison result. The fusion output module 405 is used to fuse the first read data and the second read data read from the second data signal to obtain fused video data, and output the fused video data to the video display device for display through the video output device.
[0064] Furthermore, decision module 404 is also used for: If the comparison result shows that the write progress is greater than or equal to the read threshold, then the first read data is obtained from the current cache area; Alternatively, if the comparison result shows that the write progress is less than the read threshold, then the first read data is obtained from the previous cache area.
[0065] Furthermore, the threshold determination module 403 is also used for: Calculate the ratio of the input frame rate to the output frame rate to obtain the frame rate ratio; The reading coefficient is determined based on the frame rate ratio and the preset mapping table; The read threshold is determined based on the read coefficient and the effective image pixels of the current write frame.
[0066] Furthermore, the reading coefficient is A linear combination of integers, where N is a positive integer.
[0067] Furthermore, the effective image pixels are calculated based on the first horizontal resolution and the first vertical resolution, wherein the first horizontal resolution and the first vertical resolution are video control signals of the first data signal.
[0068] Furthermore, the threshold determination module 403 is also used for: Obtain the first clock frequency of the first data signal; The input frame rate is obtained by calculating the ratio of the first clock frequency to the effective image pixels.
[0069] It is understood that the above system embodiments correspond to the method embodiments of the present invention, and can implement all the implementation methods of any of the above method embodiments of the present invention, which will not be repeated here.
[0070] It should be noted that the system embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.
[0071] Based on the embodiments of the above-described multi-video frame fusion display method, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the multi-video frame fusion display method of any of the above embodiments of the present invention, the steps of which include: Acquire at least two video data signals, including a first data signal and a second data signal; For the first data signal, the first data signal is cached sequentially into different cache areas of the memory in units of frames; Get the writing progress of the current frame in the current buffer area, and get the input frame rate of the first data signal and the output frame rate of the video output device; determine the reading threshold based on the input frame rate and the output frame rate, wherein the reading threshold is less than the data volume of a complete video frame; At the beginning of the current output frame, the write progress is compared with the read threshold, and the first read data of the current output frame is determined in the memory based on the comparison result. The first read data and the second read data read from the second data signal are fused to obtain fused video data, and the fused video data is output to the video display device for display through the video output device.
[0072] Embodiments of the present invention Furthermore, based on the comparison result, the first read data for the current output frame is determined in the memory, specifically as follows: If the comparison result shows that the write progress is greater than or equal to the read threshold, then the first read data is obtained from the current cache area; Alternatively, if the comparison result shows that the write progress is less than the read threshold, then the first read data is obtained from the previous cache area.
[0073] Furthermore, based on the input frame rate and the output frame rate, a reading threshold is determined, specifically as follows: Calculate the ratio of the input frame rate to the output frame rate to obtain the frame rate ratio; The reading coefficient is determined based on the frame rate ratio and the preset mapping table; The read threshold is determined based on the read coefficient and the effective image pixels of the current write frame.
[0074] Furthermore, the reading coefficient is A linear combination of integers, where N is a positive integer.
[0075] Furthermore, the effective image pixels are calculated based on the first horizontal resolution and the first vertical resolution, wherein the first horizontal resolution and the first vertical resolution are video control signals of the first data signal.
[0076] Further, the input frame rate of the first data signal is obtained, specifically as follows: Obtain the first clock frequency of the first data signal; The input frame rate is obtained by calculating the ratio of the first clock frequency to the effective image pixels.
[0077] For example, in this embodiment, the computer program can be divided into one or more modules, one or more modules are stored in memory and executed by a processor to complete the present invention. One or more module elements can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a terminal device.
[0078] Terminal devices can be computing devices such as desktop computers, laptops, handheld computers, and cloud servers. Terminal devices may include, but are not limited to, processors and memory.
[0079] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device through various interfaces and lines.
[0080] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium, including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the multi-video frame fusion display method of any of the above-described method embodiments of the present invention, the steps of which include: Acquire at least two video data signals, including a first data signal and a second data signal; For the first data signal, the first data signal is cached sequentially into different cache areas of the memory in units of frames; Get the writing progress of the current frame in the current buffer area, and get the input frame rate of the first data signal and the output frame rate of the video output device; determine the reading threshold based on the input frame rate and the output frame rate, wherein the reading threshold is less than the data volume of a complete video frame; At the beginning of the current output frame, the write progress is compared with the read threshold, and the first read data of the current output frame is determined in the memory based on the comparison result. The first read data and the second read data read from the second data signal are fused to obtain fused video data, and the fused video data is output to the video display device for display through the video output device.
[0081] Embodiments of the present invention Furthermore, based on the comparison result, the first read data for the current output frame is determined in the memory, specifically as follows: If the comparison result shows that the write progress is greater than or equal to the read threshold, then the first read data is obtained from the current cache area; Alternatively, if the comparison result shows that the write progress is less than the read threshold, then the first read data is obtained from the previous cache area.
[0082] Furthermore, based on the input frame rate and the output frame rate, a reading threshold is determined, specifically as follows: Calculate the ratio of the input frame rate to the output frame rate to obtain the frame rate ratio; The reading coefficient is determined based on the frame rate ratio and the preset mapping table; The read threshold is determined based on the read coefficient and the effective image pixels of the current write frame.
[0083] Furthermore, the reading coefficient is A linear combination of integers, where N is a positive integer.
[0084] Furthermore, the effective image pixels are calculated based on the first horizontal resolution and the first vertical resolution, wherein the first horizontal resolution and the first vertical resolution are video control signals of the first data signal.
[0085] Further, the input frame rate of the first data signal is obtained, specifically as follows: Obtain the first clock frequency of the first data signal; The input frame rate is obtained by calculating the ratio of the first clock frequency to the effective image pixels.
[0086] Based on the above-described method embodiments, this invention further provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the multi-video frame fusion display method of any of the above-described method embodiments, the steps of which include: Acquire at least two video data signals, including a first data signal and a second data signal; For the first data signal, the first data signal is cached sequentially into different cache areas of the memory in units of frames; Get the writing progress of the current frame in the current buffer area, and get the input frame rate of the first data signal and the output frame rate of the video output device; determine the reading threshold based on the input frame rate and the output frame rate, wherein the reading threshold is less than the data volume of a complete video frame; At the beginning of the current output frame, the write progress is compared with the read threshold, and the first read data of the current output frame is determined in the memory based on the comparison result. The first read data and the second read data read from the second data signal are fused to obtain fused video data, and the fused video data is output to the video display device for display through the video output device.
[0087] Embodiments of the present invention Furthermore, based on the comparison result, the first read data for the current output frame is determined in the memory, specifically as follows: If the comparison result shows that the write progress is greater than or equal to the read threshold, then the first read data is obtained from the current cache area; Alternatively, if the comparison result shows that the write progress is less than the read threshold, then the first read data is obtained from the previous cache area.
[0088] Furthermore, based on the input frame rate and the output frame rate, a reading threshold is determined, specifically as follows: Calculate the ratio of the input frame rate to the output frame rate to obtain the frame rate ratio; The reading coefficient is determined based on the frame rate ratio and the preset mapping table; The read threshold is determined based on the read coefficient and the effective image pixels of the current write frame.
[0089] Furthermore, the reading coefficient is A linear combination of integers, where N is a positive integer.
[0090] Furthermore, the effective image pixels are calculated based on the first horizontal resolution and the first vertical resolution, wherein the first horizontal resolution and the first vertical resolution are video control signals of the first data signal.
[0091] Further, the input frame rate of the first data signal is obtained, specifically as follows: Obtain the first clock frequency of the first data signal; The input frame rate is obtained by calculating the ratio of the first clock frequency to the effective image pixels.
[0092] The modules / units integrated into the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.
[0093] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. A method for multi-video frame fusion display, characterized in that, include: Acquire at least two video data signals, the video data signals including a first data signal and a second data signal; For the first data signal, the first data signal is cached sequentially in different cache areas of the memory in units of frames; Obtain the writing progress of the current frame in the current buffer area, and obtain the input frame rate of the first data signal and the output frame rate of the video output device; determine the reading threshold based on the input frame rate and the output frame rate, wherein the reading threshold is less than the data volume of a complete video frame; At the start of the current output frame, the write progress is compared with the read threshold, and the first read data of the current output frame is determined in the memory based on the comparison result. The first read data and the second read data read from the second data signal are fused to obtain fused video data, and the fused video data is output to the video display device for display through the video output device.
2. The multi-video frame fusion display method as described in claim 1, characterized in that, The step of determining the first read data of the current output frame in the memory based on the comparison result specifically involves: If the comparison result indicates that the write progress is greater than or equal to the read threshold, then the first read data is read from the current cache area; Alternatively, if the comparison result indicates that the write progress is less than the read threshold, then the first read data is read from the previous cache area.
3. The multi-video frame fusion display method as described in claim 1, characterized in that, The step of determining the reading threshold based on the input frame rate and the output frame rate specifically involves: Calculate the ratio of the input frame rate to the output frame rate to obtain the frame rate ratio; The reading coefficient is determined based on the frame rate ratio and the preset mapping table; The read threshold is determined based on the read coefficient and the effective image pixels of the current write frame.
4. The multi-video frame fusion display method as described in claim 3, characterized in that, The reading coefficient is A linear combination of integers, where N is a positive integer.
5. The multi-video frame fusion display method as described in claim 3, characterized in that, The effective image pixels are calculated based on the first horizontal resolution and the first vertical resolution, wherein the first horizontal resolution and the first vertical resolution are video control signals of the first data signal.
6. The multi-video frame fusion display method as described in claim 5, characterized in that, The specific steps for obtaining the input frame rate of the first data signal are as follows: Obtain the first clock frequency of the first data signal; The input frame rate is obtained by calculating the ratio of the first clock frequency to the effective image pixels.
7. A multi-video frame fusion display system, characterized in that, include: Signal acquisition module, buffer module, threshold determination module, decision module, and fusion output module; The signal acquisition module is used to acquire at least two video data signals, the video data signals including a first data signal and a second data signal; The caching module is used to cache the first data signal in frames sequentially to different cache areas of the memory. The threshold determination module is used to obtain the writing progress of the current write frame in the current buffer area, and to obtain the input frame rate of the first data signal and the output frame rate of the video output device; and to determine a reading threshold based on the input frame rate and the output frame rate, wherein the reading threshold is less than the data volume of a complete video frame; The decision module is used to compare the write progress with the read threshold at the start of the current output frame, and determine the first read data of the current output frame in the memory based on the comparison result. The fusion output module is used to fuse the first read data and the second read data read from the second data signal to obtain fused video data, and output the fused video data to the video display device for display through the video output device.
8. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the multi-video frame fusion display method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the multi-video frame fusion display method as described in any one of claims 1-6.
10. A computer program product, characterized in that, include: Computer instructions, when executed by a processor, implement the multi-video frame fusion display method as described in any one of claims 1-6.
