A video processing method and device for preventing frame loss

By introducing frame rate reduction units and FPGA detection and eliminating duplicate frames in the video processing SoC chip, and dynamically adjusting the frame rate, the frame drop problem in the video acquisition and output process is solved, the stability and integrity of the video stream are improved, and the improvement of the stability and integrity of the video stream is improved. User experience.

CN114493982BActive Publication Date: 2024-11-22SHENZHEN OCAM TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210144317.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-17
Publication Date
2024-11-22
Estimated Expiration
2042-02-17

AI Technical Summary

Technical Problem

Existing video processing SoC chips are prone to frame loss during video acquisition and output, resulting in stuttering of video display, especially in high-speed motion scenarios that affect the viewing effect and may lose important video data.

Method used

By introducing a frame rate reduction unit between the video input unit and the video output unit, repeat frame rate dynamic adjustment is performed when the CPU responds to interruption, the stability of the video stream is ensured, and repeated frame detection and removal are achieved using FPGA. , the frame rate of the video output unit is configured to be greater than the frame rate of the input unit to eliminate frame rate deviation.

Benefits of technology

It effectively reduces video frame loss, improves the stability and integrity of video acquisition, avoids the loss of video data, and improves user experience, especially in high-speed motion scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114493982B_ABST
    Figure CN114493982B_ABST
Patent Text Reader

Abstract

The present application provides a video processing method for preventing frame loss, comprising: a video input unit captures images and stores them into a memory in units of frames at a first frame rate; the video input unit generates a first hardware interrupt to a CPU; the CPU responds to the first hardware interrupt, and the corresponding first interrupt response function puts the memory address information of a new frame of image into a buffer queue corresponding to the video output unit; after the video output unit completes the transmission of a frame of image at a second frame rate, a second hardware interrupt is generated to the CPU; the CPU responds to the second hardware interrupt, and the corresponding second interrupt response function takes out the memory address information of a new frame of image from the buffer queue and passes it to the video output unit; the video output unit reads a frame of image starting from the memory address and sends it at a second frame rate; the second frame rate of the video output unit is greater than the first frame rate of the video input unit; the method also comprises a duplicate frame detection step and a duplicate frame removal step; the duplicate frame removal step removes the duplicate frames detected in the duplicate frame detection step and restores them to the same frame rate as the video input unit of the video processing SoC chip.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method and device applied to industries such as cameras, video recorders, and video matrices. Background Art

[0002] Video processing SoC chip is a chip that integrates CPU and hardware video processing unit for video processing. It has the advantages of high integration, low power consumption and low price. Most of the video devices such as cameras, video recorders, video matrices, etc. on the market are developed based on video processing SoC chips.

[0003] Figure 1 It is a typical camera solution based on video processing SoC chip. Camera hardware is generally composed of optical lens (Lens), image photoelectric sensor (Image Sensor), and video processing SoC chip. The video processing SoC chip is connected to DDR memory device externally, and internally integrates standard video interface chips such as HDMI / DP, or requires an external video interface chip (Video Interface Chip). The image photoelectric sensor outputs raw video data (raw data), which is collected and processed by the video processing SoC chip for image signal processing (ISP), and finally generates standard video interface signal output such as HDM / DP / SDI / VGA.

[0004] Figure 2 It is a typical video matrix device solution based on video SoC chip. The internal hardware of the video matrix is ​​generally composed of one or more video processing SoC chips, which collects and receives multiple standard video signal inputs, performs video processing such as scaling and format conversion inside the video SoC chip, and finally generates standard video interface signal outputs such as HDM / DP / SDI / VGA.

[0005] The video processing SoC chip is generally composed of a CPU, a video input unit, a video output unit, and a coprocessor, which includes a GPU (Graphic Process Unit), a video codec (Encoder, Decoder), an image scaling unit (Scaler), etc. In addition, the components inside the video processing SoC chip share a memory controller to access the external DDR memory. Figure 3A typical video processing SoC chip internal components share a memory controller to access the external DDR memory. The video input unit collects video data from the standard video transmission interface (MIPI interface, LVDS interface, parallel port, etc.), performs image signal processing, and then stores the video data in the DDR memory to facilitate the coprocessor to perform the next step of image processing; the video output unit reads the video data in the DDR memory and sends it out according to the standard video transmission interface (HDMI interface, DP interface, parallel port, etc.).

[0006] However, in actual applications, the above scheme will randomly cause frame loss, resulting in phenomena such as freezes during video display, seriously affecting the user experience. Even for some scenarios with high requirements for video recording, frame loss will make the video lose its value as evidence, making the scheme unsuitable for such scenarios.

[0007] For example, when using a camera or video recorder to shoot a video, in an application scenario where the target object moves a lot (such as shooting the moment a bullet is fired, a vehicle speeding on a highway, a ball moving at high speed in a sports event, etc.), if there is frame loss in the video capture content, it will affect the viewing effect and even cause the loss of important video data. Cameras and video recorders are the first link in video capture. Generally, a video capture system may also require transmission links such as a video matrix before it is finally input to the display device. If video frames are lost in any link, it will cause frame loss in the entire system. How to ensure that there is no frame loss in video capture puts higher demands on video equipment. There are corresponding detection methods for frame loss in camera equipment in the industry. For example, patent CN111405272A "A camera frame loss and repeated frame detection device and method" and patent CN105611291B "Method and device for adding marking information to video frames and detecting frame loss" both propose methods for detecting video frame loss.

[0008] Therefore, reducing frame loss while being compatible with the existing technical architecture is a technical problem that needs to be solved urgently. Summary of the invention

[0009] Currently, many video devices such as cameras, video recorders, and video matrices on the market use video processing SoC chips because SoC chips have advantages that other solutions cannot match, such as high integration, low power consumption, and low price. However, for many video processing SoC chips, video input acquisition and video output processing require the CPU to manage the video frame buffer cache. When the CPU occupancy rate is high, there is a risk of untimely processing, which ultimately leads to video frame loss. In addition, the video input frame rate and video output frame rate of the video processing SoC chip are configured to be the same, but due to different clock sources or slight deviations in their respective clock phase-locked loops (PLLs), over time, the total number of frames collected by the video input may be more than the total number of frames of the video output (for example, the configured acquisition frame rate and output frame rate are both 30fps, but in fact the average input frame rate is 30.00000fps, and the average output frame rate is 29.99999fps, so an average of 100,000 frames accumulates to 1 frame). When the accumulated number exceeds the maximum capacity of the video buffer, frame loss will occur.

[0010] To address this problem, the present application proposes a video processing method and device for preventing frame loss, which can effectively solve the problem of frame loss.

[0011] The present application provides a video processing method for preventing frame loss, comprising:

[0012] Step S1: The video input unit captures images and stores them in memory in units of frames at a first frame rate;

[0013] Step S2: the video input unit generates a first hardware interrupt to the CPU;

[0014] Step S3: The CPU responds to the first hardware interrupt, and the corresponding first interrupt response function puts the memory address information of the image stored in the memory in step S1 into the tail of the buffer queue corresponding to the video output unit;

[0015] Step S4: after the video output unit completes transmission of data corresponding to one frame of image at the second frame rate, a second hardware interrupt is generated to the CPU;

[0016] Step S5: the CPU responds to the second hardware interrupt, and the corresponding second interrupt response function takes out the memory address information of the new frame of image from the head of the buffer queue and transmits it to the video output unit, and updates the memory address information;

[0017] Step S6: the video output unit reads data corresponding to one frame of image from the memory starting from the memory address, and sends it to the frame rate reduction unit at the second frame rate;

[0018] Step S7: the frame rate reduction unit receives the image data of the video output unit, detects and removes duplicate frames, and generates a standard video stream with a first frame rate;

[0019] The second frame rate of the video output unit is greater than the first frame rate of the video input unit.

[0020] Preferably, step S3 further comprises: after multi-layer pipeline control, finally adding the display buffer receiving queue created by the video output unit driver software.

[0021] Preferably, the ratio of the second frame rate to the first frame rate is: second frame rate / first frame rate ≥ 1.2.

[0022] Preferably, the first frame rate is 50fps and the second frame rate is 60fps.

[0023] Preferably, when the cache queue length is 1 frame, the memory address of the video output unit is not updated; if it is not in the blanking period of video transmission, the memory address of the video output unit is not updated either.

[0024] Furthermore, the step of detecting and eliminating duplicate frames specifically includes:

[0025] The video input unit embeds an ID number at a first fixed position of a captured frame of image, so as to retain the ID number when the video output unit sends the frame of image to the frame rate reduction unit processing module;

[0026] The frame rate reduction unit detecting repeated frames includes: the frame rate reduction unit parsing the embedded ID serial number from the one frame of image, and detecting whether the one frame of image is a repeated frame according to the repeatability of the ID serial number;

[0027] or,

[0028] The image content of each frame is accumulated and calculated to obtain a checksum. When the checksums are equal, it is considered a "duplicate frame".

[0029] Preferably, the first fixed position is the first line of the image content.

[0030] Preferably, the frame rate reduction unit processing module is implemented using FPGA.

[0031] Preferably, the elimination step in the detecting and eliminating duplicate frames refers to removing the duplicate frames detected in the detecting step and restoring the frame rate to the same first frame rate as the video input unit.

[0032] The video data after the step of detecting and eliminating duplicate frames will be stored in the memory corresponding to the frame rate reduction unit processing module.

[0033] Due to the reason that the video processing SoC has untimely interrupt response, the cycle in which the frame rate reduction unit processing module receives video frames may be occasionally slower or faster.

[0034] To solve the problem of occasional frame rate instability in the above-mentioned video data, a frame buffer step S8 may further be included. The step S8 means: starting to send after receiving n complete frames of images; where n is a natural number greater than 2.

[0035] Preferably, the value of n is 2.

[0036] Furthermore, to solve the problem that there may be a slight deviation between the frame rate collected by the video processing SoC chip and the frame rate output by the video output interface module of the FPGA (for example, the frame rate collected by the video processing SoC chip is 50.0000001fps, while the frame rate output by the FPGA chip is 50.0000000), a frame rate dynamic adjustment step S9 is further included, specifically including:

[0037] Step S9-1: Count the number of video frames output by the FPGA duplicate frame removal module within a specific period, and calculate the frame rate fps_input;

[0038] Step S9-2: Count the number of video frames fps_output sent by the FPGA video output interface module within a specific period, and calculate the frame rate;

[0039] Step S9-3: If fps_input>fps_output, then dynamically increase the pixel clock of fps_output within the PPM range allowed by the standard video output interface; conversely, if fps_input<fps_output, then decrease the pixel clock of fps_output, so as to achieve the effect that the FPGA video output module dynamically follows the video input frame rate of the video processing SoC chip macroscopically.

[0040] It should be noted that the standard video output interface allows a certain clock deviation jitter of PPM (part per milinon). The frame rate collected by the video processing SoC chip and the frame rate sent by the FPGA chip video output module both need to meet the standard interface timing clock deviation jitter range, that is, the clock jitter floating range is not very large.

[0041] Furthermore, the present application also provides a video processing device for preventing frame loss, including:

[0042] Video input module: Collect images and store them in memory in units of frames at a first frame rate;

[0043] Video output module: Read 1 frame of image from memory and send it at a second frame rate;

[0044] The second frame rate is greater than the first frame rate.

[0045] Preferably, it also includes a frame rate reduction unit module, which specifically includes a duplicate frame detection module and a duplicate frame elimination module.

[0046] Preferably, a frame buffer module is also included.

[0047] Preferably, a frame rate dynamic adjustment module is also included.

[0048] Preferably, the frame rate dynamic adjustment module counts the video frame rate after the FPGA repeated frame rate is eliminated and the output frame rate of the FPGA video output module, and dynamically adjusts the pixel clock of the FPGA video output module according to the interpolation of the frame rate, so that the frame rate of the FPGA video output module dynamically tracks the frame rate collected by the video input module of the video processing SoC chip.

[0049] The anti-frame loss video processing method can be implemented by using an anti-frame loss video processing device.

[0050] When the configured video output frame rate is greater than the video input frame rate, the video output unit will generate a large number of "duplicate frames" (even if the CPU software interrupt responds in time), but no "frame loss" will occur. This is mainly because when 1 frame of buffer accumulation occurs, the output rate is greater than the input rate, so the accumulated buffer can be digested within a certain period of time, and it will not continue to accumulate and cause the buffer queue to burst.

[0051] In the frame buffering step, although the FPGA delays the output of 2 frames, the overall delay does not increase. The reason is that the video output unit of the video processing SoC chip also has a display receiving buffer queue, but because the configured frame rate of the video output unit is greater than the frame rate of the video input unit, the display receiving buffer queue of the video output unit usually does not have video frame accumulation; compared with the existing solution in which the video output unit of the video processing SoC chip is equal to the video input unit, the video output unit generally needs to be configured with a buffer queue of 2 or 3 frames. If accumulation occurs, it will cause delay. That is, with the new solution, the delay of the video processing SoC chip itself is reduced. Although the back-end FPGA increases the delay, the total delay does not increase significantly. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 :Decomposition of Camera hardware solution based on SoC chip

[0053] Figure 2 :Video matrix hardware solution based on SoC chip

[0054] Figure 3 :Internal diagram of video processing SoC chip

[0055] Figure 4 :Video processing SoC chip software and hardware processing flow chart

[0056] Figure 5 : Schematic diagram showing the frame loss process when the number of receiving buffer queues is 2

[0057] Figure 6 :Schematic diagram of this application plan

[0058] Figure 7 : Schematic diagram of the process of eliminating buffer accumulation when the video output frame rate is greater than the video input frame rate

[0059] Figure 8 :Schematic diagram of the solution using the same source clock DETAILED DESCRIPTION

[0060] The invention is described in detail below with reference to the embodiments.

[0061] Figure 3 A typical video SoC processor is shown, the processor includes a CPU, a coprocessor, a memory controller, a memory chipset, a video input unit and a video output unit; wherein the video input unit is used to collect video images and store them in the memory chipset, the video images are transmitted using a MIPI interface, a LVDS interface or a parallel port, and the video output unit is used to output the video images stored in the memory chipset through interfaces such as HDMI and DP; the memory chipset is controlled by a memory controller, and the CPU and the coprocessor perform instruction operations and / or data operations with the memory chipset through the memory controller. The instruction operations and / or data operations are determined by the system architecture adopted by the CPU and the coprocessor, and the system architecture can adopt architectures such as Arm and RISC-V. The coprocessor includes a GPU, an H.265 decoder, an H.265 encoder, a counter, an OSD processor, and the like.

[0062] The video processing SoC chip uses the video input unit to capture video images, passes through other video processing units (such as video scaling), and finally outputs the video to standard image transmission interfaces such as the HDMI interface, DP interface, and parallel port through the video output unit. The video input unit and the video output unit are hardware integrated, but the CPU performs initialization configuration, buffer application, and buffer management on them, which means that the CPU needs to implement the corresponding driver software, including the video input unit driver software and the video output unit driver software. The entire process of the video processing SoC chip from video acquisition to video output is as follows: Figure 4 shown.

[0063] Step 1: The video input unit collects one frame of image and puts it into DDR memory;

[0064] Step 2: The video input unit generates the first hardware interrupt to the CPU;

[0065] Step 3: The video input unit driver software running inside the CPU responds to the first hardware interrupt and puts the DDR address information of the new frame image into the Buffer receiving queue of the next video processing unit; after multi-layer pipeline control, it is finally added to the display Buffer receiving queue created by the video output unit driver software;

[0066] Step 4: After the video output unit completes the transmission of one frame of image, it generates a second hardware interrupt to the CPU;

[0067] Step 5: The video output unit driver software running inside the CPU responds to the second hardware interrupt, and takes out the DDR address of a new frame of image from the display buffer receiving queue and passes it to the video output unit;

[0068] Step 6: The video output unit reads one frame of image from the specified DDR address and sends it out.

[0069] In the above, the video output unit driver software creates a display receiving buffer queue to manage the video frame data waiting to be sent, and in order to reduce transmission delay, the number of frames in the display receiving buffer queue is limited, for example, it is generally fixed to 2 to 3 frames. Under normal circumstances, the frame rate collected by the video input unit is equal to the frame rate of the video output unit, and the CPU interrupt processing response is timely, then the video output unit does not lose frames. However, frame loss will occur in the following situations:

[0070] 1) Due to the busy CPU, the video input unit driver software did not respond to the CPU interrupt in time, resulting in the timely addition of the newly captured video frame to the display receiving buffer queue driven by the video output unit. The video output unit needs to output video data according to the standard hardware interface timing and cannot wait. That is, when the video output unit transmits one frame of image and finds that there is no new video frame in the display receiving buffer queue, it must also send the next frame of image immediately. In this case, it can only send the previous frame of image that has been sent once again (i.e., send a "duplicate frame"). In the process of sending the "duplicate frame", the CPU software responds to the interrupt and adds the late new frame to the display receiving buffer queue, which causes the buffer queue to accumulate one frame of image. When the video output unit sends a "duplicate frame", it must send a whole frame of "duplicate frame" before sending the next frame. It cannot send halfway and then cut off to the DDR address of the new frame. Otherwise, the upper and lower images will not be the same frame, resulting in screen cracking and other phenomena.

[0071] Once the buffer generates 1 accumulated frame, if the acquisition frame rate of the video input unit is equal to the frame rate of the video output unit, it will continue to accumulate. Similarly, the next time the CPU interrupt response is not handled in time, 1 more frame of buffer will be accumulated. However, the number of buffers is limited, which will eventually cause the buffer queue to burst and can only be overwritten or discarded, resulting in "frame loss". When the number of display receiving buffer queues of the video output unit is 2, the frame loss process is as follows Figure 5 shown.

[0072] 2) The frame rates configured for the video input unit and the video output unit are the same, but due to the clock PLL precision error, the actual frame rate collected by the video input unit may not be absolutely equal to the actual frame rate transmitted by the video output unit. For example, if both are configured at 30fps, but the frame rate collected by the video input is 30.00001fps, and the frame rate output by the video output unit is 30.00000fps, then over time, the display receiving buffer queue will also be full and cause frame loss.

[0073] Above, let’s summarize the reasons for frame loss:

[0074] A) Hardware constraints: The video hardware output unit of the SoC uses an industry-standard video transmission interface with a fixed output frame rate and timing. After a frame is transmitted, the next frame must be sent immediately according to the timing requirements without waiting. If the driver software of the video output unit is not ready for a new frame at this time, the video output unit must also send a data frame (generally, it chooses to repeat the previous frame, that is, send a "duplicate frame");

[0075] B) Software reasons: The video output unit driver software applies for a fixed M-frame buffer queue (the larger M is, the greater the output delay) based on the video output delay requirements. When the video output unit is about to transmit a frame of video, the interrupt triggers the software to take a new frame from the buffer queue and send the memory address to the video output unit for the next frame transmission. In the SoC, the CPU runs the software operating system, and the video input unit and video output unit require software participation. The multi-tasking operating system cannot guarantee the timeliness of software processing, especially when the CPU occupancy rate is high.

[0076] Hardware reason: There is a slight deviation between the receiving frame rate of the video input acquisition unit and the output frame rate of the video output unit.

[0077] Although video processing SoC chips cannot eliminate the phenomenon of "duplicate frames" and "frame loss" due to software or hardware reasons, video processing SoC chips still have the advantages of low power consumption and low price. Based on the research on video SoC chips, this application provides a video processing solution to prevent frame loss, which specifically includes:

[0078] Step S1: The video input unit captures images and stores them in memory in units of frames at a first frame rate;

[0079] Step S2: the video input unit generates a first hardware interrupt to the CPU;

[0080] Step S3: The CPU responds to the first hardware interrupt, and the corresponding first interrupt response function puts the memory address information of the image stored in the memory in step S1 into the tail of the buffer queue corresponding to the video output unit;

[0081] Step S4: after the video output unit completes transmission of data corresponding to one frame of image at the second frame rate, a second hardware interrupt is generated to the CPU;

[0082] Step S5: the CPU responds to the second hardware interrupt, and the corresponding second interrupt response function takes out the memory address information of the new frame of image from the head of the buffer queue and transmits it to the video output unit, and updates the memory address information;

[0083] Step S6: the video output unit reads data corresponding to one frame of image from the memory starting from the memory address, and sends it to the frame rate reduction unit at the second frame rate;

[0084] Step S7: the frame rate reduction unit receives the image data of the video output unit, detects and removes duplicate frames, and generates a standard video stream with a first frame rate;

[0085] The second frame rate of the video output unit is greater than the first frame rate of the video input unit.

[0086] Preferably, step S3 further comprises: after multi-layer pipeline control, finally adding the display buffer receiving queue created by the video output unit driver software.

[0087] Preferably, the ratio of the second frame rate to the first frame rate is: second frame rate / first frame rate ≥ 1.2.

[0088] Preferably, the first frame rate is 50fps and the second frame rate is 60fps.

[0089] Preferably, when the cache queue length is 1 frame, the memory address of the video output unit is not updated; if it is not in the blanking period of video transmission, the memory address of the video output unit is not updated either.

[0090] Furthermore, the step of detecting and eliminating duplicate frames specifically includes:

[0091] The video input unit embeds an ID number at a first fixed position of a captured frame of image, so as to retain the ID number when the video output unit sends the frame of image to the frame rate reduction unit processing module;

[0092] The frame rate reduction unit detecting repeated frames includes: the frame rate reduction unit parsing the embedded ID serial number from the one frame of image, and detecting whether the one frame of image is a repeated frame according to the repeatability of the ID serial number;

[0093] or,

[0094] The image content of each frame is accumulated and calculated to obtain a checksum. When the checksums are equal, it is considered a "duplicate frame".

[0095] Preferably, the first fixed position is the first line of the image content.

[0096] Preferably, the frame rate reduction unit processing module is implemented using FPGA.

[0097] Preferably, the elimination step in the detecting and eliminating duplicate frames refers to removing the duplicate frames detected in the detecting step and restoring the frame rate to the same first frame rate as the video input unit.

[0098] The video data after the step of detecting and eliminating duplicate frames will be stored in the memory corresponding to the frame rate reduction unit processing module.

[0099] As a preferred implementation, it is possible to consider using a video processing SoC chip + FPGA chip hardware design to implement the above solution. The architecture is as follows: Figure 6 As shown:

[0100] In order to solve the frame loss problem of the video processing SoC chip, an FPGA chip is added. It should be noted that in some solutions, FPGA may be needed by itself. For example, FPGA is used to convert HDMI video interface to SDI video interface. If the SoC chip backend already has an FPGA chip, it only needs to add the corresponding processing module. The steps for the collaborative processing of the video processing SoC chip and the FPGA chip are as follows:

[0101] A) Video processing SoC chip: In the prior art, the video output frame rate is usually configured to be the same as that of the video input unit. However, in order to solve the frame loss problem, the present invention needs to configure the frame rate of the video output unit to be greater than the frame rate captured by the video input unit, that is, the video output unit outputs video at a second frame rate, while the video input unit inputs video at a first frame rate, and the second frame rate is greater than the first frame rate; for example, as an optional implementation, the frame rate of the video input unit is 50fps, then the frame rate of the video output unit can be configured to 60fps, so that the video output unit will generate a large number of "duplicate frames" (even if the CPU software interrupt responds in time), but will not generate "frame loss".

[0102] The reason for generating a large number of "duplicate frames" is relatively easy to understand, because the video output unit is generated according to the standard interface timing and the output frame rate is fixed, and the frame rate of the front-end video input unit is lower than the frame rate of the video output unit. Then there is a fixed frame rate difference per unit time. For example, if the front-end input is 50fps and the back-end output is 60fps, then 10 frames of "duplicate frames" will be generated in 1 second.

[0103] When the second frame rate of the configured video output is greater than the first frame rate of the video input, no "frame loss" occurs. This is mainly because when 1 frame of buffer accumulation occurs, the output rate is greater than the input rate, so the accumulated buffer can be digested within a certain period of time, and the buffer queue will not be overloaded due to continuous accumulation.

[0104] Preferably, in order to better enable those skilled in the art to understand the whole process, the following description is given by taking the number of buffer queues of the video output unit as 2 and the frame rate of the video output as 2 times the frame rate of the video input as an example. It should be noted that such a process should not be interpreted as limiting the scope of protection of the present invention; Figure 7 shown.

[0105] When the video input unit does not have a CPU interrupt response delay, since the video input frame rate is only 1 / 2 of the video output frame rate, the video output unit will repeat the frame once every 1 frame. Figure 7 If the t3.1 is t3.1, then two normal frames will occur continuously in the T5 and T6 cycles, so that the accumulated 1-frame buffer will be cleared within 2 frames. The CPU's untimely response is not normal, but only occurs occasionally. In addition, the number of buffer queues of the video output unit itself is generally greater than 2, which has an anti-jitter effect. That is, as long as the accumulated buffer can be cleared within a specific time, there will be no "frame loss" problem.

[0106] B) FPGA chip: Since the video SoC chip intentionally increases the output frame rate to solve the "frame loss" problem, it generates many "duplicate frames". Therefore, the video input interface module of the FPGA chip collects video data with increased frame rate and "duplicate frames". To this end, the FPGA adds a frame rate reduction unit module, which includes a duplicate frame detection module and a duplicate frame removal module to remove duplicate frames and restore them to the same frame rate as the video input unit of the video processing SoC chip.

[0107] There are two ways to implement the duplicate frame detection module:

[0108] The first method is that after the video processing SoC chip receives a frame of image, the ID number is embedded in the first line of the image content, and the ID number information is still retained when it is sent to the FPGA through the video output unit; the FPGA then parses the embedded ID number from the image content, and based on the repeatability of the ID number, it can detect whether it is a "duplicate frame".

[0109] The second method is that the FPGA performs cumulative sum operations on the image content of each frame and calculates a checksum. When the checksums are equal, it is considered a "duplicate frame." Note that the video images captured by real cameras are generated by photoelectric sensors, and the sensors themselves have random noise, so two adjacent video data frames will not be exactly the same.

[0110] After the video data passes through the duplicate frame detection module and the duplicate frame removal module, it will be stored in the external DDR memory of the FPGA. Due to the untimely response of the video processing SoC interrupt, the cycle of the FPGA receiving the video frame may occasionally be a little slower or faster.

[0111] In order to send out video data with unstable frame rates occasionally, FPGA needs to perform frame buffer processing and start sending after receiving two complete frames of images, so that jitter can be filtered out and continuous images can be output. The specific implementation method is to control the video output interface module of FPGA to lag 2 frames for output. Although FPGA delays the output by 2 frames, the overall delay does not increase: the video output unit of the video processing SoC chip also has a display receiving buffer queue, but because the configured frame rate of the video output unit is greater than the frame rate of the video input unit, the display receiving buffer queue of the video output unit usually does not have video frame accumulation; compared with the existing solution where the video output unit of the video processing SoC chip is equal to the video input unit, the video output unit generally needs to configure a buffer queue of 2 or 3 frames. If accumulation occurs, it will cause delay. --- That is, with the new solution, the delay of the video processing SoC chip itself is reduced. Although the back-end FPGA increases the delay, the total delay does not increase significantly.

[0112] In addition, to solve the problem that there may be a slight deviation between the frame rate collected by the video processing SoC chip and the frame rate output by the video output interface module of the FPGA (for example, the frame rate collected by the video processing SoC chip is 50.0000001 fps, while the frame rate output by the FPGA chip is 50.0000000), the FPGA needs to add a frame rate dynamic adjustment module to avoid problems such as frame explosion caused by cumulative frames similar to those of the video processing SoC chip. It should be noted that the standard video output interface allows a certain PPM (part per million) of clock deviation jitter. The frame rates collected by the video processing SoC chip and sent by the video output module of the FPGA chip need to meet the clock deviation jitter range of the standard interface timing, that is, the floating range of clock jitter is not very large. The specific implementation method of the FPGA frame rate dynamic adjustment mode is as follows: 1) Count the number of video frames output by the FPGA duplicate frame removal module within a specific period and calculate the frame rate fps_input; 2) Count the number of video frames sent by the FPGA video output interface module within a specific period fps_output and calculate the frame rate; 3) If fps_input > fps_output, dynamically increase the pixel clock of fps_output within the PPM range allowed by the standard video output interface; conversely, if fps_input < fps_output, decrease the pixel clock of fps_output. To achieve the effect that the FPGA video output module dynamically follows the video input frame rate of the video processing SoC chip macroscopically.

[0113] Through the entire technical solution of this invention, a video processing method for preventing frame loss is realized, and the feasibility is verified in actual product applications. For example, when the frame rates configured for the video input unit and the video output unit of a certain video SoC chip are both 100 fps, there will be frame loss phenomena after actual testing.

[0114] After adopting the solution of this application, there is no frame loss phenomenon after statistics.

[0115] For example, in a test, for a video processing SoC chip that does not adopt the foregoing solution of this application, the frame rate configured for the video output unit of the SoC chip is the same as the frame rate collected by the video input unit. After statistics, when the CPU occupancy rate of the SoC chip is less than 15%, there will be 1 - 2 frame losses on average every about 5 minutes; when the CPU occupancy rate is greater than 60%, there will be 1 - 2 frame losses on average every 2 minutes.

[0116] After adopting the anti-frame loss video processing method described in this application, for the test condition that the frame rate (60fps) of the video output unit of the video processing SoC chip is greater than the frame rate (50fps) of the video input unit, it has been measured that when the CPU occupancy of the video processing SoC chip is less than 15% and greater than 60%, no frame loss occurs during the test of continuous operation for 24 hours.

[0117] As an optional implementation, Figure 8 As shown, you can also use the same source clock to ensure frame rate synchronization. The principle is that the clock PPM deviation, all devices using this clock will have synchronization deviation, and they are relatively static. This can delete the frame rate dynamic adjustment module inside the FPGA chip, and then other solutions remain unchanged.

[0118] The frame rate dynamic adjustment module inside the FPGA chip is to solve the problem that the actual frame rate collected by the video input unit of the video processing SoC chip has a slight deviation from the actual frame rate of the video output interface module of the FPGA chip, which may cause the video output interface module of the FPGA chip to also have frame loss. If the frame rate of the video input unit of the video processing SoC chip is exactly the same, then the FPGA frame rate dynamic adjustment module can be removed. The specific implementation method is to use the same source clock. For example, in the Camera solution, the same source clock is provided to the image sensor, video processing SoC chip, and FPGA chip.

[0119] The above is only a preferred embodiment of the present application and does not constitute any other form of limitation to the present application. Any modification or equivalent change made based on the technical essence of the present application still falls within the scope of protection required by the present application.

Claims

1. A video processing method for preventing frame loss, Features: Step S1: The video input unit captures images and stores them in memory in units of frames at a first frame rate; Step S2: the video input unit generates a first hardware interrupt to the CPU; Step S3: The CPU responds to the first hardware interrupt, and the corresponding first interrupt response function puts the memory address information of the image stored in the memory in step S1 into the tail of the buffer queue corresponding to the video output unit; Step S4: after the video output unit completes transmission of data corresponding to one frame of image at the second frame rate, a second hardware interrupt is generated to the CPU; Step S5: the CPU responds to the second hardware interrupt, and the corresponding second interrupt response function takes out the memory address information of a new frame of image from the head of the buffer queue and transmits it to the video output unit, and updates the memory address information; Step S6: the video output unit reads data corresponding to one frame of image from the memory starting from the memory address, and sends it to the frame rate reduction unit at the second frame rate; Step S7: the frame rate reduction unit receives the image data of the video output unit, detects and removes duplicate frames, and generates a standard video stream with a first frame rate; The second frame rate of the video output unit is greater than the first frame rate of the video input unit.

2. The method according to claim 1, Features: The step of detecting and eliminating duplicate frames specifically includes: The video input unit embeds an ID number at a first fixed position of a captured frame of image, so as to retain the ID number when the video output unit sends the frame of image to the frame rate reduction unit processing module; The frame rate reduction unit detecting repeated frames includes: the frame rate reduction unit parsing the embedded ID serial number from the one frame of image, and detecting whether the one frame of image is a repeated frame according to the repeatability of the ID serial number; or, The image content of each frame is accumulated and calculated to obtain a checksum. When the checksums are equal, it is considered a "duplicate frame".

3. The method according to claim 1, Features: The removing step in the detecting and removing duplicate frames refers to removing the duplicate frames detected in the detecting step and restoring the frame rate to the same first frame rate as the video input unit.

4. The method according to claim 1, Features: It further includes a frame buffering step S8, wherein the step S8 starts sending after receiving complete n frames of images; wherein n is a natural number greater than 2.

5. The method according to claim 1, Features: The frame rate dynamic adjustment step S9 is also included, which specifically includes: Step S9-1: Count the number of video frames output by the FPGA duplicate frame elimination module within a specific period, and calculate the frame rate fps_input; Step S9-2: Count the number of video frames fps_output sent by the FPGA video output interface module within a specific period, and calculate the frame rate; Step S9-3: If fps_input > fps_output, dynamically increase the pixel clock of fps_output within the PPM allowed by the standard video output interface; conversely, if fps_input < fps_output, decrease the pixel clock of fps_output; so as to achieve the effect that the FPGA video output module dynamically follows the video input frame rate of the video processing SoC chip macroscopically.

6. An apparatus for implementing the video processing method for preventing frame loss as described in claim 1, characterized in that, it includes: Video input module: Collect images and store them in memory frame by frame at a first frame rate; Video output module: Read 1 frame of image from memory and send it at a second frame rate; The second frame rate is greater than the first frame rate.

7. The apparatus according to claim 6, characterized in that: It further includes a frame rate reduction unit module, which specifically includes a duplicate frame detection module and a duplicate frame removal module.

8. The apparatus according to any one of claims 6-7, characterized in that: It further includes a frame buffer module.

9. The apparatus according to any one of claims 6-8, characterized in that: It further includes a frame rate dynamic adjustment module.

10. The apparatus according to any one of claims 6-9, characterized in that: The frame rate dynamic adjustment module statistics the video frame rate after FPGA duplicate frame removal and the output frame rate of the FPGA video output module, and dynamically adjusts the pixel clock of the FPGA video output module according to the interpolation of the frame rate, so that the frame rate of the FPGA video output module dynamically tracks the frame rate collected by the video input module of the video processing SoC chip.