A method and device for detecting and visualizing missing frames in a video raw stream file

Abstract

The application discloses a kind of video bare stream file frame loss detection and visual analysis method and device, it is related to video analysis technical field.The method includes: constructing a video analysis interface, including video picture display area, original code stream information display area, video play control area, frame visual presentation area and frame loss information display area;When video bare stream file is acquired, the frame type and frame serial number of all frames are parsed and identified;Frame serial number is traversed to check continuity, and frame loss information is generated according to the inspection result;All frames are visualized in frame visual presentation area and presented, I frame, P frame is distinguished and displayed, and frame loss position is marked;According to the operation of video play control area, visual presentation area or frame loss information display area, the display content of other each area is updated.The video bare stream file frame loss detection and visual analysis method and device provided by the application realize the visual analysis of video bare stream file.

Description

Technical Field

[0001] This invention relates to the field of video analysis technology, and in particular to a method and apparatus for detecting and visualizing dropped frames in raw video stream files. Background Technology

[0002] H.264 is currently the most widely used video coding standard in fields such as video conferencing, security monitoring, and streaming media. During the research and development and testing of video conferencing systems, developers often need to analyze the raw H.264 video stream (i.e., the .h264 raw stream file) captured from the device or network to locate problems such as video distortion, stuttering, and decoding failure caused by network packet loss, encoder malfunctions, or encapsulation errors.

[0003] However, while existing general-purpose video players (such as VLC and PotPlayer) or professional video analysis software (such as Elecard StreamEye and H.264Visa) can play or parse H.264 streams, they have the following limitations: First, they are mainly designed for complete, encapsulated media files (such as MP4 and MKV). For raw .h264 stream files without encapsulation headers, they often cannot be opened or played directly and require complex format conversion. Second, even if some tools can play raw streams, these playback tools cannot perform intelligent analysis. Developers need to manually check frame by frame or write additional scripts for analysis, which is inefficient. Third, existing tools cannot accurately locate the position of dropped frames in raw stream files. Fourth, existing tools have difficulty simultaneously viewing the raw NAL unit byte data of the frame, limiting the ability of developers to deeply analyze anomalies from the underlying bitstream. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and apparatus for detecting and visually analyzing dropped frames in raw video stream files, which enables rapid parsing of raw video stream files, automatic detection of dropped frame locations, synchronous display of the current frame's original bitstream data, and intuitive presentation of frame type distribution and dropped frame information in a visual manner.

[0005] In a first aspect, the present invention provides a method for detecting and visually analyzing dropped frames in raw video stream files, including: Interface construction process: Build a video analysis interface, including a video display area, a raw bitstream information display area, a video playback control area, a frame visualization display area, and a frame loss information display area; Video raw stream parsing process: When the video raw stream file is obtained, the frame type and frame number of all frames are parsed and identified; the frame numbers are traversed to check the continuity, and the lost frame information is generated based on the check results; all frames are visualized in the frame visualization area, I-frames and P-frames are displayed separately, and the lost frame positions are marked. Interaction process: Update the display content of other areas based on the operations performed in the video playback control area, the visualization presentation area, or the frame loss information display area.

[0006] Furthermore, the frame visualization area displays each frame as a corresponding frame bar, including frame position and frame number information, and uses different colors to represent I-frames, P-frames, and the currently selected frame.

[0007] Furthermore, the frame loss information display area is specifically a frame loss list, including the frame position and frame number of each lost frame.

[0008] Furthermore, the video playback control area includes playback control controls and frame number input controls, as well as frame rate adjustment controls.

[0009] Furthermore, updating the display content of other areas based on the operations of the video playback control area, the visualization presentation area, or the frame loss information display area specifically includes: The current frame is determined based on the operations of the video playback control area and the visualization display area, or the adjacent frames of the current frame are determined based on the operations of the frame loss information display area, and then the display content of other areas is made to correspond to the current frame or adjacent frames.

[0010] Secondly, the present invention provides a device for detecting and visualizing frame drops in raw video stream files, comprising: The interface construction module is used to build a video analysis interface, including a video display area, a raw bitstream information display area, a video playback control area, a frame visualization display area, and a frame loss information display area. The video raw stream parsing module is used to parse and identify the frame type and frame number of all frames when a video raw stream file is obtained; it traverses the frame number to check the continuity and generates lost frame information based on the check results; it visualizes all frames in the frame visualization display area, distinguishes between I-frames and P-frames, and marks the location of lost frames. The interaction module is used to update the display content of other areas based on the operations performed in the video playback control area, the visualization presentation area, or the frame loss information display area.

[0011] Furthermore, the frame visualization area displays each frame as a corresponding frame bar, including frame position and frame number information, and uses different colors to represent I-frames, P-frames, and the currently selected frame.

[0012] Furthermore, the frame loss information display area is specifically a frame loss list, including the frame position and frame number of each lost frame.

[0013] Furthermore, the video playback control area includes playback control controls and frame number input controls, as well as frame rate adjustment controls.

[0014] Furthermore, updating the display content of other areas based on the operations of the video playback control area, the visualization presentation area, or the frame loss information display area specifically includes: The current frame is determined based on the operations of the video playback control area and the visualization display area, or the adjacent frames of the current frame are determined based on the operations of the frame loss information display area, and then the display content of other areas is made to correspond to the current frame or adjacent frames.

[0015] The technical solutions provided in the embodiments of the present invention have at least the following technical effects or: 1. Significantly improved analysis efficiency: By automatically detecting and listing all lost frame locations, R&D personnel no longer need to manually check frame by frame. For video files with tens of thousands of frames, the manual investigation work that originally took several hours can be shortened to seconds.

[0016] 2. Optimizations specifically for raw video streams: Directly supports the parsing and playback of raw video stream files, eliminating the format conversion step and simplifying the analysis process for developers.

[0017] 3. Intuitive and clear information presentation: The innovative frame visualization area uses color-coded frame bars to display the distribution of frame types in the entire file. Combined with special markers for lost frames, users can quickly locate abnormal areas on the macro timeline and achieve a clear and easy-to-understand assessment of the stream's health status.

[0018] 4. Traceable underlying bitstream: Synchronously displays the original bitstream of the current frame (NAL unit hexadecimal data) and supports user copying, which facilitates R&D personnel to conduct in-depth analysis of anomalies at the bitstream level, such as start code errors, NAL type anomalies, syntax element errors, etc.

[0019] 5. Flexible jumping and speed adjustment: Supports precise jumping by directly inputting frame number and customizing FPS speed adjustment playback, which is convenient for detailed analysis or quick browsing of specific areas.

[0020] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the video analysis interface in Embodiment 1 of the present invention; Figure 3This is a flowchart of the video analysis process in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the device in Embodiment 2 of the present invention. Detailed Implementation

[0023] This application provides a method and apparatus for detecting and visually analyzing dropped frames in raw video stream files. This enables rapid parsing of raw video stream files, automatic detection of dropped frame locations, synchronous display of the current frame's raw bitstream data, and intuitive presentation of frame type distribution and dropped frame information in a visual manner.

[0024] The overall concept of the technical solution in this application is as follows: Automatic frame loss detection mechanism based on frame sequence number continuity: By traversing and checking whether the frame sequence numbers of adjacent frames in the bitstream are continuous, the mechanism automatically identifies and records the location of the lost frame and the lost frame number.

[0025] The combination of direct parsing of raw streams and visual coloring: without relying on any encapsulation container, the raw stream file is directly parsed, and the parsing results are presented intuitively on the same interface in the form of red (I-frame) / white (P-frame) frame bars and blue (currently selected frame) highlights.

[0026] Linked display of frame information and raw bitstream: When the user clicks on a frame bar or jumps to a specific frame, the decoded screen of that frame is displayed synchronously with the original NAL unit hexadecimal data, and the bitstream data can be copied, which is convenient for locating problems from both the encoding layer and the bitstream layer.

[0027] Precise frame number jump and custom frame rate playback: Supports users to directly enter the frame number to jump to a specified frame, and supports custom FPS adjustment of playback speed. Example 1

[0028] This embodiment provides a method for detecting and visually analyzing dropped frames in raw video stream files, such as... Figure 1 As shown, including; S1. Interface Construction Process: Construct a video analysis interface, including a video display area, a raw bitstream information display area, a video playback control area, a frame visualization area, and a frame loss information display area.

[0029] In one specific embodiment, the raw video stream file is specifically a raw stream file in .h264 format, and the video analysis interface is referenced. Figure 2 : A: Video display area, used to display the decoded image of the current frame; B: Raw bitstream information display area, used to display the NAL unit hexadecimal byte data of the currently selected frame. This area allows users to copy the data for further analysis.

[0030] C: Video playback control area, used to control the playback of the target video, including play / pause switch button, progress bar drag control, FPS frame rate adjustment input box and "Settings" button, frame number jump input box and "Settings" button.

[0031] D: Frame visualization area, used to visualize all frames. For example, each frame is displayed as a corresponding frame bar, showing all frames of the entire H.264 file in a horizontal "frame bar" format, including frame position and frame number information. I-frames (keyframes), P-frames (predicted frames), and the currently selected frame are represented by different colors. In this embodiment, red frame bars represent I-frames, white frame bars represent P-frames, and blue highlights the currently selected frame. Missing frame locations are highlighted with special marks on the frame bars.

[0032] E: Frame loss information display area, which lists all detected frame loss records in the format of "frame position: lost frame number" (for example, "1198:94" means that the frame with frame number 94 was lost at frame position 1198).

[0033] S2. Video raw stream parsing process: When the video raw stream file is obtained, the frame type and frame number of all frames are parsed and identified; the frame numbers are traversed to check the continuity, and the lost frame information is generated based on the check results; all frames are visualized in the frame visualization area, I-frames and P-frames are displayed separately, and the lost frame positions are marked.

[0034] The raw stream parsing module directly reads raw .h264 format files without relying on any container. This module parses the H.264 NAL units byte by byte, identifying the type of each NAL unit (including I-frame, P-frame, SPS, PPS, etc.) and extracting the frame sequence information for each video frame. Simultaneously, the module outputs the complete raw byte data of the NAL unit of the currently selected frame in hexadecimal format to the interface information display area.

[0035] The automatic frame drop detection module iterates through the frame sequence values ​​of all parsed video frames, checking the continuity of frame sequence numbers between adjacent frames in playback order. When a frame sequence number jump is detected (i.e., the difference between the current frame's frame sequence number and the previous frame's frame sequence number is greater than 1), it is determined that a frame drop has occurred at that position. The tool automatically records the frame drop position (i.e., the frame's position in the file's playback order) and the lost frame sequence number, generating a frame drop list in the format "frame position:lost frame number," which is displayed in the frame drop information display area.

[0036] S3. Interaction process: Update the display content of other areas based on the operations of the video playback control area, the visualization presentation area, or the frame loss information display area.

[0037] The current frame is determined based on the operations performed in the video playback control area and the visualization display area, and then the displayed content in other areas corresponds to the current frame or adjacent frames. For example, when the video plays to a certain frame, it becomes the current frame, or when the user clicks on a target position on the frame bar, the current frame is selected. The image of that frame is displayed in the video playback control area, and the raw bitstream data of that frame is updated synchronously in the raw bitstream information display area. The frame loss information display area can also automatically jump to the range of frame loss positions closest to the current frame.

[0038] Based on the operations in the frame loss information display area, the adjacent frames (which can be the previous or next frame) of the current frame are determined, and then the display content in other areas is made to correspond to the current frame or adjacent frames. For example, when a user clicks on a "Frame Position: Lost Frame Number" in the frame loss list, its adjacent frame (the previous or next frame) is determined as the selected current frame. At the same time, the image of that frame is displayed in the video playback control area, and the raw bitstream data of that frame is updated synchronously in the raw bitstream information display area; the frame visualization display area can also automatically jump to the current frame position range.

[0039] In one specific embodiment, the raw video stream file is specifically a raw stream file in .h264 format, and the analysis process is as follows: Figure 3 As shown.

[0040] 1. Open file: The user clicks the "Open Video" button at the top of the interface, selects the .h264 raw stream file, and the tool immediately begins parsing.

[0041] 2. Automatic Parsing and Detection: The raw stream parsing module parses the file, extracts all frames and frame type information, and displays the raw bitstream data of the current default frame in the raw bitstream information display area. The automatic frame loss detection module automatically traverses the frame sequence numbers of all frames, checks continuity, and generates a list of lost frames.

[0042] 3. Visual Presentation: The frame bars in the frame visualization area are automatically drawn—red bars represent I-frames, white bars represent P-frames, and blue highlights indicate the currently selected frame. If any frames are missing, a special marker will be displayed at the corresponding location.

[0043] 4. Playback and Viewing: Users can switch playback status at any time by clicking the play / pause button. Clicking any frame on the frame bar, or entering the frame number in the Frame input box and clicking "Settings," will jump to the corresponding frame. The frame's image will be displayed in the video display area, and the raw bitstream hexadecimal data of that frame will be displayed in the raw bitstream information display area. Users can copy this data for offline analysis.

[0044] 5. Speed ​​Adjustment and Analysis: Users can modify the frame rate value in the FPS input box in the video playback control area and click "Settings" to change the playback speed. For dropped frames, users can combine the dropped frame list information with the speed adjustment playback function to perform detailed analysis of abnormal areas and accurately locate the root cause of decoding anomalies.

[0045] Based on the same inventive concept, this application also provides an apparatus corresponding to the method in Embodiment 1, as detailed in Embodiment 2. Example 2

[0046] This embodiment provides a device for detecting and visualizing dropped frames in raw video stream files, such as... Figure 4 As shown, it includes: The interface construction module is used to build a video analysis interface, including a video display area, a raw bitstream information display area, a video playback control area, a frame visualization display area, and a frame loss information display area. The video raw stream parsing module is used to parse and identify the frame type and frame number of all frames when a video raw stream file is obtained; it traverses the frame number to check the continuity and generates lost frame information based on the check results; it visualizes all frames in the frame visualization display area, distinguishes between I-frames and P-frames, and marks the location of lost frames. The interaction module is used to update the display content of other areas based on the operations performed in the video playback control area, the visualization presentation area, or the frame loss information display area.

[0047] Preferably, the frame visualization presentation area displays each frame as a corresponding frame bar, including frame position and frame number information, and uses different colors to represent I-frames, P-frames and the currently selected frame respectively.

[0048] Preferably, the frame loss information display area is specifically a frame loss list, including the frame position and frame number of each lost frame.

[0049] Preferably, the video playback control area includes playback control controls and frame number input controls, and also includes frame rate adjustment controls.

[0050] Preferably, updating the display content of other areas based on the operations of the video playback control area, the visualization presentation area, or the frame loss information display area specifically includes: The current frame is determined based on the operations of the video playback control area and the visualization display area, or the adjacent frames of the current frame are determined based on the operations of the frame loss information display area, and then the display content of other areas is made to correspond to the current frame or adjacent frames.

[0051] Since the apparatus described in Embodiment 2 of the present invention is an apparatus used to implement the method of Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of the present invention, and therefore will not be described again here. All apparatuses used in the method of Embodiment 1 of the present invention fall within the scope of protection of the present invention.

[0052] The technical solutions provided in the embodiments of the present invention have at least the following technical effects or: 1. Significantly improved analysis efficiency: By automatically detecting and listing all lost frame locations, R&D personnel no longer need to manually check frame by frame. For video files with tens of thousands of frames, the manual investigation work that originally took several hours can be shortened to seconds.

[0053] 2. Optimizations specifically for raw video streams: Directly supports the parsing and playback of raw video stream files, eliminating the format conversion step and simplifying the analysis process for developers.

[0054] 3. Intuitive and clear information presentation: The innovative frame visualization area uses color-coded frame bars to display the distribution of frame types in the entire file. Combined with special markers for lost frames, users can quickly locate abnormal areas on the macro timeline and achieve a clear and easy-to-understand assessment of the stream's health status.

[0055] 4. Traceable underlying bitstream: Synchronously displays the original bitstream of the current frame (NAL unit hexadecimal data) and supports user copying, which facilitates R&D personnel to conduct in-depth analysis of anomalies at the bitstream level, such as start code errors, NAL type anomalies, syntax element errors, etc.

[0056] 5. Flexible jumping and speed adjustment: Supports precise jumping by directly inputting frame number and customizing FPS speed adjustment playback, which is convenient for detailed analysis or quick browsing of specific areas.

[0057] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0061] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting and visually analyzing dropped frames in raw video stream files, characterized in that, include: Interface construction process: Build a video analysis interface, including a video display area, a raw bitstream information display area, a video playback control area, a frame visualization display area, and a frame loss information display area; Video raw stream parsing process: When the video raw stream file is obtained, the frame type and frame number of all frames are parsed and identified; Iterate through the frame sequence numbers to check for continuity, and generate frame loss information based on the check results; Visualize all frames in the frame visualization area, distinguish between I-frames and P-frames, and mark the locations of dropped frames; Interaction process: Update the display content of other areas based on the operations performed in the video playback control area, the visualization presentation area, or the frame loss information display area.

2. The method according to claim 1, characterized in that: The frame visualization area displays each frame as a corresponding frame bar, including frame position and frame number information, and uses different colors to represent I-frames, P-frames and the currently selected frame.

3. The method according to claim 1, characterized in that: The frame loss information display area is specifically a frame loss list, which includes the frame position and frame number of each lost frame.

4. The method according to claim 1, characterized in that: The video playback control area includes playback control controls and frame number input controls, as well as frame rate adjustment controls.

5. The method according to claim 1, characterized in that: The process of updating the display content of other areas based on operations performed in the video playback control area, the visualization presentation area, or the frame drop information display area specifically includes: The current frame is determined based on the operations of the video playback control area and the visualization display area, or the adjacent frames of the current frame are determined based on the operations of the frame loss information display area, and then the display content of other areas is made to correspond to the current frame or adjacent frames.

6. A device for detecting and visualizing dropped frames in raw video stream files, characterized in that, include: The interface construction module is used to build a video analysis interface, including a video display area, a raw bitstream information display area, a video playback control area, a frame visualization display area, and a frame loss information display area. The video raw stream parsing module is used to parse and identify the frame type and frame number of all frames when a video raw stream file is obtained; Iterate through the frame sequence numbers to check for continuity, and generate frame loss information based on the check results; Visualize all frames in the frame visualization area, distinguish between I-frames and P-frames, and mark the locations of dropped frames; The interaction module is used to update the display content of other areas by operating the visualization area or the frame loss information display area.

7. The apparatus according to claim 6, characterized in that: The frame visualization area displays each frame as a corresponding frame bar, including frame position and frame number information, and uses different colors to represent I-frames, P-frames and the currently selected frame.

8. The apparatus according to claim 6, characterized in that: The frame loss information display area is specifically a frame loss list, which includes the frame position and frame number of each lost frame.

9. The apparatus according to claim 6, characterized in that: The video playback control area includes playback control controls and frame number input controls, as well as frame rate adjustment controls.

10. The apparatus according to claim 6, characterized in that: The process of updating the display content of other areas based on operations performed in the video playback control area, the visualization presentation area, or the frame drop information display area specifically includes: The current frame is determined based on the operations of the video playback control area and the visualization display area, or the adjacent frames of the current frame are determined based on the operations of the frame loss information display area, and then the display content of other areas is made to correspond to the current frame or adjacent frames.

Images