A method or device for quickly displaying video file thumbnails and their information in an embedded system

Abstract

The present invention discloses a method for quickly displaying video file thumbnails and their information in an embedded system. The method comprises the following steps: configuring video file information and a file encapsulation method, recording a video file; synchronously obtaining thumbnail information; after video file recording is completed, calculating another portion of video file information in combination with real-time monitoring information; calculating the contents of a private information header; then adding the private information header, video file information, thumbnail information, and file identifier to the end of the video file in a private information format; and when displaying the video file on an embedded platform graphical interface, directly jumping to the file identifier at the end of the file via a video file pointer, reading and verifying the file identifier, and then quickly calculating an offset to the end of the file based on the information format and the number of bytes occupied by each piece of information, and then reading thumbnail or video file information. This method can smoothly display thumbnails in batches on a graphical interface, and can also quickly obtain video file information individually.

Description

Technical Field

[0001] The present invention belongs to the field of video image information display of embedded systems, and in particular relates to a method or device for quickly displaying video file thumbnails and information thereof in an embedded system. Background Art

[0002] Currently, the main methods for displaying video file thumbnails in embedded system devices are:

[0003] The first type: Some embedded system devices, such as the YW5609 HDMI camera, have limited platform hardware resources. All video files are displayed using the system's built-in ordinary file icons. When 12 or more video files need to be displayed on the same graphical interface, they can only be distinguished by video file name. If users want to quickly understand the general content of the video file through the video file thumbnail, they can only decode and play each video file separately.

[0004] The second method involves some embedded system devices, such as the Hi3516AV100 platform, whose processors lack hardware decoders. While video decoding may not be required for product functionality, video thumbnail display is still required. To address this, a hidden folder can be created on the storage medium. After recording a video, a thumbnail file with the same name as the video file is placed in this hidden folder through some means. Because thumbnail files are small, when a video file thumbnail is needed, the system can quickly retrieve the thumbnail file from the hidden folder and load it for display. However, if the video file is renamed, deleted, or moved, the system background must also synchronize the thumbnail file in the hidden folder. This approach can be prone to confusion and inconvenient to maintain.

[0005] The third method: Some embedded systems with high main processor frequencies, when displaying a video file thumbnail, directly open the video file, search through the file content, parse the first keyframe, copy it to the DDR cache, use a software decoder to extract the full-resolution data, then scale it to a specified smaller resolution before displaying it in the designated location on the graphical interface. This method directly operates on the video file and is clean, but it consumes excessive CPU and DDR resources, resulting in significant delays in thumbnail display and blocking other concurrent tasks on the graphical interface, such as mouse movement lag.

[0006] Prior to the present invention, attempts to display high-definition 4K video file thumbnails in embedded systems using FFmpeg open-source code using a third approach were made. However, there was a delay of approximately 800 milliseconds between clicking a single 4K video file thumbnail and displaying it at the specified 240x136 pixel size on the graphical interface. When displaying multiple video files on the same graphical interface, mouse movement was noticeably lag-free, and each thumbnail was refreshed and displayed at intervals. Prior to the present invention, attempts were also made to improve the system by using three threads to refresh 12 4K video files, with one thread responsible for displaying four 4K video file thumbnails. This resulted in minimal improvement in refresh speed, but increased CPU and DDR resource usage.

[0007] Currently, due to hardware resource limitations, embedded systems rarely display complete video file information on a graphical interface, such as video width, video height, encoding method, data frame rate, frame rate, and video duration. Viewing the video requires inserting the storage medium into a computer, which is very inconvenient.

[0008] In view of the above problems existing in the current method, it is necessary to invent a method for quickly displaying video file information or video file thumbnail information. Summary of the Invention

[0009] The object of the present invention is to provide a method or device for quickly displaying video file thumbnails and their information in an embedded system, which can display video file thumbnails in batches very smoothly on a graphical interface and can also quickly obtain video file information individually.

[0010] The present invention provides the following technical solutions:

[0011] A method for quickly displaying video file thumbnails and information thereof in an embedded system, the method comprising the following steps:

[0012] (1) Configure the video encoding width, encoding height, encoding mode, and file encapsulation mode in the video file information, and record the video file;

[0013] (2) Encode and obtain a frame of low-resolution image data as a thumbnail, and obtain the thumbnail cache address and data length in the thumbnail information;

[0014] (3) After the video file is recorded, the duration, frame rate, and data rate of the video file information are calculated based on the real-time monitoring information; the private information header content is calculated based on the video file information, thumbnail information, and file identifier; and the private information header, video file information, thumbnail information, and file identifier are added to the end of the video file in a private information format;

[0015] (4) When displaying the video file on the embedded platform graphical interface, directly jump to the file identifier at the end of the file through the video file pointer, read and verify the file identifier, and then quickly calculate the offset to the end of the file based on the information format and the number of bytes occupied by each information, and read the video file thumbnail information or video file information.

[0016] In the present invention, the embedded system mainly includes: a real-time image acquisition sensor, a main control chip with built-in video encoder and picture encoder, a storage medium and a Linux operating system.

[0017] Preferably, in the step (1), a video encoding resolution of 4K, 1080P or 720P is adopted, an encoding method of H264 or H265 is adopted, and the file encapsulation method is uniformly MP4 based on the container principle.

[0018] Further preferably, by configuring the video encoding resolution and the video encoding mode, the width value iWidth, the height value iHeight, and the encoding mode value iCodeType in the video file information are obtained; the file header information required for MP4 file encapsulation is initialized using the libavformat library interface in the FFmpeg code; a child thread is created for real-time acquisition and storage of video code streams, and timing and frame number monitoring statistics are enabled.

[0019] In step (1), the video encoding resolution and the video encoding mode can be arbitrarily combined, and MP4 video files with different encoding resolutions and encoding modes can exist in the same embedded platform.

[0020] Preferably, in the step (2), the thumbnail is provided in JPEG encoding mode, and the thumbnail data length is reduced by optimizing PEG parameters and resolution size.

[0021] Further preferably, a frame of YUV data is obtained from the video code stream, scaled and compressed into a small-resolution picture data using a JPEG encoder, and the thumbnail data length iThumbDataLength and the thumbnail data storage address iThumbBuffer are obtained; the JPEG encoding parameters and resolution size are optimized to control the small-resolution picture data size within 5KB.

[0022] In step (2), the thumbnail and video code stream are both derived from the front-end real-time image acquisition system; by adjusting the JPEG parameters and resolution size to reduce the length of the thumbnail data, the subsequent extraction of thumbnail information and the optimization of the decoding and display speed can be facilitated.

[0023] Preferably, in the step (3), the video recording duration is calculated in seconds based on the start and end time of the video recording; the video frame rate is calculated in frames per second based on the number of frames and duration; the data rate is calculated in kilobits per second based on the size and duration of the video file; the number of bytes occupied by each video file information and the number of bytes occupied by the file identifier are agreed upon; and the information required for the MP4 container private information header is calculated based on this information and the length of the thumbnail data.

[0024] Specifically: when the video recording ends, the duration iTime is calculated by using the gettimeofday function at the beginning and end. The number of frames can be obtained from the child thread monitoring information. The file size is calculated by using the stat function. According to the video frame rate = number of frames / duration, the frame rate value iFrameRate is calculated in fps (frames / second); according to the data rate = file size / duration, the data rate value iDateRate is calculated in kbps (kilobits / second).

[0025] According to the MP4 container box principle, a box consists of a box header and a box body. The box body stores the actual information content, namely the video file information, thumbnail information and file identifier. The box header stores the information header information, including box size and box type: the box type represents the box type, which is represented by 4 characters. The private container added here cannot conflict with the public box type naming in the specification. The variable is defined as acBoxType[4]; the box size occupies 4 bytes and stores the number of bytes occupied by the entire container. The variable iBoxSize is defined. First calculate the byte size of the box body iBoxBodySize: iBoxBodySize = iInfoNum*iInfoLength+iThumbDataLength+iThumbLengthInfo+iFileIDSize;

[0026] Where iInfoNum is the number of video file information, iInfoLength is the number of bytes occupied by each video file information, iThumbDataLength is the number of bytes of thumbnail data obtained in step (2), iThumbLengthInfo indicates the number of bytes occupied by the thumbnail data length information storage, and iFileIDSize indicates the number of bytes occupied by the file identifier storage. The file identifier is used to verify whether the video file has private information stored. The variable defined by the file identifier content is acFileID[6]. The value of the box size can be calculated as: iBoxSize = iBoxBodySize + 8; the number 8 represents the number of bytes occupied by the box header itself.

[0027] The container size value iBoxSize, the container type value acBoxType[4], the width iWidth, height iHeight, encoding method iCodeType, data rate iDateRate, frame rate iFrameRate, duration iTime, thumbnail data storage address iThumbBuffer, thumbnail data length iThumbDataLength, and file identifier content acFileID[6] in the private information header are clearly defined. Then, the private information format can be written to the end of the file through the video file pointer.

[0028] Preferably, in the step (4), after reading and verifying the file identifier, the thumbnail data storage address offset is calculated by reading the thumbnail data length stored in front of the file identifier, and the video file pointer quickly jumps and reads the video file thumbnail information.

[0029] Specifically: when displaying the thumbnail of a video file, directly jump the file pointer to the position at the end of the file where the offset is the number of bytes iFileIDSize occupied by the file identifier storage, read the file identifier, and verify whether the video file is recorded by this embedded platform. If the verification fails, use the system's built-in ordinary file icon data instead; if the verification passes, you can directly read the iThumbLengthInfo byte thumbnail data length stored at the address before the file identifier, calculate the offset iOffset = iThumbDataLength + iThumbLengthInfo + iFileIDSize of the actual thumbnail data to the end of the file, quickly jump to the offset address iOffset, and read the iThumbDataLength byte thumbnail data, and send it to the graphical interface for decoding and display.

[0030] Preferably, in the step (4), after reading and verifying the file identifier, the offset from the first video file information to the end of the file is calculated by reading the thumbnail data length, combining the number of video file information and the number of bytes occupied by each video file information, and directly jumping to and reading the video file information.

[0031] Specific location: When obtaining the video file information, calculate the iBoxBodySize value in step (3). This value is the address offset of the first video file information. Jump directly to this address and read iInfoNum*iInfoLength bytes at one time, that is, all video file information. Then, according to the agreed storage order, select the required video file information from the read information.

[0032] Preferably, in step (4), when it is necessary to display multiple video file thumbnails in a graphical interface, step (4) can be repeated to decode and display the thumbnail information in different positions of the graphical interface.

[0033] In the present invention, thumbnails can be quickly displayed in batches by repeating step (4). Combined with the actual embedded platform performance, it can be further optimized to batch refresh and display thumbnails of multiple video files using multiple threads.

[0034] The present invention also provides a device for quickly displaying video file thumbnails and their information in an embedded system, the device comprising a memory and one or more processors; the one or more processors execute one or more programs in the memory to implement the method for quickly displaying video file thumbnails and their information in the embedded system.

[0035] Compared with the prior art, the method or device for quickly displaying video file thumbnails and their information in an embedded system proposed by the present invention integrates the video file information and thumbnail information into the video file after recording the video, making reading more direct and fast. It can display video file thumbnails in batches very smoothly on a graphical interface, and can also quickly obtain video file information individually; the information extraction speed is fast, suitable for the rapid display of massive video files, convenient for users to query or select, and greatly saves platform hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a diagram of the private information format;

[0037] Figure 2 Add private information flow chart to video files;

[0038] Figure 3 Flowchart for parsing private information for video files;

[0039] Figure 4 Display video file thumbnails in batches for the graphical interface;

[0040] Figure 5 Displays detailed information of video files in a graphical interface. DETAILED DESCRIPTION

[0041] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0043] As a possible embodiment of the present invention: the embedded system processor is an ARM core 1.5GHz, 1GB DDR memory; the video stream resolution collected by the front end is 3840x2160 (4K), the frame rate is 60FPS, the encoding method is H264, the video encapsulation format is MP4, and the storage medium is a high-speed CLASS10 level SD card with a capacity of 32GB.

[0044] S1. Record video files, obtain thumbnail information, and generate video files with private information

[0045] The process of recording video files is mainly divided into two function interface calls: the recording start interface and the recording end interface. The recording start interface includes the following steps: video recording initialization, video recording thread creation, and thumbnail data acquisition; the recording end interface includes the following steps: video file information calculation, private information header and file identifier configuration, and private information writing operation.

[0046] The following describes these two interface implementations.

[0047] The recording start interface performs the following operations:

[0048] like Figure 2 As shown in Figure 21, the video encoder resolution is configured to 3840x2160, resulting in width variables iWidth = 3840 and height variables iHeight = 2160. The encoding method is configured to H.264, resulting in encoding method variable iCodeType = 0. A file named VID_0013 is created on the SD card storage medium. The libavformat library interface in FFmpeg is used to initialize the file header information required for the MP4 encapsulation format. The video statistics variables are initialized: duration iTime and frame count iFrame.

[0049] Set the video flag and create a sub-thread synchronously. Figure 2 As shown in 22, it is used to obtain and save the video stream in real time and enable monitoring statistics of timing and frame number.

[0050] After the child thread is created, the recording start interface immediately obtains a frame of YUV format data from the front-end video stream, generates a frame of thumbnail data with a resolution of 240x136 through the JPEG encoder, and temporarily stores it in the iThumbBuffer dynamically allocated by the malloc function. The thumbnail data length is iThumbDataLength.

[0051] The child thread will poll the video recording flag in real time. If the flag is valid, it will obtain a frame of H264 encoded data, and immediately save it to the SD card file name VID_0013 after processing. The frame count value iFrame will be increased by 1. When entering the child thread, the gettimeofday function will be used to obtain the video recording start time. When exiting the child thread, the gettimeofday function will be used to obtain the video recording end time. The difference between the two can be used to calculate the duration iTime information.

[0052] The recording end interface performs the following operations:

[0053] like Figure 2 As shown in 23, the recording flag is reset. When the child thread polls and finds the flag invalid, it terminates the video storage process and exits. The libavformat library interface in FFmpeg is called to save and close the MP4 file, generating a standard MP4 file. Based on the statistics of the global variables iTime and iFrame in the child thread, the file size iFileSize is obtained using the stat function. The video frame rate is calculated using iFrameRate = iFrame / iTime and converted to fps (frames per second). The data rate is calculated using iDataRate = iFileSize / iTime and converted to kbps (kilobits per second).

[0054] In this embodiment, the file identifier acFileID[6] = {0x74, 0x70, 0x78, 0x63, 0x61, 0x6d}, which is the ASCII code of the letters tpxcam, occupies 6 bytes.

[0055] In this embodiment, the stored private information does not need to be parsed and obtained by the standard player. Instead, a private container type needs to be defined. The container type acBoxType[4] = {0x74, 0x68, 0x75, 0x6d}, which is the ASCII code of "thum", occupies 4 bytes. The number of bytes of the container content iBoxBodySize can be calculated by the following simple formula:

[0056] iBoxBodySize=iInfoNum*iInfoLength+iThumbDataLength+iThumbLengthInfo+iFileIDSize;

[0057] In this embodiment, iInfoNum=6 means that 6 video file information is stored; iInfoLength=4 means that each video file information occupies 4 bytes; iThumbDataLength is the actual number of bytes of thumbnail data; iThumbLengthInfo=4 means that the storage of thumbnail data length information occupies 4 bytes; iFileIDSize=6 means that the storage of file identifier occupies 6 bytes. After knowing the number of bytes of container content iBoxBodySize, the total size of the container can be calculated by iBoxSize=iBoxBodySize+8. In this way, all the information of the private information header is clear, and it can be calculated according to Figure 1 The private information content is written into the information format, the recording end interface call is completed, and a video file with private information is generated.

[0058] S2. Video file thumbnail information analysis

[0059] like Figure 3 As shown in the figure, after opening the specified MP4 video file, the fseek function is used to jump directly to the address with an offset of 6 at the end of the file, and the 6-byte file identifier is read and compared with the content {0x74, 0x70, 0x78, 0x63, 0x61, 0x6d}. If the file identifiers are inconsistent, it can be confirmed that the video file is not the MP4 video file recorded by this embedded platform. If the verification fails, the reading is directly exited and the graphical interface will use the unified file icon in the file system instead of the display; if the file identifiers are consistent, such as Figure 1 As shown in the private information format, the thumbnail data length information is stored before the file identifier. The file pointer can be jumped to the offset of iThumbLengthInfo + iFileIDSize (4 + 6 in this embodiment) to read the thumbnail length information iThumbDataLength. Once the thumbnail length information is known, the thumbnail data storage offset can be calculated as iThumbLengthInfo + iFileIDSize + iThumbDataLength. The file pointer jumps to this offset address to read the thumbnail data of length iThumbDataLength. The read data is directly placed in the cache buffer created by the graphical interface, facilitating fast decoding and display. In this embodiment, the iThumbDataLength of each video thumbnail data is generally within 5KB.

[0060] like Figure 4 As shown, 12 video file thumbnails are displayed on a graphical interface. Just repeat this step and the display can be quickly refreshed within 1 second.

[0061] S3, video file information analysis

[0062] If you only want to obtain video file information, as in step S2, first verify the file identifier. After verification, read the thumbnail data length information iThumbDataLength and calculate Figure 1 The address offset iOffset of the width of the first video file information is shown:

[0063] iOffset=iInfoNum*iInfoLength+iThumbDataLength+iThumbLengthInfo+iFileIDSize;

[0064] In this embodiment, iInfoNum=6 is the number of video file information; iInfoLength=4 is the number of bytes occupied by each video file information, iThumbDataLength is the actual number of bytes of thumbnail data; iThumbLengthInfo=4 means that the storage of thumbnail data length information occupies 4 bytes; iFileIDSize=6 means that the storage of file identifier occupies 6 bytes. After calculating the iOffset value, the file pointer jumps to iOffset and reads iInfoNum*iInfoLength=24 bytes of information. This information includes the following: Figure 1 As shown in the figure: width, height, encoding method, data rate, frame rate, duration, a total of 6 video file information. The code can be obtained and displayed according to the storage order, the display effect is as follows Figure 5 The file size can be quickly obtained using the stat function and is not stored in the video file.

[0065] In summary, the present invention stores video file information, thumbnail information and file identifier when recording a video file. When the video file thumbnail and its information need to be obtained, they can be obtained through rapid verification and analysis.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for quickly displaying video file thumbnails and their information in an embedded system, characterized in that: The method comprises the following steps: (1) Configure the video encoding width, encoding height, and encoding method in the video file information, and uniformly use the MP4 based on the container principle for file encapsulation to record the video file; (2) Encode and obtain a frame of low-resolution image data as a thumbnail, and obtain the thumbnail cache address and data length in the thumbnail information; (3) After the video file is recorded, the duration, frame rate, and data rate of the video file information are calculated in combination with the real-time monitoring information; the private information header content is calculated based on the video file information, thumbnail information, and the custom file identifier; finally, the private information is added to the end of the video file. The storage order of the private information format is private information header, video file information, thumbnail information, and file identifier; (4) When displaying the video file on the embedded platform graphical interface, directly jump to the file identifier at the end of the file through the video file pointer, read and verify the file identifier, and then quickly calculate the offset to the end of the file based on the information format and the number of bytes occupied by each information, and read the video file thumbnail information or video file information; The video file information includes encoding width, encoding height, encoding mode, data rate, frame rate and duration; In step (3), the video recording duration is calculated in seconds based on the start and end time of the video recording; the video frame rate is calculated in frames per second based on the number of frames and duration; the data rate is calculated in kilobits per second based on the size and duration of the video file; the number of bytes occupied by each video file information and the number of bytes occupied by the file identifier are agreed upon; based on this information and the length of the thumbnail data, the information required for the MP4 container private information header is calculated; the private information header includes the container size and the container type; In step (4), when displaying the video file thumbnail information, after reading and verifying the file identifier, the thumbnail data storage address offset is calculated by reading the thumbnail data length stored before the file identifier, and the video file pointer quickly jumps to read the video file thumbnail information; In step (4), when displaying the video file information, after reading and verifying the file identifier, the offset from the first video file information to the end of the file is calculated by reading the thumbnail data length, combining the number of video file information and the number of bytes occupied by each video file information, and directly jumping to and reading the video file information.

2. The method for quickly displaying video file thumbnails and their information in an embedded system according to claim 1, characterized in that: In the step (1), a video encoding resolution of 4K, 1080P or 720P is adopted, and an encoding method of H264 or H265 is adopted.

3. The method for quickly displaying video file thumbnails and their information in an embedded system according to claim 2, characterized in that: By configuring the video encoding resolution and video encoding method, the width, height, and encoding method values ​​in the video file information are obtained; the libavformat library interface in the FFmpeg code is used to initialize the file header information required for MP4 file encapsulation; a child thread is created for real-time acquisition and storage of video streams, and timing and frame count monitoring and statistics are enabled.

4. The method for quickly displaying video file thumbnails and their information in an embedded system according to claim 1, characterized in that: In the step (2), the thumbnail is provided in JPEG encoding mode, and the thumbnail data length is reduced by optimizing JPEG parameters and resolution size.

5. The method for quickly displaying video file thumbnails and their information in an embedded system according to claim 4, characterized in that: Get a frame of YUV data from the video stream, scale and compress it into a small-resolution image using the JPEG encoder, and obtain the thumbnail data length and thumbnail data storage address; optimize the JPEG encoding parameters and resolution size to keep the small-resolution image data size within 5KB.

6. The method for quickly displaying video file thumbnails and their information in an embedded system according to claim 1, characterized in that: In the step (4), when it is necessary to display multiple video file thumbnails in a graphical interface, step (4) can be repeated to decode and display the thumbnail information in different positions of the graphical interface.

7. A device for quickly displaying video file thumbnails and their information in an embedded system, characterized in that: The device includes a memory and one or more processors; the one or more processors execute one or more programs in the memory to implement the method for quickly displaying video file thumbnails and their information in an embedded system as described in any one of claims 1-6.

Images