Method and apparatus for generating auxiliary frame file, storage medium and electronic device

Abstract

The application discloses a kind of generation method and device of auxiliary frame file, storage medium, electronic device, above-mentioned method includes: video file is analyzed, obtains the auxiliary frame sequence corresponding to the video file;Determine the corresponding relationship of auxiliary frame in the auxiliary frame sequence and the video frame in the video file;According to the corresponding relationship generates auxiliary frame file and the index file of the auxiliary frame file, wherein the index file is used to find auxiliary frame in the auxiliary frame file;Using the above scheme, the video file is analyzed intelligently in the related art, and the file containing all video frames of the video file generated and the auxiliary frame generated is stored in hard disk, so that the video frame of the video file is stored twice, and the problem of increasing IO overhead of hard disk is solved.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to a method and apparatus for generating auxiliary frame files, a storage medium, and an electronic device. Background Technology

[0002] In the field of video surveillance, intelligent applications are becoming increasingly common. Retrieving video from ordinary recordings stored on other conventional monitoring devices is inefficient and time-consuming. Therefore, it is necessary to import the recordings into intelligent analysis devices for secondary intelligent analysis. After intelligent analysis, the client can retrieve the intelligent stream for playback. The playback stream contains video frames and auxiliary frames (rule frames, event frames, and trajectory frames). Based on these auxiliary frames, the client can display intelligent rules, intelligent tracking boxes, and detection attributes, among other intelligent information, on the screen.

[0003] However, in existing technologies, after intelligent analysis, the original video file and intelligent auxiliary frame data need to be written to the hard drive together. Since the original video file is already stored on the hard drive, repeatedly storing the original video file increases disk I / O overhead and affects the storage performance of the intelligent device. At the same time, many of the auxiliary frames after intelligent analysis are the same, and they need to be repeatedly inserted into different frame sequences, resulting in data redundancy.

[0004] In related technologies, after intelligent analysis of video files, a file containing all video frames and auxiliary frames of the video file is stored on the hard drive. This results in the video frames of the video file being stored twice, increasing the I / O overhead of the hard drive. No effective solution has yet been proposed. Summary of the Invention

[0005] This invention provides a method and apparatus for generating auxiliary frame files, a storage medium, and an electronic device to at least solve the problem in related technologies where, after intelligent analysis of a video file, a file containing all video frames of the video file and the generated auxiliary frames is stored on a hard disk, resulting in secondary storage of the video frames of the video file and increased hard disk I / O overhead.

[0006] According to one embodiment of the present invention, a method for generating an auxiliary frame file is provided, comprising: analyzing a video file to obtain an auxiliary frame sequence corresponding to the video file; determining the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the video file; generating an auxiliary frame file and an index file of the auxiliary frame file according to the correspondence, wherein the index file is used to search for auxiliary frames in the auxiliary frame file.

[0007] In an exemplary embodiment, before determining the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the recording file, the method further includes: parsing the index information of the video frames from the frame headers of the video frames stored in the recording file, wherein the index information includes at least one of the following: the first frame sequence number of the video frame, the first timestamp of the video frame, and the first file offset of the video frame; and storing the index information in an index database.

[0008] In an exemplary embodiment, generating an auxiliary frame file based on the correspondence includes: determining video frames corresponding to auxiliary frames in the auxiliary frame sequence from the index database according to the correspondence, thereby obtaining multiple video frames; storing the first frame sequence number of each of the multiple video frames in the frame header of each of the multiple auxiliary frames corresponding to the multiple video frames, wherein the frame header of the auxiliary frames is further used to store the second frame sequence number and the frame type of the auxiliary frames; and generating the auxiliary frame file based on the finally obtained multiple auxiliary frames.

[0009] In an exemplary embodiment, generating an index file for auxiliary frame files based on the correspondence includes: obtaining a plurality of identical first auxiliary frames in the auxiliary frame sequence; determining a third frame sequence number of a plurality of video frames corresponding to the plurality of first auxiliary frames based on the correspondence; and generating an index file for the auxiliary frame files based on the third frame sequence number, a second auxiliary frame, a second timestamp of the second auxiliary frame, and a second file offset of the second auxiliary frame, wherein the plurality of first auxiliary frames includes: the second auxiliary frame, and the second timestamp of the second auxiliary frame is the minimum value among the third timestamps of the plurality of first auxiliary frames.

[0010] In an exemplary embodiment, after generating an auxiliary frame file and an index file of the auxiliary frame file according to the correspondence, the method further includes: upon receiving a video playback task, determining a target timestamp from the index information of the video frames stored in the recording file according to the start time indicated by the video playback task; determining the target file offset of the target video frame corresponding to the target timestamp, and reading the video frame in the recording file according to the target file offset; and sending the read video frame to complete the video playback task.

[0011] In one exemplary embodiment, sending a read video frame includes: sequentially parsing out the fourth frame number of the read video frame; searching the index file for a fifth frame number corresponding to the fourth frame number; if the fifth frame number is found, sending a third auxiliary frame corresponding to the fifth frame number; and if the third auxiliary frame is sent, sending the video frame corresponding to the fourth frame number.

[0012] In an exemplary embodiment, before sending the third auxiliary frame corresponding to the fifth frame sequence number, the method further includes: obtaining a fourth timestamp and a third file offset corresponding to the fifth frame sequence number from the index file; and obtaining the third auxiliary frame from the auxiliary frame file according to the fourth timestamp and the third file offset.

[0013] According to another embodiment of the present application, an auxiliary frame file generation apparatus is also provided, comprising: an analysis module for analyzing a video file to obtain an auxiliary frame sequence corresponding to the video file; a determination module for determining the correspondence between auxiliary frames in the auxiliary frame sequence and video frames in the video file; and a generation module for generating an auxiliary frame file and an index file of the auxiliary frame file according to the correspondence, wherein the index file is used to search for auxiliary frames in the auxiliary frame file.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the above-described method for generating auxiliary frame files when running.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described method for generating auxiliary frame files through the computer program.

[0016] In this embodiment, the video file is first analyzed to obtain the corresponding auxiliary frame sequence; then, the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the video file is determined; finally, an auxiliary frame file and an index file of the auxiliary frame file are generated based on the determined correspondence, wherein the index file is used to find auxiliary frames in the auxiliary frame file. By adopting the above technical solution, by generating an auxiliary frame file separately after intelligent analysis of the video file, the video frames of the video file are avoided from being saved repeatedly. This solves the problem in related technologies where, after intelligent analysis of the video file, the generated file containing all the video frames of the video file and the generated auxiliary frames is stored on the hard disk, resulting in secondary storage of the video frames of the video file and increased hard disk I / O overhead. This achieves the technical effect of reducing hard disk I / O overhead and improving the storage performance of smart devices. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a hardware structure block diagram of a camera device for an optional auxiliary frame file generation method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating an optional method for generating auxiliary frame files according to an embodiment of this application.

[0021] Figure 3 This is a schematic diagram of the structure of an optional auxiliary frame header according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the structure of an optional index file according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of an optional intelligent bitstream merging and conversion embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the structure of an optional intelligent analysis system for video files according to an embodiment of this application;

[0025] Figure 7 This is a structural block diagram of an auxiliary frame file generation system according to an embodiment of this application. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing system. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a method of generating auxiliary frame files according to an embodiment of this application. Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the image. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. In one exemplary embodiment, the camera device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned computer terminal. For example, the camera device may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 Equivalent functions or ratios shown Figure 1 The functions shown have more different configurations.

[0029] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the auxiliary frame file generation method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage systems, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a secure text network via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the camera equipment's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet.

[0031] This embodiment provides a method for generating auxiliary frame files, applied to the aforementioned computer terminal. Figure 2 This is a flowchart of an optional auxiliary frame file generation method according to an embodiment of this application, the process including the following steps:

[0032] Step S202: Analyze the video file to obtain the auxiliary frame sequence corresponding to the video file;

[0033] Step S204: Determine the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the video recording file;

[0034] It should be noted that the above auxiliary frames correspond one-to-one with the video frames, and the auxiliary frames should be inserted before the corresponding video frames.

[0035] Step S206: Generate an auxiliary frame file and an index file of the auxiliary frame file according to the correspondence, wherein the index file is used to find the auxiliary frame in the auxiliary frame file.

[0036] Through the above steps, the video file is first analyzed to obtain the corresponding auxiliary frame sequence; then, the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the video file is determined; finally, an auxiliary frame file and an index file of the auxiliary frame file are generated based on the determined correspondence, wherein the index file is used to find auxiliary frames in the auxiliary frame file. By using the above technical solution, by generating an auxiliary frame file separately after intelligent analysis of the video file, the video frames of the video file are avoided from being saved repeatedly. This solves the problem in related technologies where, after intelligent analysis of the video file, a file containing all the video frames of the video file and the generated auxiliary frames is stored on the hard disk, resulting in secondary storage of the video frames of the video file and increased hard disk I / O overhead. This achieves the technical effect of reducing hard disk I / O overhead and improving the storage performance of smart devices.

[0037] Optionally, before performing the above-described determination step S204: determining the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the recording file, the method further includes: parsing the index information of the video frames from the frame header of the video frames saved in the recording file, wherein the index information includes at least one of the following: the first frame sequence number of the video frame, the first timestamp of the video frame, and the first file offset of the video frame; and storing the index information in an index database.

[0038] Before determining the correspondence between auxiliary frames and video frames, it is necessary to first parse the index information of the video frames. After receiving the video file to be intelligently analyzed, it is first stored on the device's hard drive. During the storage process, the frame header of the video file bitstream, i.e. the frame header of the video frame, is parsed out. The first frame sequence number, first timestamp, and first file offset of the video frame are parsed out from the frame header. The index information of the video file is formed and stored in the index database.

[0039] Optionally, the above generation step S206: generating an auxiliary frame file according to the correspondence can be implemented by the following steps: determining the video frame corresponding to the auxiliary frame in the auxiliary frame sequence from the index database according to the correspondence, thereby obtaining multiple video frames; storing the first frame number of each of the multiple video frames in the frame header of the multiple auxiliary frames corresponding to the multiple video frames, wherein the frame header of the auxiliary frame is also used to store the second frame number and the frame type of the auxiliary frame; and generating the auxiliary frame file based on the finally obtained multiple auxiliary frames.

[0040] Based on the established correspondence, multiple video frames corresponding to the auxiliary frames in the auxiliary frame sequence are determined from the index database. Then, the first frame number of each of these video frames is stored in the frame header of the corresponding auxiliary frame. The frame header of the auxiliary frame also stores the second frame number and the frame type of the auxiliary frame. Finally, the auxiliary frame file is generated based on the multiple auxiliary frames whose first frame numbers are stored in their frame headers. The structure of the auxiliary frame storing the first frame number is as follows: Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of an optional auxiliary frame header according to an embodiment of this application. The frame header of the auxiliary frame includes the frame sequence number of the auxiliary frame to identify the auxiliary frame, and also includes the frame sequence number of the corresponding video frame to indicate the position where the auxiliary frame should be inserted, as well as the frame type. The frame types include three types: regular frames, trajectory frames, and event frames, which are used to distinguish different categories of auxiliary frames.

[0041] It should be noted that the above correspondence can be determined based on the first timestamp of the video frame and the timestamp of the auxiliary frame. It is understood that the timestamp of the auxiliary frame indicates the generation time of the auxiliary frame during the intelligent analysis of the recorded file. Therefore, the timestamps of the video frame and its corresponding auxiliary frame should be the same. Thus, the correspondence between the video frame and the auxiliary frame can be determined based on the timestamps. The correspondence can also be determined in other ways, such as automatically saving the frame number of the corresponding video frame in the frame header of the auxiliary frame when it is generated; this application does not impose any limitations on this.

[0042] Optionally, the above generation step S206: generating an index file for the auxiliary frame file according to the correspondence can be implemented through the following steps, specifically including: obtaining multiple identical first auxiliary frames in the auxiliary frame sequence; determining the third frame sequence number of multiple video frames corresponding to the multiple first auxiliary frames according to the correspondence; generating the index file for the auxiliary frame file according to the third frame sequence number, the second auxiliary frame, the second timestamp of the second auxiliary frame, and the second file offset of the second auxiliary frame, wherein the multiple first auxiliary frames include: the second auxiliary frame, and the second timestamp of the second auxiliary frame is the minimum value among the third timestamps of the multiple first auxiliary frames.

[0043] Auxiliary frame data includes rule frame data, trajectory frame data, and event frame data. Rule frame data remains unchanged when the intelligent rules do not change. Typical video frames are 30 frames per second, and many trajectory frames are identical when an object is moving slowly or stationary. Furthermore, event frames change synchronously with trajectory frames and contain many object attributes, resulting in many identical frames. Therefore, to avoid consuming excessive disk memory with a large amount of duplicate auxiliary frame data, duplicate first auxiliary frames in the auxiliary frame sequence can be removed. The specific method includes: first, obtaining multiple identical first auxiliary frames from the auxiliary frame sequence; then, determining the third frame sequence number of the multiple video frames corresponding to these first auxiliary frames based on the correspondence; then, generating an index file for the auxiliary frame file based on the third frame sequence number, the second auxiliary frame, the second timestamp of the second auxiliary frame, and the second file offset of the second auxiliary frame; wherein the second auxiliary frame belongs to the multiple first auxiliary frames, and the second timestamp of the second auxiliary frame is the minimum value among the third timestamps of the multiple first auxiliary frames. The final index file is as follows: Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of an optional index file according to an embodiment of this application. For each auxiliary frame that is finally saved, it can correspond to one or more original bitstream frame numbers to be inserted (equivalent to the frame numbers of the video frames mentioned above), and also saves the data offset of the auxiliary frame in the auxiliary frame file and the UTC timestamp of the auxiliary frame.

[0044] For example, for 30 frames of data in the same second, the object may not have moved in the video frame. Therefore, the auxiliary frames (such as trajectory frames) corresponding to these 30 video frames are all the same. Therefore, when generating the index file, only the first frame of the 30 auxiliary frames corresponding to these 30 video frames is saved, that is, the first auxiliary frame generated. However, the frame numbers of the 30 video frames corresponding to these 30 auxiliary frames are all saved together with the auxiliary frames. In this way, the correct association between the video frames and the auxiliary frames can still be known through the index file.

[0045] It should be noted that the above method for generating index files can also be used to filter and merge multiple identical and adjacent auxiliary frame data, and this application does not limit this.

[0046] Optionally, after generating the index file of the auxiliary frame file through the above scheme, since the duplicate auxiliary frame data has been processed, the first auxiliary frames other than the second auxiliary frame can be deleted from the multiple first auxiliary frames to reduce the occupation of hard disk memory.

[0047] In an exemplary embodiment, after performing the above-described generation step S206: generating an auxiliary frame file and an index file of the auxiliary frame file according to the correspondence, the method further includes: upon receiving a video playback task, determining a target timestamp from the index information of the video frames stored in the recording file according to the start time indicated by the video playback task; determining the target file offset of the target video frame corresponding to the target timestamp, and reading the video frame in the recording file according to the target file offset; and sending the read video frame to complete the video playback task.

[0048] If a user wants to play back the video stream, they can select the recording file to be played back and the time period to be played back at the front end, thus issuing a video playback task. The start time is determined from the time period indicated by the video playback task, and the target timestamp is determined from the index information of the video frames saved in the recording file based on this start time. Then, the target file offset of the target video frame corresponding to the target timestamp is determined. Based on the target file offset, the storage location of the video to be played back in the recording file is determined, and the video frame is read. Finally, the read video frame is sent to complete the video playback task.

[0049] In one exemplary embodiment, sending a read video frame includes: sequentially parsing out the fourth frame number of the read video frame; searching the index file for a fifth frame number corresponding to the fourth frame number; if the fifth frame number is found, sending a third auxiliary frame corresponding to the fifth frame number; and if the third auxiliary frame is sent, sending the video frame corresponding to the fourth frame number.

[0050] During the transmission of the read video frames, the fusion of video frames and auxiliary frames also needs to be completed. The specific steps include: first, parsing out the fourth frame number of each read video frame sequentially; then, searching the index file for a fifth frame number corresponding to that fourth frame number; if a fifth frame number is found, it indicates the existence of an auxiliary frame corresponding to that video frame. Therefore, before sending the video frame, the third auxiliary frame corresponding to the fifth frame number needs to be sent, and after that, the video frame corresponding to the fourth frame number is sent; if the fifth frame number is not found, the video frame corresponding to the fourth frame number is sent directly. This transmission method allows the video frames and auxiliary frames to be fused during transmission. The above process is as follows: Figure 5 As shown, Figure 5 This is a schematic diagram of an optional intelligent stream merging and conversion embodiment of this application.

[0051] Optionally, before performing the above sending steps: before sending the third auxiliary frame corresponding to the fifth frame sequence number, the method further includes: obtaining a fourth timestamp and a third file offset corresponding to the fifth frame sequence number from the index file; and obtaining the third auxiliary frame from the auxiliary frame file according to the fourth timestamp and the third file offset.

[0052] Before sending the third auxiliary frame corresponding to the fifth frame sequence number, the following steps are also included: obtaining the fourth timestamp and the third file offset corresponding to the fifth frame sequence number from the index file; determining the storage location of the third auxiliary frame in the auxiliary frame file based on the fourth timestamp and the third file offset, thereby obtaining the third auxiliary frame.

[0053] This embodiment provides an optional intelligent analysis system for video files. Figure 6 This is a schematic diagram of the structure of an optional intelligent analysis system for video files in an embodiment of this application, such as... Figure 6 As shown, the intelligent analysis system includes:

[0054] WEB Display Module: Used for users to import video recordings and enable intelligent video analysis tasks.

[0055] Intelligent Analysis Module: Used to read video data from the storage module and perform intelligent analysis.

[0056] Storage module: Used to store video files imported from the WEB presentation module.

[0057] Database module: Used to store the index of video files and the intelligent analysis results of the intelligent analysis module.

[0058] The intelligent analysis of video files using the above modules includes the following steps:

[0059] Step 1: Import the video file from the WEB display module and save it to the storage module;

[0060] Step 2: The storage module creates the index information for the video files and stores it in the database module;

[0061] Step 3: The user starts the intelligent video analysis task through the WEB display module;

[0062] Step 4: The intelligent analysis module reads the video file data to be analyzed from the storage module;

[0063] Step 5: The intelligent analysis module generates separate auxiliary frame files from the intelligent auxiliary frame data obtained from the analysis.

[0064] Step 6: Insert the obtained intelligent analysis results into the database.

[0065] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0066] This embodiment also provides an auxiliary frame file generation apparatus, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0067] Figure 7 This is a structural block diagram of an auxiliary frame file generation apparatus according to an embodiment of this application; as shown below. Figure 7 As shown, it includes:

[0068] Analysis module 72 is used to analyze the video file to obtain the auxiliary frame sequence corresponding to the video file;

[0069] The determining module 74 is used to determine the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the video recording file;

[0070] It should be noted that the above auxiliary frames correspond one-to-one with the video frames, and the auxiliary frames should be inserted before the corresponding video frames.

[0071] The generation module 76 is used to generate an auxiliary frame file and an index file of the auxiliary frame file according to the correspondence, wherein the index file is used to find the auxiliary frame in the auxiliary frame file.

[0072] The aforementioned device first analyzes the video file to obtain the corresponding auxiliary frame sequence. Then, it determines the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the video file. Finally, it generates an auxiliary frame file and an index file based on the determined correspondence. The index file is used to find auxiliary frames within the auxiliary frame file. By generating an auxiliary frame file separately after intelligent analysis of the video file, the device avoids repeatedly saving the video frames of the video file. This solves the problem in related technologies where intelligent analysis of a video file results in the generation of a file containing all the video frames and the generated auxiliary frames, which is then stored on the hard drive, leading to secondary storage of the video frames and increased hard drive I / O overhead. This solution achieves the technical effect of reducing hard drive I / O overhead and improving the storage performance of smart devices.

[0073] In an exemplary embodiment, the determining module 74 is further configured to, before determining the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the recording file, parse the index information of the video frames from the frame header of the video frames stored in the recording file, wherein the index information includes at least one of the following: the first frame sequence number of the video frame, the first timestamp of the video frame, and the first file offset of the video frame; and store the index information in an index database.

[0074] Before determining the correspondence between auxiliary frames and video frames, it is necessary to first parse the index information of the video frames. After receiving the video file to be intelligently analyzed, it is first stored on the device's hard drive. During the storage process, the frame header of the video file bitstream, i.e. the frame header of the video frame, is parsed out. The first frame sequence number, first timestamp, and first file offset of the video frame are parsed out from the frame header. The index information of the video file is formed and stored in the index database.

[0075] In an exemplary embodiment, the generation module 76 is further configured to determine, according to the correspondence, the video frame corresponding to the auxiliary frame in the auxiliary frame sequence from the index database to obtain a plurality of video frames; respectively store the first frame sequence number of the plurality of video frames in the frame header of the plurality of auxiliary frames corresponding to the plurality of video frames, wherein the frame header of the auxiliary frame is further configured to store the second frame sequence number of the auxiliary frame and the frame type of the auxiliary frame; and generate the auxiliary frame file according to the finally obtained plurality of auxiliary frames.

[0076] Based on the established correspondence, multiple video frames corresponding to the auxiliary frames in the auxiliary frame sequence are determined from the index database. Then, the first frame number of each of these video frames is stored in the frame header of the corresponding auxiliary frame. The frame header of the auxiliary frame also stores the second frame number and the frame type of the auxiliary frame. Finally, the auxiliary frame file is generated based on the multiple auxiliary frames whose first frame numbers are stored in their frame headers. The structure of the auxiliary frame storing the first frame number is as follows: Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of an optional auxiliary frame header according to an embodiment of this application. The frame header of the auxiliary frame includes the frame sequence number of the auxiliary frame to identify the auxiliary frame, and also includes the frame sequence number of the corresponding video frame to indicate the position where the auxiliary frame should be inserted, as well as the frame type. The frame types include three types: regular frames, trajectory frames, and event frames, which are used to distinguish different categories of auxiliary frames.

[0077] It should be noted that the above correspondence can be determined based on the first timestamp of the video frame and the timestamp of the auxiliary frame. It is understood that the timestamp of the auxiliary frame indicates the generation time of the auxiliary frame during the intelligent analysis of the recorded file. Therefore, the timestamps of the video frame and its corresponding auxiliary frame should be the same. Thus, the correspondence between the video frame and the auxiliary frame can be determined based on the timestamps. The correspondence can also be determined in other ways, such as automatically saving the frame number of the corresponding video frame in the frame header of the auxiliary frame when it is generated; this application does not impose any limitations on this.

[0078] In an exemplary embodiment, the generation module 76 is further configured to: acquire multiple identical first auxiliary frames in the auxiliary frame sequence; determine the third frame sequence number of multiple video frames corresponding to the multiple first auxiliary frames according to the correspondence; and generate an index file of the auxiliary frame file according to the third frame sequence number, the second auxiliary frame, the second timestamp of the second auxiliary frame, and the second file offset of the second auxiliary frame, wherein the multiple first auxiliary frames include: the second auxiliary frame, and the second timestamp of the second auxiliary frame is the minimum value among the third timestamps of the multiple first auxiliary frames.

[0079] Auxiliary frame data includes rule frame data, trajectory frame data, and event frame data. Rule frame data remains unchanged when the intelligent rules do not change. Typical video frames are 30 frames per second, and many trajectory frames are identical when an object is moving slowly or stationary. Furthermore, event frames change synchronously with trajectory frames and contain many object attributes, resulting in many identical frames. Therefore, to avoid consuming excessive disk memory with a large amount of duplicate auxiliary frame data, duplicate first auxiliary frames in the auxiliary frame sequence can be removed. The specific method includes: first, obtaining multiple identical first auxiliary frames from the auxiliary frame sequence; then, determining the third frame sequence number of the multiple video frames corresponding to these first auxiliary frames based on the correspondence; then, generating an index file for the auxiliary frame file based on the third frame sequence number, the second auxiliary frame, the second timestamp of the second auxiliary frame, and the second file offset of the second auxiliary frame; wherein the second auxiliary frame belongs to the multiple first auxiliary frames, and the second timestamp of the second auxiliary frame is the minimum value among the third timestamps of the multiple first auxiliary frames. The final index file is as follows: Figure 4 As shown, Figure 4 This is a schematic diagram of an optional index file according to an embodiment of this application. For each auxiliary frame that is finally saved, it can correspond to one or more original bitstream frame numbers to be inserted (equivalent to the frame numbers of the video frames mentioned above), and also saves the data offset of the auxiliary frame in the auxiliary frame file and the UTC timestamp of the auxiliary frame.

[0080] For example, for 30 frames of data in the same second, the object may not have moved in the video frame. Therefore, the auxiliary frames (such as trajectory frames) corresponding to these 30 video frames are all the same. Therefore, when generating the index file, only the first frame of the 30 auxiliary frames corresponding to these 30 video frames is saved, that is, the first auxiliary frame generated. However, the frame numbers of the 30 video frames corresponding to these 30 auxiliary frames are all saved together with the auxiliary frames. In this way, the correct association between the video frames and the auxiliary frames can still be known through the index file.

[0081] It should be noted that the above method for generating index files can also be used to filter and merge multiple identical and adjacent auxiliary frame data, and this application does not limit this.

[0082] Optionally, after generating the index file of the auxiliary frame file through the above scheme, since the duplicate auxiliary frame data has been processed, the first auxiliary frames other than the second auxiliary frame can be deleted from the multiple first auxiliary frames to reduce the occupation of hard disk memory.

[0083] In an exemplary embodiment, the generation module 76 is further configured to, after generating an auxiliary frame file and an index file of the auxiliary frame file according to the correspondence, upon receiving a video playback task, determine a target timestamp from the index information of the video frames stored in the recording file according to the start time indicated by the video playback task; determine the target file offset of the target video frame corresponding to the target timestamp, and read the video frame in the recording file according to the target file offset; and send the read video frame to complete the video playback task.

[0084] If a user wants to play back the video stream, they can select the recording file to be played back and the time period to be played back at the front end, thus issuing a video playback task. The start time is determined from the time period indicated by the video playback task, and the target timestamp is determined from the index information of the video frames saved in the recording file based on this start time. Then, the target file offset of the target video frame corresponding to the target timestamp is determined. Based on the target file offset, the storage location of the video to be played back in the recording file is determined, and the video frame is read. Finally, the read video frame is sent to complete the video playback task.

[0085] In an exemplary embodiment, the generation module 76 is further configured to sequentially parse out the fourth frame number of the read video frame; search the index file for a fifth frame number corresponding to the fourth frame number; if the fifth frame number is found, send a third auxiliary frame corresponding to the fifth frame number; and if the third auxiliary frame is sent, send the video frame corresponding to the fourth frame number.

[0086] During the transmission of the read video frames, the fusion of video frames and auxiliary frames also needs to be completed. The specific steps include: first, parsing out the fourth frame number of each read video frame sequentially; then, searching the index file for a fifth frame number corresponding to that fourth frame number. If a fifth frame number is found, it indicates the existence of an auxiliary frame corresponding to that video frame. Therefore, before sending the video frame, the third auxiliary frame corresponding to the fifth frame number needs to be sent first, followed by the video frame corresponding to the fourth frame number. If the fifth frame number is not found, the video frame corresponding to the fourth frame number is sent directly. This transmission method allows the video frames and auxiliary frames to be fused during transmission. The process is as follows: Figure 5 As shown, Figure 5 This is a schematic diagram of an optional intelligent stream merging and conversion embodiment of this application.

[0087] In an exemplary embodiment, the generation module 76 is further configured to, before sending the third auxiliary frame corresponding to the fifth frame sequence number, obtain a fourth timestamp and a third file offset corresponding to the fifth frame sequence number from the index file; and obtain the third auxiliary frame from the auxiliary frame file according to the fourth timestamp and the third file offset.

[0088] Before sending the third auxiliary frame corresponding to the fifth frame sequence number, the following steps are also included: obtaining the fourth timestamp and the third file offset corresponding to the fifth frame sequence number from the index file; determining the storage location of the third auxiliary frame in the auxiliary frame file based on the fourth timestamp and the third file offset, thereby obtaining the third auxiliary frame.

[0089] Embodiments of the present invention also provide a storage medium comprising a stored program, wherein the program, when executed, performs any of the methods described above.

[0090] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0091] S1, Analyze the video file to obtain the auxiliary frame sequence corresponding to the video file;

[0092] S2, determine the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the video recording file;

[0093] S3, generate an auxiliary frame file and an index file of the auxiliary frame file according to the correspondence, wherein the index file is used to find the auxiliary frame in the auxiliary frame file.

[0094] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0095] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0096] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0097] S1, Analyze the video file to obtain the auxiliary frame sequence corresponding to the video file;

[0098] S2, determine the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the video recording file;

[0099] S3, generate an auxiliary frame file and an index file of the auxiliary frame file according to the correspondence, wherein the index file is used to find the auxiliary frame in the auxiliary frame file.

[0100] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0101] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0102] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing systems. They can be centralized on a single computing system or distributed across a network of multiple computing systems. Optionally, they can be implemented using program code executable by a computing system, thereby storing them in a storage system for execution by the computing system. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of generating an auxiliary frame file, characterized by, include: The video file is analyzed to obtain the auxiliary frame sequence corresponding to the video file; Determine the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the recorded file; An auxiliary frame file and an index file for the auxiliary frame file are generated based on the correspondence, wherein the index file is used to find the auxiliary frame in the auxiliary frame file; The index file for generating auxiliary frame files based on the correspondence includes: Obtain multiple identical first auxiliary frames from the auxiliary frame sequence, wherein the categories of the multiple first auxiliary frames include: rule frame data, trajectory frame data, and event frame data; The third frame number of the multiple video frames corresponding to the multiple first auxiliary frames is determined according to the correspondence relationship; An index file for the auxiliary frame file is generated based on the third frame sequence number, the second auxiliary frame, the second timestamp of the second auxiliary frame, and the second file offset of the second auxiliary frame, wherein the plurality of first auxiliary frames includes: the second auxiliary frame, and the second timestamp of the second auxiliary frame is the minimum value among the third timestamps of the plurality of first auxiliary frames.

2. The method of claim 1, wherein, Before determining the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the recorded file, the method further includes: The index information of the video frame is parsed from the frame header of the video frame saved in the video file, wherein the index information includes at least one of the following: the first frame number of the video frame, the first timestamp of the video frame, and the first file offset of the video frame; The index information is stored in the index database.

3. The method of claim 2, wherein, Auxiliary frame files are generated based on the aforementioned correspondence, including: Based on the correspondence, the video frame corresponding to the auxiliary frame in the auxiliary frame sequence is determined from the index database to obtain multiple video frames; The first frame number of each of the multiple video frames is stored in the frame header of a plurality of auxiliary frames corresponding to the multiple video frames. The frame header of the auxiliary frame is also used to store the second frame number of the auxiliary frame and the frame type of the auxiliary frame. The auxiliary frame file is generated based on the multiple auxiliary frames obtained at the end.

4. The method of claim 1, wherein, After generating auxiliary frame files and index files of the auxiliary frame files according to the correspondence, the method further includes: Upon receiving a video playback task, the target timestamp is determined from the index information of the video frames stored in the recording file according to the start time indicated by the video playback task. Determine the target file offset of the target video frame corresponding to the target timestamp, and read the video frame from the recording file according to the target file offset; Send the read video frames to complete the video playback task.

5. The method for generating auxiliary frame files according to claim 4, characterized in that, Send the read video frames, including: The fourth frame number of the read video frames is then parsed out sequentially; Search the index file to see if there is a fifth frame number corresponding to the fourth frame number; If the sequence number of the fifth frame is retrieved, a third auxiliary frame corresponding to the sequence number of the fifth frame is sent. Once the third auxiliary frame has been sent, the video frame corresponding to the sequence number of the fourth frame is sent.

6. The method of claim 5, wherein, Before sending the third auxiliary frame corresponding to the sequence number of the fifth frame, the method further includes: Obtain the fourth timestamp and the third file offset corresponding to the fifth frame sequence number from the index file; The third auxiliary frame is obtained from the auxiliary frame file based on the fourth timestamp and the third file offset.

7. An apparatus for generating a supplemental frame file, characterized by comprising: include: The analysis module is used to analyze the video file and obtain the auxiliary frame sequence corresponding to the video file; A determining module is used to determine the correspondence between the auxiliary frames in the auxiliary frame sequence and the video frames in the recorded file; A generation module is used to generate an auxiliary frame file and an index file of the auxiliary frame file according to the correspondence, wherein the index file is used to find auxiliary frames in the auxiliary frame file; The determining module is further configured to: acquire multiple identical first auxiliary frames in the auxiliary frame sequence, wherein the categories of the multiple first auxiliary frames include: regular frame data, trajectory frame data, and event frame data; determine the third frame sequence number of multiple video frames corresponding to the multiple first auxiliary frames according to the correspondence; generate an index file of the auxiliary frame file according to the third frame sequence number, the second auxiliary frame, the second timestamp of the second auxiliary frame, and the second file offset of the second auxiliary frame, wherein the multiple first auxiliary frames include: the second auxiliary frame, and the second timestamp of the second auxiliary frame is the minimum value among the third timestamps of the multiple first auxiliary frames.

8. A computer readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 6. 9.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 6 through the computer program.

Images