An MP4 format audio and video file generation method, system and computer storage medium

Abstract

The application relates to an MP4 format audio and video file generation method, system and computer storage medium, and the method comprises the following steps: acquiring an audio and video data stream; the audio and video data stream comprises audio and video identification information, audio and video media information, audio and video frame data information and frame index information corresponding to each frame of the audio and video frame data information; ftyp data blocks, mdat data blocks, Smoov box data blocks and moov box data blocks are respectively created, the Smoov box data block is used for storing media information and key frame index information corresponding to the audio and video frame data information; whether the index information of the corresponding frame currently written into the moov box data block is key frame index information is judged, if the index information is key frame index information, the index information of the current corresponding frame is written into the Smoov box data block and the index information of the current corresponding frame is validated; the transmission state of the audio and video data stream is monitored, if the transmission is completed, the moov box data block is validated to generate an MP4 format file. The MP4 file can be normally played under the condition of abnormal termination of video recording and when the video recording is normally completed.

Description

Technical Field

[0001] This invention relates to the field of audio and video, and in particular to a method, system, and computer storage medium for generating MP4 format audio and video files. Background Technology

[0002] MP4 is one of the most widely used audio and video recording formats. Each MP4 file contains several box data blocks; all the data in an MP4 file is assembled from these boxes. A complete MP4 file that can be played normally by a common media player contains at least three box data blocks: the ftyp block, the mdat box block, and the moov box block. The ftyp block stores the initial identifier information of the MP4 file; the mdat box block stores the actual audio and video media data; and the moov box block stores the media information of the audio and video stream and the temporal and spatial index information of each frame in the audio and video stream data, including timestamps, data size, and index address information within the file. If any of these three box data blocks is missing or incomplete, the entire MP4 recording file will not play correctly.

[0003] Real-time recording of MP4 files by hardware devices such as surveillance cameras, camcorders, mobile phones, set-top boxes, computer desktops, game consoles, and medical surgery recording devices typically involves generating audio and video media information and temporal and spatial index information at the end of the recording process after the audio and video stream data has been written. However, when the device experiences unexpected power outages, crashes, software malfunctions, or unexpected disconnections from the storage device, the recording process terminates abruptly, resulting in errors in the video file recording. The media and index information in the `mdat` data block fails to be written to the file, making it impossible to read the media data and index information from the `mdat` data block, thus rendering the recorded MP4 file unplayable. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, system, and computer storage medium for generating MP4 format audio and video files to address the problems of abnormal recording termination and MP4 recording files failing to play due to missing moov box data block information.

[0005] A method for generating MP4 format files, the method comprising:

[0006] Create a video recording file and obtain an audio and video data stream, wherein the audio and video data stream includes audio and video identification information, audio and video media information, audio and video frame data information, and frame index information corresponding to each frame of the audio and video frame data information;

[0007] Create an ftyp data block and an mdat data block. The ftyp data block is used to store the audio and video identification information and write it to the recording file. The mdat data block is used to continuously receive audio and video frame data information from the audio and video data stream and write it to the recording file.

[0008] Create a moov box data block, which is used to store the media information corresponding to the audio and video frame data information and the index information of each frame;

[0009] Create a Smoov box data block, determine whether the index information of the corresponding frame currently written to the Smoov box data block is the key frame index information, if it is the key frame index information, write the index information of the current corresponding frame to the Smoov box data block and write the Smoov box data block storing the key frame index information to the video file so that the key frame index information in the Smoov box data block takes effect.

[0010] Monitor the transmission status of the audio and video data streams. If the transmission is completed, activate the moov box data block to generate the MP4 format file.

[0011] The above-described embodiments of the present invention introduce a simplified moov box data block to write the index data of key frames during recording, and then write the complete moov box at the end of recording, ensuring that the MP4 file can be played normally in the case of abnormal termination of recording and normal termination.

[0012] In one preferred embodiment, the step of writing the index information of the current corresponding frame to the Smoov box data block and writing the Smoov box data block storing the keyframe index information to the video file includes:

[0013] Reserve storage space for the current mdat data block;

[0014] Write the Smoov box data block to the storage space and update the current mdat data block so that the keyframe index information stored in the Smoov box data block takes effect.

[0015] In one preferred embodiment, reserving storage space for the current mdat data block includes:

[0016] Determine the relationship between the reserved space of the current mdat data block and the size of the Smoov box data block. If the space value of the Smoov box data block is greater than the reserved space value of the current mdat data block, update the reserved space value of the mdat data block to the space value of the Smoov box data block.

[0017] The reserved space value of the mdat data block is added to the original space value of the mdat data block to update the space value of the mdat data block.

[0018] The above embodiments of the present invention can prevent the reduction in the ability to resist abnormal recording termination caused by the continuous growth of the moov box size and the impact of writing moov box data stutter on the real-time performance and smoothness of recording. At the same time, it can also avoid the waste of storage space caused by reserving a large space for moov box index information to prevent overflow.

[0019] In one preferred embodiment, the method for initially setting and updating the size of the reserved space for the current mdat data block includes:

[0020] The reserved space size of the mdat box data block is an integer multiple of the current maximum media data frame size or the larger of the current Smoov box data block space size.

[0021] Specifically, in one preferred embodiment, the reserved space size of the mdatbox data block is set to N times (to save reserved storage space, N can be a small value, such as N=2) the current maximum audio and video frame data size. In particular, when setting it for the first time, the maximum audio and video frame data size is the size of the first frame audio and video data, and the initial space value of the mdatbox data block is preset accordingly, and storage space is reserved for the mdat data block in the recording file.

[0022] Next, as the audio and video data streams are continuously written, the reserved space size of the mdat data block is updated to N times the current maximum media data frame size. The relationship between the current reserved space of the mdat data block and the space occupied by the Smoov box data block is determined. If the Smoov box data block space value is greater than the reserved space of the current mdat data block, the reserved space value of the mdat data block is updated to the Smoov box data block space value. The reserved space value of the mdat data block is then added to the original space value of the mdat data block to update its space value.

[0023] In one preferred embodiment, the method further includes:

[0024] Obtain the first frame data information corresponding to the audio and video data stream;

[0025] Write the first frame data information into the mdat data block, and write the media information and index information corresponding to the first frame data information into the Smoov box data block and the moov box data block.

[0026] In one preferred embodiment, the acquisition of the audio and video data stream further includes:

[0027] The audio and video data streams are decoded.

[0028] In one preferred embodiment, determining whether the index information of the corresponding frame currently written to the moov box data block is keyframe index information includes:

[0029] Based on the type of audio / video encoder and the keyframe flags, it is determined whether the current media data frame is keyframe index information.

[0030] In one preferred embodiment, the media information includes: the resolution, sampling rate, duration, and encoder type of the audio and video data stream; the index information includes: the timestamp corresponding to the data frame, the data frame size, and the data index address of the data frame in the file.

[0031] The above-described embodiments of the present invention introduce a streamlined Smoov box data block to periodically write index data of keyframes during recording, and then write the complete Smoov box data block at the end of recording. This ensures that MP4 files can be played normally in cases of abnormal recording termination and normal termination. The above-described embodiments of the present invention prevent the reduction in resistance to abnormal recording termination caused by the decreasing update frequency of the Smoov box data block due to its continuously growing size, and avoid the impact of stuttering when writing the Smoov box data block on the real-time performance and smoothness of recording. Simultaneously, it avoids the waste of storage space caused by reserving large Smoov box data block index information space to prevent overflow.

[0032] An MP4 file generation system, comprising:

[0033] The data stream acquisition module is used to create a video recording file and acquire audio and video data streams, wherein the audio and video data streams include audio and video identification information, audio and video media information, audio and video frame data information, and frame index information corresponding to each frame of the audio and video frame data information;

[0034] The mdat data block creation module is used to create ftyp data blocks and mdat data blocks. The ftyp data blocks are used to store the audio and video identification information and write it to the recording file. The mdat data blocks are used to continuously receive audio and video frame data information from the audio and video data stream and write it to the recording file.

[0035] Create a moov box data block, which is used to store the media information corresponding to the audio and video frame data information and the index information of each frame;

[0036] Create a Smoov box data block, determine whether the index information of the corresponding frame currently written to the Smoov box data block is the key frame index information, if it is the key frame index information, write the index information of the current corresponding frame to the Smoov box data block and write the Smoov box data block storing the key frame index information to the video file so that the key frame index information in the Smoov box data block takes effect.

[0037] The file generation module is used to monitor the transmission status of the audio and video data stream. If the transmission is completed, the moovbox data block is activated to generate the MP4 format file.

[0038] The above-described embodiments of the present invention introduce a simplified Smoov box data block to periodically write the index data of key frames during the recording process, and then write the complete Smoov box at the end of the recording, ensuring that the MP4 file can be played normally in the case of abnormal termination of recording and normal termination.

[0039] A computer storage medium, wherein the computer-executable instructions, when executed by a computer processor, implement any of the above-described methods for generating MP4 format files.

[0040] The above-described embodiments of the present invention execute the above-described MP4 format file generation method through the above-described computer storage medium, and introduce a simplified Smoov box data block to periodically write the index data of key frames during the recording process, and write the complete Smoov box at the end of the recording, so as to ensure that the MP4 file can be played normally under abnormal termination of recording and normal termination. Attached Figure Description

[0041] Figure 1 This is a flowchart of a method for generating MP4 format audio and video files according to a first preferred embodiment of the present invention;

[0042] Figure 2 This is a flowchart detailing step S40 of a method for generating MP4 format audio and video files according to a first preferred embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of a module of an MP4 format audio and video file generation system according to a second preferred embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the Smoovbox data block creation module of an MP4 format audio and video file generation system according to a second preferred embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] It should be noted that when an element is referred to as being "set on" another element, it can be directly on the other element or there may be an intervening element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or there may be an intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only and do not represent the only possible implementation.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0048] like Figure 1 As shown, the first preferred embodiment of the present invention discloses a method for generating MP4 format files, the method comprising:

[0049] S10: Create a video recording file and obtain an audio and video data stream, wherein the audio and video data stream includes audio and video identification information, audio and video media information, audio and video frame data information, and frame index information corresponding to each frame of the audio and video frame data information.

[0050] In this embodiment, the creation of the video file is for acquiring the audio and video data stream. This acquisition step may further include decoding the audio and video data stream. Specifically, at the start of video recording, the recording audio and video encoder type is set, and the recording audio and video encoder is used to acquire the video identifier information, audio and video media information, audio and video frame data information, and frame index information for each frame of the audio and video frame data corresponding to the audio and video data stream.

[0051] In this step, when video recording begins, step S10 above can obtain the audio and video data stream of the video. The audio and video data stream may include audio and video identification information, audio and video media information, audio and video frame data information, and frame index information corresponding to each frame of the audio and video frame data information.

[0052] Furthermore, in this embodiment, the aforementioned audio and video identification information is used to identify the header information of the MP4 format file to be generated. In other words, the audio and video identification information is the ftyp field of the aforementioned audio and video data stream. The ftyp field is used to record encoding-related information of the MP4 format, such as what audio and video data storage specifications (file format) the file to be stored must follow, as well as the version number and compatibility information of the specification.

[0053] The aforementioned audio and video media information includes information such as the resolution, frame rate, sampling rate, duration, and encoder type of the audio and video data stream. The aforementioned frame index information includes the timestamp of the corresponding frame in the aforementioned video data stream, the data frame size, and the data index address of the corresponding data frame in the MP4 file to be generated.

[0054] S20: Create an ftyp data block and an mdat data block. The ftyp data block is used to store the audio and video identification information and write it to the recording file. The mdat data block is used to continuously receive audio and video frame data information from the audio and video data stream and write it to the recording file.

[0055] In this step, an FTYP data block is created and written to the recording file. The FTYP data block is used to store the audio and video identification information. The FTYP data block is also used to store the audio and video data storage specification (file format) that the mp4 file follows, as well as the version number, compatibility, and other information of the specification.

[0056] Create an mdat data block. The mdat data block is used to continuously receive audio and video frame data information from the audio and video data stream and write it to the recording file. The audio and video frame data information mainly refers to the collection information of several video frames or audio frames.

[0057] S30: Create a moov box data block, which is used to store the media information corresponding to the audio and video frame data information and the index information of each frame.

[0058] A moov box data block is created to store the media information corresponding to the audio and video frame data information and the index information of each frame. In this embodiment, the media information may include the resolution, frame rate, sampling rate, duration, encoder type of the MP4 file to be created, as well as the timestamp, duration, frame size and other information of each audio or video frame in the file, and the data index address of each audio and video frame in the file.

[0059] S40: Create a Smoov box data block, determine whether the index information of the corresponding frame currently written to the Smoov box data block is the key frame index information, if it is the key frame index information, write the index information of the current corresponding frame to the Smoov box data block and write the Smoov box data block storing the key frame index information to the video file so that the key frame index information in the Smoov box data block takes effect.

[0060] In this step, the Smoov box data block created has the same format as the moov box data block. The Smoov box data block is used to store the media information and keyframe index information corresponding to the audio and video frame data information. Similar to the moov box data block, in this embodiment, the media information may include the file's resolution, sampling rate, duration, encoder type, and information such as the timestamp, duration, and frame size of each keyframe audio or video frame in the file, as well as the data index address of each audio and video frame in the file.

[0061] Specifically, in this embodiment, the current media data frame can be identified and determined as keyframe index information based on the type of video encoder and the video keyframe flag. If the current media data frame is determined to be a video keyframe, the frame's timestamp, data size, and corresponding index information in the file are updated to the Smoovbox data block to obtain new Smoovbox index information. Taking video recording encoded by an H.246 video encoder as an example, the 3-byte or 4-byte video frame start code 0000 01 or 00 00 00 01 is first found. Then, it is determined whether the value of the next byte is equal to 0x65. If so, the video data frame is keyframe index information; otherwise, it is non-keyframe index information.

[0062] More specifically, in this step, the method for determining whether the index information of the corresponding frame currently written to the moov box data block is the keyframe index information further includes:

[0063] Obtain the first frame data information corresponding to the audio and video data stream;

[0064] The first frame data information is written into the mdat data block, and the media information and index information corresponding to the first frame data information are written into the Smoov box data block and the moov box data block.

[0065] In this embodiment, the first frame of the aforementioned audio and video data stream is assumed to be keyframe data. The first frame of audio and video data is stored in the aforementioned mdat data block, and the media information and index information corresponding to the first frame are stored in the aforementioned moov box and Smoov box data blocks. Then, in subsequent judgments of the audio and video data stream, the aforementioned judgment step of identifying whether the current media data frame is keyframe index information based on the type of video encoder and video keyframe flags is referenced.

[0066] Combination Figure 2 As shown, the steps described above, which involve writing the index information of the current frame to the Smoov box data block and then writing the Smoov box data block to the recording file to make the index information of all previous keyframes effective, can specifically include the following sub-steps:

[0067] S41: Reserve storage space for the currently described mdat data block;

[0068] In this detailed step, when the audio and video data stream is started to be received, the reserved space size of the mdat box data block is set to N times the current maximum audio and video frame data size. Specifically, when setting it for the first time, the maximum audio and video frame data size is the size of the first frame audio and video data, and the initial space value of the mdat box data block is preset accordingly, and storage space is reserved for the mdat data block in the recording file.

[0069] Specifically, in one preferred embodiment, the reserved space size of the mdatbox data block is set to N times (to save reserved storage space, N can be a small value, such as N=2) the current maximum audio and video frame data size. In particular, when setting it for the first time, the maximum audio and video frame data size is the size of the first frame audio and video data, and the initial space value of the mdatbox data block is preset accordingly, and storage space is reserved for the mdat data block in the recording file.

[0070] Next, as the audio and video data streams are continuously written, the reserved space size of the mdat data block is updated to the current maximum media data frame size. The relationship between the current reserved space of the mdat data block and the space occupied by the Smoov box data block is determined. If the Smoov box data block space value is greater than the reserved space of the current mdat data block, the reserved space value of the mdat data block is updated to the Smoov box data block space value. The reserved space value of the mdat data block is then added to the original space value of the mdat data block to update its space value.

[0071] S42: Write the Smoov box data block to the storage space and update the current mdat data block so that the keyframe index information stored in the Smoov box data block takes effect.

[0072] In this detailed step, the index information of the corresponding frame of the current Smoov box data block is written into the reserved space of the updated mdat data block in the video file, so that the media information and index information corresponding to the audio and video streams in the current Smoov box data block take effect.

[0073] Next, the current media data frames of the input stream are continuously written to the reserved space after the mdat box data block. The moov box index information cache is updated for each frame of data. The media information corresponding to the key frame, including the timestamp, data size, and index information written in the file, is updated to the Smoov box data block. The Smoov box data block after the reserved storage space of the mdat box data block in the recording file is also updated, and the new current Smoov box size value is obtained.

[0074] Before writing each media data frame, it is determined whether the sum of the size of the media data frame to be written and the size of the already written data is greater than or equal to the size of the current mdat data block. If it is greater than the current mdat data block, the above process is performed: if the Smoov box data block space value is greater than the reserved space of the current mdat data block, the reserved space value of the mdat data block is updated to the space value of the Smoov box data block, and new storage space is reserved after the current mdat data block in the recording file according to the size of the reserved space value of the mdat data block. Then, the current Smoovbox data block is rewritten after the newly reserved storage space in the recording file, and the new reserved space value of the mdat data block is added to the original space value of the mdat data block. The current mdat data block size field in the recording file is updated with the new mdat data block size to make the media information and index information corresponding to the audio and video streams in the current Smoov box data block effective. If it is less than or equal to the current mdat data block, the above process of writing the current media data frame to the reserved space after the mdat box data block continues, updating the moov for each frame of data. The box index information is cached, and the media information corresponding to the key frame, including the timestamp, data size, and index information written in the file, is updated to the Smoov box data block. The Smoov box data block after the reserved storage space of the mdat box data block written to the recording file is also updated, and the new current Smoov box size value is obtained; until the audio and video data stream transmission and writing are completed.

[0075] The amount of new storage space reserved for the mdat data block in the video file must be large enough to ensure that when a new moov box data block is written after the new mdat data block has reserved storage space, the old moov box data block data will not be overwritten (overflow). This ensures that a valid moov box data block exists before the new moov box block is successfully written, and thus ensures that the video can still play normally if the recording terminates abnormally during this period.

[0076] On the one hand, for shorter video files, due to the smaller amount of index information and smaller moov box data blocks, choosing a smaller mdat data block for reserved storage space will not cause overflow and will save reserved storage space. For example, when choosing a smaller N=2, the size of the reserved storage space for the mdat data block can be controlled within a very small range. On the other hand, for longer video files, due to the huge amount of index information and the dramatically increased size of the moov box data blocks, using a simplified Smoov box data block and setting the reserved space value of the mdat data block to the space value of the Smoov box data block can also minimize the reserved storage space. At the same time, a simplified Smoov box data block can greatly reduce the time spent rewriting the moov box data block each time.

[0077] The steps described above in this embodiment can prevent the reduction in the ability to resist abnormal recording termination and the impact of writing moov box data block stuttering on the real-time performance and smoothness of recording caused by the rapid growth of the moov box size (storing all frame indexes instead of key frame indexes) leading to a decrease in the moov box update frequency (the update condition is that the size of the audio and video data written to Mdat is greater than the size of the moov box data block). At the same time, it can also avoid the waste of storage space caused by reserving a large moov box (moov box data block continues to grow and its size cannot be estimated) index information space to prevent overflow.

[0078] S50: Monitor the transmission status of the audio and video data stream. If the transmission is completed, activate the moov box data block to generate the MP4 format file.

[0079] In this step, after the normal transmission of the audio and video data stream is completed as described above, a moov box data block (containing media information of the audio and video data stream and index information of each frame) is written after the current Smoov box data block in the recording file, and the size of the written mdat box is updated to the sum of the current mdat box size value and the last written Smoov box size value, and finally the MP4 format file is generated.

[0080] The above-described embodiments of the present invention introduce a streamlined moov box data block to periodically write index data of keyframes during recording, and then write the complete moov box data block at the end of recording, ensuring that MP4 files can be played normally in cases of abnormal recording termination and normal termination. The above-described embodiments of the present invention can prevent the decrease in resistance to abnormal recording termination caused by the continuous growth of the moov box data block size leading to a decrease in the update frequency of the moov box data block, and the impact of writing the moov box data block stuttering on the real-time performance and smoothness of recording. At the same time, it can also avoid the waste of storage space caused by reserving a large space for the index information of the moov box data block to prevent overflow.

[0081] like Figure 3 As shown, the second preferred embodiment of the present invention discloses an MP4 format file generation system 100, which includes a data stream acquisition module 110, an mdat data block creation module 120, a moov box data block creation module 130, a Smoov box data block creation module 140, and a file generation module 150.

[0082] The aforementioned data stream acquisition module 110 is used to create a video recording file and acquire audio and video data streams. The audio and video data streams include audio and video identification information, audio and video media information, audio and video frame data information, and frame index information corresponding to each frame of the audio and video frame data information.

[0083] When video recording begins, the data stream acquisition module 110 can acquire the audio and video data stream of the video. Specifically, the creation of the recording file is for the purpose of acquiring the audio and video data stream, which may include audio and video identification information, audio and video media information, audio and video frame data information, and frame index information corresponding to each frame of the audio and video frame data information.

[0084] Furthermore, in this embodiment, the aforementioned audio and video identification information is used to identify the header information of the MP4 format file to be generated. In other words, the audio and video identification information is the ftyp field of the aforementioned audio and video data stream. The ftyp field is used to record encoding-related information of the MP4 format, such as what audio and video data storage specifications (file format) the file to be stored must follow, as well as the version number and compatibility information of the specification.

[0085] The aforementioned audio and video media information includes the resolution, sampling rate, duration, encoder type, etc. of the audio and video data stream; the aforementioned frame index information includes the data frame size of the corresponding frame in the aforementioned video data stream and the data index address of the corresponding data frame in the MP4 file to be generated.

[0086] In this implementation, the acquisition of the audio and video data stream may further include decoding the audio and video data stream. Specifically, at the start of video recording, a recording audio and video encoder type is set, and the recording audio and video encoder is used to acquire the video identification information, audio and video media information, audio and video frame data information, and frame index information for each frame of the audio and video frame data information corresponding to the audio and video data stream.

[0087] The aforementioned mdat data block creation module 120 is used to create ftyp data blocks and mdat data blocks. The ftyp data blocks are used to store the audio and video identification information and write it to the recording file. The mdat data blocks are used to continuously receive audio and video frame data information from the audio and video data stream and write it to the recording file.

[0088] The aforementioned mdat data block creation module 120 creates an ftyp data block and writes it to the recording file. The ftyp data block is used to store the audio and video identification information. The ftyp data block is also used to store the audio and video data storage specification (file format) that the mp4 file follows, as well as the version number, compatibility, and other information of the specification.

[0089] The aforementioned mdat data block creation module 120 creates mdat data blocks. The mdat data blocks are used to continuously receive audio and video frame data information from the audio and video data stream and write it to the recording file. The audio and video frame data information mainly refers to the collection information of several video frames or audio frames.

[0090] The aforementioned moov box data block creation module 130 is used to create moov box data blocks, which are used to store media information corresponding to the audio and video frame data information and index information of each frame.

[0091] The moov box data block is used to store the media information corresponding to the audio and video frame data information and the index information of each frame. In this embodiment, the media information may include the resolution, frame rate, sampling rate, duration, encoder type of the MP4 file to be created, as well as the timestamp, duration, frame size and other information of each audio or video frame in the file, and the data index address of each audio and video frame in the file.

[0092] The Smoov box data block creation module 140 is used to create Smoov box data blocks and determine whether the index information of the corresponding frame currently written to the Smoov box data block is the key frame index information. If it is the key frame index information, the index information of the current corresponding frame is written to the Smoov box data block and the Smoov box data block storing the key frame index information is written to the video file so that the key frame index information in the Smoov box data block becomes effective.

[0093] Here, the Smoov box data block format is the same as the moov box data block. The Smoov box data block is used to store the media information and keyframe index information corresponding to the audio and video frame data information. Similar to the moov box data block, in this embodiment, the media information may include the file resolution, sampling rate, duration, encoder type, and information such as the timestamp, duration, and frame size of each keyframe audio or video frame in the file, as well as the data index address of each audio and video frame in the file.

[0094] Specifically, the Smoov box data block format can identify whether the current media data frame is keyframe index information based on the video encoder type and video keyframe flags. More specifically, if the current media data frame is determined to be a video keyframe, the frame's timestamp, data size, and corresponding index information in the file are updated in the Smoov box data block to obtain new Smoov box index information. In this embodiment, taking video recording encoded by an H.246 video encoder as an example, first, the 3-byte or 4-byte video frame start code 00 00 01 or 00 00 00 01 is found. Then, it is determined whether the value of the immediately following byte is equal to 0x65. If so, the video data frame is keyframe index information; otherwise, it is non-keyframe index information.

[0095] In this embodiment, the Smoov box data block format creation module 140 further includes a first frame data acquisition unit, which is used to acquire the first frame data information corresponding to the audio and video data stream; and write the first frame data information into the mdat data block, and write the media information and index information corresponding to the first frame data information into the Smoovbox data block and the moov box data block.

[0096] In this embodiment, the first frame of the audio and video data stream is assumed to be the key frame data. The first frame of the audio and video data stream is stored in the above-mentioned mdat data block, and the media information and index information corresponding to the first frame of the audio and video data stream are stored in the above-mentioned moov box data block and Smoov box data block.

[0097] In more detail, combined Figure 2 As shown, the Smoov box data block creation module 140 includes an mdat data block space reservation unit 141 and a key frame index information activation unit 142.

[0098] The aforementioned mdat data block space reservation unit 141 is used to reserve storage space for the current mdat data block. Specifically, when receiving audio and video data streams, the aforementioned mdat data block space reservation unit 141 sets the size of the reserved space for the mdat box data block to N times the current maximum audio and video frame data size. In particular, when setting it for the first time, the maximum audio and video frame data size is the size of the first frame audio and video data, and thus presets the initial space value of the mdat box data block, and reserves storage space for the mdat data block in the recording file.

[0099] Next, as the audio and video data streams are continuously written, the reserved space size of the mdat data block is updated to the current maximum media data frame size. The relationship between the current reserved space of the mdat data block and the space occupied by the Smoov box data block is determined. If the Smoov box data block space value is greater than the reserved space of the current mdat data block, the reserved space value of the mdat data block is updated to the Smoov box data block space value. The reserved space value of the mdat data block is then added to the original space value of the mdat data block to update its space value.

[0100] The aforementioned keyframe index information activation unit 142 is used to write the index information of the current frame after the current storage space of the aforementioned mdat data block, and update the current mdat data block so that the index information of all previous keyframes becomes effective.

[0101] The keyframe index information activation unit 142 writes the index information of the corresponding frame of the current Smoov box data block into the reserved space of the updated mdat data block in the recording file, so that the media information and index information corresponding to the audio and video streams in the current Smoov box data block take effect.

[0102] Next, the current media data frames of the input stream are continuously written to the reserved space after the mdat box data block. The moov box index information cache is updated for each frame of data. The media information corresponding to the key frame, including the timestamp, data size, and index information written in the file, is updated to the Smoov box data block. The Smoov box data block after the reserved storage space of the mdat box data block in the recording file is also updated, and the new current Smoov box size value is obtained.

[0103] Before writing each media data frame, it is determined whether the sum of the size of the media data frame to be written and the size of the already written data is greater than or equal to the size of the current mdat data block. If it is greater than the current mdat data block, the above process is performed: if the Smoov box data block space value is greater than the reserved space of the current mdat data block, the reserved space value of the mdat data block is updated to the space value of the Smoov box data block, and new storage space is reserved after the current mdat data block in the recording file according to the size of the reserved space value of the mdat data block. Then, the current Smoovbox data block is rewritten after the newly reserved storage space in the recording file, and the new reserved space value of the mdat data block is added to the original space value of the mdat data block. The current mdat data block size field in the recording file is updated with the new mdat data block size to make the media information and index information corresponding to the audio and video streams in the current Smoov box data block effective. If it is less than or equal to the current mdat data block, the above process of writing the current media data frame to the reserved space after the mdat box data block continues, updating the moov for each frame of data. The box index information is cached, and the media information corresponding to the key frame, including the timestamp, data size, and index information written in the file, is updated to the Smoov box data block. The Smoov box data block after the reserved storage space of the mdat box data block written to the recording file is also updated, and the new current Smoov box size value is obtained; until the audio and video data stream transmission and writing are completed.

[0104] The amount of new storage space reserved for the mdat data block in the video file must be large enough to ensure that when a new moov box data block is written after the new mdat data block has reserved storage space, the old moov box data block data will not be overwritten (overflow). This ensures that a valid moov box data block exists before the new moov box block is successfully written, and thus ensures that the video can still play normally if the recording terminates abnormally during this period.

[0105] On the one hand, for shorter video files, due to the smaller amount of index information and smaller moov box data blocks, choosing a smaller mdat data block for reserved storage space will not cause overflow and will save reserved storage space. For example, when choosing a smaller N=2, the size of the reserved storage space for the mdat data block can be controlled within a very small range. On the other hand, for longer video files, due to the huge amount of index information and the dramatically increased size of the moov box data blocks, using a simplified Smoov box data block and setting the reserved space value of the mdat data block to the space value of the Smoov box data block can also minimize the reserved storage space. At the same time, a simplified Smoov box data block can greatly reduce the time spent rewriting the moov box data block each time.

[0106] The above implementation can prevent the reduction in resistance to abnormal recording termination caused by the rapid growth of the moov box size (storing all frame indexes instead of key frame indexes), which leads to a decrease in the moov box update frequency (the update condition is that the size of the audio and video data written to Mdat is greater than the size of the moov box data block). It can also prevent the impact of writing moov box data blocks on the real-time performance and smoothness of recording due to the stuttering caused by storing all frame indexes, which causes the moov box data block to grow rapidly during recording and increases the time spent rewriting the moov box. At the same time, it can also avoid the waste of storage space caused by reserving a large moov box (moov box data block continues to grow and its size cannot be estimated) index information space to prevent overflow.

[0107] The file generation module 150 is used to monitor the transmission status of the audio and video data stream. If the transmission is completed, the moovbox data block is activated to generate the MP4 format file.

[0108] The file generation module 150 monitors the normal transmission of the audio and video data stream and writes a moov box data block (containing media information of the audio and video data stream and index information of each frame) after the current Smoovbox data block in the recording file. It also updates the size of the written mdat box to the sum of the current mdat box size and the last written Smoov box size, and finally completes the generation of the MP4 format file.

[0109] The above-described embodiments of the present invention introduce a streamlined moov box data block to periodically write index data of keyframes during recording, and then write the complete moov box data block at the end of recording, ensuring that MP4 files can be played normally in cases of abnormal recording termination and normal termination. The above-described embodiments of the present invention can prevent the decrease in resistance to abnormal recording termination caused by the continuous growth of the moov box data block size leading to a decrease in the update frequency of the moov box data block, and the impact of writing the moov box data block stuttering on the real-time performance and smoothness of recording. At the same time, it can also avoid the waste of storage space caused by reserving a large space for the index information of the moov box data block to prevent overflow.

[0110] A computer storage medium, wherein the computer-executable instructions, when executed by a computer processor, implement any of the above-described methods for generating MP4 format files.

[0111] The above-described embodiments of the present invention execute the above-described MP4 format file generation method through the above-described computer storage medium, and introduce a simplified Smoov box data block to periodically write the index data of key frames during the recording process, and write the complete Smoov box at the end of the recording, so as to ensure that the MP4 file can be played normally under abnormal termination of recording and normal termination.

[0112] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0113] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0114] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0115] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0116] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The above embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

[0118] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0119] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for generating MP4 format files, characterized in that, The method includes: Create a video recording file and obtain an audio and video data stream, wherein the audio and video data stream includes audio and video identification information, audio and video media information, audio and video frame data information, and frame index information corresponding to each frame of the audio and video frame data information; Create an ftyp data block and an mdat data block. The ftyp data block is used to store the audio and video identification information and write it to the recording file. The mdat data block is used to continuously receive audio and video frame data information from the audio and video data stream and write it to the recording file. Create a moov box data block. The moov box data block is used to store the media information corresponding to the audio and video frame data information and the index information of each frame. Create a Smoov box data block, determine whether the index information of the corresponding frame currently written to the Smoov box data block is keyframe index information. If it is keyframe index information, write the index information of the current corresponding frame to the Smoov box data block and write the Smoov box data block storing the keyframe index information to the video file so that the keyframe index information in the Smoov box data block becomes effective. Monitor the transmission status of the audio and video data streams. If the transmission is completed, activate the moov box data block to generate the MP4 format file. The steps of writing the index information of the current corresponding frame to the Smoov box data block and writing the Smoov box data block storing the keyframe index information to the video file include: Reserve storage space for the current mdat data block; Write a Smoov box data block into the storage space and update the current mdat data block space value to make the keyframe index information stored in the Smoov box data block effective.

2. The method for generating MP4 format files according to claim 1, characterized in that, The step of writing the Smoov box data block to the storage space and updating the current mdat data block space value includes: Determine the relationship between the reserved space of the current mdat data block and the size of the Smoov box data block. If the space value of the Smoov box data block is greater than the reserved space value of the current mdat data block, update the reserved space value of the mdat data block to the space value of the Smoov box data block. The reserved space value of the mdat data block is added to the original space value of the mdat data block to update the space value of the mdat data block.

3. The method for generating MP4 format files according to claim 2, characterized in that, The reserved storage space in the current mdat data block includes: The reserved space size of the mdat data block is an integer multiple of the current maximum media data frame size or the larger of the current Smoov box data block size.

4. The method for generating MP4 format files according to claim 1, characterized in that, The method further includes: Obtain the first frame data information corresponding to the audio and video data stream; Write the first frame data information into the mdat data block, and write the media information and index information corresponding to the first frame data information into the Smoov box data block and the moov box data block.

5. The method for generating MP4 format files according to claim 1, characterized in that, The acquisition of audio and video data streams also includes: The audio and video data streams are decoded.

6. The method for generating MP4 format files according to claim 5, characterized in that, The step of determining whether the index information of the corresponding frame currently written to the moovbox data block is keyframe index information includes: Based on the type of audio / video encoder and the keyframe flags, it is determined whether the current media data frame is keyframe index information.

7. The method for generating MP4 format files according to claim 1, characterized in that, The media information includes: the resolution, sampling rate, duration, and encoder type of the audio and video data stream; the index information includes: the timestamp corresponding to the data frame, the data frame size, and the data index address of the data frame in the file.

8. An MP4 format file generation system, characterized in that, include: The data stream acquisition module is used to create a video recording file and acquire audio and video data streams, wherein the audio and video data streams include audio and video identification information, audio and video media information, audio and video frame data information, and frame index information corresponding to each frame of the audio and video frame data information; The mdat data block creation module is used to create ftyp data blocks and mdat data blocks. The ftyp data blocks are used to store the audio and video identification information and write it to the recording file. The mdat data blocks are used to continuously receive audio and video frame data information from the audio and video data stream and write it to the recording file. The moov box data block creation module stores the media information corresponding to the audio and video frame data information and the index information of each frame. The Smoov box data block creation module determines whether the index information of the corresponding frame currently written to the Smoov box data block is keyframe index information. If it is keyframe index information, it writes the index information of the current corresponding frame to the Smoov box data block and writes the Smoov box data block storing the keyframe index information to the video file, so that the keyframe index information in the Smoov box data block becomes effective. The Smoov box data block creation module reserves storage space in the current mdat data block; it writes the Smoov box data block to the storage space and updates the space value of the current mdat data block, so that the keyframe index information stored in the Smoov box data block becomes effective. The file generation module is used to monitor the transmission status of the audio and video data stream. If the transmission is completed, the moov box data block is activated to generate the MP4 format file.

9. A computer storage medium, characterized in that, The computer-executable instructions, when executed by a computer processor, implement the MP4 format file generation method as described in any one of claims 1-7.

Images