Real-time video stream processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a real-time video stream processing method and device, electronic equipment and a storage medium. According to the technical scheme provided by the embodiment of the invention, when the user browser needs to store the real-time video stream, the streaming media data in the specified format issued to the user browser for playing the real-time video stream is multiplexed, and de-encapsulation is performed based on the streaming media data to obtain the video data in the first encapsulation format similar to the streaming media data; and re-packaging the video data into the video data in the second packaging format. In this way, the processing efficiency of the real-time video stream to be stored is improved in a de-encapsulation and re-encapsulation mode, the situation that the coding process consumes too long time due to direct transcoding of the real-time video stream is avoided, then the calculation pressure of a video server is reduced, the storage efficiency of the real-time video stream is improved, and the video storage experience of a user is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video processing technology, and in particular to a method, device, electronic device, and storage medium for processing real-time video streams. Background Art

[0002] Currently, in real-time video streaming scenarios such as live broadcasts, classroom inspections, and security patrols, users need to watch the real-time images captured by the camera while also downloading a specific segment of the live video stream to their local computer. To achieve this, the video server needs to encode the live video stream captured by the camera into a specified streaming format (such as FLV or HLS) and then provide it to the user's browser for playback. Since streaming media in a specified format cannot be downloaded and saved directly to a local disk, if a user needs to record and play a specific segment of the live video stream, the live video stream needs to be encoded into a format suitable for storage, then transmitted to the user's browser and saved to the local disk.

[0003] However, when the user's browser needs to save the real-time video stream, the video server needs to encode the real-time video stream into the specified streaming format on the one hand, and on the other hand, it needs to allocate computing resources to encode the real-time video stream into the format for storage. The computing pressure on the video server is too high, and the video stream encoding process is time-consuming, resulting in low efficiency in implementing video stream storage, affecting the user's video download experience. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, electronic device, and storage medium for processing real-time video streams, which can shorten the time required to convert real-time video streams to be stored, reduce the computing pressure on video servers, and solve the technical problem of excessively long conversion time required for real-time video streams to be stored.

[0005] In a first aspect, an embodiment of the present application provides a method for processing a real-time video stream, comprising:

[0006] Extract the streaming media data of the specified format obtained by encoding the real-time video stream, and transmit the streaming media data to the user's browser in real time for playback;

[0007] Decapsulating the streaming media data into video data in a first encapsulation format, the video data in the first encapsulation format comprising a plurality of first sub-files storing description data and a plurality of second sub-files storing media data;

[0008] Extract description data from each first sub-file and splice them into a description data file; extract media data from each second sub-file and splice them into a media data file; construct second encapsulation format video data based on the description data file and the media data file; and transmit the second encapsulation format video data to the user browser for storage.

[0009] As can be seen, when a user's browser needs to save a real-time video stream, this application multiplexes the streaming media data in a specified format sent to the user's browser for real-time video streaming playback, decapsulates the streaming media data, obtains video data in a first encapsulation format similar to the streaming media data, and recapsulates it into video data in a second encapsulation format. This improves the processing efficiency of the real-time video stream to be saved by decapsulating and recapsulating it, avoiding the situation where the encoding process is too time-consuming due to direct transcoding of the real-time video stream. This reduces the computing pressure on the video server, improves the efficiency of saving the real-time video stream, and enhances the user's video saving experience.

[0010] Furthermore, the present application can adapt the video playback format to the local player corresponding to the user's browser by transpackaging the video data in the first packaging format into the data in the second packaging format, thereby improving video compatibility.

[0011] Furthermore, the plurality of first sub-files are composed of a first metadata file and a plurality of media object files;

[0012] Extract description data from each first sub-file and splice them into a description data file, including:

[0013] Parsing the first metadata file to obtain file information data and storage container data;

[0014] Parse each media object file in turn to obtain segment sequence data and index information data;

[0015] The file information data, storage container data, fragment sequence data and index information data are used as description data to construct a second metadata file, and the second metadata file is used as the description data file.

[0016] Furthermore, constructing the second metadata file includes:

[0017] Perform data conversion on the fragment sequence data and index information data, and fill the converted data into the specified information file.

[0018] Furthermore, the second sub-file is a media byte file;

[0019] Extracting media data from each second sub-file and splicing them into a media data file includes:

[0020] Delete the header content of each media byte file, calculate the total data length of each media byte file, and create new header content based on the total data length;

[0021] The media byte content of each media byte file is extracted, and the new title content and the media byte content are concatenated into a new media byte file as a media data file.

[0022] Furthermore, constructing the second encapsulation format video data based on the description data file and the media data file includes:

[0023] A type file is extracted from the video data in the first encapsulation format, and the type file, the description data file and the media data file are merged to generate video data in the second encapsulation format.

[0024] Furthermore, decapsulating the streaming media data into video data in a first encapsulation format includes:

[0025] Parse the video frames of streaming media data frame by frame;

[0026] Obtain media data corresponding to the audio track and video track, and put the media data into the second sub-file;

[0027] Obtain description data of the corresponding audio track and video track, and put the description data into the first sub-file.

[0028] Furthermore, the streaming media data of a specified format obtained by encoding the real-time video stream is extracted, including:

[0029] In response to a video saving request from a user browser, a request period of the video saving request is determined, and streaming media data of corresponding segments is extracted based on the request period.

[0030] By extracting and processing streaming media data corresponding to the user request period, accurate processing of the real-time video stream to be saved can be achieved, avoiding redundant processing and calculation of streaming media data.

[0031] In a second aspect, an embodiment of the present application provides a real-time video stream processing device, comprising:

[0032] An extraction module is configured to extract streaming media data of a specified format obtained by encoding the real-time video stream, and the streaming media data is used to be transmitted in real time to the user's browser for playback;

[0033] a decapsulation module configured to decapsulate the streaming media data into video data in a first encapsulation format, the video data in the first encapsulation format comprising a plurality of first sub-files storing description data and a plurality of second sub-files storing media data;

[0034] The construction module is configured to extract description data from each first sub-file and splice them into a description data file, extract media data from each second sub-file and splice them into a media data file, construct second encapsulation format video data based on the description data file and the media data file, and transmit the second encapsulation format video data to the user browser for storage.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0036] memory and one or more processors;

[0037] The memory is used to store one or more programs;

[0038] When the one or more programs are executed by the one or more processors, the one or more processors implement the real-time video stream processing method as described in the first aspect.

[0039] In a fourth aspect, an embodiment of the present application provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the real-time video stream processing method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of a method for processing a real-time video stream provided in Example 1 of the present application;

[0041] Figure 2 This is a flowchart for constructing Fmp4 format video data in Example 1 of the present application;

[0042] Figure 3 This is a schematic diagram of the file structure of Fmp4 format video data in Example 1 of the present application;

[0043] Figure 4 This is a schematic diagram of the file structure of the MP4 format video data in Example 1 of the present application;

[0044] Figure 5 This is a flow chart describing the generation of a data file in Example 1 of the present application;

[0045] Figure 6 This is a schematic diagram of the moof file structure in Example 1 of this application;

[0046] Figure 7 This is a schematic diagram of the moov file structure in Example 1 of the present application;

[0047] Figure 8 This is a flow chart of generating a media data file in Example 1 of the present application;

[0048] Figure 9 This is a structural diagram of a real-time video stream processing device provided in Example 2 of the present application;

[0049] Figure 10 This is a structural diagram of an electronic device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. It should also be noted that, for ease of description, only parts related to the present application, not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0051] This application provides a method for processing real-time video streams. This method multiplexes streaming media data in a specified format, which is delivered to a user's browser for real-time video streaming playback. It then decapsulates the streaming media data to obtain video data in a first encapsulation format that approximates the streaming media data, and then recapsulates the data into video data in a second encapsulation format. This decapsulation and recapsulation process improves the processing efficiency of the real-time video stream to be stored, reducing the computing pressure on the video server.

[0052] In the real-time video streaming usage scenario, the video server pulls the real-time video stream to the video server based on standard protocols such as GB28181 or RTSP, and then encodes the real-time video stream on the video server to obtain the FLV or HLS format supported by the user's browser, and provides it to the user's browser for real-time playback.

[0053] The process of users downloading and saving recorded videos requires users to first request to download the MP4 file through the browser, and then the MP4 format video data is transmitted to the user's browser from the video server and saved to the local disk.

[0054] Because FLV or HLS video formats and MP4 video formats are completely different encapsulation formats, the former is suitable for live broadcasts, while the latter is suitable for on-demand playback. In some business scenarios, cameras operate 24 / 7, generating large amounts of video data. Due to storage costs, live video streams are typically not stored on the server, meaning that the video data is lost once viewed. If you want to support playback of recorded videos and allow users to download them, you need to store the live video stream in MP4 format on the server. When a user's browser requests to save a live video stream, the video server must encode the live video stream into the specified streaming format and allocate computing resources to convert the live video stream into the storage format. This places excessive computing pressure on the video server, and the video stream encoding process is time-consuming, resulting in low live video stream storage efficiency and a poor user experience in video downloading.

[0055] To address this situation, a live video stream conversion server is typically added. When a user requests to download and replay the video, the server converts the live video stream into MP4 format and provides it to the user for download and storage. Alternatively, the live video stream is transcoded in the user's browser to obtain the MP4 format. However, both adding a conversion server and performing live video stream transcoding in the user's browser require additional computing resources to transcode the live video stream, and the entire process is still time-consuming.

[0056] Based on this, a method for processing real-time video streams according to an embodiment of the present application is provided. When a user browser needs to save a real-time video stream, the method multiplexes the streaming media data in a specified format sent to the user browser for real-time video streaming playback, decapsulates the streaming media data, and recapsulates it into MP4 format video data. This decapsulation and recapsulation improves the processing efficiency of the real-time video stream to be saved, avoids the situation where the encoding process is too time-consuming due to direct transcoding of the real-time video stream, and thus reduces the computing pressure on the video server, improves the efficiency of saving the real-time video stream, and enhances the user's video saving experience.

[0057] In the embodiments of the present application, the video formats involved include video coding formats and encapsulation formats (also known as container formats). Among them, H264, H265, etc. are video coding formats, which compress and encode the original YUV data of the real-time video stream. Fmp4, mp4, FLV and HLS are video encapsulation formats, which are containers for packaging already encoded H264 or H265 video data. For example, in the mp4 encapsulation format standard, there is a moov box file, which contains most of the encapsulation information, such as video bit rate, frame rate, size and index of each video frame, while the mdat box file saves the already encoded H264 or H265 media data.

[0058] Example 1:

[0059] Figure 1 A flowchart of a method for processing a real-time video stream provided in Example 1 of the present application is provided. The method for processing a real-time video stream provided in this embodiment can be executed by a real-time video stream processing device. The real-time video stream processing device can be implemented through software and / or hardware. The real-time video stream processing device can be composed of two or more physical entities or a single physical entity. Generally speaking, the real-time video stream processing device can be a video processing device such as a video server or a user browser device.

[0060] The following description will be made by taking the real-time video stream processing device as an example to execute the real-time video stream processing method. Figure 1 , the real-time video stream processing method specifically includes:

[0061] S110 , extracting streaming media data of a specified format obtained by encoding the real-time video stream, and transmitting the streaming media data to the user's browser in real time for playback.

[0062] When the user browser needs to save the real-time video stream, this application reuses the streaming media data of the specified format (such as FLV or HLS format) sent to the user browser for real-time video streaming playback, so as to decapsulate this part of the streaming media data and recapsulate it into video data suitable for saving the real-time video stream, such as mp4 format video data. Compared with using the real-time video stream to directly encode into mp4 format video data, the time consumption of decapsulation and recapsulation is shorter, thereby achieving the effect of improving the processing efficiency of the real-time video stream and reducing the computing pressure of the video server. According to the actual processing requirements, the streaming media data can also be other different specified formats. This application does not impose fixed restrictions on the specific streaming media data format and will not go into details here.

[0063] It should be noted that when this application processes a real-time video stream to obtain video data in a format to be saved, the video server can obtain the corresponding streaming media data for processing, or the user's browser device can process the streaming media data based on the streaming media data currently played by the browser. This application does not impose any fixed restrictions on the processing device of the real-time video stream, and will not be described in detail here. The following describes in detail the real-time video stream processing method of this application using a video browser as an example.

[0064] Exemplarily, when processing real-time video stream, by receiving the video preservation request of the user browser, the extraction of the streaming media data of the specified format obtained by encoding the real-time video stream is triggered. It is understandable that, for the picture collected by the camera in real time, it will be pulled to the video server through standard protocols such as GB28181 or RTSP, and then on the video server, the real-time video stream is transcoded to obtain the FLV or HLS format that the user browser can support, to provide the user browser for real-time playback. Therefore, when the user needs to save the corresponding real-time video stream, the video preservation request is input through the user browser, and the video preservation request is reported to the video server. Then, the video server can reuse the streaming media data of the specified format obtained by encoding the real-time video stream based on the video preservation request, so as to improve the generation efficiency of the format video data to be saved and shorten the processing time.

[0065] Specifically, extracting streaming media data in a specified format obtained by encoding a real-time video stream includes:

[0066] In response to a video saving request from a user browser, a request period of the video saving request is determined, and streaming media data of corresponding segments is extracted based on the request period.

[0067] Taking the live broadcast scenario as an example, when a user watches a real-time live video through a browser, if they need to record the current live video and download it to a local disk, they can click the "Save" option provided by the browser, enter the time period of the live video to be saved, that is, the request period, and then generate a corresponding video save request to the video server. Based on the received video save request, the video server can extract the streaming media data corresponding to the request period by determining the request period in the video save request, and then transpack it into the video data in the format to be saved. By extracting and processing the streaming media data corresponding to the user's request period, accurate processing of the real-time video stream to be saved can be achieved, avoiding redundant processing and calculation of streaming media data. There is no need for users to crop the video, and the corresponding video segment can be played on demand through the request period.

[0068] Furthermore, the streaming media data is transmitted to the user's browser for real-time playback, allowing the user to watch the live video through the browser. By reusing this streaming media data, the encoding process of the real-time video stream into the format to be saved can be omitted. By transpackaging, the time spent on processing the real-time video stream can be reduced.

[0069] S120: Decapsulate the streaming media data into video data in a first encapsulation format, where the video data in the first encapsulation format includes a plurality of first sub-files storing description data and a plurality of second sub-files storing media data.

[0070] Based on the acquired streaming media data, video data in the required storage format is obtained by decapsulation. Furthermore, considering that the streaming media data is a streaming segment encapsulation structure, when encapsulating the streaming media data into the video data in the required storage format, the present application first decapsulates the streaming media data into video data in the form of streaming segments with a similar structure, which is defined as the first encapsulation format video data, and then transcapsulates the first format video data into video data in the required storage format, which is defined as the second encapsulation format video data.

[0071] Below, this application is described in detail by taking Fmp4 format video data as the first encapsulation format video data and mp4 format video data as the second format video data as an example.

[0072] This decapsulates streaming media data in a specified format into Fmp4 format video data, facilitating subsequent recapturing back into MP4 format video data. It should be noted that Fmp4 (Fragmented MP4) is a video and audio streaming format, an extension of the MPEG-4 Part 12 standard. Unlike traditional MP4, Fmp4 divides media files into multiple fragments, each of which is a complete MP4 file containing media data, description data, and other information. Since streaming media data in FLV or HLS format also divides media files into several segments, the encapsulation structures of the two are similar, and the encapsulation structures of Fmp4 format video data and mp4 format video data are also similar. Therefore, when transpackaging streaming media data in FLV or HLS format into mp4 format video data, you can first quickly decapsulate the streaming media data in FLV or HLS format into Fmp4 format video data, and then convert the Fmp4 format video data into mp4 format video data. Compared with directly decapsulating it into mp4 format video data, its encapsulation efficiency is higher, thereby shortening the conversion time of the real-time video stream to be saved.

[0073] It should be noted that when the user browser needs to save the real-time video stream, this application chooses to multiplex the streaming media data and transpack it into mp4 format video data for the user browser to download and save. This is because mp4 format video data is suitable for most video saving scenarios (i.e., on-demand), while Fmp4 format video data is suitable for live video scenarios. In the case where the user browser needs to save the real-time video stream, in order to improve the compatibility of the user browser's local storage of the real-time video stream, it is necessary to further transpack it into mp4 format video data based on the Fmp4 format video data.

[0074] The file that defines the storage description data of Fmp4 format video data is the first subfile, and the file that stores media data is the second subfile. The process of decapsulating streaming media data into Fmp4 format video data is mainly the process of constructing the above-mentioned first subfile and second subfile.

[0075] Specifically, decapsulating the streaming media data into the first encapsulation format video data includes:

[0076] Parse the video frames of streaming media data frame by frame;

[0077] Obtain media data corresponding to the audio track and video track, and put the media data into the second sub-file;

[0078] Obtain description data of the corresponding audio track and video track, and put the description data into the first sub-file.

[0079] Reference Figure 2 Taking FLV streaming media data as an example, a flow chart of decapsulating streaming media data in an embodiment of the present application is provided. According to the user's request period, the corresponding segmented FLV streaming media data is selected. The FLV streaming media data is then decapsulated, and description data and media data are obtained for each video frame data. Among them, by parsing the video track and audio track of the video frame, the media data therein is obtained, and the media data is written into the second sub-file. At the same time, by parsing the description data of the video track and audio track of the video frame, the processed description data is written into the first sub-file, thereby obtaining the first sub-file and the second sub-file of the Fmp4 format video data. For streaming media data in other specified formats, the description data and media data are obtained by referring to the above decapsulation process, and then the first sub-file and the second sub-file of the Fmp4 format video data are obtained.

[0080] Specifically, Fmp4 and mp4 format video data encapsulate their description data and media data in corresponding file boxes, namely ftyp, moov, moof, mdat, and their underlying files. The ftyp file is a type file used to store file type information; the moov file is used to store media metadata, including file information data (mvhd) and storage container data (trak); the moof file is used to store media object information, including fragment sequence data (mfhd) and index information data (traf); and the mdat file is used to store the actual media byte content.

[0081] Reference Figure 3, a schematic diagram of the composition structure of Fmp4 format video data in an embodiment of the present application is provided. In the process of converting FLV streaming media into Fmp4 format video data, the present application is a process of continuously processing streaming media video fragment data. Except for the ftyp+moov file contained in the initial fragment of the streaming media, the other fragments can be considered as a combination of moof+mdat. An Fmp4 file is composed of sub-files such as ftyp+moov+moof_1+mdat_1+...+moof_n+mdat_n, wherein moov, moof_1, moof_2...moof_n is the first sub-file, mdat_1, mdat_2...mdat_n is the second sub-file, and the ftyp file is used to store file type information. The description data and media data obtained by decapsulating FLV streaming media are stored in the above-mentioned sub-files accordingly, thereby completing the decapsulation of FLV streaming media and obtaining the Fmp4 format video data.

[0082] S130: Extract description data from each first sub-file and splice them into a description data file; extract media data from each second sub-file and splice them into a media data file; construct video data in a second encapsulation format based on the description data file and the media data file; and transmit the video data in the second encapsulation format to the user browser for storage.

[0083] Based on the Fmp4 format video data obtained by the above decapsulation, by extracting the description data from the first sub-file and the media data from the second sub-file, these two parts of data can be spliced ​​together according to the format requirements of the mp4 format video data to obtain the description data file and the media data file, and then construct the mp4 format video data.

[0084] Reference Figure 4 This provides a schematic diagram of the file structure of MP4 video data. MP4 files typically consist of a single mdatbox file, consisting of ftyp, moov, and mdat. Therefore, the process of converting an FMP4 file to an MP4 file can be understood as converting the file structure of ftyp, moov, moof_1, mdat_1, ..., moof_n, and mdat_n into a file structure of ftyp, moov, and mdat.

[0085] Specifically, since MP4 files are primarily divided into media data and description data, media data is typically placed in the mdat box, while description data is placed in the moov file. However, FMP4's description data is only partially placed in the moov file, with the majority placed in the moof file. Therefore, transcoding an FMP4 file to an MP4 file can be divided into two steps: processing the media data, which involves concatenating the multiple mdat boxes in the FMP4 file into a complete mdat box; and processing the description data, which involves parsing the moof box in the FMP4 file, extracting the data within, and then converting it back into a complete moov box in the MP4 file.

[0086] Wherein, the plurality of first sub-files are composed of a first metadata file and a plurality of media object files; Figure 5 , extract description data from each first sub-file and splice them into a description data file, including:

[0087] S1301: Parse the first metadata file to obtain file information data and storage container data;

[0088] S1302: parse each media object file in sequence to obtain segment sequence data and index information data;

[0089] S1303: Use the file information data, storage container data, fragment sequence data, and index information data as description data to construct a second metadata file, and use the second metadata file as the description data file.

[0090] When generating the description data file, it is necessary to first parse the metadata file in the first encapsulation format video data, which is defined as the first metadata file, that is, the moov file in Fmp4. The file information data mvhd and storage container data trak of the audio and video are obtained from the moov file in Fmp4, and then the media object file in Fmp4, that is, the moof file, is parsed in turn to obtain the segment sequence data mfhd and index information data traf. The file structure of moof is as follows Figure 6 As shown, the file structure of moov is as follows Figure 7 shown.

[0091] Specifically, the traf box of the moof file contains data such as tfhd, tfdt, trun, and sdtp. The specific descriptions are as follows:

[0092] tfhd, full box, mainly stores **trackId**, similar to mvex.trex;

[0093] tfdt, full box, mainly stores baseMediaDecodeTime, which is **firstDts**;

[0094] trun, full box, mainly stores sample information;

[0095] sdtp, full box, mainly stores sample flags information, including isLeading, sampleDependsOn, sampleIsDependedOn, and sampleHasRedundancy.

[0096] The moov and moof data parsed based on Fmp4 need to be repackaged into a complete moov box, defined as the second metadata file, that is, the moov file in the mp4. Constructing the second metadata file includes: converting the segment sequence data and index information data, and filling the converted data into the specified information file.

[0097] Based on the file information data mvhd and storage container data trak obtained by parsing the moov file in Fmp4, the fragment sequence data mfhd and index information data traf obtained by parsing the moof file are converted and filled into the specified information file stbl box. Among them, stbl mainly contains information such as stts, stss, ctts, stsc, stsz, stco, etc.

[0098] Where sample represents a video frame or audio frame, the conversion relationship is as follows:

[0099] stts, the duration of the sample, corresponds to the sample_duration of moof's trun;

[0100] stss, sample type is the sequence number of the I frame, corresponding to the sdtp of moof;

[0101] ctts, pts of sample, corresponds to sample_composition_time_offset of trun in moof;

[0102] stsc, the number of samples in each chunk, corresponds to the number of samples recorded in each moof;

[0103] stsz, the byte size of sample, corresponding to sample_size of trun;

[0104] Stco, the offset of each chunk, corresponds to the starting position of the data of mdat associated with moof in the first part of the merging process, which refers to the position in the entire file.

[0105] On the other hand, refer to Figure 8 , the second sub-file is a media byte file; extracting media data from each second sub-file and splicing it into a media data file includes:

[0106] S1304: Delete the title content of each media byte file, calculate the total data length of each media byte file, and create new title content based on the total data length;

[0107] S1305: Extract the media byte content of each media byte file, and concatenate the new title content and the media byte content into a new media byte file as a media data file.

[0108] Media data file concatenation involves merging the mdat boxes of each Fmp4 media byte file. Specifically, the header portion of each media byte file's mdat box is removed, and the total length of the data in all the mdat boxes is calculated. Based on this total length, a new mdat box header is created and then concatenated with the mdat content of the previously removed media byte file. This completes the concatenation of the Fmp4 mdat files into a single MP4 mdat file.

[0109] It's important to note that the entire data conversion process isn't a simple data concatenation process. In the MP4 encapsulation format, boxes are nested in multiple layers, and changes to inner boxes can affect the data in outer boxes. For example, if the length of an inner box changes, the lengths of all outer boxes must be modified. Furthermore, the MP4 video data encapsulation format version must be considered during the transpackaging process. For example, the byte size of some boxes used to store data varies between versions, so the byte size of the data needs to be adjusted during the conversion process.

[0110] Furthermore, constructing the second encapsulation format video data based on the description data file and the media data file includes: extracting the type file from the first encapsulation format video data, and merging the type file, the description data file and the media data file to generate the second encapsulation format video data.

[0111] Based on the description data file and media data file obtained above, combined with the type file ftyp extracted from the Fmp4 format video data, the mp4 format video data required by the user to save the real-time video stream can be merged to generate the file structure of ftyp+moov+mdat.

[0112] Furthermore, during the MP4 format video data transcoding process, the converted data can be transmitted to the user's browser for storage in real time. By continuously parsing the FMP4 format video data, calculating the moov data, writing the mdat data, and finally writing the complete moov to disk, a complete MP4 file can be obtained.

[0113] The entire process requires only JavaScript to decapsulate and recapsulate the data. This process involves parsing the source and target encapsulation formats, respectively. This process is significantly shorter than video encoding, simplifying the MP4 video data generation process and improving video processing efficiency.

[0114] Optionally, when transpackaging video data in the first encapsulation format into video data in the second encapsulation format, the second encapsulation format video data may also be in an encapsulation format such as mov or mkv, depending on the format in which the user's browser needs to save the downloaded video. It is only necessary to extract the corresponding content from the subfile storing the description data and media data in the first encapsulation format video data according to the encapsulation structure of the second encapsulation format video, and then transpackaging the data according to the encapsulation structure of the second encapsulation format video data. This application does not impose any fixed restrictions on the specific transpackaging format, and will not be elaborated on here.

[0115] As described above, when the user browser needs to save the real-time video stream, the streaming media data of the specified format sent to the user browser for real-time video stream playback is multiplexed, and decapsulated based on the streaming media data to obtain the first encapsulation format video data that is similar to the streaming media data, and re-encapsulated into the second encapsulation format video data. In this way, the processing efficiency of the real-time video stream to be saved is improved by decapsulation and re-encapsulation, avoiding the situation where the encoding process takes too long due to direct transcoding of the real-time video stream, thereby reducing the computing pressure of the video server, improving the saving efficiency of the real-time video stream, and improving the user's video saving experience. In addition, the present application can adapt the video playback format to the local player corresponding to the user's browser by transcapsulating the first encapsulation format video data into the second encapsulation format data, thereby improving video compatibility.

[0116] Example 2:

[0117] Based on the above embodiments, Figure 9This is a structural diagram of a real-time video stream processing device provided in Example 2 of this application. Figure 9 The real-time video stream processing device provided in this embodiment specifically includes: an extraction module 21, a decapsulation module 22 and a construction module 23.

[0118] The extraction module 21 is configured to extract the streaming media data of a specified format obtained by encoding the real-time video stream, and the streaming media data is used to be transmitted to the user's browser in real time for playback;

[0119] The decapsulation module 22 is configured to decapsulate the streaming media data into a first encapsulation format video data, the first encapsulation format video data including a plurality of first sub-files storing description data and a plurality of second sub-files storing media data;

[0120] The construction module 23 is configured to extract description data from each first sub-file, splice them into a description data file, extract media data from each second sub-file, splice them into a media data file, construct second encapsulation format video data based on the description data file and the media data file, and transmit the second encapsulation format video data to the user browser for storage.

[0121] Furthermore, the plurality of first sub-files are composed of a first metadata file and a plurality of media object files;

[0122] Extract description data from each first sub-file and splice them into a description data file, including:

[0123] Parsing the first metadata file to obtain file information data and storage container data;

[0124] Parse each media object file in turn to obtain segment sequence data and index information data;

[0125] The file information data, storage container data, fragment sequence data and index information data are used as description data to construct a second metadata file, and the second metadata file is used as the description data file.

[0126] Furthermore, constructing the second metadata file includes:

[0127] Perform data conversion on the fragment sequence data and index information data, and fill the converted data into the specified information file.

[0128] Furthermore, the second sub-file is a media byte file;

[0129] Extracting media data from each second sub-file and splicing them into a media data file includes:

[0130] Delete the header content of each media byte file, calculate the total data length of each media byte file, and create new header content based on the total data length;

[0131] The media byte content of each media byte file is extracted, and the new title content and the media byte content are concatenated into a new media byte file as a media data file.

[0132] Furthermore, constructing the second encapsulation format video data based on the description data file and the media data file includes:

[0133] A type file is extracted from the video data in the first encapsulation format, and the type file, the description data file and the media data file are merged to generate video data in the second encapsulation format.

[0134] Furthermore, decapsulating the streaming media data into video data in a first encapsulation format includes:

[0135] Parse the video frames of streaming media data frame by frame;

[0136] Obtain media data corresponding to the audio track and video track, and put the media data into the second sub-file;

[0137] Obtain description data of the corresponding audio track and video track, and put the description data into the first sub-file.

[0138] Furthermore, the streaming media data of a specified format obtained by encoding the real-time video stream is extracted, including:

[0139] In response to a video saving request from a user browser, a request period of the video saving request is determined, and streaming media data of corresponding segments is extracted based on the request period.

[0140] By extracting and processing streaming media data corresponding to the user request period, accurate processing of the real-time video stream to be saved can be achieved, avoiding redundant processing and calculation of streaming media data.

[0141] As described above, when the user browser needs to save the real-time video stream, the streaming media data of the specified format sent to the user browser for real-time video stream playback is multiplexed, and decapsulated based on the streaming media data to obtain the first encapsulation format video data that is similar to the streaming media data, and re-encapsulated into the second encapsulation format video data. In this way, the processing efficiency of the real-time video stream to be saved is improved by decapsulation and re-encapsulation, avoiding the situation where the encoding process takes too long due to direct transcoding of the real-time video stream, thereby reducing the computing pressure of the video server, improving the saving efficiency of the real-time video stream, and improving the user's video saving experience. In addition, the present application can adapt the video playback format to the local player corresponding to the user's browser by transcapsulating the first encapsulation format video data into the second encapsulation format data, thereby improving video compatibility.

[0142] The real-time video stream processing device provided in the second embodiment of the present application can be used to execute the real-time video stream processing method provided in the above-mentioned first embodiment, and has corresponding functions and beneficial effects.

[0143] Example 3:

[0144] The third embodiment of the present application provides an electronic device, referring to Figure 10 The electronic device includes: a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The number of processors in the electronic device may be one or more, and the number of memories in the electronic device may be one or more. The processor, memory, communication module, input device, and output device of the electronic device may be connected via a bus or other means.

[0145] As a computer-readable storage medium, the memory can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the real-time video stream processing method described in any embodiment of the present application (for example, the extraction module, decapsulation module and construction module in the real-time video stream processing device). The memory may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function; the data storage area can store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the device via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0146] The communication module is used for data transmission.

[0147] The processor executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory, that is, realizes the above-mentioned real-time video stream processing method.

[0148] The input device can be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device can include a display device such as a display screen.

[0149] The electronic device provided above can be used to execute the real-time video stream processing method provided in the above embodiment 1, and has corresponding functions and beneficial effects.

[0150] Example 4:

[0151] An embodiment of the present application also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute a method for processing a real-time video stream. The method for processing a real-time video stream includes: extracting streaming media data in a specified format obtained by encoding the real-time video stream, and the streaming media data is used to transmit the streaming media data in real time to a user browser for playback; decapsulating the streaming media data into video data in a first encapsulation format, and the first encapsulation format video data includes multiple first sub-files storing description data and multiple second sub-files storing media data; extracting description data from each first sub-file and splicing them into a description data file, extracting media data from each second sub-file and splicing them into a media data file, constructing video data in a second encapsulation format based on the description data file and the media data file, and transmitting the second encapsulation format video data to the user browser for storage.

[0152] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROMs, floppy disks, or tape drives; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or it may be located in a different second computer system that is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) that can be executed by one or more processors.

[0153] Of course, the storage medium containing computer-executable instructions provided in an embodiment of the present application is not limited to the real-time video stream processing method described above, and can also execute related operations in the real-time video stream processing method provided in any embodiment of the present application.

[0154] The real-time video stream processing device, storage medium, and electronic device provided in the above embodiments can execute the real-time video stream processing method provided in any embodiment of the present application. For technical details not fully described in the above embodiments, please refer to the real-time video stream processing method provided in any embodiment of the present application.

[0155] The above are only preferred embodiments of the present application and the technical principles employed. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that are possible for those skilled in the art will not depart from the scope of protection of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments and may include more other equivalent embodiments without departing from the concept of the present application. The scope of the present application is determined by the scope of the claims.

Claims

1. A method for processing real-time video stream, characterized in that: include: Extracting streaming media data of a specified format obtained by encoding a real-time video stream, and transmitting the streaming media data to a user's browser in real time for playback; Decapsulating the streaming media data into video data in a first encapsulation format, where the video data in the first encapsulation format includes a plurality of first sub-files storing description data and a plurality of second sub-files storing media data; The description data is extracted from each of the first sub-files and spliced ​​into a description data file; the media data is extracted from each of the second sub-files and spliced ​​into a media data file; second encapsulation format video data is constructed based on the description data file and the media data file; and the second encapsulation format video data is transmitted to the user browser for storage.

2. The method for processing real-time video stream according to claim 1, wherein: The plurality of first sub-files are composed of a first metadata file and a plurality of media object files; The step of extracting the description data from each of the first sub-files and splicing the description data into a description data file includes: Parsing the first metadata file to obtain file information data and storage container data; Parsing each of the media object files in sequence to obtain fragment sequence data and index information data; A second metadata file is constructed using the file information data, the storage container data, the fragment sequence data, and the index information data as the description data, and the second metadata file is used as the description data file.

3. The method for processing real-time video stream according to claim 2, wherein: The constructing of the second metadata file includes: Perform data conversion on the fragment sequence data and the index information data, and fill the converted data into a specified information file.

4. The method for processing real-time video stream according to claim 1, wherein: The second sub-file is a media byte file; The extracting the media data from each of the second sub-files and splicing them into a media data file includes: deleting the title content of each of the media byte files, calculating the total data length of each of the media byte files, and creating new title content based on the total data length; The media byte content of each of the media byte files is extracted, and the new title content and the media byte content are spliced ​​into a new media byte file as the media data file.

5. The method for processing real-time video stream according to any one of claims 1 to 4, characterized in that: The constructing the second encapsulation format video data based on the description data file and the media data file includes: A type file is extracted from the first encapsulation format video data, and the type file, the description data file and the media data file are merged to generate a second encapsulation format video data.

6. The method for processing real-time video stream according to claim 1, wherein: Decapsulating the streaming media data into video data in a first encapsulation format includes: Parsing the video frames of the streaming media data frame by frame; Obtaining media data corresponding to the audio track and the video track, and putting the media data into the second sub-file; Obtain description data corresponding to the audio track and the video track, and put the description data into the first sub-file.

7. The method for processing real-time video stream according to claim 1, wherein: The extracting of streaming media data in a specified format obtained by encoding the real-time video stream includes: In response to a video saving request from a user browser, a request period of the video saving request is determined, and the streaming media data of the corresponding segment is extracted based on the request period.

8. A real-time video stream processing device, characterized in that: include: An extraction module is configured to extract streaming media data of a specified format obtained by encoding a real-time video stream, and the streaming media data is used to be transmitted in real time to a user's browser for playback; a decapsulation module configured to decapsulate the streaming media data into video data in a first encapsulation format, wherein the video data in the first encapsulation format includes a plurality of first subfiles storing description data and a plurality of second subfiles storing media data; A construction module is configured to extract the description data from each of the first sub-files, splice them into a description data file, extract the media data from each of the second sub-files, splice them into a media data file, construct second encapsulation format video data based on the description data file and the media data file, and transmit the second encapsulation format video data to the user browser for storage.

9. An electronic device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the real-time video stream processing method according to any one of claims 1 to 7.

10. A storage medium containing computer-executable instructions, characterized in that: When executed by a computer processor, the computer executable instructions are used to perform the real-time video stream processing method according to any one of claims 1 to 7.

Images