A method and system for continuous MP4 recording across URL HLS playback
By establishing an independent recording context in cloud storage HLS playback and utilizing first keyframe gating, the problems of video file fragmentation and misjudgment of recording end notification caused by playback address switching are solved, thus achieving the continuity of recording and the complete playability of files.
Patent Information
- Application Number
- CN202611449468.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-17
- Publication Date
- 2026-11-10
AI Technical Summary
In cloud storage HLS playback scenarios, when switching playback addresses, existing technologies cause the recorded files to be split into multiple fragmented files, and there are problems such as inconsistent encoding formats, timeline reversal, and misjudgment of recording end notifications, which affect the continuity and playability of the recording.
By establishing a recording context independent of the playback address task, using the first keyframe of the new address as the write gate, and combining encoding parameter compatibility and media time rollback judgment, the continuity and playability of the recording file are achieved, avoiding the impact of the asynchronous exit of the old task on the recording of the new task.
It achieves the goal of maintaining the continuity of user recording when switching playback addresses, reducing file fragmentation, ensuring the integrity and playability of each output MP4 file, and avoiding misjudgment of recording end notification and mixed file parameters.
Smart Images

Figure CN123075152A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of audio and video storage and playback technology, and more specifically, to a method and system for continuous MP4 recording with cross-URL HLS playback. Background Technology
[0002] In HLS cloud storage playback scenarios, users can initiate local manual recording during playback. Cloud playback addresses are typically time-sensitive; after the current playback address ends normally, expires, or experiences a network error, the client needs to obtain a new playback address to continue playback. If the recording file is still bound to a single playback address task, each address switch will close the current MP4 file, causing a single user recording operation to be split into multiple files.
[0003] Existing solutions typically map a playback address task directly to a recording file. When the playback address is updated, the old file is closed and a new file is created, resulting in the same user recording being split into multiple fragmented MP4 files. These files then need to be manually stitched together for subsequent viewing, archiving, or sharing. Some solutions retain the old file handle whenever the playback address changes, but they do not wait for the first valid keyframe of the new address. If the new file continues writing from a predicted frame or other non-random access media unit, the player may not be able to decode correctly from the beginning of the file.
[0004] In addition, the existing solution has the following shortcomings: it only judges whether the address has changed, without simultaneously checking the new and old audio and video encoding configurations and the continuity of media time. If the new address has switched the encoding format, resolution, audio configuration, or the playback position has rolled back, continuing to write to the same MP4 file may cause parameter inconsistencies or timeline reversal within the file; there is a parallel window for new and old tasks when switching playback addresses. The old task may only report completion, failure, or failure after the new task has taken over the recording. If the old task can still close the file or send a recording end notification, the upper layer may mistakenly believe that the user's recording has stopped; during high-speed playback, the display link may actively discard some non-critical video frames to maintain playback performance. If the recording is written after this frame dropping logic, the local recording content will be unexpectedly truncated by the display strategy.
[0005] Therefore, a method is needed to determine whether to continue writing the original file or create a new file when the playback address changes during HLS playback on the same device and in the same channel, while ensuring the integrity and playability of the MP4 encapsulation, and to prevent the asynchronous exit of the old task from affecting the recording process of the new task. Summary of the Invention
[0006] The purpose of this application is to provide a method and system for continuous MP4 recording across URL HLS playback. By establishing a recording context independent of the playback address task, handing over the recording context when the address is switched, using the first keyframe of the new address as the write gate, performing continuous file determination when the first keyframe arrives, and performing the owner's final state closing, the system maintains the continuity of user recording and ensures the playability of each output MP4 file when the playback address is switched.
[0007] In a first aspect, embodiments of this application provide a method for continuous MP4 recording with cross-URL HLS playback, comprising the following steps: Establish a recording context, which is independent of the playback address task. The recording context includes the logical playback identity corresponding to the same device identifier and channel identifier, the recording request status, the current MP4 file handle, the baseline of audio and video encoding parameters of the opened file, the last time the media was written, the keyframe status, the most recent keyframe group cache, and the current owner task number. In response to a playback address switch, the new playback address is compared with the old playback address: if the new playback address is the same as the old playback address and the old task has not ended, the current task remains unchanged; if the new playback address is different from the old playback address, a new task with a new task number is created, the new task takes over the recording context and updates the current owner task number, the recording context is set to wait for the first keyframe of the new address, the old task is marked as replaced, and the recording end status notification of the old task is suppressed; In the state of waiting for the first keyframe of the new address, a media packet with a new address is received, and video keyframe identification is performed on the media packet. Before the video keyframe is identified, the video content of the new address is not written to the current MP4 file. When a video keyframe is detected while waiting for the first keyframe of the new address, a continuous file determination is performed: if the audio and video encoding parameters of the new task are incompatible with the baseline of the audio and video encoding parameters of the already opened file, the original file is closed and a new MP4 file is created; if the media time of the new keyframe rolls back more than a preset threshold relative to the most recently written media time, the original file is closed and a new MP4 file is created; if the encoding parameters are compatible and the media time has not rolled back more than the preset threshold, the original file handle is maintained and the audio and video media packet of the new address is appended to the current MP4 file. After the continuous file determination is completed, the MP4 file is closed and a recording end notification is sent only by the current owner task of the recording context, and the recording end notification is sent only once.
[0008] Preferably, the recording context further includes: a mutex lock, a recording start notification state, a recording end notification state, a file sequence number, a file path, latched audio and video parameters of the current playback task, valid status of opened file parameters, address switching status, and MP4 write information.
[0009] Preferably, after the new task takes over the recording context, the method further includes: if the old task does not stop within a preset waiting time, not writing new address data to the current MP4 file, restoring the recording ownership of the old task, and rejecting the switch; switching requests for the same logical playback identity are processed serially, and if the previous round of switching is not finished, subsequent switching requests return "session busy".
[0010] Preferably, the identification of the video keyframes includes: When a media packet is a video packet and the media packet extension identifier indicates a keyframe, it is determined to be a keyframe; when the media packet extension identifier does not indicate a keyframe, the network abstraction layer units in the video payload are parsed. Units with a network abstraction layer type of 5 in H.264 encoding are determined to be keyframes, and units with a network abstraction layer type of 16 to 21 in H.265 encoding are determined to be random access frames.
[0011] Preferably, after establishing the recording context, the method further includes: Maintain a cache of the most recent keyframe group, which stores the most recent group of images from the start of a keyframe to the current media pack; When a new keyframe is received, the old buffer is cleared and a new buffer is started with the new keyframe. When a user starts recording midway, if the cache contains an image group starting with a keyframe, the MP4 writer is opened, and the image group in the cache is first written to the MP4 file in the cache order, and then the current and subsequent media packets are written; if the cache does not contain the image group, the MP4 writer is opened after receiving a keyframe, and the image group in the cache and the current keyframe are written.
[0012] Preferably, the audio and video encoding parameters include video parameters and audio parameters. The video parameters include video encoding type, frame rate, and video encoding configuration data. The audio parameters include whether audio is enabled, audio encoding type, sampling rate, sampling bit width, number of channels, audio frame rate, and bit rate. The audio and video encoding parameters of the new task are compared with each item in the baseline of the audio and video encoding parameters of the already opened file. If any item is different, the audio and video encoding parameters are determined to be incompatible. If the audio encoding is not supported, or the sampling rate and number of channels do not meet the writer conditions, the MP4 file is opened as if there is no audio.
[0013] Preferably, the determination of media time rollback includes: adding the preset threshold to the media timestamp of the new keyframe; when the sum is less than the most recently written media time, media time rollback is determined, the original file is closed, and a new MP4 file is created.
[0014] Preferably, the operation of closing the MP4 file includes: Retrieve the last remaining video and audio data blocks not yet written to disk; update the actual length of mdat; update the video and audio sample table and timeline index, where the sample table includes stts, stsz, stsc, stss, and stco; write moov based on the current sample table; close the current MP4 file.
[0015] Preferably, before appending the audio and video media packet from the new address to the current MP4 file, the method further includes: sending the same encoded media packet first to the recording link and then to the display link for high-speed frame loss judgment; the recording link receives the complete encoded media packet and performs keyframe recognition, image group caching and MP4 writing; the display link discards non-critical video frames according to the playback speed; and the frame loss processing of the display link does not change the content of the MP4 file written by the recording link.
[0016] Secondly, embodiments of this application provide a continuous MP4 recording system for cross-URL HLS playback, comprising: The recording context management module is used to establish a recording context independent of the playback address task. The recording context includes the logical playback identity corresponding to the same device identifier and channel identifier, the recording request status, the current MP4 file handle, the baseline of audio and video encoding parameters of the opened file, the time of the most recent media write, the keyframe status, the most recent keyframe group cache, and the current owner task sequence number. The task handover module is used to respond to playback address switching by comparing the new playback address with the old playback address: when the new playback address is the same as the old playback address and the old task has not ended, the current task remains unchanged; when the new playback address is different from the old playback address, a new task with a new task number is created, the new task takes over the recording context and updates the current owner task number, the recording context is set to wait for the first keyframe of the new address, the old task is marked as replaced, and the recording end status notification of the old task is suppressed; The keyframe gating module is used to receive media packets with a new address in the state of waiting for the first keyframe of the new address, to identify video keyframes in the media packets, and not to write video content with the new address to the current MP4 file before the video keyframe is identified. The continuous file determination module is used to perform continuous file determination when a video keyframe is identified in the state of waiting for the first keyframe of the new address: when the audio and video encoding parameters of the new task are incompatible with the baseline of the audio and video encoding parameters of the already opened file, the original file is closed and a new MP4 file is created; when the media time of the new keyframe rolls back more than a preset threshold relative to the most recently written media time, the original file is closed and a new MP4 file is created; when the encoding parameters are compatible and the media time has not rolled back more than the preset threshold, the original file handle is maintained and the audio and video media packet of the new address is appended to the current MP4 file; The final state closing module is used to close the MP4 file and send a recording final state notification only by the current owner task of the recording context after the continuous file determination is completed, and the recording final state notification is sent only once.
[0017] Compared with the solutions of the prior art, the present invention has at least the following beneficial technical effects: (1) By establishing a recording context independent of the playback address task, and having the new task take over the recording context when the playback address is switched, a user manual recording can maintain the same MP4 file across multiple short-lifecycle HLS playback addresses, reducing file fragmentation and manual splicing operations caused by playback address switching; (2) By using first keyframe gating, encoding parameter compatibility judgment and media time rollback judgment, the original file is only used when it is safe to continue writing, avoiding the new file from starting to continue writing from non-random access media units such as prediction frames, reducing the risk of file not being able to be decoded at the beginning, parameter mixing and internal timeline rollback; (3) By controlling the cross-task handover of the recording context, the owner's task sequence number and the silent final state of the old task, the delayed completion, failure or failure notification of the replaced old task cannot close the recording file being used by the new task, and also avoids the upper layer receiving repeated recording end notifications. (4) By placing the recording input before the frame drop judgment in high-speed display, the degradation strategy for user viewing performance does not change the complete audio and video content of the local MP4. (5) By controlling the closing and creating new files when the encoding parameters change or the media time rolls back, instead of forcibly splicing discontinuous data, each output MP4 file has a corresponding sample table and moov index information, ensuring that each output MP4 file is playable while maintaining the continuous semantics of the user's recording.
[0018] In summary, this application organically combines key technologies such as cross-task handover of recording context, gating of first keyframe writing, joint determination of encoding parameter compatibility and media time rollback, owner final state termination, and recording link front-end to form a complete cross-URL HLS playback continuous MP4 recording solution. It can run stably in various scenarios such as cloud storage HLS playback, long-term monitoring playback, and live playback. Under the condition of frequent switching of playback address, it not only ensures the continuous semantics of user recording, but also ensures the independent playability of each output MP4 file. Attached Figure Description
[0019] Exemplary embodiments of the present invention can be more fully understood by referring to the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain the present invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.
[0020] Figure 1 A flowchart illustrating a method for continuous MP4 recording with cross-URL HLS playback provided as an exemplary embodiment of this application; Figure 2 A timing diagram illustrating task handover and recording context handover during playback address switching, provided as an exemplary embodiment of this application; Figure 3 A schematic diagram of the recording context state transition provided in an exemplary embodiment of this application; Figure 4 A schematic diagram illustrating the writing and closing process of an MP4 container, provided as an exemplary embodiment of this application; Figure 5 A schematic diagram of the structure of a continuous MP4 recording system with cross-URL HLS playback provided as an exemplary embodiment of this application; Figure 6 A schematic diagram of an electronic device provided for an exemplary embodiment of this application; Figure 7 This is a schematic diagram of a computer-readable medium provided for an exemplary embodiment of this application. Detailed Implementation
[0021] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0022] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0023] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0024] Before providing a detailed description of the embodiments of this application, the key terms involved will first be explained. HLS (HTTP Live) Streaming refers to a streaming media transmission protocol based on the Hypertext Transfer Protocol, where the playback client obtains media segments one by one according to the playlist; the playback address (URL) refers to an accessible address issued by the cloud for the playback session, which is usually time-sensitive and needs to be obtained to continue playback after expiration or network failure; MP4 refers to a media container based on the ISO basic media file format, with its sample data and index information stored in the mdat and moov areas respectively; a keyframe refers to a video frame that can be decoded independently without relying on other frames; a group of pictures (GOP) refers to a group of continuous media samples that starts with a keyframe and ends before the next keyframe; logical playback identity refers to a playback session identifier determined by the same device identifier and channel identifier, used to associate multiple playback address tasks under the same session; the owner task sequence number refers to the playback task sequence number recorded in the recording context that has the right to write to and close the MP4 file and send recording end status notification; recording end status notification refers to a one-time notification sent to the upper layer when recording is completed, recording fails, or recording stops; the network abstraction layer unit (NAL unit) refers to the basic bitstream unit in encoding standards such as H.264 and H.265.
[0025] This embodiment uses a local manual recording scenario with HLS playback in cloud storage as an example to provide a detailed explanation of the continuous MP4 recording method of this application. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating a method for continuous MP4 recording across URL HLS provided in an embodiment of this application. The method in this embodiment is executed by a processor in the playback client and includes the following steps: S110, Establish a recording context. The recording context is independent of the playback address task. The recording context includes the logical playback identity corresponding to the same device identifier and channel identifier, recording request status, current MP4 file handle, audio and video encoding parameter baseline of the opened file, last written media time, keyframe status, last keyframe group cache, and current owner task sequence number. Specifically, the recording context is independent of the playback address task. The recording state of the same playback is saved by the recording context and is not directly destroyed when a single HLS address or a single playback task ends. The recording context is protected by a mutex lock and includes: recording request state, indicating that a recording start request has been received (not equivalent to the MP4 being successfully opened); recording start and recording end notification states, limiting the recording start notification and recording stop or failure notification to be sent only once each; first frame control state, indicating that a keyframe must be waited before opening the MP4 for the first time; file state, indicating that the file is open and the MP4 writer handle is valid; file attributes, including file sequence number, duration slice parameters, and output path; latched audio and video parameters of the current playback task; baseline of audio and video parameters of the opened file and their validity status; address switching state, including flags such as context handover, waiting for a new address keyframe, and the requirement to reopen the MP4 after parameter changes; owner task sequence number, used to record the sequence number of the task currently authorized to write, close the MP4, and process the recording end state; the time of the most recently written media; and the most recently cached keyframe group. The unique index of a playback task consists of a cloud ID and a channel number. Each time a new playback task is created, the task manager increments the task number. When a user starts recording, the owner's task number is set to the current task's task number.
[0026] S120, in response to the playback address switching, compare the new playback address with the old playback address: when the new playback address is the same as the old playback address and the old task has not ended, keep the current task unchanged; when the new playback address is different from the old playback address, create a new task with a new task number, let the new task take over the recording context and update the current owner task number, set the recording context to wait for the first keyframe of the new address, the old task is marked as replaced, and the recording end status notification of the old task is suppressed; Specifically, requests for the same HLS address with the same cloud ID and channel are processed idempotently; requests with different addresses are processed by creating a new task and entering a handover process. Please refer to [link / reference]. Figure 2 , Figure 2This is a timing diagram illustrating the task handover and recording context handover during playback address switching, as provided in an embodiment of this application. During handover, the old task disables its final state notification, marks itself as replaced, and requests to stop. The new task takes over the recording context, sets a cross-address continuation flag and a flag waiting for a new address keyframe, and updates the owner task number to the new task's task number. The old task's stop waiting window is set to 3000 milliseconds to allow limited time for the old task to asynchronously stop and the queue to close. When the old task stops normally within the waiting window, the handover is complete; when the old task times out, a rollback is executed: no new task is started, no new address data is written to the current MP4 file, the old task is re-registered as the current task, the owner task number is restored to the old task's task number, the cross-address continuation, waiting for a new address keyframe, and parameter reopen flags are cleared, the current switch occupation flag is cleared, and the session is busy. Therefore, if the old task fails to exit abnormally for an extended period, the original recording control remains with the old task, preventing two tasks from simultaneously controlling the same MP4 file. Switching between the same logical playback identities is serialized by the switching occupation flag. If the previous round of switching has not yet ended, subsequent switching requests will return a busy session, and the recording context will not be handed over concurrently.
[0027] S130, in the state of waiting for the first keyframe of the new address, a media packet with a new address is received, and video keyframe identification is performed on the media packet. Before the video keyframe is identified, the video content of the new address is not written to the current MP4 file. Specifically, after receiving a new media packet, the new task does not write video content at the new address to the existing MP4 file until a video keyframe is received. Keyframe identification includes two methods: a first-level fast determination, checking whether the media packet is a video packet and whether the media packet extension identifier is true; if both conditions are met, it is identified as a keyframe. A second-level supplementary determination, for video packets waiting for the first keyframe or waiting for a new address keyframe, if the media packet extension identifier does not provide a keyframe, then the Network Abstraction Layer (NAL) units in the video payload are parsed: scanning the NAL units after the 3-byte or 4-byte Annex-B start code; in H.264 encoding, a NAL unit type of 5 is identified as a keyframe; in H.265 encoding, a NAL unit type between 16 and 21 is identified as a random access frame (keyframe). After a payload hit, the current media packet is marked as a keyframe, and then the recording start or cross-address continuation determination proceeds. The above two methods can be used individually or in combination.
[0028] In an alternative implementation of this embodiment, keyframe identification can also be performed based solely on media packet extension identifiers or solely on network abstraction layer unit parsing. These two methods can be used individually or in combination.
[0029] S140, when a video keyframe is identified in the state of waiting for the first keyframe of the new address, a continuous file determination is performed: when the audio and video encoding parameters of the new task are incompatible with the baseline of the audio and video encoding parameters of the already opened file, the original file is closed and a new MP4 file is created; when the media time of the new keyframe rolls back more than a preset threshold relative to the most recently written media time, the original file is closed and a new MP4 file is created; when the encoding parameters are compatible and the media time has not rolled back more than the preset threshold, the original file handle is maintained and the audio and video media packet of the new address is appended to the current MP4 file; Specifically, the continuous file determination includes file creation determination, encoding parameter compatibility determination, and media time rollback determination. If a recording file has not yet been opened, a new MP4 file is created. After the audio and video parameters of the new address arrive, they are first normalized to the parameters actually used by the MP4 writer, and then compared item by item with the parameter baseline of the currently opened MP4. If any comparison item is different, it is determined to be incompatible. This embodiment does not compare independent resolution fields or independent video bitrate fields, but uses the video configuration data, encoding type, and frame rate actually used by the MP4 writer as the video parameter baseline. The video configuration data includes SPS / PPS or corresponding H.265 configuration for MP4 encapsulation. Therefore, when the encoding information such as resolution and profile changes and causes changes in the configuration data, it is determined to be incompatible. Audio parameter determination items include whether audio is enabled, audio encoding type, sampling rate, sampling bit width, number of channels, audio frame rate, and bitrate. G.711A, G.711U, and AAC are supported. If the audio encoding is not supported, or the AAC sampling rate and number of channels do not meet the writer's conditions, the MP4 is opened as if there is no audio.
[0030] When the first keyframe of the new address arrives, its media timestamp is compared with the timestamp of the most recently successfully written MP4. The current threshold is 500 milliseconds: when the timestamp of the new address keyframe plus 500 milliseconds is less than the most recently written timestamp, it is determined that the media time has become discontinuous, the current MP4 is forcibly closed, and a new MP4 is created; slight jitter within 500 milliseconds does not trigger a split. The judgment condition is shown in formula (1): (1) in, The media timestamp representing the new keyframe. This indicates a preset threshold (500 milliseconds in this example). This indicates the most recent media write time. If none of the above forced splitting conditions are met, the original file handle is retained, and audio / video packets at the new address are appended. For fixed MP4 output paths, this scheme does not use file duration as the splitting condition for cross-address continuation; duration-based slicing is only an optional implementation for non-MP4 recording modes.
[0031] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the state transition of a recording context provided in an embodiment of this application. After the user starts recording, the recording context enters a waiting state for the first keyframe; upon receiving the keyframe, it enters a recording state. When the address changes, it enters a waiting state for the new address keyframe; upon receiving the new address keyframe, it performs a continuous file determination. If writing can continue, it returns to the recording state to continue appending; if splitting is necessary, it closes the original MP4 file, creates a new MP4 file, and returns to the recording state. When the user stops, there is no continuing address, or takeover fails, it enters a closed state. When the old task times out, ownership of the old task is restored.
[0032] S150, after the continuous file determination is completed, only the current owner task of the recording context closes the MP4 file and sends a recording end notification, and the recording end notification is sent only once.
[0033] Specifically, only the current owner's task in the recording context can close the file and send a recording completion notification, subject to three checks before execution: 1. Task permission check: The task sequence number matches the current owner's task sequence number (ownerTaskSeq), or the owner's task sequence number is zero; 2. Sending permission check: The task is allowed to send a completion notification; 3. Status check: The task has not yet sent a completion notification. If the completion, playback address invalidation, or failure notification of the replaced old task is not met, the above checks are not satisfied if the context has taken over or the conditions for resuming playback are still met, and the file cannot be closed or a completion notification cannot be sent. The file is only closed and a recording completion notification is sent only once when the user actively stops the task, there is no resuming playback address, takeover fails, or a new task determines that closure is necessary. Similarly, if the completion, playback address invalidation, or failure notification of the replaced old task is not closed if the context has taken over or the conditions for resuming playback are still met, the file is only closed and a recording completion notification is sent only once when the user actively stops the task, there is no resuming playback address, takeover fails, or a new task determines that closure is necessary. A task termination notification must also meet the following conditions: the task is allowed to send termination notifications, the task number is still the current task, and the task has not yet sent a termination notification. Therefore, an old task that has been replaced by a new task cannot close an MP4 file being used by the new task.
[0034] Please refer to Figure 4 , Figure 4This is a schematic diagram of the MP4 container writing and closing process provided in an embodiment of this application. When writing can continue, the current MP4 writer is not closed, and the file header is not recreated. When the MP4 writer is open, it first writes the ftyp, then the mdat header, and continuously appends video and audio samples to the mdat. When splitting or finally stopping recording is required, closing occurs in the following order: flushing out the last remaining video and audio data blocks; updating the actual length of the mdat; updating the video and audio sample table and timeline index, including stts, stsz, stsc, stss, and stco; writing moov according to the current sample table; and closing the current MP4 file. In cases of parameter incompatibility, timestamp rollback, or user-stopped recording, the current MP4 is closed first; if recording still needs to continue, a new MP4 writer is created. In this way, each closed MP4 has a corresponding sample table and moov index information, and can be decoded and played as an independent MP4 file.
[0035] The recording and display links are layered: In the distribution thread, the same encoded media packet first enters the recording link, then the display link for high-speed frame drop detection. The recording link directly receives the complete encoded media packet and performs keyframe recognition, image group buffering, MP4 continuation or splitting; the display link then decides whether to discard non-critical video frames based on the playback speed. Therefore, in high-speed display scenarios, even if the display link discards non-critical video frames, the recording link has already obtained the corresponding encoded media packet and is unaffected by display frame drops. Keyframes, audio frames, and the first frame continue to be distributed according to the display link rules.
[0036] This embodiment achieves continuous MP4 recording control across URL HLS playback through the organic integration of the above steps: the recording context, independent of the playback address task, decouples the recording state from the single-address playback task, allowing the same recording to persist across multiple short-lifetime addresses; the address switching stage completes the cross-task handover of the recording context, while simultaneously performing silent final state control on the replaced old task to prevent the asynchronous exit of the old task from interfering with the recording of the new task; gating verification before writing is implemented using the first keyframe of the new address as the write trigger point, and no new address content is written to the existing file before the keyframe is received; after the first keyframe arrives, the file continuation or split is determined by the joint judgment result of encoding parameter compatibility and media time rollback, and the original file is appended only when the parameters are compatible and the time is continuous; finally, only the task currently holding the recording ownership executes the file closing and final state notification, ensuring the uniqueness of file operation rights. In scenarios with frequent playback address switching, the entire mechanism not only ensures the continuous semantics of user recording, but also ensures the encapsulation integrity and independent playability of each output MP4 file.
[0037] To further illustrate the above method, this application will further elaborate on the solution by combining two scenarios: long-term playback and high-speed playback.
[0038] This embodiment uses a long-duration cross-address playback scenario as an example. In the current cloud playback scenario, one HLS address typically corresponds to a playback span of approximately 10 seconds, and the same playback continuously passes through N HLS addresses. The traditional address switching, i.e., file splitting, would create N MP4 files of approximately 10 seconds each, requiring the user to open, switch, and reposition among these files during playback. This solution maintains the same MP4 writer for address switching with compatible parameters and consecutive timestamps. If there are K consecutive files after file termination in this playback, then 1≤K≤N is satisfied. When all addresses can be continued writing, the number of MP4 files is reduced from N to 1, and the reduction in the number of files is proportional to... As shown in formula (2): (2); Where N represents the number of HLS addresses experienced during this playback. The number of times a user switches between MP4s is reduced from N-1 to 0. In cases of parameter incompatibility, timestamp rollback, or abnormal switching, the solution preserves the independent playable structure of the written portion by first closing the current MP4 and then creating a new MP4, instead of forcibly writing incompatible media into the same file. In abnormal scenarios: when there are no keyframes for a long time at a new address, the system remains in a state of waiting for the keyframe of the new address and does not write new address data to the current MP4; when encoding parameters change frequently, the audio and video parameters of each new address are compared with the baseline of the current MP4 parameters, and any incompatibility will close the current MP4 at the keyframe of the new address and create a new MP4; when media time rollback is repeated, each new address keyframe is compared according to a 500-millisecond threshold, and if the threshold is exceeded, the file is split; when playback addresses are switched frequently, the switching occupation flag is serialized, and subsequent requests will return a busy session if the previous round of switching has not ended.
[0039] This embodiment uses a high-speed playback scenario as an example. When a user watches playback at a speed higher than normal, the display link will discard some non-critical video frames to maintain playback performance. In this embodiment, the recording link is placed before the display link's high-speed frame dropping judgment. The same encoded media packet first enters the recording link completely, and then enters the display link. The recording link performs keyframe recognition, image group caching, and MP4 writing, writing the complete audio and video media packet; the display link then discards non-critical video frames according to the speed rule. Therefore, the local MP4 recording content is not affected by the display degradation strategy. When a user starts recording midway through playback, the recording context first enters a waiting state for the first keyframe: if the most recent keyframe group cache already has an image group starting with the keyframe, the MP4 writer is opened, and the image group is written to the MP4 in cache order first, and then the current and subsequent media packets are written; if the cache is not yet available, media packets are continued to be received until a keyframe is received, then the MP4 is opened, and the cached image group and the current keyframe are written. In this way, the starting video sample of the MP4 starts from the keyframe, avoiding empty files or loss of the starting screen due to waiting for the next keyframe. It should be noted that when switching across URLs, the most recent keyframe group of the old address is not used to write to the current MP4. The new address must actually receive the new keyframe before it can perform the continuation or split determination.
[0040] This embodiment applies to HLS address changes caused by address refresh, signature address update, or continuous playback window switching within the same cloud playback session. If the video encoding configuration, frame rate, or audio parameters remain compatible after the address change, and the timestamp of the new address keyframe does not roll back beyond a preset threshold, then the same MP4 file is continued. Switches involving resolution changes, encoding profile changes, or other changes that alter the video encoding configuration data do not continue writing the same MP4 file; instead, a new MP4 file is created at the new address keyframe. Changes in audio sampling rate, sampling bit width, number of channels, frame rate, bitrate, or audio encoding also result in splitting. If a change in video bitrate does not alter the video configuration data, encoding type, or frame rate used by the writer, writing can continue; if it causes changes in the aforementioned parameters, then splitting occurs.
[0041] For typical abnormal scenarios, the system's handling strategies are as follows: 1. No keyframes for a new address for a long time: Maintain the state of waiting for keyframes for the new address and do not write new address data to the current MP4; a timeout split strategy can be configured, after which the current MP4 is closed and a new file is created or manual processing is initiated; 2. Frequent changes in encoding parameters: The audio and video parameters of each new address are compared with the baseline parameters of the current MP4; if any incompatibility occurs, the current MP4 is closed at the keyframe of the new address and a new MP4 is created; 3. Repeated media time rewind: Each new address keyframe is compared according to a preset threshold; if the threshold is exceeded, the file is split to avoid media timeline rewind within an MP4; 4. High-frequency switching of playback addresses: Switching between the same cloud ID and the same channel is serialized by the switching occupation marker; if the previous round of switching has not yet ended, subsequent requests will return a busy session, and the recording context will not be handed over concurrently; 5. Old task stops timeout or arrives late at the end state: Restore the recording control of the old task; the late end state is intercepted by the task sequence number and the end state notification switch, and does not affect the current recording file.
[0042] In a preferred embodiment, this solution can be further extended in the following directions: First, a configurable timeout splitting strategy is set for waiting for keyframes at new addresses. When a keyframe is not received at a new address within a preset time, the current MP4 is captured and a new MP4 is created or manual processing is initiated to adapt to abnormal scenarios where the live source has no keyframes for a long time. Second, a persistence and crash recovery mechanism is added to the recording context. The key fields of the recording context are periodically written to disk. When the client exits abnormally and restarts, the recording state is restored based on the persistent fields, and the uncaptured MP4s are captured, thereby improving the reliability of the recording. Third, this solution is extended to playback scenarios of other segmented transmission protocols, such as playback scenarios based on MPEG-DASH, to handle continuous recording when segmented address switching using the same approach.
[0043] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of a continuous MP4 recording system 20 with cross-URL HLS playback, as described in some embodiments of this application. This embodiment further illustrates the aforementioned method embodiments in conjunction with the detailed working process of each module of the system. The continuous MP4 recording system includes a recording context management module, a task handover module, a keyframe gating module, a continuous file determination module, and a final state termination module. Each module corresponds one-to-one with each step in the aforementioned method embodiments, as detailed below: The recording context management module 210 is used to establish a recording context independent of the playback address task. The recording context includes the logical playback identity corresponding to the same device identifier and channel identifier, the recording request status, the current MP4 file handle, the baseline of audio and video encoding parameters of the opened file, the most recent media writing time, the key frame status, the most recent key frame group cache, and the current owner task sequence number, corresponding to method step S110. The task handover module 220 is used to respond to the playback address switch by comparing the new playback address with the old playback address. If the playback addresses are different, a new task with a new task number is created. The new task takes over the recording context and updates the current owner task number. The old task is marked as replaced and its recording end status notification is suppressed, corresponding to method step S120. The keyframe gating module 230 is used to receive media packets at a new address while waiting for the first keyframe of the new address, and not to write video content at the new address to the current MP4 file before receiving the video keyframe, corresponding to method step S130. The continuous file determination module 240 is used to perform continuous file determination when the first keyframe of the new address arrives. If the encoding parameters are incompatible or the media time rollback exceeds the preset threshold, the original file is closed and a new MP4 file is created. Otherwise, the original file handle is kept and audio and video media packets are appended, corresponding to method step S140. The final state closing module 250 is used to close the MP4 file and send a recording final state notification only by the current owner task of the recording context, and the recording final state notification is sent only once, corresponding to method step S150.
[0044] It should be noted that this embodiment achieves a complete recording control link from recording context establishment to final state termination through the collaborative work of the above modules. Each module has a clear responsibility and a clear interface, making it easy to deploy in monitoring clients, cloud playback clients, browser playback plugins, or integrated recording and broadcasting machines.
[0045] Please refer to Figure 6 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 6 As shown, the electronic device 30 includes: a processor 300, a memory 301, a bus 302, and a communication interface 303. The processor 300, the communication interface 303, and the memory 301 are connected via the bus 302. The memory 301 stores a computer program that can run on the processor 300. When the processor 300 runs the computer program, it executes the method described above in this application.
[0046] The memory 301 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.
[0047] Bus 302 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs. After receiving an execution instruction, the processor 300 executes the program. The methods disclosed in any of the foregoing embodiments of this application can be applied to the processor 300, or implemented by the processor 300.
[0048] The processor 300 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 300 or by instructions in software form. The processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 301. The processor 300 reads the information in memory 301 and, in conjunction with its hardware, completes the steps of the above method.
[0049] The electronic device and the method provided in this application are based on the same inventive concept and have the same beneficial effects as the methods used, operated, or implemented therein. The electronic device can be a monitoring client, cloud playback client, browser playback plugin, recording and broadcasting all-in-one machine, or edge recording server, etc., that has audio and video playback and local recording capabilities.
[0050] This application also provides a computer-readable medium corresponding to the method provided in the foregoing embodiments. Please refer to... Figure 7 The computer-readable storage medium shown is an optical disc 40, on which a computer program (i.e., a program product) is stored, which, when run by a processor, performs the aforementioned method.
[0051] The computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. In a specific application scenario, the computer-readable storage medium may be built into a monitoring client or recording and broadcasting unit as part of the firmware; or it may be a standalone software product.
[0052] The computer-readable storage medium provided in the above embodiments of this application and the method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.
[0053] This embodiment also provides a computer program product, which includes a computer program. When executed by a processor, the computer program implements the methods described in any of the above embodiments.
[0054] The computer program product may include any combination of one or more program modules.
[0055] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0056] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0057] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0058] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0059] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application.
Claims
1. A method for continuous MP4 recording with cross-URL HLS playback, characterized in that, Includes the following steps: Establish a recording context, which is independent of the playback address task. The recording context includes the logical playback identity corresponding to the same device identifier and channel identifier, the recording request status, the current MP4 file handle, the baseline of audio and video encoding parameters of the opened file, the last time the media was written, the keyframe status, the most recent keyframe group cache, and the current owner task number. In response to a playback address switch, the new playback address is compared with the old playback address: if the new playback address is the same as the old playback address and the old task has not ended, the current task remains unchanged; When the new playback address is different from the old playback address, a new task with a new task number is created, the new task takes over the recording context and updates the current owner task number, the recording context is set to wait for the first keyframe of the new address, the old task is marked as replaced, and the recording end status notification of the old task is suppressed. In the state of waiting for the first keyframe of the new address, a media packet with a new address is received, and video keyframe identification is performed on the media packet. Before the video keyframe is identified, the video content of the new address is not written to the current MP4 file. When a video keyframe is detected while waiting for the first keyframe of the new address, a continuous file determination is performed: if the audio and video encoding parameters of the new task are incompatible with the baseline of the audio and video encoding parameters of the already opened file, the original file is closed and a new MP4 file is created; if the media time of the new keyframe rolls back more than a preset threshold relative to the most recently written media time, the original file is closed and a new MP4 file is created; if the encoding parameters are compatible and the media time has not rolled back more than the preset threshold, the original file handle is maintained and the audio and video media packet of the new address is appended to the current MP4 file. After the continuous file determination is completed, the MP4 file is closed and a recording end notification is sent only by the current owner task of the recording context, and the recording end notification is sent only once.
2. The method according to claim 1, characterized in that, The recording context also includes: a mutex lock, recording start notification status, recording end notification status, file sequence number, file path, latched audio and video parameters of the current playback task, valid status of opened file parameters, address switching status, and MP4 write information.
3. The method according to claim 1, characterized in that, After the new task takes over the recording context, the method further includes: if the old task does not stop within a preset waiting time, no new address data is written to the current MP4 file, the recording ownership of the old task is restored, and the current switch is rejected; switching requests for the same logical playback identity are processed serially, and if the previous round of switching is not finished, subsequent switching requests return "session busy".
4. The method according to claim 1, characterized in that, The identification of video keyframes includes: When a media packet is a video packet and the media packet extension identifier indicates a keyframe, it is determined to be a keyframe; when the media packet extension identifier does not indicate a keyframe, the network abstraction layer units in the video payload are parsed. Units with a network abstraction layer type of 5 in H.264 encoding are determined to be keyframes, and units with a network abstraction layer type of 16 to 21 in H.265 encoding are determined to be random access frames.
5. The method according to claim 1, characterized in that, Following the establishment of the recording context, the following is also included: Maintain a cache of the most recent keyframe group, which stores the most recent group of images from the start of a keyframe to the current media pack; When a new keyframe is received, the old buffer is cleared and a new buffer is started with the new keyframe. When a user starts recording midway, if the cache contains an image group starting with a keyframe, the MP4 writer is opened, and the image group in the cache is first written to the MP4 file in the cache order, and then the current and subsequent media packets are written; if the cache does not contain the image group, the MP4 writer is opened after receiving a keyframe, and the image group in the cache and the current keyframe are written.
6. The method according to claim 1, characterized in that, The audio and video encoding parameters include video parameters and audio parameters. The video parameters include video encoding type, frame rate, and video encoding configuration data. The audio parameters include whether audio is enabled, audio encoding type, sampling rate, sampling bit width, number of channels, audio frame rate, and bit rate. The audio and video encoding parameters of the new task are compared with each item in the baseline of the audio and video encoding parameters of the already opened file. If any item is different, it is determined that the audio and video encoding parameters are incompatible. If the audio encoding is not supported, or the sampling rate or number of channels does not meet the writer's conditions, the MP4 file will be opened as if there is no audio.
7. The method according to claim 1, characterized in that, The determination of media time rollback includes: adding the preset threshold to the media timestamp of the new keyframe; when the sum is less than the most recently written media time, media time rollback is determined, the original file is closed and a new MP4 file is created.
8. The method according to claim 1, characterized in that, The operation of closing the MP4 file includes: Retrieve the last remaining video and audio data blocks not yet written to disk; update the actual length of mdat; update the video and audio sample table and timeline index, where the sample table includes stts, stsz, stsc, stss, and stco; write moov based on the current sample table; close the current MP4 file.
9. The method according to claim 1, characterized in that, Before appending the audio and video media packet from the new address to the current MP4 file, the method further includes: sending the same encoded media packet first to the recording link, and then sending it to the display link for high-speed frame loss judgment; the recording link receives the complete encoded media packet and performs keyframe recognition, image group caching and MP4 writing; the display link discards non-critical video frames according to the playback speed; the frame loss processing of the display link does not change the content of the MP4 file written by the recording link.
10. A continuous MP4 recording system with cross-URL HLS playback, characterized in that, include: The recording context management module is used to establish a recording context independent of the playback address task. The recording context includes the logical playback identity corresponding to the same device identifier and channel identifier, the recording request status, the current MP4 file handle, the baseline of audio and video encoding parameters of the opened file, the time of the most recent media write, the keyframe status, the most recent keyframe group cache, and the current owner task sequence number. The task handover module is used to compare the new playback address with the old playback address in response to a playback address switch: when the new playback address is the same as the old playback address and the old task has not ended, the current task remains unchanged. When the new playback address is different from the old playback address, a new task with a new task number is created, the new task takes over the recording context and updates the current owner task number, the recording context is set to wait for the first keyframe of the new address, the old task is marked as replaced, and the recording end status notification of the old task is suppressed. The keyframe gating module is used to receive media packets with a new address in the state of waiting for the first keyframe of the new address, to identify video keyframes in the media packets, and not to write video content with the new address to the current MP4 file before the video keyframe is identified. The continuous file determination module is used to perform continuous file determination when a video keyframe is identified in the state of waiting for the first keyframe of the new address: when the audio and video encoding parameters of the new task are incompatible with the baseline of the audio and video encoding parameters of the already opened file, the original file is closed and a new MP4 file is created; when the media time of the new keyframe rolls back more than a preset threshold relative to the most recently written media time, the original file is closed and a new MP4 file is created; when the encoding parameters are compatible and the media time has not rolled back more than the preset threshold, the original file handle is maintained and the audio and video media packet of the new address is appended to the current MP4 file; The final state closing module is used to close the MP4 file and send a recording final state notification only by the current owner task of the recording context after the continuous file determination is completed, and the recording final state notification is sent only once.
