Media stream reorganization method, system, electronic device, and readable storage medium

Abstract

The application discloses a media stream reorganization method and system, electronic equipment and a readable storage medium, and relates to the technical field of media streams. The method comprises the following steps: obtaining a target media frame and a corresponding target timestamp, and obtaining a previous valid media frame and a corresponding valid timestamp; verifying the validity of the target timestamp according to the valid timestamp; if the target timestamp passes the validity verification, performing frame loss detection based on the valid timestamp and the target timestamp; if there is frame loss, generating one or more compensation media frames and corresponding compensation timestamps according to the previous valid media frame, the target media frame and a reference time increment, and writing the compensation media frames into the media stream; and repeatedly performing the above steps for each target media frame in the media stream to complete the media stream reorganization. In this way, the timing integrity and synchronization accuracy of the media stream are significantly improved, frame loss is effectively identified and filled, and audio and video synchronization, lag or jump are avoided.

Description

Technical Field

[0001] This invention relates to the field of media streaming technology, and in particular to a media streaming reconstruction method, system, electronic device, and readable storage medium. Background Technology

[0002] In the entire video recording chain of embedded camera devices, from image sensor acquisition, image preprocessing, multi-threaded encoding, queue caching to local writing / network streaming, any performance fluctuation, resource contention, or external interference in any link can cause frame anomalies and timestamp anomalies.

[0003] Currently, industry solutions for handling abnormal timestamps in media frames within media streams are generally quite basic. For example, directly removing frames with invalid timestamps or abnormal intervals can avoid player errors, but it artificially creates image gaps, shortens video length, and results in a strong sense of jumpiness in dynamic scenes, which is completely unacceptable for scenarios with high requirements for image integrity, such as motion shooting and security evidence collection. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a media stream reconstruction method, system, electronic device and readable storage medium.

[0005] This invention provides the following technical solution: In a first aspect, the present invention provides a media stream reassembly method, the method comprising: Get the target media frame and its corresponding target timestamp, and get the previous valid media frame and its corresponding valid timestamp; Verify the validity of the target timestamp based on the valid timestamp. If the target timestamp passes the validity verification, then frame loss detection is performed based on the valid timestamp and the target timestamp; If there are lost frames, one or more compensation media frames and corresponding compensation timestamps are generated based on the previous valid media frame, the target media frame and the reference time increment, and the compensation media frames are written to the media stream. For each target media frame in the original media stream, repeat the above steps to complete the media stream reconstruction.

[0006] In an optional implementation, the validity of the target timestamp is verified based on the valid timestamp, including: Determine whether the target timestamp is less than or equal to the valid timestamp; If the target timestamp is less than or equal to the valid timestamp, the target timestamp is deemed to have failed the validity verification; otherwise, the target timestamp is deemed to have passed the validity verification.

[0007] In an optional implementation, the method further includes: If the target timestamp fails the validity verification, a reference time increment will be added to the valid timestamp to obtain a new target timestamp for the target media frame.

[0008] In an optional implementation, the method further includes: If the target timestamp fails the validity verification, the target timestamp will be matched with the compensation timestamps of all compensation media frames written in the media stream. If a match is found, the compensation media frame corresponding to the matching compensation timestamp will be replaced with the target media frame, and the matching compensation timestamp will be used as the new target timestamp of the target media frame.

[0009] In an optional implementation, frame loss detection is performed based on the valid timestamp and the target timestamp, including: If the difference between the target timestamp and the valid timestamp is greater than the maximum allowable error threshold, then it is determined that there is a frame loss. The maximum allowable error threshold is a preset multiple of the reference time increment.

[0010] In an optional implementation, one or more compensated media frames and corresponding compensated timestamps are generated based on the previous valid media frame, the target media frame, and the reference time increment, including: Use the previous valid media frame as the reference media frame; Copy the reference media frame as the compensation media frame, and overlay a reference time increment onto the timestamp corresponding to the reference media frame as the compensation timestamp. Write the compensation media frame into the media stream and update the compensation media frame to the reference media frame; If the difference between the target timestamp and the compensation timestamp is greater than the maximum allowable error threshold, return to the step of copying the reference media frame as the compensation media frame until the difference between the target timestamp and the compensation timestamp is less than or equal to the maximum allowable error threshold.

[0011] In an optional implementation, the method further includes: After generating each compensation media frame, determine whether the number of compensation media frames generated has reached the preset maximum number of compensation frames per time; if so, terminate the step of copying the reference media frame as the compensation media frame in advance.

[0012] In a second aspect, the present invention provides a media stream reassembly system, the system comprising: The acquisition module is used to acquire the target media frame and its corresponding target timestamp, and to acquire the previous valid media frame and its corresponding valid timestamp. The verification module is used to verify the validity of the target timestamp based on the valid timestamp. The detection module is used to detect dropped frames based on the valid timestamp and the target timestamp; The generation module is used to generate one or more compensation media frames and corresponding compensation timestamps based on the previous valid media frame, the target media frame, and the reference time increment if frame loss occurs, and write the compensation media frames into the media stream.

[0013] Thirdly, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the media stream reconstruction method as described in any of the foregoing embodiments.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a media stream reconstruction method as described in any of the foregoing embodiments.

[0015] This invention discloses a media stream reassembly method, system, electronic device, and readable storage medium. The method involves acquiring a target media frame and its corresponding target timestamp, as well as the previous valid media frame and its corresponding valid timestamp. The target timestamp is validated based on its valid timestamp. If the target timestamp passes validation, frame loss detection is performed based on both the valid and target timestamps. If frame loss exists, one or more compensation media frames and their corresponding compensation timestamps are generated based on the previous valid media frame, the target media frame, and a reference time increment, and these compensation media frames are written into the media stream. For each target media frame in the media stream, the above steps are repeated to complete the media stream reassembly. This significantly improves the temporal integrity and synchronization accuracy of the media stream through timestamp validity verification and dynamic compensation mechanisms; effectively identifies and fills in lost frames, avoiding audio-visual asynchrony, stuttering, or jumps; eliminates the need for retransmission from the original encoding end, reducing network bandwidth and latency sensitivity, and enhancing terminal playback stability and user experience consistency. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope of protection of the present invention. In the various drawings, similar components are numbered similarly.

[0017] Figure 1 A flowchart of the media stream reassembly method proposed in this embodiment is shown; Figure 2 Another flowchart of the media stream reassembly method proposed in this embodiment is shown; Figure 3 A schematic diagram of the frame loss compensation proposed in this embodiment is shown; Figure 4 A schematic diagram of the media stream proposed in this embodiment is shown; Figure 5This embodiment illustrates a recording pause and resume diagram. Figure 6 A schematic diagram of the media stream reassembly system proposed in this embodiment is shown.

[0018] Explanation of reference numerals in the attached diagram: 600 - Media Stream Reassembly System; 601 - Acquisition Module; 602 - Verification Module; 603 - Detection Module; 604 - Generation Module. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0020] The components of the embodiments of the invention described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0021] In the following, the terms “comprising,” “having,” and their cognates, which may be used in various embodiments of the invention, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as excluding, firstly, the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations thereof.

[0022] Furthermore, the terms "first," "second," and "third" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0023] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0024] Example 1 This disclosure provides a media stream reassembly method specifically for embedded camera devices' mobile video capture links, primarily applicable to high-definition video recording devices such as action cameras, PTZ cameras, live streaming terminals, and security cameras. It is adaptable to standard definition, high definition, and ultra-high definition resolutions, as well as high frame rate recording scenarios such as 30fps, 60fps, and 120fps. It is also compatible with all service links including local storage recording, real-time network streaming, and local playback, resolving issues such as frame loss, timestamp corruption, and audio-visual asynchrony throughout the entire video capture, encoding, and transmission process.

[0025] Please see Figure 1 The media stream reassembly method includes steps S101 to S104, and each step is described in detail below.

[0026] Step S101: Obtain the target media frame and its corresponding target timestamp, and obtain the previous valid media frame and its corresponding valid timestamp.

[0027] In this embodiment, the media stream includes a video stream and / or an audio stream; the video stream is used as an example below.

[0028] The system receives the current media frame to be processed during video recording as the target media frame, uses the target media frame's timestamp (PTS) as the target timestamp, and simultaneously obtains the previous valid media frame and its corresponding valid timestamp. The previous valid media frame is the last media frame successfully written to the media stream timeline.

[0029] Understandably, the previous valid media frame serves as the sole comparison object for determining whether the target media frame is in the correct timing sequence, which is a prerequisite for ensuring the correctness of all subsequent processing logic. At the same time, it can also serve as the data source for timestamp correction and frame loss compensation, restoring the monotonically increasing timing sequence and filling in the blank screen and timeline gaps caused by frame loss.

[0030] Step S102: Verify the validity of the target timestamp based on the valid timestamp.

[0031] In this embodiment, the timing validity of the target timestamp is verified using a valid timestamp, and all abnormal frames with reversed or reverted timing are intercepted.

[0032] It should be noted that due to asynchronous encoding, out-of-order output of frame data by multiple threads, reversed read / write order of the buffer queue, delay in processing the preceding frame, and priority output of the following frame, the target timestamp may be out of order.

[0033] Therefore, using timestamp validity verification avoids issues such as player decoding logic errors, screen tearing, and player crashes caused by out-of-order timestamps, thereby preventing terminal playback compatibility failures, video file corruption, and recording task interruptions.

[0034] In one specific embodiment, step S102 includes: determining whether the target timestamp is less than or equal to the valid timestamp; if the target timestamp is less than or equal to the valid timestamp, then the target timestamp fails the validity verification; otherwise, the target timestamp passes the validity verification.

[0035] In this embodiment, validity is verified by comparing the target timestamp of the current target media frame to be processed with the valid timestamp of the previous valid media frame. Specifically, if the target timestamp is less than or equal to the valid timestamp, it is known that the media frame corresponding to the valid timestamp is output as a later frame first, while the media frame corresponding to the target timestamp is output as a earlier frame with a delay. Therefore, the target timestamp is determined to be an invalid timestamp and fails the validity verification. Conversely, if the target timestamp is greater than the valid timestamp, it is known that the target timestamp passes the validity verification and can proceed to the subsequent frame loss detection process.

[0036] In an optional embodiment, for step S102 above, the method further includes: if the target timestamp fails the validity verification, then superimposing a reference time increment on the valid timestamp to obtain a new target timestamp for the target media frame.

[0037] In this embodiment, if the target timestamp fails the validity verification, the target timestamp will be corrected to obtain a new target timestamp, and the target media frame will be written into the media stream based on the new target timestamp to eliminate timing anomalies caused by timestamp rollback or duplication, and to force the global frame PTS to strictly monotonically increase, fully complying with mainstream encoding standards such as H.264 / H.265.

[0038] It should be noted that the correction process includes: using the valid timestamp as a reference, superimposing a reference time increment to obtain the new target timestamp of the target media frame.

[0039] The reference time increment can be one of the following: the frame rate of the media stream, the sampling rate, the actual number of sampling points, the target output frame rate, the average frame timestamp interval, the encoder time base, and the container stream time base. This embodiment does not limit the specific time increment.

[0040] Understandably, existing methods that directly discard frames with illegal PTS or abnormal intervals, while avoiding player errors, artificially create image fragmentation, shorten video length, and produce a strong sense of jumpiness in dynamic scenes, making them completely unsuitable for scenarios with high requirements for image integrity, such as motion shooting and security evidence collection. This embodiment, however, forces a strict incrementing PTS through timestamp correction, conforming to international audio and video coding standards, ensuring compatibility with all types of players and editing software, and preventing decoding crashes and file corruption.

[0041] Step S103: If the target timestamp passes the validity verification, then frame loss detection is performed based on the valid timestamp and the target timestamp.

[0042] In this embodiment, if the target timestamp passes the validity verification, the time interval between the valid timestamp and the target timestamp is used to detect whether a frame drop has occurred between the valid timestamp and the target timestamp.

[0043] In one specific embodiment, step S103 includes: if the difference between the target timestamp and the valid timestamp is greater than the maximum error allowable threshold, then it is determined that there is frame loss.

[0044] In this embodiment, the difference between the target timestamp and the valid timestamp is calculated, and it is determined whether the difference is greater than the maximum error allowable threshold. If the difference is greater than the maximum error allowable threshold, it is determined that there is currently a frame loss.

[0045] It should be noted that the maximum allowable error threshold is a preset multiple of the reference time increment. For example, when the reference time increment is the frame rate, the maximum allowable error threshold is 3 / frame rate seconds. However, when the media device is a low-stability network / old hardware, the maximum allowable error threshold can be 5 / frame rate seconds; when the media device is a high-performance device, the maximum allowable error threshold can be 2 / frame rate seconds. This embodiment does not impose specific limitations.

[0046] It should be noted that when a frame is lost, a time interval log can be printed to facilitate backend problem tracing and performance analysis.

[0047] In step S104, if there are lost frames, one or more compensation media frames and corresponding compensation timestamps are generated based on the previous valid media frame, the target media frame and the reference time increment, and the compensation media frames are written into the media stream.

[0048] In this embodiment, frame replacement is triggered when frame loss is detected. The replacement frame reuses the complete image data of the previous valid media frame as the content of the compensation media frame, and uses a reference time increment to synchronously and incrementally assign a continuous compensation timestamp to each generated compensation media frame until the time interval difference after compensation returns to the normal range, thereby filling the timeline gap caused by frame loss.

[0049] If there are no dropped frames, the target media frame is written to the video stream according to the target timestamp, and the target media frame is updated to the previous valid media frame.

[0050] For each target media frame to be processed currently read from the original media stream, repeat steps S101 to S104 to complete the media stream reassembly.

[0051] It is understandable that frame drops can be triggered by insufficient hardware computing power, system scheduling, link blockage, external interference, service switching, etc. Existing methods can batch-fill frames at preset fixed frame intervals. When the device frame rate fluctuates or there are pause intervals, the frame filling timeline is distorted, which further aggravates the audio-visual desynchronization. Alternatively, they may only log the data without making any repairs.

[0052] In this embodiment, the frame interval is detected to identify dropped frames, and the previous valid media frame is reused for compensation. This systematically solves problems such as frame loss, timestamp disorder, and audio-visual asynchrony in video recording with low computing power.

[0053] Please see Figure 2 In one specific embodiment, step S104 includes steps S1041 to S1045, and each step is described in detail below.

[0054] Step S1041: Use the previous valid media frame as the reference media frame.

[0055] In this embodiment, the previous valid media frame is used as the reference media frame. The reference media frame is used to provide the video image content and display timestamp reference for compensating the lost frame, thereby ensuring that the image content source on which the compensation operation depends is reliable and clear, avoiding abnormal compensation frame images due to errors in the reference source itself, and thus ensuring that the subsequent filled frames have good visual continuity and consistency.

[0056] Step S1042: Copy the reference media frame as the compensation media frame, and overlay a reference time increment onto the timestamp corresponding to the reference media frame as the compensation timestamp.

[0057] In this embodiment, the complete and valid image data of the reference media frame is copied to obtain a new media frame. This new media frame is used as a compensation media frame to compensate for the lost frame. The timestamp corresponding to the reference media frame is used as a reference, and a reference time increment is superimposed as the timestamp of the compensation media frame, i.e., the compensation timestamp.

[0058] Understandably, by copying the content of the same reference frame, a frame can be quickly generated without the need for complex interpolation algorithms, reducing computational overhead and preventing the generation of false image pixels, thus ensuring image quality. Simultaneously, by overlaying a fixed reference time increment to assign a timestamp to the compensation frame, the interval between the compensation frame and the preceding and following frames on the timeline is ensured to be uniform, maintaining the smoothness of the media stream from a temporal perspective and avoiding playback stuttering or image jumps caused by missing frames.

[0059] It should be noted that if the compensation timestamp generation fails, the current frame interpolation action is skipped, and the next target media frame is processed.

[0060] Step S1043: Write the compensation media frame into the media stream and update it to the reference media frame.

[0061] In this embodiment, a compensation media frame is written into the media stream to fill the gaps caused by frame loss or missing frames, ensuring the integrity of the number of frames and the continuity of time in the media stream; at the same time, the compensation media frame is updated as a reference media frame, replacing the original reference media frame, and serving as the benchmark for the next round of compensation operations.

[0062] It should be added that special tags or records can be added to all compensation media frames to identify them.

[0063] Step S1044: Determine whether the difference between the target timestamp and the compensation timestamp is greater than the maximum error allowable threshold.

[0064] If yes, return to step S1042 until the difference between the target timestamp and the compensation timestamp is less than or equal to the maximum error allowable threshold; if no, proceed to step S1045.

[0065] Step S1045: Stop the cyclic compensation operation.

[0066] In this embodiment, if the difference between the target timestamp being processed and the currently generated compensation timestamp is greater than the maximum error allowable threshold, then the time point of the current compensation media frame has not yet reached the vicinity of the target time point. It is necessary to return to step S1042 to generate a new compensation media frame and its corresponding compensation timestamp, until the difference between the target timestamp and the newly generated compensation timestamp is less than or equal to the maximum error allowable threshold, then the cyclic compensation operation is stopped.

[0067] Understandably, by setting a maximum allowable error threshold and performing compensation iteratively, the number of compensation frames can be precisely controlled, preventing both insufficient compensation leading to missing frames and excessive compensation introducing redundant frames. This method ensures seamless temporal alignment between the compensated media stream and the original valid frames, guaranteeing high synchronization between the video and timeline during playback, thus improving the accuracy of media stream reconstruction and the playback experience. Furthermore, this iterative mechanism can adaptively adjust the number of compensation frames based on the actual duration of missing frames, exhibiting strong robustness.

[0068] For example, please see Figure 3 When 3 frames are lost between the target timestamp PTS4 and the effective timestamp PTS1, 3 compensation frames are added sequentially based on PTS1. The timeline progresses at a constant speed, the picture does not jump, and the total video length is consistent with the theoretical length.

[0069] In one specific embodiment, the method further includes: after generating each compensation media frame, determining whether the number of compensation media frames generated has reached a preset maximum number of single compensation frames; if so, terminating the step of copying a reference media frame as a compensation media frame in advance.

[0070] In this embodiment, a preset maximum number of single compensation frames is used to constrain the maximum number of consecutive compensation frames.

[0071] After each compensation media frame is successfully generated, the total number of compensation media frames generated consecutively in the current single compensation session is counted and compared with the preset maximum number of compensation frames per session. If the count value has reached the preset maximum number of compensation frames per session, a termination operation is immediately triggered, forcibly exiting the loop execution step S1042, and no new compensation media frames are generated, even if the difference between the target timestamp and the last compensation timestamp still does not meet the maximum error allowable threshold condition, in order to prevent the compensation operation from continuing indefinitely under abnormal conditions and avoid infinitely consuming resources.

[0072] For security and long-duration recording scenarios, the maximum single-time compensation frame count can be preset to a range of 20 to 30 to prevent extreme frame loss loops; for live streaming scenarios, the maximum single-time compensation frame count can be preset to a range of less than 10. This embodiment does not impose specific limitations.

[0073] It should be added that the compensation frames are dynamically generated based on the iOS / macOS standard CMSampleBuffer interface. This interface natively supports multiple platforms, ensuring a unified frame data format, strong universality, and low processing latency. At the same time, the generation process adopts a fault-tolerant mechanism. If the historical frame buffer is empty, the call returns a null value (i.e., the optional type is nil), and the current frame compensation operation is immediately terminated. Only the log is recorded to prevent program crashes and ensure the stability of media stream reconstruction.

[0074] In an optional embodiment, the method further includes: if the target timestamp fails the validity verification, matching the target timestamp with the compensation timestamps of all compensation media frames written in the media stream; if the match is successful, replacing the compensation media frame corresponding to the successfully matched compensation timestamp with the target media frame, and using the successfully matched compensation timestamp as the new target timestamp of the target media frame.

[0075] In this embodiment, if the target timestamp fails the validity verification, before correcting the target timestamp, the target timestamp can be compared one by one with the compensation timestamps of all written compensation media frames in the current media stream to achieve timestamp matching. If the target timestamp is equal to or differs from a certain compensation timestamp within a preset tolerance range, the match is considered successful.

[0076] When a target timestamp successfully matches a compensation timestamp, the compensation media frame corresponding to that compensation timestamp is replaced with the target media frame, and the compensation timestamp is used as the new target timestamp for the replaced target media frame. This restores the placeholder content originally filled by duplicate frames to the true original valid frame. Simultaneously, the successfully matched compensation timestamp is assigned to the target media frame as its new target timestamp, ensuring that the frame maintains consistent timing with preceding and following frames on the timeline. Conversely, if the target timestamp fails to match any compensation timestamps, the target timestamp is corrected.

[0077] Understandably, when media frame timestamps are out of order, a frame that should be later on the world timeline may be processed earlier due to being assigned a smaller PTS value. After compensating for this preceding frame, subsequent processed media frames may have a world timeline position earlier than the preceding frame. However, this earlier time position has already been occupied by the compensation frame generated by the aforementioned compensation process, leading to a conflict in frame display positions and necessitating an automatic error correction mechanism. Therefore, this embodiment achieves automatic error correction by matching all written compensation timestamps with the target timestamp.

[0078] In other embodiments, after the aforementioned matching and replacement are completed, the method may further include replacing the compensation frame between the target media frame and the next valid media frame (i.e., the compensation frame located after the target media frame within the continuous compensation segment where the target media frame is located) with a copy of the target media frame.

[0079] For example, please see Figure 4 If the target timestamp that fails the verification is successfully matched with PTS1+2 in all the compensation timestamps (PTS1+1, PTS1+2, PTS1+3, PTS5+1) written in the media stream (PTS7 is the timestamp of the previous valid media frame), then the media frame of PTS1+2 will be replaced with the target media frame corresponding to the target timestamp. At the same time, the compensation frame (PTS1+3) will be replaced with a copy of the target media frame.

[0080] In other implementations, when a pause is triggered during video recording, the timestamp of the last media frame is recorded to obtain the pause timestamp as the reference frame timestamp; a reference time increment is superimposed on the reference frame timestamp to obtain the pause reference time point; during the pause of video recording, the actual pause duration is accumulated based on the pause reference time point; for all subsequent media frames captured after video recording resumes, the original timestamp of the subsequent media frames is subtracted from the actual pause duration to obtain the actual timestamp of the subsequent media frames.

[0081] Understandably, existing methods that directly reset the entire video timeline after an anomaly occurs during video recording can lead to a break in the video timeline, resulting in extremely poor compatibility when playing segmented videos or splicing them together. This embodiment, however, records the last frame's timestamp (PTS) as the recovery benchmark and cumulatively deducts the pause duration, ensuring that the timestamps of the resumed frames remain continuous with the frame sequence before the pause, with uniform frame intervals. This allows the player to smoothly decode and render, achieving a seamless recording experience with synchronized audio and video.

[0082] For example, please see Figure 5 When recording reaches PTS2, a pause is triggered, and this frame is set as the reference frame. The pause period is a window period, and the pause duration is recorded cumulatively. After recording resumes, the timing sequence of subsequent PTS3 and PTS4 frames is generated based on the reference frame PTS to ensure the continuity and correctness of the video timeline.

[0083] It should be noted that mainstream action cameras, handheld gimbal cameras, and vehicle-mounted live streaming terminals were used as test devices. High-load encoding, weak network conditions, frequent pause / resume, and high / low frame rate switching were used as test conditions. After testing and verification, the frame interpolation reliability of this embodiment reached 99.8%, and there were no obvious gaps in the picture under 1000 random frame drops. The pause / resume mechanism is accurate. Even with more than 20 frequent pauses in a single recording, the timestamp did not jump, and the audio-visual synchronization error was less than 10ms. The performance is excellent. When recording at high load of 60fps / 120fps, the CPU usage increase is less than 3%, with no additional heat generation or stuttering. It has wide compatibility. The generated video files can be parsed normally in mainstream mobile and PC players and editing software.

[0084] The media stream reassembly method proposed in this embodiment obtains the target media frame and its corresponding target timestamp, and the previous valid media frame and its corresponding valid timestamp; it verifies the validity of the target timestamp based on the valid timestamp; if the target timestamp passes the validity verification, it performs frame loss detection based on the valid timestamp and the target timestamp; if frame loss exists, it generates one or more compensation media frames and their corresponding compensation timestamps based on the previous valid media frame, the target media frame, and the reference time increment, and writes the compensation media frames into the media stream; for each target media frame in the media stream, the above steps are repeated to complete the media stream reassembly. In this way, through timestamp validity verification and dynamic compensation mechanisms, the temporal integrity and synchronization accuracy of the media stream are significantly improved; frame loss is effectively identified and filled, avoiding audio-visual asynchrony, stuttering, or jumps; it does not rely on retransmission from the original encoding end, reducing network bandwidth and latency sensitivity, and enhancing terminal playback stability and user experience consistency.

[0085] Example 2 Furthermore, this disclosure provides a media stream reconstruction system 600, please refer to [link to relevant documentation]. Figure 6 The system includes: The acquisition module 601 is used to acquire the target media frame and its corresponding target timestamp, and to acquire the previous valid media frame and its corresponding valid timestamp. Verification module 602 is used to verify the validity of the target timestamp based on the valid timestamp; Detection module 603 is used for frame loss detection based on valid timestamp and target timestamp; The generation module 604 is used to generate one or more compensation media frames and corresponding compensation timestamps based on the previous valid media frame, the target media frame and the reference time increment if frame loss occurs, and write the compensation media frames into the media stream.

[0086] In an optional implementation, the verification module 602 is further configured to determine whether the target timestamp is less than or equal to the valid timestamp; if the target timestamp is less than or equal to the valid timestamp, the target timestamp is determined to have failed the validity verification; otherwise, the target timestamp is determined to have passed the validity verification.

[0087] In an optional implementation, the system further includes: The correction module is used to add a reference time increment to the valid timestamp if the target timestamp fails the validity verification, so as to obtain a new target timestamp for the target media frame.

[0088] In an optional implementation, the system further includes: The matching module is used to match the target timestamp with the compensation timestamps of all compensation media frames written in the media stream if the target timestamp fails the validity verification; if the match is successful, the compensation media frame corresponding to the successfully matched compensation timestamp is replaced with the target media frame, and the successfully matched compensation timestamp is used as the new target timestamp of the target media frame.

[0089] In an optional implementation, the detection module 603 is further configured to determine that there is frame loss if the difference between the target timestamp and the valid timestamp is greater than the maximum allowable error threshold, wherein the maximum allowable error threshold is a preset multiple of the reference time increment.

[0090] In an optional implementation, the generation module 604 is further configured to: use the previous valid media frame as a reference media frame; copy the reference media frame as a compensation media frame, and superimpose a reference time increment on the timestamp corresponding to the reference media frame as a compensation timestamp; write the compensation media frame into the media stream, and update the compensation media frame to the reference media frame; if the difference between the target timestamp and the compensation timestamp is greater than the maximum allowable error threshold, then return to the step of copying the reference media frame as a compensation media frame, until the difference between the target timestamp and the compensation timestamp is less than or equal to the maximum allowable error threshold.

[0091] In an optional implementation, the generation module 604 is further configured to determine, after generating each compensation media frame, whether the number of compensation media frames generated has reached the preset maximum number of single compensation frames; if so, terminate the step of copying the reference media frame as the compensation media frame in advance.

[0092] The system provided in this embodiment can execute the steps of the media stream reconstruction method provided in Embodiment 1. To avoid repetition, the steps will not be repeated.

[0093] The media stream reassembly system proposed in this embodiment acquires the target media frame and its corresponding target timestamp, as well as the previous valid media frame and its corresponding valid timestamp. It then verifies the validity of the target timestamp based on the valid timestamp. If the target timestamp passes the validity verification, it performs frame loss detection based on the valid timestamp and the target timestamp. If frame loss exists, it generates one or more compensation media frames and their corresponding compensation timestamps based on the previous valid media frame, the target media frame, and the reference time increment, and writes the compensation media frames into the media stream. For each target media frame in the media stream, the above steps are repeated to complete the media stream reassembly. In this way, through timestamp validity verification and dynamic compensation mechanisms, the system significantly improves the temporal integrity and synchronization accuracy of the media stream; effectively identifies and fills in lost frames, avoiding audio-visual asynchrony, stuttering, or jumps; and eliminates the need for retransmission from the original encoding end, reducing network bandwidth and latency sensitivity, and enhancing terminal playback stability and user experience consistency.

[0094] Example 3 Furthermore, embodiments of this disclosure provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the media stream reconstruction method described in Embodiment 1.

[0095] It should be noted that electronic devices can include action cameras, PTZ cameras, live streaming terminals, security cameras, etc.

[0096] For action cameras, in high-speed recording scenarios such as outdoor cycling, diving, and extreme sports, the solution has high frame rate (60fps / 120fps) adaptability, which can effectively resist instantaneous hardware fluctuations and ensure smooth footage without frame drops.

[0097] For gimbal cameras, in scenarios such as aerial photography, portrait tracking, and still life time-lapse recording, it is common to frequently switch shooting angles or pause to compose the shot. The pause-resume mechanism can ensure that the timestamp does not change after multiple pauses, resulting in excellent audio-visual synchronization.

[0098] For live streaming terminals, network jitter can easily lead to data packet loss in scenarios such as outdoor live streaming, vehicle-mounted live streaming, and mobile streaming. The built-in automatic frame interpolation mechanism can dynamically generate compensation frames when frame loss occurs, avoiding gaps or stuttering in the live stream and improving the viewing experience for viewers.

[0099] For security cameras, in long-term monitoring and recording scenarios such as parks, vehicles, and homes, the equipment needs to operate stably 24 / 7. The frame loss detection and completion capabilities can ensure that critical images are not lost, while the CPU usage increase is less than 3% and there is no additional heat generation, ensuring the reliability and stability of long-term recording.

[0100] The device provided in this embodiment can perform the steps of the media stream reconstruction method provided in Embodiment 1. To avoid repetition, the steps will not be repeated.

[0101] Example 4 This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the media stream reconstruction method described in Embodiment 1.

[0102] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0103] The computer-readable storage medium provided in this embodiment can implement the media stream reconstruction method provided in Embodiment 1. To avoid repetition, it will not be described again here.

[0104] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0105] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0106] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A media stream reassembly method, characterized in that, The method includes: Get the target media frame and its corresponding target timestamp, and get the previous valid media frame and its corresponding valid timestamp; The validity of the target timestamp is verified based on the valid timestamp. If the target timestamp passes the validity verification, then frame loss detection is performed based on the valid timestamp and the target timestamp; If there are lost frames, one or more compensation media frames and corresponding compensation timestamps are generated based on the previous valid media frame, the target media frame and the reference time increment, and the compensation media frames are written into the media stream. For each target media frame in the original media stream, repeat the above steps to complete the media stream reconstruction.

2. The media stream reconstruction method according to claim 1, characterized in that, The step of validating the target timestamp based on the valid timestamp includes: Determine whether the target timestamp is less than or equal to the valid timestamp; If the target timestamp is less than or equal to the valid timestamp, the target timestamp is determined to have failed the validity verification; otherwise, the target timestamp is determined to have passed the validity verification.

3. The media stream reconstruction method according to claim 2, characterized in that, The method further includes: If the target timestamp fails the validity verification, the valid timestamp is superimposed with the reference time increment to obtain a new target timestamp for the target media frame.

4. The media stream reconstruction method according to claim 2, characterized in that, The method further includes: If the target timestamp fails the validity verification, the target timestamp is matched with the compensation timestamps of all compensation media frames written in the media stream. If a match is successful, the compensation media frame corresponding to the successfully matched compensation timestamp is replaced with the target media frame, and the successfully matched compensation timestamp is used as the new target timestamp of the target media frame.

5. The media stream reconstruction method according to claim 1, characterized in that, The frame loss detection based on the valid timestamp and the target timestamp includes: If the difference between the target timestamp and the valid timestamp is greater than the maximum allowable error threshold, then it is determined that there is frame loss, wherein the maximum allowable error threshold is a preset multiple of the reference time increment.

6. The media stream reconstruction method according to claim 5, characterized in that, The step of generating one or more compensated media frames and corresponding compensated timestamps based on the previous valid media frame, the target media frame, and the reference time increment includes: Use the previous valid media frame as the reference media frame; The reference media frame is copied as the compensation media frame, and the timestamp corresponding to the reference media frame is superimposed with the reference time increment as the compensation timestamp. Write the compensation media frame into the media stream, and update the compensation media frame to the reference media frame; If the difference between the target timestamp and the compensation timestamp is greater than the maximum allowable error threshold, then return to the step of copying the reference media frame as the compensation media frame until the difference between the target timestamp and the compensation timestamp is less than or equal to the maximum allowable error threshold.

7. The media stream reconstruction method according to claim 6, characterized in that, The method further includes: After generating each of the compensation media frames, it is determined whether the number of compensation media frames generated has reached the preset maximum number of compensation frames per time; if so, the step of copying the reference media frame as the compensation media frame is terminated in advance.

8. A media stream reconfiguration system, characterized in that, The system includes: The acquisition module is used to acquire the target media frame and its corresponding target timestamp, and to acquire the previous valid media frame and its corresponding valid timestamp. The verification module is used to verify the validity of the target timestamp based on the valid timestamp; The detection module is used to perform frame loss detection based on the valid timestamp and the target timestamp; The generation module is used to generate one or more compensation media frames and corresponding compensation timestamps based on the previous valid media frame, the target media frame and the reference time increment if frame loss occurs, and write the compensation media frames into the media stream.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the media stream reconstruction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the media stream reconstruction method as described in any one of claims 1 to 7.

Images