Audio and video track duration alignment method and device based on write budget

By dynamically controlling the amount of AAC audio frames written and the number of frames attached in an embedded recording device, the problem of audio tracks being longer than video tracks in embedded multi-channel video recording is solved, achieving accurate matching of audio and video track durations and playback. This method is suitable for vehicle dashcams and security monitoring recording equipment.

CN123075142APending Publication Date: 2026-11-1070MAI CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611297484.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-25
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In embedded multi-channel video recording, when the video encoding channel and the audio encoding channel run independently, fluctuations in the write speed of the storage medium increase the actual frame interval of the video encoding, while the AAC encoding still produces a constant output at the sampling rate. This results in the audio track having a longer playable duration than the video track within the same container file, leading to problems such as abnormal player progress bars, audio-visual asynchrony, and misalignment during evidence playback.

Method used

Before the container file is encapsulated and written, the number of encapsulated and written video encoded frames N, the number of AAC audio frames written to the container A, and the upper limit of AAC audio frame writing B are maintained. During the frame-by-frame encapsulation process, the writing of AAC audio frames is dynamically controlled. By calculating B = N × R_aac / (S_aac × fps), the amount of AAC audio frames written is limited to ensure that the duration of the video track and the audio track match, and the number of AAC audio frames attached is limited when the timestamp is abnormal.

Benefits of technology

It achieves audio and video track length alignment within a container file without relying on post-processing corrections by the player, improving playback compatibility, progress bar display accuracy, and the reliability of evidence playback, while reducing audio and video track length deviations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123075142A_ABST
    Figure CN123075142A_ABST
Patent Text Reader

Abstract

The application aims to provide a method and device for audio and video track duration alignment based on a writing budget, which not only realizes audio and video track duration alignment in a container file in an embedded real-time segment recording scenario, but also improves playing compatibility, accuracy of progress bar display, and reliability of evidence playback without relying on subsequent correction by a player and without performing full file remapping on a container timestamp.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of embedded audio and video real-time encoding and container encapsulation technology, and in particular to an audio and video track duration alignment method and device based on write budget. Background Technology

[0002] In existing technologies, within the field of embedded multi-channel video recording, H.264 / H.265 video encoded streams are typically mixed with AAC audio encoded streams and encapsulated into container files such as MP4 and MOV, widely used in vehicle dashcams, security monitoring recording equipment, etc. Existing technologies for audio-video synchronization processing mainly include the following solutions: PTS / DTS timestamp remapping scheme: During transcoding or post-processing, the timestamps of audio and video packets are reallocated according to the timeline to ensure audio and video alignment during playback. This scheme requires buffering or writing back the container index information for the entire media segment. It is suitable for non-real-time transcoding and editing scenarios, but not for real-time recording on embedded devices with 30-second segments and loop writing.

[0003] Player-side truncation or padding solution: This involves discarding excess audio at the end of the video stream or adding silence when video is missing, ensuring audio-visual consistency during playback. However, this solution does not modify the container file itself, and the playable duration of the audio and video tracks within the file may still be inconsistent, failing to meet the file consistency requirements for evidence playback, forensic identification, and third-party transcoding.

[0004] Fixed-ratio mixing scheme: Audio frames are attached to video frames at a fixed ratio, for example, at 30fps, every 2 video frames are attached to 1 AAC frame. This scheme is effective when the video encoder outputs frames stably at the nominal frame rate; however, when storage write backpressure, GOP expansion, or link recovery causes the actual video frame rate to be lower than the nominal frame rate, the consistency of the duration of the two tracks within the container cannot be guaranteed.

[0005] Timestamp-driven audio retrieval scheme: During encapsulation, the nearest neighbor AAC frame is matched from the audio buffer based on the video frame timestamp and written to the container. This scheme is usable when the audio and video encoding rates are basically the same; however, when the audio encoder continuously outputs at wall-clock intervals while the video encoder actually slows down due to write card backpressure, excessive AAC will still be written, resulting in the audio track being longer than the video track.

[0006] Cache level management and dynamic bitrate reduction solutions: These solutions monitor the remaining data in the cache or write latency, and dynamically adjust the encoding bitrate or write parameters to avoid cache overflow and data loss. These solutions address issues like "inability to write" or "frame dropping," but do not address the problem of "excessive audio writing causing inconsistent audio and video track durations within the container."

[0007] The aforementioned embedded recording devices typically have the following characteristics: the video encoding channel and the audio encoding channel operate independently without hardware phase-locked loop; multiple video recordings can share the same audio encoding output stream; fluctuations in the write speed of the storage medium can lead to an increase in the actual frame interval of video encoding, while AAC encoding still produces a constant output at the sampling rate.

[0008] Therefore, when the writing speed of storage media such as SD cards slows down, the encoding frame queue is pushed back, or the actual length of the GOP increases, a common phenomenon is that the number of video frames in the same container file increases according to the actual pack progress, while AAC continues to write according to the wall-clock, resulting in the audio track having a longer playable duration than the video track, causing abnormal progress bars in the player, audio and video desynchronization, and misalignment in evidence playback. Summary of the Invention

[0009] One objective of this application is to provide a method and device for aligning audio and video track durations based on write budget. Without relying on post-processing corrections by the player or performing full file remapping of container timestamps, this method not only achieves audio and video track duration alignment within container files in embedded real-time segmented recording scenarios, but also improves playback compatibility, progress bar display accuracy, and the reliability of evidence playback.

[0010] According to one aspect of this application, a method for aligning audio and video track durations based on a write budget is provided, characterized in that the method is applied to an embedded recording device in which video encoding channels and audio encoding channels operate independently, wherein the method includes: Before writing the container file, maintain the number of video encoded frames N that have been written, the number of AAC audio frames A that have been written to the container, and the upper limit of writing AAC audio frames B. During the frame-by-frame encapsulation process, the following operation is repeated for each video encoded frame to be written until the container file is closed, so that the playable duration of the video track in the container file matches the playable duration of the AAC audio track: When writing the current video encoding frame, N is incremented by 1, and the upper limit B of the AAC audio frame that is currently allowed to be written is dynamically determined based on the real-time N and the nominal frame rate fps of the current video recording channel. If A is greater than or equal to B, then the current video encoded frame is directly written to the container file, and the AAC audio frame is not obtained from the AAC audio cache. If A is less than B, then retrieve audio data not exceeding B from the AAC audio buffer. A frame (AAC audio frame) is attached to the current video encoding frame. The current video encoding frame and the attached AAC audio frame are mixed and encapsulated and written into the container file. The A frame is updated according to the actual number of AAC audio frames written into the container.

[0011] Furthermore, in the above method, the formula for calculating the upper limit B of the AAC audio frame write is: B = N×R_aac / (S_aac×fps) ; Where R_aac is the AAC sampling rate, and S_aac is the number of sampling points corresponding to each AAC audio frame. This indicates rounding up to the nearest integer.

[0012] Furthermore, in the above method, the integer arithmetic formula for the upper limit B of the AAC audio frame write is: B = (N × R_aac + S_aac × fps) 1) / (S_aac×fps).

[0013] Furthermore, in the above method, if A is less than B, then no more than B is obtained from the AAC audio buffer. Before the A-frame AAC audio frame, the method further includes: Obtain the timestamp curr_ts of the current video encoded frame and the timestamp prev_ts of the previous video encoded frame, and calculate the time interval curr_ts. prev_ts; If curr_ts If prev_ts is greater than or equal to (1000 / fps)×K, it is determined to be an abnormal timestamp jump, where K is a preset positive integer; When the timestamp jumps abnormally, the number of AAC audio frames attached to the current video encoded frame is limited to a maximum of one frame.

[0014] Furthermore, the above method further includes: When no abnormal timestamp jump occurs, the number of AAC audio frames attached to the current video encoded frame is limited to a maximum of M, where M is an integer greater than 1. The number of AAC audio frames actually acquired in the current video encoding frame is min(M, B). A).

[0015] Furthermore, in the above method, when closing the container file, the method further includes: Record the time span T between the timestamp of the first and last video encoded frames in the container file; The theoretical output number E of AAC audio frames is estimated based on the time span T. Max(0, E) A) Determine the number D of AAC audio frames that have not been written to the container file, which is used to diagnose storage write performance and encoding anomalies. The formula for calculating the theoretical output number E of the AAC audio frames is as follows: E = T×R_aac / (S_aac×1000) .

[0016] Furthermore, in the above method, the embedded recording device includes multiple video recording channels, and the multiple video recording channels share the same AAC encoded stream and a corresponding AAC audio buffer; Each video recording channel independently maintains its corresponding N, A, and B; When each video recording channel retrieves an AAC audio frame from the shared AAC audio buffer, the retrieved AAC audio frame is marked as occupied by the channel. The same AAC audio frame is not written repeatedly by more than two video recording channels.

[0017] Furthermore, the above method further includes: When a new container file is successfully created, N and A corresponding to the new container file are reset to zero, and the timestamps of the first and last video encoded frames corresponding to the new container file are set to zero.

[0018] According to another aspect of this application, a non-volatile storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a processor, cause the processor to implement the audio / video track duration alignment method based on write budget as described above.

[0019] According to another aspect of this application, an audio / video track duration alignment device based on write budget is also provided, wherein the device includes: One or more processors; Computer-readable medium for storing one or more computer-readable instructions. When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement the audio / video track duration alignment method based on write budget as described above.

[0020] Compared with existing technologies, this application is applied to embedded recording devices where video encoding channels and audio encoding channels operate independently, including: before containerizing and writing a container file, maintaining the number N of encapsulated and written video encoding frames, the number A of AAC audio frames written to the container, and the upper limit B of AAC audio frames to be written; during the frame-by-frame encapsulation process, the following operations are repeatedly performed for each video encoding frame to be written until the container file is closed, so that the playable duration of the video track in the container file matches the playable duration of the AAC audio track: when writing the current video encoding frame, N is incremented by 1, and the upper limit B of the currently allowed AAC audio frames to be written is dynamically determined based on the real-time N and the nominal frame rate fps of the current video recording channel; if A is greater than or equal to B, the current video encoding frame is directly written to the container file without obtaining AAC audio frames from the AAC audio buffer; if A is less than B, no more than B is obtained from the AAC audio buffer. A frame (AAC audio frame) is attached to the current video encoded frame. The current video encoded frame and the attached AAC audio frame are then mixed and encapsulated into the container file, and A is updated according to the actual number of AAC audio frames written into the container. In this application, without relying on post-processing corrections by the player or performing full file remapping of the container timestamp, not only is the duration alignment of audio and video tracks within the container file achieved in embedded real-time segmented recording scenarios, but playback compatibility, progress bar display accuracy, and reliability of evidence playback are also improved. Attached Figure Description

[0021] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating an audio / video track duration alignment method based on a write budget according to one aspect of this application is shown. Figure 2 A flowchart illustrating AAC frame-by-frame encapsulation control is shown in an audio / video track duration alignment method based on write budget according to one aspect of this application. Figure 3 The diagram illustrates the AAC discard estimation flowchart when a container file is closed in an audio / video track duration alignment method based on write budget according to one aspect of this application. Figure 4 This diagram illustrates the independent budget status and occupancy flags in a multi-channel recording sharing AAC encoded stream during an audio / video track duration alignment method based on a write budget, according to one aspect of this application. Figure 5 This diagram illustrates the processing of audio and video track durations in the prior art when budget control is not employed. Figure 6This diagram illustrates the alignment process of audio and video track durations using budget control in a method for aligning audio and video track durations based on a write budget according to one aspect of this application. Figure 7 The diagram shows the overall system architecture of an audio / video track duration alignment method based on a write budget according to one aspect of this application. Detailed Implementation

[0022] The present application will now be described in further detail with reference to the accompanying drawings.

[0023] In a typical configuration of this application, the terminal, the device of the service network, and the trusted party all include one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0024] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0025] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0026] Existing technologies have not yet provided a systematic AAC write total control scheme on the container encapsulation write side to address the "multiple audio writes under dual encoder independent clock domains" problem. In embedded multi-channel recording scenarios where video and audio encoding channels operate independently and multiple recordings share the same audio encoding output stream, when storage write backpressure, GOP expansion, or link recovery causes the actual output frame rate of the video encoder to be lower than the nominal frame rate, while the AAC audio encoder continues to output at wall-clock, existing solutions cannot effectively guarantee consistent playback duration for audio and video tracks within the same container file. This application aims to: On the container encapsulation writing side, the upper limit of AAC audio frames to be written is dynamically determined based on the number of video frames already packed. Once the upper limit is reached, the fetching of frames from the AAC audio buffer is stopped to prevent the audio track from being longer than the video track. When the video timestamp jumps abnormally, the number of AAC audio frames that can be attached to a single video frame is limited to prevent the AAC accumulated during the jump interval from being over-compressed and attached, which would aggravate the track length deviation. When the container file is closed, the number of AAC audio frames not written to the container is estimated based on the timestamp span of the first and last video frames actually written, which helps to evaluate storage performance and encoding anomalies. For each recording channel, an independent write budget state is maintained, ensuring that the duration of each container file's audio and video tracks is consistent while sharing the same audio encoding output stream.

[0027] To achieve the above technical objectives, such as Figure 1 As shown, Figure 1 This document presents a flowchart illustrating a method for aligning audio and video track durations based on a write budget, as proposed in one aspect of this application. This method is applied to embedded recording devices where video and audio encoding channels operate independently. Specifically, it addresses the issue of inconsistent audio and video track durations caused by fluctuations in the video encoding frame rate when real-time mixing and encapsulating H.264 / H.265 video streams and AAC audio streams into MP4 or other container files in an embedded recording device where video and audio encoding channels operate independently and multiple recordings share the same audio encoding output stream. This method can be applied to recording scenarios in embedded recording devices such as vehicle dashcams and security monitoring recording equipment. The method includes steps S1, S2, S3, S4, and S5, specifically comprising the following steps: Step S1: Before writing the container file, maintain the number of video encoded frames N that have been written, the number of AAC audio frames A that have been written to the container, and the upper limit of writing AAC audio frames B. Step S2: During the frame-by-frame encapsulation process, the following operation is repeated for each video encoded frame to be written until the container file is closed, so that the playable duration of the video track in the container file matches the playable duration of the AAC audio track: Step S3: When writing the current video encoding frame, increment N by 1, and dynamically determine the upper limit B of the AAC audio frames that are currently allowed to be written based on the real-time N and the nominal frame rate fps of the current video recording channel. Step S4: If A is greater than or equal to B, then the current video encoded frame is directly written to the container file without obtaining the AAC audio frame from the AAC audio cache. Step S5: If A is less than B, then obtain an audio buffer containing no more than B from the AAC audio buffer. A frame (AAC audio frame) is attached to the current video encoding frame. The current video encoding frame and the attached AAC audio frame are mixed and encapsulated and written into the container file. The A frame is updated according to the actual number of AAC audio frames written into the container.

[0028] Through the above steps S1 to S5, this application not only achieves the alignment of audio and video track durations within the container file in embedded real-time segmented recording scenarios without relying on post-processing corrections by the player or performing full file remapping of the container timestamp, but also improves playback compatibility, progress bar display accuracy, and the reliability of evidence playback.

[0029] For example, during the real-time encapsulation and writing of container files, the audio and video timestamps are not modified, and no truncation occurs at the playback end. Instead, before container encapsulation / mux writing, the number N of video encoded frames already packed and written and the number AAC audio frames already written to the container file are maintained. The upper limit B of the currently allowed AAC audio frames to be written is dynamically determined based on N and the nominal frame rate (fps) of the current video recording channel. When A is greater than or equal to B, audio data retrieval from the AAC audio buffer is stopped; when A is less than B, only data up to (B) of the AAC audio buffer is retrieved. A) The AAC audio frame is attached to the current video encoding frame to actively limit the total amount of AAC audio frames written when the video frame output slows down, ensuring that the audio and video tracks in the container file have the same playable duration.

[0030] Continuing with the above embodiments of this application, the formula for calculating the upper limit B of the AAC audio frame write is as follows: B = N×R_aac / (S_aac×fps) ; Where R_aac is the AAC sampling rate, and S_aac is the number of sampling points corresponding to each AAC audio frame. This indicates rounding up to the nearest integer.

[0031] In this embodiment, for each video encoded frame to be written, for the Nth video encoded frame that has already been packed, the upper limit B of the AAC audio frames currently allowed to be written to the container can preferably be determined according to the following formula: B = N × R_aac / (S_aac × fps)

[0032] Wherein, R_aac is the AAC sampling rate, and in a preferred embodiment of this application, R_aac = 16000Hz is preferred; S_aac is the number of sampling points corresponding to each AAC audio frame, and in a preferred embodiment of this application, S_aac = 1024 is preferred; fps is the nominal frame rate of the current video encoding channel of the video encoding frame to be written, and in a preferred embodiment of this application, it is preferred to be 15fps or 30fps; This indicates rounding up. It should be noted that the preferred embodiment in this example is merely illustrative; other values ​​for R_aac, S_aac, and the nominal frame rate (fps) are all included within the scope of this application.

[0033] Furthermore, the integer arithmetic formula for the upper limit B of the AAC audio frame write is as follows: B = (N × R_aac + S_aac × fps) 1) / (S_aac×fps).

[0034] The theoretical duration of the video corresponding to the N video encoded frames at the nominal frame rate (fps) is N / fps seconds. The number of AAC theoretically produced frames within this theoretical duration is rounded up to B, thus ensuring that the total number of AAC audio frames allowed to be written matches the theoretical video duration corresponding to the already packed video encoded frames.

[0035] Example using a preferred embodiment: When recording at full speed for 30s@15fps, N=450, then B= 450×16000 / (1024×15) =469; When recording at full speed (30s@30fps), N=900, then B= 900×16000 / (1024×30) =469. The above preferred values ​​are merely illustrative examples; other possible values ​​are included within the scope of protection of this application.

[0036] For each video encoded frame to be written, N is incremented by 1, and the upper limit B of AAC audio frame to be written is dynamically determined based on real-time N and fps. B increases or remains unchanged as N increases.

[0037] In a preferred embodiment of the present application, during the process of frame-by-frame encapsulation, the following steps are performed for the current video encoded frame to be written: Increment N by 1, and determine B according to N and fps; If A≥B, audio data will not be acquired from the AAC audio buffer for the current video encoded frame, and the current video encoded frame is directly written into the container; If A<B, acquire at most (B A) AAC audio frames from the AAC audio buffer and attach them to the current video encoded frame, then increment A, where the incremented A does not exceed B; if the actually available quantity is less than (B A), the actual acquired quantity shall prevail; Repeat the above process until the current container file is closed.

[0038] Continuing with the above embodiment of the present application, before acquiring no more than B from the AAC audio buffer when A is less than B A AAC audio frames, the method further comprises: Acquire the timestamp curr_ts of the current video encoded frame and the timestamp prev_ts of the previous video encoded frame, and calculate the time interval curr_ts prev_ts; If curr_ts prev_ts is greater than or equal to (1000 / fps)×K, it is determined that an abnormal timestamp jump occurs, where K is a preset positive integer; When the abnormal timestamp jump occurs, the number of AAC audio frames attached to the current video encoded frame is limited to a maximum of one frame.

[0039] In a preferred embodiment of the present application, as Figure 2 shown, before writing each video encoded frame into the container, the following pack process (corresponding to pack_audio_for_frame) is executed: Step S101: Determine the current recording type; if it is a no-AAC mode such as time-lapse photography, skip audio attachment; Step S102: N←N+1, calculate B according to the real-time N and fps; Step S103: If A≥B, set audio_pack_count of the current video encoded frame to 0, no longer call the AAC acquisition interface, and directly write into the container; Step S104: If A<B, determine whether the timestamp interval between adjacent video frames is abnormal. If (curr_ts If prev_ts)≥(1000 / fps)×K, it is determined to be an abnormal timestamp jump. In a preferred embodiment of this application, K=3, at which time the upper limit of the number of AAC audio frames in a single frame (i.e., the upper limit of single frame splicing) is 1; otherwise, the upper limit of the number of AAC audio frames in a single frame is M, and in this embodiment M=2. Step S105: Calculate the actual number of AAC audio frames that can be taken this time: max_take = min(maximum number of frames that can be attached per frame, B) A); Step S106: Repeatedly call get_aac to retrieve up to max_take AAC audio frames from the AAC audio buffer and attach them to the current video encoding frame. Each time an A←A+1 is attached; Step S107: Update prev_video_ts and proceed to the processing of the next video encoding frame.

[0040] The basis for M=2 is as follows: the duration of a single AAC audio frame is approximately 64ms (1024 / 16000), the duration of a 15fps video frame is approximately 66.7ms, and the duration of a 30fps video frame is approximately 33.3ms; the total duration of two AAC audio frames is 128ms, which covers approximately the cycle of two 15fps video frames. This is the minimum integer upper limit for satisfying the AAC / video sampling rate ratio under budget constraints. Of course, the values ​​used in this preferred embodiment are merely illustrative examples, and other values ​​of M can be included within the protection scope of this application.

[0041] In a preferred embodiment of this application, when the SD card write speed slows down, if the embedded recording device records normally at 15fps, AAC audio frames continuously enter the buffer at approximately 64ms / frame. When the SD card write speed decreases, the video encoding frame queue becomes full and frames are dropped according to GOP alignment, the video pull interval increases, and the actual pack frame rate is lower than 15fps.

[0042] For example, if only 380 video encoded frames are packed within a 30-second wall-clock, then B = 380×16000 / (1024×15) =396. Without budget control, it is still possible to write 469 AAC audio frames, with an audio track length of approximately 30s and a video track length of approximately 25.3s, resulting in a deviation of approximately 4.7s. After adopting the alignment method of this application, AAC audio frames are stopped being fetched when A reaches 396, and subsequent video encoded frames have no audio appended. The two tracks within the container file have the same playable duration, with a deviation of less than 50ms.

[0043] In another preferred embodiment of this application, when a timestamp abrupt change occurs, after a write card backpressure or frame loss in the queue, the time interval between the timestamps of adjacent video encoded frames may reach more than 200ms (greater than 3×66.7ms at 15fps). If two AAC audio frames are still allowed to be attached to a single frame at this time, multiple AAC audio frames buffered during the abrupt change may be compressed and attached to the same video frame, exacerbating the track length deviation. This preferred embodiment forces max_take=1 when the time interval gap is abnormal; if A≥B at the same time, AAC acquisition is still skipped. This strategy shares the same timestamp abrupt change threshold as video stream health monitoring, and is semantically consistent.

[0044] Continuing with the above embodiments of this application, the method further includes: When no abnormal timestamp jump occurs, the number of AAC audio frames attached to the current video encoded frame is limited to a maximum of M, where M is an integer greater than 1. The number of AAC audio frames actually acquired in the current video encoding frame is min(M, B). A).

[0045] In the embodiments of this application, during the processing of abnormal timestamp transitions, when the time interval between the timestamps of two adjacent video encoded frames is greater than or equal to (1000 / fps) × K, it is determined to be an abnormal timestamp transition, where K is a preset positive integer, preferably K=3. During an abnormal timestamp transition, the number of AAC audio frames attached to a single video encoded frame is limited to 1 to avoid excessive compression of accumulated AAC within the transition interval and its attachment to the same video frame.

[0046] In the absence of a non-timestamp aberration (i.e., when the aforementioned timestamp aberration does not occur), the maximum number of AAC audio frames that can be attached to a single video encoded frame is M, where M is an integer greater than 1, preferably M=2, and the actual number of attachments is min(M, B). A).

[0047] Continuing with the above embodiments of this application, when closing the container file, the method further includes: Record the time span T between the timestamp of the first and last video encoded frames in the container file; The theoretical output number E of AAC audio frames is estimated based on the time span T. Max(0, E) A) Determine the number D of AAC audio frames that have not been written to the container file, which is used to diagnose storage write performance and encoding anomalies. The formula for calculating the theoretical output number E of the AAC audio frames is as follows: E = T×R_aac / (S_aac×1000) .

[0048] In the preferred embodiment of this application, such as Figure 3 As shown, when performing discard estimation after closing the container file, the time span T between the timestamp of the first and last video encoded frames in the container file is recorded. The theoretical number of AAC audio frames E produced within this time span T is estimated, and the number of unwritten AAC audio frames D = max(0, E) is calculated. A) Used for diagnosing storage write performance and encoding anomalies. Preferably, the theoretical output number E of the AAC audio frames is calculated using the following formula: E= T × R_aac / (S_aac × 1000) .

[0049] Continuing with the above embodiments of this application, the embedded recording device includes multiple video recording channels, and the multiple video recording channels share the same AAC encoded stream and a corresponding AAC audio buffer; Each video recording channel independently maintains its corresponding N, A, and B; When each video recording channel retrieves an AAC audio frame from the shared AAC audio buffer, the retrieved AAC audio frame is marked as occupied by the channel. The same AAC audio frame is not written repeatedly by more than two video recording channels.

[0050] In the preferred embodiment of this application, such as Figure 4 As shown, for multiple video recording channels, each video recording channel independently maintains its own N, A, and B, while sharing the same AAC encoded stream and AAC audio buffer. When each channel obtains AAC audio frames from the shared AAC encoded stream, it marks the already obtained AAC audio frames as occupied by the channel, ensuring that the same AAC audio frame is not written repeatedly by more than two video recording channels.

[0051] In a specific preferred embodiment, the embedded recording device of this embodiment supports simultaneous encapsulation of multiple MP4 streams, such as pre-recording, post-recording, and sub-streams, while sharing an AAC encoded stream and an AAC circular buffer queue. Each rec_file independently maintains an AvSyncPackStat, which may include, but is not limited to, packed_video_cnt(N), packed_aac_cnt(A), prev_video_ts, file_first_video_ts, file_last_video_ts, etc. B is calculated independently for each video recording channel based on its own fps and N.

[0052] When get_aac retrieves AAC audio frames from the shared queue, it marks the AAC audio frames by channel occupancy using isUsed[rec_file]. The same AAC audio frame is not written repeatedly by more than two video recording channels. Each container file implements track length alignment.

[0053] Continuing with the above embodiments of this application, the method further includes: When a new container file is successfully created, N and A corresponding to the new container file are reset to zero, and the timestamps of the first and last video encoded frames corresponding to the new container file are set to zero.

[0054] In a preferred embodiment of this application, after file_create is successful, file_first_video_ts and file_last_video_ts are set to zero, and budget_skip_video_cnt is set to zero; N and A are reset to 0 when a new container file is created.

[0055] Specifically, after each successful write of a video encoded frame by file_write, the timestamps of the first and last video encoded frames, file_first_video_ts, are updated to perform the initialization operation.

[0056] When closing the container file file (file_close), perform discard estimation: span_ms=(file_last_video_ts file_first_video_ts) / 1000 E= span_ms × R_aac / (S_aac × 1000)

[0057] D=max(0, E A) The log outputs the number of frames (V), the number of AACs written (A), the estimated number of dropped files (D), and the file size. Then, av_sync_pack_reset resets the state of this path.

[0058] In parking pre-recording mode, if the trigger type does not require pre-buffering, the encoding queue is cleared and the av_sync_pack state is reset before creating and opening the container file to avoid using the previous budget count.

[0059] In a preferred embodiment of this application, such as Figure 5As shown, when there is no budget control, the number of packed video frames decreases due to reduced write card backpressure, while AAC audio frames are still written according to wall-clock time, making the audio track longer than the video track; as shown in Figure 6 shown, after adopting the budget of the present application, the write volume of AAC audio frames is dynamically tightened with N, so that the audio and video tracks have the same length. Among them, Table 1 shows the design verification data (15fps, 30s segments, pre-recording).

[0060] Table 1 Comparison table with and without budget control

[0061] In one aspect of the present application, an audio and video track duration alignment system based on write budget is further proposed, as shown in Figure 7 shown. The system is deployed in an embedded recording device, in which the video encoding channel and the audio encoding channel operate independently, and multiple recording paths share the same audio encoding output stream. The embedded recording device includes: A video encoding module, configured to perform H.264 / H.265 encoding on collected images and output video encoded frames; affected by ISP processing, link status and storage write speed, the actual frame output interval may be greater than 1 / fps; An AAC encoding module: configured to continuously encode PCM audio at the wall-clock sampling rate, output AAC audio frames and store them in the AAC circular buffer queue; A container encapsulation module: configured to mix and write the video encoded frames and the attached AAC audio frames into a container file such as MP4; A processor: configured to execute the audio and video track duration alignment method of the present application during the real-time container encapsulation process, maintain the packed video frame count N, the number A of AAC audio frames already written into the container, and the dynamic write upper limit B for each recording path, and complete AAC budget gating before container encapsulation / mux writing.

[0062] Among them, the data flow is: the video encoded frames enter the encapsulation process through the encoded frame queue, and the AAC budget gating logic is called at the packing stage; audio is obtained from the AAC buffer only when A<B; and finally written into the container file. When closing the container file, the discard estimated value is recorded.

[0063] Preferably, the embedded recording device is an on-vehicle multi-channel dashcam, the AAC encoding module continuously outputs AAC-LC frames at a 16kHz sampling rate, and the actual frame output rate of the video encoding module is lower than the nominal fps due to the influence of SD card write backpressure or GOP structure.

[0064] In scenarios where video encoding frame output slows down while AAC still produces at wall-clock speeds, this application can ensure consistent playback duration of audio and video tracks within the container on the encapsulation write side. When backpressure from the write card causes the number of video frames to fall short of the full segment, the deviation in audio and video track length can be reduced from several seconds to milliseconds. At the same time, the number of audio frames not written to AAC can be estimated and quantized through discard, facilitating engineering diagnosis.

[0065] According to another aspect of this application, a non-volatile storage medium is also provided, on which computer-readable instructions are stored, which, when executed by a processor, cause the processor to implement the above-described method for aligning audio and video track durations based on write budget.

[0066] According to another aspect of this application, an audio / video track duration alignment device based on write budget is also provided, wherein the device includes: One or more processors; Computer-readable medium for storing one or more computer-readable instructions. When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors implement the audio / video track duration alignment method based on write budget as described above.

[0067] For details of the various embodiments of the audio / video track duration alignment device based on write budget, please refer to the corresponding parts of the above-described embodiments of the audio / video track duration alignment method based on write budget, which will not be repeated here.

[0068] In summary, this application is applied to an embedded recording device where the video encoding channel and audio encoding channel operate independently, including: before containerizing and writing a container file, maintaining the number N of encapsulated and written video encoding frames, the number A of AAC audio frames written to the container, and the upper limit B of AAC audio frames to be written; during the frame-by-frame encapsulation process, the following operation is repeatedly performed for each video encoding frame to be written until the container file is closed, so that the playable duration of the video track in the container file matches the playable duration of the AAC audio track: when writing the current video encoding frame, N is incremented by 1, and the upper limit B of the currently allowed AAC audio frames to be written is dynamically determined based on the real-time N and the nominal frame rate fps of the current video recording channel; if A is greater than or equal to B, the current video encoding frame is directly written to the container file without obtaining AAC audio frames from the AAC audio cache; if A is less than B, no more than B is obtained from the AAC audio cache. A frame (AAC audio frame) is attached to the current video encoded frame. The current video encoded frame and the attached AAC audio frame are then mixed and encapsulated into the container file, and A is updated according to the actual number of AAC audio frames written into the container. In this application, without relying on post-processing corrections by the player or performing full file remapping of the container timestamp, not only is the duration alignment of audio and video tracks within the container file achieved in embedded real-time segmented recording scenarios, but playback compatibility, progress bar display accuracy, and reliability of evidence playback are also improved.

[0069] Compared with the prior art, this application has the following beneficial effects: (1) Ensure that the duration of audio and video tracks within the container is consistent. In scenarios where the video encoding frame rate is lower than the nominal frame rate, but AAC is still continuously produced according to the wall-clock, the total amount of AAC written is dynamically limited on the encapsulation writing side based on the number of already packed video encoded frames, so that the playable duration of the audio track within the same container file is consistent with that of the video track, thus avoiding abnormal progress bars in the player and audio-visual asynchrony.

[0070] (2) Suitable for embedded real-time segmented recording. It adopts frame-by-frame O(1) decision-making, without the need for full file buffering or write-back container index, making it suitable for embedded scenarios such as vehicle recorders with 30-second segmented and cyclic writing, with minimal impact on system memory and write latency.

[0071] (3) Effectively address write back pressure scenarios. When the writing speed of storage media such as SD cards slows down, resulting in a reduction in the actual number of video pack frames, the upper limit of AAC audio frame writing dynamically tightens with N, which can reduce the audio and video track length deviation from several seconds to milliseconds, thereby improving the reliability of evidence playback and forensic identification.

[0072] (4) Suppress excessive splicing of AAC audio frames caused by timestamp jumps. When the timestamps of adjacent video frames jump abnormally, limit the splicing of a single frame to a maximum of one AAC audio frame to avoid the AAC audio frames accumulated during the jump interval being compressed and spliced ​​to the same video frame, thereby further controlling the track length deviation.

[0073] (5) Supports multi-channel recording without interference. Each video recording channel maintains its own write budget state independently. Under the premise of sharing the same AAC encoded output stream, it ensures that the audio and video track durations of each container file are aligned, and avoids the same AAC audio frame being written repeatedly by using an occupation flag.

[0074] (6) Facilitates engineering diagnosis and performance evaluation. When closing the file, the number of AAC audio frames not written to the container is estimated based on the timestamp span of the first and last video frames. This can quantify the number of AAC audio frames skipped due to budget control, and is used to evaluate storage write performance and encoding anomalies.

[0075] (7) No sacrifice in video quality. Compared with solutions such as reducing video bitrate or pausing audio encoding, this application only limits the amount of AAC audio frames written in the mux layer without changing the video encoding parameters, which is beneficial to maintaining the quality of evidence collection.

[0076] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0077] Furthermore, a portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions invoking the methods of this application may be stored in a fixed or removable recording medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the working memory of a computer device operating according to the program instructions. Here, one embodiment of this application includes an apparatus comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the apparatus is triggered to operate the methods and / or technical solutions based on the foregoing embodiments of this application.

[0078] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

Claims

1. A method for aligning audio and video track durations based on write budget, characterized in that, This method is applied to embedded recording devices where video encoding channels and audio encoding channels operate independently, wherein the method includes: Before writing the container file, maintain the number of video encoded frames N that have been written, the number of AAC audio frames A that have been written to the container, and the upper limit of writing AAC audio frames B. During the frame-by-frame encapsulation process, the following operation is repeated for each video encoded frame to be written until the container file is closed, so that the playable duration of the video track in the container file matches the playable duration of the AAC audio track: When writing the current video encoding frame, N is incremented by 1, and the upper limit B of the AAC audio frame that is currently allowed to be written is dynamically determined based on the real-time N and the nominal frame rate fps of the current video recording channel. If A is greater than or equal to B, then the current video encoded frame is directly written to the container file, and the AAC audio frame is not obtained from the AAC audio cache. If A is less than B, then retrieve audio data not exceeding B from the AAC audio buffer. A frame (AAC audio frame) is attached to the current video encoding frame. The current video encoding frame and the attached AAC audio frame are mixed and encapsulated and written into the container file. The A frame is updated according to the actual number of AAC audio frames written into the container.

2. The method according to claim 1, characterized in that, The formula for calculating the upper limit B of the AAC audio frame write is as follows: B= N×R_aac / (S_aac×fps) ; Where R_aac is the AAC sampling rate, and S_aac is the number of sampling points corresponding to each AAC audio frame. This indicates rounding up to the nearest integer.

3. The method according to claim 2, characterized in that, The integer formula for the upper limit B of the AAC audio frame write is: B=(N×R_aac+S_aac×fps 1) / (S_aac×fps)。 4. The method according to claim 1, characterized in that, If A is less than B, then retrieve audio data not exceeding B from the AAC audio buffer. Before the A-frame AAC audio frame, the method further includes: Obtain the timestamp curr_ts of the current video encoded frame and the timestamp prev_ts of the previous video encoded frame, and calculate the time interval curr_ts. prev_ts; If curr_ts If prev_ts is greater than or equal to (1000 / fps)×K, it is determined to be an abnormal timestamp jump, where K is a preset positive integer; When the timestamp jumps abnormally, the number of AAC audio frames attached to the current video encoded frame is limited to a maximum of one frame.

5. The method according to claim 4, characterized in that, The method further includes: When no abnormal timestamp jump occurs, the number of AAC audio frames attached to the current video encoded frame is limited to a maximum of M, where M is an integer greater than 1. The number of AAC audio frames actually acquired in the current video encoding frame is min(M, B). A).

6. The method according to claim 1, characterized in that, When closing the container file, the method further includes: Record the time span T between the timestamp of the first and last video encoded frames in the container file; The theoretical output number E of AAC audio frames is estimated based on the time span T. Max(0, E) A) Determine the number D of AAC audio frames that have not been written to the container file, which is used to diagnose storage write performance and encoding anomalies. The formula for calculating the theoretical output number E of the AAC audio frames is as follows: E= T×R_aac / (S_aac×1000) 。 7. The method according to claim 1, characterized in that, The embedded recording device includes multiple video recording channels, and the multiple video recording channels share the same AAC encoded stream and a corresponding AAC audio buffer; Each video recording channel independently maintains its corresponding N, A, and B; When each video recording channel retrieves an AAC audio frame from the shared AAC audio buffer, the retrieved AAC audio frame is marked as occupied by the channel. The same AAC audio frame is not written repeatedly by more than two video recording channels.

8. The method according to claim 1, characterized in that, The method further includes: When a new container file is successfully created, N and A corresponding to the new container file are reset to zero, and the timestamps of the first and last video encoded frames corresponding to the new container file are set to zero.

9. A non-volatile storage medium having stored computer-readable instructions thereon, which, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 8.

10. An audio / video track duration alignment device based on write budget, wherein, The device includes: One or more processors; Computer-readable medium for storing one or more computer-readable instructions. When the one or more computer-readable instructions are executed by the one or more processors, the one or more processors perform the method as described in any one of claims 1 to 8.