A method for continuous recording of multi-source audio and video on a unified timeline for glasses

By setting acquisition source identifiers and generation identifiers for multi-source audio and video recordings of smart glasses, identifying candidate breakpoints and establishing time correspondences, the problem of time differences caused by the pause or resumption of acquisition sources is solved, realizing continuous recording of multi-source audio and video data along a unified timeline, and improving the accuracy and continuity of recording.

CN123123526APending Publication Date: 2026-11-17HANGZHOU LAWAKEN TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611619667.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

During the continuous recording of multi-source audio and video in smart glasses, pausing, resuming, or re-initializing the acquisition source causes a difference between the data reception order and the actual acquisition order, affecting the time correspondence and the continuity of the recording file. Furthermore, the delayed output of cached data causes timing disorder.

Method used

By setting acquisition source identifiers and acquisition generation identifiers for each acquisition source, a multi-source acquisition sequence is generated, candidate breakpoints are identified and a two-sided observation interval is established, the cached data before the breakpoint and the recovered data after the breakpoint are distinguished, the time correspondence between the source timestamp and the continuous clock of the glasses device is established, a unified timeline record is generated, and video keyframes or audio data packet boundaries are selected for segmented saving according to the segment position.

Benefits of technology

It improves the accuracy of multi-source time alignment, enhances the temporal consistency of continuous records, improves the adaptability of segmented storage, reduces the impact of out-of-order cached data and interrupted periods, and ensures the continuity and availability of records.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123526A_ABST
    Figure CN123123526A_ABST
Patent Text Reader

Abstract

This invention discloses a method for continuous recording of multi-source audio and video data on a unified timeline for glasses, relating to the field of audio and video data processing. The method includes: acquiring multi-source audio and video data and the operating status of the acquisition sources; setting acquisition source identifiers and acquisition generation identifiers; associating source timestamps with the device's continuous clock to form an acquisition time stamp; identifying candidate breakpoints; distinguishing between cached data before the breakpoint and recovered data after the breakpoint and determining the gap interval; establishing a time correspondence according to the acquisition generation and generating a unified timeline record; and selecting video keyframe boundaries or normal audio data packet boundaries for segmented storage based on the video gap status of the candidate segment positions, avoiding cache misordering and gap compression, and improving recording time consistency. Establishing a time correspondence through acquisition generation identifiers and acquisition time stamps, distinguishing between cached data and recovered data, and retaining gap intervals helps improve time alignment accuracy, timing consistency, and segmentation adaptability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of audio and video data processing, specifically to a method for continuous recording of multi-source audio and video on a unified timeline for glasses. Background Technology

[0002] With the development of smart glasses and other wearable devices, cameras, microphones, and other acquisition components are widely used for first-person perspective shooting, on-site work documentation, and daily life recording. Different acquisition components typically generate video frames or audio data packets independently and mark the acquisition time according to their respective time bases. The recording module synchronizes, encodes, and segments the received audio and video data for storage.

[0003] During long-term continuous recording, factors such as user operation, power consumption control, device temperature, system resource scheduling, or abnormalities in the acquisition components may cause some acquisition sources to pause, resume, or reinitialize. At the same time, data that has not yet been output from the acquisition driver or data buffer may continue to arrive after acquisition resumes, causing a difference between the data reception order and the actual acquisition order. This may be accompanied by timestamp rollback, jumps, or interruptions, thereby affecting the temporal correspondence of audio and video data and the continuity of the recording file.

[0004] Therefore, how to accurately reflect the actual acquisition sequence and interruption period of each acquisition source during the continuous recording of multi-source audio and video in smart glasses, reduce the timing disorder caused by the delayed output of cached data, and maintain the continuity and availability of recording segments when some acquisition sources are temporarily lacking data has become a technical problem that needs to be solved. Summary of the Invention

[0005] Based on the shortcomings of the prior art described above, the purpose of this invention is to provide a method for continuous recording of multi-source audio and video on a unified timeline for glasses, so as to solve the above-mentioned technical problems.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for continuous recording of multi-source audio and video on a unified timeline for eyeglasses, comprising: S1: Acquire the continuous recording task of glasses, including the recording segment duration, collect multi-source audio and video data and the running status of the collection source, set the collection source identifier for each collection source, set and update the collection generation identifier according to the running status of the collection source, associate the source timestamp with the continuous clock time of the glasses device to form a collection time stamp, and generate a multi-source collection sequence. S2: Based on the multi-source acquisition sequence, identify the candidate breakpoints of each acquisition source, establish a double-sided observation interval before and after the candidate breakpoint, distinguish between the cached data before the breakpoint and the recovered data after the breakpoint according to the acquisition generation identifier and acquisition time stamp, and determine the time interval between the two as the gap interval of the corresponding acquisition source to generate a continuous sorting sequence. S3: Based on the continuous sorting sequence and acquisition time stamp, establish the time correspondence between the source timestamp and the continuous clock of the glasses device according to the acquisition generation, arrange the multi-source audio and video data to the corresponding time position and retain the gap interval, and generate a unified time axis record; S4: Based on the continuous recording task of glasses, determine the candidate segment position according to the recording segment duration. When the candidate segment position is outside the gap interval of the video acquisition source, select the video keyframe boundary. When the candidate segment position is inside the gap interval of the video acquisition source, select the boundary of the normally acquired audio data packet. Segment the recording of the unified time axis and save it to generate continuous audio and video recording data of glasses from multiple sources.

[0007] The present invention is further configured such that the continuous recording task of the glasses includes: a recording task identifier, a list of acquisition sources, and a recording segment duration; Multi-source audio and video data includes: video frames generated by video acquisition sources and audio data packets generated by audio acquisition sources, with each video frame and audio data packet carrying a source timestamp; The data acquisition source's operating status includes: start-up status, normal acquisition status, paused status, resumed status, and reinitialization status.

[0008] The present invention is further configured such that S1 includes: Start video and audio acquisition sources based on the acquisition source list, and set acquisition source identifiers for each acquisition source; When the acquisition source enters the start state, the acquisition generation identifier is set. When the acquisition source changes from the pause state to the resume state or enters the re-initialization state, the acquisition generation identifier is updated, and the original acquisition generation identifier is retained for the multi-source audio and video data acquired before the update. The source timestamps carried by the video frames and audio data packets are associated with the continuous clock time of the glasses device at the corresponding acquisition time to form an acquisition time stamp; The multi-source audio and video data are associated with the recording module in the order they are received, along with the multi-source audio and video data, the operating status of the acquisition source, the acquisition source identifier, the acquisition generation identifier, and the acquisition time stamp, to generate a multi-source acquisition sequence.

[0009] The present invention is further configured such that identifying candidate breakpoints for each acquisition source includes: Extract multi-source audio and video data from the multi-source acquisition sequence according to the acquisition source identifier; Check the acquisition generation identifier, acquisition source running status, source timestamp, and acquisition time stamp of adjacent multi-source audio and video data; The corresponding position is determined as a candidate breakpoint when any of the following conditions are met; The conditions include: a change in the acquisition generation identifier; the acquisition source's operating status indicating that acquisition is paused or resumed; a source timestamp rollback or acquisition timestamp interruption; and a change in the acquisition generation identifier of subsequent audio and video data.

[0010] The present invention is further configured such that establishing the bilateral observation interval includes: According to the arrangement order in the multi-source acquisition sequence, with the candidate breakpoint as the boundary, the audio and video data of the corresponding acquisition source located before the candidate breakpoint and with continuous acquisition time stamps are extracted to form the observation interval before the breakpoint; Extract the audio and video data from the first audio and video data after the candidate breakpoint to the point where the data segment carrying the updated acquisition generation identifier and the acquisition time stamp is continuous, and form the observation interval after the breakpoint; The observation interval before the breakpoint and the observation interval after the breakpoint are combined to form a two-sided observation interval.

[0011] The present invention is further configured such that generating a continuous arrangement sequence includes: The audio and video data carrying the generation identifier of the data collected before the breakpoint within the observation interval after the breakpoint are identified as the buffered data before the breakpoint. The audio and video data that carry the updated acquisition generation identifier and have continuous acquisition time stamps within the observation interval after the breakpoint are identified as the data recovered after the breakpoint. According to the acquisition time stamp, the cached data before the breakpoint and the audio and video data in the observation interval before the breakpoint are merged and arranged, and the last audio and video data after the arrangement is determined as the last data before the breakpoint. The time interval after the acquisition time of the last data item before the breakpoint and before the earliest acquisition time in the recovered data after the breakpoint is determined as the gap interval of the corresponding acquisition source. The data is organized in the order of the audio and video data before the breakpoint, the gap interval, and the recovered data after the breakpoint, and a continuous organized sequence is generated.

[0012] The present invention is further configured such that establishing the time correspondence between the source timestamp and the continuous clock of the glasses device includes: The sequence is divided into continuous data collection sequences based on the data collection source identifier and the data collection generation identifier; Extract the acquisition timestamps corresponding to the same acquisition generation, and establish a time correspondence based on the source timestamps associated in the acquisition timestamps and the continuous clock time of the glasses device; When the data collection generation identifier changes, the time correspondence of the previous data collection generation is no longer used for the updated data collection generation, and a new time correspondence is established for the updated data collection generation.

[0013] The present invention is further configured such that generating a unified timeline record includes: The temporal location of multi-source audio and video data is determined based on the temporal correspondence of each acquisition generation. Arrange the cached data before the breakpoint before the corresponding gap interval, and arrange the recovered data after the breakpoint after the corresponding gap interval, while retaining the gap interval of each acquisition source. Multi-source audio and video data from different acquisition sources are merged according to time and location to generate a unified timeline record.

[0014] The present invention is further configured such that determining the candidate segment position includes: The starting position of the unified timeline is used as the starting position of the first recording segment. The ending position of each recording segment is determined in sequence according to the duration of the recording segment, and each ending position is determined as a candidate segment position.

[0015] The present invention is further configured such that generating continuous multi-source audio and video recording data for glasses includes: Determine whether the candidate segment position is located within the gap interval of the video acquisition source; For candidate segment locations located outside the gap interval of the video acquisition source, the boundary of the first video keyframe no earlier than the candidate segment location is selected as the segment boundary; For candidate segment positions located within the gap interval of the video acquisition source, the boundary of the first audio data packet output by the audio acquisition source in normal acquisition state, which is no earlier than the candidate segment position, is selected as the segment boundary. The unified timeline is divided according to the segment boundaries, and the corresponding gap intervals are retained to generate continuous recording data of multi-source audio and video for glasses.

[0016] This invention provides a method for continuous recording of multi-source audio and video on a unified timeline for eyeglasses. The method comprises: S1: acquiring a continuous recording task for eyeglasses, including recording segment durations; collecting multi-source audio and video data and the operating status of the acquisition sources; setting acquisition source identifiers for each acquisition source; setting and updating acquisition generation identifiers based on the acquisition source operating status; associating source timestamps with the continuous clock time of the eyeglasses device to form an acquisition time stamp; and generating a multi-source acquisition sequence. S2: identifying candidate breakpoints for each acquisition source based on the multi-source acquisition sequence; establishing bilateral observation intervals before and after the candidate breakpoints; distinguishing between buffered data before the breakpoint and recovered data after the breakpoint based on the acquisition generation identifier and the acquisition time stamp; and determining the time interval between the two as the gap interval for the corresponding acquisition source. S3: Based on the continuous processing sequence and acquisition time stamp, establish the time correspondence between the source timestamp and the continuous clock of the glasses device according to the acquisition generation, arrange the multi-source audio and video data to the corresponding time positions and retain the gap interval, and generate a unified timeline record; S4: Based on the glasses continuous recording task, determine the candidate segment positions according to the recording segment duration, select the video keyframe boundary when the candidate segment position is outside the gap interval of the video acquisition source, and select the normally acquired audio data packet boundary when the candidate segment position is inside the gap interval of the video acquisition source, segment and save the unified timeline record, and generate glasses multi-source audio and video continuous recording data. The beneficial effects include: Improve the accuracy of multi-source time alignment: By establishing time correspondences through acquisition generation identifiers and acquisition timestamps, it helps to reduce the impact of pausing, resuming, or re-initializing the acquisition source on audio and video time alignment.

[0017] Enhance the temporal consistency of continuous recording: By distinguishing between cached data before the breakpoint and recovered data after the breakpoint and retaining the gap interval, it helps to reduce the out-of-order cached data and the compression of interruption periods, making the recording time closer to the actual acquisition process.

[0018] Improve the adaptability of segmented storage: Selecting video keyframe boundaries or audio data packet boundaries based on the video gap state where the candidate segment position is located helps to improve the segmented continuity and the availability of recorded data in the event of temporary video interruption.

[0019] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 The flowchart illustrates a method for continuous recording of multi-source audio and video on a unified timeline for glasses, as an exemplary embodiment of the present invention. Detailed Implementation

[0021] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0022] It should be noted that the accompanying drawings provided in the following embodiments are only for illustrating the basic concept of the present invention. The drawings only show components related to the present invention and are not drawn according to the actual number, shape, and size of the components in the actual implementation. In the actual implementation process, the type, quantity, proportion, and layout of each component can be adjusted according to the specific application scenario, and its actual structure and layout can be more complex.

[0023] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0024] Example:

[0025] A method for continuous recording of multi-source audio and video on a unified timeline for glasses, such as Figure 1 As shown, it includes: S1: Acquire the continuous recording task of the glasses, including the recording segment duration, collect multi-source audio and video data and the running status of the collection sources, set the collection source identifier for each collection source, set and update the collection generation identifier according to the running status of the collection source, associate the source timestamp with the continuous clock time of the glasses device to form a collection time stamp, and generate a multi-source collection sequence.

[0026] Specifically, this embodiment is executed by the recording control module built into the glasses device. The glasses device includes, but is not limited to, smart glasses, augmented reality glasses, industrial inspection glasses, or law enforcement recording glasses. The recording control module includes a task management unit, a video acquisition adapter unit, an audio acquisition adapter unit, a status management unit, a time stamp generation unit, and a sequence recording unit. The video acquisition adapter unit can use a Camera2 or V4L2 acquisition interface, and the audio acquisition adapter unit can use an AudioRecord or ALSA acquisition interface.

[0027] The task management unit receives the recording start request through physical buttons, voice commands, or application programming interfaces, and establishes a continuous recording task for glasses, including a recording task identifier, a list of acquisition sources, and the recording duration of each segment. The recording task identifier can be generated using a universally unique identifier. The list of acquisition sources records the acquisition source type, hardware channel, data format, and source timestamp unit. In this embodiment, it is assumed that the list of acquisition sources includes one front-facing camera and one microphone, with the front-facing camera serving as the video acquisition source and the microphone serving as the audio acquisition source.

[0028] The recording segment duration indicates the target duration of subsequent recording segments. This invention defaults to 60 seconds, but can be set to a range of 30 to 300 seconds depending on storage capacity and single-segment file recovery time. When the recording start request does not provide a recording segment duration, 60 seconds is used. When the recording segment duration is less than 30 seconds or more than 300 seconds, it can be adjusted to 30 seconds or 300 seconds respectively.

[0029] Multi-source audio and video data includes video frames and audio data packets. Video frames can use uncompressed YUV format or H.264 / H.265 encoding formats; this embodiment uses a resolution of 1920×1080 and 30 frames per second. Audio data packets can use pulse code modulation format or AAC encoding format; this embodiment uses a sampling rate of 48000 Hz, 16-bit sampling depth, and mono configuration. Video frames carry a source timestamp indicating the start time of exposure or the time of video frame generation, and audio data packets carry a source timestamp indicating the time of acquisition of the first sampling point. When video frames or audio data packets are encoded, the encoded data inherits the source timestamp of the input data.

[0030] The acquisition source's operating status includes startup, normal acquisition, paused, resumed, and reinitialization states. After the acquisition interface is established, it enters the startup state; after outputting the first valid data item, it enters the normal acquisition state; when it temporarily stops generating new data, it enters the paused state; when data output resumes from the paused state, it enters the resumed state; and when the acquisition interface is re-established due to driver errors, resource reallocation, or device reconstruction, it enters the reinitialization state. When the acquisition callback is successful, the data length is greater than 0, and the source timestamp can be read, the video frame or audio data packet is determined to be valid data.

[0031] The task management unit creates acquisition interfaces based on the acquisition source list, sets acquisition source identifier V01 for video acquisition sources, and sets acquisition source identifier A01 for audio acquisition sources. The acquisition source identifier remains unchanged when the acquisition source status changes. When an acquisition source first enters the start state, the acquisition generation number is set to 1, with video acquisition sources corresponding to V01-G001 and audio acquisition sources corresponding to A01-G001. The acquisition generation identifier is not updated when the acquisition source maintains a normal acquisition state or enters a pause state. The acquisition generation number is incremented by 1 when the acquisition source changes from a pause state to a resume state or enters a reinitialization state.

[0032] The status management unit only updates the acquisition generation identifier when the operating status of the acquisition source actually changes. Repeated status notifications do not trigger updates. After re-initialization, the identifier is not updated again when entering the recovery or normal acquisition state. The acquisition generation identifier update operation and the marking operation of the first multi-source audio and video data after recovery are completed through the same serial processing thread. Different acquisition sources maintain their own acquisition generation identifiers.

[0033] After obtaining multi-source audio and video data through the acquisition callback, the acquisition adaptation unit associates the current acquisition generation identifier with the data before writing it into the acquisition buffer. Data that has already entered the acquisition buffer before the update retains the original acquisition generation identifier. The acquisition buffer can directly save the acquisition generation identifier. When the acquisition interface does not support additional markers, it can establish the correspondence between multi-source audio and video data and acquisition source identifier and acquisition generation identifier through the incrementing buffer sequence number.

[0034] The continuous clock time of the glasses device uses a monotonic time that is unaffected by system date, network time synchronization, or time zone adjustment. In this embodiment, the CLOCK_BOOTTIME continuous timing interface with nanosecond precision is used. The Android system can use the SystemClock.elapsedRealtimeNanos interface, and the embedded Linux system can read CLOCK_BOOTTIME through the clock_gettime interface. The time stamp generation unit reads the source timestamp and the continuous clock time of the glasses device in the same acquisition callback to form the acquisition time stamp, and then writes the acquisition time stamp and multi-source audio and video data together into the acquisition buffer.

[0035] The video capture interface can obtain the source timestamp of the video frame through Camera2's SENSOR_TIMESTAMP or V4L2 buffer timestamp, and the audio capture interface can obtain the source timestamp of the audio data packet through AudioTimestamp or ALSA hardware timestamp. When the source timestamp and the continuous clock time of the glasses device use the same time base, they are directly associated. When the time base cannot be confirmed to be consistent, the continuous clock time of the glasses device is read immediately after the capture callback obtains multi-source audio and video data.

[0036] The status management unit generates a data acquisition source operation status record item when the operation status of the acquisition source changes. The sequence recording unit uses a producer-consumer queue to receive audio and video record items and data acquisition source operation status record items, and assigns an incrementing reception sequence number through atomic write operations. The audio and video record item is associated with multi-source audio and video data, data acquisition source operation status, data acquisition source identifier, data acquisition generation identifier, and data acquisition timestamp. The sequence recording unit saves the record items according to the reception sequence number, without reordering them according to the source timestamp, and generates a multi-source acquisition sequence.

[0037] Assuming that the video frame generated before the video acquisition source V01 pauses has entered the acquisition buffer and carries V01-G001, it is updated to V01-G002 when the video acquisition source resumes; the buffered video frame still retains V01-G001 when it arrives at the sequence recording unit after the recovery, and the newly generated video frame after the recovery carries V01-G002, thus providing a basis for distinguishing between the buffered data before the breakpoint and the data recovered after the breakpoint.

[0038] S2: Based on the multi-source acquisition sequence, identify the candidate breakpoints of each acquisition source, establish a double-sided observation interval before and after the candidate breakpoint, distinguish between the cached data before the breakpoint and the recovered data after the breakpoint according to the acquisition generation identifier and acquisition time stamp, and determine the time interval between the two as the gap interval of the corresponding acquisition source to generate a continuous sorting sequence.

[0039] Specifically, the recording control module uses hash grouping, sliding window, bidirectional scanning and stable merge sort algorithms to process multi-source acquisition sequences; it groups audio and video recording items and acquisition source running status recording items according to the acquisition source identifier, and the recording items corresponding to the same acquisition source identifier form an acquisition source sequence. The audio and video recording items are used as data to be processed, and the acquisition source running status recording items are used as the basis for status changes.

[0040] The recording control module checks adjacent multi-source audio and video data in the same acquisition source sequence according to the received sequence number. The check content includes acquisition generation identifier, acquisition source running status, source timestamp and acquisition timestamp. If the continuous clock time of the glasses device within the same acquisition generation keeps increasing, the interval between the continuous clock times of adjacent glasses devices does not exceed the continuity threshold, and the source timestamp keeps increasing or meets the timestamp wrap-around rule, the acquisition timestamp is determined to be continuous.

[0041] The continuity threshold defaults to 3 nominal output cycles. For video acquisition sources with 30 frames per second, 100 milliseconds is used, and for audio acquisition sources with a 20-millisecond encapsulation time, 60 milliseconds is used. When the acquisition configuration changes, the continuity threshold can be selected within the range of 2 to 5 nominal output cycles. When the output jitter is small, a smaller value is used, and when the output jitter is large, a larger value is used.

[0042] When the acquisition generation identifier of adjacent multi-source audio and video data changes, the position between adjacent multi-source audio and video data is determined as a candidate breakpoint; when there are pause state records or resume state records between adjacent multi-source audio and video data, the position corresponding to the acquisition source running status record item is determined as a candidate breakpoint; when source timestamp rollback or acquisition timestamp interruption occurs, and the acquisition generation identifier of subsequent multi-source audio and video data changes, the source timestamp rollback position or acquisition timestamp interruption position is determined as a candidate breakpoint.

[0043] The default backward check duration after source timestamp rollback or data acquisition time stamp interruption is 2 seconds, which can be set within the range of 1 to 5 seconds depending on the maximum recovery time of the acquisition source. If no updated acquisition generation identifier appears within the backward check duration, the candidate breakpoint is not confirmed, and the check continues after new data is acquired. When multiple candidate breakpoint conditions are triggered simultaneously during the same pause and recovery process, duplicate results are merged based on the acquisition generation identifier before the update, the acquisition generation identifier after the update, and the data acquisition source running status record.

[0044] The recording control module starts scanning forward from the last multi-source audio and video data before the candidate breakpoint, extracting the last multi-source audio and video data with the same acquisition generation identifier and continuous acquisition time stamp, forming the observation interval before the breakpoint; the default observation duration is 500 milliseconds, which can be set in the range of 200 milliseconds to 2000 milliseconds according to the maximum retention time of the acquisition buffer. The forward scan stops when it encounters a change in the acquisition generation identifier, an interruption in the acquisition time stamp, or the previous candidate breakpoint.

[0045] The recording control module starts scanning backward from the first multi-source audio and video data after the candidate breakpoint until a stable data segment is formed, carrying the updated acquisition generation identifier and continuous acquisition time stamp. The multi-source audio and video data obtained by scanning forms an observation interval after the breakpoint. The duration of the stable data segment is set to 200 milliseconds by default, which can be set in the range of 100 milliseconds to 500 milliseconds. It also contains at least 3 multi-source audio and video data. The default duration of a single backward scan is 2 seconds. If a stable data segment is not formed, the scan continues in the next processing.

[0046] Multi-source audio and video data within the observation interval after the breakpoint, carrying the acquisition generation identifier before the breakpoint and with the continuous clock time of the glasses device earlier than the start time of the stable data segment, are identified as pre-breakpoint buffered data. Multi-source audio and video data carrying the updated acquisition generation identifier and with continuous acquisition timestamps are identified as post-breakpoint recovered data. The recording control module merges and arranges the pre-breakpoint buffered data and the multi-source audio and video data within the observation interval before the breakpoint according to the order of the continuous clock time of the glasses device, the source timestamp, and the receiving sequence number.

[0047] The last multi-source audio and video data after sorting is determined as the last data before the breakpoint, and the multi-source audio and video data with the earliest acquisition time in the recovered data after the breakpoint is determined as the first data after the breakpoint. A gap interval is formed after the acquisition time corresponding to the last data before the breakpoint and before the acquisition time corresponding to the first data after the breakpoint. The gap interval records the acquisition source identifier, the gap start time, and the gap end time, and is not filled with fictitious video frames or audio data packets.

[0048] The recording control module organizes the multi-source audio and video data before the breakpoint, the gap interval, and the recovered data after the breakpoint in the order of the merged arrangement. The multi-source audio and video data outside the observation interval on both sides maintains the original order, and the gap interval is retained for different acquisition sources, generating a continuous organized sequence.

[0049] S3: Based on the continuous sorting sequence and acquisition time stamp, establish the time correspondence between the source timestamp and the continuous clock of the glasses device according to the acquisition generation, arrange the multi-source audio and video data to the corresponding time position and retain the gap interval, and generate a unified time axis record.

[0050] Specifically, the recording control module divides the continuous sorting sequence according to the acquisition source identifier and acquisition generation identifier, extracts the acquisition timestamp corresponding to the same acquisition generation, and arranges the source timestamp and the continuous clock time of the glasses device according to the source timestamp to form a time correspondence table; each group of source timestamps and continuous clock time of the glasses device in the time correspondence table forms a time anchor point, and the internal time precision follows the nanosecond precision set in step S1.

[0051] When both the source timestamp and the continuous clock time of the glasses device are increasing, the acquisition timestamp is determined as a valid time anchor point. When the source timestamp is repeated, the first acquisition timestamp is determined as the time anchor point, and subsequent multi-source audio and video data maintain the sequential relationship according to the reception sequence number. When the continuous clock time of the glasses device rolls back, the corresponding acquisition timestamp is marked as an abnormal acquisition timestamp and is not used as a time anchor point. The corresponding time position is determined based on adjacent valid time anchor points and source timestamps within the same acquisition generation. When there are not enough adjacent valid time anchor points, the corresponding multi-source audio and video data is temporarily stored, and the corresponding time position is determined after obtaining the next valid time anchor point.

[0052] Within the same acquisition generation, a piecewise linear interpolation algorithm is used to establish the time correspondence between adjacent time anchors. When the source timestamp of multi-source audio and video data is consistent with the time anchor, the continuous clock time of the glasses device corresponding to the time anchor is directly used. When the source timestamp is located between two adjacent time anchors, the continuous clock time of the glasses device is determined according to the position of the source timestamp between the two time anchors. The piecewise linear interpolation does not cross the boundary or gap interval of the acquisition generation.

[0053] When multi-source audio and video data already carries complete acquisition timestamps, the continuous clock time of the glasses device in the acquisition timestamp is used first. When the encoded data only retains the source timestamp, the continuous clock time of the glasses device is determined through the time correspondence table corresponding to the same acquisition generation. When the acquisition generation identifier changes, the time correspondence table of the previous acquisition generation is discontinued, and the time correspondence table is re-established with the first valid acquisition timestamp of the updated acquisition generation.

[0054] The recording control module determines the earliest valid continuous clock time of the glasses device in the continuous sorting sequence as the starting position of the unified time axis recording. The continuous clock time of the glasses device corresponding to each multi-source audio and video data item forms a corresponding time position relative to the starting position of the unified time axis recording. Multi-source audio and video data from different acquisition sources are arranged according to the corresponding time position using a priority queue multi-path merging algorithm. When the corresponding time positions are the same, they are arranged according to the acquisition source identifier and the receiving sequence number.

[0055] The start and end times of the gaps are converted into corresponding time positions in a unified timeline record. Data cached before the breakpoint is arranged before the gap interval, and data recovered after the breakpoint is arranged after the gap interval. Gap intervals from different acquisition sources are retained separately. When a gap occurs in one acquisition source, the corresponding time position of another acquisition source is not changed, and a unified timeline record is finally generated.

[0056] Assuming that the three video frames of video acquisition source V01 within V01-G001 correspond to 100000 milliseconds, 100033 milliseconds, and 100066 milliseconds respectively, the first video frame within V01-G002 corresponds to 101233 milliseconds, and the first audio data packet of audio acquisition source A01 corresponds to 100005 milliseconds; the recording control module determines 100000 milliseconds as the starting position for unified timeline recording, arranges the video frames at 0 milliseconds, 33 milliseconds, 66 milliseconds, and 1233 milliseconds, arranges the first audio data packet at 5 milliseconds, and retains the gap interval of video acquisition source V01 from 66 milliseconds to 1233 milliseconds.

[0057] S4: Based on the continuous recording task of glasses, determine the candidate segment position according to the recording segment duration. When the candidate segment position is outside the gap interval of the video acquisition source, select the video keyframe boundary. When the candidate segment position is inside the gap interval of the video acquisition source, select the boundary of the normally acquired audio data packet. Segment the recording of the unified time axis and save it to generate continuous audio and video recording data of glasses from multiple sources.

[0058] Specifically, the recording control module uses fixed-length position generation, gap interval binary search, video keyframe search, audio data packet search, and segmented media encapsulation methods to process the unified timeline recording; the segmented saving program takes the starting position of the unified timeline recording as the starting position of the first recording segment, and determines the candidate segment positions in sequence according to the recording segment duration, and the subsequent candidate segment positions are not offset by the actual segment boundaries.

[0059] When multiple candidate segment positions correspond to the same segment boundary, only one is retained. Candidate segment positions that are earlier than or located at the confirmed segment boundary are marked as processed. The segment saving program arranges the gap intervals of the video acquisition source according to the gap start position. The gap start position is included in the gap interval, and the gap end position is not included in the gap interval. Then, a binary search is used to determine whether the candidate segment position is located within the gap interval of the video acquisition source.

[0060] When the candidate segment position is outside the gap interval of the video acquisition source, video frames are retrieved from the candidate segment position backwards, and the first video keyframe boundary no earlier than the candidate segment position is selected as the segment boundary. The video keyframe located at the video keyframe boundary is included in the next recorded segment. The default video keyframe interval is 2 seconds, which can be set within the range of 1 to 5 seconds according to the allowable offset duration of the segment boundary.

[0061] When the candidate segment position is located within the gap interval of the video acquisition source, audio data packets are retrieved starting from the candidate segment position. The boundary of the first audio data packet output by an audio acquisition source in normal acquisition state, which is no earlier than the candidate segment position, is selected as the segment boundary. Audio data packets located at the audio data packet boundary are assigned to the next recording segment. When both the video acquisition source and the audio acquisition source have gaps, the segment boundary is not confirmed temporarily. If no audio data packets that meet the conditions are obtained before the end of the recording task, the end position of the unified timeline record is taken as the end position of the last recording segment.

[0062] When the video acquisition source recovers from the gap interval, the recording control module sends a synchronization frame request before submitting the first recovered video frame to the video encoder, so that the first recoverable encodeable video frame is given priority to form the video key frame; if the synchronization frame request does not take effect immediately, the next recording segment continues to save audio data packets, and starts saving video frames after obtaining the first video key frame.

[0063] This embodiment uses a segmented MP4 encapsulation format to save the continuous audio and video recording data of the glasses from multiple sources. It initializes the video track parameters, audio track parameters, and time base of the segment recording. The media segment saves the video frames and audio data packets within the corresponding recording segment. When the remaining time of the unified timeline recording is insufficient for the recording segment duration, the remaining content is saved as the last recording segment without filling in blank data.

[0064] When the gap in the video capture source crosses the segment boundary, the gap before the segment boundary is recorded in the time index of the previous record segment, and the gap after the segment boundary is recorded in the time index of the next record segment. The two gaps continue to be associated with the same capture source identifier, and the gap duration is not compressed.

[0065] Assuming the unified timeline recording starts at 0 seconds and ends at 205 seconds, with each recording segment lasting 60 seconds, and candidate segment positions at 60 seconds, 120 seconds, and 180 seconds, the gap interval of the video acquisition source starts at 119.8 seconds and ends at 122.4 seconds; the boundary of the first video keyframe after 60 seconds is at 60.8 seconds, the boundary of the first audio data packet output by the audio acquisition source in normal acquisition state after 120 seconds is at 120.005 seconds, and the boundary of the first video keyframe after 180 seconds is at 180.6 seconds.

[0066] The segmented saving procedure thus forms four recording segments: 0 seconds to 60.8 seconds, 60.8 seconds to 120.005 seconds, 120.005 seconds to 180.6 seconds, and 180.6 seconds to 205 seconds. The gap from 119.8 seconds to 120.005 seconds is recorded in the second recording segment, and the gap from 120.005 seconds to 122.4 seconds is recorded in the third recording segment, generating continuous multi-source audio and video recording data for the glasses.

[0067] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for recording a multi-source audio-video unified timeline of eyeglasses, characterized in that, include: S1: Acquire the continuous recording task of glasses, including the recording segment duration, collect multi-source audio and video data and the running status of the collection source, set the collection source identifier for each collection source, set and update the collection generation identifier according to the running status of the collection source, associate the source timestamp with the continuous clock time of the glasses device to form a collection time stamp, and generate a multi-source collection sequence. S2: Based on the multi-source acquisition sequence, identify the candidate breakpoints of each acquisition source, establish a double-sided observation interval before and after the candidate breakpoint, distinguish between the cached data before the breakpoint and the recovered data after the breakpoint according to the acquisition generation identifier and acquisition time stamp, and determine the time interval between the two as the gap interval of the corresponding acquisition source to generate a continuous sorting sequence. S3: Based on the continuous sorting sequence and acquisition time stamp, establish the time correspondence between the source timestamp and the continuous clock of the glasses device according to the acquisition generation, arrange the multi-source audio and video data to the corresponding time position and retain the gap interval, and generate a unified time axis record; S4: Based on the continuous recording task of glasses, determine the candidate segment position according to the recording segment duration. When the candidate segment position is outside the gap interval of the video acquisition source, select the video keyframe boundary. When the candidate segment position is inside the gap interval of the video acquisition source, select the boundary of the normally acquired audio data packet. Segment the recording of the unified time axis and save it to generate continuous audio and video recording data of glasses from multiple sources.

2. The method of claim 1, wherein, The continuous recording task for glasses includes: recording task identifier, data source list, and recording segment duration; Multi-source audio and video data includes: video frames generated by video acquisition sources and audio data packets generated by audio acquisition sources, with each video frame and audio data packet carrying a source timestamp; The data acquisition source's operating status includes: start-up status, normal acquisition status, paused status, resumed status, and reinitialization status.

3. The method of claim 2, wherein, S1 includes: Start video and audio acquisition sources based on the acquisition source list, and set acquisition source identifiers for each acquisition source; When the acquisition source enters the start state, the acquisition generation identifier is set. When the acquisition source changes from the pause state to the resume state or enters the re-initialization state, the acquisition generation identifier is updated, and the original acquisition generation identifier is retained for the multi-source audio and video data acquired before the update. The source timestamps carried by the video frames and audio data packets are associated with the continuous clock time of the glasses device at the corresponding acquisition time to form an acquisition time stamp; The multi-source audio and video data are associated with the recording module in the order they are received, along with the multi-source audio and video data, the operating status of the acquisition source, the acquisition source identifier, the acquisition generation identifier, and the acquisition time stamp, to generate a multi-source acquisition sequence.

4. The method of claim 1, wherein, Candidate breakpoints for each acquisition source include: Extract multi-source audio and video data from the multi-source acquisition sequence according to the acquisition source identifier; Check the acquisition generation identifier, acquisition source running status, source timestamp, and acquisition time stamp of adjacent multi-source audio and video data; The corresponding position is determined as a candidate breakpoint when any of the following conditions are met; The conditions include: a change in the acquisition generation identifier; the acquisition source's operating status indicating that acquisition is paused or resumed; a source timestamp rollback or acquisition timestamp interruption; and a change in the acquisition generation identifier of subsequent audio and video data.

5. The method of claim 4, wherein, Establishing a two-sided observation interval includes: According to the arrangement order in the multi-source acquisition sequence, with the candidate breakpoint as the boundary, the audio and video data of the corresponding acquisition source located before the candidate breakpoint and with continuous acquisition time stamps are extracted to form the observation interval before the breakpoint; Extract the audio and video data from the first audio and video data after the candidate breakpoint to the point where the data segment carrying the updated acquisition generation identifier and the acquisition time stamp is continuous, and form the observation interval after the breakpoint; The observation interval before the breakpoint and the observation interval after the breakpoint are combined to form a two-sided observation interval.

6. The method of claim 5, wherein the method further comprises: Generating a continuous arrangement sequence includes: The audio and video data carrying the generation identifier of the data collected before the breakpoint within the observation interval after the breakpoint are identified as the buffered data before the breakpoint. The audio and video data that carry the updated acquisition generation identifier and have continuous acquisition time stamps within the observation interval after the breakpoint are identified as the data recovered after the breakpoint. According to the acquisition time stamp, the cached data before the breakpoint and the audio and video data in the observation interval before the breakpoint are merged and arranged, and the last audio and video data after the arrangement is determined as the last data before the breakpoint. The time interval after the acquisition time of the last data item before the breakpoint and before the earliest acquisition time in the recovered data after the breakpoint is determined as the gap interval of the corresponding acquisition source. The data is organized in the order of the audio and video data before the breakpoint, the gap interval, and the recovered data after the breakpoint, and a continuous organized sequence is generated.

7. The method of claim 1, wherein, Establishing the time correspondence between the source timestamp and the continuous clock of the glasses device includes: The sequence is divided into continuous data collection sequences based on the data collection source identifier and the data collection generation identifier; Extract the acquisition timestamps corresponding to the same acquisition generation, and establish a time correspondence based on the source timestamps associated in the acquisition timestamps and the continuous clock time of the glasses device; When the data collection generation identifier changes, the time correspondence of the previous data collection generation is no longer used for the updated data collection generation, and a new time correspondence is established for the updated data collection generation.

8. The method of claim 7, wherein the method further comprises: Generating a unified timeline record includes: The temporal location of multi-source audio and video data is determined based on the temporal correspondence of each acquisition generation. Arrange the cached data before the breakpoint before the corresponding gap interval, and arrange the recovered data after the breakpoint after the corresponding gap interval, while retaining the gap interval of each acquisition source. Multi-source audio and video data from different acquisition sources are merged according to time and location to generate a unified timeline record.

9. The method of claim 1, wherein, Determining the candidate segment position includes: The starting position of the unified timeline is used as the starting position of the first recording segment. The ending position of each recording segment is determined in sequence according to the duration of the recording segment, and each ending position is determined as a candidate segment position.

10. The method of claim 9, wherein, The generated multi-source audio and video continuous recording data for glasses includes: Determine whether the candidate segment position is located within the gap interval of the video acquisition source; For candidate segment locations located outside the gap interval of the video acquisition source, the boundary of the first video keyframe no earlier than the candidate segment location is selected as the segment boundary; For candidate segment positions located within the gap interval of the video acquisition source, the boundary of the first audio data packet output by the audio acquisition source in normal acquisition state, which is no earlier than the candidate segment position, is selected as the segment boundary. The unified timeline is divided according to the segment boundaries, and the corresponding gap intervals are retained to generate continuous recording data of multi-source audio and video for glasses.