A multi-source data storage and processing method supporting multi-rate synchronous playback

Abstract

The application discloses a kind of multi-source data storage and processing method supporting multi-rate synchronous playback, S1: establishing unified time axis and carrying out time normalization;S2: generating time sampling point sequence according to unified time axis;S3: construct multi-resolution unified time index layer;S4: write original data and index layer in predetermined method to storage medium;S5: according to target speed selection corresponding time index layer, and execute multi-source synchronous playback processing.Through multi-resolution time index, playback end does not need to carry out complex time calculation or scanning under different speed, but directly selects appropriate index layer, and the playback is driven by the time step of index layer, to reduce runtime calculation burden;Meanwhile, unified index is carried out on the storage format of multi-source data, and the position of multi-source data at the same time point can be obtained by index access once during playing, to avoid a large number of time alignment, interpolation / lookup operation during playing stage, to improve the accuracy and efficiency of multi-source synchronization.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for storing and processing multi-source data that supports multi-rate synchronous playback. Background Technology

[0002] With the development of video acquisition and multi-sensor fusion technology, in fields such as motion analysis, intelligent monitoring, human-computer interaction, and virtual reality / augmented reality, it is usually necessary to record multiple data sources simultaneously on the same timeline, such as video images (RGB video stream), posture data (such as human skeleton key points, IMU posture, 3D pose, etc.), event tagging data (such as behavior labels, detection events, semantic annotations), and other metadata (such as audio features, depth map indexes, etc.). In the process of data playback and analysis, in order to browse faster or analyze more in detail, multi-speed playback is very common, such as 0.5x slow motion, 2x or 4x fast motion. At present, the mainstream implementations are mainly in the following categories: (1) Multi-speed playback method based on a single video timeline: This type of method usually uses the timestamp of the video frame as a basis and attaches metadata (such as posture, events) to the video timeline. During playback, the speed is achieved by changing the number of inter-frame skips or the decoding rate, such as 2x, 4x, 0.5x. However, most of these methods only index keyframes of the video, and the metadata only relies on its own timestamp, lacking a multi-resolution time index structure pre-built for multiple speeds; and after each video frame jump, it is necessary to search or interpolate in the metadata sequence. As the types and number of data sources increase, the runtime synchronous computation complexity increases. (2) Method of pre-generating multiple sets of data versions for different speeds: This method generates multiple sets of video versions with different frame rates in the offline stage, such as 1x video stream, 2x video stream, and 4x video stream, but retains a set of data streams for each speed, which increases the storage cost exponentially. In addition, the timestamp correspondence between different speed versions is complex, and slight drift between versions is easy to occur. At the same time, each version is indexed independently, and additional alignment calculations are required when switching speeds or analyzing across versions. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for storing and processing multi-source data that supports multi-rate synchronous playback.

[0004] The objective of this invention is achieved through the following technical solution: a multi-source data storage and processing method supporting multi-rate synchronous playback, comprising the following steps:

[0005] S1: Establish a unified timeline and perform time normalization;

[0006] S2: Generate a time sampling point sequence based on a unified time axis;

[0007] S3: Construct a multi-resolution unified time index layer;

[0008] S4: Write the raw data and index layer to the storage medium using a predetermined method;

[0009] S5: Select the corresponding time index layer according to the target speed and perform multi-source synchronous playback processing.

[0010] Preferably, step S1 further includes the following step:

[0011] S11: Record the original timestamp of the collected multi-source data;

[0012] S12: Map the original timestamp to the same internal timeline The above forms a unified logical time sequence, where the time unit is a preset time step unit;

[0013] S13: Record a unified timeline The correspondence between the start time, end time and physical time.

[0014] Preferably, in step S2, the predetermined base time granularity is used. Generate a sequence of basic sampling points on a unified time axis. Then, the basic sampling point sequence is downsampled or resampled.

[0015] Preferably, step S3 further includes the following step:

[0016] S31: For each layer of time sampling point sequence Define an index layer The index layer contains several time index entries, and each time index entry corresponds to a sampling point on a unified time axis. ;

[0017] S32: For sampling points Process it;

[0018] S33: Write the metadata to this index layer.

[0019] Preferably, in step S32, the search is performed in the video data. The nearest target video frame, recording the frame's position offset and frame number information in the video data file or data stream;

[0020] In the attitude data, find... The nearest attitude data packet is recorded, along with its position offset and sequence number within the attitude data stream.

[0021] In the event tag data, determine the sampling point. The set of events within the time window, and record the offset information or index range information used to locate these events;

[0022] The offset or index information corresponding to the video data, posture data, and event data mentioned above is compared with the timestamps on a unified timeline. Record them together in the time index item.

[0023] Preferably, step S4 further includes the following step:

[0024] S41: Separate storage is used to write video data, attitude data, and event data into the storage medium in their respective data formats;

[0025] S42: Write the multi-resolution time index layer generated in step S3 as the index part corresponding to the original data into the storage medium;

[0026] S43: Record the location, level, and associated unified timeline information of each index layer in the storage medium.

[0027] Preferably, step S5 further includes the following step:

[0028] S51: Based on the sampling step size corresponding to each layer and the preset recommended speed information, select the speed that matches the target speed. Most matching time index layer ;

[0029] S52: Using the selected time index layer as the driving time axis, based on the timestamp difference between adjacent time index entries in the index layer and the target speed multiplication... Calculate the time index item number or timestamp corresponding to each playback moment, and extract video data offset, attitude data offset and event data offset information from the corresponding time index item;

[0030] S53: Decode the necessary video frames based on the video data offset information recorded in the time index;

[0031] S54: Read the corresponding attitude data packet based on the attitude data offset information in the time index item;

[0032] S55: Based on the event data offset information or index range information in the time index item, read the corresponding event marker and display or trigger it in the playback progress.

[0033] The present invention has the following advantages: By using multi-resolution time indexing, the playback end does not need to perform complex time calculations or scanning at different speeds. Instead, it directly selects the appropriate index layer and drives playback with the time step of the index layer, thereby reducing the computational burden during runtime and improving the speed response speed. At the same time, it performs unified indexing of multi-source data in storage format, so that only one index access is needed each time during playback to obtain the position of multi-source data at the same time point. This avoids a large number of time alignment, interpolation / lookup operations during playback, and improves the accuracy and efficiency of multi-source synchronization. Attached Figure Description

[0034] Figure 1 A schematic diagram illustrating the process of multi-source data storage and processing to support multi-rate synchronous playback. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0036] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0037] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other.

[0038] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0039] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0040] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0041] In this embodiment, as Figure 1 As shown, a method for storing and processing multi-source data that supports multi-rate synchronous playback includes the following steps:

[0042] S1: Establish a unified timeline and perform time normalization;

[0043] S2: Generate a time sampling point sequence based on a unified time axis;

[0044] S3: Construct a multi-resolution unified time index layer;

[0045] S4: Write the original data and index layer to the storage medium using a predetermined method; preferably, step S4 further includes the following step:

[0046] S41: Separate storage is used to write video data, attitude data, and event data into the storage medium in their respective data formats;

[0047] S42: Write the multi-resolution time index layer generated in step S3 as the index part corresponding to the original data into the storage medium;

[0048] S43: Record the location, level, and associated unified timeline information of each index layer in the storage medium.

[0049] S5: Select the corresponding time index layer based on the target playback speed and perform multi-source synchronous playback processing. Through multi-resolution time indexing, the playback end does not need to perform complex time calculations or scans at different playback speeds. Instead, it directly selects the appropriate index layer, using the index layer's time step to drive playback, thereby reducing runtime computational burden and improving playback speed response. Simultaneously, unified indexing of multi-source data in the storage format ensures that only one index access is needed each time during playback to obtain the location of multi-source data at the same time point. This avoids numerous time alignment, interpolation / lookup operations during playback, improving the accuracy and efficiency of multi-source synchronization.

[0050] Furthermore, step S1 also includes the following steps:

[0051] S11: Record the original timestamp of the collected multi-source data; preferably, the multi-source data includes video frames, attitude data packets, event marker data, etc.

[0052] S12: Map the original timestamp to the same internal timeline The above forms a unified logical time sequence, where the time unit is a preset time step unit, so that video frame timestamps, attitude timestamps and event timestamps are all represented by a unified time axis;

[0053] S13: Record a unified timeline The correspondence between the start time, end time and physical time.

[0054] Furthermore, in step S2, based on the predetermined base time granularity... Generate a sequence of basic sampling points on a unified time axis. This sequence is used for 1x speed playback, and then the base sample point sequence is downsampled or resampled. Specifically, the base sample point sequence is downsampled or resampled to generate a time sample point sequence suitable for higher speed playback, such as a 2x speed layer sample point sequence. Its sampling interval is approximately ; 4x speed layer sampling point sequence Its sampling interval is approximately Higher speed layers are generated recursively based on the required speed. A higher temporal resolution sequence of sampling points is generated for slow playback (0.5x). Its sampling interval is less than ,For example This invention enables more precise timing control during slow-motion playback. Specifically, it balances fast forward and slow motion, achieving denser sampling points and smoother time interpolation in slow-motion mode, thus providing more refined and continuous attitude and event information during slow-motion analysis.

[0055] In this embodiment, step S3 further includes the following step:

[0056] S31: For each layer of time sampling point sequence Define an index layer The index layer contains several time index entries, and each time index entry corresponds to a sampling point on a unified time axis. ;

[0057] S32: For sampling points Processing is performed; specifically, in step S32, the video data is searched for... The nearest target video frame, preferably a keyframe or a frame close to a keyframe, is recorded, along with its position offset and frame number information in the video data file or data stream.

[0058] In the attitude data, find... The nearest attitude data packet is recorded, along with its position offset and sequence number within the attitude data stream.

[0059] In the event tag data, determine the sampling point. The set of events within the time window, and record the offset information or index range information used to locate these events;

[0060] The offset or index information corresponding to the video data, posture data, and event data mentioned above is compared with the timestamps on a unified timeline. Record them together in the time index item.

[0061] S33: Write the metadata of the index layer, specifically including sampling step size information, recommendation speed information (e.g., 1.0, 2.0, 4.0, 0.5, etc.) and index item quantity information. Through steps S31 to S33, a multi-resolution unified time index layer set is constructed for multiple time layers with different sampling granularities.

[0062] Furthermore, step S5 also includes the following steps:

[0063] S51: Based on the sampling step size corresponding to each layer and the preset recommended speed information, select the speed that matches the target speed. Most matching time index layer Preferred speed and recommended playback speed. An index layer that is close to and has an index granularity no coarser than the required resolution.

[0064] S52: Using the selected time index layer as the driving time axis, based on the timestamp difference between adjacent time index entries in the index layer and the target speed multiplication... Calculate the time index item number or timestamp corresponding to each playback moment, and extract video data offset, attitude data offset and event data offset information from the corresponding time index item;

[0065] S53: Decode the necessary video frames according to the video data offset information recorded in the time index; specifically, at high speed, directly jump to and decode only the corresponding key frame or a small number of related frames according to the key frame or near the key frame indicated by the index; at low speed or normal speed, decode the required number of intermediate frames in sequence starting from the key frame to obtain a smooth picture.

[0066] S54: Based on the attitude data offset information in the time index item, read the corresponding attitude data packet, and perform interpolation calculations if necessary to obtain the accurate attitude value corresponding to the current playback time point;

[0067] S55: Based on the event data offset information or index range information in the time index item, read the corresponding event marker and display or trigger it in the playback progress. Specifically, through steps S51 to S55, during the playback stage, there is no need to perform a large-scale scan of the original multi-source data. It is only necessary to directly complete the synchronous playback of multi-source data at multiple speeds based on the pre-constructed multi-resolution time index layer. In other words, this invention directly jumps to keyframes based on the high-speed dedicated index layer, which greatly reduces the decoding link length across keyframe sequences. Especially for video encoding formats with dependent structures such as H.264 / H.265, the CPU decoding burden is greatly reduced, and the number of index accesses to metadata is also reduced accordingly, because each playback moment only needs to access one unified time index item, without having to search multiple independent index files separately.

[0068] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for storing and processing multi-source data that supports multi-rate synchronous playback, characterized in that: Includes the following steps: S1: Establish a unified timeline and perform time normalization; S2: Generate a time sampling point sequence based on a unified time axis; S3: Construct a multi-resolution unified time index layer; S4: Write the raw data and index layer to the storage medium using a predetermined method; S5: Select the corresponding time index layer according to the target speed and perform multi-source synchronous playback processing.

2. The multi-source data storage and processing method supporting multi-rate synchronous playback according to claim 1, characterized in that: Step S1 further includes the following steps: S11: Record the original timestamp of the collected multi-source data; S12: Map the original timestamp to the same internal timeline The above forms a unified logical time sequence, where the time unit is a preset time step unit; S13: Record a unified timeline The correspondence between the start time, end time and physical time.

3. The multi-source data storage and processing method supporting multi-rate synchronous playback according to claim 2, characterized in that: In step S2, according to the predetermined base time granularity Generate a sequence of basic sampling points on a unified time axis. Then, the basic sampling point sequence is downsampled or resampled.

4. The multi-source data storage and processing method supporting multi-rate synchronous playback according to claim 3, characterized in that: Step S3 further includes the following steps: S31: For each layer of time sampling point sequence Define an index layer The index layer contains several time index entries, and each time index entry corresponds to a sampling point on a unified time axis. ; S32: For sampling points Process it; S33: Write the metadata to this index layer.

5. The multi-source data storage and processing method supporting multi-rate synchronous playback according to claim 4, characterized in that: In step S32, the video data is searched for... The nearest target video frame, recording the frame's position offset and frame number information in the video data file or data stream; In the attitude data, find... The nearest attitude data packet is recorded, along with its position offset and sequence number within the attitude data stream. In the event tag data, determine the sampling point. The set of events within the time window, and record the offset information or index range information used to locate these events; The offset or index information corresponding to the video data, posture data, and event data mentioned above is compared with the timestamps on a unified timeline. Record them together in the time index item.

6. The multi-source data storage and processing method supporting multi-rate synchronous playback according to claim 5, characterized in that: Step S4 also includes the following steps: S41: Separate storage is used to write video data, attitude data, and event data into the storage medium in their respective data formats; S42: Write the multi-resolution time index layer generated in step S3 as an index part corresponding to the original data into the storage medium; S43: Record the location, level, and associated unified timeline information of each index layer in the storage medium.

7. The multi-source data storage and processing method supporting multi-rate synchronous playback according to claim 6, characterized in that: Step S5 also includes the following steps: S51: Based on the sampling step size corresponding to each layer and the preset recommended speed information, select the speed that matches the target speed. Most matching time index layer ; S52: Using the selected time index layer as the driving time axis, based on the timestamp difference between adjacent time index entries in the index layer and the target speed multiplication... Calculate the time index item number or timestamp corresponding to each playback moment, and extract video data offset, attitude data offset and event data offset information from the corresponding time index item; S53: Decode the necessary video frames based on the video data offset information recorded in the time index; S54: Read the corresponding attitude data packet based on the attitude data offset information in the time index item; S55: Based on the event data offset information or index range information in the time index item, read the corresponding event marker and display or trigger it in the playback progress.

Images