Video multi-stream recording and playback method, device and electronic device

Abstract

The present application provides a method, device and electronic device for recording and playback of multiple video data streams, and relates to the field of data processing. The method for recording and playback of multiple video data streams includes calculating the target video encoding time of the target image frame based on multiple image frames in the video data stream and multiple metadata frames in the original metadata stream that respectively correspond to the multiple image frames; writing the target video encoding time into the original metadata stream to obtain a metadata file; and recording and playback of the video based on the video data stream, the metadata file and other data streams. The method for recording and playback of multiple video data streams provided by the embodiment of the present application can record and playback a camera with multiple lenses and attached sensors, and realize the playback of multiple data streams in synchronization.

Description

Technical Field

[0001] The present application relates to a field, and in particular to a method, device and electronic device for recording and playing back multiple video data streams. Background Art

[0002] Video recording and playback are widely used in fields such as surveillance, creative production, learning, and entertainment. Cameras and webcams achieve this by saving video data to local files. During recording, video is captured, compressed, and stored in real time. During playback, the video is read, decoded, and displayed to reproduce the video.

[0003] However, the recording and playback functions of existing cameras and webcams only operate on the video data of a single camera and cannot recover other sensor data and metadata information transmitted by the camera. In addition, for cameras with multiple sensors (for example, a standard RGB sensor and specialized sensors such as ToF sensors or LiDAR sensors), which have multiple data streams, current playback and recording technologies cannot synchronize the data streams of multiple cameras. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method, device and electronic device for recording and playing back multiple video data streams, which can realize the synchronous playback of multiple data streams by recording and playing back a camera with multiple lenses and accompanying sensors.

[0005] In a first aspect, an embodiment of the present application provides a method for recording and playing back multiple video data streams, the method comprising: calculating a target video encoding time of a target image frame based on multiple image frames in a video data stream and multiple metadata frames in an original metadata stream that respectively correspond to the multiple image frames; writing the target video encoding time into the original metadata stream to obtain a metadata file; and recording and playing back the video based on the video data stream, the metadata file, and other data streams.

[0006] In the above implementation process, the video multi-stream recording and playback method provided in the embodiment of the present application calculates the target video encoding time of the target image frame, writes it into the original metadata stream, and performs recording and playback operations based on the video data stream, metadata file, and other data streams. The video multi-stream recording and playback method provided in the embodiment of the present application achieves precise synchronization of multiple data streams, ensuring time consistency during the multi-stream recording and playback process.

[0007] Calculating a target video encoding time of a target image frame based on multiple image frames in a video data stream and multiple metadata frames in an original metadata stream that respectively correspond to the multiple image frames includes: obtaining a target recording timestamp and a previous recording timestamp from the target frame and metadata frames corresponding to a previous frame of the target frame, as well as a previous video encoding time corresponding to the previous frame of the target frame; wherein an initial value of the video encoding time is 0; calculating a time difference between the target recording timestamp and the previous recording timestamp; and adding the time difference to the previous video encoding time to obtain the target video encoding time.

[0008] In the above implementation process, the recording and playback method of video multiple data streams provided in the embodiment of the present application can calculate the target video encoding time of the target image frame based on multiple image frames in the video data stream and multiple metadata frames corresponding to these image frames in the original metadata stream; the calculated target video encoding time corresponds to the recording timestamp of the image frame, thereby realizing time synchronization between the image frames in the video data stream and the metadata frames in the original metadata stream, and the relevant information of each image frame can be accurately obtained during the playback operation.

[0009] Optionally, in an embodiment of the present application, the target video encoding time is written into the original metadata stream to obtain a metadata file, including: for each metadata frame in the original metadata stream, each metadata frame is divided into a metadata header and metadata for storage; the video encoding time corresponding to each metadata frame is written into the corresponding metadata header to obtain a metadata file in which the metadata header and metadata are alternately stored.

[0010] In the above implementation process, the video multi-stream data recording and playback method provided by the embodiment of the present application writes the target video encoding time into the original metadata stream, ultimately generating a metadata file containing time information. This also enables binding of image frames with corresponding metadata, allowing accurate acquisition of relevant information for each image frame during playback.

[0011] Optionally, in an embodiment of the present application, based on the video data stream, metadata file and other data streams, recording and playing back the video includes: storing the video data stream as a video file including image frames, and storing other data streams as other data files including other recording timestamps; storing metadata files, video files and other data files to complete the recording of the video.

[0012] In the above-mentioned implementation process, the recording and playback method of multiple video data streams provided by the embodiment of the present application completes the recording of the video by storing the video data stream as a video file including image frames, and storing other data streams as other data files including other recording timestamps. In this process, the original metadata is written into the encoding time, corresponding one-to-one with the image frame, forming a metadata file. In addition, the sensor data and control data are directly saved to the corresponding binary file, ensuring the integrity and consistency of the data. During the playback process, the synchronous playback of multiple data streams is achieved through the synchronization principle of the recording time and the current time flow rate. Even if the stream changes, it will not affect the normal synchronization. In addition, due to the dynamic adaptability, real-time processing capability and correct association with metadata of the solution, it is also allowed to adjust the video stream during the recording process, such as modifying the resolution and format, without affecting the normal recording and playback.

[0013] Optionally, in an embodiment of the present application, the video data stream is stored as a video file including image frames, including: performing image compression, graphic encoding and image multiplexing on the image frames, and storing the video data stream as multiple video files including image frames; and separately storing the metadata files corresponding to the image frames in each video file to establish a video file and metadata file pair.

[0014] In this implementation, multiple image frames within each video file are stored separately along with their corresponding metadata files, ensuring timestamp continuity during playback. This ensures smooth and accurate playback even in the event of unusual circumstances during recording. This facilitates data management and processing when handling unusual situations, and also makes data backup and recovery easier.

[0015] Optionally, in an embodiment of the present application, recording and playing back videos based on video data streams, metadata files and other data streams also includes: reading other recording timestamps in other data files; determining the recording timestamp of the image frame and the metadata corresponding to the image frame in the metadata file based on the video encoding time of the image frame in the video data stream; synchronizing the recording timestamp of the image frame with the image frames, metadata and other data corresponding to other timestamps to play back the video.

[0016] In the above implementation process, the video multi-data stream recording and playback method provided by the embodiment of the present application can accurately obtain the recording timestamps of the image frames in the video data stream and the timestamps in other data streams during the playback process, and match and synchronize them; keep the time between different data streams consistent, so that the video playback process has temporal coherence, and correctly match the image frames with the data frames in other data streams in time sequence, so that the video playback process has data accuracy.

[0017] Optionally, in an embodiment of the present application, the recording timestamp of the image frame and the image frames, metadata and other data corresponding to other timestamps are synchronized for video playback, including: calculating the simulated recording time based on the actual time and the recording start time; when the recording timestamp reaches the simulated recording time, the image frames, metadata and other data corresponding to the recording timestamp are transmitted for video playback.

[0018] In the above implementation, when acquiring metadata, the recording timestamp and encoding time of the image frame are simultaneously obtained. This way, a set of corresponding frames and metadata, along with the recording timestamp, is acquired. By repeating this process, all image streams can be acquired. Furthermore, when reading sensor data frames and control data frames, their recording timestamps can also be directly obtained. When the recording timestamp reaches the analog recording time, the data is transmitted, thus achieving data stream synchronization. This is unaffected even if the rates of the individual streams differ.

[0019] In a second aspect, an embodiment of the present application provides a recording and playback device for multiple video data streams, and the recording and playback device for multiple video data streams includes: a data processing module and a recording and playback module; the data processing module is used to calculate the target video encoding time of the target image frame based on multiple image frames in the video data stream and multiple metadata frames in the original metadata stream corresponding to the multiple image frames respectively; the data processing module is also used to write the target video encoding time into the original metadata stream to obtain a metadata file; the recording and playback module is used to record and play back videos based on the video data stream, the metadata file and other data streams.

[0020] In a third aspect, an embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein program instructions are stored in the memory, and when the processor reads and runs the program instructions, it executes the steps in any of the above implementation methods.

[0021] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer program instructions are stored in the computer-readable storage medium. When the computer program instructions are read and executed by a processor, the steps in any of the above implementation methods are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0023] Figure 1Provided for the embodiment of the present application is a flowchart of a method for recording and replaying multiple video data streams;

[0024] Figure 2 A schematic diagram of data transmission provided in an embodiment of the present application;

[0025] Figure 3 A flowchart for calculating video encoding time provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of the metadata file structure provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of storing multi-stream data provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of the sensor data file structure provided in an embodiment of the present application;

[0029] Figure 7 A schematic diagram of a file pair provided in an embodiment of the present application;

[0030] Figure 8 Flowchart of video playback provided for implementation of this application;

[0031] Figure 9 A schematic diagram of video playback provided in an embodiment of the present application;

[0032] Figure 10 A schematic diagram of calculating the simulated recording time provided in an embodiment of the present application;

[0033] Figure 11 A flowchart for calculating the simulated recording time provided in an embodiment of the present application;

[0034] Figure 12 A schematic diagram of a module for a video multi-stream recording and playback device provided in an embodiment of the present application;

[0035] Figure 13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the drawings. For example, two consecutive boxes may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, may be implemented using a dedicated hardware-based system that performs the specified functions or actions, or may be implemented using a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present invention may be integrated together to form a separate part, or each module may exist separately, or two or more modules may be integrated to form a separate part.

[0037] Video recording and playback refers to the process of capturing video data using a camera or webcam and saving it to a local storage device for later reading, decoding, and display. During recording, video is captured, compressed, and stored in real time. During playback, the video content is read, decoded, and displayed to allow for a replay and review. Video recording and playback are widely used in a variety of fields, including surveillance, creative production, learning, and entertainment.

[0038] The inventors have discovered that the recording and playback functions of existing cameras and webcams can only operate on the video data of a single camera and are unable to recover other sensor data and metadata information transmitted by the camera.

[0039] Furthermore, current technology cannot synchronize data streams from multiple sensors, such as devices that combine standard RGB sensors with specialized sensors like ToF or LiDAR. This makes it impossible to synchronize multiple data streams, integrate multiple data types, or effectively recover sensor data and metadata during recording and playback.

[0040] Therefore, although the recording and playback functions of existing cameras and webcams perform well in capturing and playing back single video data, there are still significant limitations and challenges when facing the demand for multi-sensor data and metadata information.

[0041] Based on this, the present application proposes a method, device, and electronic device for recording and playback of multiple video data streams. This method, during recording, binds the time of each video frame to the recording time in the corresponding metadata; during playback, the corresponding image is found through a frame of metadata; and, utilizing the principle that the recording time and the current time flow rate are the same, synchronized playback of multiple data streams is achieved.

[0042] Please see Figure 1 , Figure 1 The embodiment of the present application provides a flowchart of a method for recording and replaying multiple video data streams; the present application provides a method for recording and replaying multiple video data streams, which can be recorded and replayed by Figure 13 electronic equipment to perform.

[0043] The video multi-stream recording and playback method provided in the embodiment of the present application includes the following steps:

[0044] Step S100: Calculating a target video encoding time of a target image frame according to a plurality of image frames in a video data stream and a plurality of metadata frames in an original metadata stream respectively corresponding to the plurality of image frames.

[0045] In step S100 , it is first necessary to obtain an image frame and a metadata frame corresponding to the image frame, so as to encode the time of a certain image frame in the recorded video to obtain the video encoding time.

[0046] It should be understood that a video is composed of a series of continuously played image frames, usually played at a certain speed (frame rate) to form a dynamic visual effect. The main purpose of the video multi-stream recording and playback method provided in the embodiment of the application is to synchronize the image frames, metadata and other data streams.

[0047] The video data stream in the embodiment of the present application may include RGB data stream and other image data streams, such as infrared (IR) image data stream, thermal imaging data stream, laser radar (LiDAR) data stream, ultrasonic data stream, etc.; wherein the infrared (IR) image data stream corresponds to infrared image frames, the thermal imaging data stream corresponds to thermal image frames, the laser radar (LiDAR) data stream corresponds to point cloud data, the ultrasonic data stream corresponds to ultrasonic image frames, etc.

[0048] Camera metadata is additional information that describes the images or videos captured by the camera, typically including shooting time, shooting location, camera model, exposure parameters (such as aperture, shutter speed, ISO), focal length, white balance settings, focus information, etc. In the embodiment of the present application, stream synchronization is achieved by binding the time of the image frame in the video (video encoding time) and the recording time (the actual recording time recorded in the metadata).

[0049] That is to say, the target video encoding time in the embodiment of the present application is the time when the target image frame is recorded in the video, and this time corresponds to the actual recording time in the metadata corresponding to the target image frame.

[0050] Step S200: writing the target video encoding time into the original metadata stream to obtain a metadata file.

[0051] In step S200, after obtaining the target video encoding time of the target image frame in the recorded video, the target video encoding time is further written into the original metadata stream, thereby generating a metadata file. During playback, the corresponding image frame can be found through the metadata in the metadata file to achieve time alignment.

[0052] Step S300: Record and play back the video based on the video data stream, metadata file and other data streams.

[0053] In the above step S300, based on the obtained video data stream (including several image frames), metadata file (including target video encoding time), and other data streams, the video is recorded, and the video can be played back based on the recorded file.

[0054] It should be noted that the other data streams in the embodiments of the present application can be data detected by various sensors in the camera, that is, sensor data streams, such as temperature sensors, vibration sensors, etc. Since the image may change during the transmission process when the video is recorded as well as with the addition of algorithms, the other data streams in the embodiments of the present application can also be control data streams corresponding to camera control operations.

[0055] Please Figure 1 See the basis of Figure 2 , Figure 2 A data transmission diagram provided for an embodiment of the present application; the camera includes RGB data streams, special image data streams, and data streams of various sensors. The host may have some operations on the camera, corresponding to the control stream; these data streams are stored in the host (can also be stored in the camera). After processing through multiple data streams, when playing the video on the display device, both real-time images transmitted from the camera and processed by the host can be played, and recorded videos previously saved in the host storage space can be played.

[0056] pass Figure 1 and Figure 2As can be seen, the video multi-stream recording and playback method provided in the embodiment of the present application calculates the target video encoding time of the target image frame, writes it into the original metadata stream, and performs recording and playback operations based on the video data stream, metadata file, and other data streams. The video multi-stream recording and playback method provided in the embodiment of the present application achieves precise synchronization of multiple data streams, ensuring time consistency during the multi-stream recording and playback process.

[0057] Please see Figure 3 , Figure 3 A flowchart for calculating video encoding time is provided in an embodiment of the present application. In an optional implementation of the embodiment of the present application, step S100 calculates the target video encoding time of the target image frame based on the multiple image frames in the video data stream and the multiple metadata frames in the original metadata stream that respectively correspond to the multiple image frames. This can be achieved by the following steps:

[0058] Step S110: acquiring a target recording timestamp and a previous recording timestamp from the target frame and metadata frames corresponding to the previous frame of the target frame, as well as a previous video encoding time corresponding to the previous frame of the target frame.

[0059] In step S110, the target recording timestamp and the previous recording timestamp, as well as the previous video encoding time corresponding to the previous frame, are obtained from the metadata frame corresponding to the target frame and the previous frame. The target video encoding time of the target frame can be calculated using the recording timestamp of the target frame, the recording timestamp of the previous frame, and the video encoding time corresponding to the previous frame. It should be noted that the initial value of the video encoding time is 0 because the recorded video time usually starts at 0.

[0060] Step S120: Calculate the time difference between the target recording timestamp and the previous recording timestamp.

[0061] Step S130: Add the time difference to the previous video encoding time to obtain the target video encoding time.

[0062] In steps S120-S130, the time difference between the target recording timestamp and the previous recording timestamp is calculated, i.e., the time interval between the target frame and the previous frame. Furthermore, the calculated time difference is added to the previous video encoding time to obtain the target video encoding time of the target frame. It should be understood that the obtained target video encoding time will be used to mark the temporal position of the target frame in the final recorded video.

[0063] That is to say, the target video encoding time of the target frame can be calculated in the following way: target video encoding time = previous video encoding time + (target recording timestamp - previous recording timestamp); for example, the previous video encoding time is the 1st second, the target recording timestamp is 12:00:02 on day A, and the previous recording timestamp is 12:00:01 on day A, then the target video encoding time = 1 second + 1 second = 2 seconds, that is, the target image frame should be the 2nd second of the recorded video.

[0064] pass Figure 3 It can be seen that the recording and playback method of video multiple data streams provided in the embodiment of the present application can calculate the target video encoding time of the target image frame based on multiple image frames in the video data stream and multiple metadata frames corresponding to these image frames in the original metadata stream; the calculated target video encoding time corresponds to the recording timestamp of the image frame, thereby realizing the time synchronization between the image frames in the video data stream and the metadata frames in the original metadata stream, and the relevant information of each image frame can be accurately obtained during the playback operation.

[0065] Please see Figure 4 , Figure 4 Schematic diagram of the metadata file structure provided in an embodiment of the present application; in an optional implementation of an embodiment of the present application, step S200 writes the target video encoding time into the original metadata stream to obtain a metadata file, which can be achieved in the following manner:

[0066] For each metadata frame in the original metadata stream, each metadata frame is divided into a metadata header and metadata for storage; the video encoding time corresponding to each metadata frame is written into the corresponding metadata header to obtain a metadata file in which the metadata header and metadata are stored alternately.

[0067] like Figure 4 As shown in the figure, each metadata frame in the original metadata stream is divided into a metadata header and a metadata part. The metadata header stores the binding information between the frame and metadata, including metadata size, recording timestamp, video encoding time, and other information, while the metadata part stores the specific metadata content. The target video encoding time corresponding to each metadata frame is written into the corresponding metadata header. The processed metadata header and metadata are stored alternately to form a complete metadata file.

[0068] The metadata file includes a metadata header, such as Figure 4 As shown in Figure 1, the total metadata header stores the necessary information for simulating camera devices and provides basic information related to the entire recording process, such as camera model, shooting time, etc.

[0069] pass Figure 4As can be seen, the video multi-stream recording and playback method provided in the embodiments of the present application implements the writing of the target video encoding time into the original metadata stream, ultimately generating a metadata file containing time information. This also enables the binding of image frames with corresponding metadata, allowing accurate access to relevant information for each image frame during playback.

[0070] Please see Figure 5 , Figure 5 A schematic diagram of saving multi-stream data provided in an embodiment of the present application; in this embodiment of the present application, step S300 records and plays back a video based on a video data stream, metadata file, and other data streams, including the following:

[0071] The video data stream is stored as a video file including image frames, and other data streams are stored as other data files including other recording time stamps; metadata files, video files and other data files are stored to complete the recording of the video.

[0072] Other data files Figure 5 In the example, the sensor data file corresponds to the sensor data, and the control data file corresponds to the control data; according to Figure 5 It can be seen that the image frames corresponding to the metadata one by one calculate the video encoding time and then write the encoding time into the metadata; further, after the image is compressed, encoded, multiplexed and other operations are performed on the image, a video file is formed, and the video file is stored in the storage space.

[0073] The original metadata includes the recording timestamp, camera information, and corresponding image information. After the encoding time is written into the metadata and the encoding time and metadata are bound, it is stored as a metadata file.

[0074] Please refer to Figure 6 , Figure 6 Schematic diagram of the sensor data file structure provided by the embodiment of the present application; the storage of sensor and control data is relatively simple, that is, the timestamp of the data is directly bound together with the size of the data frame and the data and saved in a binary file, such as Figure 6 As shown in the figure, taking the sensor file as an example, each frame of sensor data contains the recording timestamp, the length of the sub-frame data, and the sensor data. The control data file structure is similar to the sensor file structure, and each frame contains the recording timestamp and control data.

[0075] It should be noted that the data stream in the embodiment of the present application is generated from the camera and transmitted to the host, and the host processes the data and saves it locally; in some embodiments, the data stream can be saved in the camera, and image data can be obtained from the camera, and after calculation by the host, it can be put on the display device.

[0076] pass Figure 5 and Figure 6 It can be seen that the recording and playback method of video multiple data streams provided in the embodiment of the present application completes the recording of the video by storing the video data stream as a video file including image frames, and storing other data streams as other data files including other recording timestamps. In this process, the original metadata is written into the encoding time, corresponding one-to-one with the image frame, to form a metadata file. In addition, the sensor data and control data are directly saved to the corresponding binary file, ensuring the integrity and consistency of the data. During the playback process, the synchronous playback of multiple data streams is achieved through the synchronization principle of the recording time and the current time flow rate. Even if the stream changes, it will not affect the normal synchronization. In addition, due to the dynamic adaptability, real-time processing capability and correct association with metadata of the solution, it is also allowed to adjust the video stream during the recording process, such as modifying the resolution and format, without affecting the normal recording and playback.

[0077] Please see Figure 7 , Figure 7 A schematic diagram of a file pair provided in an embodiment of the present application; in an optional embodiment, storing a video data stream as a video file including image frames includes:

[0078] Image compression, graphic encoding and image multiplexing are performed on the image frames, and the video data stream is stored as multiple video files including the image frames; the metadata files corresponding to the image frames in each video file are stored separately to establish a video file and metadata file pair.

[0079] During the recording process, some special circumstances may arise, such as changing the image format or resolution, or the saved file being too large. These circumstances may make it impossible to continue saving the image in the original video file. Therefore, it is necessary to create a new video file and metadata file pair to achieve uninterrupted storage of video data and ensure the continuity of timestamps during playback.

[0080] It should be noted that since the sensor data and control data files have a relatively small amount of data and a fixed format, similar special cases will not occur, and therefore the same processing does not need to be performed.

[0081] Separately storing multiple frames and their corresponding metadata within each video file ensures timestamp consistency during playback. This ensures smooth and accurate playback even in the event of a recording error. This makes data management and processing more convenient when dealing with unusual situations, while also making data backup and recovery easier.

[0082] Please see Figure 8 , Figure 8 Flowchart of video playback provided for implementation of this application; please Figure 8See the basis of Figure 9 , Figure 9 Schematic diagram of video playback provided in an embodiment of the present application; in an optional implementation of an embodiment of the present application, step S300 records and plays back a video based on a video data stream, metadata file, and other data streams, which can be achieved by the following steps:

[0083] Step S310: Read other recording timestamps in other data files.

[0084] Step S320: determining the recording timestamp of the image frame and the metadata corresponding to the image frame in the metadata file according to the video encoding time of the image frame in the video data stream.

[0085] Step S330: Synchronize the recording timestamp of the image frame with the image frames, metadata, and other data corresponding to other timestamps to play back the video.

[0086] In steps S310-S330, the recording timestamps of other data files are first read. These timestamps correspond to data frames in other data streams and are used for time synchronization with the image frames in the video data stream. Next, based on the video encoding time of the image frame in the video data stream, the recording timestamp and corresponding metadata of the image frame are determined from the metadata file. The information in the metadata file allows the recording timestamp and related metadata of each image frame to be obtained, providing the necessary reference for subsequent time synchronization. The recording timestamps of the image frames are then synchronized with the other timestamps for video playback.

[0087] That is, the image frames, metadata in the video data stream and data frames in other data streams are matched and synchronized according to their timestamps to achieve the playback process of the entire video.

[0088] pass Figure 8 and Figure 9 It can be seen that the video multi-data stream recording and playback method provided in the embodiment of the present application can accurately obtain the recording timestamps of the image frames in the video data stream and the timestamps in other data streams during the playback process, and match and synchronize them; make the time between different data streams consistent, so that the video playback process has temporal coherence, and the image frames and the data frames in other data streams are correctly matched in time sequence, so that the video playback process has data accuracy.

[0089] Please see Figure 10 , Figure 10 This is a schematic diagram of the simulation recording time calculation provided in the embodiment of the present application; please Figure 10 See the basis of Figure 11 , Figure 11A flowchart for calculating the simulated recording time provided in an embodiment of the present application; in an optional implementation of an embodiment of the present application, step S330 synchronizes the recording timestamp of an image frame with image frames, metadata, and other data corresponding to other timestamps for video playback, which may include the following steps:

[0090] Step S331: Calculate the simulated recording time according to the actual time and the recording start time.

[0091] In step S331, the difference between the actual time (i.e., the current actual time) and the recording start time is calculated to obtain the time interval. Furthermore, the simulated recording time is obtained by subtracting the time interval from the actual time. By continuously updating the actual time and substituting it into the formula, the simulated recording time corresponding to the current time can be obtained.

[0092] Step S332: When the recording timestamp reaches the simulated recording time, the image frame, metadata and other data corresponding to the recording timestamp are transmitted for video playback.

[0093] When implementing data transmission control, a controller can be used to manage data transmission. The controller controls data transmission according to changes in the analog recording time. Specifically:

[0094] When the video needs to be paused, simply pause the update of the simulated recording time and synchronize the time interval based on the difference between the current time and the recording start time. This allows the video to be paused. When the video resumes playing, the simulated recording time is resumed.

[0095] When the video needs to jump, the difference between the jump time and the simulated recording time is calculated and reflected in the time interval. Furthermore, the file reading position of each data stream is modified to the corresponding playback position to realize the video jump function.

[0096] When acquiring metadata, the recording timestamp and encoding time of the image frame can be obtained simultaneously. This way, a set of corresponding frames and metadata is acquired, along with the recording timestamp. This process is repeated to acquire all image streams. Simultaneously, when reading sensor data and control data frames, their recording timestamps can also be directly obtained. When the recording timestamp reaches the analog recording time, the data is transmitted, thus achieving data stream synchronization. This is unaffected even if the streams have different rates.

[0097] In an optional embodiment, during playback, data can be sent at fixed time intervals to ensure data continuity and accuracy. By sending data at regular intervals, the synchronous playback of the data stream can be achieved regardless of how the data stream rate changes.

[0098] For slower data streams, data padding can be performed to bring them up to the same frame rate as the other data streams. Padding can be done by repeating the previous frame's data or inserting blank frames to keep the frame rates of the various data streams consistent.

[0099] In addition, during playback, data may become out of sync due to different data stream rates. To handle this, a buffer can be used to cache and adjust data to ensure that the data of each stream remains synchronized during playback.

[0100] Please see Figure 12 , Figure 12 This is a module diagram of a video multi-stream recording and playback device provided in an embodiment of the present application; the present application also provides a video multi-stream recording and playback device, and the video multi-stream recording and playback device 100 includes a data processing module 110 and a recording and playback module 120.

[0101] The data processing module 110 is configured to calculate a target video encoding time of a target image frame according to multiple image frames in the video data stream and multiple metadata frames in the original metadata stream corresponding to the multiple image frames.

[0102] The data processing module 110 is further configured to write the target video encoding time into the original metadata stream to obtain a metadata file.

[0103] The recording and playback module 120 is used to record and play back videos based on video data streams, metadata files, and other data streams.

[0104] In an optional embodiment, in the process of calculating the target video encoding time of the target image frame based on multiple image frames in the video data stream and multiple metadata frames in the original metadata stream that respectively correspond to the multiple image frames, the data processing module 110 is used to: obtain the target recording timestamp and the previous recording timestamp from the metadata frames corresponding to the target frame and the previous frame of the target frame, as well as the previous video encoding time corresponding to the previous frame of the target frame; wherein the initial value of the video encoding time is 0; calculate the time difference between the target recording timestamp and the previous recording timestamp; and add the time difference to the previous video encoding time to obtain the target video encoding time.

[0105] In an optional embodiment, in the process of writing the target video encoding time into the original metadata stream to obtain the metadata file, the data processing module 110 is used to: for each metadata frame in the original metadata stream, divide each metadata frame into a metadata header and metadata for storage; write the video encoding time corresponding to each metadata frame into the corresponding metadata header to obtain a metadata file in which the metadata header and metadata are alternately stored.

[0106] In an optional embodiment, during the process of recording and playing back a video based on a video data stream, metadata files, and other data streams, the recording and playback module 120 is used to: store the video data stream as a video file including image frames, and store other data streams as other data files including other recording timestamps; store metadata files, video files, and other data files to complete the recording of the video.

[0107] In an optional embodiment, in the process of storing the video data stream as a video file including image frames, the recording and playback module 120 is used to: perform image compression, graphic encoding and image multiplexing on the image frames, and store the video data stream as multiple video files including image frames; and separately store the metadata files corresponding to the image frames in each video file to establish a video file and metadata file pair.

[0108] In an optional embodiment, when recording and playing back videos based on video data streams, metadata files and other data streams, the recording and playback module 120 is used to: read other recording timestamps in other data files; determine the recording timestamp of the image frame and the metadata corresponding to the image frame in the metadata file based on the video encoding time of the image frame in the video data stream; synchronize the recording timestamp of the image frame with the image frames, metadata and other data corresponding to other timestamps to play back the video.

[0109] In an optional embodiment, in the process of synchronizing the recording timestamp of the image frame and the image frames, metadata and other data corresponding to other timestamps for video playback, the recording and playback module 120 is used to: calculate the simulated recording time based on the actual time and the recording start time; when the recording timestamp reaches the simulated recording time, the image frames, metadata and other data corresponding to the recording timestamp are transmitted for video playback.

[0110] See Figure 13 , Figure 13 The electronic device 200 provided in the embodiment of the present application includes a processor 201 and a memory 202 , wherein the memory 202 stores machine-readable instructions executable by the processor 201 , and when the machine-readable instructions are executed by the processor 201 , the method described above is performed.

[0111] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are read and executed by a processor, the steps in any implementation method of the above-mentioned video multi-stream recording and playback method are executed.

[0112] The computer-readable storage medium can be a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other media that can store program code.

[0113] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the elements.

[0114] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for recording and replaying multiple video data streams, characterized in that: The recording and playback method comprises: Obtaining a target recording timestamp and a previous recording timestamp from the target image frame and the metadata frame corresponding to the previous frame of the target image frame, as well as a previous video encoding time corresponding to the previous frame of the target image frame, respectively; wherein the initial value of the video encoding time is 0; calculating a time difference between the target recording timestamp and the previous recording timestamp; and adding the time difference to the previous video encoding time to obtain a target video encoding time; Writing the target video encoding time into the original metadata stream to obtain a metadata file; Record and play back videos based on video data streams, metadata files, and other data streams; The recording and playback of the video based on the video data stream, metadata file and other data streams includes: storing the other data stream as another data file including another recording time stamp; Reading the other recording timestamps in the other data files; determining the recording timestamp of the image frame and the metadata corresponding to the image frame in the metadata file according to the video encoding time of the image frame in the video data stream; The simulated recording time is calculated according to the actual time and the recording start time; when the recording timestamp reaches the simulated recording time, the image frame, metadata and other data corresponding to the recording timestamp are transmitted for video playback.

2. The method according to claim 1, characterized in that Writing the target video encoding time into the original metadata stream to obtain a metadata file includes: For each metadata frame in the original metadata stream, dividing the metadata frame into a metadata header and metadata for storage; The video encoding time corresponding to each metadata frame is written into the corresponding metadata header to obtain the metadata file in which the metadata header and the metadata are alternately stored.

3. The method according to claim 1, characterized in that Recording and playing back videos based on the video data stream, metadata file and other data streams also includes: storing the video data stream as a video file including image frames; The metadata file, video file and other data files are stored to complete the recording of the video.

4. The method according to claim 3, characterized in that The step of storing the video data stream as a video file including image frames comprises: Performing image compression, graphic coding, and image multiplexing on the image frames, and storing the video data stream as a plurality of video files including the image frames; The metadata file corresponding to each image frame in the video file is stored separately to create a video file and metadata file pair.

5. A video multi-stream recording and playback device, characterized in that: The video multi-stream recording and playback device includes: a data processing module and a recording and playback module; The data processing module is configured to obtain a target recording timestamp and a previous recording timestamp from the target image frame and the metadata frame corresponding to the previous frame of the target image frame, as well as a previous video encoding time corresponding to the previous frame of the target image frame; wherein the initial value of the video encoding time is 0; calculate a time difference between the target recording timestamp and the previous recording timestamp; and add the time difference to the previous video encoding time to obtain the target video encoding time; The data processing module is further configured to write the target video encoding time into the original metadata stream to obtain a metadata file; The recording and playback module is used to record and play back videos based on video data streams, metadata files and other data streams; wherein, the recording and playback of videos based on the video data streams, metadata files and other data streams includes: storing the other data streams as other data files including other recording timestamps; reading the other recording timestamps in the other data files; determining the recording timestamp of the image frame and the metadata corresponding to the image frame in the metadata file according to the video encoding time of the image frame in the video data stream; calculating the simulated recording time according to the actual time and the recording start time; when the recording timestamp reaches the simulated recording time, the image frame, metadata and other data corresponding to the recording timestamp are transmitted for video playback.

6. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein program instructions are stored in the memory, and when the processor runs the program instructions, the steps of the method according to any one of claims 1 to 4 are executed.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 4 are executed.

Images