Video synchronization method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a video synchronization method and device, equipment and a storage medium. The method comprises the following steps: acquiring video data acquired by at least two channels, adding timestamps to video frames in the video data, determining the video frames corresponding to the at least two channels as synchronous frames based on the timestamps, encoding the video frames, and then encoding the video frames based on the channels corresponding to the video data. And carrying out interleaving packaging processing on the coded video frame, and uploading the packaged video data. According to the technical scheme provided by the embodiment of the invention, the problem that the playing pictures are easy to be asynchronous when the pictures corresponding to the multi-view camera are browsed at the terminal in the related technology is solved, the synchronism of the video data of different channels is ensured to the greatest extent, and the finally uploaded video data are enabled to keep synchronous in each frame of picture in the synchronous playing process; and the user experience is obviously improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a video synchronization method, device, equipment and storage medium. Background Art

[0002] With the development of sensor technology, in the field of camera-based scene monitoring, the solution of monitoring through multi-cameras is increasingly used. Multi-cameras can simultaneously capture images from multiple angles of the same scene, and users can browse images from multiple angles on the terminal at the same time, giving users a more comprehensive understanding of the scene, thereby making the monitoring of the scene more comprehensive and reliable.

[0003] In the related technology, during the transmission of video data, frame loss is easy to occur due to network instability, and the video data corresponding to different cameras are easy to arrive at different times during the transmission process, which causes the corresponding pictures of the video data received by the terminal to be out of sync. When the user browses the video data shot by multiple cameras on the terminal, if the field of view angles of different pictures overlap and the playback pictures are out of sync, it will seriously affect the user's perception and judgment of the scene information, resulting in a poor user experience. Summary of the invention

[0004] The embodiments of the present disclosure provide a video synchronization method, apparatus, device and storage medium to solve the problem in the related art that corresponding images of multiple cameras are easily out of sync when viewed by a terminal.

[0005] In a first aspect, an embodiment of the present disclosure provides a video synchronization method, the video synchronization method comprising:

[0006] Obtain video data collected by at least two channels, and add time stamps to video frames in the video data, wherein each channel corresponds to a video sensor;

[0007] Based on the timestamp, determine that the video frames corresponding to at least two channels are synchronization frames, and encode the video frames, where the synchronization frames are used to represent the video frames collected by the video sensors corresponding to the at least two channels at the same time;

[0008] Based on the channel corresponding to the video data, the encoded video frames are interleaved and packaged, and the packaged video data is uploaded.

[0009] Optionally, before obtaining video data collected by at least two channels and adding timestamps to video frames in the video data, the method further includes: setting the working modes of all sensors corresponding to the at least two channels to slave modes; or determining one of the sensors corresponding to the at least two channels as a master sensor; setting the working mode of the master sensor to a master control mode, and setting the working modes of other sensors except the master sensor to slave modes.

[0010] Optionally, based on the timestamp, the video frames corresponding to at least two channels are determined to be synchronization frames, and the video frames are encoded, including: calculating the time difference between the timestamps of the video frames corresponding to different channels; if the time difference is less than the set time difference threshold, determining that the video frames corresponding to at least two channels are synchronization frames; and encoding the video frames.

[0011] Optionally, before encoding the video frame, it also includes: if the time difference is greater than a set time difference threshold, determining the video frame with the smallest timestamp as the target video frame, and determining the channel corresponding to the target video frame as the target channel; removing the target video frame in the target channel; obtaining the video frame in the target channel that is located after the target video frame, and calculating the time difference between the video frame and the video frames in other channels other than the target channel; if the time difference is greater than the set time difference threshold, repeating the process of removing the target video frame, obtaining the video frame in the target channel and calculating the time difference until the time difference is less than the set time difference threshold; determining that the video frames corresponding to at least two channels are synchronized frames.

[0012] Optionally, encoding the video frames includes: synchronously configuring an encoder for processing the video frames; and encoding the video frames based on the configured encoder.

[0013] Optionally, the video frames in the video data include key frames and non-key frames, and the key frame interval length corresponding to the video data is used to indicate the number of non-key frames between adjacent key frames in the same video data; the encoder used to process the video frames is synchronously configured, including: determining that the encoder used to encode the video frames corresponding to each channel is a synchronous encoder, and the synchronous encoder is used to indicate an encoder that starts processing and / or stops processing at the same time; setting the frame rate of all encoders to the same value, and setting the key frame interval length of all encoders to the same value.

[0014] Optionally, based on the channel corresponding to the video data, the encoded video frames are interleaved and packaged, and the packaged video data is uploaded, including: single-frame packaging of the encoded video frames; alternately storing the packaged video frames corresponding to different channels in turn to a set storage position, wherein any two adjacent video frames in the set storage position belong to different channels; and uploading all the packaged video frames corresponding to the set storage position.

[0015] Optionally, based on the timestamp, determining that the video frames corresponding to at least two channels are synchronized frames, and before encoding the video frames, the method further includes: preprocessing the video frames corresponding to the at least two channels, wherein the preprocessing includes resolution adjustment processing.

[0016] In a second aspect, an embodiment of the present disclosure provides a video synchronization device, the video synchronization device comprising:

[0017] An acquisition module, used to acquire video data collected by at least two channels and add time stamps to video frames in the video data, wherein each channel corresponds to a video sensor;

[0018] A determination module, used to determine, based on the timestamp, that the video frames corresponding to at least two channels are synchronization frames, and to encode the video frames, wherein the synchronization frames are used to represent the video frames collected by the video sensors corresponding to at least two channels at the same time;

[0019] The processing module is used to interleave and encapsulate the encoded video frames based on the channels corresponding to the video data, and upload the encapsulated video data.

[0020] Optionally, the acquisition module is also used to acquire video data collected by at least two channels, and before adding timestamps to video frames in the video data, set the working modes of all sensors corresponding to at least two channels to slave modes; or, determine one of the sensors corresponding to at least two channels as a main sensor; set the working mode of the main sensor to the master control mode, and set the working modes of other sensors except the main sensor to the slave modes.

[0021] Optionally, the determination module is specifically used to calculate the time difference between the timestamps of video frames corresponding to different channels; if the time difference is less than a set time difference threshold, determine that at least two video frames corresponding to the channels are synchronized frames; and encode the video frames.

[0022] Optionally, the determination module is also used to, before encoding the video frame, if the time difference is greater than a set time difference threshold, determine the video frame with the smallest timestamp as the target video frame, and determine the channel corresponding to the target video frame as the target channel; remove the target video frame in the target channel; obtain the video frame in the target channel that is located after the target video frame, and calculate the time difference between the video frame and the video frames in other channels other than the target channel; if the time difference is greater than the set time difference threshold, repeat the process of removing the target video frame, obtaining the video frame in the target channel and calculating the time difference until the time difference is less than the set time difference threshold; determine that the video frames corresponding to at least two channels are synchronized frames.

[0023] Optionally, the determination module is specifically used to synchronously configure an encoder for processing video frames; and encode the video frames based on the configured encoder.

[0024] Optionally, the determination module is specifically used to, if the video frames in the video data include key frames and non-key frames, the key frame interval length corresponding to the video data is used to indicate the number of non-key frames spaced between adjacent key frames in the same video data, determine that the encoder used to encode the video frames corresponding to each channel is a synchronous encoder, and the synchronous encoder is used to indicate an encoder that starts processing and / or stops processing at the same time; set the frame rates of all encoders to the same value, and set the key frame interval lengths of all encoders to the same value.

[0025] Optionally, the processing module is specifically used to perform single-frame encapsulation processing on the video frames after encoding; alternately store the video frames corresponding to different channels after encapsulation processing in a set storage position in turn, wherein any two adjacent video frames in the set storage position belong to different channels; and upload all the encapsulated video frames corresponding to the set storage position.

[0026] Optionally, the determination module is also used to determine, based on the timestamp, that the video frames corresponding to at least two channels are synchronized frames, and pre-process the video frames corresponding to at least two channels before encoding the video frames, wherein the pre-processing includes resolution adjustment processing.

[0027] In a third aspect, an embodiment of the present disclosure further provides a control device, the control device comprising:

[0028] at least one processor;

[0029] and a memory communicatively coupled to the at least one processor;

[0030] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the control device to perform the video synchronization method as described in the first aspect of the present disclosure.

[0031] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the video synchronization method of the first aspect of the present disclosure.

[0032] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the video synchronization method as in the first aspect of the present disclosure.

[0033] The video synchronization method, device, equipment and storage medium provided by the embodiments of the present disclosure obtain video data collected by at least two channels, add timestamps to video frames in the video data, and then determine the video frames corresponding to at least two channels as synchronization frames based on the timestamps, and encode the video frames, and then interleave and encapsulate the encoded video frames based on the channels corresponding to the video data, and upload the encapsulated video data. In this way, by taking different corresponding measures in multiple links such as video data collection, encoding, encapsulation and uploading, the synchronization of video data of different channels can be ensured to the maximum extent, so that each frame of the finally uploaded video data can be synchronized during the synchronous playback process, significantly improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0035] Figure 1 An application scenario diagram of the video synchronization method provided by the embodiment of the present disclosure;

[0036] Figure 2 A flowchart of a video synchronization method provided by an embodiment of the present disclosure;

[0037] Figure 3a A flowchart of a video synchronization method provided by yet another embodiment of the present disclosure;

[0038] Figure 3b for Figure 3a A flow chart of another method for implementing a synchronization frame provided in the illustrated embodiment;

[0039] Figure 3c for Figure 3a A flow chart of a method for encoding a video frame provided by the illustrated embodiment;

[0040] Figure 3d for Figure 3a A flow chart of a method for synchronously configuring an encoder provided by the illustrated embodiment;

[0041] Figure 4 A schematic diagram of the structure of a video synchronization device provided by another embodiment of the present disclosure;

[0042] Figure 5 A schematic diagram of the structure of a control device provided in one embodiment of the present disclosure.

[0043] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0044] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0045] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0046] The following is a description of the concepts involved in the embodiments of the present disclosure:

[0047] Multi-camera: A monitoring device that contains multiple cameras (or camera sensors) at the same time. Each camera will perform image acquisition at the same time. When users watch the image data output by the multi-camera, they usually ensure that the images acquired by different cameras are played synchronously to achieve multi-angle image acquisition and monitoring.

[0048] With the development of sensor technology, in the field of camera-based scene monitoring, the solution of monitoring through multi-cameras is increasingly used. Multi-cameras can simultaneously capture images from multiple angles of the same scene, and users can browse images from multiple angles on the terminal at the same time, giving users a more comprehensive understanding of the scene, thereby making the monitoring of the scene more comprehensive and reliable.

[0049] In the related art, during the collection and transmission of video data, frame loss is easy to occur due to the instability of signal transmission and communication network, and the video data corresponding to different cameras are easy to arrive at different times during the transmission process (for example, the video data corresponding to different cameras are sent to the cloud server independently, and it is easy for one stream to arrive first and the other to arrive later, and the cloud server will integrate the video data according to the time of receiving the video data), which will cause the corresponding pictures of the video data received by the terminal to be out of sync. When the user browses the video data taken by multiple cameras on the terminal, the field of view of different pictures overlaps and the playback pictures are out of sync, which will seriously affect the user's perception and judgment of the scene information and lead to a poor user experience.

[0050] In order to solve this problem, the embodiment of the present disclosure provides a video synchronization method, which takes multiple measures in the process of video data acquisition, encoding, and packaging to ensure the synchronization of image data multiple times, thereby fully ensuring the synchronization of uploaded video data to ensure the user experience.

[0051] The application scenarios of the embodiments of the present disclosure are explained below:

[0052] Figure 1 FIG. 1 is an application scenario diagram of the video synchronization method provided by the embodiment of the present disclosure. Figure 1 As shown, during the plug-in development process, the sensor 101 in the multi-eye camera 100 transmits the collected video data to the processor 110 for preprocessing, encoding, packaging and other processing, and then uploads it to the cloud server 120. The synchronization of the video data is ensured through the processing in the processor 110.

[0053] It should be noted that Figure 1 In the illustrated scenario, the multi-camera, sensor, processor, and cloud server are illustrated by only one or a specific number, but the present disclosure is not limited to this. That is, the number of multi-camera, sensor, processor, and cloud server can be arbitrary.

[0054] The video synchronization method provided by the present disclosure is described in detail below through specific embodiments.

[0055] Figure 2 The following is a flow chart of a video synchronization method provided by an embodiment of the present disclosure. Figure 2 As shown, the video synchronization method provided in this embodiment includes the following steps:

[0056] Step S201: Obtain video data collected by at least two channels, and add time stamps to video frames in the video data.

[0057] Among them, each channel corresponds to a video sensor.

[0058] Specifically, during the image acquisition phase, it is necessary to ensure that the video data corresponding to each channel is synchronously acquired data, which can be achieved by simultaneously starting the video sensors of all channels to perform acquisition operations.

[0059] After the video data is collected, it is necessary to add a timestamp to the video data of each channel to ensure the synchronization between videos by comparing the timestamps of the video data of different channels.

[0060] Step S202: based on the timestamp, determine that the video frames corresponding to at least two channels are synchronized frames, and perform encoding processing on the video frames.

[0061] The synchronization frame is used to represent the video frames collected by the video sensors corresponding to at least two channels at the same time.

[0062] Specifically, to ensure the synchronization of video data before encoding, it is necessary to check the video data corresponding to different channels frame by frame (actually, sampling inspection can also be performed at a set distance). By comparing the timestamps of video frames with the same sequence number of the video data, or comparing the ordinals of the same timestamps, if the timestamps of video frames with the same sequence number are the same, it can be confirmed that these video frames are synchronized frames. The ordinal number of a video frame in the video data is the number of video frames from the first frame of the video data to the current video frame being compared.

[0063] For example, in video data that captures video frames at a rate of 12 frames per second, the 25th frame, which is the first video frame captured at the 3rd second, should have a timestamp of 0:00:03.00 (i.e., 3.00 seconds from the start of the video). If the timestamp of the 25th frame of video data from another channel is also 0:00:03.00, then these two video frames are synchronized frames. If their timestamp is 0:00:02.50, then these two video frames are not synchronized frames.

[0064] By checking whether the video frames in the video data are synchronous frames frame by frame or sampling, it is ensured that the video data of different channels are synchronous frames from the beginning to the end, thereby effectively ensuring the synchronization of the collected video data, so as to ensure the synchronization of the encoded video data.

[0065] Step S203: interleave and encapsulate the encoded video frames based on the channels corresponding to the video data, and upload the encapsulated video data.

[0066] Specifically, since the encoded video data may be lost in the upload process due to communication network reasons, and the video data of different channels may be uploaded to the cloud server at a time that is not synchronized, the video data of different channels need to be packaged together in an interleaved manner during the packaging process, and then all the video data need to be uploaded to the cloud server as a whole, rather than uploaded separately.

[0067] The interleaved encapsulation is to encapsulate the video frames with the same sequence number corresponding to different channels into storage positions alternately according to the sequence number of the video frames. Thus, the video data of each storage position contains the video frames with the same sequence number of all channels.

[0068] Therefore, even if data loss occurs during the upload process, it is not a single frame loss in a certain channel, but the video data of all channels lose the same corresponding number of frames of video data, so that the video can still meet the synchronization when it is played, and effectively avoid the time asynchrony that may occur when different channels upload video data to the cloud server separately, thereby maximizing the synchronization characteristics of the video data.

[0069] The video synchronization method provided by the embodiment of the present disclosure obtains video data collected by at least two channels, adds timestamps to video frames in the video data, and then determines the video frames corresponding to at least two channels as synchronization frames based on the timestamps, and encodes the video frames, and then interleaves and encapsulates the encoded video frames based on the channels corresponding to the video data, and uploads the encapsulated video data. In this way, by taking different corresponding measures in multiple links such as video data collection, encoding, encapsulation and uploading, the synchronization of video data of different channels can be ensured to the maximum extent, so that each frame of the finally uploaded video data can be synchronized during the synchronous playback process, significantly improving the user experience.

[0070] Figure 3a The following is a flow chart of a video synchronization method provided by an embodiment of the present disclosure. Figure 3a As shown, the video synchronization method provided in this embodiment includes the following steps:

[0071] Step S301: setting the working modes of sensors corresponding to at least two channels to slave modes.

[0072] Specifically, this embodiment is used to further illustrate the video synchronization method in the entire process from video data acquisition to uploading based on the above embodiments.

[0073] In order to ensure the synchronization of video data in the acquisition phase, it is necessary to enable the video sensors corresponding to different channels to start data acquisition synchronously according to the same instruction.

[0074] At this time, the working mode of each sensor can be set to slave mode, that is, to accept instructions from a single controller or control system to start data collection. Therefore, by sending a single start command to the multi-eye camera, the sensors of all channels can start data collection at the same time, thereby effectively ensuring the synchronization of the collected video data.

[0075] Step S302: determine one of the sensors corresponding to at least two channels as a main sensor.

[0076] Specifically, in another case, one sensor can be selected from multiple sensors as the main sensor, and the other sensors can be set as slave sensors, so that when a start instruction is sent to the main sensor, the main sensor will automatically distribute the start instruction to the other slave sensors, so that each sensor starts data collection synchronously. This can also ensure the synchronization of the collected video data.

[0077] Step S303: setting the working mode of the main sensor to the master mode, and setting the working modes of other sensors except the main sensor to the slave mode.

[0078] Specifically, when the master sensor and slave sensors are configured, the working mode of the master sensor needs to be set to the master mode, and the working modes of other sensors need to be set to the slave mode, so as to ensure that the master sensor can automatically distribute the start command to other slave sensors in the slave mode when receiving the start command, so as to ensure the realization of synchronous acquisition of video data, and then ensure that the video frame images collected by each sensor at the same time are the same, so as to achieve frame synchronization.

[0079] Step S301 is an optional step parallel to step S302 to step S303. Those skilled in the art may select corresponding steps to perform according to actual conditions, and no limitation is made here.

[0080] Step S304: Obtain video data collected by at least two channels, and add time stamps to video frames in the video data.

[0081] Among them, each channel corresponds to a video sensor.

[0082] Specifically, after the video sensor collects data, a timestamp is added to each frame of the video data for subsequent processing.

[0083] In some embodiments, after adding a timestamp to the video frame, the video frames corresponding to at least two channels are usually preprocessed, and the preprocessing includes resolution adjustment processing.

[0084] Specifically, since different sensors of a multi-view camera may use components of different models, the captured video data may have differences in resolution, brightness, etc. Therefore, after adding the timestamp, the video data also needs to be processed frame by frame (i.e., preprocessing) by adjusting the resolution, brightness, and video cropping to ensure that the video frames corresponding to different channels remain unified in terms of resolution, brightness, and size, so that users can view them and improve the user experience.

[0085] Step S305, calculating the time difference between the timestamps of the video frames corresponding to different channels.

[0086] Specifically, since frame loss may occur during the preprocessing process, the preprocessed video data needs to be synchronized at the video frame level (even if no preprocessing is performed, the collected video data needs to be synchronized at the frame level).

[0087] Synchronization processing is used to ensure synchronization between video data corresponding to different channels. If it is determined that different video data are in a synchronized state, no processing is required. If it is determined that there are differences in video frames between different video data (i.e., not synchronized), the video data needs to be further processed.

[0088] In determining whether different video data are in a synchronized state, it is first necessary to select video frames for comparison from the video data, which can be frame-by-frame comparison or sampled video frames for comparison. The comparison method is to first calculate the time difference between the timestamps of these video frames to determine whether they are synchronized frames based on the time difference.

[0089] In some embodiments, considering that video frames in video data exist the difference between key frames (referring to video frames that contain complete picture data, so the data volume is large) and predicted frames (referring to video frames that are close to key frames and whose pictures only have some differences relative to key frames, so the predicted frames usually only have difference data relative to the previous frame, so the data volume is small, and by increasing the predicted frames, the compression rate of the video data can be improved), therefore, when sampling and selecting video frames for comparison, key frames can be preferably used for comparison.

[0090] Step S306: if the time difference is less than the set time difference threshold, it is determined that the video frames corresponding to at least two channels are synchronized frames.

[0091] Specifically, when the time difference between the video frames corresponding to different channels for comparison is less than the set time difference threshold, the video frames for comparison can be determined to be synchronized frames, and then the next group of video frames is selected for comparison.

[0092] The time difference threshold can be selected as 1 / 2 of the time difference between the timestamps of two adjacent video frames in any video data, that is, 1 / 2 frame interval time. If the time difference between video frames corresponding to different channels is not 0 and is less than this time difference threshold, this time difference can be considered as the time difference caused by the signal transmission process, but it does not affect the fact that these video frames belong to synchronous frames.

[0093] Furthermore, if Figure 3b As shown, it is a flowchart of another implementation method of the synchronization frame, which includes the following steps:

[0094] Step S3061: If the time difference is greater than the set time difference threshold, the video frame with the smallest timestamp is determined as the target video frame, and the channel corresponding to the target video frame is determined as the target channel.

[0095] Specifically, if the time difference is greater than the set time difference threshold, it means that these video frames do not belong to the synchronization frame. In order to ensure the synchronization of the video data, it is necessary to process the video data corresponding to these video frames.

[0096] The specific processing method is to select the video frame with the smallest timestamp from the compared video frames, and determine the corresponding channel as the target channel, so as to achieve synchronization of the video data by processing the video data in the target channel.

[0097] Step S3062, removing the target video frame in the target channel.

[0098] Specifically, since the video data corresponding to the video frame with the largest timestamp in a group of video frames for comparison can be considered as video data with lost frames (because frame loss causes the number of frames in the video data to decrease, video frames with larger video frame numbers and larger timestamps will appear at positions with smaller video frame numbers, and video frames with timestamps larger than those corresponding to other channels will appear everywhere), and the lost video frames cannot be recovered. Therefore, it is necessary to discard part of the video frames in the target channel until they are synchronized with the video data corresponding to the video frame with the largest timestamp.

[0099] The specific implementation method is to remove the target video frame in the target channel video data, that is, the video frame currently used for comparison.

[0100] Step S3063, obtaining a video frame in the target channel that is located after the target video frame, and calculating a time difference between the video frame and video frames in other channels except the target channel.

[0101] Specifically, after removing a video frame, the video frame for comparison in the target channel can be determined again, that is, a video frame after the removed target video frame, and the time difference between the timestamps of the determined video frame and the video frames for comparison in other channels can be calculated.

[0102] Step S3064, if the time difference is greater than the set time difference threshold, repeat the process of removing the target video frame, obtaining the video frame in the target channel and calculating the time difference until the time difference is less than the set time difference threshold.

[0103] Specifically, if the time difference calculated in the previous step is still greater than the set time difference threshold, the video frame is removed again. The process of removing a video frame and calculating the time difference is repeated until the calculated time difference is less than the set time difference threshold. At this time, the video frame of the target channel is synchronized with the video frames of other channels for comparison.

[0104] Step S3065: Determine that video frames corresponding to at least two channels are synchronized frames.

[0105] Specifically, in a similar case, if there are multiple video frames for comparison whose time difference values ​​are all greater than the set time difference threshold, the video frame with the largest timestamp can be used as the standard, and each other channel can be used as the target channel in turn to perform the process of extracting the video frames until the video frames of all other channels are synchronized frames with the video frame with the largest timestamp.

[0106] Repeating the process of determining whether all video frames used for sampling comparison are synchronous frames and processing them can effectively ensure the synchronization of video frames used for encoding processing.

[0107] Step S307: encode the video frame.

[0108] Specifically, after the synchronization of the video frames is completed, the video frames need to be encoded to compress the video data to facilitate subsequent packaging and uploading.

[0109] Furthermore, if Figure 3c As shown, it is a flow chart of a method for encoding a video frame, which includes the following steps:

[0110] Step S3071, synchronously configuring an encoder for processing video frames.

[0111] Specifically, the tool for encoding video data is called an encoder, which encodes the video data according to the configured parameters. For multi-channel video data, a separate encoder is usually assigned to each channel for encoding to ensure data processing efficiency. Therefore, in order to ensure the synchronization of video data corresponding to different channels, the encoder needs to be synchronized first.

[0112] In some embodiments, considering that the video frames in the video data include key frames and non-key frames, the key frame interval length corresponding to the video data is used to represent the number of non-key frames between adjacent key frames in the same video data; in this case, Figure 3d As shown, it is a flow chart of a method for synchronous configuration of an encoder, which includes the following steps:

[0113] Step A1: determine that the encoder used for encoding the video frame corresponding to each channel is a synchronous encoder.

[0114] The synchronous encoder is used to refer to an encoder that starts processing and / or stops processing at the same time.

[0115] Specifically, it is first necessary to ensure that each encoder can be started and stopped synchronously through a single command to ensure synchronization of video data processing corresponding to each channel.

[0116] Step A2, setting the frame rates of all encoders to the same value, and setting the key frame interval lengths of all encoders to the same value.

[0117] Specifically, secondly, it is also necessary to ensure that the configuration parameters corresponding to different encoders are the same, that is, the format, frame rate, key frame interval length, etc. of the encoders are all the same values.

[0118] Step S3072: Encode the video frame based on the configured encoder.

[0119] Specifically, the encoder with the above synchronization configuration can ensure synchronization when processing video data, and ensure that the types of video frames of different channels processed synchronously are the same (such as all are key frames or prediction frames), so that the encoding processing speed of the video frames can also be synchronized.

[0120] Step S308, performing single frame encapsulation processing on the encoded video frames.

[0121] Specifically, after the encoding process, the video data can be packaged for uploading. Since data loss may still occur during the video data uploading process, the video frames are interleaved and packaged, that is, each video frame is packaged as a single frame first, and then stored alternately. When uploading, they are uploaded in sequence according to the storage location, rather than uploading all the video frames of a certain channel first.

[0122] Step S309, alternately storing the video frames corresponding to different channels after the encapsulation process in the set storage position.

[0123] Wherein, any two adjacent video frames at the storage locations are set to belong to different channels.

[0124] Specifically, the specific alternating storage process is that the video frame with the same timestamp of each channel is stored in an adjacent position with the synchronization frame of another channel, until all the synchronization frames are stored, and then the video frame with the next timestamp is stored.

[0125] Exemplarily, as shown in Table 1 below, it is a schematic table of the alternating storage method of two-channel video frames. It can be seen from the table below that the adjacent two sides of each video frame are video frames of another channel.

[0126] <![CDATA[I 0-0 ]]> <![CDATA[I 1-0 ]]> <![CDATA[P 0-1 ]]> <![CDATA[P 1-1 ]]> <![CDATA[P 0-2 ]]> <![CDATA[P 1-2 ]]> …… <![CDATA[I 0-15 ]]> <![CDATA[I 1-15 ]]> <![CDATA[P 0-16 ]]> <![CDATA[P 1-16 ]]> <![CDATA[P 0-17 ]]> <![CDATA[P 1-17 ]]> ……

[0127] Table 1 Schematic diagram of the alternating storage method of two-channel video frames

[0128] Among them, I represents the key frame, P represents the predicted frame, and I 0-0 0-0 in the figure represents the 0th frame corresponding to channel 0. Similarly, 1-16 represents the 16th frame corresponding to channel 1.

[0129] Step S310, uploading all packaged video frames corresponding to the set storage location.

[0130] Specifically, by uploading the data in the above-mentioned set storage location as a whole to the cloud server, it is convenient for users to obtain video data from the cloud server, and after unpacking and decoding, multi-channel simultaneous playback can be achieved. And because the uploaded data is interleaved and packaged into a single file, there will be no frame loss in the video data of a single channel. Even if there is a frame loss, it is the case that the video data of each channel is lost together, and there will be no situation where the upload time of video data of different channels is not synchronized, thereby maximizing the synchronization of the screen of the multi-channel video data browsed by the user, and effectively ensuring the user's browsing experience.

[0131] The video synchronization method provided by the embodiment of the present disclosure configures sensors of different channels through the data acquisition link, adds timestamps to the acquired video data, and then judges and processes the synchronization frame after preprocessing the video data. Then, the video data is encoded based on the encoder configured with synchronization, and then the video data of different channels are encapsulated together and uploaded by interleaving and encapsulating. Thus, the synchronization of video data can be guaranteed to the maximum extent through the combination of synchronization processing measures in multiple links, thereby ensuring the synchronization of frame images when users browse video data of different channels, and significantly improving user experience.

[0132] Figure 4 The structure diagram of the video synchronization device provided by one embodiment of the present disclosure is as follows. Figure 4 As shown, the video synchronization device 400 includes: an acquisition module 410 , a determination module 420 and a processing module 430 .

[0133] in:

[0134] An acquisition module 410 is used to acquire video data collected by at least two channels and add time stamps to video frames in the video data, wherein each channel corresponds to a video sensor;

[0135] A determination module 420 is used to determine, based on the timestamp, that the video frames corresponding to at least two channels are synchronization frames, and to encode the video frames, where the synchronization frames are used to indicate that the video frames corresponding to the video sensors of the at least two channels are collected at the same time;

[0136] The processing module 430 is used to perform interleaving and encapsulation processing on the encoded video frames based on the channels corresponding to the video data, and upload the encapsulated video data.

[0137] Optionally, the acquisition module 410 is also used to acquire video data collected by at least two channels, and before adding timestamps to video frames in the video data, set the working modes of all sensors corresponding to at least two channels to slave modes; or, determine one of the sensors corresponding to at least two channels as a main sensor; set the working mode of the main sensor to the master control mode, and set the working modes of other sensors except the main sensor to the slave modes.

[0138] Optionally, the determination module 420 is specifically used to calculate the time difference between the timestamps of video frames corresponding to different channels; if the time difference is less than a set time difference threshold, determine that at least two video frames corresponding to the channels are synchronized frames; and encode the video frames.

[0139] Optionally, the determination module 420 is also used to, before encoding the video frame, if the time difference is greater than a set time difference threshold, determine the video frame with the smallest timestamp as the target video frame, and determine the channel corresponding to the target video frame as the target channel; remove the target video frame in the target channel; obtain the video frame in the target channel that is located after the target video frame, and calculate the time difference between the video frame and the video frames in other channels other than the target channel; if the time difference is greater than the set time difference threshold, repeat the process of removing the target video frame, obtaining the video frame in the target channel and calculating the time difference until the time difference is less than the set time difference threshold; determine that the video frames corresponding to at least two channels are synchronized frames.

[0140] Optionally, the determination module 420 is specifically configured to synchronously configure an encoder for processing video frames; and perform encoding processing on the video frames based on the configured encoder.

[0141] Optionally, the determination module 420 is specifically used to, if the video frames in the video data include key frames and non-key frames, the key frame interval length corresponding to the video data is used to represent the number of non-key frames between adjacent key frames in the same video data, determine that the encoder used to encode the video frames corresponding to each channel is a synchronous encoder, and the synchronous encoder is used to represent an encoder that starts processing and / or stops processing at the same time; set the frame rate of all encoders to the same value, and set the key frame interval length of all encoders to the same value.

[0142] Optionally, the processing module 430 is specifically used to perform single-frame encapsulation processing on the encoded video frames; alternately store the video frames corresponding to different channels after the encapsulation processing in a set storage position in turn, wherein any two adjacent video frames in the set storage position belong to different channels; and upload all the encapsulated video frames corresponding to the set storage position.

[0143] Optionally, the determination module 420 is further used to determine, based on the timestamp, that the video frames corresponding to at least two channels are synchronization frames, and pre-process the video frames corresponding to at least two channels before encoding the video frames, wherein the pre-processing includes resolution adjustment processing.

[0144] In this embodiment, the video synchronization device solves the problem in the related art that the corresponding images of multiple cameras are easily out of sync when browsed on the terminal through the combination of various modules, and ensures the synchronization of video data of different channels to the greatest extent, so that each frame of the finally uploaded video data can be synchronized during the synchronous playback process, significantly improving the user experience.

[0145] Figure 5 A schematic diagram of the structure of a control device provided by an embodiment of the present disclosure, such as Figure 5 As shown, the control device 500 includes: a memory 510 and a processor 520 .

[0146] The memory 510 stores a computer program that can be executed by at least one processor 520. The computer program is executed by at least one processor 520 to enable the control device to implement the material removal method provided in any of the above embodiments or the video synchronization method provided in any of the above embodiments.

[0147] The memory 510 and the processor 520 may be connected via a bus 530 .

[0148] The relevant instructions can be understood by referring to the relevant descriptions and effects corresponding to the method embodiments, which will not be repeated here.

[0149] The relevant instructions can be understood by referring to the relevant descriptions and effects corresponding to the method embodiments, which will not be repeated here.

[0150] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement a material removal method as provided in any of the above method embodiments or a video synchronization method as provided in any of the above embodiments.

[0151] Among them, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0152] An embodiment of the present disclosure provides a computer program product, which includes computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the material removal method in the above method embodiment or the video synchronization method provided in any of the above embodiments.

[0153] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0154] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0155] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video synchronization method, characterized in that: The video synchronization method comprises: Obtain video data collected by at least two channels, and add time stamps to video frames in the video data, wherein each channel corresponds to a video sensor; Based on the timestamp, determine that the video frames corresponding to the at least two channels are synchronization frames, and encode the video frames, wherein the synchronization frames are used to represent the video frames collected by the video sensors corresponding to the at least two channels at the same time; Based on the channel corresponding to the video data, the encoded video frames are interleaved and packaged, and the packaged video data is uploaded.

2. The method according to claim 1, characterized in that Before acquiring the video data collected by at least two channels and adding time stamps to the video frames in the video data, the method further includes: The working modes of the sensors corresponding to the at least two channels are all set to slave mode; or, Determine one of the sensors corresponding to the at least two channels as a main sensor; The working mode of the main sensor is set to the master mode, and the working modes of other sensors except the main sensor are set to the slave mode.

3. The method according to claim 1, characterized in that The determining, based on the timestamp, that the video frames corresponding to the at least two channels are synchronization frames, and encoding the video frames, includes: Calculate the time difference between the timestamps of the corresponding video frames of different channels; If the time difference is less than a set time difference threshold, determining that the video frames corresponding to the at least two channels are synchronized frames; The video frame is encoded.

4. The method according to claim 3, characterized in that Before encoding the video frame, the method further includes: If the time difference is greater than a set time difference threshold, the video frame with the smallest timestamp is determined as the target video frame, and the channel corresponding to the target video frame is determined as the target channel; Removing the target video frame in the target channel; Acquire a video frame in the target channel that is located after the target video frame, and calculate a time difference between the video frame and video frames in other channels other than the target channel; If the time difference is greater than the set time difference threshold, repeat the process of removing the target video frame, obtaining the video frame in the target channel and calculating the time difference until the time difference is less than the set time difference threshold; Determine that the video frames corresponding to the at least two channels are synchronization frames.

5. The method according to claim 3, characterized in that: The encoding process of the video frame includes: Synchronously configuring an encoder for processing the video frame; Based on the configured encoder, the video frame is encoded.

6. The method according to claim 5, characterized in that The video frames in the video data include key frames and non-key frames, and the key frame interval length corresponding to the video data is used to represent the number of non-key frames between adjacent key frames in the same video data; The synchronous configuration of the encoder for processing the video frame includes: Determine that an encoder used for encoding a video frame corresponding to each channel is a synchronous encoder, where the synchronous encoder is used to indicate an encoder that starts processing and / or stops processing at the same time; Set the frame rate of all encoders to the same value, and set the keyframe interval length of all encoders to the same value.

7. The method according to any one of claims 1 to 6, characterized in that The step of interleaving and encapsulating the encoded video frames based on the channel corresponding to the video data, and uploading the encapsulated video data, includes: Performing single-frame encapsulation processing on the encoded video frames; The video frames corresponding to different channels after the encapsulation process are alternately stored in a set storage position in sequence, wherein any two adjacent video frames in the set storage position belong to different channels; Upload all packaged video frames corresponding to the set storage location.

8. The method according to any one of claims 1 to 6, characterized in that Before determining, based on the timestamp, that the video frames corresponding to the at least two channels are synchronization frames and encoding the video frames, the method further includes: The video frames corresponding to the at least two channels are preprocessed, and the preprocessing includes resolution adjustment processing.

9. A video synchronization device, characterized in that: The video synchronization device comprises: An acquisition module, used to acquire video data collected by at least two channels and add time stamps to video frames in the video data, wherein each channel corresponds to a video sensor; A determination module, configured to determine, based on the timestamp, that the video frames corresponding to the at least two channels are synchronization frames, and to encode the video frames, wherein the synchronization frames are used to represent the video frames collected by the video sensors corresponding to the at least two channels at the same time; The processing module is used to perform interleaving and encapsulation processing on the encoded video frames based on the channels corresponding to the video data, and upload the encapsulated video data.

10. A control device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the control device to perform the video synchronization method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the video synchronization method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, characterized in that The computer program product comprises computer-executable instructions, and when the computer-executable instructions are executed by a processor, the computer-executable instructions are used to implement the video synchronization method according to any one of claims 1 to 8.

Images