Multi-camera spatio-temporal joint synchronous acquisition processing method and device, and storage medium

Abstract

The invention provides a multi-camera space-time joint synchronous acquisition processing method and device and a storage medium, and the method comprises the steps: receiving data packets uploaded by each camera, and generating a corresponding video data packet queue; obtaining a UTC absolute timestamp of a first frame video data packet of each queue and a PTS of each video data packet; calculating the incremental offset of the PTS of each video data packet relative to the PTS of the first frame of video data packet, and calculating the UTC absolute timestamp of each video data packet according to the UTC absolute timestamp of the first frame of video data packet and the incremental offset corresponding to each video data packet; and determining the video receiving moment with the optimal relative synchronism degree as a video synchronization time point according to the UTC absolute timestamps of the video data packets of the cameras received at different video receiving moments, and generating a synchronous multi-view video sequence based on the multiple cameras according to the video data packets corresponding to the video synchronization time point. And space-time joint synchronous acquisition of multi-camera video data is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and information communication technology, and in particular to a multi-camera spatiotemporal joint synchronous acquisition processing method, device and storage medium. Background Art

[0002] Multi-camera systems for different physical locations provide basic capabilities for the construction of digital sports venues. Multi-point multi-camera devices at different physical locations are deployed on the playing field of the sports venue, which helps to capture the field conditions in all directions, especially supporting the synchronization of multi-camera video shooting, which is crucial for capturing fast-paced ball sports. The multi-camera synchronization system based on physical multi-points is different from the multi-camera output stereo vision acquisition system similar to a physical single point with a common substrate or a common bracket. The problem is more complicated and has certain research value.

[0003] The existing technologies for realizing multi-camera synchronization mainly include the following methods: for example, the subject adopts a plurality of image acquisition cards in series to trigger the synchronization control between the host and multiple cameras, and a high-precision timer module is used to generate a high-precision clock signal for communication synchronization between devices; for another example, a hardware synchronizer main board and a hardware synchronizer slave board are used to control and communicate the acquisition trigger signal, and drive the camera to acquire the host to output a multi-viewpoint video sequence.

[0004] The existing methods for achieving multi-camera synchronization have at least the following shortcomings: 1. The technical solution uses more electronic device modules, and the overall cost is relatively high; 2. Based on the multi-camera with a long physical multi-point space distance, the image acquisition card in series in the existing technology, and the connection between the main board and the slave board of the hardware synchronizer are not very friendly to the engineering wiring in the case of long multi-point space distance. The signal attenuation of long-distance wiring is serious, the adaptability is not high, and the space appears to be relatively messy and disordered, and the solution is not conducive to implementation. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a multi-camera spatiotemporal joint synchronous acquisition processing method, device and storage medium that solve the above technical problems or at least partially solve the above technical problems.

[0006] One aspect of the present invention provides a multi-camera spatiotemporal joint synchronous acquisition and processing method, the method comprising:

[0007] Receive data packets uploaded by each camera, wherein the data packets include video data packets, and generate video data packet queues corresponding to each camera;

[0008] Obtain the UTC absolute timestamp and display timestamp PTS of the first frame of video data packet in each video data packet queue, as well as the PTS of each subsequent video data packet in the queue except the first frame of video data packet;

[0009] Calculate the incremental offset of the PTS of each subsequent video data packet corresponding to the PTS of the first frame video data packet, and calculate the UTC absolute timestamp of the corresponding subsequent video data packet according to the UTC absolute timestamp of the first frame video data packet and the incremental offset corresponding to each subsequent video data packet;

[0010] Determine the relative synchronization degree of the video data collected by each camera according to the UTC absolute timestamp of the video data packets received by each camera at different video receiving times, and use the video receiving time with the best relative synchronization degree as the video synchronization time point;

[0011] The video data packets received from each camera at the video synchronization time point are used as the first frame of video data packets to be output by the corresponding camera to generate a synchronized multi-viewpoint video sequence based on multiple cameras.

[0012] Further, the data packet also includes an audio data packet;

[0013] The method further comprises:

[0014] Generate audio data packet queues corresponding to each camera;

[0015] Get the PTS of each audio data packet in the audio data packet queue, calculate the incremental offset of the PTS of each audio data packet corresponding to the PTS of the first frame of the video data packet, and calculate the UTC absolute timestamp of each video data packet according to the UTC absolute timestamp of the first frame of the video data packet and the incremental offset corresponding to each audio data packet;

[0016] Correspondingly, the method of using the video data packets received from each camera at the video synchronization time point as the first frame of video data packets to be output by the corresponding camera to generate a synchronized multi-viewpoint video sequence based on multiple cameras includes:

[0017] Obtain target audio data packets synchronized with the video data packets received by each camera at the video synchronization time point, and for the video data packet queue and audio data packet queue corresponding to each camera, generate a synchronized multi-viewpoint video sequence based on multiple cameras with the video data packet received at the video synchronization time point and the target audio data packet synchronized therewith as the starting positions.

[0018] Further, after calculating the UTC absolute timestamps of the subsequent video data packets according to the UTC absolute timestamp of the first frame of the video data packet and the incremental offsets corresponding to the subsequent video data packets, the method further includes:

[0019] The UTC absolute timestamp and PTS of each data packet are bound to the corresponding data packet, and the host UTC timestamp when the host receives each data packet is used as the video receiving time of the corresponding data packet to update the video data packet queue.

[0020] Further, the method of determining the relative synchronization degree of the video data collected by each camera according to the UTC absolute timestamp of the video data packets received by each camera at different video receiving times, and taking the video receiving time with the best relative synchronization degree as the video synchronization time point, includes:

[0021] The UTC absolute timestamp of the video data packets uploaded by each camera received by the host at the same video reception moment is used to construct a synchronization vector T = [t1, t2, ..., t n ], where t i is the UTC absolute timestamp of the video data packet captured by the i-th camera;

[0022] According to the synchronization vector T at different video receiving moments, the relative synchronization error index α of the video data collected by each camera at each video receiving moment is calculated. The calculation formula A of α is:

[0023]

[0024] Where t is the video receiving time corresponding to the synchronization vector T;

[0025] Find the video receiving time corresponding to the minimum value of the relative synchronization error index, and use the current video receiving time as the video synchronization time point.

[0026] Furthermore, before calculating the relative synchronization error index α of the video data collected by each camera at each video receiving moment according to the synchronization vector T at different video receiving moments, the method further includes:

[0027] According to the synchronization vector T corresponding to different video receiving moments, the asynchronous deviation index β of the video data collected by each camera at each video receiving moment is calculated. The calculation formula B of β is:

[0028] B=max i |t i -t|

[0029] When it is detected that the asynchronous deviation index β corresponding to a certain video receiving moment is less than the preset system nominal error, the step of calculating the relative synchronization error index α of the video data collected by each camera at each video receiving moment is performed for the synchronization vector T corresponding to subsequent different video receiving moments starting from the current video receiving moment.

[0030] Further, the step of acquiring a target audio data packet synchronized with the video data packets received from each camera at the video synchronization time point includes:

[0031] Search the video data packet queue corresponding to each camera to obtain the host UTC timestamp and PTS of the target video data packet corresponding to the current camera. The target video data packet is the video data packet of the corresponding camera received at the video synchronization time point.

[0032] The audio data packet queue of each camera is searched respectively to obtain a target audio data packet in the audio data packet queue whose host UTC timestamp is equal to or greater than the host UTC timestamp of the target video data packet corresponding to the current camera.

[0033] Furthermore, the method further comprises:

[0034] After searching the audio data packet queue of each camera to obtain the target audio data packet whose host UTC timestamp is equal to or greater than the host UTC timestamp of the target video data packet corresponding to the current camera in the audio data packet queue, it is determined whether the PTS of the target video data packet corresponding to each camera is modified and rewritten when it is output as the first frame of video data;

[0035] If the PTS of the target video data packet is modified and rewritten when it is output as the first frame of video data, the PTS of the target audio data packet is synchronously updated according to the offset time difference of the target audio data packet of the camera relative to the target video data packet;

[0036] For the video data packet queue and audio data packet queue corresponding to each camera, the PTS of each data packet is synchronously updated packet by packet toward the tail of each queue according to the PTS of the target video data packet and the target audio data packet.

[0037] Further, the receiving of data packets uploaded by each camera includes:

[0038] Before the host sends the acquisition instruction to each camera, perform video preloading processing on each camera to receive the data packets uploaded by each camera; and / or

[0039] Send acquisition instructions to each camera to receive data packets uploaded by each camera.

[0040] Another aspect of the present invention provides a multi-camera spatiotemporal joint synchronous acquisition and processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above multi-camera spatiotemporal joint synchronous acquisition and processing method when executing the computer program.

[0041] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above multi-camera spatiotemporal joint synchronous acquisition and processing method.

[0042] The multi-camera spatiotemporal joint synchronous acquisition processing method, device and storage medium provided by the embodiment of the present invention can realize multi-camera spatiotemporal joint synchronous acquisition without setting other external electronic devices, which can not only save the overall cost of hardware equipment, but also make the multi-camera engineering wiring based on physical multi-points more concise and highly operable. Moreover, the present invention fully embodies the spatiotemporal coordination and joint synchronization of the host and multiple cameras, supports the detection and calculation of the relative synchronization degree of the video data collected by each camera to update and determine the video synchronization time point, can effectively ensure the realization accuracy of the spatiotemporal joint synchronization of multiple cameras, and improve the stability and reliability of the system.

[0043] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only used for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. In the accompanying drawings:

[0045] Figure 1 It is a flow chart of a multi-camera spatiotemporal joint synchronous acquisition and processing method according to an embodiment of the present invention;

[0046] Figure 2 This is a flowchart of a multi-camera spatiotemporal joint synchronous acquisition processing method according to another embodiment of the present invention. DETAILED DESCRIPTION

[0047] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0048] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined.

[0049] The image sensor used in the camera involved in the present invention is not limited to a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), a linear scan sensor, an infrared sensor, a thermal imager, a 3D depth sensor, an X-ray sensor, an ultraviolet sensor, etc.

[0050] The connection between the camera and the host computer involved in the present invention is not limited to interfaces such as USB, MIPI, HDMI, GigE, Ethernet, Wi-Fi, SDI, VGA, AV, Thunderbolt and FireWire (IEEE 1394).

[0051] The multi-camera spatiotemporal joint synchronous acquisition processing method provided by the present invention is applicable to a multi-camera synchronous acquisition system. The real-time system includes at least two or more cameras and a host terminal. The host terminal sends an acquisition instruction to each camera. The camera obtains the acquisition instruction and starts to collect live video data and uploads it to the host. The host obtains the video data packet of the camera and performs spatiotemporal joint synchronous alignment processing to generate an aligned multi-view video sequence. The multi-camera spatiotemporal joint synchronous acquisition processing method provided by an embodiment of the present invention is described in detail below.

[0052] like Figure 1 As shown, the multi-camera spatiotemporal joint synchronous acquisition and processing method proposed in the embodiment of the present invention includes the following steps:

[0053] S11, receiving data packets uploaded by each camera, wherein the data packets include video data packets, and generating video data packet queues corresponding to each camera.

[0054] In this embodiment, after receiving the video streams transmitted from each camera, the host generates a data packet queue and obtains parameter characteristics of the video stream, where the video parameter characteristics include video resolution, frame rate, encoder type and time base, etc.

[0055] S12: Obtain the UTC absolute timestamp and display timestamp PTS of the first frame of video data packet in each video data packet queue, and the PTS of each subsequent video data packet in the queue except the first frame of video data packet.

[0056] S13, calculating the incremental offset of the PTS of each subsequent video data packet corresponding to the PTS of the first frame video data packet, and calculating the UTC absolute timestamp of the corresponding subsequent video data packet according to the UTC absolute timestamp of the first frame video data packet and the incremental offset corresponding to each subsequent video data packet.

[0057] In this embodiment, the host calculates and generates the UTC absolute timestamp utc_timestamp and the video start display timestamp Presentation Time Stamp (PTS) of the first frame of the video data packet based on the video data packet received from the camera, as well as the PTS corresponding to each video data packet. For each video data packet, based on the UTC absolute timestamp of the first frame of video and the PTS of the first frame of video data packet and the PTS corresponding to each data packet mentioned above, the incremental offset of the PTS of each data packet corresponding to the PTS of the first frame of video is calculated, and the UTC absolute timestamp of each data packet is calculated by superimposing this offset and the UTC absolute timestamp of the first frame of video.

[0058] Furthermore, the method further comprises: binding the UTC absolute timestamp and PTS of each data packet to the corresponding data packet. Moreover, the host records the local host UTC timestamp local_utc_timestamp of each collected data packet, and updates the host UTC timestamp when the host receives each data packet as the video receiving time of the corresponding data packet to the video data packet queue.

[0059] The processing logic for calculating the UTC absolute timestamp of the video data packet is as follows:

[0060] Get the UTC absolute timestamp of the first frame of video data packet:

[0061] first_frame_utc=first_video_packet.utc_timestamp;

[0062] Get the PTS of the first frame of video data packet:

[0063] first_frame_pts=first_video_packet.pts;

[0064] Traverse all video data packets and calculate the incremental offset of the PTS of the current data packet relative to the PTS of the first frame of video:

[0065] delta_video_pts=packet.pts-first_frame_pts;

[0066] Among them, video_time_unit is the video clock frequency, its value is equivalent to the video time base, and its typical value is 1 / 90000;

[0067] Calculate the UTC absolute timestamp packet_utc of the current data packet:

[0068] packet_utc=first_frame_utc+delta_video_pts*video_time_unit, where video_time_unit is a conversion factor, for example, conversion from PTS unit to UTC unit.

[0069] Bind the calculated UTC absolute timestamp and PTS to the data packet:

[0070] packet.utc_timestamp=packet_utc

[0071] Update to the video packet queue:

[0072] video_packets_queue.append(packet).

[0073] S14. Determine the relative synchronization degree of the video data collected by each camera according to the UTC absolute timestamp of the video data packets received by each camera at different video receiving times, and use the video receiving time with the best relative synchronization degree as the video synchronization time point.

[0074] S15: Using the video data packets received from each camera at the video synchronization time point as the first frame of video data packets to be output by the corresponding camera to generate a synchronized multi-viewpoint video sequence based on multiple cameras.

[0075] The multi-camera spatiotemporal joint synchronous acquisition processing method provided by the embodiment of the present invention can realize multi-camera spatiotemporal joint synchronous acquisition without setting up other external electronic devices, which can not only save the overall cost of hardware equipment, but also make the multi-camera engineering wiring based on physical multi-points more concise and highly operable. Moreover, the present invention fully embodies the spatiotemporal coordination and joint synchronization of the host and multiple cameras, supports the detection and calculation of the relative synchronization degree of the video data collected by each camera to update and determine the video synchronization time point, can effectively ensure the realization accuracy of the spatiotemporal joint synchronization of multiple cameras, and improve the stability and reliability of the system.

[0076] In the embodiment of the present invention, the video data packets uploaded by each camera are received by performing video preloading processing on each camera to receive the video data packets uploaded by each camera before the host sends the acquisition instruction to each camera; or the acquisition instruction can be directly sent to each camera to receive the video data packets uploaded by each camera.

[0077] Since the first frame of the video data packet uploaded by the camera is often large after being compressed by the video encoder, the present invention adopts preloading processing before formally acquiring the video data to avoid network congestion for receiving the first frame, thereby improving the first frame experience. Specifically, before the host sends the acquisition instruction to each camera, the host obtains the video data packet of each camera through a video preloading process, and pre-fetches the video header data of each camera to predetermine the video synchronization time point. Among them, the duration d of the preloaded video, d is a floating point number greater than 0, in seconds. In the preloading stage, the host side receives a certain amount of video data packets from each camera, thereby forming a corresponding video data packet queue. In the preloading stage, in order to reduce the computational overhead, the data packet can be decoded without first performing the decoding operation, but only the packet header information of the data packet is parsed, including at least the packet type PacketType, the packet sequence number Packet Index, the display time stamp Presentation Time Stamp (PTS), etc., and is bound to the corresponding data packet and updated to the video data packet queue in the preloading stage.

[0078] Furthermore, the present invention pre-synchronizes the system reference coordinated universal time Universal Time Coordinated (UTC) clock of each camera by synchronous timing, such as using the Precision Time Protocol (PTP) or Network Time Protocol (NTP) network clock protocol to synchronize the local system clock of each camera. NTP provides millisecond accuracy, while PTP can provide up to sub-microsecond accuracy. More preferably, the present invention synchronizes the UTC clock by using the host and each camera to synchronize timing with the same world time server, so that the host and each camera have a local clock synchronized with the global clock, so that the data communication between the host and the camera is based on a global clock, thereby ensuring that the session is in the same pace.

[0079] In the embodiment of the present invention, the step S14 determines the relative synchronization of the video data collected by each camera according to the UTC absolute timestamps of the video data packets received by each camera at different video receiving times, and uses the video receiving time with the best relative synchronization as the video synchronization time point, including:

[0080] The UTC absolute timestamp of the video data packets uploaded by each camera received by the host at the same video reception moment is used to construct a synchronization vector T = [t1, t2, ..., t n ], where t i is the UTC absolute timestamp of the video data packet captured by the i-th camera;

[0081] According to the synchronization vector T at different video receiving moments, the relative synchronization error index α of the video data collected by each camera at each video receiving moment is calculated. The calculation formula A of α is:

[0082]

[0083] Where t is the video receiving time corresponding to the synchronization vector T;

[0084] Find the video receiving time corresponding to the minimum value of the relative synchronization error index, and use the current video receiving time as the video synchronization time point.

[0085] Furthermore, before calculating the relative synchronization error index α of the video data collected by each camera at each video receiving moment according to the synchronization vector T at different video receiving moments, the asynchronous deviation index β of the video data collected by each camera at each video receiving moment can be calculated according to the synchronization vector T corresponding to the different video receiving moments. The calculation formula B of β is:

[0086] B=max i |t i -t|

[0087] When it is detected that the asynchronous deviation index β corresponding to a certain video receiving moment is less than the preset system nominal error, the step of calculating the relative synchronization error index α of the video data collected by each camera at each video receiving moment is performed for the synchronization vector T corresponding to subsequent different video receiving moments starting from the current video receiving moment.

[0088] In this embodiment, multi-camera relative synchronization is used to measure the degree of time alignment of frames captured by all cameras in a multi-camera system. Assume that there are n cameras, each camera captures a frame at time t, and each frame has a UTC absolute timestamp. We can define a vector T = [t1, t2, ..., t n ], where t i is the UTC absolute timestamp of the frame captured by the i-th camera. Ideally, if all cameras are perfectly synchronized, all elements of this vector should be equal, i.e., t1 = t2 = ... = t n . However, in practical applications, this ideal situation is difficult to achieve due to factors such as hardware limitations, network delays and processing delays. The present invention uses the optimal synchronization equation A to measure the relative synchronization degree of the optimal conditions of the camera, characterizing the Euclidean distance between the perception vectors T and t. The present invention uses the worst asynchrony equation B to measure the worst asynchrony degree of the camera, characterizing the maximum deviation of the vector T element sample from the true value.

[0089]

[0090] B=maxi |t i -t|

[0091] As in the above equation A, B, t i is the utc_timestamp of the frame captured by the i-th camera, and t is the local UTC timestamp local_utc_timestamp corresponding to the camera data packet received by the host.

[0092] The worst allowable synchronization error of the preset multi-camera system is defined as σ. Optionally, when the camera video stream frame rate is 120 Hz, σ is 0.005 seconds, that is, 5 milliseconds accuracy; when the camera video stream frame rate is 60 Hz, σ is 0.010 seconds, that is, 10 milliseconds accuracy; when the camera video stream frame rate is 30 Hz, σ is 0.020 seconds, that is, 20 milliseconds accuracy.

[0093] In actual application, conditional judgment can be used to determine whether relative synchronization error index detection is required. For example, a multi-camera system needs to perform spatiotemporal joint synchronization processing during initialization, and spatiotemporal joint synchronization processing is required if asynchrony occurs during the acquisition process. The specific conditions can be set to perform relative synchronization error index detection calculation when the system is initialized or when the β value calculated by the multi-camera worst asynchrony index module does not reach the system nominal error. The asynchrony deviation index β is calculated according to equation B. When β<σ, the relative synchronization error index α is calculated according to equation A. The specific calculation of the relative synchronization error index includes traversing the video data packet queue from new to old, that is, from the tail end of the queue to the head end of the queue to obtain the calculated value α of equation A at a certain moment, that is, when there is a similar value t, so that the related synchronization error of the joint multi-camera space is minimized, that is, α min This corresponds to the optimal synchronization point of each camera. In this way, this synchronization point is the alignment point of the physical objective world time of each camera, and is also the first frame of video processed by the camera synchronization logic.

[0094] Furthermore, the present invention can also define a period for normalizing the relative synchronization error index detection, which is used to start the normalized relative synchronization error index detection calculation and feedback in a period of time. This can effectively eliminate the influence of camera hardware processing and network transmission jitter, and further improve the robustness and reliability of the multi-camera system synchronization performance.

[0095] The present invention supports real-time detection, calculation and feedback closed loop of optimal synchronization index and worst asynchronous index related to multiple cameras, and proposes an anti-jitter processing method. It has the characteristics of high system stability and reliability, can effectively ensure the realization accuracy of multi-camera spatiotemporal joint synchronization, and improve system stability and reliability.

[0096] For a multi-camera system that also has an audio option, the present invention also provides another embodiment for detailed description. Figure 2 As shown, the multi-camera spatiotemporal joint synchronous acquisition and processing method proposed in the embodiment of the present invention includes the following steps:

[0097] S21, receiving video data packets and audio data packets uploaded by each camera, and generating video data packet queues and audio data packet queues corresponding to each camera.

[0098] After receiving the video and audio data packets transmitted by each camera, the host generates video and audio data packet queues respectively, and obtains the parameter characteristics of the audio and video streams. The video parameter characteristics include video resolution, frame rate, encoder type and time base, etc. The audio parameter characteristics include audio sampling rate, time base, encoder type, number of channels and sample format, etc.

[0099] S22, obtaining the UTC absolute timestamp and display timestamp PTS of the first frame of video data packet in each video data packet queue, and the PTS of each subsequent video data packet and audio data packet in the queue except the first frame of video data packet.

[0100] S23. Calculate the incremental offset of the PTS of each subsequent video data packet and audio data packet relative to the PTS of the first frame video data packet, and calculate the UTC absolute timestamp of the corresponding subsequent video data packet and audio data packet based on the UTC absolute timestamp of the first frame video data packet and the incremental offset corresponding to each subsequent video data packet and audio data packet.

[0101] In this embodiment, the host calculates and generates the UTC absolute timestamp utc_timestamp and the video start display timestamp Presentation Time Stamp (PTS) of the first frame of the video data packet based on the video data packet received from the camera, as well as the PTS corresponding to each video and audio data packet. For each video or audio data packet, based on the UTC absolute timestamp of the first frame of video and the PTS of the first frame of video data packet and the PTS corresponding to each data packet mentioned above, the incremental offset of the PTS of each data packet corresponding to the PTS of the first frame of video is calculated, and the UTC absolute timestamp of each data packet is calculated by superimposing the incremental offset and the UTC absolute timestamp of the first frame of video.

[0102] Furthermore, the method further comprises: binding the UTC absolute timestamp and PTS of each data and audio data packet to the corresponding data packet. Moreover, the host records the local host UTC timestamp local_utc_timestamp of each collected data packet, and updates the host UTC timestamp when the host receives each data packet as the video receiving time of the corresponding data packet to the video data packet queue.

[0103] The processing logic for calculating the UTC absolute timestamps of video and audio data packets is as follows:

[0104] Get the UTC absolute timestamp of the first frame of video data packet:

[0105] first_frame_utc=first_video_packet.utc_timestamp;

[0106] Get the PTS of the first frame of video data packet:

[0107] first_frame_pts=first_video_packet.pts;

[0108] Traverse all video data packets and calculate the incremental offset of the PTS of the current data packet relative to the PTS of the first frame of video:

[0109] delta_video_pts=packet.pts-first_frame_pts;

[0110] Among them, video_time_unit is the video clock frequency, its value is equivalent to the video time base, and its typical value is 1 / 90000;

[0111] Calculate the UTC absolute timestamp packet_utc of the current data packet:

[0112] packet_utc=first_frame_utc+delta_video_pts*video_time_unit, where video_time_unit is a conversion factor, for example, conversion from PTS unit to UTC unit.

[0113] Bind the calculated UTC absolute timestamp and PTS to the data packet:

[0114] packet.utc_timestamp=packet_utc

[0115] Update to the video packet queue:

[0116] video_packets_queue.append(packet)

[0117] Traverse all audio data packets and calculate the elapsed time offset of the current data packet's PTS relative to the first frame of video PTS, in seconds:

[0118] since_1st_packet=packet.pts*audio_time_unit-first_frame_pts*video_time_unit;

[0119] Among them, the audio clock frequency audio_time_unit, its value is equivalent to the audio time base, and its typical value is the reciprocal of the audio sampling rate. For example, if the sampling rate is 16000 samples / second, it is 1 / 16000;

[0120] Calculate the UTC absolute timestamp of the current packet:

[0121] packet_utc=first_frame_utc+since_1st_packet;

[0122] Bind the calculated UTC absolute timestamp and PTS to the data packet:

[0123] packet.utc_timestamp=packet_utc;

[0124] Update to the audio packet queue: audio_packets_queue.append(packet).

[0125] S24. Determine the relative synchronization degree of the video data collected by each camera according to the UTC absolute timestamp of the video data packets received by each camera at different video receiving times, and use the video receiving time with the best relative synchronization degree as the video synchronization time point.

[0126] S25. Obtain a target audio data packet synchronized with the video data packets received by each camera at a video synchronization time point, and for a video data packet queue and an audio data packet queue corresponding to each camera, generate a synchronized multi-viewpoint video sequence based on multiple cameras, taking the video data packet received at the video synchronization time point and the target audio data packet synchronized therewith as starting positions.

[0127] In the embodiment of the present invention, the step S24 determines the relative synchronization of the video data collected by each camera according to the UTC absolute timestamps of the video data packets received by each camera at different video reception times, and uses the video reception time with the best relative synchronization as the video synchronization time point, including:

[0128] The UTC absolute timestamp of the video data packets uploaded by each camera received by the host at the same video reception moment is used to construct a synchronization vector T = [t1, t2, ..., t n ], where t i is the UTC absolute timestamp of the video data packet captured by the i-th camera;

[0129] According to the synchronization vector T at different video receiving moments, the relative synchronization error index α of the video data collected by each camera at each video receiving moment is calculated. The calculation formula A of α is:

[0130]

[0131] Where t is the video receiving time corresponding to the synchronization vector T;

[0132] Find the video receiving time corresponding to the minimum value of the relative synchronization error index, and use the current video receiving time as the video synchronization time point.

[0133] Furthermore, before calculating the relative synchronization error index α of the video data collected by each camera at each video receiving moment according to the synchronization vector T at different video receiving moments, the method further includes:

[0134] According to the synchronization vector T corresponding to different video receiving moments, the asynchronous deviation index β of the video data collected by each camera at each video receiving moment is calculated. The calculation formula B of β is:

[0135] B=max i |t i -t|

[0136] When it is detected that the asynchronous deviation index β corresponding to a certain video receiving moment is less than the preset system nominal error, the step of calculating the relative synchronization error index α of the video data collected by each camera at each video receiving moment is performed for the synchronization vector T corresponding to the subsequent different video receiving moments starting from the current video receiving moment. Otherwise, the target video data packet corresponding to the historical video synchronization time point can be used as the first frame of video data packet to be output to continue generating a synchronous multi-viewpoint video sequence based on multiple cameras.

[0137] In the embodiment of the present invention, obtaining the target audio data packet synchronized with the video data packets received by each camera at the video synchronization time point in step S25 specifically includes: searching the video data packet queue corresponding to each camera to obtain the host UTC timestamp and PTS of the target video data packet corresponding to the current camera, the target video data packet being the video data packet received by the corresponding camera at the video synchronization time point; searching the audio data packet queue of each camera to obtain the target audio data packet in the audio data packet queue whose host UTC timestamp is equal to or greater than the host UTC timestamp of the target video data packet corresponding to the current camera.

[0138] In order to achieve synchronization of the camera time domain dimension, the present invention uses the audio and video synchronization technology to process the audio stream and video stream output by a single camera respectively according to the video synchronization time point, that is, the optimal synchronization point. In order to achieve the optimal audio and video synchronization effect, it is necessary to calculate the audio data packet corresponding to the corresponding video synchronization time point. The specific implementation process is as follows: obtain the host UTC timestamp local_utc_timestamp and pts of the target video data packet corresponding to the video synchronization time point from the video data packet queue, and search and calculate the audio data packet whose host UTC timestamp local_utc_timestamp is greater than or equal to the host UTC timestamp local_utc_timestamp of the target video data packet in the audio data packet queue of the same camera, and use it as the target audio data packet.

[0139] In order to achieve the best audio-visual synchronization effect, further, after searching the audio data packet queue of each camera to obtain the target audio data packet whose host UTC timestamp is equal to or greater than the host UTC timestamp of the target video data packet corresponding to the current camera in the audio data packet queue, it is determined whether the PTS of the target video data packet corresponding to each camera is corrected and rewritten when it is output as the first frame of video data; if the PTS of the target video data packet is corrected and rewritten when it is output as the first frame of video data, the PTS of the target audio data packet is synchronously updated according to the offset time difference of the target audio data packet of the camera relative to the target video data packet; then for the video data packet queue and audio data packet queue corresponding to each camera, the PTS of each data packet is synchronously updated one by one toward the tail of each queue according to the PTS of the target video data packet and the target audio data packet. If the PTS of the target video data packet is not corrected and rewritten when it is output as the first frame of video data, the PTS of the target audio data packet obtained by searching and calculating shall prevail.

[0140] Calculate the offset time difference of the target audio data packet relative to the target video data packet:

[0141] audio_elapsed_since_video_sync=audio_sync.pts*audio_time_unit–video_sync.pts*video_time_unit;

[0142] Taking the video reference pts as 0, calculate and update the pts of the audio synchronization point to achieve the best synchronization between audio and video:

[0143] audio.sync.pts=round(audio_elapsed_since_video_sync / audio_time_unit);

[0144] Among them: audio_sync is the target audio data packet, audio_sync.pts is the PTS of the target audio data packet, video_sync.pts is the PTS of the video packet of the optimal synchronization point, audio_elapsed_since_video_sync: the natural time elapsed value of the audio synchronization point based on the video synchronization point, in seconds; audio_time_unit is the audio clock frequency, and video_time_unit is the video clock frequency.

[0145] Correspondingly, a synchronous multi-view video sequence based on multiple cameras is generated with the video data packet received at the video synchronization time point and the target audio data packet synchronized therewith as the starting position, including: the audio data packet and the video data packet in the queue are processed into corresponding audio frames and video frames according to the timeline corresponding to their respective PTSs, respectively, with the video data packet received at the video synchronization time point and the target audio data packet synchronized therewith as the starting position, so as to generate a synchronous multi-view video sequence based on multiple cameras. The present invention updates the PTS corresponding to each data packet one by one toward the tail of each cache queue according to the position of the best video synchronization point and audio synchronization point, and then restores the audio and video data packets of each camera queue to the corresponding audio frames and video frames according to the multiple cameras and the timeline, thereby generating a synchronous multi-view video sequence based on multiple cameras.

[0146] In one embodiment of the present invention, the camera is a network-based camera, and what is transmitted is a byte stream data packet compressed by an audio or video encoder. In this case, the host end performs packet assembly processing of the relevant data packets according to the corresponding protocol standard and performs decoding processing of the data packets according to the relevant audio or video compression standard. Preferably, for the video stream sent by the camera, such as ITU-T H.264, ITU-TH.265 and AOMedia Video 1 standard protocols, the host end performs hardware video decoding processing according to its own hardware capability specifications to reduce processing delay and improve performance. For example, Nvidia video decoding capabilities are used.

[0147] The multi-camera spatiotemporal joint synchronous acquisition and processing method provided by the present invention, if the multi-camera acquisition system is configured not to generate an audio stream, the present invention only outputs a sequence of pure video parts of multiple viewpoints; it also includes the generated multi-viewpoint video sequence only bypassing the original camera stream parts, in which case the present invention outputs a composite audio and video output multi-viewpoint video sequence; it also includes the case where the generated multi-viewpoint video needs to be rewritten or transcoded, such as the need to perform correction processing on the original video or superimpose a LOGO and other rewritten video, then it is necessary to perform corresponding video processing on the basis of the original video and then re-compositely output the video sequence with multiple viewpoints.

[0148] Preferably, in the embodiment of the present invention, when the host performs video and audio related processing, such as camera distortion correction processing, audio or video resampling, video format conversion and other processing technologies, GPU CUDA / OpenCL technology is preferentially used to release or reduce the host CPU computing pressure and bottleneck, and the host can support multi-camera 60HZ, 120HZ or higher ultra-high frame rate video processing capabilities. Therefore, the system has the ability to expand the number of multiple cameras according to the host performance configuration capability, which is conducive to the technical solution of the present invention by selecting different numbers of cameras to be assembled in venues of different sizes.

[0149] The multi-camera spatiotemporal joint synchronous acquisition processing method proposed in the present invention has a hardware system that only includes multiple cameras and a host component unit, does not involve unnecessary cost, and can achieve multi-camera spatiotemporal joint synchronous acquisition without setting up other external electronic devices. This not only saves hardware equipment costs, but also makes the multi-camera wiring based on physical multi-points more concise and highly operational.

[0150] In addition, the present invention fully reflects the characteristics of spatiotemporal coordination and joint synchronization of the host and multiple cameras, has high accuracy and complies with the system nominal, supports real-time detection, calculation and feedback closed loop of the optimal synchronization index and the worst asynchrony index related to multiple cameras, proposes an anti-jitter processing method, has the characteristics of high system stability and reliability, adaptive low, high or ultra-high frame rate multi-camera synchronous acquisition and processing, and has the ability to expand the number of multiple cameras according to the host performance configuration capability, which is conducive to the invention being implemented in venues of different space sizes.

[0151] for Figure 2 For the embodiment shown, due to its Figure 1 The embodiments shown are basically similar, so the description is relatively simple. For relevant parts, see Figure 1 The partial description of the illustrated embodiment is sufficient and has corresponding technical effects.

[0152] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0153] Another embodiment of the present invention further provides a multi-camera spatiotemporal joint synchronous acquisition and processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the methods provided in the above-mentioned method embodiments when executing the computer program. Figure 1 Steps S11 to S15 shown, or, Figure 2 Steps S21 to S25 shown

[0154] Another embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the methods provided by the above method embodiments are implemented.

[0155] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0157] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, any one of the claimed embodiments may be used in any combination.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-camera spatiotemporal joint synchronous acquisition and processing method, characterized in that: The method comprises: Receive data packets uploaded by each camera, wherein the data packets include video data packets, and generate video data packet queues corresponding to each camera; Obtain the UTC absolute timestamp and display timestamp PTS of the first frame of video data packet in each video data packet queue, as well as the PTS of each subsequent video data packet in the queue except the first frame of video data packet; Calculate the incremental offset of the PTS of each subsequent video data packet corresponding to the PTS of the first frame video data packet, and calculate the UTC absolute timestamp of the corresponding subsequent video data packet according to the UTC absolute timestamp of the first frame video data packet and the incremental offset corresponding to each subsequent video data packet; Determine the relative synchronization degree of the video data collected by each camera according to the UTC absolute timestamp of the video data packets received by each camera at different video receiving times, and use the video receiving time with the best relative synchronization degree as the video synchronization time point; The video data packets received from each camera at the video synchronization time point are used as the first frame of video data packets to be output by the corresponding camera to generate a synchronized multi-viewpoint video sequence based on multiple cameras.

2. The method according to claim 1, characterized in that The data packet also includes an audio data packet; The method further comprises: Generate audio data packet queues corresponding to each camera; Get the PTS of each audio data packet in the audio data packet queue, calculate the incremental offset of the PTS of each audio data packet corresponding to the PTS of the first frame of the video data packet, and calculate the UTC absolute timestamp of each video data packet according to the UTC absolute timestamp of the first frame of the video data packet and the incremental offset corresponding to each audio data packet; The method of using the video data packets received from each camera at the video synchronization time point as the first frame of video data packets to be output by the corresponding camera to generate a synchronized multi-viewpoint video sequence based on multiple cameras includes: Obtain target audio data packets synchronized with the video data packets received by each camera at the video synchronization time point, and for the video data packet queue and audio data packet queue corresponding to each camera, generate a synchronized multi-viewpoint video sequence based on multiple cameras with the video data packet received at the video synchronization time point and the target audio data packet synchronized therewith as the starting positions.

3. The method according to claim 1 or 2, characterized in that: After calculating the UTC absolute timestamps of the subsequent video data packets according to the UTC absolute timestamp of the first frame of the video data packet and the incremental offsets corresponding to the subsequent video data packets, the method further includes: The UTC absolute timestamp and PTS of each data packet are bound to the corresponding data packet, and the host UTC timestamp when the host receives each data packet is used as the video receiving time of the corresponding data packet to update the video data packet queue.

4. The method according to claim 1 or 2, characterized in that: The method of determining the relative synchronization degree of the video data collected by each camera according to the UTC absolute timestamps of the video data packets received by each camera at different video receiving times, and taking the video receiving time with the best relative synchronization degree as the video synchronization time point, includes: The UTC absolute timestamp of the video data packets uploaded by each camera received by the host at the same video reception moment is used to construct a synchronization vector T = [t1, t2, ..., t n ], where t i is the UTC absolute timestamp of the video data packet captured by the i-th camera; According to the synchronization vector T at different video receiving moments, the relative synchronization error index α of the video data collected by each camera at each video receiving moment is calculated. The calculation formula A of α is: Where t is the video receiving time corresponding to the synchronization vector T; Find the video receiving time corresponding to the minimum value of the relative synchronization error index, and use the current video receiving time as the video synchronization time point.

5. The method according to claim 4, characterized in that Before calculating the relative synchronization error index α of the video data collected by each camera at each video receiving moment according to the synchronization vector T at different video receiving moments, the method further includes: According to the synchronization vector T corresponding to different video receiving moments, the asynchronous deviation index β of the video data collected by each camera at each video receiving moment is calculated. The calculation formula B of β is: B=max i |t i -t| When it is detected that the asynchronous deviation index β corresponding to a certain video receiving moment is less than the preset system nominal error, the step of calculating the relative synchronization error index α of the video data collected by each camera at each video receiving moment is performed for the synchronization vector T corresponding to subsequent different video receiving moments starting from the current video receiving moment.

6. The method according to claim 2, characterized in that The step of obtaining a target audio data packet synchronized with the video data packets received from each camera at the video synchronization time point comprises: Search the video data packet queue corresponding to each camera to obtain the host UTC timestamp and PTS of the target video data packet corresponding to the current camera. The target video data packet is the video data packet of the corresponding camera received at the video synchronization time point. The audio data packet queue of each camera is searched respectively to obtain a target audio data packet in the audio data packet queue whose host UTC timestamp is equal to or greater than the host UTC timestamp of the target video data packet corresponding to the current camera.

7. The method according to claim 6, characterized in that The method further comprises: After searching the audio data packet queue of each camera to obtain the target audio data packet whose host UTC timestamp is equal to or greater than the host UTC timestamp of the target video data packet corresponding to the current camera in the audio data packet queue, it is determined whether the PTS of the target video data packet corresponding to each camera is modified and rewritten when it is output as the first frame of video data; If the PTS of the target video data packet is modified and rewritten when it is output as the first frame of video data, the PTS of the target audio data packet is synchronously updated according to the offset time difference of the target audio data packet of the camera relative to the target video data packet; For the video data packet queue and audio data packet queue corresponding to each camera, the PTS of each data packet is synchronously updated packet by packet toward the tail of each queue according to the PTS of the target video data packet and the target audio data packet.

8. The method according to claim 1 or 2, characterized in that: The receiving of data packets uploaded by each camera includes: Before the host sends the acquisition instruction to each camera, perform video preloading processing on each camera to receive the data packets uploaded by each camera; and / or Send acquisition instructions to each camera to receive data packets uploaded by each camera.

9. A multi-camera spatiotemporal joint synchronous acquisition and processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images