Real-time multi-camera synchronization method and system based on multiple processes

Abstract

The embodiment of the invention discloses a multi-process-based real-time multi-path camera synchronization method and system, according to the embodiment of the invention, frames with the same time are paired according to timestamps, real-time unified processing of multi-path incoming frames is realized through a shared memory circular buffer area, each camera acquires the frames in an independent process and attaches the timestamps, and the real-time synchronization of the multi-path cameras is realized. And interacting data through a shared memory circular queue, and dynamically matching frames according to a timestamp by a synchronization process. Compared with an existing scheme, data interaction real-time performance is improved through shared memory zero copy, synchronization precision is improved through timestamp matching and dynamic error control, expansibility is enhanced through a multi-process parallel framework, the precision and real-time performance of multi-camera data synchronization are greatly improved, and the method is suitable for large-scale popularization and application. The requirement of three-dimensional reconstruction for multi-view data time consistency is met through targeted optimization, high efficiency and accuracy of subsequent three-dimensional reconstruction and data processing are ensured, and synchronization precision and real-time performance are remarkably improved.

Description

TECHNICAL FIELD

[0001] The embodiment of the application relates to the technical field of camera synchronization, in particular to a real-time multi-camera synchronization method and system based on multi-processes. BACKGROUND

[0002] The existing tennis electronic referee system generally adopts a monocular vision positioning scheme: the system acquires a venue video stream through a fixed single camera, and relies on an image processing module to realize tennis detection and landing point positioning. The core process includes camera calibration, moving target tracking and two-dimensional coordinate mapping.

[0003] The monocular vision scheme has inherent limitations. A single camera cannot obtain depth information, resulting in that landing point three-dimensional coordinate calculation relies on motion trajectory speculation. Such speculation can produce significant errors when the ball speed changes sharply. In addition, the single-view system is susceptible to occlusion interference, affecting the reliability of key frame capture.

[0004] Some improvement schemes attempt to introduce multi-camera, but these schemes mostly adopt non-real-time synchronization mechanisms. For example, time stamps are matched through software playback, or rely on simple network time protocol synchronization. Such methods have time errors of more than milliseconds, which cannot meet the precise capture requirements of high-speed moving objects.

[0005] The different synchronization of multiple cameras in time directly destroys the accuracy of three-dimensional reconstruction. When the tennis ball moves at a speed of more than 200 kilometers per hour, a time deviation of 1 millisecond can cause a spatial coordinate deviation of more than 5 centimeters. Existing synchronization technologies cannot control the time error of multiple video streams within milliseconds.

[0006] The data processing architecture of the existing system also has bottlenecks: the traditional scheme adopts inter-process communication to transfer video frames, frequent data copying increases transmission delay, and there is a lack of efficient shared resource management mechanism when multiple processes are collected in parallel, which is easy to cause access conflict or data loss. SUMMARY

[0007] To this end, the embodiment of the application provides a real-time multi-camera synchronization method and system based on multi-processes, to solve the technical problem that the existing technology has a large time error between each camera, cannot accurately capture the position of a fast-moving tennis ball, and thus affects the real-time and accuracy of trajectory reconstruction, and cannot meet the requirements of high-speed games.

[0008] In order to achieve the above purpose, the embodiment of the application provides the following technical scheme:

[0009] According to the first aspect of the embodiment of the application, a real-time multi-camera synchronization method based on multi-processes is provided, and the method comprises:

[0010] Create independent acquisition processes for each camera, acquire video streams and generate video frames in real time;

[0011] Attach accurate timestamps and frame sequence identifiers to each video frame;

[0012] Each acquisition process writes video frames into a circular buffer in shared memory, which contains a write index and a read index, and implements multi-process access control through a process lock;

[0013] The synchronization process monitors the shared memory of the camera in real time, matches the timestamps of the video frames output by multiple cameras, and selects frame pairs with timestamp differences within the set allowed error range for synchronization pairing;

[0014] Real-time output of successfully matched frame pairs to the processing queue.

[0015] Further, the synchronization process monitors all circular buffers in real time, and matches the timestamps of the video frames output by multiple cameras, including:

[0016] During the first synchronization, a pair of frames with a timestamp difference less than a first threshold value is selected as the initial synchronization frame;

[0017] During subsequent synchronization, a frame pair with a timestamp difference less than a second threshold value is selected based on a dynamic tolerance window.

[0018] Further, the circular buffer contains a write index and a read index, and implements multi-process access control through a process lock, including:

[0019] The structure of the circular buffer includes:

[0020] Each camera corresponds to an independent circular queue;

[0021] Each queue stores frame data packets, which contain image data, timestamps, and frame sequence identifiers;

[0022] The write index points to the current writable position, and the read index points to the current readable position.

[0023] Further, the dynamic tolerance window is generated in the following way:

[0024] According to the timestamp difference of the last successfully matched frame pair, the range of the second threshold value is dynamically adjusted.

[0025] Further, the first threshold value is a fixed value in milliseconds, and the second threshold value is an adaptive value based on historical synchronization errors.

[0026] Further, the synchronization process monitors the shared memory of the camera in real time, performs timestamp matching on the video frames output by the multiple cameras, selects frames with a timestamp difference within a set allowable error range for synchronous pairing, and further includes:

[0027] If no valid frame pair is matched within the preset time, a buffer overflow processing mechanism is triggered, the oldest frame is discarded, and the read index is updated.

[0028] Further, the timestamp is generated by a high-precision system clock, with a precision of microseconds.

[0029] According to a second aspect of an embodiment of the present application, a multi-process-based real-time multi-camera synchronization system is provided, and the system includes:

[0030] The acquisition module is configured to create an independent acquisition process for each camera, acquire a video stream in real time, and generate a video frame;

[0031] The frame data attachment module is configured to attach an accurate timestamp and a frame sequence identifier to each video frame;

[0032] The shared memory writing module is configured to write the video frame into a circular buffer in the shared memory by each acquisition process, the circular buffer includes a write index and a read index, and the multi-process access control is realized by a process lock;

[0033] The synchronization pairing module is configured to monitor the shared memory of the camera in real time by the synchronization process, perform timestamp matching on the video frames output by the multiple cameras, and select frames with a timestamp difference within a set allowable error range for synchronous pairing;

[0034] The output module is configured to output the matched frame pair to a processing queue in real time.

[0035] The embodiment of the present application has the following advantages:

[0036] The embodiment of the present application pairs the frames with equivalent time according to the timestamp, realizes real-time unified processing of multiple stream-in frames through a shared memory circular buffer, each camera independent process acquires a frame and attaches a timestamp, data is interacted through a shared memory circular queue, and the synchronization process dynamically matches the frames according to the timestamp. Compared with the existing scheme, it uses shared memory zero-copy to improve the real-time performance of data interaction, uses timestamp matching and dynamic error control to improve synchronization accuracy, uses a multi-process parallel architecture to enhance scalability, greatly improves the accuracy and real-time performance of multi-camera data synchronization, optimizes the demand for time consistency of multi-view data for three-dimensional reconstruction, ensures the efficiency and accuracy of subsequent three-dimensional reconstruction and data processing, and significantly improves the synchronization accuracy and real-time performance. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only exemplary, and for those skilled in the art, other drawings can also be obtained from the provided drawings without creative labor.

[0038] The structures, proportions, sizes, etc. shown in the specification are only used to cooperate with the content disclosed in the specification, so that those skilled in the art can understand and read, and are not used to limit the conditions that the present application can be implemented, so they do not have technical significance. Any modification of structure, change of proportion relationship or adjustment of size, without affecting the effect and purpose that the present application can produce, should still fall within the scope of the technical content disclosed by the present application.

[0039] Figure 1 It is a structural schematic diagram of the prior art;

[0040] Figure 2 It is a logical structure schematic diagram of a real-time multi-camera synchronization system based on multi-process provided by an embodiment of the present application;

[0041] Figure 3 It is a flowchart of a real-time multi-camera synchronization method based on multi-process provided by an embodiment of the present application. DETAILED DESCRIPTION

[0042] The embodiments of the present application will be described below by specific specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the specification. Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0043] The "Research on Tennis Landing Point Positioning Based on Monocular Vision" proposes a method for positioning the landing point of a tennis ball based on monocular vision, which is used to replace or supplement the high-priced professional landing point tracking system that is difficult to deploy in a low-cost environment, and reference Figure 1 Its core technology includes:

[0044] Camera calibration: using the corner points of the court lines as calibration feature points, solving the camera internal and external parameters and the inverse shadow parameter; according to the re-projection error, the calibration feature points are optimized.

[0045] Tracking landing point: identifying the foreground target through the background model and tracking algorithm, and determining the landing point frame according to the channel direction change of adjacent frames.

[0046] Drop point positioning: the drop point in the picture is relieved from the shadow, a projection model is applied, camera parameters are applied, and the drop point coordinates are converted to the three-dimensional world coordinate system.

[0047] The system monitors the tennis court through a single fixed camera. First, the image acquisition module continuously acquires video frames. The image preprocessing module performs grayscale, denoising, and edge enhancement processing on the image. Then, the tennis detection module detects the position of the moving object (tennis ball) using frame difference method, and extracts the two-dimensional pixel coordinates of the ball in the image frame through the centroid positioning method. The trajectory extraction module linearly fits the pixel positions of the ball in consecutive frames to establish a two-dimensional trajectory.

[0048] To determine whether the tennis ball has landed or is within the court, the motion analysis module extracts the motion trend of the ball, and determines the possible drop point frame according to the sharp decrease in speed or the extreme point in the y direction. Then, the court line detection module detects the boundary line in the image based on Hough transform, and maps the drop point pixel coordinates to the actual court coordinate range.

[0049] Finally, the drop point judgment module compares the mapping result with the predefined court area, and outputs the penalty result.

[0050] This technology is mainly applicable to standard tennis court environment, and can realize a certain degree of automation in drop point detection through monocular vision to complete the recognition and judgment of tennis drop point, which seriously depends on stable image acquisition and image processing algorithm.

[0051] The existing tennis electronic referee system usually uses a single camera or multiple cameras in a non-real-time synchronous manner, and the penalty depends on a large number of image processing and playback operations, which results in a large time error between cameras, cannot accurately capture the position of the fast-moving tennis ball, and further affects the real-time and accuracy of trajectory reconstruction, and cannot meet the requirements of high-speed competition.

[0052] In order to solve the technical problems that there is a large time error between each camera, the position of the fast-moving tennis ball cannot be accurately captured, which further affects the real-time and accuracy of trajectory reconstruction, and cannot meet the requirements of high-speed competition.

[0053] The application adopts a cyclic buffer structure based on shared memory, realizes real-time synchronization of multiple camera video frames through a strict timestamp matching mechanism, effectively reduces the time error to the millisecond level, significantly improves the accuracy and real-time of data synchronization, and can efficiently solve the real-time synchronization problem of multi-camera data acquisition.

[0054] Reference Figure 2The embodiment of the application discloses a multi-process-based real-time multi-channel camera synchronization system, which comprises a collection module 1, a frame data appending module 2, a shared memory writing module 3, a synchronization pairing module 4 and an output module 5.

[0055] Corresponding to the above-mentioned multi-process-based real-time multi-channel camera synchronization system, the embodiment of the application further discloses a multi-process-based real-time multi-channel camera synchronization method.

[0056] Reference Figure 3 The embodiment of the application discloses a multi-process-based real-time multi-channel camera synchronization method, which creates an independent collection process for each camera, collects video streams in real time and generates video frames, appends accurate time stamps and frame sequence identifiers to each video frame, writes the video frames into a circular buffer in shared memory by each collection process, the circular buffer comprises a write index and a read index, and realizes multi-process access control through a process lock, a synchronization process monitors the shared memory of the camera in real time, performs time stamp matching on the video frames output by the multi-channel camera, selects frames with a time stamp difference within a set allowable error range for synchronization pairing, and outputs the matched frame pair to a processing queue in real time.

[0057] Each camera collects video frames in real time through an independent process, and each frame is accompanied by an accurate time stamp. A shared memory circular queue structure is adopted, each camera process writes the collected frames into shared memory, and the frame data comprises a frame number, a time stamp and image data.

[0058] The shared memory structure comprises a write index and a read index, and a process lock (Lock) is used to prevent multi-process access competition. The synchronization process monitors the shared memory of the two cameras in real time, selects frames with a time stamp difference within a set allowable error range for synchronization pairing by comparing the time stamp of each frame.

[0059] Strict time matching logic is adopted for initial synchronization to ensure that the time stamps of the first paired frames of the two cameras are strictly close. The frame pair after synchronization is output to a processing queue in real time for subsequent three-dimensional reconstruction algorithm.

[0060] The application greatly improves the accuracy and real-time performance of multi-camera data synchronization through an efficient shared memory mechanism, accurate frame index and time stamp pairing, and ensures the efficiency and accuracy of subsequent three-dimensional reconstruction and data processing.

[0061] In the embodiment of the application, the synchronization of the multi-channel camera can also be realized through a PTP clock synchronization protocol or an NTP network synchronization mechanism, and a hardware trigger rule can also be used to solve the combination shooting synchronization problem.

[0062] While the application has been described in detail and with reference to specific embodiments thereof, it will be apparent to one skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope thereof. It is therefore intended that whatever lies within the scope of the application be covered by the appended claims and that there be no intent of any sort to limit the application except as can be explicitly expressed in those claims.

Claims

1. A real-time multi-channel camera synchronization method based on multi-process, characterized in that, The method includes: Create an independent acquisition process for each camera to capture video streams and generate video frames in real time; Attach a precise timestamp and frame sequence identifier to each video frame; Each acquisition process writes video frames into a circular buffer in shared memory. The circular buffer contains write and read indices and uses process locks to implement multi-process access control. The synchronization process monitors the shared memory of the cameras in real time, performs timestamp matching on the video frames output by multiple cameras, and selects frames with timestamp differences within the set allowable error range for synchronization pairing. Successfully matched frame pairs are output to the processing queue in real time.

2. The real-time multi-channel camera synchronization method based on multi-process as described in claim 1, characterized in that, The synchronization process monitors all circular buffers in real time and performs timestamp matching on video frames output from multiple cameras, including: During the initial synchronization, a pair of frames with a timestamp difference less than the first threshold are selected as the initial synchronization frames. During subsequent synchronization, frame pairs with timestamp differences less than the second threshold are selected based on a dynamic tolerance window.

3. The real-time multi-channel camera synchronization method based on multi-process as described in claim 1, characterized in that, The circular buffer contains write and read indices and implements multi-process access control through process locks, including: The structure of the circular buffer includes: Each camera corresponds to an independent circular queue; Each queue stores a frame data packet, which contains image data, a timestamp, and a frame sequence identifier. The write index points to the current writable position, and the read index points to the current readable position.

4. The real-time multi-channel camera synchronization method based on multi-process as described in claim 2, characterized in that, The dynamic tolerance window is generated as follows: The range of the second threshold is dynamically adjusted based on the timestamp difference of the previous successfully matched frame pair.

5. The real-time multi-channel camera synchronization method based on multi-process as described in claim 4, characterized in that, The first threshold is a fixed value at the millisecond level, and the second threshold is an adaptive value based on historical synchronization errors.

6. The real-time multi-channel camera synchronization method based on multi-process as described in claim 1, characterized in that, The synchronization process monitors the shared memory of the cameras in real time, performs timestamp matching on video frames output from multiple cameras, selects frames with timestamp differences within a set allowable error range for synchronization pairing, and also includes: If no valid frame pair is matched within the preset time, the buffer overflow handling mechanism is triggered, the oldest frame is discarded and the read index is updated.

7. The real-time multi-channel camera synchronization method based on multi-process as described in claim 1, characterized in that, The timestamp is generated by a high-precision system clock with an accuracy of microseconds.

8. A real-time multi-channel camera synchronization system based on multi-process, characterized in that, The system includes: The acquisition module is used to create an independent acquisition process for each camera, acquire video streams in real time and generate video frames; The frame data appending module is used to append precise timestamps and frame sequence identifiers to each video frame; The shared memory write module is used by each acquisition process to write video frames into a circular buffer in shared memory. The circular buffer contains a write index and a read index, and multi-process access control is implemented through a process lock. The synchronization pairing module is used to synchronize the shared memory of the cameras in real time, perform timestamp matching on the video frames output by multiple cameras, and select frames whose timestamp differences are within the set allowable error range for synchronization pairing. The output module is used to output successfully matched frame pairs to the processing queue in real time.

Images