A camera video acquisition time synchronization device and method based on external triggering

Abstract

The invention discloses a camera video acquisition time synchronization device based on external triggering, which has high precision, low cost and is easy to implement. It includes: a video acquisition processor, which is composed of a control module, a data processing module, a recording and output module; and a plurality of cameras; the control module receives an external standard time signal, generates an external trigger signal of a set frequency according to the high-precision clock inside the control module based on the time signal, generates a timestamp corresponding to the time point of each trigger signal, and transmits the timestamp data to the data processing module; the data processing module receives the timestamp information of the control module, and communicates and interacts with each camera, and realizes the correspondence of the timestamp to each frame image of the video with the camera, and records or outputs the video data after time synchronization; the recording and output module receives the video data of the data processing module, records the video data according to the file management requirements, and converts the video data according to the output format requirements and outputs it.

Description

Technical Field

[0001] The present invention relates to the technical field of embedded data processing, in particular to a camera video acquisition time synchronization device based on external triggering. The present invention also provides a camera video acquisition time synchronization method based on external triggering. Background Art

[0002] An embedded system is a special-purpose computer with specific functions. With the rapid development of the chip electronics industry, embedded systems have been increasingly used in various fields such as industrial control, data acquisition, remote monitoring, smart terminals, and network communications. The acquisition of video data by embedded systems is the basis of various applications. When different data sources (such as video cameras, sensors, control systems, etc.) need to be analyzed or operated on the same time basis to ensure data accuracy and consistency, time synchronization is required during video data acquisition. For example, the collaborative work of multiple sensors (such as cameras, radars, lidars, etc.) for autonomous driving, industrial manufacturing vibration monitoring and image synchronization, and synchronous acquisition of scientific experimental data (such as images, temperature, pressure, flow, etc.).

[0003] When data acquisition requires high time synchronization accuracy, the collected data is usually transmitted to the processing unit through a high-speed bus. The processing unit performs time alignment or adds time information such as timestamps to the data according to the received external time code information or the system time after periodic synchronization with the external time code. This type of method is suitable for high-frequency and low-bandwidth data acquisition application scenarios, such as temperature, pressure and other physical quantity data acquisition; however, the video image data is large, and the bus bandwidth and transmission speed are required to be very high after the collection of multiple different video data is collected and synchronized, and the scalability is not good. Therefore, the time synchronization processing can be performed separately in each acquisition front end, for example, the camera realizes time synchronization according to its own synchronized system time. In this method, the system time of each camera can be synchronized with high accuracy through the IEEE 1588 time protocol, and the IEEE1588 time synchronization function requires a specific chip to be implemented, which increases the cost of the embedded system, and the system time of multiple cameras will drift, and it needs to be synchronized periodically; on the other hand, the camera performs time alignment or attachment timestamp processing after imaging, and the process of calling to obtain the system time is a software operation, which is difficult to control all camera processing processes or the same camera processing time to be strictly consistent. Therefore, it is urgent to develop a camera video acquisition time synchronization device in view of the characteristics of camera large data volume and low frequency data acquisition. Summary of the invention

[0004] In view of the above problems, the present invention provides a camera video acquisition time synchronization device based on external triggering, which utilizes the high-precision controllable characteristics of the camera hardware external triggering, and obtains the high-precision moment of each frame of video image according to the time period of the external trigger signal generated by the system itself. It has the advantages of high precision, low cost and easy implementation.

[0005] A camera video acquisition time synchronization device based on external triggering, characterized in that it comprises:

[0006] Video acquisition processor, which consists of a control module, a data processing module, a recording and output module;

[0007] and several cameras;

[0008] The control module receives an external standard time signal, uses the time signal as a reference to generate an external trigger signal of a set frequency according to a high-precision clock inside the control module, generates a timestamp corresponding to the time point of each trigger signal, and transmits the timestamp data to the data processing module; the data processing module receives the timestamp information of the control module, communicates and interacts with each camera, and works with the camera to correspond the timestamp to each frame image of the video, and records or outputs the video data after time synchronization; the recording and output module receives the video data of the data processing module, records the video data according to the file management requirements, and converts the video data according to the output format requirements and outputs it;

[0009] Each camera receives an external trigger signal from a control module of a video acquisition processor, performs imaging exposure according to the external trigger signal, and transmits the acquired data to a data processing module of the video acquisition processor.

[0010] It is further characterized by:

[0011] The control module of the video acquisition processor is implemented by an FPGA with clock management and control capabilities. The control module receives a high-precision external time signal, analyzes it, and adjusts the reference time according to the processing time. Taking the reference time as the starting point, a periodic external trigger signal is generated and output to the camera. At the same time, the time value corresponding to each external trigger signal is formed according to the reference time value and the generated external trigger signal count value, and is output to the data processing module together with the external trigger count value;

[0012] The external trigger signal uses the FPGA internal clock as the clock source and uses the FPGA built-in differential pins or differential drivers to generate differential signals to ensure the time accuracy is at the nanosecond level;

[0013] All cameras expose and image each frame of video image according to the external trigger signal, and count them sequentially, and record the count value as the image serial number;

[0014] The data processing module receives each frame of raw video image data and each frame image serial number collected by the camera, compares the image serial number with the external trigger signal count value received from the control module, obtains the external trigger signal time value corresponding to each frame image, binds the time value with the image data, and completes subsequent processing;

[0015] If the data transmission bandwidth between the data processing module and the camera is limited, the time information of each external trigger is sent from the data processing module to each camera, and each camera performs the above-mentioned corresponding processing of matching each frame of video image with the time value of the external trigger signal locally, and after completing the relevant image processing on the camera, the processed data is transmitted to the data processing module;

[0016] Each time the camera is exposed by external trigger, it also records the local system time of the camera and binds it to the serial number of each frame of the video image. When the external trigger is disturbed and the camera exposure is inaccurate, the difference between the local system time of each frame image is the inverse of the frame rate to determine whether the serial number of the frame image matches the external trigger count value generated by the video acquisition processor control module. If they do not match, the image serial number is corrected according to the local time of the camera system.

[0017] The camera video acquisition time synchronization method based on external triggering adopts a camera video acquisition time synchronization device based on external triggering, and is characterized in that it comprises the following steps:

[0018] Step 1: System startup preparation: After the system is started, wait for the external time signal for time synchronization;

[0019] Step 2: determine whether the control module of the video acquisition processor receives an external time signal. If no external time signal is received, go to step 1 and continue waiting; otherwise, go to step 3.

[0020] Step 3: The control module starts the clock cycle counting and parses the time information;

[0021] Step 4: The control module adjusts the calculation reference time;

[0022] Step 5: The control module generates a periodic external trigger signal and accumulates and counts the number of signals;

[0023] Step 6: Each camera generates a video image by exposing the image according to the external trigger signal, and records the video image serial number;

[0024] Step 7: The control module calculates the timestamp corresponding to each external trigger signal and outputs it to the data processing module. This step is performed simultaneously with step 6.

[0025] Step 8: Perform time synchronization and subsequent image processing on the video image according to the external trigger signal timestamp and the video image serial number;

[0026] Step 9: The recording and output module records the video data according to the file management requirements, and converts the video data according to the output format requirements and then outputs it.

[0027] After adopting the present invention, it takes the external time synchronization signal as a reference, utilizes the high-precision controllable characteristics of the camera hardware external trigger, and obtains the high-precision moment of each frame of video image according to the time period of the external trigger signal generated by the system itself, which has the advantages of high precision, low cost and easy implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram showing the structure of the device of the present invention;

[0029] Figure 2 is a flow chart of the method of the present invention;

[0030] Figure 3 A schematic diagram of the signal timing corresponding to the method of the present invention; DETAILED DESCRIPTION

[0031] A camera video acquisition time synchronization device based on external triggering, see Figure 1 , which includes a video acquisition processor and several cameras;

[0032] The video acquisition processor consists of a control module, a data processing module, and a recording and output module;

[0033] The control module receives an external standard time signal, and uses the time signal as a reference to generate an external trigger signal of a set frequency according to the high-precision clock inside the control module, and generates a timestamp corresponding to the time point of each trigger signal, and transmits the timestamp data to the data processing module; the data processing module receives the timestamp information of the control module, and communicates and interacts with each camera, and works with the camera to correspond the timestamp to each frame image of the video, and records or outputs the video data after time synchronization; the recording and output module receives the video data of the data processing module, records the video data according to the file management requirements, and converts the video data according to the output format requirements and outputs it;

[0034] Each camera receives an external trigger signal from a control module of a video acquisition processor, performs imaging exposure according to the external trigger signal, and transmits the acquired data to a data processing module of the video acquisition processor.

[0035] In specific implementation, the control module of the video acquisition processor is implemented by an FPGA with clock management and control capabilities. The control module receives high-precision external time signals, such as pulse signals or IRIG-B time codes, analyzes them, and adjusts the reference time according to the processing time. Taking the reference time as the starting point, a periodic external trigger signal is generated and output to the camera. At the same time, a time value corresponding to each external trigger signal is formed according to the reference time value and the count value of the generated external trigger signal, and is output to the data processing module together with the external trigger count value.

[0036] The external trigger signal uses the FPGA internal clock as the clock source and uses the FPGA built-in differential pins or differential drivers to generate differential signals to ensure the time accuracy is at the nanosecond level;

[0037] All cameras expose and image each frame of video image according to the external trigger signal, and count them sequentially, and record the count value as the image serial number;

[0038] The data processing module receives each frame of raw video image data and the serial number of each frame of image collected by the camera, compares the image serial number with the count value of the external trigger signal received from the control module, obtains the time value of the external trigger signal corresponding to each frame of image, binds the time value with the image data, and completes subsequent processing, such as OSD superimposition of time code label, image compression, etc.

[0039] If the data transmission bandwidth between the data processing module and the camera is limited, the time information of each external trigger is sent from the data processing module to each camera, and each camera performs the corresponding processing of matching each frame of the video image with the time value of the external trigger signal locally, and completes the OSD superimposed time code label, image compression and other related image processing on the camera, and then transmits the processed data to the data processing module;

[0040] Each time the camera is exposed by external trigger, it also records the local system time of the camera and binds it to the serial number of each frame of the video image. When the external trigger is disturbed and the camera exposure is inaccurate, the difference between the local system time of each frame image is the inverse of the frame rate to determine whether the serial number of the frame image matches the external trigger count value generated by the video acquisition processor control module. If they do not match, the image serial number is corrected according to the local time of the camera system.

[0041] The camera video acquisition time synchronization method based on external triggering adopts the camera video acquisition time synchronization device based on external triggering, see Figure 2 , which comprises the following steps:

[0042] Step 1: System startup preparation: After the system is started, wait for the external time signal for time synchronization;

[0043] Step 2: determine whether the control module of the video acquisition processor receives an external time signal. If no external time signal is received, go to step 1 and continue waiting; otherwise, go to step 3.

[0044] Step 3: The control module starts the clock cycle counting and parses the time information;

[0045] The control module is implemented using an FPGA with strong clock management and control capabilities. The external time signal uses a high-precision pulse signal or IRIG-B time code signal. Due to the different types and protocols of external time signals, it takes different amounts of time to receive and parse the external time signal. Therefore, when the FPGA receives an external clock signal (such as Figure 3 The external time signal (t0) starts counting the clock cycle, i.e., starts timing. When the parsed time signal is completed, the clock cycle counting ends, i.e., ends timing. The parsed synchronization time is t0.

[0046] Step 4: The control module adjusts the calculation reference time;

[0047] Calculate the FPGA processing time T based on the number of FPGA clock cycles in which the FPGA receives and processes the external time signal in step 3. FP =Num T *ΔT FP , where Num T is the number of FPGA clock counts, ΔT FP is the processing clock cycle length of the FPGA, the reference time t1 = t0 + T FP ;

[0048] Step 5: The control module generates a periodic external trigger signal and accumulates and counts the number of signals;

[0049] The control module generates an external trigger signal by FPGA according to the set video image acquisition frequency (i.e., image frame rate). In a specific embodiment, the external trigger signal uses the internal clock of FPGA as the clock source, and uses the built-in differential pin or differential driver of FPGA to generate a differential signal to ensure the time accuracy at the nanosecond level. FPGA counts the generated external trigger signal, such as Figure 3 As shown in p i =i is the count value (i.e., serial number) of the external trigger signal, where i=1, 2, 3, ..., N;

[0050] Step 6: Each camera generates a video image by exposing the image according to the external trigger signal, and records the video image serial number; the video image serial number starts from the first external trigger signal received and counts up in sequence, such as Figure 3 As shown in j =j, where j=1, 2, 3, ..., M;

[0051] Step 7: The control module calculates the timestamp corresponding to each external trigger signal and outputs it to the data processing module. This step is performed simultaneously with step 6.

[0052] The timestamp of the external trigger signal is (p i , t i ), p i is the count value (i.e., serial number) of the external trigger signal in step 5, t i is the time relative to the reference time when the FPGA generates the i-th external trigger signal. Since the FPGA clock has high precision and high processing frequency, it can be directly obtained by t i =t1+(i-1)*T p Calculated, where T p The exposure interval of the set video image, that is, the inverse of the video frame rate;

[0053] Step 8: Perform time synchronization and subsequent image processing on the video image according to the timestamp of the external trigger signal and the serial number of the video image; the method includes the following:

[0054] Method 1: The data processing module receives each frame of raw video image data and each frame image serial number v collected by the camera j , compare and match the image serial number v j And the timestamp of the external trigger signal received from the control module i , when the video image v j =p i When , the exposure time of the video frame is t i , that is, the time code of this frame is t i , bind it with the image data, and the data processing module completes the subsequent processing, such as OSD overlay time code tag and image compression;

[0055] Method 2: If the data transmission bandwidth between the data processing module and the camera is limited, the data processing module can send the timestamp of each external trigger to each camera in real time, and each camera will perform local processing on the received timestamp (p i , t i ) and its self-generated image serial number v j The above matching process is performed to obtain the time code of each frame of the image, and after completing the relevant image processing of OSD superimposed time code label and image compression on the camera, the processed data is transmitted to the data processing module;

[0056] When the specific method is implemented, the camera also records the camera local system time tc each time the external trigger exposure j , bound to the serial number, so that adjustment and correction can be made when the external trigger is inaccurate. Specifically, in the above methods 1 and 2, when the camera exposure is inaccurate due to interference with the external trigger, for example Figure 3 The last group of figures in the figure shows the case of under-exposure triggering of the camera. The difference between the local system time of each frame image is the inverse of the frame rate to determine whether the serial number of the frame image matches the external trigger count value generated by the video acquisition processor control module. If not, the image serial number is corrected according to the local time of the camera system. The specific processing process is: each time v j 、p i Before matching, calculate the difference between the current frame exposure time and the previous frame exposure time Δtc=tc according to the camera local system time j -tc j-1 , if Δtc=T p , the camera triggers the exposure normally and no adjustment or correction is required; if Δtc<T p , the current frame is a false trigger, and the image data of the current frame is removed; if Δtc>T p , then the trigger exposure imaging omission occurs before the current frame, such as Figure 3 The camera in the last group of pictures in the figure misses the trigger exposure imaging in the third frame, so we can further judge Δtc and T p Multiple relationship, when Δtc=L*T p When , it means that the L frame image trigger exposure imaging is missed, then the external trigger signal timestamp (p i , t i ) and then make adjustments at intervals of L timestamps, i.e. v j Match corresponding p i+L , video image v j The synchronized time code is t i+L ;

[0057] Step 9: The recording and output module records the video data according to the file management requirements, and converts the video data according to the output format requirements and then outputs it.

[0058] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0059] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A camera video acquisition time synchronization device based on external triggering, characterized in that: It includes: Video acquisition processor, which consists of a control module, a data processing module, a recording and output module; and several cameras; The control module receives an external standard time signal, generates an external trigger signal of a set frequency based on the time signal and the high-precision clock inside the control module, generates a timestamp corresponding to the time point of each trigger signal, and transmits the timestamp data to the data processing module; The data processing module receives the timestamp information from the control module, communicates and interacts with each camera, and works with the camera to correspond the timestamp to each frame of the video, and records or outputs the time-synchronized video data; The recording and output module receives the video data from the data processing module, records the video data according to the file management requirements, and converts and outputs the video data according to the output format requirements; Each camera receives an external trigger signal from a control module of a video acquisition processor, performs imaging exposure according to the external trigger signal, and transmits the acquired data to a data processing module of the video acquisition processor.

2. The camera video acquisition time synchronization device based on external triggering according to claim 1, characterized in that: The control module of the video acquisition processor is implemented using an FPGA with clock management and control capabilities. The control module receives a high-precision external time signal, parses it, and adjusts the reference time according to the processing time. Taking the reference time as the starting point, it generates a periodic external trigger signal and outputs it to the camera. At the same time, the time value corresponding to each external trigger signal is formed according to the reference time value and the count value of the generated external trigger signal, and is output to the data processing module together with the external trigger count value.

3. The camera video acquisition time synchronization device based on external triggering according to claim 2, characterized in that: The external trigger signal uses the FPGA internal clock as the clock source and uses the FPGA built-in differential pins or differential drivers to generate differential signals.

4. The camera video acquisition time synchronization device based on external triggering according to claim 1, characterized in that: All cameras expose and image each frame of video image according to the external trigger signal, and count them sequentially, and record the count value as the image serial number.

5. The camera video acquisition time synchronization device based on external triggering according to claim 4, characterized in that: The data processing module receives each frame of original video image data and the serial number of each frame of image collected by the camera, compares the image serial number and the external trigger signal count value received from the control module, obtains the external trigger signal time value corresponding to each frame of image, binds the time value with the image data, and completes subsequent processing.

6. The camera video acquisition time synchronization device based on external triggering according to claim 5, characterized in that: If the data transmission bandwidth between the data processing module and the camera is limited, the time information of each external trigger is sent from the data processing module to each camera, and each camera performs the corresponding processing of matching each frame of video image with the time value of the external trigger signal locally, and after completing the relevant image processing on the camera, the processed data is transmitted to the data processing module.

7. A camera video acquisition time synchronization device based on external triggering according to claim 5 or 6, characterized in that: Each time the camera is exposed by external trigger, it also records the local system time of the camera and binds it to the serial number of each frame of the video image. When the external trigger is disturbed and the camera exposure is inaccurate, the difference between the local system time of each frame image is the inverse of the frame rate to determine whether the serial number of the frame image matches the external trigger count value generated by the video acquisition processor control module. If they do not match, the image serial number is corrected according to the local time of the camera system.

8. A camera video acquisition time synchronization method based on external triggering, which adopts a camera video acquisition time synchronization device based on external triggering as claimed in any one of claims 1 to 7, characterized in that: It includes the following steps: Step 1: System startup preparation: After the system is started, wait for the external time signal for time synchronization; Step 2: determine whether the control module of the video acquisition processor receives an external time signal. If no external time signal is received, go to step 1 and continue waiting; otherwise, go to step 3. Step 3: The control module starts the clock cycle counting and parses the time information; Step 4: The control module adjusts the calculation reference time; Step 5: The control module generates a periodic external trigger signal and accumulates and counts the number of signals; Step 6: Each camera generates a video image by exposing the image according to the external trigger signal, and records the video image serial number; Step 7: The control module calculates the timestamp corresponding to each external trigger signal and outputs it to the data processing module. This step is performed simultaneously with step 6. Step 8: Perform time synchronization and subsequent image processing on the video image according to the external trigger signal timestamp and the video image serial number; Step 9: The recording and output module records the video data according to the file management requirements, and converts the video data according to the output format requirements and then outputs it.

Images