Video synchronization method and apparatus applicable to vehicles

Abstract

This invention provides a video synchronization method and apparatus applicable to vehicles, solving the problem of video synchronization issues caused by different transmission delays. [Solution] A video synchronization method comprising the steps of: calculating the number of buffer frames for each of a plurality of buffer queues in the memory on the vehicle based on a plurality of delay times between a plurality of shooting circuits and a processor; controlling a plurality of shooting circuits with a transmission circuit to shoot the area around the vehicle and generate a plurality of video streams, and controlling the transmission circuit to send each of the plurality of video streams to a plurality of buffer queues; detecting the number of queue frames of the video streams temporarily stored in each buffer queue; and determining whether or not to retrieve the first frame of the video streams temporarily stored in each of the plurality of buffer queues based on the number of buffer frames and the number of queue frames of each buffer queue, thereby integrating all first frames into a composite image.

Description

Technical Field

[0001] The present invention relates to video processing technology, and particularly to a video synchronization method and apparatus applicable to vehicles.

Background Art

[0002] Currently, in many cases, surveillance cameras are installed in vehicles. The most common installation method is to mount the surveillance camera in front of or behind the vehicle so that the user can view the captured video on a display inside the vehicle. A plurality of conventional surveillance cameras can simultaneously capture the road conditions in the front and the rear, but these videos are separately transmitted to and stored in the in-vehicle system, and the in-vehicle system usually does not record the time of the videos captured by each surveillance camera. Also, when these surveillance cameras transmit videos to the in-vehicle system, there may be different transmission delays between these surveillance cameras and the in-vehicle system. Different transmission delays cause a synchronization deviation in these videos captured simultaneously. Therefore, how to prevent the synchronization deviation of videos caused by different transmission delays is a problem that those skilled in the art urgently need to solve.

Summary of the Invention

Problems to be Solved by the Invention

[0003] The main object of the present invention is to provide a video synchronization method and apparatus applicable to vehicles, and to solve the problem of synchronization deviation of videos caused by different transmission delays.

Means for Solving the Problems

[0004] To achieve the above object, the video synchronization method applicable to vehicles proposed by the present invention includes the following steps. Step a) Detect a plurality of delay times between a plurality of imaging circuits and the processor by a processor on the vehicle and a transmission circuit on the vehicle. Step b) The processor calculates the number of buffer frames for each of the multiple buffer queues in the memory on the vehicle based on the multiple delay times, where each of the multiple buffer queues corresponds to one of the multiple imaging circuits. Step c) The processor and the transmission circuit control the plurality of shooting circuits to capture images of the area around the vehicle from multiple shooting directions to generate multiple video streams, and further control the transmission circuit to transmit each of the multiple video streams to the plurality of buffer queues in the memory. Step d) The processor detects the number of queue frames of the video stream temporarily stored in each buffer queue. Step e) The processor determines whether or not to retrieve the first frame of the video stream temporarily stored in each of the plurality of buffer queues, based on the number of buffer frames and the number of queue frames in each buffer queue, where the first frame refers to the video frame in each video stream that was first stored in the corresponding buffer queue. Step f) If it is determined to retrieve the first frame of the video stream temporarily stored in each of the plurality of buffer queues, the processor integrates all of the first frames into the composite video. Step g) If it is determined that the first frame of the video stream temporarily stored in each buffer queue is not to be retrieved, steps c) through e) are executed again.

[0005] To achieve the above objective, the video synchronization device applied to a vehicle as proposed by the present invention includes the following devices. A device comprising multiple imaging circuits, installed on a vehicle, and configured to capture images of the area around the vehicle from multiple imaging directions. A transmission circuit, which is connected to the plurality of imaging circuits and configured to receive a video stream. A memory device connected to the transmission circuit and configured to store a plurality of commands and a plurality of buffer queues, wherein each of the plurality of buffer queues corresponds to one of the plurality of imaging circuits. A processor, configured to be connected to the transmission circuit and the memory, and to access the plurality of commands and to perform the following steps. Step a) The transmission circuit detects multiple delay times between the multiple imaging circuits and the processor. Step b) Based on the multiple delay times, calculate the number of buffer frames for each of the multiple buffer queues in the vehicle. Step c) The transmission circuit controls the multiple shooting circuits to capture images of the area around the vehicle from multiple shooting directions to generate multiple video streams, and further controls the transmission circuit to transmit each of the multiple video streams to the multiple buffer queues. Step d) Detect the number of queue frames of the video stream temporarily stored in each buffer queue. Step e) Based on the number of buffer frames and the number of queue frames in each of the buffer queues, it is determined whether or not to retrieve the first frame of the video stream temporarily stored in each of the plurality of buffer queues, where the first frame refers to the video frame in each of the video streams that was first stored in the corresponding buffer queue. Step f) If it is determined to retrieve the first frame of the video stream temporarily stored in each of the multiple buffer queues, all of the first frames are integrated into the composite video. Step g) If it is determined that the first frame of the video stream temporarily stored in each buffer queue is not to be retrieved, steps c) through e) are executed again.

[0006] Compared to conventional technology, the present invention first detects the delay time of the shooting circuit for capturing images around the vehicle, sets the number of buffer frames for each buffer queue, and then compares the number of queue frames and buffer frames for each buffer queue to determine the timing for extracting the first frame from each buffer queue and synchronizing the surrounding image. Therefore, the present invention can solve the problem of image synchronization errors caused by different transmission delays between each shooting circuit. [Brief explanation of the drawing]

[0007] [Figure 1] This is a block diagram of a video synchronization device applied to a vehicle in some embodiments of the present invention. [Figure 2] This is a schematic diagram of the installation method of multiple imaging circuits in some embodiments of the present invention. [Figure 3] This is a schematic diagram of a plurality of buffer queues in some embodiments of the present invention. [Figure 4] This is a flowchart of a video synchronization method applied to a vehicle in some embodiments of the present invention. [Figure 5] This is a flowchart of several steps included in one step of a video synchronization method applied to a vehicle in some embodiments of the present invention. [Figure 6] This is a flowchart of several steps included in another step of a video synchronization method applied to a vehicle in some embodiments of the present invention. [Figure 7] This is a flowchart of several steps included in another step of a video synchronization method applied to a vehicle in some embodiments of the present invention. [Figure 8] This is a schematic diagram of a composite image in some embodiments of the present invention. [Modes for carrying out the invention]

[0008] Please refer to Figure 1, which is a block diagram of a video synchronization device 100 applied to a vehicle in some embodiments of the present invention. As shown in Figure 1, the video synchronization device 100 applied to a vehicle (hereinafter referred to as the video synchronization device 100) includes a plurality of shooting circuits 111 to 11n, a transmission circuit 120, a memory 130, and a processor 140, where n is a positive integer greater than 1. The transmission circuit 120 is connected to the shooting circuits 111 to 11n, the memory 130, and the processor 140. The memory 130 is connected to the processor 140. In some embodiments, the shooting circuits 111 to 11n are all connected to the transmission circuit 120 by wireless connection. It should be noted that the wireless connection method may result in multiple transmission delays between the shooting circuits 111 to 11n and the transmission circuit 120, and these transmission delays may be the same or different (e.g., 200 milliseconds, 150 milliseconds, and 100 milliseconds).

[0009] In some other embodiments, some of the imaging circuits 111-11n are connected to the transmission circuit 120 by wireless connection, and other parts of the imaging circuits 111-11n are connected to the transmission circuit 120 by wired connection. It should be noted that the wireless connection method may result in multiple different transmission delays between some of the imaging circuits 111-11n and the transmission circuit 120 (e.g., 200 milliseconds and 100 milliseconds), while the wired connection method does not result in any transmission delay between other parts of the imaging circuits 111-11n and the transmission circuit 120 (i.e., the transmission delay is 0 seconds). It should be noted that if all of the imaging circuits 111-11n are connected to the transmission circuit 120 by wired connection to avoid transmission delay, a large amount of wiring cost will be incurred. Conversely, if all of the imaging circuits 111-11n are connected to the transmission circuit 120 by wireless connection, wiring costs can be significantly reduced, but transmission delay will occur.

[0010] In this embodiment, the shooting circuits 111 to 11n are each installed at the same or different locations on the vehicle and are used to capture images of the area around the vehicle from multiple shooting directions. In some embodiments, the shooting circuits 111 to 11n each correspond to multiple shooting directions. In other words, each shooting circuit can capture images from its corresponding shooting direction on the vehicle, thereby generating its own video stream (i.e., multiple consecutive images). In some embodiments, each shooting circuit can be installed at any location on the vehicle capable of capturing images of the area around the vehicle. In some embodiments, the shooting direction may be any direction toward the area around the vehicle. In some embodiments, the vehicle may be an automobile, electric bicycle, motorcycle, or bicycle. In some embodiments, each shooting circuit can be implemented by any video acquisition circuit (e.g., optical shooting circuit, infrared shooting circuit, or three-dimensional shooting circuit).

[0011] The installation methods for each shooting circuit will be explained below using specific examples. Please also refer to Figure 2, which is a schematic diagram of the installation methods for shooting circuits 111 to 114 in some embodiments of the present invention. Assuming that the video synchronization device 100 has four shooting circuits 111 to 114, as shown in Figure 2, shooting circuit 111 can be installed on the front edge of the vehicle VH, shooting circuit 112 can be installed on one side pillar of the vehicle VH, shooting circuit 113 can be installed on the other side pillar of the vehicle VH, and shooting circuit 114 can be installed on the tail edge of the vehicle VH.

[0012] Furthermore, each of the imaging circuits 111 to 114 has multiple imaging ranges 210 to 240, and the imaging directions of each imaging circuit 111 to 114 correspond to imaging ranges 210 to 240, respectively. That is, the imaging direction of imaging circuit 111 may be directly facing the imaging range 210 of imaging circuit 111, the imaging direction of imaging circuit 112 may be directly facing the imaging range 220 of imaging circuit 112, the imaging direction of imaging circuit 113 may be directly facing the imaging range 230 of imaging circuit 113, and the imaging direction of imaging circuit 114 may be directly facing the imaging range 240 of imaging circuit 114.

[0013] Returning to FIG. 1, in this embodiment, the transmission circuit 120 is used to receive video streams from each of the imaging circuits 111-11n. In some embodiments, the transmission circuit 120 can be implemented by any communication circuit for communication (e.g., a communication circuit of a controller area network (CAN), a Wi-Fi communication circuit, or a Bluetooth communication circuit).

[0014] In this embodiment, the memory 130 stores a plurality of buffer queues, and the plurality of buffer queues respectively correspond to the imaging circuits 111-11n. For example, the memory 130 can store n buffer queues. Here, the first buffer queue corresponds to the imaging circuit 111, the second buffer queue corresponds to the imaging circuit 112, the third buffer queue corresponds to the imaging circuit 11n, and so on by analogy. In some embodiments, the plurality of buffer queues are all first-in-first-out (FIFO) buffer areas in the memory 130. In some embodiments, each buffer queue temporarily stores a plurality of consecutive frames in the video stream generated by the corresponding imaging circuit. In other words, each time the transmission circuit 120 sequentially receives a plurality of frames of a video stream from one imaging circuit, the processor 140 can control the transmission circuit 120 to sequentially transmit these frames to the buffer queue corresponding to the imaging circuit. Therefore, the corresponding buffer queue can store these frames in a first-in-first-out manner.

[0015] Hereinafter, the buffer queue will be described using specific examples. Referring to FIG. 3 together, FIG. 3 is a schematic diagram of the buffer queues 131-13n in some embodiments of the present invention. As shown in FIG. 3, assuming that the buffer queues 131-13n respectively correspond to the imaging circuits 111-11n, the buffer queues 131-13n can temporarily store a plurality of frames of a plurality of video streams img1-imgn respectively generated by the imaging circuits 111-11n in a first-in-first-out manner.

[0016] Returning to FIG. 1, in some embodiments, the memory 130 further stores a plurality of commands, and the processor 140 executes the detailed steps described in the following paragraphs based on these commands. In some embodiments, the memory 130 can be implemented by, but is not limited to, a flash memory, a read-only memory, a hard disk, or other equivalent storage components. In some embodiments, the processor 140 can be implemented by, but is not limited to, a central processing unit (CPU), a micro control unit (MCU), a programmable logic controller (PLC), a system on chip (SoC), or a field programmable gate array (FPGA). In some embodiments, the memory 130 is further used to store the composite video described in the following paragraphs.

[0017] In some embodiments, the video synchronization device 100 may further include a display 150. The display 150 is connected to the processor 140 and is used to display the composite video described in the following paragraphs. In some embodiments, the display 150 can be implemented by any display panel or display circuit (e.g., a liquid crystal screen or a touch panel). In some embodiments, the transmission circuit 120, the memory 130, the processor 140, and the display 150 can be installed in an in-vehicle system (e.g., a controller area network) on a vehicle.

[0018] Referring to FIG. 4 together, FIG. 4 is a flowchart of a video synchronization method applied to a vehicle in some embodiments of the present invention. Here, the video synchronization method is applied to the video synchronization device 100 shown in FIG. 1.

[0019] As shown in Figure 4, the video synchronization method applied to the vehicle includes steps S410 to S460. First, in step S410, the processor 140 detects multiple delay times between the imaging circuits 111 to 11n and the processor 140 via the transmission circuit 120. In some embodiments, each delay time represents the time required for the corresponding imaging circuits 111 to 11n to transmit the video stream to the processor 140.

[0020] The detection of delay time will be explained further below. Please also refer to Figure 5, which is a flowchart of several steps S411 to S413 included in step S410 of a video synchronization method applied to a vehicle in some embodiments of the present invention. As shown in Figure 5, step S410 in Figure 4 includes steps S411 to S413.

[0021] In step S411, the processor 140 controls the transmission circuit 120 to send a timestamp to the imaging circuits 111-11n. In some embodiments, the processor 140 periodically generates a timestamp and simultaneously transmits the timestamp to the imaging circuits 111-11n, where the timestamp indicates the system time of the in-vehicle system connected to the video synchronization device 100. In other words, the processor 140 can periodically use the timestamp to confirm the delay time corresponding to each imaging circuit 111-11n.

[0022] In step S412, the processor 140 controls the transmission circuit 120 to receive multiple feedback messages from each of the imaging circuits 111 to 11n. In some embodiments, when each imaging circuit 111 to 11n receives a timestamp, each imaging circuit sends the received timestamp back to the transmission circuit 120 as a feedback message, thereby allowing the processor 140 to receive feedback messages from each of the imaging circuits 111 to 11n through the transmission circuit 120.

[0023] In step S413, the processor 140 calculates multiple time differences between the transmission time when the timestamp was sent and the multiple reception times when multiple feedback messages were received, and uses these as multiple delay times between the imaging circuits 111~11n and the processor. In some embodiments, the processor 140 records the time when the timestamp was sent to the outside as the transmission time. In some embodiments, the processor 140 records the time when a feedback message was received from each imaging circuit as each reception time. In other words, the processor 140 calculates the time difference between the transmission time and each reception time, and uses these as the delay time corresponding to each imaging circuit.

[0024] Returning to Figure 4, in step S420, the processor 140 calculates the number of buffer frames for each of the multiple buffer queues 131 to 13n in the memory 130 based on a plurality of delay times. In some embodiments, the number of buffer frames for each buffer queue indicates the number of frames that each buffer queue needs to buffer (i.e., the number of frames that the buffer queue needs to buffer in order to achieve video synchronization).

[0025] The calculation of the buffer frame count will be explained further below. Please also refer to Figure 6, which is a flowchart of several steps S421 to S422 included in step S420 of a video synchronization method applied to a vehicle in some embodiments of the present invention. As shown in Figure 6, step S410 in Figure 4 includes steps S421 to S422.

[0026] In step S421, the processor 140 extracts the maximum value (i.e., the maximum delay time) from the multiple delay times calculated in step S413, and calculates multiple time differences between each of the multiple delay times and the maximum value. In other words, the processor 140 can first obtain the maximum delay time and then calculate the time differences between the maximum delay time and each delay time. For example, the processor 140 calculates the first time difference between the first delay time and the maximum delay time (corresponding to the first imaging circuit), the second time difference between the second delay time and the maximum delay time (corresponding to the second imaging circuit), and the nth time difference between the nth delay time and the maximum delay time (corresponding to the nth imaging circuit), and then makes an inference based on this.

[0027] In step S422, the processor 140 calculates multiple products between multiple time differences and the frame rates of the imaging circuits 111 to 11n, and these are used as the number of buffer frames for the buffer queues 131 to 13n. In some embodiments, the frame rates of the imaging circuits 111 to 11n may all be the same. In other words, the processor 140 can calculate the product between each time difference and the frame rate, and these can be used as the number of buffer frames for each buffer queue corresponding to each time difference. More specifically, a single video frame input to a buffer queue cannot be retrieved from that buffer queue until a certain number of other video frames have been input to the same buffer queue, where the certain number is the number of buffer frames, and the number of buffer frames is inversely proportional to the delay time of the imaging circuit.

[0028] For example, consider a video synchronization device 100 having three shooting circuits 111 to 113 corresponding to three buffer queues 131 to 133. Assuming that the delay times of the three shooting circuits 111 to 113 are 200 milliseconds, 100 milliseconds, and 300 milliseconds, respectively, and that the frame rates of all three shooting circuits 111 to 113 are 30 frames / second, the processor 140 first obtains 300 milliseconds as the maximum delay time, and then calculates the time differences between the maximum delay time and each delay time, i.e., 100 milliseconds (300-200), 200 milliseconds (300-100), and 0 milliseconds (300-300), respectively. Subsequently, the processor 140 calculates the product of the time differences of 100 milliseconds, 200 milliseconds, and 0 milliseconds and the frame rate of 30 frames / second, and sets these as the number of buffer frames for the three buffer queues 131 to 133 (i.e., 3 frames, 6 frames, and 0 frames), respectively.

[0029] Returning to Figure 4, in step S430, the processor 140 controls the shooting circuits 111 to 11n via the transmission circuit 120 to capture images of the area around the vehicle from multiple shooting directions, generating multiple video streams. The processor 140 then controls the transmission circuit 120 to send the multiple video streams to buffer queues 131 to 13n in the memory 130. In other words, the processor 140 can store all the video frames of the video streams captured by the shooting circuits 111 to 11n in buffer queues 131 to 13n, which correspond to the shooting circuits 111 to 11n.

[0030] In step S440, the processor 140 detects the number of queue frames of the video stream temporarily stored in each buffer queue. In some embodiments, the processor 140 periodically detects the number of video frames of the video stream currently temporarily stored in each buffer queue and uses this as the queue frame count.

[0031] In step S450, the processor 140 determines whether or not to retrieve the first frame of the video stream temporarily stored in buffer queues 131 to 13n, based on the number of buffer frames and queue frames of each buffer queue. In the above embodiment, the first frame refers to the first video frame stored in the corresponding buffer queue for each video stream. If the processor 140 determines to retrieve the first frame of the video stream temporarily stored in buffer queues 131 to 13n, the processor 140 executes step S460. Conversely, if the processor 140 determines not to retrieve the first frame of the video stream temporarily stored in buffer queues 131 to 13n, the processor 140 executes step S430 again.

[0032] The decision to remove the data will be explained further below. Please also refer to Figure 7, which is a flowchart of several steps S451 to S454 included in step S450 of a video synchronization method applied to a vehicle in some embodiments of the present invention. As shown in Figure 7, step S450 in Figure 4 includes steps S451 to S454.

[0033] In step S451, the processor 140 compares the number of queue frames for each buffer queue with the number of buffer frames for each buffer queue plus 1. If the number of queue frames for a certain buffer queue is greater than the number of buffer frames plus 1, the processor 140 executes step S452. On the other hand, if the number of queue frames for each buffer queue is not greater than the number of buffer frames for each buffer queue plus 1, and none of them are equal, the processor 140 executes step S453. Also, if the number of queue frames for each buffer queue is equal to the number of buffer frames for each buffer queue plus 1, the processor 140 executes step S454.

[0034] In step S452, the processor 140 deletes the first frame of a video stream temporarily stored in a buffer queue whose queue frame count is greater than the corresponding buffer frame count plus 1. In other words, if the processor 140 detects that the number of video frames temporarily stored in any buffer queue is greater than the buffer frame count of the buffer queue plus 1, it deletes the first frame of the video stream temporarily stored in the buffer queue. Subsequently, the processor 140 executes step S451 again.

[0035] In step S453, the processor 140 decides not to retrieve the first frame of the video stream temporarily stored in buffer queues 131 to 13n. In other words, if the processor 140 detects that the number of queue frames in all buffer queues 131 to 13n is not greater than the number of buffer frames plus 1, and is not equal to the number of buffer frames plus 1, it does not retrieve the first frame temporarily stored in buffer queues 131 to 13n. Subsequently, the processor 140 can execute step S430 again (i.e., control buffer queues 131 to 13n to wait for a new video stream to be received from the corresponding imaging circuit). In some embodiments, if the processor 140 detects after a certain period of time (e.g., 30 minutes) that the number of queue frames in any buffer queue is always less than the number of buffer frames in that buffer queue plus 1, the processor 140 determines that an abnormality has occurred in the buffer queue, and as a result does not need to compare the number of queue frames in that buffer queue with the number of buffer frames plus 1 (i.e., it only compares the number of queue frames with the number of buffer frames plus 1 for the other buffer queues).

[0036] In step S454, the processor 140 decides to retrieve the first frame of the video stream temporarily stored in buffer queues 131 to 13n. In other words, if the processor 140 detects that the number of queue frames in all buffer queues 131 to 13n is equal to the number of buffer frames plus 1, it decides that it can retrieve the first frame temporarily stored in buffer queues 131 to 13n and execute step S460.

[0037] For example, looking back at the previous example, assuming that the buffer frames of buffer queues 131 to 133 are 3, 6, and 0 frames respectively, and the queue frames of buffer queues 131 to 133 are 5, 6, and 0 frames respectively, then processor 140 can determine that the queue frame count of buffer queue 131 is greater than the number of buffer frames of buffer queue 131 plus 1 (i.e., 5 frames is greater than 4 frames). Based on this, processor 140 deletes the first frame of the video stream temporarily stored in buffer queue 131 (i.e., 4 frames remain), and then compares the number of queue frames of buffer queues 131 to 133 with the number of buffer frames of buffer queues 131 to 133 plus 1 again.

[0038] At this time, the processor 140 can determine that the number of queue frames in buffer queue 131 is equal to the number of buffer frames in buffer queue 131 plus 1 (i.e., 4 frames and 4 frames are equal), and that the number of queue frames in buffer queue 132 is less than the number of buffer frames in buffer queue 131 plus 1 (i.e., 6 frames is less than 7 frames), and that the number of queue frames in buffer queue 133 is less than the number of buffer frames in buffer queue 131 plus 1 (i.e., 0 frames is less than 1 frame). In other words, the processor 140 can determine that the number of queue frames in each of the buffer queues 131 to 133 is not greater than the number of buffer frames in each of the buffer queues 131 to 133 plus 1, and that they are not all equal. Based on this, the processor 140 controls buffer queues 131 to 133 to wait for a new video stream to be received from the imaging circuits 111 to 113 (i.e., it executes step S430 again).

[0039] It should be noted that the reason the number of queue frames in buffer queue 131 is greater than the number of buffer frames in buffer queue 131 is that the shooting circuit 111 started shooting earlier than the shooting circuits 112-113. The video stream captured by shooting circuit 111 before the shooting circuits 112-113 started shooting cannot be synchronized with the video captured by shooting circuits 112-113, so the processor 140 needs to delete the video stream captured earlier. The reason the number of queue frames in buffer queue 132 is less than the number of buffer frames in buffer queue 132 is that the first frame of buffer queue 132 cannot be output until another new video frame is input to buffer queue 132. The reason the number of queue frames in buffer queue 133 is less than the number of buffer frames in buffer queue 133 is that no video frames have yet been input to buffer queue 133.

[0040] Furthermore, when buffer queues 131-133 receive one video frame from a new video stream from the imaging circuits 111-113, the number of queue frames for buffer queues 131-133 becomes 5 frames, 7 frames, and 1 frame, respectively. Processor 140 compares the number of queue frames for each buffer queue with the number of buffer frames for each buffer queue plus 1. Subsequently, processor 140 can determine that the number of queue frames for buffer queue 131 is greater than the number of buffer frames for buffer queue 131 plus 1 (i.e., 5 frames is greater than 4 frames). Based on this, processor 140 deletes the first frame of the video stream temporarily stored in buffer queue 131 (i.e., 4 frames remain), and compares the number of queue frames for each buffer queue 131-133 with the number of buffer frames for each buffer queue 131-133 plus 1 again.

[0041] At this time, the processor 140 can determine that the number of queue frames in buffer queue 131 is equal to the number of buffer frames in buffer queue 131 plus 1 (i.e., 4 frames and 4 frames are equal), the number of queue frames in buffer queue 132 is equal to the number of buffer frames in buffer queue 131 plus 1 (i.e., 7 frames and 7 frames are equal), and the number of queue frames in buffer queue 133 is equal to the number of buffer frames in buffer queue 131 plus 1 (i.e., 1 frame and 1 frame are equal). In other words, the processor 140 can determine that the number of queue frames in each of buffer queues 131 to 133 is equal to the number of buffer frames in each of buffer queues 131 to 133 plus 1. Based on this, the processor 140 determines that it can retrieve the first frame of the video stream temporarily stored in buffer queues 131 to 133 and execute the subsequent integration process (i.e., step S460).

[0042] Returning to Figure 4, in step S460, the processor 140 integrates all the first frames into a composite image. In some embodiments, the processor 140 controls the display 150 to display the composite image, or directly stores the composite image in memory 130 for subsequent applications (e.g., image optimization or image recognition). In other words, the processor 140 can not only display the composite image on the display 150, but can also store the composite image in memory 130 without displaying it on the display 150, or even store the composite image in memory 130 and display it on the display 150 at the same time.

[0043] In some embodiments, multiple regions in the composite image each contain multiple starting frames (i.e., screen compositing). In other embodiments, if the shooting ranges of each shooting circuit 111~11n overlap, the composite image may be a panoramic image. In other embodiments, the composite image may further contain multiple starting frames corresponding to multiple shooting ranges having the same time marker (e.g., 3:10 p.m.) (i.e., timeline compositing). When a user selects one shooting range corresponding to one of the time markers (e.g., selection using the touch display 150), the processor 140 can control the display 150 to display the starting frame corresponding to the selected shooting range. It should be noted that the composite image can be applied to certain application scenarios (e.g., stereoscopic vision, panoramic stitching, high-speed photography), thereby providing the effect of maintaining image consistency and coherence. In other words, the present invention can be used to perform time synchronization or image stitching on images from different sources for vehicles.

[0044] The generation of a composite image will be explained below using a specific example. Please also refer to Figure 8, which is a schematic diagram of a composite image 800 in some embodiments of the present invention. As shown in Figure 8, assuming that the number of shooting circuits and buffer queues n is 2, the composite image 800 includes a first region 810 and a second region 820. The first region 810 includes the first frame generated by the shooting circuit 111 (e.g., an image of the front of the vehicle taken at a specific time), and the second region 820 includes the first frame generated by the shooting circuit 112 (e.g., an image of the rear of the vehicle taken at a specific time). It should be noted that here, a composite image 800 including the first region 810 and the second region 820 displayed vertically is given as an example, but in other embodiments, a composite image 800 including the first region 810 and the second region 820 displayed horizontally may be adopted.

[0045] As described above, the video synchronization method and apparatus for vehicles proposed by the present invention utilize the delay time of the shooting circuit for shooting the area around the vehicle to set the number of buffer frames for each buffer queue, and by comparing the number of queue frames and buffer frames for each buffer queue, it determines the timing to extract the first frame from each buffer queue and synchronize the surrounding video. Therefore, the present invention can solve the problem of video synchronization misalignment caused by different transmission delays between each shooting circuit. Furthermore, the video synchronization method and apparatus for vehicles proposed by the present invention can handle the delay of the wireless network when a wireless connection method is adopted for the shooting circuit, and the instability of the time it takes for the video frames generated by the shooting circuit to reach the system, and can avoid the increased wiring costs when a wired connection method is adopted for the shooting circuit. On the other hand, the video synchronization method and apparatus for vehicles proposed by the present invention can ensure video consistency and coherence when multiple shooting circuits shoot the same scene simultaneously.

[0046] The above are merely preferred examples of the present invention, and the scope of the claims of the present invention is not limited thereto. Accordingly, all equivalent modifications made based on the content of the present invention shall be included within the scope of protection of the present invention. [Explanation of Symbols]

[0047] 100: Video synchronization device applied to vehicles 111~11n: Imaging circuit 120: Transmission Circuit 130: Memory 140: Processor 150: Display VH: Vehicle 210~240: Shooting range img1~imgn: Video stream 131~13n: Buffer queue S410~S460, S411~S413, S421~S422, S451~S454: Step 800: Composite video 810: First area 820:Second area

Claims

1. Step a, in which a processor on the vehicle and a transmission circuit on the vehicle detect multiple delay times between multiple imaging circuits and the processor, The processor performs the following steps: b) calculates the number of buffer frames for each of the multiple buffer queues in the memory on the vehicle based on the multiple delay times (where each of the multiple buffer queues corresponds to one of the multiple imaging circuits), Step c involves controlling the plurality of shooting circuits using the processor and the transmission circuit to capture images of the area around the vehicle from multiple shooting directions to generate multiple video streams, and further controlling the transmission circuit to transmit the plurality of video streams to the plurality of buffer queues in the memory, The processor performs the step d of detecting the number of queue frames of the video stream temporarily stored in each of the buffer queues, The processor performs the following steps: 1) determine whether or not to retrieve the first frame of the video stream temporarily stored in each of the plurality of buffer queues based on the number of buffer frames and the number of queue frames in each buffer queue (where the first frame refers to the video frame initially stored in the corresponding buffer queue in each video stream), When the processor determines to retrieve the first frame of the video stream temporarily stored in each of the plurality of buffer queues, the processor performs step f, which involves integrating all of the first frames into a composite image. A video synchronization method applied to a vehicle, comprising: step g, if it is determined not to retrieve the first frame of the video stream temporarily stored in each of the buffer queues, step g, which involves repeating steps c through e.

2. The aforementioned step a is, The processor controls the transmission circuit and transmits a timestamp to the plurality of imaging circuits. The processor controls the transmission circuit to receive multiple feedback messages from each of the multiple imaging circuits, A video synchronization method applicable to a vehicle according to claim 1, comprising the step of using the processor to calculate a plurality of time differences between the transmission time when the timestamp was transmitted and the plurality of reception times when each of the plurality of feedback messages was received, and using these differences as the plurality of delay times between the plurality of shooting circuits and the processor.

3. Step b is, The processor performs the steps of calculating multiple time differences between the multiple delay times and the maximum value among them, A video synchronization method applicable to a vehicle according to claim 1, comprising the step of using the processor to calculate a plurality of products between the plurality of time differences and the frame rates of the plurality of shooting circuits, and using these as the plurality of buffer frames for the plurality of buffer queues.

4. The aforementioned step e is Step e1, the processor compares the number of queue frames in each buffer queue with the number of buffer frames in each buffer queue plus 1. A video synchronization method applicable to a vehicle according to claim 1, comprising step e2, where the processor determines to retrieve the first frame of the video stream temporarily stored in each of the plurality of buffer queues if the number of queue frames in each buffer queue is equal to the corresponding number of buffer frames plus 1.

5. The aforementioned step e is If the number of queue frames in a certain buffer queue is greater than the number of buffer frames in that buffer queue plus 1, the processor deletes the first frame of the video stream temporarily stored in the buffer queue whose number of queue frames is greater than the corresponding number of buffer frames plus 1, and then performs step e1 again. A video synchronization method applicable to a vehicle according to claim 4, further comprising the step of determining that the processor will not retrieve the first frame of the video stream temporarily stored in each of the plurality of buffer queues if the number of queue frames in each buffer queue is not greater than the number of buffer frames in each buffer queue plus 1, and the numbers of queue frames in each buffer queue are not equal.

6. Multiple imaging circuits are installed in the vehicle and configured to capture images of the area around the vehicle from multiple shooting directions, A transmission circuit connected to the plurality of imaging circuits and configured to receive the video stream from each of the imaging circuits, A memory connected to the transmission circuit and configured to store multiple commands and multiple buffer queues (where each of the multiple buffer queues corresponds to one of the multiple imaging circuits), It is a processor, It is connected to the transmission circuit and the memory, and accesses the plurality of commands, The transmission circuit performs a step of detecting multiple delay times between the multiple imaging circuits and the processor, Step b: Based on the aforementioned multiple delay times, calculate the number of buffer frames for each of the aforementioned multiple buffer queues in the vehicle. Step c involves controlling the plurality of shooting circuits using the transmission circuit, generating a plurality of video streams by shooting the area around the vehicle from a plurality of shooting directions, and further controlling the transmission circuit to transmit the plurality of video streams to the plurality of buffer queues, Step d: Detect the number of queue frames of the video stream temporarily stored in each of the buffer queues. Step e, which determines whether or not to retrieve the first frame of the video stream temporarily stored in each of the plurality of buffer queues based on the number of buffer frames and the number of queue frames of each buffer queue (where the first frame refers to the video frame that was first stored in the corresponding buffer queue in each video stream), If it is determined to retrieve the first frame of the video stream temporarily stored in each of the aforementioned plurality of buffer queues, step f is to integrate all of the first frames into the composite video, If it is determined that the first frame of the video stream temporarily stored in each of the buffer queues is not to be retrieved, then step g is performed again, from step c to step e. A video synchronization device applied to a vehicle, including a processor configured to perform the following:

7. In step a, the processor The steps include: transmitting a timestamp to the plurality of imaging circuits using the transmission circuit; The transmission circuit receives multiple feedback messages from each of the multiple imaging circuits, A video synchronization device for a vehicle according to claim 6, configured to perform the steps of: calculating a plurality of time differences between the transmission time at which the timestamp was transmitted and the plurality of reception times at which each of the plurality of feedback messages was received, and using these as the plurality of delay times between the plurality of shooting circuits and the processor.

8. In step b, the processor The steps include: calculating multiple time differences between the multiple delay times and the maximum value among them; A video synchronization device for a vehicle according to claim 6, configured to perform the steps of: calculating a plurality of products between the plurality of time differences and the frame rates of the plurality of shooting circuits, and using these as the plurality of buffer frames for the plurality of buffer queues.

9. In step e, the processor Step e1 involves comparing the number of queue frames in each buffer queue with the number of buffer frames in each buffer queue plus 1, A video synchronization device for a vehicle according to claim 6, configured to perform step e2 of determining to retrieve the first frame of the video stream temporarily stored in each of the plurality of buffer queues if the number of queue frames in each of the buffer queues is equal to the number of buffer frames in each of the buffer queues plus 1.

10. In step e, the processor further If the number of queue frames in a certain buffer queue is greater than the number of buffer frames in that buffer queue plus 1, the processor deletes the first frame of the video stream temporarily stored in the buffer queue whose number of queue frames is greater than the corresponding number of buffer frames plus 1, and then performs step e1 again. A video synchronization device for a vehicle according to claim 9, configured to perform the step of determining, if the number of queue frames in each of the buffer queues is not greater than the number of buffer frames in each of the buffer queues plus 1, and if the numbers of each buffer queue are not equal, then the processor determines not to retrieve the first frame of the video stream temporarily stored in each of the plurality of buffer queues.

Images