Dual-camera synchronous counting method and device

Abstract

The invention provides a double-camera synchronous counting method and device, and the method comprises the steps: S1, receiving data, and enabling the data of double cameras to enter two video interface controller VIC modules; s2, counting synchronization processing is carried out in the two VIC modules, wherein a double-counter synchronization mechanism is adopted, and each VIC module comprises an 8-bit counter count 8 and a 32-bit counter count 32; timestamps are added, the timestamps are added into the two data streams, and after a specific time period, the corresponding timestamps can be hit into the heads of the data streams; the timestamp is divided into 8-bit counting and 32-bit counting, and when the 8-bit counting reaches the maximum value, the 8-bit counting will carry to the 32-bit counting, S3, outputting the counted and synchronized data to an ISP module for image processing; s4, after processing is completed, data of the double cameras are output to a bus; and S5, merging the two pictures as long as the difference between the two timestamps is within a reasonable threshold range. Accurate image synchronization, high-resolution timestamp generation and accurate time information are realized. The low-power-consumption design is stable and reliable.

Description

Technical Field

[0001] The present invention belongs to the technical field of dual-camera image processing and synchronization, and particularly relates to a method and device for dual-camera synchronous counting. Background Art

[0002] In the prior art, with the development of digital image processing technology, camera applications have become increasingly widespread. In camera applications, the dual-camera solution has been widely used. The dual-camera solution can improve image quality and increase the diversity of shooting scenes by simultaneously capturing images from two perspectives. However, in the dual-camera solution, due to the asynchronous timing between the two cameras, the image quality will decline and the visual effect will be poor. Ensuring that the images captured by the two cameras are synchronized in time is a challenge.

[0003] To solve this problem, some synchronization schemes have been proposed in the current prior art, including hardware-based synchronization and software-based synchronization. In the hardware synchronization scheme, synchronization is achieved by adding a synchronization signal in the hardware circuit. In the software synchronization scheme, synchronization is achieved by software algorithms.

[0004] Many existing dual-camera synchronization schemes adopt a similar timestamp mechanism to synchronize the images captured by the cameras. These schemes usually use hardware logic and counter devices to assign a timestamp to each captured image and provide the timestamp for software use to perform synchronization processing on the images.

[0005] For example, some existing schemes use hardware platforms such as FPGA or ASIC to implement dual-camera synchronization. These schemes usually use hardware logic such as counters or PLLs to generate timestamps and embed the timestamps into the image data. On the software side, the timestamps can be used to synchronize the images of the two cameras.

[0006] However, the prior art usually designs complex circuit designs or software designs, or relies on a single counter or a simple synchronization mechanism to achieve image synchronization, resulting in problems such as inaccurate synchronization and low efficiency.

[0007] In addition, common technical terms include:

[0008] ASIC (Application-Specific Integrated Circuit): Application-specific integrated circuit, referring to a customized integrated circuit designed according to specific application requirements.

[0009] ISP (Image Signal Processor): Image signal processor, a dedicated hardware or software module for processing digital image data.

[0010] Timestamp: A timestamp refers to the numerical representation of a point in time, usually an integer or a floating-point number. In the present invention, the timestamp is used to record the time information of the images captured by the camera, so as to synchronize the images of two cameras.

[0011] Dual-camera synchronization technology: Dual-camera synchronization technology is a technology used to synchronize the images captured by two cameras.

[0012] VIC (Video Interface Controller): A video interface controller, which is a controller dedicated to video input, used to receive signals from an external video source and transmit them to a processor or other video devices for processing.

[0013] Merging pictures: Merging pictures refers to fusing the images captured by two cameras in order to obtain a more complete and clear image. Summary of the Invention

[0014] To solve the above problems, the purpose of the present application is to solve the problems of image quality degradation and poor visual effects caused by out-of-sync timing between cameras in existing dual-camera solutions. Existing synchronization solutions mainly rely on software synchronization, or single-hardware counting, or designing complex circuit logics. The present invention proposes a dual-counter hardware synchronization solution based on ASIC, realizing a hardware device for precise synchronization and timestamp generation, which is simple and easy to use, thereby achieving high-frame-rate and high-quality visual effects of images.

[0015] Specifically, the present invention provides a method for dual-camera synchronous counting, and the method includes the following steps:

[0016] S1. Receiving data. To implement binocular vision processing, two camera controllers built into an ASIC chip receive the inputs of two external cameras, that is, the data of the dual-cameras enter two Video Interface Controller (VIC) modules.

[0017] S2. Performing counting synchronization processing in the two VIC modules:

[0018] A dual-counter synchronization mechanism is adopted. Each VIC module includes an 8-bit counter count8 and a 32-bit counter count32.

[0019] Adding timestamps. Timestamps are added to the two data streams. After a specific time period, the corresponding timestamps will be inserted into the head of the data stream. The specific time period refers to the time of one frame or half a frame of an image. The timestamps are divided into 8-bit counting and 32-bit counting. When the 8-bit counting reaches the maximum value, it will carry over to the 32-bit counting, that is, when the 8-bit counter reaches the maximum value, the 32-bit counter starts counting.

[0020] S3. Output the data after counting and synchronization in step S2 to the ISP module for image processing;

[0021] S4. After the processing is completed, output the data of the dual cameras to the bus;

[0022] S5. As long as the gap between the two timestamps is within a reasonable threshold range, merge the two pictures. The reasonable threshold is determined by the application using the image including the camera management tool and includes 1 or 2 frame images.

[0023] The step S2 further includes:

[0024] S2.1. The VIC module 1 and the VIC module 2 respectively receive the data input by the camera 1 and the camera 2; S2.2. The Count8 and count32 in each VIC module respectively perform synchronous counting on the input data; add timestamps respectively.

[0025] In the step S5, the processor CPU processes the merged image; if the value of the timestamp is within the range allowed by the operating system, the system determines that the data can be processed accordingly, and the CPU will perform data processing operations on the corresponding data stream, and the two pictures can be merged.

[0026] The step S5 further includes:

[0027] S5.1. Read the 32-bit timestamp;

[0028] S5.2. Judge whether the 32-bit difference is out of range,

[0029] If the difference is higher than or equal to the threshold, perform exception handling. The threshold is determined by the application using the image including the camera management tool, and the threshold includes 1 or 2 frame images;

[0030] If the difference is lower than the threshold, read the 8-bit timestamp;

[0031] S5.3. If after reading the 8-bit timestamp, the difference is higher than the threshold, the threshold is determined by the specific software, and the threshold includes 1 or 2 frame images, then merge the image data and discard part of the data. The merged image data is the merged image data of 2 VICs, and discard the data of one VIC.

[0032] The step S5.3 further includes that if there are differences in the 32-bit counter, the operating system can immediately determine that the working states of the two data streams are significantly abnormal. By using historical information, it can be determined whether both cameras have problems or a certain camera has problems. Assume that the historical information has always been that one VIC is greater than the other VIC by 1 frame, but if it becomes one VIC is greater than the other VIC by 2 frames, it indicates that the other VIC has an error; if it becomes one VIC is less than the other VIC, it indicates that this VIC has an error; if the value of the time stamp, which is the value obtained by adding the 32-bit and 8-bit counters here, is within the range allowed by the application using the image in the operating system, the system determines that the data can be processed accordingly, including that the relevant computing components of the CPU will perform data processing operations on the corresponding data stream and can merge the two pictures.

[0033] In the method, as long as the time stamps of the two pictures are within the controllable range, the pictures obtained by the two ISPs can be merged. The controllable range is determined by the application using the image and includes 1 frame.

[0034] This application also relates to a device for dual-camera synchronous counting. The device is applicable to any of the above methods and further includes:

[0035] An ISP dual-camera implemented based on an ASIC chip;

[0036] Two camera controllers built in the ASIC chip, which are used to respectively receive the inputs of the two external cameras, that is, the data input from the dual-camera enters two video interface controller VIC modules for counting and synchronous processing;

[0037] It includes a dual-counter for completing synchronous counting, including an 8-bit counter count8 and a 32-bit counter count32;

[0038] An ISP module, and the data after counting and synchronous processing is output to the ISP module for image processing;

[0039] A CPU, which is used to perform data processing operations on the corresponding data stream and can merge the two pictures.

[0040] The ISP mentioned refers to a hardware module for processing dual-camera images.

[0041] Therefore, the advantages of this application are as follows:

[0042] 1. Precise image synchronization, applicable to dual-camera systems.

[0043] 2. High-resolution time stamp generation, providing accurate time information.

[0044] 3. Low-power design, applicable to mobile devices and embedded systems.

[0045] 4. Stability and reliability, suitable for various environments and applications. Brief Description of the Drawings

[0046] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not limit the present invention.

[0047] Figure 1 It is a schematic diagram of data transmitted by two cameras that cannot be fully synchronized in the prior art.

[0048] Figure 2 It is a schematic diagram of step S5 in the method of this application.

[0049] Figure 3 It is a schematic diagram of the hardware module for the device of this application to process dual-camera images.

[0050] Figure 4 It is a schematic diagram of the process flow of the method of this application. Detailed Description of the Preferred Embodiment

[0051] In order to more clearly understand the technical content and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0052] This application relates to dual-camera image processing and synchronization technology, especially a hardware device for realizing image synchronization and timestamp generation in dual-camera applications. It is an ISP dual-camera VIC module counting synchronization device based on ASIC implementation. By counting, a timestamp data is stored in each frame of the picture. Hardware counting has high-speed usability, simplicity, and reliability.

[0053] In the prior art, as Figure 1 shown, the data transmitted by two cameras cannot be fully synchronized. Even if the designer tries to make the two cameras start working simultaneously as much as possible, there will still be a time error between the actual generated data;

[0054] Most of the time, this error may alternate between positive and negative. Sometimes the left side may be faster, and sometimes the right side may be faster; it is very difficult to be completely consistent;

[0055] At special time points, such as the time point when the circuit just starts to work, the data streams of the two cameras may be quite different, and it is difficult for the CPU inside the ASIC to correctly process such data streams;

[0056] At other special time points, such as when a certain camera is interfered, it will also cause the two data streams to be unable to be synchronized.

[0057] In order to implement binocular vision processing, two camera controllers are built into the ASIC chip, respectively receiving the inputs from two external cameras;

[0058] The present invention provides a method for generating timestamps. Timestamps are added to two data streams to provide a flexible basis for the CPU inside the ASIC to process the data streams. When the data stream of a certain camera is generated, after a specific time period, such as one image, the corresponding timestamp will be inserted into the head of the data stream. The timestamp is divided into 8-bit counting and 32-bit counting. When the 8-bit counting reaches the maximum value, it will carry over to the 32-bit counting. This counting method provides a flexible way for the operating system to judge the differences. The 8-bit counter is accurate but changes quickly and can accurately judge the differences at the millisecond level. Among them, the 32-bit counter changes slowly and can judge the differences in large data streams. If there are differences in the 32-bit counter, the operating system can immediately judge that the working states of the two data streams are significantly abnormal. Using historical information, it can be judged whether both cameras have problems at the same time or whether a certain camera has problems. If the value of the timestamp is within the range allowed by the operating system, the system will judge that the data can be processed accordingly, and relevant computing components such as the CPU will perform data processing operations on the corresponding data stream, and two pictures can be merged.

[0059] Therefore, as Figure 4 shown, an embodiment is a method for synchronizing an ISP dual-camera controller based on ASIC implementation. The method includes:

[0060] S1. Receive data. In order to implement binocular vision processing, the input of two external cameras is received through two camera controllers built into the ASIC chip, that is, the data of the dual cameras enters two Video Interface Controller (VIC) modules.

[0061] S2. Perform counting synchronization processing in the two VIC modules:

[0062] A dual-counter synchronization mechanism is adopted. Each VIC module includes an 8-bit counter count8 and a 32-bit counter count32.

[0063] Add timestamps. Timestamps are added to the two data streams. After a specific time period, the corresponding timestamp will be inserted into the head of the data stream. The specific time period refers to the time of one frame or half a frame of an image, such as the time period of one image. The timestamp is divided into 8-bit counting and 32-bit counting. When the 8-bit counting reaches the maximum value, it will carry over to the 32-bit counting, that is, when the 8-bit counter reaches the maximum value, the 32-bit counter starts counting.

[0064] S3. Output the data after counting and synchronization in step S2 to the ISP module for image processing.

[0065] S4. After the processing is completed, output the data of the dual cameras to the bus.

[0066] S5, as long as the gap between two timestamps is within a reasonable threshold range, two images are merged. The reasonable threshold is determined by the application using the images. The application using the images, such as a camera management tool, includes 1 or 2 frames of images.

[0067] The step S2 further includes:

[0068] S2.1, VIC module 1 and VIC module 2 respectively receive the data input by camera 1 and camera 2; S2.2, Count8 and count32 in each VIC module respectively perform synchronous counting on the input data; timestamps are added respectively.

[0069] As Figure 2 shown, in the step S5, the processor CPU processes the merged image; if the value of the timestamp is within the range allowed by the operating system, the system determines that the data can be processed accordingly, and the CPU will perform data processing operations on the corresponding data stream, and two images can be merged.

[0070] The step S5 further includes:

[0071] S5.1, read the 32-bit timestamp;

[0072] S5.2, judge whether the 32-bit difference is out of range,

[0073] If the difference is higher than or equal to the threshold, exception handling is performed. The threshold is determined by the application using the images including the camera management tool. The threshold includes 1 or 2 frames of images;

[0074] If the difference is lower than the threshold, read the 8-bit timestamp;

[0075] S5.3, if after reading the 8-bit timestamp, the difference is higher than the threshold, the threshold is determined by the specific software, and the threshold includes 1 or 2 frames of images, then merge the image data and discard part of the data. The merged image data is the merger of the image data of 2 VICs, and discard the data of one VIC.

[0076] The step S5.3 further includes that if there are differences in the 32-bit counter, the operating system can immediately determine that the working states of the two data streams are significantly abnormal. Using historical information, it can be determined whether both cameras have problems or a certain camera has a problem: Assume that the historical information has always been that one VIC is greater than the other VIC by 1 frame, but if it becomes one VIC is greater than the other VIC by 2 frames, it means that the other VIC has an error; if it becomes one VIC is less than the other VIC, it means that this VIC has an error; if the value of the time stamp, which is the value obtained by adding the 32-bit and 8-bit counters here, is within the range allowed by the application that uses the image in the operating system, the system determines that the data can be processed accordingly, including that the relevant computing components of the CPU will perform data processing operations on the corresponding data stream, and two pictures can be merged.

[0077] In the method, as long as the time stamps of the two pictures are within the controllable range, the pictures obtained by the two ISPs can be merged. The controllable range is determined by the application that uses the image and includes 1 frame.

[0078] This application further includes a device for dual-camera synchronous counting. The device is applicable to any of the above methods and further includes:

[0079] An ISP dual-camera implemented based on an ASIC chip;

[0080] Two camera controllers built in the ASIC chip, used to respectively receive the inputs of the two external cameras, that is, the data input from the dual-camera enters two video interface controller VIC modules for counting and synchronous processing;

[0081] It includes a dual-counter for completing synchronous counting, including an 8-bit counter count8 and a 32-bit counter count32;

[0082] An ISP module, and the data after counting and synchronous processing is output to the ISP module for image processing;

[0083] A CPU, used to perform data processing operations on the corresponding data stream, and two pictures can be merged.

[0084] The ISP mentioned refers to a hardware module for processing dual-camera images.

[0085] As Figure 3 shown, among which,

[0086] 1). The data of the dual-camera enters two VIC modules;

[0087] 2). Counting and synchronous processing are performed in the VIC module;

[0088] 3). The data after counting and synchronization is output to the ISP module for image processing;

[0089] 4). After the processing is completed, output the data of the dual cameras to the bus.

[0090] In summary, the technical solution of this application lies in: a hardware dual-counter synchronization mechanism implemented for the synchronization of dual cameras. The solutions in the prior art may include using other software synchronization methods or more complex hardware designs, but these existing methods are not as simple, reliable, and efficient as the hardware dual-counter design. Specifically, it includes the following aspects:

[0091] Counter synchronization mechanism: The key innovation of the present invention is the adoption of a dual-counter synchronization mechanism. This includes an 8-bit counter (count8) and a 32-bit counter (count32). When the 8-bit counter reaches its maximum value, the 32-bit counter starts counting. This mechanism ensures the precise synchronization of image frames and the accurate generation of timestamps, providing highly reliable time synchronization for the dual-camera system. The protection points include the hardware design of the dual-counter system and the related control logic.

[0092] Reasonable threshold range: The present invention introduces the concept of a reasonable threshold range during the synchronization process. As long as the difference between the two timestamps is within the reasonable threshold range, the two pictures can be merged. This innovation makes the synchronization more flexible and adaptable to the requirements of different application scenarios.

[0093] Low-power design: The present invention takes into account the requirements of mobile devices and embedded systems and adopts a low-power design. The design of the counter system ensures that precise synchronization can still be achieved in the low-power state, which helps to extend the battery life of the device.

[0094] Stability and reliability: The present invention emphasizes the stability and reliability of the hardware device. The dual-counter synchronization mechanism is not easily affected by external interference or device differences, thus ensuring the stability of the system and the reliability of image synchronization.

[0095] Wide application range: The key point of the present invention also lies in its wide applicability. It is applicable to various dual-camera systems, including mobile devices, monitoring systems, autonomous driving vehicles, etc., providing a solution for precise synchronization and timestamp generation in these fields.

[0096] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for synchronous counting of dual cameras, characterized in that, The method includes the following steps: S1. Receive data. To implement binocular vision processing, the input from two external cameras is received through two camera controllers built into the ASIC chip, that is, the data of the dual cameras enters two Video Interface Controller (VIC) modules. S2. Perform counting and synchronization processing in the two VIC modules: A dual-counter synchronization mechanism is adopted. Each VIC module includes an 8-bit counter count8 and a 32-bit counter count32. Add timestamps. Timestamps are added to the two data streams. After a specific time period, the corresponding timestamps will be inserted into the head of the data stream. The specific time period refers to the time of one frame or half a frame of an image. The timestamps are divided into 8-bit counting and 32-bit counting. When the 8-bit counting reaches the maximum value, it will carry over to the 32-bit counting, that is, when the 8-bit counter reaches the maximum value, the 32-bit counter starts counting. S3. Output the data after counting and synchronization in step S2 to the ISP module for image processing. S4. After the processing is completed, output the data of the dual cameras to the bus. S5. As long as the difference between the two timestamps is within a reasonable threshold range, the two pictures are merged. The reasonable threshold is determined by the application using the image, including the camera management tool, and includes 1 frame or 2 frames of images.

2. The dual-camera synchronous counting method according to claim 1, wherein Step S2 further includes: S2.

1. VIC module 1 and VIC module 2 respectively receive the data input from camera 1 and camera 2; S2.

2. The Count8 and count32 in each VIC module respectively perform synchronous counting on the input data and add timestamps respectively.

3. A dual-camera synchronous counting method according to claim 1, characterized in that, In step S5, the processor CPU processes the merged image; if the value of the timestamp is within the range allowed by the operating system, the system determines that the data can be processed accordingly, and the CPU will perform data processing operations on the corresponding data stream, and the two pictures can be merged.

4. A dual-camera synchronous counting method according to claim 3, characterized in that, Step S5 further includes: S5.

1. Read the 32-bit timestamp. S5.

2. Judge whether the 32-bit difference is out of range. If the difference is higher than or equal to the threshold, abnormal processing is performed. The threshold is determined by the application using the image, including the camera management tool, and the threshold includes 1 frame or 2 frames of images. If the difference is lower than the threshold, read the 8-bit timestamp. S5.

3. If after reading the 8-bit timestamp, the difference is higher than the threshold, the threshold is determined by the specific software, and the threshold includes 1 frame or 2 frames of images, then merge the image data and discard part of the data. The merged image data is the merger of the image data of the two VICs, and the data of one VIC is discarded.

5. A dual-camera synchronous counting method according to claim 4, characterized in that The step S5.3 further includes that if there are differences in the 32-bit counter, the operating system can immediately determine that the working states of the two data streams are significantly abnormal. By using historical information, it can be determined whether both cameras have problems or a certain camera has a problem. Assume that the historical information has always been that one VIC is greater than the other VIC by 1 frame, but if it becomes one VIC is greater than the other VIC by 2 frames, it indicates that the other VIC has an error; if it becomes one VIC is less than the other VIC, it indicates that this VIC has an error; if the value of the time stamp, which is the value obtained by adding the 32-bit and 8-bit counters here, is within the range allowed by the application that uses the image in the operating system, the system determines that the data can be processed accordingly, including that the relevant computing components of the CPU will perform data processing operations on the corresponding data stream and can merge two pictures.

6. A dual-camera synchronous counting method according to claim 1, characterized in that, In the method, as long as the time stamps of the two pictures are within the controllable range, the pictures obtained by the two ISPs can be merged. The controllable range is determined by the application that uses the image and includes 1 frame.

7. A device for synchronous counting of dual cameras, characterized in that, The device is applicable to any of the methods described in the above claims 1-6 and further includes: An ISP dual camera implemented based on an ASIC chip; Two camera controllers built in the ASIC chip, which are used to respectively receive the inputs of two external cameras, that is, the data input from the dual camera enters two video interface controller VIC modules for counting and synchronization processing; It includes a dual counter for completing synchronous counting, including an 8-bit counter count8 and a 32-bit counter count32; An ISP module, and the data after counting and synchronization processing is output to the ISP module for image processing; A CPU, which is used to perform data processing operations on the corresponding data stream and can merge two pictures.

8. A device for synchronous counting of dual cameras according to claim 7, characterized in that The ISP refers to a hardware module for processing dual-camera images.

Images