Multi-camera imaging system based on microsecond-level dynamic synchronization control

Abstract

The application discloses a kind of multi-camera imaging system based on microsecond level dynamic synchronization control, comprising: multi-camera imaging array includes several orderly arranged imaging camera, for imaging;Synchronization control module includes synchronization controller, for each imaging camera provides independent microsecond level time delay configuration;Timing management module includes time-sharing scheduler, for the exposure time slot of each imaging camera;Data acquisition and temporary storage module includes acquisition circuit and memory, for image acquisition and cache;Processing module includes image registration and alignment module and image fusion and interpolation module, wherein, the image registration and alignment module is used for image cooperation and alignment, the image fusion and interpolation module is used for image sequence interpolation image fusion.This application has the characteristics of high time synchronization accuracy, multi-channel acquisition, multi-resolution interpolation fusion space-time reconstruction, with ultra-high resolution and million frame per second level frame rate image acquisition and timing restoration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly, to a multi-camera imaging system based on microsecond-level dynamic synchronization control. BACKGROUND

[0002] With the improvement of computing power and the progress of image sensor technology, multi-camera systems are widely used in three-dimensional reconstruction, depth estimation, super-resolution computational photography, light field imaging and environmental perception.

[0003] However, existing multi-camera systems mainly focus on "spatial dimension redundancy" (such as different viewing angles) for reconstruction, and generally ignore the challenge of "time dimension accurate alignment". The traditional system has the following problems:

[0004] 1. Limited synchronization capability, traditional systems achieve camera synchronous acquisition through master-slave triggering (trigger bus), but trigger delay is subject to IO transmission link, drive delay and acquisition clock, and uncontrollable jitter usually reaches dozens of microseconds or more;

[0005] 2. Motion target imaging blur, in high-speed target scenes, inter-frame time difference will cause stitching artifacts, even wrong frame problems, affecting the consistency of the fused image;

[0006] 3. Fusion algorithm is not suitable for high-speed scenes, traditional image stitching and interpolation algorithms (such as optical flow method, bilinear interpolation) cannot work stably under high-speed changing content, especially for image sequences with discontinuous time dimension. SUMMARY

[0007] The purpose of the present application is to provide a multi-camera imaging system based on microsecond-level dynamic synchronization control, which realizes image acquisition and time sequence restoration with ultra-high resolution (such as more than 10K) and million frames per second (10 6 fps) level frame rate through multi-channel high-speed data acquisition and the spatio-temporal modeling capability of neural networks, and is widely applicable to high-end application scenarios such as high-speed physical experiments, femtosecond / picosecond chemical reaction observation, aerospace visual measurement, industrial ultrafast defect detection, etc.

[0008] The present application provides a multi-camera imaging system based on microsecond-level dynamic synchronization control, comprising:

[0009] A multi-camera imaging array, a synchronization control module, a time sequence management module, a data acquisition and temporary storage module, and a processing module, wherein,

[0010] The multi-camera imaging array comprises a plurality of orderly arranged imaging cameras for imaging.

[0011] The synchronization control module includes a synchronization controller, which provides an independent microsecond-level delay configuration for each imaging camera;

[0012] The timing management module includes a time-sharing scheduler for configuring the exposure time slots of each imaging camera;

[0013] The data acquisition and storage module includes an acquisition circuit and a memory, used to acquire images and cache them;

[0014] The processing module includes an image registration and alignment module and an image fusion and interpolation module. The image registration and alignment module is used to perform image matching and alignment, and the image fusion and interpolation module is used to perform image fusion and interpolation on the image sequence.

[0015] In this solution, the array structure of the multi-camera imaging array supports one-dimensional linear structure, two-dimensional rectangular structure and three-dimensional spherical structure.

[0016] In this scheme, the trigger time expression of the imaging camera is as follows:

[0017] ;

[0018] in, For the first Triggering time of each camera, For reference trigger time, For the first Programmable delay time for each imaging camera.

[0019] In this scheme, the programmable delay time of the imaging camera is calculated as follows:

[0020] ;

[0021] in, For the first Programmable delay time for each imaging camera For the first The delay steps of each camera, This is the master clock frequency of the synchronous controller.

[0022] In this scheme, the synchronization controller includes a programmable delay array (PDA) and a digital phase-locked loop (DPLL) circuit. Through delay line adjustment, the maximum time difference among all imaging cameras is achieved. .

[0023] In this scheme, the target image obtained during target observation is a continuous function. The specific location is The time is The brightness value, the first The imaging camera in the first The image at the second sampling time is expressed as follows:

[0024] ;

[0025] in, For the first The imaging camera in the first The image at the next sampling time, For position parameters, For time parameters, Spatial offset caused by the current frame's viewpoint shift. For the first The camera captured the first... Time offset during frames;

[0026] An approximate target image is obtained by jointly approximating and restoring images from different imaging cameras.

[0027] In this scheme, the processing module uses a spatiotemporal interpolation neural network for image matching, alignment, and fusion. The spatiotemporal interpolation neural network includes a feature extraction layer, a spatial feature alignment module, a temporal fusion module, and an upsampling module.

[0028] In this scheme, the loss function of the spatiotemporal interpolation neural network is as follows:

[0029] ;

[0030] in, To reconstruct the error loss, The total variation smoothing loss, For inter-frame consistency regularization loss, For weight hyperparameters.

[0031] In this scheme, when performing image matching, alignment, and image fusion, multiple time-slice image frames from the imaging camera are specifically used. The input is fed into a spatiotemporal interpolation neural network to obtain temporally continuous interpolated image frames. ,in, / For pixel height, / The width is in pixels. This represents the number of network feature channels.

[0032] In this scheme, the resolution of the output image is... ,in, Sampling factor on the interpolation network, This is the original image size.

[0033] The present invention discloses a multi-camera imaging system based on microsecond-level dynamic synchronization control, which features high time synchronization accuracy, multi-channel acquisition, and multi-resolution interpolation fusion spatiotemporal reconstruction. It combines ultra-high resolution with image acquisition and temporal restoration at a frame rate of millions of frames per second, and can be widely applied to high-end application scenarios such as high-speed physics experiments, femtosecond / picosecond chemical reaction observation, aerospace vision measurement, and industrial ultrafast defect detection. Attached Figure Description

[0034] Figure 1 A flowchart of a multi-camera imaging system based on microsecond-level dynamic synchronization control according to the present invention is shown;

[0035] Figure 2 This invention illustrates a microsecond-level synchronization triggering timing diagram for a multi-camera imaging system based on microsecond-level dynamic synchronization control.

[0036] Figure 3 This invention illustrates a spatial-temporal complementary imaging method for a multi-camera imaging system based on microsecond-level dynamic synchronization control.

[0037] Figure 4 A schematic diagram of the STINet interpolation network application of a multi-camera imaging system based on microsecond-level dynamic synchronization control according to the present invention is shown. Detailed Implementation

[0038] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0039] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0040] This invention belongs to the cross-technical field of image sensors, parallel image acquisition, high-performance imaging systems and deep vision computing. Specifically, it relates to a multi-camera array image acquisition and fusion system that combines microsecond-level high-precision synchronous control circuit, time-sharing exposure strategy and deep learning interpolation reconstruction network. The purpose is to improve the existing shortcomings of traditional technologies. The specific innovations are as follows: (1) A microsecond-level synchronous trigger control module based on delay-locked array (DLL) + programmable logic (FPGA) is constructed to realize the "software and hardware combination" timing alignment at the camera level; (2) A time-sharing micro-shift exposure mechanism (TSE) is proposed, in which each camera acquires different time slices of the same moving target to achieve spatial-temporal complementarity; (3) An end-to-end spatio-temporal interpolation neural network (STINet) is constructed to realize the reconstruction of low frame image sequences into ultra-high frame rate, high-resolution dynamic image streams.

[0041] Specifically, Figure 1 A schematic diagram of the structure of a multi-camera imaging system based on microsecond-level dynamic synchronization control according to this application is shown.

[0042] like Figure 1 As shown, this application discloses a multi-camera imaging system based on microsecond-level dynamic synchronization control, comprising:

[0043] The system comprises a multi-camera imaging array, a synchronization control module, a timing management module, a data acquisition and temporary storage module, and a processing module.

[0044] The multi-camera imaging array includes several imaging cameras arranged in an orderly manner for imaging;

[0045] The synchronization control module includes a synchronization controller, which provides an independent microsecond-level delay configuration for each imaging camera;

[0046] The timing management module includes a time-sharing scheduler for configuring the exposure time slots of each imaging camera;

[0047] The data acquisition and storage module includes an acquisition circuit and a memory, used to acquire images and cache them;

[0048] The processing module includes an image registration and alignment module and an image fusion and interpolation module. The image registration and alignment module is used to perform image matching and alignment, and the image fusion and interpolation module is used to perform image fusion and interpolation on the image sequence.

[0049] It should be noted that, in this embodiment, the multi-camera imaging array includes several ordered imaging cameras for imaging. In application, the array structure of the imaging array supports one-dimensional linear structure, two-dimensional rectangular structure, and three-dimensional spherical structure. The synchronization control module includes a synchronization controller based on PDA (Programmable Delay Array) + DPLL (Digital Phase-Locked Loop) circuit, used to provide independent microsecond-level delay configuration for each imaging camera. The timing management module includes a time-division scheduler used to configure the exposure time slot of each imaging camera. The data acquisition and temporary storage module includes an acquisition circuit and a memory used to acquire images and cache them. Each data stream is cached in local SRAM (Static Random-Access Memory) to avoid I / O port blockage.

[0050] Furthermore, the processing module includes an image registration and alignment module and an image fusion and interpolation module. The image registration and alignment module is used to perform image matching and alignment, specifically based on an improved optical flow + feature alignment method. The image fusion and interpolation module is used to perform image fusion interpolation on the image sequence, specifically using a deep neural network to spatiotemporally interpolate the image sequence to generate continuous, high-quality frame images. The process will be described in detail in the subsequent specification.

[0051] According to an embodiment of the present invention, the trigger time expression of the camera of the imaging camera is as follows:

[0052] ;

[0053] in, For the first Triggering time of each camera, For reference trigger time, For the first Programmable delay time for each imaging camera.

[0054] It should be noted that, in this embodiment, the current imaging system includes: For each camera, the system's reference trigger time is... Accordingly, the first Trigger time of each camera ,in, For the first The programmable delay time of each imaging camera, in microseconds.

[0055] Furthermore, the formula for calculating the programmable delay time of the imaging camera is as follows:

[0056] ;

[0057] in, For the first Programmable delay time for each imaging camera For the first The delay steps of each camera, This refers to the master clock frequency of the synchronous controller, measured in MHz. Represents the set of positive integers. Represents the set of positive real numbers, specifically as follows: Figure 2 As shown, the timing diagram is a microsecond-level synchronous triggering diagram, specifically using three cameras. , and Let's take an example to illustrate.

[0058] According to an embodiment of the present invention, the synchronization controller includes a programmable delay array (PDA) and a digital phase-locked loop (DPLL) circuit, and through delay line adjustment, the maximum time difference among all imaging cameras is achieved. .

[0059] It should be noted that, in this embodiment, the synchronization controller includes a programmable delay array (PDA) and a digital phase-locked loop (DPLL) circuit. Adjustment of the resting delay line requires determining the maximum time difference among all imaging cameras. To achieve synchronous control.

[0060] According to an embodiment of the present invention, the target image obtained when observing the target is a continuous function. The specific location is The time is The brightness value, the first The imaging camera in the first The image at the second sampling time is expressed as follows:

[0061] ;

[0062] in, For the first The imaging camera in the first The image at the next sampling time, For position parameters, For time parameters, Spatial offset caused by the current frame's viewpoint shift. For the first The camera captured the first... Time offset during frames;

[0063] An approximate target image is obtained by jointly approximating and restoring images from different imaging cameras.

[0064] It should be noted that, in this embodiment, as Figure 3 As shown, this is a schematic diagram of spatial-temporal complementary imaging. The Temporal Shift Exposure (TSE) mechanism essentially involves offsetting the camera's acquisition window on a microscopic time scale, allowing different imaging cameras to observe images of the target at different time slices. In this case, a multi-view camera captures the same target at different times to achieve spatial-temporal complementary imaging.

[0065] Specifically, the target image obtained when observing the target is a continuous function. The specific location is The time is The brightness value, the first The imaging camera in the first The image at the second sampling time is expressed as follows: ;in, For the first The imaging camera in the first The image at the next sampling time, For position parameters, For time parameters, Spatial offset caused by the current frame's viewpoint shift. For the first The camera captured the first... With the time offset at frame rate, an approximate target image can then be obtained by jointly approximating and reconstructing images from different imaging cameras. STINet corresponds to the Spatio-Temporal Interpolation Network.

[0066] According to an embodiment of the present invention, the processing module uses a spatiotemporal interpolation neural network for image matching, alignment and fusion, wherein the spatiotemporal interpolation neural network includes a feature extraction layer, a spatial feature alignment module, a temporal fusion module and an upsampling module.

[0067] It should be noted that, in this embodiment, as Figure 4 The diagram shows an application of the STINet interpolation network. The STINet spatiotemporal interpolation neural network includes a feature extraction layer, a spatial feature alignment module, a temporal fusion module, and an upsampling module. The feature extraction layer includes a residual backbone and multiscale features, corresponding to a multiscale residual feature extraction layer. The spatial feature alignment module performs image alignment based on learnable optical flow. The temporal fusion module performs fusion based on temporal attention and convolution. Correspondingly, the upsampling module performs subpixel convolution or deconvolution.

[0068] Furthermore, the loss function of the spatiotemporal interpolation neural network STINet is as follows:

[0069] ;

[0070] in, To reconstruct the error loss, This corresponds to the L1 error. To predict the image, For real images, The total variation smoothing loss is denoted as , where , , For gradient operators, and This is the inter-frame consistency regularization loss (specifically, regularization calculation is performed based on the difference between neighboring frames). For weight hyperparameters.

[0071] Furthermore, when performing image matching, alignment, and fusion, such as Figure 4 As shown, this specifically involves multiple time-slice image frames from the imaging camera. The input is fed into a spatiotemporal interpolation neural network to obtain temporally continuous interpolated image frames. ,in, / For pixel height, / The width is in pixels. This represents the number of network feature channels.

[0072] According to an embodiment of the present invention, the resolution of the output image ,in, Sampling factor on the interpolation network, This is the original image size.

[0073] It should be noted that, in this embodiment, the imaging system includes: There are 1 camera, and the frame rate of each camera is 1000. The unit is fps, and the sampling factor on the interpolation network is . Correspondingly, the equivalent frame rate The resolution of the output image ,in, This is the original image size.

[0074] Specifically, as shown in Table 1, this table presents a comparison of the performance dimensions of this application and traditional prior art.

[0075] Table 1. Comparison of performance metrics

[0076] .

[0077] The present invention discloses a multi-camera imaging system based on microsecond-level dynamic synchronization control, which features high time synchronization accuracy, multi-channel acquisition, and multi-resolution interpolation fusion spatiotemporal reconstruction. It combines ultra-high resolution with image acquisition and temporal restoration at a frame rate of millions of frames per second, and can be widely applied to high-end application scenarios such as high-speed physics experiments, femtosecond / picosecond chemical reaction observation, aerospace vision measurement, and industrial ultrafast defect detection.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0079] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0080] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0081] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A multi-camera imaging system based on microsecond-level dynamic synchronization control, characterized in that, include: The system comprises a multi-camera imaging array, a synchronization control module, a timing management module, a data acquisition and temporary storage module, and a processing module. The multi-camera imaging array includes several imaging cameras arranged in an orderly manner for imaging; The synchronization control module includes a synchronization controller, which provides an independent microsecond-level delay configuration for each imaging camera; The timing management module includes a time-sharing scheduler for configuring the exposure time slots of each imaging camera; The data acquisition and storage module includes an acquisition circuit and a memory, used to acquire images and cache them; The processing module includes an image registration and alignment module and an image fusion and interpolation module. The image registration and alignment module is used to perform image matching and alignment, and the image fusion and interpolation module is used to perform image fusion and interpolation on the image sequence. The trigger time expression for the imaging camera's camera is as follows: ； in, For the first Triggering time of each camera, For reference trigger time, For the first Programmable delay time for each imaging camera; The formula for calculating the programmable delay time of an imaging camera is as follows: ； in, For the first Programmable delay time for each imaging camera For the first The delay steps of each camera, This refers to the master clock frequency of the synchronous controller. The synchronization controller includes a programmable delay array (PDA) and a digital phase-locked loop (DPLL) circuit, which adjusts the delay lines to achieve the maximum time difference among all imaging cameras. ; The processing module uses a spatiotemporal interpolation neural network for image matching, alignment, and fusion. The spatiotemporal interpolation neural network includes a feature extraction layer, a spatial feature alignment module, a temporal fusion module, and an upsampling module. The loss function of the spatiotemporal interpolation neural network is as follows: ； in, To reconstruct the error loss, The total variation smoothing loss, For inter-frame consistency regularization loss, These are the weight hyperparameters; When performing image matching, alignment, and fusion, multiple time-slice image frames from the imaging camera are specifically combined. The input is fed into a spatiotemporal interpolation neural network to obtain temporally continuous interpolated image frames. ,in, The pixel height of the original time-slice image frame. The pixel height of the time-continuously interpolated image frame. The pixel width of the original time-slice image frame. The pixel width of the time-continuously interpolated image frame. This represents the number of network feature channels.

2. The multi-camera imaging system based on microsecond-level dynamic synchronization control according to claim 1, characterized in that, The array structure of the multi-camera imaging array supports one-dimensional linear structure, two-dimensional rectangular structure and three-dimensional spherical structure.

3. The multi-camera imaging system based on microsecond-level dynamic synchronization control according to claim 1, characterized in that, The target image obtained when observing the target is a continuous function. The specific location is The time is The brightness value, the first The imaging camera in the first The image at the second sampling time is expressed as follows: ； in, For the first The imaging camera in the first The image at the next sampling time, For position parameters, For time parameters, Spatial offset caused by the current frame's viewpoint shift. For the first The camera captured the first... Time offset during frames; An approximate target image is obtained by jointly approximating and restoring images from different imaging cameras.

4. The multi-camera imaging system based on microsecond-level dynamic synchronization control according to claim 1, characterized in that, Output image resolution ,in, Sampling factor on the interpolation network, This is the original image size.

Images