Light-weight high-speed imaging system and method based on coding exposure

Abstract

The invention relates to the technical field of high-speed imaging, in particular to a lightweight high-speed imaging system and method based on coding exposure, and the system comprises imaging hardware which is used for collecting an asymmetric exposure coding sequence for random coding exposure of a target scene in a time domain dimension, generating a coded blurred image according to the asymmetric exposure coding sequence; the upper computer is used for acquiring an exposure coding sequence, a serial number of an initial video frame corresponding to the exposure coding sequence and a coding blurred image, generating time domain information according to the exposure coding sequence and the video frame serial number, generating spatial domain information according to the coding blurred image, and analyzing a target video frame according to the time domain information and the spatial domain information, the definition of the target video frame is greater than that of the initial video frame. Therefore, the problems that an imaging system in the prior art is high in cost, heavy and the like are solved.

Description

Technical Field

[0001] This application relates to the field of high-speed imaging technology, and particularly to a lightweight high-speed imaging system and method based on coded exposure. Background Art

[0002] High-speed imaging has wide application value in fields such as driverless, sports live broadcast, scientific research observation, military security, etc. At the same time, it is also a difficult problem in the field of imaging. The main difficulties of this technology are mainly two. One is the challenge to the image signal-to-noise ratio brought by the short exposure time required for high-speed imaging, and the other is the pressure on the data transmission bandwidth caused by high-throughput data acquisition.

[0003] Currently, high-speed and high-resolution imaging systems are generally realized by adopting high-quality hardware devices, such as high-sensitivity and low-noise sensors, dedicated high-bandwidth data transmission lines, etc. The overall cost is very high, and continuous long-term acquisition is generally not possible. In scientific research, there are also some dedicated high-speed imaging devices such as streak cameras. It performs high-speed scanning imaging through voltage deflection, and can complete the observation of one-dimensional signals with nanosecond resolution, and is often used to observe some high-speed physical phenomena. However, its resolution is relatively low and it requires a laser as the illumination light source, so the application scenario is limited and the price is also very expensive. There are also some high-speed imaging systems that rely on multi-camera interleaved acquisition and post-fusion to achieve, that is, the acquisition moments of each camera are staggered from each other, and finally the videos collected by all cameras are fused into a high-speed video through algorithms. The disadvantage of this scheme is that the system is relatively bulky and inflexible. Summary of the Invention

[0004] This application provides a lightweight high-speed imaging system and method based on coded exposure to solve problems such as the high cost and bulkiness of the existing imaging systems.

[0005] In a first aspect embodiment of this application, a lightweight high-speed imaging system based on coded exposure is provided, including: an imaging hardware for collecting an asymmetric exposure coding sequence of a target scene with random coded exposure in the time domain dimension, and generating a coded blurred image according to the asymmetric exposure coding sequence; a host computer for obtaining the exposure coding sequence, the serial number of the initial video frame corresponding to the exposure coding sequence, and the coded blurred image, generating time domain information according to the exposure coding sequence and the video frame serial number, generating spatial domain information according to the coded blurred image, and parsing out the target video frame according to the time domain information and the spatial domain information, where the clarity of the target video frame is greater than that of the initial video frame.

[0006] Optionally, the imaging hardware includes a camera module and a controller, where the controller controls the camera module to encode and compress the acquisition of the coded exposure image of the target scene, and generates an asymmetric exposure coding sequence according to the coded exposure image.

[0007] Optionally, the camera module includes a camera and a sensor that support the external trigger control mode of the shutter.

[0008] Optionally, the camera module includes a camera that does not support the external trigger control mode of the shutter, a programmable external shutter, and a sensor. The controller controls the programmable external shutter based on a shutter control pulse and synchronously controls the camera that does not support the external trigger control mode of the shutter based on a synchronous trigger signal.

[0009] An embodiment of the second aspect of the present application provides a lightweight high-speed imaging method based on coded exposure. The method uses the lightweight high-speed imaging system based on coded exposure in the above embodiment for imaging. The method includes the following steps: obtaining an asymmetric exposure coding sequence of a target scene with random coded exposure in the time domain dimension; generating time domain information according to the exposure coding sequence and the serial number of the initial video frame corresponding to the exposure coding sequence, generating a coded blurred image according to the asymmetric exposure coding sequence, and generating spatial domain information according to the coded blurred image; parsing out a target video frame according to the time domain information and the spatial domain information, where the clarity of the target video frame is greater than that of the initial video frame.

[0010] Optionally, parsing out the target video frame according to the time domain information and the spatial domain information includes: inputting the time domain information and the spatial domain information into a BDINR reconstruction network, and the BDINR reconstruction network outputs the target video frame.

[0011] Optionally, the BDINR reconstruction network includes a spatial domain information embedding module, a time domain information embedding module, and an implicit neural representation module. The spatial domain information embedding module extracts spatial domain features from the spatial domain information; the time domain information embedding module extracts time domain features from the time domain information; the fusion feature of the spatial domain feature and the time domain feature of the last frame of the initial video frame is input into the implicit neural representation module, and the target video frame is reconstructed based on the fusion feature by the spatial domain feature and the time domain feature.

[0012] Optionally, the time domain information embedding module includes a linear layer and a GELU loss function.

[0013] Optionally, the network structures of the spatial domain information embedding module and the implicit neural representation module are the same, where the network structure includes a convolutional layer, a transposed convolutional layer, a residual module, and a ReLU loss function.

[0014] Optionally, the BDINR reconstruction network also reconstructs the current frame based on the fusion feature of the previous frame.

[0015] Thus, the present application has the following beneficial effects:

[0016] An embodiment of the present application constructs a lightweight high-speed imaging system based on coded exposure, including imaging hardware and a host computer. The imaging hardware collects an asymmetric exposure coding sequence of a target scene with random coded exposure in the time domain dimension, and generates a coded blurred image according to the asymmetric exposure coding sequence. The host computer generates time domain information according to the exposure coding sequence, the serial number of the initial video frame corresponding to the exposure coding sequence, and the coded blurred image, generates spatial domain information according to the coded blurred image, and resolves the target video frame according to the time domain information and the spatial domain information. By generating a coded blurred image through an asymmetric exposure coding sequence with random coded exposure in the time domain dimension, it is not necessary to introduce complex two-dimensional coding devices, with lower cost, lighter weight, better stability, and the collected asymmetric exposure coding sequence can solve the problem of uncertainty in the physical motion direction in the image during subsequent analysis and reconstruction. Moreover, coded exposure can avoid the loss of high-frequency information and improve the signal-to-noise ratio of the image. Thus, the technical problems of high cost and heaviness of the existing imaging system are solved.

[0017] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, where:

[0019] Figure 1 is a schematic diagram of a lightweight high-speed imaging system based on coded exposure according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of the principle of coded exposure to avoid high-frequency information loss according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of the principle of coded exposure to solve the uncertainty of the motion direction according to an embodiment of the present application;

[0022] Figure 4 is a flowchart of a lightweight high-speed imaging method based on coded exposure according to an embodiment of the present application;

[0023] Figure 5 is a structural diagram of a BDINR reconstruction network according to an embodiment of the present application;

[0024] Figure 6 is a specific implementation diagram of a BDINR reconstruction network according to an embodiment of the present application;

[0025] Figure 7The structural flowchart of a lightweight high-speed imaging system based on coded exposure provided according to an embodiment of the present application;

[0026] Figure 8 The schematic diagram of implementing coded exposure based on a programmable external shutter provided according to an embodiment of the present application. Detailed implementation manners

[0027] The embodiments of the present application are described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals indicate the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as a limitation to the present application.

[0028] Before describing the solutions of the embodiments of the present application, the following imaging technologies similar to the present application, coded-aperture single-exposure compressive imaging, are introduced first.

[0029] Coded-aperture single-exposure compressive imaging is similar to the lightweight high-speed imaging based on coded exposure proposed in the present application. The basic processes of both are to first use an ordinary low-speed camera to encode and acquire a motion-blurred image, and then decode and recover multiple consecutive clear video frames from it. Compared with the traditional scheme, since both first encode, compress and acquire, and then decode and recover multiple frames through algorithms, the amount of data during the acquisition process is small, and it will not cause data transmission bandwidth pressure. It can directly use an ordinary low-speed sensor to complete the acquisition, and the application scenarios are relatively wide. The difference between the two is that in coded-aperture single-exposure compressive imaging, spatio-temporal dimension random coding is performed, so a high-speed two-dimensional coding device (such as a digital micromirror array or ferroelectric liquid crystal) needs to be additionally introduced into the imaging system. Such a device is generally expensive and sensitive to factors such as system vibration. A slight deviation in position will affect the subsequent decoding and reconstruction. Therefore, such a system needs to be recalibrated before each data acquisition. While the present application only needs to perform random coding on the time domain dimension. Physically, only the control of the shutter opening and closing needs to be performed to achieve the regulation of exposure. Some cameras that support external shutter trigger control can directly implement coded exposure, and cameras that do not support this function can also achieve coded exposure by additionally introducing an external shutter and synchronizing it with the camera. The overall cost is relatively low, the spatial resolution is high, and the stability is relatively good.

[0030] Specifically, Figure 1 The schematic diagram of a lightweight high-speed imaging system based on coded exposure provided according to an embodiment of the present application.

[0031] As Figure 1 shown, the lightweight high-speed imaging system 10 based on coded exposure includes: an imaging hardware 11 and a host computer 12.

[0032] Among them, the imaging hardware 11 is used to collect an asymmetric exposure coding sequence of the target scene with random coded exposure in the time domain dimension, and generate a coded blurred image according to the asymmetric exposure coding sequence; the host computer 12 is used to obtain the exposure coding sequence, the serial number of the initial video frame corresponding to the exposure coding sequence, and the coded blurred image, generate time domain information according to the exposure coding sequence and the video frame serial number, generate spatial domain information according to the coded blurred image, and resolve the target video frame according to the time domain information and the spatial domain information, wherein the clarity of the target video frame is greater than that of the initial video frame.

[0033] It can be understood that the embodiment of the present application constructs a lightweight high-speed imaging system 10 based on coded exposure, including an imaging hardware 11 and a host computer 12. The imaging hardware 11 collects an asymmetric exposure coding sequence of the target scene with random coded exposure in the time domain dimension, and generates a coded blurred image according to the asymmetric exposure coding sequence. The host computer 12 generates time domain information according to the exposure coding sequence, the serial number of the initial video frame corresponding to the exposure coding sequence, and the coded blurred image, generates spatial domain information according to the coded blurred image, and resolves the target video frame according to the time domain information and the spatial domain information. By generating a coded blurred image through an asymmetric exposure coding sequence with random coded exposure in the time domain dimension, there is no need to introduce complex two-dimensional coding devices, the cost is relatively low, it is more lightweight, has better stability, and the collected asymmetric exposure coding sequence can solve the problem of physical motion direction uncertainty in the image during subsequent parsing and reconstruction, and coded exposure can avoid the loss of high-frequency information and improve the signal-to-noise ratio of the image.

[0034] In the embodiment of the present application, the imaging hardware 10 includes a camera module and a controller. Among them, the controller controls the camera module to encode and compress the collected coded exposure image of the target scene, and generates an asymmetric exposure coding sequence according to the coded exposure image.

[0035] It can be understood that the imaging hardware 10 in the embodiment of the present application includes a camera module and a controller. The controller controls the camera module to encode and compress the collected coded exposure image of the target scene, generates an asymmetric exposure coding sequence according to the coded exposure image, and encodes and compresses the multi-frame high-speed scene for collection, which is equivalent to compressing the original multi-frame high-speed video frames into one frame and then collecting them on the sensor, reducing the amount of data, thereby reducing the requirement for data transmission bandwidth.

[0036] In the embodiment of the present application, the camera module includes a camera and a sensor that support the fast shutter external trigger control mode.

[0037] It can be understood that the camera module in the embodiments of the present application supports a camera sensor in a shutter trigger control mode, that is, in the embodiments of the present application, it is necessary to control the opening and closing of the shutter to achieve coded exposure, without the need to additionally introduce a high-speed two-dimensional coding device, and the cost is relatively low. For example, a camera in an external shutter trigger control mode can be IEEE DCAM Trigger Mode 5.

[0038] In the embodiments of the present application, the camera module includes a camera that does not support the external shutter trigger control mode, a programmable external shutter, and a sensor. Among them, the controller controls the programmable external shutter based on a shutter control pulse, and synchronously controls the camera that does not support the external shutter trigger control mode based on a synchronous trigger signal.

[0039] It can be understood that the camera module in the embodiments of the present application may also include a camera that does not support the external shutter trigger mode, a programmable external shutter, and a sensor. The controller controls the programmable external shutter based on a shutter control pulse, and synchronously controls the camera that does not support the external shutter trigger control mode based on a synchronous trigger signal.

[0040] Specifically, the camera module in the embodiments of the present application includes a camera that supports the external shutter trigger control mode and also includes a camera that does not support the external shutter trigger control mode. For a camera that does not support the external shutter trigger control mode, only one external shutter needs to be introduced, and the system flexibility is relatively high. Among them,

[0041] For the method of coded exposure implemented based on an external shutter + a general camera, the opening and closing state of this external shutter is controlled by an input pulse, and the controller can output a specified shutter control pulse to control it. At the same time, the microcontroller also inputs a synchronous trigger signal to the camera to control it to cooperate with the external shutter for image acquisition.

[0042] In addition, it should be noted that the principle of coded exposure to avoid the loss of high-frequency information is as Figure 2 shown. The motion blur caused by one-dimensional linear motion is used to simplify the explanation. For the traditional ordinary exposure method, the motion blur kernel formed is a continuous straight line, and there are periodic zeros in the frequency domain of the blur kernel. These zeros will cause the loss of information of the corresponding frequency components in the blurred image. If the coded exposure method is adopted, the exposure method can be designed to affect the formation of the blur kernel (become a discontinuous straight line), so that there are no zeros in the frequency domain of the blur kernel, and the loss of the corresponding frequency components can be avoided.

[0043] The principle of coded exposure to solve the problem of uncertainty in the motion direction is as Figure 3As shown in the figure, there are two moving objects (a square and a circle) in the figure, moving along two possible directions from left to right (from the solid part to the dashed box), with the same moving distance, only the starting and ending points are different, and there are a total of 4 possible combinations of motion states. If conventional exposure is used, the resulting motion-blurred images are exactly the same, and the indistinguishable nature will lead to the problem of motion direction uncertainty during reconstruction; if asymmetric sequence coding exposure is used (such as 11101, the sequence is asymmetric), then the 4 resulting motion-blurred images are different and distinguishable, which can solve the problem of motion direction uncertainty during reconstruction. It should be noted that an asymmetric exposure coding sequence must be sampled because if a symmetric coding sequence is used (such as 11011), the 4 blurred images are still exactly the same and indistinguishable. Therefore, the embodiment of the present application uses asymmetric sequence coding exposure.

[0044] According to the lightweight high-speed imaging system based on coded exposure proposed in the embodiment of the present application, the imaging hardware acquires an asymmetric exposure coding sequence of random coded exposure of the target scene in the time domain dimension, and generates a coded blurred image according to the asymmetric exposure coding sequence. The host computer generates time domain information according to the exposure coding sequence, the serial number of the initial video frame corresponding to the exposure coding sequence, and the coded blurred image, generates spatial domain information according to the coded blurred image, resolves the target video frame according to the time domain information and the spatial domain information, generates a coded blurred image through an asymmetric exposure coding sequence of random coded exposure in the time domain dimension, without introducing complex two-dimensional coding devices, with lower cost, being more lightweight, having better stability, and collecting an asymmetric exposure coding sequence, which can solve the problem of physical motion direction uncertainty in the subsequent analysis and reconstruction of the image, and coded exposure can avoid the loss of high-frequency information and improve the signal-to-noise ratio of the image.

[0045] Next, refer to the accompanying drawings to describe the lightweight high-speed imaging method based on coded exposure proposed in the embodiment of the present application.

[0046] Figure 4 It is a flowchart of the lightweight high-speed imaging method based on coded exposure in the embodiment of the present application.

[0047] As shown in Figure 4 the figure, the lightweight high-speed imaging method based on coded exposure uses the lightweight high-speed imaging system based on coded exposure in the above embodiment for imaging, including the following steps:

[0048] In step S101, an asymmetric exposure coding sequence of random coded exposure of the target scene in the time domain dimension is obtained.

[0049] Among them, the target scene can be a high-speed moving scene.

[0050] It can be understood that the embodiments of the present application can obtain an asymmetric exposure coding sequence for random coded exposure of the target scene in the time domain dimension.

[0051] In step S102, time domain information is generated according to the exposure coding sequence and the serial number of the initial video frame corresponding to the exposure coding sequence, a coded blurred image is generated according to the asymmetric exposure coding sequence, and spatial domain information is generated according to the coded blurred image.

[0052] Among them, the spatial domain information is the coded blurred graph, and the time domain information is the exposure coding sequence and the serial number of the corresponding initial video frame.

[0053] It can be understood that the embodiments of the present application can generate time domain information according to the exposure coding sequence and the serial number of the initial video frame corresponding to the exposure coding sequence, generate a coded blurred graph according to the asymmetric exposure coding sequence, and further generate spatial domain information according to the coded blurred image.

[0054] In step S103, the target video frame is parsed according to the time domain information and the spatial domain information, wherein the clarity of the target video frame is greater than that of the initial video frame.

[0055] It can be understood that the embodiments of the present application can parse the target video frame according to the time domain information and the spatial domain information to realize the reconstruction of the video frame in the target scene. By generating a coded blurred image through an asymmetric exposure coding sequence for random coded exposure in the time domain dimension, there is no need to introduce a complex two-dimensional coding device, the cost is low, the stability is better, and since the obtained exposure coding sequence is asymmetric, the problem of uncertainty in the physical motion direction in the image during subsequent parsing and reconstruction can be solved, and coded exposure can avoid the loss of high-frequency information and improve the signal-to-noise ratio of the image.

[0056] In the embodiments of the present application, parsing the target video frame according to the time domain information and the spatial domain information includes: inputting the time domain information and the spatial domain information into the BDINR reconstruction network, and the BDINR reconstruction network outputs the target video frame.

[0057] It can be understood that the embodiments of the present application can input the time domain information and the spatial domain information into the BDINR reconstruction network, and the BDINR reconstruction network outputs the target video frame. The specific structure of the BDINR reconstruction network is as follows.

[0058] In the embodiments of the present application, the BDINR reconstruction network includes a spatial domain information embedding module, a time domain information embedding module, and an implicit neural representation module. Among them, the spatial domain information embedding module extracts spatial domain features from the spatial domain information; the time domain information embedding module extracts time domain features from the time domain information; the fusion feature of the spatial domain feature and the time domain feature of the last frame of the initial video frame is input into the implicit neural representation module, and the target video frame is reconstructed based on the fusion feature by the spatial domain feature and the time domain feature.

[0059] It can be understood that the BDINR reconstruction network in the embodiments of the present application includes a spatial domain information embedding module, a time domain information embedding module, and an implicit neural representation module. The spatial domain information completes feature extraction through the spatial domain information embedding module (SEM). After the time domain information is mapped to a high dimension through sine position encoding, it then completes feature extraction through the time domain information embedding module (TEM). Finally, the time domain features and spatial domain features of each frame are fused and input into the implicit neural representation module (INRV) to complete the reconstruction of the corresponding frame.

[0060] In the embodiments of the present application, the time domain information embedding module includes a linear layer and a GELU loss function.

[0061] In the embodiments of the present application, the network structures of the spatial domain information embedding module and the implicit neural representation module are the same. Among them, the network structure includes a convolutional layer, a transposed convolutional layer, a residual module, and a ReLU loss function.

[0062] In the embodiments of the present application, the BDINR reconstruction network also reconstructs the current frame based on the fused features of the previous frame.

[0063] It can be understood that the BDINR reconstruction network also reconstructs the current frame based on the fused features of the previous frame. The autoregressive reconstruction strategy improves the smoothness between the reconstructed video frames and enhances the coherence and naturalness of the video.

[0064] Specifically, the structure of the BDINR reconstruction network in the embodiments of the present application is as Figure 5 shown.

[0065] The overall structure of the BDINR reconstruction network. The input information of the network includes two parts - the spatial domain information is the encoded blurred image, and the time domain information is the video frame number and the corresponding exposure encoding. The spatial domain information completes feature extraction through the spatial domain information embedding module (SEM). After the time domain information is mapped to a high dimension through sine position encoding, it then completes feature extraction through the time domain information embedding module (TEM). Finally, the time domain features and spatial domain features of each frame are fused and input into the implicit neural representation module (INRV) to complete the reconstruction of the corresponding frame. To improve the smoothness between the reconstructed video frames, an autoregressive reconstruction strategy is also introduced here, that is, the features of the previous frame are also used as input when reconstructing each frame, so as to provide auxiliary information for adjacent frames.

[0066] The specific implementation details of each module in the BDINR network are as Figure 6 shown. The time domain embedding module (TEM) is composed of a linear layer (Linear) and a GELU loss function; the other modules are composed of a convolutional layer (Conv), a transposed convolutional layer (DeConv), a residual module (ResBlock), and a ReLU loss function.

[0067] It should be noted that the foregoing explanation of the embodiments of the lightweight high-speed imaging system based on coded exposure also applies to the method of lightweight high-speed imaging based on coded exposure in this embodiment, and will not be elaborated here.

[0068] According to the method of lightweight high-speed imaging based on coded exposure proposed in the embodiments of the present application, an asymmetric exposure coding sequence that randomly codes and exposes in the time domain dimension for the target scene can be used. Time domain information is generated based on the exposure coding sequence and the serial number of the initial video frame corresponding to the exposure coding sequence. A coded blurred image is generated based on the asymmetric exposure coding sequence, and spatial domain information is generated based on the coded blurred image. Furthermore, the target video frame is parsed based on the time domain information and the spatial domain information. There is no need to introduce complex two-dimensional coding devices, the cost is relatively low, and the stability is better. Moreover, since the obtained exposure coding sequence is asymmetric, the problem of uncertainty in the physical motion direction in the image during subsequent parsing and reconstruction can be solved, and coded exposure can avoid the loss of high-frequency information, improving the signal-to-noise ratio of the image.

[0069] The solutions of the embodiments of the present application will be comprehensively described below in combination with the above-mentioned lightweight high-speed imaging system and method based on coded exposure, as Figure 7 shown, which is mainly divided into two parts: a hardware imaging system and a software reconstruction algorithm. At the hardware imaging system end, through the method of coded exposure, the high-speed scene is encoded, compressed, and collected to obtain a coded blurred image. At the software reconstruction algorithm end, through the BDINR reconstruction network, the coded blurred image is decoded and reconstructed to restore the original clear video frame.

[0070] I. Hardware Imaging System

[0071] A camera that supports the fast shutter external trigger control mode (IEEE DCAM Trigger Mode 5) or a conventional camera + programmable external shutter (as Figure 8 shown, Figure 8 is a schematic diagram of the coded exposure implementation based on a programmable external shutter), both can achieve the acquisition of coded exposure images. The core lies in being able to control the exposure of the camera shutter during the acquisition process. In the ordinary exposure method, the shutter is continuously open during the acquisition of one frame of image, but in the coded exposure method, the shutter needs to perform multiple rapid opening and closing switches according to the given 0, 1 sequence during the acquisition of one frame of image. The role played by coded exposure in high-speed imaging is mainly reflected in two aspects: 1) Avoid the loss of high-frequency information in the blurred image. As Figure 2 shown, there is a loss of high-frequency information in the ordinary blurred image collected under conventional exposure, but by using coded exposure (asymmetric coding sequence) to regulate the exposure process, it is possible to avoid the loss of high-frequency information in the collected coded blurred image, which is beneficial to subsequent algorithm reconstruction. 2) Solve the problem of uncertainty in the motion direction, as Figure 3As shown, when trying to reconstruct clear video frames from ordinary blurred images, it is essentially impossible to determine the motion direction of the reconstructed object because the same blurred image may be generated when the object moves forward and backward. Conversely, given a blurred image, the motion direction of the object cannot be determined. However, this problem can be solved by coded exposure (asymmetric coding sequence). When objects with different motion directions are captured using coded exposure, the resulting blurred images are different and distinguishable, thus solving the problem of motion direction uncertainty.

[0072] II. Software Reconstruction Algorithm

[0073] To recover multiple clear video frames from the motion-blurred images captured by coding, we designed a corresponding reconstruction algorithm BDINR, which achieved a relatively fast reconstruction speed and good reconstruction quality. As Figure 5 and Figure 6 shown, this algorithm is based on learnable implicit neural representation, which has good flexibility and can flexibly adjust the number of clear video frames recovered from each captured coded image. Implicit neural representation is a method of representing data (such as pictures, videos, 3D scenes, etc.) using neural networks, and it has shown better performance than conventional matrix representation methods in some image restoration tasks. Here, we use it to represent the video frames to be reconstructed and embed it into a trainable deep learning network framework for data-driven training. After training, as long as the captured motion-blurred image, the exposure coding sequence, and the serial number of the video frame to be reconstructed are input, the corresponding clear video frame can be reconstructed.

[0074] In summary, the solution of the embodiment of the present application uses simple and low-cost coded exposure technology to achieve lightweight, stable, and flexible high-speed imaging, and has the advantages of small data transmission bandwidth occupancy, high signal-to-noise ratio, low cost, and wide application. That is, the coded exposure method is used to encode and compress the acquisition of high-speed scenes. It is equivalent to compressing the original multi-frame high-speed video frames into one frame and then capturing them on the sensor. Therefore, the occupied data bandwidth is smaller, and continuous long-time acquisition can be performed based on ordinary low-speed sensors. Compared with traditional coded aperture single-exposure compression imaging, it is easier to implement, lower in cost, and better in stability. The BDINR reconstruction algorithm can complete the decoding and reconstruction of the captured video at a relatively fast speed and high quality.

[0075] In the description of this specification, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or N embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0076] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0077] Any process or method description shown in a flowchart or described in other ways herein can be understood to represent a module, segment, or part of code including one or N executable instructions for implementing a customized logical function or process, and the scope of the preferred embodiments of this application includes additional implementations, where the functions can be executed in a manner that is not in the order shown or discussed, including in a substantially simultaneous manner according to the functions involved or in the reverse order, which should be understood by those skilled in the art to which the embodiments of this application pertain.

[0078] It should be understood that each part of this application can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, the steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware as in another embodiment, any one of the following techniques well known in the art or a combination of them can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays, field programmable gate arrays, etc.

[0079] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method for implementing the above embodiments can be completed by instructing relevant hardware through a program, and the above program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0080] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A lightweight high-speed imaging system based on coded exposure, characterized in that: include: Imaging hardware, used for collecting an asymmetric exposure coding sequence of a target scene for random coding exposure in a time domain dimension, and generating a coded blurred image according to the asymmetric exposure coding sequence; The host computer is used to obtain an exposure coding sequence, a serial number of an initial video frame corresponding to the exposure coding sequence, and a coded blurred image, generate temporal information according to the exposure coding sequence and the video frame serial number, generate spatial information according to the coded blurred image, and parse a target video frame according to the temporal information and the spatial information, wherein the clarity of the target video frame is greater than the clarity of the initial video frame.

2. The lightweight high-speed imaging system based on coded exposure according to claim 1, characterized in that: The imaging hardware includes a camera module and a controller, wherein the controller controls the camera module to collect the coded exposure image of the target scene through coding compression, and generates an asymmetric exposure coding sequence according to the coded exposure image.

3. The lightweight high-speed imaging system based on coded exposure according to claim 2, characterized in that: The camera module includes a camera and a sensor that support a shutter external trigger control mode.

4. The lightweight high-speed imaging system based on coded exposure according to claim 2, characterized in that: The camera module includes a camera that does not support an external shutter trigger control mode, a programmable external shutter and a sensor, wherein the controller controls the programmable external shutter based on a shutter control pulse, and synchronously controls the camera that does not support an external shutter trigger control mode based on a synchronous trigger signal.

5. A lightweight high-speed imaging method based on coded exposure, characterized in that: The method uses the lightweight high-speed imaging system based on coded exposure according to any one of claims 1 to 4 to perform imaging, wherein the method comprises the following steps: Obtain an asymmetric exposure coding sequence for randomly coding and exposing the target scene in the time domain dimension; Generate temporal information according to the exposure coding sequence and the serial number of the initial video frame corresponding to the exposure coding sequence, generate a coded blurred image according to the asymmetric exposure coding sequence, and generate spatial information according to the coded blurred image; A target video frame is parsed according to the time domain information and the spatial domain information, wherein the definition of the target video frame is greater than the definition of the initial video frame.

6. The lightweight high-speed imaging method based on coded exposure according to claim 5, characterized in that: The step of parsing a target video frame according to the time domain information and the spatial domain information includes: The temporal information and the spatial information are input into a BDINR reconstruction network, and the BDINR reconstruction network outputs the target video frame.

7. The lightweight high-speed imaging method based on coded exposure according to claim 6, characterized in that: The BDINR reconstruction network includes a spatial information embedding module, a temporal information embedding module and an implicit neural representation module, wherein: The spatial information embedding module extracts spatial features from the spatial information; The time domain information embedding module extracts time domain features from the time domain information; The fused features of the spatial domain features and the temporal domain features of the last frame of the initial video frame are input into the implicit neural representation module, and the spatial domain features and the temporal domain features are reconstructed based on the fused features to obtain the target video frame.

8. The lightweight high-speed imaging method based on coded exposure according to claim 7, characterized in that: The time domain information embedding module includes a linear layer and a GELU loss function.

9. The lightweight high-speed imaging method based on coded exposure according to claim 7, characterized in that: The network structure of the spatial information embedding module and the implicit neural representation module is the same, wherein the network structure includes a convolutional layer, a deconvolutional layer, a residual module and a ReLU loss function.

10. The lightweight high-speed imaging method based on coded exposure according to claim 7, characterized in that: The BDINR reconstruction network also reconstructs the current frame based on the fused features of the previous frame.

Images