Time domain mapping encryption imaging method and system based on event camera

Abstract

The invention provides a time domain mapping encryption imaging method and system based on an event camera, and belongs to the field of computer vision hardware encryption imaging. The method comprises the steps that a receiver builds an asynchronous time domain mapping imaging system and constructs a space coding secret key sequence, then obtains an SLM modulation pattern sequence, obtains a noise SLM modulation pattern sequence and sends the SLM modulation pattern sequence to a sender; and the receiver carries out calibration based on the SLM modulation pattern sequence to obtain a space decoding secret key sequence. And the sender obtains the imaging event information and the encrypted time decoding secret key sequence based on the noise SLM modulation pattern sequence and sends the imaging event information and the encrypted time decoding secret key sequence to the receiver. And the receiver obtains a fragmented grey-scale map based on the spatial decoding secret key sequence and the received information, and then obtains a dense and complete grey-scale image through a video completion network based on the fragmented grey-scale map. According to the invention, real-time encryption can be realized, and the leakage risk before encrypted information transmission is reduced; moreover, the method is not sensitive to the collection environment, is high in decoding efficiency, and is low in implementation difficulty.

Claims

1. A time domain mapping encrypted imaging method based on an event camera, characterized in that: include: 1) The receiving party builds an asynchronous time domain mapping imaging system for obtaining imaging event information, wherein the system includes an imaging objective lens, a front relay mirror, a spatial light modulator, a rear relay mirror, and an event camera; 2) The receiver constructs a spatial coding key sequence, obtains an SLM modulation pattern sequence based on the spatial coding key sequence, then adds temporal and spatial noise to the SLM modulation pattern sequence, obtains a noisy SLM modulation pattern sequence and sends it to the sender; 3) The spatial light modulator of the receiver plays the SLM modulation pattern sequence, and at the same time uses the system to shoot a uniformly illuminated flat plate to calibrate and obtain the spatial decoding key sequence; 4) The sender builds the same asynchronous time domain mapping imaging system as the receiver. The sender's spatial light modulator plays the noise SLM modulation pattern sequence and uses its own system to shoot the target scene. The event camera records the imaging event information and the moment of switching the noise SLM modulation pattern in the noise SLM modulation pattern sequence. The moment of switching the noise SLM modulation pattern is the time decoding key sequence required for encrypted imaging decoding. The sender then encrypts the time decoding key sequence and finally sends the imaging event information and the encrypted time decoding key sequence to the receiver. 5) The receiver decodes the encrypted time decoding key sequence, decrypts the imaging event information based on the decoding result and the spatial decoding key sequence, and obtains a fragmented grayscale image; 6) The receiver obtains a dense and complete grayscale image based on the fragmented grayscale image through a video completion network, that is, obtains an image corresponding to the target scene.

2. The time domain mapping encrypted imaging method based on event camera according to claim 1 is characterized in that: In the asynchronous time-domain mapping imaging system, the imaging objective lens images the external scene onto a virtual image plane, the front relay mirror images the image on the virtual image plane onto the surface of the spatial light modulator, the spatial light modulator can select some pixels on its surface to turn on, and then continue to send part of the field of view light to the rear relay mirror, the rear relay mirror receives the field of view light modulated by the spatial light modulator, and images it onto the image plane of the event camera, the event camera collects brightness change information on the image plane of the event camera, and obtains imaging event information; wherein the switching frequency of the spatial light modulator is greater than or equal to 1000 Hz, and the contrast of the spatial light modulator is greater than or equal to 100:1; the spatial light modulator is a digital micromirror device.

3. The method according to claim 1, characterized in that In step 2), a spatial coding key sequence is constructed, including: The spatial coding key sequence is composed of p×p spatial coding keys arranged in a preset order, and the spatial coding keys are used to control the position of the newly turned-on pixel of the spatial light modulator; Let the spatial encoding secret key be an image with width \(W\) and height \(H\). The spatial encoding secret key is divided into grid blocks with a total number of \(W\times H / (s\times p\times s\times p)\). Each grid block has a size of \((s\times p)\times(s\times p)\). Then each grid block is split into \(p\times p\) conducting blocks, and each conducting block has a size of \(s\times s\). For each spatial encoding secret key, among the \(p\times p\) conducting blocks in each grid block, one conducting block is randomly selected to be conducting, that is, this conducting block is set to 1, and the remaining conducting blocks are non - conducting, that is, the remaining conducting blocks are set to 0. Among them, the conducting blocks conducted in the same grid block for different spatial encoding secret keys are different. Finally, \(p\times p\) spatial encoding secret keys with non - repeating and non - missing conducting positions are obtained.

4. The method according to claim 1, characterized in that In step 2), construct a spatial encoding secret key sequence, including: The spatial encoding secret key sequence is composed of \(p\times p\) spatial encoding secret keys arranged in a preset order. The spatial encoding secret key is used to control the positions of the newly - conducting pixels of the spatial light modulator. Let the spatial encoding secret key be an image with width \(W\) and height \(H\). Randomly generate \(d\) points in the image with width \(W\) and height \(H\). Among them, the \(d\) points include the 4 endpoints of the image, and \(0.001\times W\times H < d < 0.01\times W\times H\). Then generate a Voronoi diagram for the \(d\) points. The Voronoi diagram divides the image into \(d\) regions, and each region corresponds to a point. Divide all points into \(p\times p\) groups on average. The regions corresponding to the points with the same serial number are set to 1, representing the conducting regions of the spatial encoding secret key of this serial number, and the remaining regions are set to 0, representing non - conducting. Finally, \(p\times p\) spatial encoding secret keys with non - repeating and non - missing conducting positions are obtained.

5. The method according to claim 3, characterized in that: In step 2), let the switching time interval of the SLM modulation pattern in the SLM modulation pattern sequence by the spatial light modulator be \(\tau\). In the system, the conduction duration of each pixel of the spatial light modulator each time is \(M\times\tau\), and the reset duration is \(N\times\tau\), where \(0 < M,N < p\times p\), and \(M + N=p\times p\). Therefore, the initial SLM modulation pattern in the SLM modulation pattern sequence is an image where all pixels are non - conducting, that is, the SLM modulation pattern at this time is an all - 0 image. The \(k\) - th image in the SLM modulation pattern sequence is the superposition result of the first \(k\) spatial encoding keys, where \(0 < k\leq M\). When \(k > M\), the \(k\) - th SLM modulation pattern in the SLM modulation pattern sequence is the superposition result of the \((k - M)\%\ (p\times p)\) - th spatial encoding secret key to the \(k\%\ (p\times p)\) - th spatial encoding secret key. Add time and spatial noise to the SLM modulation pattern sequence, including: Add random spatial noise to each SLM modulation pattern in the SLM modulation pattern sequence. The form of the random spatial noise is \(r\) discrete pixel points with a value of 1, where \(0 < r <(W\times H)\times N / (10\times(M + N))\). The random spatial noise is added to the pixels in the reset state and within the first \((N / 2)\times\tau\) time of the reset duration. Change the switching time interval of the SLM modulation pattern from τ to in, is a Gaussian random number with mean 0 and range ±σ, where σ is less than τ / 2.

6. The method according to claim 5, characterized in that Step 3) includes: 3.1) Place a uniformly illuminated flat plate in front of the imaging objective lens of the asynchronous time domain mapping imaging system of the receiver, and adjust the brightness of the flat plate so that each pixel on the image plane of the event camera in the asynchronous time domain mapping imaging system that enters the on state can trigger a positive event within τ time, where the positive event refers to an event generated when the pixel becomes brighter and reaches the trigger threshold of the event camera; 3.2) The spatial light modulator plays the SLM modulation pattern sequence, and captures the moment of each SLM modulation pattern switching and the positive events triggered by each pixel point on the image plane of the event camera during the switching of the SLM modulation pattern through an event camera that is hard synchronized with the spatial light modulator; 3.3) Intercept the i-th SLM modulation pattern switching time t i Until the next switching time t i +τ, construct a binary image, set the spatial position of the positive event to 1, and the other positions to 0, and obtain the i-th spatial decoding key D i ; 3.4) Obtain the spatial decoding keys of p×p switching moments in sequence, use the morphological opening operation to filter out the bright spot noise in the spatial decoding key, and then use the morphological closing operation to fill the holes in the spatial decoding key to obtain the corrected spatial decoding key. All the corrected spatial decoding keys are arranged in sequence to form a spatial decoding key sequence; At the same time, starting from the first corrected spatial decoding key in the spatial decoding key sequence, each p×p corrected spatial decoding keys generates an intra-frame time deviation indication; the intra-frame time deviation indication is a matrix, and for the conduction position of the i-th corrected spatial decoding key in each p×p corrected spatial decoding key, the corresponding position of the corresponding intra-frame time deviation indication is assigned a value of i×τ.

7. The method according to claim 1, characterized in that In step 4), the time decoding key sequence is encrypted using the AES symmetric encryption method.

8. The method according to claim 6, characterized in that The step 5) comprises: 5.1) The receiver decodes the encrypted time decoding key sequence to obtain the time decoding key sequence; 5.2) Let the switching time of a noise SLM modulation pattern be c in the entire time decoding key sequence, then the switching time is t c , and then select the modified spatial decoding key with sequence number i=c%(p×p) from the spatial decoding key sequence obtained in step 3); 5.3) Obtain the time range [t c ,t c +Mτ] and the set of positive events whose spatial position in the modified spatial decoding key is set to 1; obtain the set of positive events at the spatial position (x, y) and whose timestamp is closest to t c positive event, calculate the timestamp of the positive event and the switching time t c The time difference t * (x, y), and then calculate the gray value I(x, y) at the location where the positive event occurs according to the time domain mapping imaging principle. The calculation formula is: Among them, C PD is the capacitance of the photodiode of the event camera; V ref is the reference voltage of the event camera, which is 0; TR(t) is the light intensity modulation function, which is a unit step function ε(t) that jumps from t = 0 to 1; V thd is the threshold voltage preset by the event camera; β is the constant for adjusting the position of the inverse mapping interval; 5.4) In a modified spatial decoding key, each pixel whose spatial position is set to 1 has a corresponding grayscale value I(x,y). Each grayscale value is normalized according to the range [1 / (M×τ+β), 1 / β] to obtain the normalized grayscale value All normalized grayscale values ​​in the modified spatial decoding key A fragmented grayscale image is formed, which is the fragmented grayscale image of the switching moment of the noise SLM modulation pattern corresponding to the corrected spatial decoding key; wherein, a fragmented grayscale image can be obtained at the switching moment of each noise SLM modulation pattern.

9. The method according to claim 8, characterized in that The step 6) comprises: 6.1) Starting from the first fragmented grayscale image, each p×p fragmented grayscale images are grouped, and then the p×p fragmented grayscale images in each group are stacked and synthesized into a complete asynchronous grayscale frame, and finally multiple asynchronous grayscale frames are obtained; wherein each asynchronous grayscale frame corresponds to p×p corrected spatial decoding keys, that is, corresponds to an intra-frame time deviation indication; 6.2) Use the RAFT model to estimate the optical flow of asynchronous grayscale frames and obtain the coarse optical flow between frames with deviation; 6.3) Normalize each asynchronous grayscale frame and its corresponding intra-frame temporal deviation first, and then stack them into an image stack tensor; input the image stack tensor into 1 input convolution layer and 3 residual convolution layers in sequence to extract features and obtain a feature tensor; stack the feature tensor and the corresponding inter-frame coarse optical flow with deviation into an optical flow correction input tensor, and then input the optical flow correction input tensor into 1 input convolution layer, 1 residual convolution layer and 1 output convolution layer in sequence, and finally obtain an optical flow correction output tensor with a width of W, a height of H and a number of channels of 2×p×p; reshape the optical flow correction output tensor into p×p debiased intra-frame optical flows with a width of W, a height of H and a number of channels of 2; 6.4) Pair the fragmented grayscale image obtained in step 5.4) with the debiased intra-frame optical flow obtained in step 6.3) one by one, and input them into the video completion network together after pairing to obtain a dense and complete grayscale image, that is, obtain the image corresponding to the target scene.

10. A time domain mapping imaging system for implementing the method of claim 1, characterized in that: It includes a modulation pattern acquisition module, a spatial decoding key sequence acquisition module, an acquisition module, a fragmented grayscale image acquisition module and a dense grayscale image acquisition module; The modulation pattern acquisition module is used to build an asynchronous time domain mapping imaging system, which includes an imaging objective lens, a front relay mirror, a spatial light modulator, a rear relay mirror and an event camera; and constructing a spatial coding key sequence, obtaining an SLM modulation pattern sequence based on the spatial coding key sequence, adding temporal and spatial noise to the SLM modulation pattern sequence, obtaining a noisy SLM modulation pattern sequence and sending it to an acquisition module; The spatial decoding key sequence acquisition module is used to make the spatial light modulator play the SLM modulation pattern sequence, and at the same time use the asynchronous time domain mapping imaging system to shoot a uniformly illuminated flat plate to calibrate and obtain the spatial decoding key sequence; The acquisition module is used to build an asynchronous time domain mapping imaging system, and the spatial light modulator in the asynchronous time domain mapping imaging system plays the noise SLM modulation pattern sequence, and uses its own asynchronous time domain mapping imaging system to shoot the target scene. The event camera records the imaging event information and the moment of switching the noise SLM modulation pattern in the noise SLM modulation pattern sequence. The moment of switching the noise SLM modulation pattern is the time decoding key sequence required for encrypted imaging decoding, and then the time decoding key sequence is encrypted; finally, the imaging event information and the encrypted time decoding key sequence are sent to the fragmented grayscale image acquisition module; The fragmented grayscale image acquisition module is used to decode the encrypted time decoding key sequence, decrypt the imaging event information in combination with the decoding result and the space decoding key sequence, and obtain a fragmented grayscale image; The dense grayscale image acquisition module is used to obtain a dense and complete grayscale image based on the fragmented grayscale image through a video completion network, that is, to obtain an image corresponding to the target scene.

Images