3D light field video encryption method based on salient object detection

Abstract

The application discloses a 3D light field video encryption method based on salient target detection, which comprises the following steps: salient target detection of a 2D video frame, and binarization processing to obtain a saliency region mask; extracting the 2D video frame pixels corresponding to the mask region and rearranging into a square image; using a cellular automaton to encrypt the square image; replacing the mask region of the 2D video frame with the encrypted square image; transmitting the encrypted 2D video frame and the mask as ciphertext; receiving the ciphertext and decrypting according to the reverse process; performing monocular depth rendering on the decrypted 2D video frame to obtain a 3D video frame; and reconstructing the 3D light field video through an optical display system. Through monocular depth rendering and saliency detection, the large data volume of the 3D light field video is converted into a small data volume of 2D video, only the saliency region of the 2D video is encrypted, the data volume to be encrypted is reduced, and the efficiency of the 3D light field video encryption is improved.

Description

Technical Field

[0001] This invention relates to the field of 3D video encryption technology, specifically to a 3D light field video encryption method based on salient target detection. Background Technology

[0002] As an emerging information carrier, 3D images can present the depth and three-dimensionality of objects, bringing viewers a more realistic visual experience, thus attracting increasing research interest. Compared to static 3D images, 3D light field videos can present dynamic information, provide naked-eye 3D visual effects, and bring richer scene perception, leading to increasing research and market applications of 3D light field videos. With the widespread research on 3D images, the security of 3D image transmission over public channels is also a pressing issue. Encryption is an effective means of information security protection, and many encryption methods for 3D images have been proposed. However, the secure transmission of 3D light field videos over public channels has received little research. The reason for this is that, compared to images, videos contain far more information, requiring a massive amount of data processing, which severely limits research in video processing. Videos are synthesized frame by frame from images, and due to the persistence of vision in the human eye, the frame rate is typically set at 30fps. If video is processed as images, a 1-second video would require processing 30 images, resulting in a huge amount of data. It is worth noting that 3D light field video is composed of micro-image arrays, which has a larger data volume than 2D video. Therefore, there is an urgent need for an efficient encryption method to be deployed in practical security tasks to protect 3D light field video. Summary of the Invention

[0003] This invention proposes a 3D light field video encryption method based on salient target detection. This method can convert large amounts of 3D light field video into small amounts of 2D video, encrypting only the salient target areas of the 2D video, greatly reducing the amount of data to be encrypted and effectively improving the efficiency of 3D light field video encryption.

[0004] The objective of this invention is mainly achieved through the following technical solutions: A 3D light field video encryption method based on salient object detection includes the following steps: S1: The original 2D video frames are processed by a spatiotemporal consistency video salient object extraction network based on an attention mechanism to detect salient objects, and then binarized to obtain the salient region mask of the video frames. S2: Apply the mask obtained in S1 to all pixels of the salient region of the original 2D video frame, and rearrange all these pixels into an original square image; S3: Encrypt the original square image using the iterative rules of cellular automata, specifically including: S3.1: Initialize the cellular automata using the original square image, cell The initial state is set to the original square image. The pixel value of the location; S3.2: Generate a random matrix of the same size as the original square image to control the iteration rules for each cell; S3.3: Iterate according to the iteration rules The image with obfuscated pixel values ​​is obtained in the next step. Greater than or equal to half the maximum value of the row or column of the square image; S3.4: Apply the Arnold transform to the obfuscated image to obtain the scrambled encrypted image.

[0005] S4: Replace the salient regions of the original 2D video frame with the encrypted square image to obtain the encrypted 2D video frame; S5: The encrypted 2D video frame and the mask obtained in S1 are transmitted to the receiver as ciphertext; S6: After receiving the ciphertext, the receiver decrypts it according to the reverse process of S2-S4 to obtain the decrypted 2D video frame; S7: The decrypted 2D video frames are rendered using monocular depth rendering to obtain 3D video frames; S8: 3D video frames are used to reconstruct 3D light field video through an optical display system.

[0006] This attention-based spatiotemporal consistency video salient object extraction network includes intra-frame feature extraction and inter-frame feature extraction modules, employing an attention mechanism to enhance feature extraction effectiveness. Specifically, the intra-frame feature extraction module extracts spatial features from the input video frames and uses spatial attention to guide the network to focus on semantically relevant regions during feature learning. Channel attention is used to explicitly model the dependencies between convolutional feature channels within the network. The inter-frame feature extraction module extracts the correlation features between adjacent video frames and uses temporal attention to guide the weight allocation operation of video frames.

[0007] In step S1, the binarization process uses the Sobel operator to extract image edges.

[0008] The iteration rule in step S3.3 is as follows: (1) in, It is a cell Iteration The subsequent state. It is a Boolean function whose value is determined by the cell. The state of cells in the four neighboring regions is determined.

[0009] Step S7 includes: S7.1: Use a depth estimation network to predict the depth map corresponding to the video frame; S7.2: Generate 3D object point clouds using depth maps and video frames; S7.3: The point cloud of a 3D object is mapped to a micro-image array plane by a microlens array to obtain a micro-image array, i.e., a 3D video frame.

[0010] Furthermore, the depth estimation network in step S7.2 is designed based on a CNN network framework, with a color RGB image as input and a corresponding depth map as output. There is a one-to-one correspondence between the pixels of the color RGB image and the depth map. The value of each pixel in the depth map represents the relative spatial distance to each corresponding pixel in the RGB image.

[0011] The pixel mapping in step S7.3 is based on the lens imaging principle, mapping the pixels of the three-dimensional object image plane to the micro-image array plane through a microlens array to obtain a micro-image array image, as shown in the attached figure. Figure 3 As shown. The center depth of this pixel mapping system. d It can be calculated using the Gaussian imaging formula, as follows: (2) in, f It is the focal length of the microlens array. g It is the distance between the central depth plane and the microlens array plane. Indicates coordinates as The distance between the object and the plane of the microlens array. According to the appendix Figure 3 Geometric relationships in It can be defined by the following formula: (3) in, Represents a depth map. It is a pixel The true depth at that location. Finally, the object pixels. After the light rays pass through the center of the lens, they will be located on the plane of the micro-image array. Location. Pixel coordinates Given by the following formula: (4) in, It is the spacing between the element lenses. It is the pixel size of the image of the object to be displayed.

[0012] The optical display system in step S8 includes: A display is used to emit different colors of light to show images; Microlens arrays are used to deflect light emitted by a display. An optical diffuser is used to ensure that light passing through a microlens array is distributed evenly.

[0013] The microlens array is formed by stacking two cylindrical lens gratings facing each other.

[0014] The beneficial effects of this invention are: (1) The present invention converts 3D light field video into 2D video and processes the 2D video, which significantly reduces the amount of data to be processed.

[0015] (2) The present invention uses a significant target detection algorithm to encrypt only the region of interest in 2D video, which further reduces the amount of data to be processed and meets the high-efficiency encryption requirements of 3D light field video. Attached Figure Description

[0016] Figure 1 This is a process diagram of the efficient encryption method for light field 3D video based on salient target detection according to the present invention.

[0017] Figure 2 This is a diagram of the salient target detection network structure in this invention.

[0018] Figure 3 This is a schematic diagram of pixel mapping from a three-dimensional object plane to a micro-image array plane.

[0019] Figure 4 The diagram shows the result obtained by using the method of the present invention in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0021] like Figures 1-4 As shown, this embodiment relates to a 3D light field video encryption method based on salient target detection, including the following steps: S1: The original 2D video frames are processed by a spatiotemporal consistency video salient object extraction network based on an attention mechanism to detect salient objects, and then binarized to obtain the salient region mask of the video frames. S2: Apply the mask obtained in S1 to all pixels of the salient region of the original 2D video frame, and rearrange all these pixels into an original square image; S3: Encrypt the original square image using the iterative rules of cellular automata; S3.1: Initialize the cellular automata using the original square image, cell The initial state is set to the original square image. The pixel value of the location; S3.2: Generate a random matrix of the same size as the original square image to control the iteration rules for each cell; S3.3: Iterate according to the iteration rules The image with obfuscated pixel values ​​is obtained in the next step. Greater than or equal to half the maximum value of the row or column of the square image; S3.4: Apply the Arnold transform to the obfuscated image to obtain the scrambled encrypted image.

[0022] S4: Replace the salient regions of the original 2D video frame with the encrypted square image to obtain the encrypted 2D video frame; S5: The encrypted 2D video frame and the mask obtained in S1 are transmitted to the receiver as ciphertext; S6: After receiving the ciphertext, the receiver decrypts it according to the reverse process of S2-S4 to obtain the decrypted 2D video frame; S7: The decrypted 2D video frames are rendered using monocular depth rendering to obtain 3D video frames; S7.1: Use a depth estimation network to predict the depth map corresponding to the video frame; S7.2: Generate 3D object point clouds using depth maps and video frames; S7.3: The point cloud of a 3D object is mapped to a micro-image array plane by a microlens array to obtain a micro-image array, i.e., a 3D video frame.

[0023] S8: 3D video frames are used to reconstruct 3D light field video through an optical display system.

[0024] This invention relates to a spatiotemporally consistent video salient object extraction network based on an attention mechanism, as shown in the attached figure. Figure 2 As shown, the system includes an intra-frame feature extraction module and an inter-frame feature extraction module, employing an attention mechanism to improve the effectiveness of feature extraction. Specifically, the intra-frame feature extraction module extracts spatial features from the input video frames and uses a spatial attention mechanism to ensure the network focuses on semantically relevant regions when learning features. A channel attention mechanism is used to explicitly model the dependencies between convolutional feature channels in the network. The inter-frame feature extraction module extracts the correlation features between adjacent video frames and uses a temporal attention mechanism to guide the weight allocation operation of video frames.

[0025] In detail, the iteration rule in step S3.3 is as follows: (5) in, It is a cell Iteration The subsequent state. It is a Boolean function whose value is determined by the cell. The state of cells in the four neighboring regions is determined.

[0026] Specifically, the iteration rule for each cell is selected from three rules (represented using binary numbers): (0110100110010110)2, (0101101001101010)2, and (0101011001011010)2, which can be represented by the following logical operation equations: (6) in, This represents the logical XOR operation. This indicates a logical addition operation. This indicates a logical multiplication operation.

[0027] In detail, in step S7.1, the 2D video frame serves as the input to the depth estimation network. After passing through this network, a corresponding depth map is obtained. The depth map and the 2D video frame are then processed by a pixel mapping algorithm to obtain an image source micro-image array for 3D light field reconstruction. The pixels of the 2D video frame and the depth map correspond one-to-one. The value of each pixel in the depth map represents the relative spatial distance between each corresponding pixel in the 2D video frame. The depth map can discretize a 3D object point cloud without color information. Each pixel value in the 2D video frame determines the color information of each point in the 3D object point cloud. A colored 3D object point cloud can be generated from the depth map and the 2D video frame. Then, a microlens array is used to map each point of the colored 3D object point cloud onto the micro-image array plane to obtain the micro-image array. Figure 3 This diagram illustrates pixel mapping from a 3D object plane to a micro-image array plane. The center depth of this pixel mapping system is shown. d It can be calculated using the Gaussian imaging formula, as follows: (7) in, f It is the focal length of the microlens array. g It is the distance between the central depth plane and the microlens array plane. Indicates coordinates as The distance between the object and the plane of the microlens array. According to the appendix Figure 3 Geometric relationships in It can be defined by the following formula: (8) in, Represents a depth map. It is a pixel The actual depth at that location. Object pixels. After the light rays pass through the center of the lens, they will be located on the plane of the micro-image array. Location. Pixel coordinates Given by the following formula: (9) in, It is the spacing between the element lenses. It is the pixel size of the image of the object to be displayed.

[0028] The micro-image array after pixel mapping is used to reconstruct a stereoscopic 3D light field video through a light field display device.

[0029] The light field display device includes: a display for emitting light of different colors to display an image; a microlens array for deflecting the light emitted by the display; and an optical diffuser for uniformly distributing the light passing through the microlens array. The display has a resolution of 3836×2156. pixels The size of a single pixel is 0.09. mm ×0.09 mm The size of a single lens in the lens array is 2.54. mm ×2.54 mm The number of lenses is 137. w )×77 ( h The diffusion angle of the optical diffusion screen is 10°. 。 .

[0030] To illustrate the effectiveness of the method of the present invention, a resolution of 1920×1080 was used. pixels The test was conducted in a 2D video, with L being 1000 iterations. The test results are attached. Figure 4 As shown, attached Figure 4 The results are presented for six frames from different times in the video, including: the original 2D video frame, saliency map, binarized image, an image showing salient target regions arranged in a square, an encrypted salient target region image, an encrypted 2D video frame, a decrypted 2D video frame, and a 3D video frame after monocular depth rendering. These are the 3D light field reconstruction results captured from three different perspectives, as well as the reconstruction result of the encrypted image. (See attached...) Figure 4 As can be seen, the method of the present invention can convert a large amount of 3D light field video into a small amount of 2D video by using a monocular depth rendering algorithm and a saliency algorithm, and only encrypt the saliency region of the 2D video. The saliency reduces the amount of data to be encrypted, which greatly improves the efficiency of 3D light field video encryption.

Claims

1. A 3D light field video encryption method based on salient object detection, characterized in that, Includes the following steps: S1: The original 2D video frames are processed by a spatiotemporal consistency video salient object extraction network based on an attention mechanism to detect salient objects, and then binarized to obtain the salient region mask of the video frames. S2: Apply the mask obtained in S1 to all pixels of the salient region of the original 2D video frame, and rearrange all these pixels into an original square image; S3: Encrypt the original square image using the iterative rules of cellular automata, specifically including: S3.1: initialize the cellular automaton with the original square image, the cells of which are set to the pixel values of the original square image at the respective position; S3.2: Generate a random matrix of the same size as the original square image to control the iteration rules for each cell; S3.3: iterate according to iteration rule a pixel value confusion image is obtained, wherein, greater than or equal to half of the maximum value of the square image row or column; S3.4: Apply the Arnold transform to the obfuscated image to obtain the scrambled encrypted image; S4: Replace the salient regions of the original 2D video frame with the encrypted square image to obtain the encrypted 2D video frame; S5: The encrypted 2D video frame and the mask obtained in S1 are transmitted to the receiver as ciphertext; S6: After receiving the ciphertext, the receiver decrypts it according to the reverse process of S2-S4 to obtain the decrypted 2D video frame; S7: The decrypted 2D video frames are rendered using monocular depth rendering to obtain 3D video frames; S8: 3D video frames are used to reconstruct 3D light field video through an optical display system.

2. The 3D light field video encryption method based on salient object detection according to claim 1, characterized in that, The spatiotemporal consistency video salient object extraction network based on the attention mechanism in step S1 includes an intra-frame feature extraction module and an inter-frame feature extraction module, and uses an attention mechanism to improve the effectiveness of feature extraction. Specifically, the intra-frame feature extraction module extracts the spatial features of the input video frame and uses a spatial attention mechanism to make the network focus on regions related to semantic information when learning features, and uses a channel attention mechanism to explicitly model the interdependencies between convolutional feature channels in the network. The inter-frame feature extraction module extracts the correlation features between adjacent video frames and uses a temporal attention mechanism to guide the weight allocation operation of video frames.

3. The method of claim 1, wherein, In step S1, the binarization process uses the Sobel operator to extract image edges.

4. The method of claim 1, wherein, The iteration rule in step S3.3 is as follows: in, It is a cell Iteration The subsequent state It is a Boolean function whose value is determined by the cell. The state of cells in the four neighboring regions is determined.

5. The method of claim 1, wherein, Step S7 includes: S7.1: Use a depth estimation network to predict the depth map corresponding to the video frame; S7.2: Generate 3D object point clouds using depth maps and video frames; S7.3: The point cloud of a 3D object is mapped to a micro-image array plane by a microlens array to obtain a micro-image array, i.e., a 3D video frame.

6. The 3D light field video encryption method based on salient object detection according to claim 5, characterized in that, The depth estimation network in step S7.2 is designed based on the CNN network framework. The input is a color RGB image and the output is the corresponding depth map. The pixels of the color RGB image and the depth map are in one-to-one correspondence. The value of each pixel in the depth map represents the relative spatial distance of each corresponding pixel in the RGB image.

7. The method of claim 1, wherein, The optical display system in step S8 includes: A display is used to emit different colors of light to show images; Microlens arrays are used to deflect light emitted by a display. An optical diffuser is used to ensure that light passing through a microlens array is distributed evenly.

8. The 3D light field video high efficient encryption method based on salient object detection of claim 7, wherein, The microlens array is formed by stacking two cylindrical lens gratings facing each other.

Images