3D light field video efficient encryption method based on salient target detection

Abstract

The invention discloses a 3D light field video efficient encryption method based on salient target detection, and the method comprises the following steps: carrying out the salient target detection of a 2D video frame, and carrying out the binarization processing, thereby obtaining a salient region mask; extracting 2D video frame pixels corresponding to the mask area, and rearranging the 2D video frame pixels into a square image; encrypting the square image by using a cellular automaton; replacing the mask area of the 2D video frame with the encrypted square image; transmitting the encrypted 2D video frame and the mask as a ciphertext; the receiver decrypts the ciphertext according to an inverse process after receiving the ciphertext; performing monocular depth rendering on the decrypted 2D video frame to obtain a 3D video frame; and reconstructing a 3D light field video from the 3D video frame through an optical display system. Through monocular depth rendering and saliency detection, the 3D light field video with a large data volume is converted into the 2D video with a small data volume, only the saliency area of the 2D video is encrypted, the data volume to be encrypted is reduced, and the encryption efficiency of the 3D light field video is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D video encryption, and in particular to a 3D light field video efficient encryption method based on salient object detection. Background Art

[0002] As an emerging information carrier, 3D images can present the depth and stereoscopic sense of objects, bringing viewers a more realistic visual experience. Therefore, they have attracted more and more people's research interest. Compared with static 3D images, 3D light field videos can present dynamic information, provide naked eye 3D visual effects, and bring richer situational perception. The research and market application of 3D light field videos will increase. With the extensive research of 3D images, the transmission security of 3D images in public channels is also an urgent problem to be solved. Encryption is an effective means of information security protection. At present, many encryption methods for 3D images have been proposed. However, few people have studied the problem of secure transmission of 3D light field videos in public channels. The reason is that compared with images, videos contain more information and the amount of information that needs to be processed is huge, which seriously limits the research on video processing. Videos are synthesized frame by frame by images. Due to the visual persistence effect of the human eye, the frame rate of videos is usually set at 30fps. If videos are processed as images, for a video of only 1s, 30 images need to be processed, and the amount of data is huge. It is worth noting that 3D light field videos are composed of micro-image arrays and have a larger amount of data than 2D videos. Therefore, for the protection of 3D light field videos, an efficient encryption method is urgently needed to be deployed in actual security tasks. Summary of the invention

[0003] The present invention proposes an efficient encryption method for 3D light field video based on salient target detection. The method can convert a large amount of 3D light field video into a small amount of 2D video, and only encrypt the salient target area of ​​the 2D video, which greatly reduces the amount of data to be encrypted and effectively improves the efficiency of 3D light field video encryption.

[0004] The purpose of the present invention is mainly achieved through the following technical solutions:

[0005] The efficient encryption method of 3D light field video based on salient object detection includes the following steps:

[0006] S1: Detect salient objects on the original 2D video frame and perform binarization processing to obtain the salient region mask of the video frame;

[0007] S2: Apply the mask obtained in S1 to the original 2D video frame to obtain all pixels in the salient area, and rearrange all the pixels into an original square image;

[0008] S3: Encrypt the original square image using the iterative rule of cellular automata;

[0009] S4: replacing the salient area of ​​the original 2D video frame with the encrypted square image to obtain an encrypted 2D video frame;

[0010] S5: The encrypted 2D video frame and the mask obtained in S1 are transmitted to the receiver as ciphertext;

[0011] S6: After receiving the ciphertext, the receiver performs decryption according to the reverse process of S2-S4 to obtain the decrypted 2D video frame;

[0012] S7: Performing monocular depth rendering on the decrypted 2D video frame to obtain a 3D video frame;

[0013] S8: The 3D video frame is reconstructed into a 3D light field video through an optical display system.

[0014] Salient object detection uses a spatiotemporal consistency video salient object extraction network based on the attention mechanism, including an intra-frame feature extraction module and an inter-frame feature extraction module, and uses the attention mechanism to improve the effectiveness of feature extraction. Among them, the intra-frame feature extraction module extracts the spatial features of the input video frame, and uses the spatial attention mechanism to make the network pay more attention to the areas related to semantic information when learning features, and uses the channel attention mechanism to explicitly model the interdependence between the convolutional feature channels in the network. The inter-frame feature extraction module extracts the correlation features between adjacent frames of the video, and uses the temporal attention mechanism to guide the weight allocation operation of the video frame.

[0015] In the binarization process in step S1, the Sobel operator is used to extract the edge of the image.

[0016] The step S3 comprises:

[0017] S3.1: Initialize the cellular automaton using the original square image. The initial state of cell C(i,j) is set to the pixel value at position I(i,j) of the original square image.

[0018] S3.2: Generate a random matrix with the same size as the original square image to control the iteration rule of each cell;

[0019] S3.3: Iterate L times according to the iteration rule to obtain an image after pixel value confusion, where L is greater than or equal to half of the maximum value of the square image row or column;

[0020] S3.4: Apply Arnold transform to the obfuscated image to obtain the scrambled encrypted image.

[0021] Furthermore, the iteration rule in step S3.3 is:

[0022]

[0023] in, is the state of cell C(i,j) after t iterations. i,j (·) is a Boolean function whose value is determined by the cell states of the four neighborhoods of cell C(i,j).

[0024] The step S7 comprises:

[0025] S7.1: Use the depth estimation network to predict the depth map corresponding to the video frame;

[0026] S7.2: Generate 3D object point cloud using depth map and video frames;

[0027] S7.3: For the three-dimensional object point cloud, each point of the point cloud is mapped to the micro-image array plane through the micro-lens array to obtain a micro-image array, that is, a 3D video frame.

[0028] Furthermore, the depth estimation network in step S7.2 is designed based on the CNN network framework, with the input being a color RGB image and the output being a corresponding depth map. The pixels of the color RGB image and the depth map correspond one to one. The value of each pixel in the depth map represents the relative spatial distance of each corresponding pixel in the RGB image.

[0029] The pixel mapping in step S7.3 is based on the lens imaging principle, and the pixels of the three-dimensional object image plane are mapped to the micro-image array plane through the micro-lens array to obtain the micro-image array image, as shown in the attached figure. Figure 3 As shown. The center depth d of the pixel mapping system can be calculated by the Gaussian imaging formula, which is as follows:

[0030]

[0031] Where f is the focal length of the microlens array and g is the distance between the central depth plane and the microlens array plane. (i,j) represents the distance between the object with coordinates (i, j) and the microlens array plane. Figure 3 The geometric relationship in L (i,j) It can be defined by the following formula:

[0032]

[0033] Among them, Z represents the depth map, Z (i,j) is the true depth at pixel (i,j). Finally, the light from object pixel A(i,j) will be located at the micro-image array plane A'(u,v) after passing through the center of the lens. The pixel coordinates (u,v) are given by the following formula:

[0034]

[0035] Among them, P L is the pitch of the element lenses, P I is the pixel size of the object image to be displayed.

[0036] The optical display system in step S8 includes:

[0037] A display, which emits light of different colors to display images;

[0038] a microlens array to bend the light produced by the display;

[0039] The optical diffuser screen is used to make the light passing through the microlens array evenly distributed.

[0040] The microlens array is formed by stacking two cylindrical lens gratings facing each other.

[0041] The beneficial effects of the present invention are:

[0042] (1) The present invention converts 3D light field video into 2D video and processes the 2D video, thereby significantly reducing the amount of data to be processed.

[0043] (2) The present invention adopts a salient object detection algorithm and only encrypts the region of interest in the 2D video, further reducing the amount of data to be processed and meeting the requirements for efficient encryption of 3D light field video. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a process diagram of the light field 3D video efficient encryption method based on salient object detection of the present invention.

[0045] Figure 2 This is a structural diagram of the salient target detection network in the present invention.

[0046] Figure 3 Schematic diagram of pixel mapping from the three-dimensional object plane to the micro-image array plane.

[0047] Figure 4 This is a result diagram obtained by using the method of the present invention in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0049] like Figures 1 to 4 As shown, this embodiment relates to a 3D light field video efficient encryption method based on salient object detection, comprising the following steps:

[0050] S1: Detect salient objects on the original 2D video frame and perform binarization processing to obtain the salient region mask of the video frame;

[0051] S2: Apply the mask obtained in S1 to the original 2D video frame to obtain all pixels in the salient area, and rearrange all the pixels into an original square image;

[0052] S3: Encrypt the original square image using the iterative rule of cellular automata;

[0053] S3.1: Initialize the cellular automaton using the original square image. The initial state of cell C(i,j) is set to the pixel value at position I(i,j) of the original square image.

[0054] S3.2: Generate a random matrix with the same size as the original square image to control the iteration rule of each cell;

[0055] S3.3: Iterate L times according to the iteration rule to obtain an image after pixel value confusion, where L is greater than or equal to half of the maximum value of the square image row or column;

[0056] S3.4: Apply Arnold transform to the obfuscated image to obtain the scrambled encrypted image.

[0057] S4: replacing the salient area of ​​the original 2D video frame with the encrypted square image to obtain an encrypted 2D video frame;

[0058] S5: The encrypted 2D video frame and the mask obtained in S1 are transmitted to the receiver as ciphertext;

[0059] S6: After receiving the ciphertext, the receiver performs decryption according to the reverse process of S2-S4 to obtain the decrypted 2D video frame;

[0060] S7: Performing monocular depth rendering on the decrypted 2D video frame to obtain a 3D video frame;

[0061] S7.1: Use the depth estimation network to predict the depth map corresponding to the video frame;

[0062] S7.2: Generate 3D object point cloud using depth map and video frames;

[0063] S7.3: For the three-dimensional object point cloud, each point of the point cloud is mapped to the micro-image array plane through the micro-lens array to obtain a micro-image array, that is, a 3D video frame.

[0064] S8: The 3D video frame is reconstructed into a 3D light field video through an optical display system.

[0065] The salient object detection in this invention adopts a spatiotemporal consistency video salient object extraction network based on the attention mechanism, as shown in the attached Figure 2 As shown in the figure, it includes an intra-frame feature extraction module and an inter-frame feature extraction module, and uses an attention mechanism to improve the effectiveness of feature extraction. The intra-frame feature extraction module extracts the spatial features of the input video frame, and uses the spatial attention mechanism to make the network pay more attention to the areas related to semantic information when learning features, and uses the channel attention mechanism to explicitly model the interdependence between convolutional feature channels in the network. The inter-frame feature extraction module extracts the correlation features between adjacent frames of the video, and uses the temporal attention mechanism to guide the weight allocation operation of the video frames.

[0066] In detail, the iteration rule in step S3.3 is:

[0067]

[0068] in, is the state of cell C(i,j) after t iterations. i,j (·) is a Boolean function whose value is determined by the cell states of the four neighborhoods of cell C(i,j).

[0069] Specifically, the iteration rule of each cell is selected from three rules (using binary numbers to represent the rules): (0110100110010110)2, (0101101001101010)2 and (0101011001011010)2, which can be expressed as the following logical operation equations:

[0070]

[0071] in, represents a logical exclusive OR operation, + represents a logical addition operation, and · represents a logical multiplication operation.

[0072] In detail, in step S7.1, the 2D video frame is used as the input of the depth estimation network, and the corresponding depth map is obtained after passing through the network. The depth map and the 2D video frame are subjected to a pixel mapping algorithm to obtain an image source micro-image array for 3D light field reconstruction. The pixels of the 2D video frame and the depth map correspond one-to-one, and the value of each pixel in the depth map represents the relative spatial distance of each corresponding pixel in the 2D video frame. The depth map can discretize a three-dimensional object point cloud without color information. Each pixel value in the 2D video frame determines the color information of each point in the three-dimensional object point cloud. A colored three-dimensional object point cloud can be generated from the depth map and the 2D video frame, and then each point of the colored three-dimensional object point cloud is mapped to the micro-image array plane through the microlens array to obtain a micro-image array, attached. Figure 3The schematic diagram of pixel mapping from the three-dimensional object plane to the micro-image array plane is shown. The center depth d of the pixel mapping system can be calculated by the Gaussian imaging formula, which is as follows:

[0073]

[0074] Where f is the focal length of the microlens array and g is the distance between the central depth plane and the microlens array plane. (i,j) represents the distance between the object with coordinates (i, j) and the microlens array plane. Figure 3 The geometric relationship in L (i,j) It can be defined by the following formula:

[0075]

[0076] Among them, Z represents the depth map, Z (i,j) is the true depth at pixel (i,j). After the light from object pixel A(i,j) passes through the center of the lens, it will be located at the micro-image array plane A'(u,v). The pixel coordinates (u,v) are given by the following formula:

[0077]

[0078] Among them, P L is the pitch of the element lenses, P I is the pixel size of the object image to be displayed.

[0079] The micro-image array after pixel mapping is used to reconstruct a 3D light field video with a stereoscopic sense through a light field display device.

[0080] The light field display device includes: a display, which is used to emit light of different colors to display an image; a microlens array, which is used to deflect the light generated by the display; and an optical diffusion screen, which is used to evenly distribute the light passing through the microlens array. The resolution of the display is 3836×2156 pixels, and the size of a single pixel is 0.09mm×0.09mm. The size of a single lens of the lens array is 2.54mm×2.54mm, the number of lenses is 137(w)×77(h), and the diffusion angle of the optical diffusion screen is 10. .

[0081] In order to illustrate the effectiveness of the method of the present invention, a test was conducted on a 2D video with a resolution of 1920×1080 pixels, and the number of iterations L in the test was 1000. The test results are shown in the attached Figure 4 As shown, attached Figure 4The results of six frames of images at different times in the video are given, including: the original 2D video frame, the saliency map, the binary map, the effect map of the salient target area arranged in a square, the encrypted salient target area map, the encrypted 2D video frame, the decrypted 2D video frame, the 3D video frame after monocular depth rendering, the 3D light field reconstruction results taken from three different perspectives, and the reconstruction results of the encrypted map. Figure 4 It can be seen that the method of the present invention can convert a large amount of 3D light field video into a small amount of 2D video through a monocular depth rendering algorithm and a saliency algorithm, and only encrypt the salient area of ​​the 2D video, which significantly reduces the amount of data to be encrypted and greatly improves the efficiency of 3D light field video encryption.

Claims

1. An efficient encryption method for 3D light field video based on salient object detection, characterized in that: The following steps are involved: S1: Detect salient objects on the original 2D video frame and perform binarization processing to obtain the salient region mask of the video frame; S2: Apply the mask obtained in S1 to the original 2D video frame to obtain all pixels in the salient area, and rearrange all the pixels into an original square image; S3: Encrypt the original square image using the iterative rule of cellular automata; S4: replacing the salient area of ​​the original 2D video frame with the encrypted square image to obtain an encrypted 2D video frame; S5: The encrypted 2D video frame and the mask obtained in S1 are transmitted to the receiver as ciphertext; S6: After receiving the ciphertext, the receiver performs decryption according to the reverse process of S2-S4 to obtain the decrypted 2D video frame; S7: Performing monocular depth rendering on the decrypted 2D video frame to obtain a 3D video frame; S8: The 3D video frame is reconstructed into a 3D light field video through an optical display system.

2. The efficient encryption method for 3D light field video based on salient object detection according to claim 1 is characterized in that: The salient target detection in step S1 adopts a spatiotemporal consistency video salient target extraction network based on an attention mechanism, including an intra-frame feature extraction module and an inter-frame feature extraction module, and adopts an attention mechanism to improve the effectiveness of feature extraction; wherein, the intra-frame feature extraction module extracts the spatial features of the input video frame, and uses the spatial attention mechanism to make the network pay more attention to the areas related to semantic information when learning features, and uses the channel attention mechanism to explicitly model the interdependence between the convolutional feature channels in the network; the inter-frame feature extraction module extracts the correlation features between adjacent frames of the video, and uses the temporal attention mechanism to guide the weight allocation operation of the video frame.

3. The efficient encryption method for 3D light field video based on salient object detection according to claim 1 is characterized in that: In the binarization process in step S1, the Sobel operator is used to extract the edge of the image.

4. The efficient encryption method for 3D light field video based on salient object detection according to claim 1, characterized in that: The step S3 comprises: S3.1: Initialize the cellular automaton using the original square image. The initial state of cell C(i,j) is set to the pixel value at position I(i,j) of the original square image. S3.2: Generate a random matrix with the same size as the original square image to control the iteration rule of each cell; S3.3: Iterate L times according to the iteration rule to obtain an image after pixel value confusion, where L is greater than or equal to half of the maximum value of the square image row or column; S3.4: Apply Arnold transform to the obfuscated image to obtain the scrambled encrypted image.

5. The efficient encryption method for 3D light field video based on salient object detection according to claim 4 is characterized in that: The iteration rule in step S3.3 is: Among them, S i t ,j is the state of cell C(i,j) after t iterations, f i,j (·) is a Boolean function whose value is determined by the cell states of the four neighbors of cell C(i,j).

6. The efficient encryption method for 3D light field video based on salient object detection according to claim 1, characterized in that: The step S7 comprises: S7.1: Use the depth estimation network to predict the depth map corresponding to the video frame; S7.2: Generate 3D object point cloud using depth map and video frames; S7.3: For the three-dimensional object point cloud, each point of the point cloud is mapped to the micro-image array plane through the micro-lens array to obtain a micro-image array, that is, a 3D video frame.

7. The efficient encryption method for 3D light field video based on salient object detection according to claim 6 is characterized in that: The depth estimation network in step S7.2 is designed based on the CNN network framework, with the input being a color RGB image and the output being a corresponding depth map. The pixels of the color RGB image and the depth map correspond one to one, and the value of each pixel in the depth map represents the relative spatial distance of each corresponding pixel in the RGB image.

8. The efficient encryption method for 3D light field video based on salient object detection according to claim 1, characterized in that: The optical display system in step S8 includes: A display, which emits light of different colors to display images; a microlens array to bend the light produced by the display; The optical diffuser screen is used to make the light passing through the microlens array evenly distributed.

9. The efficient encryption method for 3D light field video based on salient object detection according to claim 8, characterized in that: The microlens array is formed by stacking two cylindrical lens gratings facing each other.

Images