Trusted integrated imaging method capable of realizing light field tampering detection and information self-recovery

Abstract

The invention relates to a credible integrated imaging method capable of realizing light field tampering detection and information self-recovery, which can realize clear multi-view light field display and consider tampering detection and recovery at the same time. Firstly, depth and angle information in a monocular scene is extracted by combining light field mapping of deep learning; then, a neural network is built, and a region of interest (ROI) and a region of interest (non-ROI) of the 3D light field are further extracted; and then, embedding recovery information constructed based on multiple view angles into the extracted ROI. And after the receiver receives the light field data, tampering detection is firstly carried out on the light field data. And when the recorded light field is tampered, the system can position a tampered area by using the random relationship among the recovery sub-blocks, and realizes recovery. And finally, the recovered micro-image array passes through a self-made 3D light field display device, so that credible 3D light field imaging is realized.

Description

Technical Field

[0001] This invention relates to the fields of integrated imaging and optical security, specifically to a reliable three-dimensional integrated imaging method based on multi-viewpoint tamper detection of light fields. Background Technology

[0002] Integrated imaging can provide full parallax, true color, and continuous viewing angles for target scenes, demonstrating enormous application potential in next-generation 3D light field imaging systems. However, as more and more integrated imaging data is uploaded to the internet, these images still face the risk of tampering, potentially causing significant losses in fields such as telemedicine and military applications.

[0003] Trustworthy 3D integrated imaging technology emphasizes the ability to detect modified regions without compromising image quality. Current research has constructed self-supervised networks to adaptively generate steganographic keys, with watermarks generated and embedded in various viewpoints of the micro-image array to ensure image quality. Other studies have proposed compressing the micro-image array to expand the key space before directly embedding the watermark using discrete Fresnel diffraction. Furthermore, by unifying the different colors acquired through a unified light field, the embedding factor can be precisely controlled, thus balancing the watermark's invisibility and robustness. Further, researchers have proposed applying three-dimensional cellular automata transformations to the temporal processing of individual micro-images to generate watermarked integrated imaging videos.

[0004] However, the aforementioned integrated imaging methods based on watermark embedding focus more on copyright protection than tamper detection, and do not take into account the data reliability of the receiving end and the corresponding imaging quality. Summary of the Invention

[0005] In view of the above problems, this invention proposes a reliable integrated imaging method that can realize light field tampering detection and information self-recovery, and build a more reliable three-dimensional light field imaging system while ensuring imaging quality.

[0006] On the one hand, this application provides a reliable integrated imaging method capable of detecting light field tampering and self-recovering information. The method includes:

[0007] By combining pixel-level light field mapping with deep learning, a monocular scene can be mapped into a sub-aperture image array.

[0008] A neural network is constructed and trained by feeding the sub-aperture image array to extract the region of interest in the three-dimensional light field;

[0009] Generate a multi-view light field recovery watermark and embed it into the region of interest;

[0010] Perform tamper detection and recovery operations on the light field data, and reconstruct the micro-image array through multi-view relationships;

[0011] Reliable 3D light field reconstruction is achieved through self-developed 3D light field display devices.

[0012] Optionally, in one embodiment of this application, the step of mapping a monocular scene into a sub-aperture image array using pixel-level light field mapping combined with deep learning includes:

[0013] A neural network model is used to estimate the depth of a scene captured by a single camera.

[0014] The estimated depth map is then converted to the light field depth within the human eye's visible depth:

[0015] Based on the depth map obtained by the transformation, the pixel coordinates of a single field of view are projected into the pixel coordinates of a sub-aperture image array.

[0016] Optionally, in one embodiment of this application, the neural network model includes a coarse estimation-fine estimation concatenated structure. The coarse estimation module is an encoder-decoder stack structure, where the encoder uses basic residual blocks as its main constituent units, and the decoder uses convolution-batch normalization-activation modules as the main deconvolutional layer structure. The first three convolutional layers in the coarse estimation module introduce Squeeze-and-Excitation (SE attention) modules. The fine estimation module is also an encoder-decoder stack structure.

[0017] Optionally, in one embodiment of this application, the loss function for training the neural network model is:

[0018]

[0019] Among them, l (k) Indicates the kth th The loss corresponding to each branch output, K=8 represents the total number of network outputs, α k Let l be the loss weight coefficient for the k-th branch. (k) Defined as: This represents the binary cross-entropy loss function. The structural similarity index loss function is represented. The loss function for standard similarity evaluation is:

[0020]

[0021] Where S(m, n) ∈ {0, 1} represents the saliency map label, and I(m, n) represents the network output saliency map.

[0022] Optionally, in one embodiment of this application, generating a multi-view light field recovery watermark and embedding it into the region of interest includes:

[0023] Each sub-aperture image is divided into several non-overlapping image blocks, with each block consisting of four pixels.

[0024] Each image block is copied into two copies and two identical light field recovery information are generated independently;

[0025] Generate a corresponding watermark bit sequence based on the light field recovery information;

[0026] The watermark bit sequence is divided into four different subgroups;

[0027] Calculate the overall difference between the host image block pixels and the watermark bits, and embed the watermark bits into the host image block pixels.

[0028] Optionally, in one embodiment of this application, the step of performing tamper detection and recovery operations on the light field data and reconstructing the micro-image array through multi-view relationships includes:

[0029] Extract the watermark bit sequence from the light field data;

[0030] The validity of the extracted watermark bit sequence is determined using a triple rule.

[0031] If valid, the corresponding bit value can be directly used as the recovery information of the tampered pixel;

[0032] If it is invalid, according to the formula The recovery result of the current invalid block is obtained, where μ = k / (n o k+n r ), η = 1 / (n o k+n r ), k=2 represents the confidence ratio of the unaltered block to the recovered block. n o and n r These are defined as the number of unaltered blocks and the number of recoverable blocks in a 3×3 image block array, respectively. m and p n These represent the pixel values ​​in the unaltered block and the restored block, respectively;

[0033] After restoration, the reconstructed sub-aperture image is remapped into a complete micro-image array.

[0034] Optionally, in one embodiment of this application, the self-developed 3D light field display device includes: a 4K liquid crystal display screen with a resolution of 3836×2156, a microlens array, and an optical diffusion screen. The size P of each element lens... L The focal length is 2.54mm, and the focal length f is 2.8mm. Pixel pitch P dThe aperture size is set to 0.09mm, corresponding to a total lens array size of 137×77. The micro-image size is 28×28, and the sub-aperture image size is 137×77. Finally, the generated micro-image array is placed in the display device for imaging, realizing the restoration of the tampered naked-eye 360° light field display.

[0035] The aforementioned reliable integrated imaging method, capable of detecting light field tampering and self-recovering information, firstly, combines pixel mapping technology with deep learning to collaboratively extract scene depth and angle data pixel by pixel; then, it achieves initial verification by extracting the region of interest (ROI) of the 3D light field; subsequently, it generates multi-view light field recovery information and embeds it into the ROI; next, it reconstructs a complete micro-image array; finally, it builds a self-developed 3D light field display to achieve naked-eye 360° light field display after tampering. When the recorded light field is tampered with, this scheme can locate the tampered area and achieve dual repair through random correlations between image blocks, while also exhibiting good parallax effects in imaging. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of a reliable integrated imaging method for realizing light field tampering detection and information self-recovery in one embodiment;

[0037] Figure 2 This is a schematic diagram illustrating the principle of sub-aperture image array generation and 3D light field region of interest extraction in one embodiment;

[0038] Figure 3 This is a schematic diagram of a neural network structure used to extract regions of interest in a 3D light field in one embodiment.

[0039] Figure 4 This is a schematic diagram of 3D light field recovery watermark embedding in one embodiment;

[0040] Figure 5 The image shows a schematic diagram (left) and a physical image (right) of a 3D light field display device in one embodiment.

[0041] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] In one embodiment, such as Figure 1 As shown, a reliable integrated imaging method capable of detecting light field tampering and self-recovering information is provided, comprising the following steps:

[0044] S101: Using pixel-level light field mapping combined with deep learning, a monocular scene is mapped into a sub-aperture image array.

[0045] In this embodiment, firstly, an existing or technologically mature neural network is used to obtain the original depth map Z of the monocular shooting scene. (x，y) Subsequently, Z is calculated according to the following formula. (x，y) Further conversion to the range of human vision:

[0046]

[0047] Where g represents the distance between the display screen and the microlens array, L (x，y) This represents the (x, y)th position in the true depth map Z. th The transformed depth value corresponding to each pixel. Based on the depth map L (x，y) By projecting the pixel coordinates of a single optical field onto the pixel coordinates of a sub-aperture image array, the optical field angle information under multiple viewpoints is obtained. Here, m, n, u, and v represent the angles at the (u, v)th optical field. th In the sub-aperture image, the (m,n)th... th The coordinates of each pixel, P I and P L These represent the pixel size of the target being displayed and the size of the lens array, respectively.

[0048]

[0049] S102: Construct a neural network and train the network by feeding it into the sub-aperture image array to extract the region of interest of the three-dimensional light field.

[0050] In this embodiment, a network model is constructed, and a sub-aperture image is used as the network input to extract the ROI and RONI regions of the 3D light field, such as... Figure 2 As shown.

[0051] Specifically, the network model consists of two modules: a coarse estimation module and a fine estimation module. For example... Figure 3 As shown, the coarse estimation encoder uses basic residual blocks as its main structure; the decoder uses a convolutional-batch normalization-activation (Conv+BN+ReLU) module as its main structure with seven deconvolutional layers. The number of channels in the bridging stage is 64, 128, 256, 512, 512, and 512 respectively. Additionally, a Squeeze-and-Excitation (SE attention) module is introduced after the first three convolutional layers in the coarse estimation module to enhance the network's attention to shallow features. In the fine estimation module, the number of channels is fixed at 64 to maintain structural consistency.

[0052] Specifically, in one embodiment of this application, the construction of the neural network, by feeding the sub-aperture image array into the training network to extract the region of interest of the three-dimensional light field, further includes:

[0053] Design a network loss function to help the network converge quickly during training.

[0054] In this embodiment, the function is composed of the intermediate estimation results from the seven decoder deconvolution layers and the final fine estimation output, specifically defined as follows: Among them, l (k) Indicates the kth th The loss corresponding to each branch output, K=8 represents the total number of network outputs, α k Let l be the loss weight coefficient for the k-th branch. (k) Defined as: This represents the binary cross-entropy loss. This represents the structural similarity index loss. The loss function for standard similarity evaluation is: Where S(m,n)∈{0,1} represents the saliency map label, and I(m,n) represents the network output saliency map.

[0055] S103: Generate a multi-view light field recovery watermark and embed it into the region of interest.

[0056] In the embodiments of this application, such as Figure 4 As shown, each sub-aperture image is divided into several non-overlapping image blocks, each consisting of four pixels, denoted as I = [p0, p1, p2, p3]. Each image block is copied into two copies, denoted as B′ = [p′0, p′1, p′2, p′3] and B″ = [p″0, p″1, p″2, p″3]. For the two image blocks B′ and B″, two completely identical reconstruction information R′ and R″ are generated: (R′ and R″ are generated using completely identical logic, so R′ will be used as an example here). Next, calculate the high five bits of R′: For r in sequence i ′ and r i Perform a bitwise XOR operation to generate the first authentication bit p: The second authentication bit q is obtained from the following formula: q = ~p. Finally, the complete watermark is constructed in bit-level form as: W = [r i ′, i The watermark bit sequence W is divided into four distinct subgroups, and each subgroup is embedded into the four pixels corresponding to image blocks I = [p0, p1, p2, p3] in the host sub-aperture image. Finally, the host pixel p is calculated.i With the corresponding recovery bit [w 3i w 3i+1 w 3i+2 The overall difference d between them i , where 0≤i≤3:d i =2 2 ·w 3i +2 1 ·w 3i+1 +2 0 ·w 3i+2 -mod(p i ,8). According to d i The value of (where 0≤i≤3) will be used to recover the corresponding bit w. 3i w 3i+1 w 3i+2 Embedded into the ROI host pixel:

[0057]

[0058] S104: Perform tamper detection and recovery operations on the light field data, and reconstruct the micro-image array through multi-view relationships;

[0059] In this embodiment, 12 recovery bits W are extracted from y0 to y3. d = [dw0, dw1, ..., dw 11 ],in Subsequently, the authentication bits p are reconstructed sequentially. d and q d The calculation method is as follows: and q d =~p d If p d ≠dw 10 or q d ≠dw 11 If the image block is invalid, then the recovered image block is determined to be invalid. Convert the watermark bits [dw0,dw1,...,dw4] and [dw5,dw6,...,dw9] corresponding to B′ and B″ into decimal form, for example: R′ e =2 7 ·dw0+2 6 ·dw1+...+2 3 ·dw4. If or If the recovered block is still deemed unusable, then the current block is also considered invalid if there are more than 5 unusable recovered blocks within a 3×3 neighborhood. If recovered block R′ or R″ is usable, its corresponding bits can be directly used as the recovery information for the tampered pixel; if both R′ and R″ are unusable, then the number of untampered blocks and the number of usable recovered blocks in the 3×3 image block array are defined as n respectively.o and n r Their corresponding recovery weights are set as μ = k / (n o k+n r ) and η = 1 / (n o k+n r Where k=2 represents the confidence ratio of the unaltered block to the recovered block. Finally, it can be determined according to the formula... Obtain the recovery result of the current invalid block, where I r p represents the final recovery block. m and p n These represent the pixel values ​​in the unaltered block and the restored block, respectively. After restoration, the reconstructed sub-aperture image is processed according to the formula EIA(u+mP). L ,v+nP L ) = Sub u,v (m,n) is remapped to the complete micro-image array, where Sub u，v (m, n) represents the (u, v)th (u, v) th The (m, n)th sub-aperture image th 1 pixel.

[0060] S105: Achieves reliable 3D light field reconstruction through self-developed 3D light field display devices.

[0061] In this embodiment of the application, a piece is assembled as follows: Figure 5 The 3D light field display device shown consists of a 4K liquid crystal display screen with a resolution of 3836×2156, a microlens array, and an optical diffusion screen. The size P of each element lens is... L The focal length is 2.54mm, and the focal length f is 2.8mm. Meanwhile, the pixel pitch P... d The setting is 0.09mm, and the corresponding total number of lens arrays is 137×77.

[0062] Finally, the generated micro-image array is placed in the display device for imaging, ultimately restoring the tampered naked-eye 360° light field display.

[0063] The aforementioned reliable integrated imaging method, capable of detecting light field tampering and self-recovering information, firstly, combines pixel mapping technology with deep learning to collaboratively extract scene depth and angle data pixel by pixel; then, it achieves initial verification by extracting the region of interest (ROI) of the 3D light field; subsequently, it generates multi-view light field recovery information and embeds it into the ROI; next, it reconstructs a complete micro-image array; finally, it builds a self-developed 3D light field display to achieve naked-eye 360° light field display after tampering. When the recorded light field is tampered with, this scheme can locate the tampered area and achieve dual repair through random correlations between image blocks, while also exhibiting good parallax effects in imaging.

[0064] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0065] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a ghost imaging encryption analysis method based on neural differential analysis under selected plaintext attack conditions. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device casing, or an external keyboard, touchpad, or mouse.

[0066] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0067] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0068] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0069] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0070] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A trusted integrated imaging method capable of light field tamper detection and information self-recovery, characterized in that, The method includes: By combining pixel-level light field mapping with deep learning, a monocular scene can be mapped into a sub-aperture image array. A neural network is constructed and trained by feeding the sub-aperture image array to extract the region of interest in the three-dimensional light field; Generate a multi-view light field recovery watermark and embed it into the region of interest; Perform tamper detection and recovery operations on the light field data, and reconstruct the micro-image array through multi-view relationships; Reliable 3D light field reconstruction is achieved through self-developed 3D light field display devices.

2. The trusted integrated imaging method capable of light field tamper detection and information self-recovery according to claim 1, characterized in that, The method of mapping a monocular scene into a sub-aperture image array using pixel-level light field mapping combined with deep learning includes: A neural network model is used to estimate the depth of a scene captured by a single camera. The estimated depth map is then converted to the light field depth within the human eye's visible depth: Based on the depth map obtained by the transformation, the pixel coordinates of a single field of view are projected into the pixel coordinates of a sub-aperture image array.

3. The trusted integrated imaging method capable of light field tamper detection and information self-recovery according to claim 1, characterized in that, The neural network model includes a "coarse estimation-fine estimation" cascaded structure. The coarse estimation module is an encoder-decoder stack structure. The encoder is mainly composed of basic residual blocks, and the decoder is mainly composed of convolution-batch normalization-activation modules as deconvolution layers. The first three convolutional layers in the coarse estimation module introduce Squeeze-and-Excitation (SE attention) modules respectively. The fine estimation module is also an encoder-decoder stack structure.

4. The trusted integrated imaging method capable of light field tamper detection and information self-recovery according to claim 1, characterized in that, The loss function for training the neural network model is: Among them, l (k) Indicates the kth th The loss corresponding to each branch output, K=8 represents the total number of network outputs, a k Let l be the loss weight coefficient for the k-th branch. (k) Defined as: This represents the binary cross-entropy loss function. The structural similarity index loss function is represented. The loss function for standard similarity evaluation is: Where S(m,n)∈{0,1} represents the saliency map label, and I(m,n) represents the network output saliency map.

5. The trusted integrated imaging method capable of light field tamper detection and information self-recovery according to claim 1, characterized in that, The step of generating a multi-view light field recovery watermark and embedding it into the region of interest includes: Each sub-aperture image is divided into several non-overlapping image blocks, with each block consisting of four pixels. Each image block is copied into two copies and two identical light field recovery information are generated independently; Generate a corresponding watermark bit sequence based on the light field recovery information; The watermark bit sequence is divided into four different subgroups; Calculate the overall difference between the host image block pixels and the watermark bits, and embed the watermark bits into the host image block pixels.

6. The trusted integrated imaging method capable of light field tamper detection and information self-recovery according to claim 1, characterized in that, The step of performing tamper detection and recovery operations on the light field data, and reconstructing the micro-image array through multi-view relationships, includes: Extract the watermark bit sequence from the light field data; The validity of the extracted watermark bit sequence is determined using a triple rule. If valid, the corresponding bit value can be directly used as the recovery information of the tampered pixel; If it is invalid, according to the formula The recovery result of the current invalid block is obtained, where μ = k / (n o k+n r ), η = 1 / (n o k+n r ), k=2 represents the confidence ratio of the unaltered block to the recovered block, n o and n r p is defined as the number of unaltered blocks and the number of recoverable blocks in a 3×3 image block array, respectively. m and p n These represent the pixel values ​​in the unaltered block and the restored block, respectively; After restoration, the reconstructed sub-aperture image is remapped into a complete micro-image array.

7. The reliable integrated imaging method for realizing light field tampering detection and information self-recovery according to claim 1, characterized in that, The self-developed 3D light field display device comprises a 4K liquid crystal display screen with a resolution of 3836x2156, a microlens array and an optical diffusion screen, wherein the size P of each microlens is 2.54mm, the focal length f is 2.8mm, and the pixel pitch P is 0.09mm L The size P of each microlens is 2.54mm, the focal length f is 2.8mm, and the pixel pitch P is 0.09mm d The size P of each microlens is 2.54mm, the focal length f is 2.8mm, and the pixel pitch P is 0.09mm

Images