Image encryption and decoding method based on deep learning

Abstract

The invention provides an image encryption and decoding method based on deep learning. The encryption method comprises the following steps: inputting hidden information and a carrier image; converting the hidden information to generate a corresponding feature map; converting the carrier image to generate a corresponding reference image; inputting the carrier image, the feature image and the reference image to an encryptor to generate a corresponding residual image; and performing pixel scale operation on the residual image and the carrier image by the encryptor to generate an encrypted image with hidden information. In this way, the visual effect after the hidden information is embedded is improved, and similar decoding accuracy is maintained.

Description

Technical Field

[0001] The present invention relates to encryption and decoding methods, and in particular to an image encryption and decoding method based on deep learning. Background Art

[0002] With the popularity of smart devices, they have not only become a part of people's lives, but also a major source of information. Many manufacturers often use two-dimensional barcodes as a channel to convey information and interact with customers. Among them, QR Code (Quick Response Code) is the most mainstream two-dimensional barcode currently used in daily life. It can be seen almost everywhere in life. Whether it is merchant web page traffic diversion, digital payment and other data transmission, relevant information is obtained or transmitted through QR Code links.

[0003] However, the shortcoming of QR Code is that it is limited by its own design. Generally speaking, QR Code must be composed of black and white modules and finding patterns, so QR Code will appear quite abrupt in the application scenario. For this reason, in recent years, many studies have hoped to use watermarking technology to hide information directly in the image, using the trademark advertising image itself as a substitute for the two-dimensional barcode. In addition, in view of the rise of convolutional neural networks (CNN) in various fields of computer vision, they have also begun to focus on watermark generation by deep learning.

[0004] In the field of anti-counterfeiting technology, most manufacturers use invisible ink to hide special two-dimensional barcodes on images to achieve the goal of anti-counterfeiting identification. However, the use of invisible ink printing increases the complexity of the process. Not only that, in order to identify the content printed with invisible ink, special infrared cameras and other equipment must be used on the machine, which also increases the cost of the machine.

[0005] To solve the above problems, the present invention provides an image encryption and decoding method based on deep learning, which takes human visual perception into consideration to reduce the visibility of hidden information. By converting the carrier image as a reference image, the hidden information in the generated encrypted image is close to the high-frequency area of ​​the original image, avoiding the low-frequency area where the human eye can more easily perceive changes, thereby achieving visual invisibility and realizing the purpose of hiding. Summary of the invention

[0006] The main purpose of the present invention is to provide an image encryption method based on deep learning, which takes into account human visual perception to reduce the visibility of hidden information, and uses the converted carrier image as a reference image to achieve visual invisibility and realize the purpose of hiding.

[0007] Another object of the present invention is to provide an image decoding method based on deep learning, which improves the recognition rate of hidden information by training decoding to resist attacks of image distortion.

[0008] In order to achieve the above-mentioned purpose, an embodiment of the present invention discloses an image encryption method based on deep learning, characterized in that the steps include: inputting a hidden information and a carrier image; converting the hidden information to generate a corresponding feature map; converting the carrier image to generate a corresponding reference image; inputting the carrier image, the feature map and the reference image to an encryptor to generate a corresponding residual map; and performing pixel-scale operations on the residual map and the carrier image with the encryptor to generate an encrypted image with hidden information.

[0009] Preferably, in the step of converting the hidden information to generate a corresponding feature map, the hidden information is converted into the feature map having three channels through a preprocessing network, and a feature vector length and a resolution of the feature map are adjusted according to a resolution of the carrier image.

[0010] Preferably, the hidden information is selected from binary information, and an information length of the hidden information is 1 to 200 bits.

[0011] Preferably, in the step of converting the carrier image by the processing unit to generate a reference image, the processing unit converts the carrier image by a noise visibility function to generate the reference image.

[0012] Preferably, in the step of inputting the carrier image, the feature map and the reference image into an encryptor to generate a corresponding residual map, a loss function is used to operate on the residual map and the reference image as network supervision.

[0013] Preferably, in the step of inputting the carrier image, the feature map and the reference image into an encryptor to generate a corresponding residual map, the encryptor performs calculations on the carrier image, the feature map and the reference image according to an initial weight to generate the residual map.

[0014] In order to achieve the other purpose mentioned above, an embodiment of the present invention discloses a decoding method based on deep learning, characterized in that the steps include: obtaining an encrypted image with hidden information; correcting the encrypted image with a decoder; calculating the corrected encrypted image to extract a corresponding feature map; and calculating the feature map to obtain the corresponding hidden information.

[0015] Preferably, the decoder generates a corresponding damaged image by attacking the encrypted image, and performs decoding training using the damaged image.

[0016] Preferably, the method of attacking the encrypted image is selected from perspective bending, motion blur and out-of-focus blur, color adjustment, adding noise, compression or a combination of any two or more of the above.

[0017] Preferably, in the step of calculating the corrected encrypted image to extract a corresponding feature map, the decoder calculates the corrected encrypted image according to an initial weight to extract the corresponding feature map.

[0018] The beneficial effect of the present invention is that the generated encrypted image is close to the high-frequency area of ​​the original image, avoiding the low-frequency area where the human eye can easily perceive changes, achieving visual invisibility and realizing the purpose of hiding. At the same time, by training to resist various distorted images, the recognition rate of hidden information is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1A A flow chart of an image encryption method according to an embodiment of the present invention;

[0020] Figure 1B A schematic diagram of an implementation of an image encryption method according to an embodiment of the present invention;

[0021] Figure 1C It is a schematic diagram of an implementation process of another embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a reference image before and after calculation according to an embodiment of the present invention;

[0023] Figure 3A A flowchart of an image decoding method according to an embodiment of the present invention;

[0024] Figure 3B A schematic diagram of an implementation of an image decoding method according to an embodiment of the present invention;

[0025] Figure 4 The carrier image and the encrypted images and residual images of different architectures of the embodiment 1 of the present invention;

[0026] Figure 5 The carrier image and the encrypted images and residual images of different architectures according to the second embodiment of the present invention; and

[0027] Figure 6 A diagram comparing the encoding results of an embodiment of the present invention and other model architectures. DETAILED DESCRIPTION

[0028] In order to make the above and / or other purposes, effects, and features of the present invention more clearly understood, preferred embodiments are specifically described below in detail:

[0029] See also Figure 1A to Figure 1B , which is a flow chart of an image encryption method according to an embodiment of the present invention and a schematic diagram of its implementation. As shown in the figure, the image encryption method based on deep learning according to an embodiment of the present invention has the following steps:

[0030] Step S1: input a hidden message and a carrier image;

[0031] Step S2: converting the hidden information to generate a corresponding feature map;

[0032] Step S3: converting the carrier image to generate a corresponding reference image;

[0033] Step S4: inputting the carrier image, the feature map and the reference image into an encryptor to generate a corresponding residual map; and

[0034] Step S5: Using the encryptor to perform pixel-scale operations on the residual image and the carrier image to generate an encrypted image with hidden information.

[0035] As shown in step S1, information and a carrier image are hidden in the input unit. In one embodiment, the hidden information may be a website link, anti-counterfeiting information or other information to be hidden, and the carrier image may be a trademark or any other image. The hidden information may be presented in the form of binary information, but is not limited thereto.

[0036] As shown in step S2, converting the hidden information is to convert the hidden information into a feature vector through a preprocessing network and generate a corresponding feature map. In one embodiment, the preprocessing network is composed of a fully connected layer, and then the feature vector is rearranged into a three-channel feature map and upsampled to the same size as the carrier image, but not limited to this.

[0037] As shown in step S3, the converted carrier image is operated through a noise visibility function (NVF) to generate a corresponding reference image.

[0038] Among them, the noise visibility function (NVF) is a mathematical model that describes the complexity of the texture in the image area, and its formula is as follows:

[0039]

[0040] Among them, i,j are the pixel coordinates of the encrypted image, is the regional variance in the n*n region centered at coordinate (i, j), and θ is a parameter that can adjust the contrast of the output image. Its expression is as follows:

[0041]

[0042] in, is the maximum value of the variance of all regions in the encrypted image, D∈[50,100]. Preferably, this D value can be set to 75, but is not limited thereto.

[0043] From the above Noise Visibility Function formula, we can know that it distinguishes the areas with complex textures and flat areas of the image by calculating the regional variance of the image. The more complex the texture, the larger the regional variance, and the closer the response value is to 0, which means that the human eye is less likely to notice the changes in the area. On the contrary, the closer the response value is to 1, the flatter the texture is, and only a slight change can be noticed by the human eye. The result is as follows Figure 2 As shown, Figure 2 Part (a) is the input carrier image. Figure 2 Part (b) is the reference image calculated by the noise visibility function, where the black part is the area with a smaller noise visibility function response value, that is, the area with smoother texture. The reference image can mark the areas with different texture complexity in the input image according to the size of the response value, thereby reducing the degree of image damage to the carrier image caused by the embedded hidden information, and at the same time maintaining a certain decoding stability.

[0044] Preferably, in order to concentrate the responsive part of the residual image on the image edge and the area with complex texture, the black area of ​​the reference image generated by the noise visibility function is replaced with the white part to describe the image edge and the area with complex texture during calculation, and this is used as the reference image for actual use, such as Figure 2 Part (c) is the reference image after the black and white areas are reversed.

[0045] As shown in step S4, the carrier image, feature map and reference image are input to the encryptor to generate a corresponding residual map. In one embodiment, the encryptor can be composed of a U-Net architecture, which merges the carrier image, feature map and reference image and inputs them into the encryptor, so that the encryptor outputs a feature map with three channels, wherein the feature vector length and resolution of the feature map need to be adjusted according to the resolution size of the target carrier image data set. Preferably, the information length of the hidden information can be set to 1 to 200 bits, and the feature vector size is 1x7200, but not limited thereto.

[0046] In one embodiment, the encryptor may perform computational processing on the carrier image, the feature map, and the reference image according to the initial weights to generate a residual map, but the present invention is not limited thereto.

[0047] In one embodiment, in order to concentrate the changes in the residual image generated by the encryptor on the image edges and areas with complex textures, a loss function calculation model is added to perform calculations on the residual image and the reference image as network supervision, but this is not limited to this.

[0048] In one embodiment, the encryptor can perform calculations based on the carrier image to generate an edge map and calculate the loss function between the residual map and the edge map, so that the hidden information in the encrypted image generated by the encryptor is close to the distribution of the edge map, thereby achieving the goal of improving image quality and achieving a balance between visual effects and decoding stability. In addition, the area with values ​​in the residual map can be distributed in the edge, and the edge map can be taken as the calculation object with the opposite black and white results.

[0049] Preferably, the loss function can adopt the loss function between the residual image and the reference image, or adopt the smoothing loss function (Smoothness Loss), so that the encryptor can learn to generate a residual image that is closer to the reference image.

[0050] The purpose of the smoothing loss function is to improve the image quality of the generated encrypted image and make the texture changes of the hidden information on the encrypted image smoother, so as to avoid the large gradient changes in the residual image and the shadow-like texture in the encrypted image. The smoothing loss function (Smoothness Loss) has the following formula:

[0051]

[0052] Where N is the total number of pixels of the encrypted image, is the gradient of the residual graph, is the gradient of the carrier image, i and j are the coordinates of the pixels. The goal of this loss function is to minimize the gradient of the residual map to achieve the purpose of smoothing, and use the gradient of the carrier image as a weight to adjust the smoothing intensity according to the original image content. That is, the residual map area corresponding to the more complex area of ​​the carrier image content is allowed to produce a larger gradient, and the residual map area corresponding to the less textured area in the carrier image content is smoothed as much as possible. The residual map generated after adding the smoothing loss function does indeed significantly reduce the change amplitude of the watermark pattern, making the change of the overall residual map value smoother, and at the same time can significantly improve the image quality data.

[0053] In one embodiment, five different loss functions may be used to train the encryptor at the same time. For example, the first loss function (L c ) to minimize the difference between each pixel of the final encrypted image and the carrier image; the second can be LPIPS perceptual loss (L p ), which is based on deep learning image similarity scoring, taking human visual perception into consideration, so that it can generate hidden information based on areas that are less noticeable to the human eye; the third one can be the probability of using the discriminator output (L D ), which is one of the perceptual losses, prompting the encryptor to produce an inconspicuous hidden information pattern; the fourth type can be the cross entropy loss (L M ); and the fifth may be to use the loss function between the residual image and the reference image (L R ), and the weighted sum of the above five loss functions is as follows:

[0054] L T =λ1L C +λ2L P +λ3L D +λ4L M +λ5L R

[0055] Among them, L T is the weighted sum of the five loss functions; λ1~λ5 are the weights of the above loss functions.

[0056] As shown in step S5, after the residual image and the carrier image are finally added pixel-wise, an encrypted image embedded with hidden information can be obtained, thereby completing the image encryption process.

[0057] See also Figure 1C , which is a schematic diagram of the implementation process of another embodiment of the present invention. As shown in the figure, the carrier image can be separated into layers to generate multiple layer images, and one or more of the layer images are used to perform the subsequent image encryption process (i.e., the aforementioned steps S2 to S5), and after the encrypted image is generated, the encrypted image is composited with the remaining unencrypted layer images, so that the encrypted image range is smaller, so that the overall encrypted image has a higher quality visual effect.

[0058] See also FIG. 3A to FIG. 3B , which is a flowchart of a decoding method according to an embodiment of the present invention and a schematic diagram of its implementation. As shown in the figure, the decoding method based on deep learning according to an embodiment of the present invention has the following steps:

[0059] Step S6: Obtaining an encrypted image with hidden information;

[0060] Step S7: Correcting the encrypted image with a decoder;

[0061] Step S8: Calculating the corrected encrypted image to extract a corresponding feature map; and

[0062] Step S9: Calculate the feature map to obtain corresponding hidden information.

[0063] As shown in step S6, an encrypted image with a hidden code is input into the input unit. In one embodiment, the encrypted image includes hidden information and a carrier image. Therefore, the hidden information can also be a website link, anti-counterfeiting information or other information to be hidden, and the carrier image can be a trademark or any other image, but is not limited thereto.

[0064] As shown in step S7, the encrypted image is corrected by a decoder. In one embodiment, the decoder includes a spatial transformation module, a convolutional layer, and a fully connected layer to correct slight image skew of the encrypted image and increase the decoding stability of the network when the image is affected by perspective transformation, but not limited to this.

[0065] In one embodiment, the decoder generates a corresponding damaged image by attacking the encrypted image, and uses the damaged image for decoding training, so as to train the overall decoding to resist image distortion caused by physical transmission, wherein the method of attacking the encrypted image is selected from perspective warp, motion blur and out-of-focus blur, color adjustment, adding noise, compression, or a combination of any two or more of the above, but not limited to these.

[0066] The methods of attacking encrypted images are described in detail as follows:

[0067] The purpose of perspective warp is that users may not always shoot from the front when shooting the printed encrypted image. Therefore, in actual applications, the images used for decoding are mostly affected by perspective deformation. Therefore, it is necessary to simulate this situation through perspective warp to improve the decoding accuracy. The method can be to project the image onto planes at different angles through the homography matrix under the assumption of the pinhole camera model to simulate images shot at different angles. To generate the homography matrix, the coordinates of the four corners of the encrypted image are randomly moved within a certain range. This range is set to ±10% of the width of the encrypted image, which is ±32 pixels for the data set used. Then, the corresponding homography matrix is ​​calculated using two sets of point sets, so that the encrypted image can be projected onto planes at different angles to simulate the situation of skewed shooting angles.

[0068] The purpose of motion blur and out-of-focus blur is that users usually hold the camera when shooting, so there is often shaking, which causes motion blur effect, and the camera itself may also produce incorrect autofocus results and cause blur. Among them, motion blur can be achieved by using a linear motion blur convolution kernel, and its formula is as follows:

[0069]

[0070] Among them, r is the distance between a straight line passing through the center point of the convolution kernel, the angle between this straight line and the horizontal axis is a random value, the standard deviation size is set to a value uniformly sampled in the range [0.25, 1.0], σ is the standard deviation parameter, and the standard deviation size is set to a value uniformly sampled in the range [1, 3].

[0071] The out-of-focus blur can be achieved by using a Gaussian blur convolution kernel. Its formula is the same as that of motion blur. The difference is that r is the radius distance from the center of the convolution kernel. The size of the two convolution kernels used for motion blur and out-of-focus blur are both set to 7x7, but this is not limited to this.

[0072] The purpose of color adjustment is that different types of copiers and monitors have different color gamut spaces, and the color ranges they can express are not the same. In addition, cameras may use different exposure settings, white balance or color correction functions when shooting. These factors may cause color shift between the image actually input to the decoder and the original encrypted image. The color adjustment method can be, for example, hue shift, desaturation, and contrast and brightness adjustment. The pixel value range of this part of the encrypted image is between [0,1], and after all the following color adjustment steps, the pixel value of each channel is intercepted to the range of [0,1] to avoid pixel values ​​beyond the original value range after color adjustment. The above three color adjustment methods are described in detail as follows:

[0073] Hue Shift is to add random color shifts to the three RGB channels of the image. The shift value is uniformly sampled from the range [-0.1, 0.1], but not limited to this.

[0074] Desaturation is to linearly interpolate the RGB color image and its corresponding grayscale image at a random ratio, which is uniformly sampled from the range [0,1), but not limited to this.

[0075] Contrast and brightness adjustment can be done by using a common linear transformation method. The formula is as follows:

[0076] g(i,j)=αf(i,j)+β

[0077] Wherein, g(i,j) is the adjusted encrypted image; f(i,j) is the original encrypted image, α is the contrast adjustment parameter, and its value is uniformly sampled from the range [0.5,1.5]; β is the brightness adjustment parameter, and its value is uniformly sampled from the range [-0.3,0.3], but not limited thereto.

[0078] The purpose of adding noise is that various noises may be generated during the camera shooting process. Therefore, Gaussian noise is added during the image distortion simulation process to train the model to resist the attack of noise. The standard deviation parameter size of this part of Gaussian noise is uniformly sampled from the range of [0,0.2], but not limited to this.

[0079] The purpose of compression is that images are mostly stored using lossy compression, such as JPEG format, so compression can be added to the JPEG format compression process to simulate the distortion. The JPEG compression process is non-differentiable, so the JPEG compression approximation method is used to simulate JPEG compression, where the JPEG compression quality parameters are uniformly sampled from the range of [50,100], but not limited to this.

[0080] As shown in step S8, the encrypted image is corrected by the convolutional layer operation of the decoder to extract the corresponding feature map. In one embodiment, the decoder can be based on the initial weight operation to correct the encrypted image to extract the corresponding feature map.

[0081] As shown in step S9, the feature map is finally restored to the corresponding hidden information through the fully connected layer of the decoder.

[0082] In order to more clearly evaluate the quality of the model trained by the image encryption and decoding method provided by the present invention, for example, the following two evaluation methods can be used as quality evaluation.

[0083] The first is to evaluate the visual invisibility of the encrypted image produced by image encryption, that is, in order to evaluate whether the hidden information in the encrypted image generated by the image encryption method of an embodiment of the present invention is imperceptible to the human eye, three image quality evaluations including SSIM, PSNR and LPIPS are used. Through these evaluations, it can be confirmed whether the encrypted image generated by this model is highly similar to the input carrier image, and the degree of damage to the carrier image by the hidden information in the encrypted image can be evaluated. In one embodiment, 734 images are used for image quality evaluation calculation.

[0084] Among them, SSIM is an indicator used to measure the structural similarity of two images. It takes into account the sensitivity of the human eye to structural information. In other words, it can be imagined as a method to evaluate whether two images "look" similar. The calculation method is to divide it into three parts and then multiply them to get the value, namely brightness (Luminance), contrast (Contrast), and structure (Structure). The highest score is 1, that is, the closer the value is to 1, the closer the overall structure is.

[0085] PSNR is mainly used to measure the noise level of an image. It can tell the difference between corresponding pixels between images. Its calculation method needs to be based on the specific image content and application scenario. The following data can be used as a rough reference guide.

[0086] 20-25dB: The difference is usually visible to the naked eye and indicates poor image quality.

[0087] 25-30dB: The image quality is average. Although some distortion of details may still be visible, the overall effect is acceptable.

[0088] 30-35dB: The image quality is relatively good, and most of the detail distortion may be difficult to detect with the naked eye.

[0089] 35-40dB: The image quality is very good and most people may have difficulty telling the difference from the original image.

[0090] 40dB and above: Indicates that the image quality is extremely good and it is almost impossible to tell the difference from the original image.

[0091] The second is decoding stability, that is, in order to evaluate whether the image decoding method of an embodiment of the present invention can generate a carrier image that can resist image distortion so as to provide users with decoding after actual printing, a corresponding encrypted image is generated through an encryptor, and a random image distortion process simulates the image after the actual printing and shooting process, and the distorted encrypted image is input into the decoder for decoding to evaluate the decoding stability of the decoder. A fixed number of 10 images randomly selected from the data set are used as a decoding test data set. Each image is tested 20 times through such a process, and the average bit accuracy of the decoding is calculated to evaluate the decoding stability of this setting.

[0092] To illustrate the influence of the initial weight on the encrypted image result in more detail, in one embodiment, the image encryption method and the image decoding method of the present invention are trained with the ImageNet 2012 verification data set, wherein two sizes of data sets are used respectively, one of which is 1000 randomly selected images from the ImageNet 2012 verification data set, and the other is the entire ImageNet 2012 verification data set, a total of 50,000 images. The encrypted image results generated by the two different sizes of data set training mentioned above differ only in that the number of images in the data set during training is increased, but the encrypted image results finally generated produce obvious grain patterns, resulting in a decrease in image quality. Therefore, in another embodiment, the weights of 10,000 iterations in the model trained by randomly selecting 1000 images from the ImageNet 2012 verification data set are used as the initial weights, and then fine-tuning training is performed with the entire ImageNet 2012 verification data set. The image quality evaluation data obtained is shown in Table 1, and it can be clearly seen that no obvious watermark grains are generated.

[0093] Table 1. Image quality evaluation data

[0094]

[0095] Example 1

[0096] It converts the carrier image with the noise visibility function to generate a reference image, and uses the loss function to operate on the reference image and the residual image as network supervision. The value range of the noise visibility function is between [0,1], which can be used to describe areas with different degrees of human eye perception. The continuous value range distribution is also closer to the style of the residual image generated by the encryptor. Therefore, the encryptor can concentrate the patterns with larger changes in the residual image in the area with larger noise visibility function response values, thereby reducing the possibility of being detected by the human eye. At the same time, some information can also be hidden in the area with smaller noise visibility function response values, providing the encryptor with sufficient range to hide information, thereby making the distribution patterns of the two close.

[0097] See also Figure 4 , which is the carrier image of Example 1 of the present invention and the encrypted images and residual images of different architectures. As shown in the figure, part (a) is the original carrier image; part (b) is the encrypted image using the StegaStamp architecture; part (c) is the residual image using the StegaStamp architecture; part (d) is the encrypted image of Example 1; and part (e) is the residual image of Example 1. Figure 4 It can be clearly seen that the present embodiment has a significant improvement in the visual quality of the encrypted image and the residual image, and does not cause visual interference of the stripe pattern. From the image quality evaluation data in Table 2, it can be seen that the present embodiment achieves better results in various image quality data compared with the watermark algorithm architecture of StegaStamp proposed by Tancik et al., and through the encryption method of the present embodiment, the encrypted image finally obtained is visually very close to the input carrier image, and at the same time, the bit accuracy is only slightly reduced, indicating that the present embodiment can indeed achieve a balance between visual effect and decoding stability.

[0098] Table 2. Image quality evaluation data

[0099] SSIM↑ PSNR↑ LPIPS↓ Bit accuracy StegaStamp 0.937 28.59 0.030 0.988 Example 1 0.953 30.20 0.023 0.986

[0100] Example 2

[0101] The difference between Example 2 and Example 1 is that, based on Example 1, a smoothing loss function is added so that the encryptor can learn to generate a residual image that is closer to the reference image. The encrypted image result generated by this example can be found in Figure 5, which is the carrier image and the encrypted images and residual images of different architectures of Example 2 of the present invention. As shown in the figure, part (a) is the original carrier image; part (b) is the encrypted image using the StegaStamp architecture; part (c) is the residual image using the StegaStamp architecture; part (d) is the encrypted image of Example 2; and part (e) is the residual image of Example 2, as shown in the figure, wherein, Figure 5 It can be found that the residual map generated after adding the smoothing loss function significantly reduces the amplitude of the pattern change, making the overall residual map smoother in value change. And from the image quality evaluation data in Table 3, it can be seen that the image quality data can be significantly improved after adding the smoothing loss function (Smoothness Loss).

[0102] Table 3. Image quality assessment data

[0103] SSIM↑ PSNR↑ LPIPS↓ Bit accuracy StegaStamp 0.937 28.59 0.030 0.988 Example 2 0.967 32.04 0.020 0.982

[0104] In one embodiment, the stability-related data of the image decoding method are shown in Tables 4 to 7. Similarly, compared with the watermark algorithm architecture of StegaStamp proposed by Tancik et al., the digital image distortion simulation decoding test data in Table 4 shows that while the image quality is improved, the decoding success rate of this embodiment only decreases slightly, successfully achieving a balance between visual invisibility and decoding stability; and the physical printing test data in Table 5 is counted by the camera shooting and decoding from the front. Although it is slightly lower than the image distortion simulation decoding test data, it can still maintain a certain decoding success rate, which is sufficient to provide a stable decoding function when applied in real scenarios.

[0105] Table 4. Digital image distortion simulation decoding test data

[0106] Average number of successfully decoded images (1000 images in total) Bit accuracy StegaStamp 981 0.989 This embodiment 966 0.986

[0107] Table 5. Physical printing test data

[0108] Average number of successfully decoded images (1000 images in total) Bit accuracy StegaStamp 987 0.985 This embodiment 926 0.973

[0109] In addition, we also tested the decoding success rate when shooting from a side angle, and tested two shooting positions at a horizontal positive and negative 30 degrees. The decoding test data of shooting at different angles are shown in Table 6. From this table, it can be found that even if shooting from a side angle, the decoding effect can be maintained similar to that when shooting from the front, indicating that the decoding method of this embodiment can effectively resist the perspective distortion caused by the camera viewing angle. From the above decoding stability related data, it can be seen that the image decoding method of the present invention can indeed maintain a fairly stable decoding effect while greatly improving the image quality.

[0110] Table 6. Decoding test data shot at different angles

[0111]

[0112]

[0113] As shown in Table 7, the decoder execution speed takes the lower-end GTX-970 display adapter and the professional graphics Quadro RTX 4000 display adapter as examples. Even the lower-end GTX-970 display adapter can achieve a fairly high frame rate per second (FPS).

[0114] Table 7. Decoder execution speed data table

[0115] GTX-970 Quadro RTX 4000 FPS 53 137

[0116] In one embodiment, the image quality evaluation data of the hidden information length is shown in Table 8. The hidden information length of the image encryption method of the present invention is 50 bits, 100 bits and 200 bits as an example. As the hidden information length increases, the corresponding generated encrypted image will be easier to see the image texture generated by the hidden information. In other words, the longer the hidden information is, the more likely it is to cause visual impact. It can be seen from the image quality evaluation data in Table 8 that when the hidden information is 200 bits in length using the image encryption method of the present invention, its image quality is slightly better than the image quality obtained when the hidden information is 100 bits in length using the StegaStamp architecture. Among them, from the decoding test success rate results of different hidden information lengths, it can be clearly seen that the bit accuracy rate will decrease with the increase of the message length.

[0117] Table 8. Image quality assessment data

[0118] SSIM↑ PSNR↑ LPIPS↓ Bit accuracy StegaStamp(100bits) 0.937 28.59 0.030 0.988 50 bits 0.972 32.40 0.021 0.990 100 bits 0.953 30.20 0.023 0.986 200 bits 0.941 28.84 0.032 0.872

[0119] In order to make the beneficial effects of the deep learning-based image encryption and decoding method of the present invention more clear, the following will compare relevant data with the HiDDeN model architecture, ARW-GAN model architecture, StegaStamp model architecture, and RIW-GAN model architecture.

[0120] The HiDDeN model architecture comes from Hiding Data With Deep Networks, Zhu et al. in CVPR 2018. The difference between this model architecture and the present invention is that the hidden information is not converted, and the decoding method architecture is also different. In addition, due to the lack of color adjustment, it cannot resist printing damage.

[0121] The ARW-GAN model architecture comes from Attention-Guided Robust Image Watermarking ModelBased on GAN, Huang et al.in CVPR 2023. The difference between this model architecture and the present invention is that the hidden information is also not converted. Although the decoding method architecture is adjusted according to the HiDDeN model, it also lacks color adjustment, so it is still unable to resist printing damage.

[0122] The StegaStamp model architecture comes from Invisible Hyperlinks in Physical Photographs, Tancik et al. in CVPR 2020. The difference between this model architecture and the present invention is that it does not use the establishment of a reference image, so its encoding effect is worse than that of the present invention, and its ability to resist damage by printing and photography is also worse than that of the present invention.

[0123] The RIW-GAN model architecture comes from Robust Image Watermarking Based on Generative Adversarial Networks for Copyright Protection, Gao et al. in IEEE 2020. The difference between this model architecture and the present invention is that it also does not use the establishment of a reference image, and also lacks color adjustment in the decoding method, so it is still unable to resist printing damage.

[0124] See also Figure 6 , which is a comparison diagram of the encoding results of an embodiment of the present invention and other model architectures, and can be referred to together with Table 9. It can be clearly seen that when the above four model architectures are compared with the image quality assessment data of the present invention, while maintaining the bit accuracy, both PSNR and SSIM have good performance. At the same time, when the hidden information length becomes longer, the difference is not large, which shows that the present invention has a very good encoding and decoding effect.

[0125]

[0126]

[0127] Note: L is the length of hidden information

[0128] In summary, the present invention provides an image encryption and decoding method based on deep learning to strike a balance between visual quality and decoding stability of hidden content. It must be possible to decode encrypted images containing hidden information using image processing technology while the human eye cannot perceive that the encrypted image contains hidden information. This method can be applied to fields such as authenticity identification, secret message transmission, or copyright protection to achieve the purpose of the present invention.

[0129] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art may make several improvements and modifications without departing from the concept of the present invention, and these improvements and modifications should also be regarded as within the scope of the present invention.

Claims

1. A deep learning-based image encryption method, characterized in that: The steps include: Input a hidden message and a carrier image; Converting the hidden information to generate a corresponding feature map; Converting the carrier image to generate a corresponding reference image; Inputting the carrier image, the feature map and the reference image into an encryptor to generate a corresponding residual map; and The encryptor performs pixel-scale operations on the residual image and the carrier image to generate an encrypted image with hidden information.

2. The image encryption method based on deep learning according to claim 1, characterized in that: In the step of converting the hidden information to generate a corresponding feature map, the hidden information is converted into the feature map with three channels through a preprocessing network, and a feature vector length and a resolution of the feature map are adjusted according to a resolution of the carrier image.

3. The image encryption method based on deep learning according to claim 1, characterized in that: The hidden information is selected from binary information, and an information length of the hidden information is 1 to 200 bits.

4. The image encryption method based on deep learning according to claim 1, characterized in that: In the step of converting the carrier image by the processing unit to generate a reference image, the processing unit converts the carrier image by a noise visibility function to generate the reference image.

5. The image encryption method based on deep learning according to claim 1, characterized in that: In the step of inputting the carrier image, the feature map and the reference image into an encryptor to generate a corresponding residual map, a loss function is used to operate the residual map and the reference image as network supervision.

6. The image encryption method based on deep learning according to claim 1, characterized in that: In the step of inputting the carrier image, the feature map and the reference image into an encryptor to generate a corresponding residual map, the encryptor performs calculations on the carrier image, the feature map and the reference image according to an initial weight to generate the residual map.

7. A decoding method based on deep learning, characterized in that: The steps include: Acquire an encrypted image with hidden information; Correcting the encrypted image with a decoder; Calculating the corrected encrypted image to extract a corresponding feature map; and The feature map is operated to obtain corresponding hidden information.

8. The deep learning-based decoding method according to claim 7, characterized in that: The decoder attacks the encrypted image to generate a corresponding damaged image, and performs decoding training using the damaged image.

9. The deep learning-based decoding method according to claim 8, characterized in that: The method of attacking the encrypted image is selected from perspective bending, motion blur and out-of-focus blur, color adjustment, adding noise, compression or a combination of any two or more of the above.

10. The deep learning-based decoding method according to claim 7, characterized in that: In the step of calculating the corrected encrypted image to extract a corresponding feature map, the decoder calculates the corrected encrypted image according to an initial weight to extract the corresponding feature map.

Images