An image steganography method based on reinforcement learning

Abstract

The present invention provides a method for improving the efficiency and security of image steganography by utilizing a reinforcement learning algorithm and a GAN model, comprising the following steps: performing image steganography pre-training using a GAN model. Utilizing the Div2K dataset as the original cover image and selecting a random information set, the model is enabled to obtain a steganographic image similar to the original image; and optimizing the steganographic parameters using a reinforcement learning algorithm. Initial parameters are randomly confirmed, and then the optimal parameters are selected and replaced based on the pre-training results; and the GAN model is trained and the image steganography effect is evaluated. Image steganography is implemented using the pre-trained model and the optimal parameters, and the performance is evaluated based on the steganographic performance and steganographic security. The present invention effectively improves the efficiency of the steganographic task by selecting a suitable steganographic method based on an analysis of the task situation, including embedded information and original image features.

Description

Technical Field

[0001] The present invention relates to the field of network security, and in particular to an image steganography method based on reinforcement learning. Background Art

[0002] Steganography is a technique for covertly embedding specific information within other forms of information. In communications, information to be transmitted can be written into carriers such as text, images, audio and video, and protocols to ensure confidential and secure transmission. Therefore, steganography has important and widespread applications in the field of information security. Unlike cryptography, which focuses on concealing the content of information and encrypting messages before transmission, steganography focuses on concealing the transmission process itself. In network communications, secret information encrypted using cryptographic algorithms is unreadable but visible. Therefore, attackers can decrypt and obtain secret information by attempting to guess the information or stealing the key. Steganography overcomes this limitation by making the communication itself invisible.

[0003] Thanks to the ubiquity of image information on the internet, image steganography, compared to other steganographic methods, offers higher information redundancy, greater hidden information capacity, a wider range of steganographic algorithms, and support for a variety of common image formats. It is currently the most widely used steganographic method. Based on the method of embedding information into pixels, image steganography can be broadly divided into two traditional methods: spatial domain-based and frequency domain-based.

[0004] In recent years, image steganography has been increasingly combined with deep learning algorithms to improve the efficiency, detection, and security of steganography algorithms, such as their resistance to statistical analysis and active attacks. Liu et al. proposed an adversarial embedding method to enhance the security of image steganography. By analyzing the gradients of the mask and stego images, they select fewer embedding units for stegoing, reducing the embedding cost. This method also improves resistance to analysis compared to traditional full or random selection methods. However, the selection of specific embedding points affects the efficiency of stegoing, and excessive modification may affect stegoing performance. Tan et al. proposed a channel-attention image steganography method based on a generative adversarial network. This method uses the XuNet network as a discriminator to enhance stegoing security and implements error correction through error correction codes. However, the additional computational and storage overhead associated with this approach reduces the efficiency and effective embedding capacity of the stegoing. Chinese Patent Authorization Publication No. CN 106530200B proposes a stego-image detection method based on a deep learning model. This method pre-trains a convolutional neural network using supervised learning to detect the carrier image with lower computational dimensions and less computational effort, thus improving detection speed.

[0005] Reinforcement learning, a machine learning paradigm, trains decision-making agents to interact with their environment and optimize their strategy selection based on the real-time state of the environment and feedback from their actions. Combining reinforcement learning with image steganography improves the security and stability of steganography by comprehensively considering both the actual steganographic image and embedded information. Chinese Patent Publication No. CN 111882476B proposes a method for automatically learning embedding costs based on deep reinforcement learning. This method utilizes a policy network to calculate embedding costs and select modification points, combined with encoding methods to generate a stegoscopy image.

[0006] Unlike the above-mentioned method of combining deep learning to optimize steganography efficiency or steganography security alone, this patent considers the cost and efficiency of the steganography algorithm and the security of the secret image at the same time. By analyzing the characteristics of the original image and the length of the embedded information, the cost and effect of steganography are estimated, and the steganography method and depth are assigned to the task. While ensuring security, a steganography process with smaller computational complexity and complexity is achieved. Summary of the Invention

[0007] The purpose of this invention is to optimize the image steganography pattern and the depth of embedded information by analyzing the original image and hidden information using a reinforcement learning algorithm, while ensuring the security of image steganography, so as to improve the efficiency and security of the communication process and realize the secure transmission of information.

[0008] Technical solution: An image steganography method based on reinforcement learning, including:

[0009] Initialize the SteganoGAN model and steganography method, use the Div2K dataset as the original cover image set, process the original cover image, and prepare the steganographic information;

[0010] Initialize the reinforcement learning agent, use the reinforcement learning algorithm to select the steganographic working mode, and perform the steganographic operation; evaluate the steganographic effect of the steganographic image and the original image, and construct an experience sequence and put it into the experience playback buffer pool;

[0011] Draw samples from the experience replay buffer pool to update the reinforcement learning agent network parameters.

[0012] Preferably, initialize M steganography methods, obtain the SteganoGAN generative steganography model based on GAN, set the initial working parameters of the model, select the Div2K dataset as the original cover image set U; for the mth (1≤m≤M) steganography method, define q m is the image quality threshold vector corresponding to the mth steganographic method, defined as Represents the maximum value that the current image quality index needs to meet; collect the information set B to be embedded, the number of information items contained in the information set B to be embedded is N = |B|, and traverse all the information b to be communicatedn , b n ∈B, determine the maximum hidden information length L, L=max{|b n |} 1≤n≤N ; Initialize the original cover picture set U, and |U|≥|B|, for any j-th (1≤j≤|U|) original cover picture U j , its quality must satisfy max{|q m |} 1≤m≤M indicators.

[0013] Preferably, each communication round k transmits message b (k) ∈B, length l (k) , in bits, and select U from the original image set U (k) As the original cover image, the number of pixels of the original cover image is p (k) , and calculate the image frequency ξ (k) , which constitutes the k-th round of image steganography task t (k) = (k) , U (k) >

[0014] According to the steganographic task t (k) , construct the current task state vector in is the RS analysis result of the secret image in the previous state, is the relative entropy between the original image and the hidden image in the previous state, and is input into the deep Q network of the reinforcement learning algorithm, where the average embedding capacity Represents the density of information steganography in the previous round of communication, general image quality index where m u and m v Represent the mean of the hidden image signal of the original cover image, and Represent the variance of the two image signals, s uv is the standard deviation of the two image signals, ρ∈[-1,1], the higher the image similarity, the larger the value;

[0015] According to the value of the current state-action pair estimated by the deep Q network, the steganalysis method m is selected for the current task (k) and the embedding information depth d (k) , select the action a (k) =[m (k) ,d (k) ], there is a probability of ε to make a random selection, and a probability of 1-ε to select the action with the greatest value, that is, A is a set of optional actions, where ​is the action that maximizes the Q value in the current state, that is, the selected steganographic working mode, including the steganographic method and embedding bit depth, θ (k) is the agent's deep Q network parameter;

[0016] According to the selected working mode a (k) Embed the information into the original cover image to get the hidden image V (k) ;

[0017] Compare the original cover image with the hidden image, perform security analysis on the hidden image, and calculate the reward for choosing this action for steganography through the reward function;

[0018] The comparison results are obtained by calculating the peak signal-to-noise ratio of the two images and structural similarity To express the peak signal-to-noise ratio in is the maximum value of the current image point color, is the mean square error between the original cover image and the hidden image, p (k) ,q (k) Represents the number of row and column pixels respectively. The higher the peak signal-to-noise ratio, the better the steganographic effect. Among them, h1 and h2 are bias constants to prevent division by zero errors. The higher the structural similarity, the better the steganographic effect. The security analysis uses RS analysis and calculation of the relative entropy of the two images. The results of the analysis are updated respectively. and In order to balance security and steganography efficiency, the reward function is where t (k) It is m (k) The corresponding computational complexity, w1, w2, w3 are the corresponding weight parameters; construct the empirical sequence η (k) ={s (k) ,a (k) ,r (k)} and store it in the experience replay buffer pool D.

[0019] Preferably, Z experiences are randomly sampled from the experience replay buffer pool D to construct sample B= And use the optimization algorithm to update the deep Q network parameters θ (k) :

[0020]

[0021] in U(1,Z) is a uniform distribution.

[0022] Repeat the above steps until all the information to be transmitted is embedded into the image through steganography, and the communication task is completed;

[0023] In order to improve the security of the agent learning, the RS analysis results of the hidden image Relative entropy of the original cover image and the hidden image The analysis result is set to fail the analysis.

[0024] The beneficial effects of the present invention are:

[0025] 1. The present invention selects a suitable steganographic method by analyzing the task situation including the embedded information and the original image features, thereby effectively improving the efficiency of the steganographic task.

[0026] 2. By considering indicators such as peak signal-to-noise ratio and structural similarity, the present invention can more accurately evaluate the task effect, which is beneficial to the optimization process of the decision-making intelligent agent.

[0027] 3. The present invention considers using a variety of security analysis methods including RS analysis and relative entropy, so that the steganographic stegosaurus has better anti-analysis ability and stronger anti-interference characteristics, further ensuring the security of information transmission. DETAILED DESCRIPTION

[0028] In order to more clearly understand the technical content of the present invention, the following embodiments are given to illustrate in detail. An image steganography method based on deep reinforcement learning comprises the following steps:

[0029] The present invention comprises the following steps:

[0030] Step 1: Initialize M kinds of steganography methods, such as the least significant bit (LSB) method based on the spatial domain, the discrete cosine transform (DCT), discrete Fourier transform (DFT), discrete wavelet transform (DWT) based on the frequency domain, and the convolutional neural network based on deep learning, the generative adversarial network and other methods, and set their initial working parameters, including the number of pixels to be modified, the error correction code encoding method, the key, the compression rate, the number of network layers and the weight bias parameters. For the mth (1≤m≤M) mode, q m is the image quality threshold vector corresponding to this method, Represents the minimum value that the current quality indicator must meet, such as image size, resolution, format, etc.

[0031] Step 2: Collect the information set B to be embedded, the number of information is N = |B|, and traverse all the information b to be communicated n ∈B, determine the maximum hidden information length L, L=max{|b n |} 1≤n≤N .

[0032] Step 3: Initialize the original cover image set U, and |U|≥|B|, for any j-th (1≤j≤|U|) picture U j , its quality must satisfy max{|q m |} 1≤m≤M indicators.

[0033] Step 4: Based on the original image set U and the embedded information set B, considering the difficulty of the steganography task, set the reinforcement learning agent and construct a deep Q network Q(·,·;θ) for it, initialize the learning rate α, discount factor γ, exploration rate e, and neural network parameters q (0) .

[0034] Step 5: Initialize the Average Embedded Capacity (AEC) c (0) ,Universal Image Quality Index (UIQI)ρ (0) , the initial value is usually set to 0, and includes the RS (Regular Singular) analysis results of the hidden image Relative entropy of the original image and the hidden image The security analysis results, including the ones from the above, are usually set to fail the analysis to increase the emphasis on the security of the agent's learning.

[0035] Step 6: Transmit message b in each communication round k (k) ∈B, length l (k) , in bits, and select U from the original image set U (k) As the original cover image, the number of pixels of the image is p (k) and calculate the image frequency ξ by methods such as wavelet transform (k) , which constitutes the k-th round of image steganography task t (k) = (k) ,U (k) >.

[0036] Step 7: According to the steganographic task t (k) , construct the current task state vector And input into the deep Q network. Among them, the average embedding capacity Represents the density of information steganography in the previous round of communication, general image quality index where m​u and m v represent the mean of the original image and hidden image signals respectively, and Represent the variance of the two image signals, s uv is the standard deviation of the two image signals, ρ∈[-1,1], and the higher the image similarity, the larger the value.

[0037] Step 8: Based on the value of the current state-action pair estimated by the Q network, select the steganalysis method m for the current task (k) and the embedding information depth d (k) , select the action a (k) =[m (k) ,d (k) ], there is a probability of ε to make a random selection, and a probability of 1-ε to select the action with the greatest value, that is, A is a set of optional actions, where the elements are all steganographic working modes supported in the current communication task. The working mode includes the steganographic method and the embedding bit depth.

[0038] Step 9: According to the selected working mode a (k) Embed the information into the original cover image to get the hidden image V (k) .

[0039] Step 10: Compare the original image with the hidden image, perform security analysis on the hidden image, and calculate the reward for choosing this action for steganography through the reward function.

[0040] In step 10, the comparison results are calculated by calculating the Peak Signal to Noise Ratio (PSNR) of the two images. and Structural Similarity Index Metric (SSIM) Peak signal-to-noise ratio in is the maximum value of the current image point color, is the mean square error between the original image and the hidden image, p (k) ,q (k) Represents the number of pixels in rows and columns respectively. The higher the peak signal-to-noise ratio, the better the steganographic effect. Structural similarity Among them, h1 and h2 are bias constants to prevent division by zero errors. The higher the structural similarity, the better the steganographic effect. The security analysis uses RS analysis and calculation of the relative entropy of the two images. The results of the analysis are updated respectively. and In order to balance security and steganography efficiency, the reward function is

[0041] where t (k) It is m (k) The corresponding computational complexity, w1, w2, w3 are the corresponding weight parameters.

[0042] Step 11: Construct the experience sequence η (k) ={s (k) ,a (k) ,r (k)} and store it in the experience replay buffer pool D.

[0043] Step 12: Randomly sample Z experiences from the experience replay buffer pool D to construct a sample And use optimization algorithms (such as stochastic gradient descent) to update the agent's deep Q network parameters θ (k) .

[0044]

[0045] Step 13: Repeat steps 6-12 until all the information to be transmitted is embedded into the image through steganography, and the communication task is completed.

[0046] Example 1

[0047] Step 1: Initialize the image steganography generative adversarial network SteganoGAN and initialize three steganography methods: basic encoder, residual encoder, and dense encoder, i.e., M = 3.

[0048] Step 2: Select the Div2K dataset as the original cover image, and crop and integrate the original image according to the requirements of the SteganoGAN input layer. The original image adjusted as required is the original image in the dataset, cropped into a square shape, and spliced ​​into a two-row and two-column task image U i .

[0049] Step 3: Prepare embedded information set B for subsequent communication, with the maximum information length L = 1024 bits = 1 Mb.

[0050] Step 4: Initialize the reinforcement learning agent and construct a deep Q network Q(·,·;θ) for it. Initialize the learning rate α = 0.0001, the discount factor γ = 0.9, the exploration rate ε = 0.2, and the neural network parameters q (0) .

[0051] Step 5: Initialize the average embedding capacity c (0) =0, general image quality index ρ (0) =0, RS analysis results of hidden images Relative entropy of the original image and the hidden image

[0052] Step 6: In the kth round of communication, construct the state And input into the deep Q network.

[0053] Step 7: Select the corresponding steganographic working mode according to the Q value output by the Q network. SteganoGAN supports an embedding bit depth of 1 to 8 per pixel, i.e., d (k) ∈[1,8]. That is, each time a steganographic task is performed, the agent needs to select from |A|=24 steganographic methods to perform steganography. The agent has a probability of 0.8 to select a steganographic task action based on the Q value. There is a 0.2 probability of making a random selection.

[0054] Step 8: Perform steganography to obtain the secret image V of this round of communication (k) , and compare with the original picture U (k) , calculate the peak signal-to-noise ratio between two images Structural similarity relative entropy And perform RS analysis on the results and update the results

[0055] Step 9: Calculate the reward r at time k (k) , and construct the empirical sequence η (k) ={s (k) ,a (k) ,r (k)}, and store it in the experience replay buffer pool D.

[0056] Step 10: Extract Z samples from the buffer pool, traverse each sample, use the deep Q network algorithm to estimate the cumulative return, and use the Adam optimizer to update the Q network parameters.

[0057] Step 11: Repeat steps 6-10 until all information is embedded and transmitted.

[0058] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. An image steganography method based on reinforcement learning, characterized in that: include: Initialize the SteganoGAN model and steganography method, use the Div2K dataset as the original cover image set, process the original cover image, and prepare the steganographic information; Initialize the reinforcement learning agent, use the reinforcement learning algorithm to select the steganographic working mode, and perform the steganographic operation; Evaluate the steganographic effect of the steganographic image and the original image, build an experience sequence, and put it into the experience playback buffer pool; specifically, each communication round k transmits message b (k) ∈B, length l (k) , in bits, and select U from the original image set U (k) As the original cover image, the number of pixels of the original cover image is p (k) , and calculate the image frequency ξ (k) , which constitutes the k-th round of image steganography task t (k) = (k) ,U (k) >​ According to the steganographic task t (k) , construct the current task state vector in is the RS analysis result of the secret image in the previous state, is the relative entropy between the original image and the hidden image in the previous state, and is input into the deep Q network of the reinforcement learning algorithm, where the average embedding capacity Represents the density of information steganography in the previous round of communication, general image quality index Among them, μ u and μ v Represent the mean of the hidden image signal of the original cover image, and Represent the variance of the two image signals, σ uv is the standard deviation of the two image signals, ρ∈[-1,1], the higher the similarity of the images, the larger the value; According to the value of the current state-action pair estimated by the deep Q network, the steganalysis method m is selected for the current task (k) and the embedding information depth d (k) , select the action a (k) =[m (k) ,d (k) ], there is a probability of ε to make a random selection, and a probability of 1-ε to select the action with the greatest value, that is, A is a set of optional actions, where is the action that maximizes the Q value in the current state, that is, the selected steganographic working mode, including the steganographic method and embedding bit depth, θ (k) is the agent's deep Q network parameter; According to the selected working mode a (k) Embed the information into the original cover image to get the hidden image V (k) ; Compare the original cover image with the hidden image, perform security analysis on the hidden image, and calculate the reward for choosing this action for steganography through the reward function; The comparison results are obtained by calculating the peak signal-to-noise ratio of the two images and structural similarity To express the peak signal-to-noise ratio in, is the maximum value of the current image point color, is the mean square error between the original cover image and the hidden image, p (k) ,q (k) Represents the number of row and column pixels respectively. The higher the peak signal-to-noise ratio, the better the steganographic effect. Among them, h1 and h2 are bias constants to prevent division by zero errors. The higher the structural similarity, the better the steganographic effect. The security analysis uses RS analysis and calculation of the relative entropy of the two images. The results of the analysis are updated respectively. and In order to balance security and steganography efficiency, the reward function is where τ (k) It is m (k) The corresponding computational complexity, ω1, ω2, ω3 are the corresponding weight parameters; construct the empirical sequence η (k) ={s (k) ,a (k) ,r (k) }, and store it in the experience replay buffer pool D; Draw samples from the experience replay buffer pool to update the reinforcement learning agent network parameters.

2. The image steganography method based on reinforcement learning according to claim 1, characterized in that: Initialize M steganography methods, obtain the GAN-based generative steganography model SteganoGAN, set the initial working parameters of the model, and select the Div2K dataset as the original cover image set U; for the mth steganography method, 1≤m≤M, define q m is the image quality threshold vector corresponding to the mth steganographic method, defined as Represents the maximum value that the current image quality index must meet, 1≤i≤|q m |; Collect the information set B to be embedded, the number of information items contained in the information set B to be embedded is N = |B|, and traverse all the information b to be communicated n , b n ∈B, determine the maximum hidden information length L, L=max{|b n |} 1≤n≤N ; Initialize the original cover image set U, and |U|≥|B|, for any j-th original cover image U j , its quality must satisfy max{|q m |} 1≤m≤M kind of indicators, 1≤j≤|U|.

3. The image steganography method based on reinforcement learning according to claim 1, characterized in that: Randomly sample Z experiences from the experience replay buffer pool D to construct a sample And use the optimization algorithm to update the deep Q network parameters θ (k) : Among them, g(ζ)~U(1,Z), U(1,Z) is uniformly distributed; Repeat the above steps until all the information to be transmitted is embedded into the image through steganography, and the communication task is completed; In order to improve the security of the agent learning, the RS analysis results of the hidden image Relative entropy of the original cover image and the hidden image The analysis result is set to fail the analysis.

Images