A reinforcement learning-based image DCT steganography optimization method

Abstract

The present invention proposes an image DCT steganography optimization method based on reinforcement learning, which relates to the field of image processing and information hiding technology. By dynamically selecting appropriate image blocks in the discrete cosine transform (DCT) domain for steganographic operation, the security of information embedding and image quality are optimized. The method first divides the image into several DCT coefficient blocks, and constructs a strategy network for image blocks in different regions. According to the DCT coefficients, energy distribution and image quality indicators of each image block, the system state is constructed, and the DCT coefficient embedding block for the steganographic operation is dynamically selected using a hierarchical reinforcement learning algorithm. While ensuring the image quality, it can not only make the embedded information more concealed, but also effectively avoid being easily identified in conventional detection algorithms.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and information hiding, and in particular to a method for optimizing discrete cosine transform (DCT) steganography of an image based on reinforcement learning. Background Art

[0002] With the widespread application of digital images and the increasing demand for information security, image steganography has attracted widespread attention as an important means of information hiding. Existing image steganography methods embed secret information in the frequency domain of the image (such as the DCT domain) to hide information. However, how to select appropriate frequency domain blocks for steganography without significantly affecting the image quality is still an important research issue.

[0003] Traditional image steganography technology mainly relies on fixed rules or simple random selection, which makes it difficult to achieve the best steganography effect in complex environments. These methods often cannot adapt to the diversity and complexity of image content, making the steganographic information easy to detect and attack. Steganography methods based on reinforcement learning use the adaptability and dynamic decision-making ability of reinforcement learning. By continuously learning and adjusting strategies, they can achieve the best steganography effect in complex environments. For example, the Chinese patent application publication number CN111882476A invented an image steganography method based on deep learning to automatically learn the embedding cost. By building a reinforcement learning steganography framework, the embedding position is dynamically adjusted to optimize the steganography effect; the Chinese patent application publication number CN108460405A invented an image steganalysis integrated classification optimization method based on deep reinforcement learning, which uses a deep reinforcement learning model to analyze and classify steganographic information, thereby improving the security and robustness of steganographic information.

[0004] With the development of machine learning technology, especially the application of reinforcement learning technology, it has become possible to dynamically and intelligently select steganographic blocks. However, the existing image steganography methods based on reinforcement learning fail to make full use of the local features of the image and the correlation between blocks, resulting in unsatisfactory steganographic efficiency and effect. In order to solve this problem, the present invention proposes an image DCT steganography optimization method based on hierarchical reinforcement learning, which improves the concealment and robustness of the steganographic information by hierarchically selecting DCT blocks for information embedding, while ensuring image quality. Summary of the invention

[0005] In view of the defects and shortcomings of the prior art, the purpose of the present invention is to provide an image DCT steganography optimization method based on reinforcement learning, which selects appropriate DCT blocks for information embedding through reinforcement learning, thereby improving the concealment and robustness of the stego information.

[0006] An image DCT steganography optimization method based on reinforcement learning comprises the following steps:

[0007] Step 1: Receive a carrier image, input the carrier image into a pre-established DCT transform module, select N blocks therein as candidate blocks, and divide the N blocks into four parts;

[0008] Step 2: Define an independent policy network for each part. Each network is responsible for selecting a specific block in its corresponding part for steganography. The weight parameter of the Q network in each policy network is θ, and the learning rate α and discount factor γ are set.

[0009] Step 3: Construct the state vector s for k time slots (k) ;

[0010] Step 4: Each layer selects the state vector s constructed according to (k) , through the strategy set X, select the appropriate block and perform encryption operation;

[0011] Step 5: After the steganographic operation is performed, the current reward u is calculated based on the information between the carrier image and the original image and the embedded image quality index. (k) ;

[0012] Step 6: Store the current state, selected strategy, and calculated reward value into the experience pool.

[0013] Step 7: Randomly sample Z experiences from the experience playback buffer pool D to form a batch sample, and construct sample B = {η (g(ζ))} 1≤ζ≤Z ;

[0014] Step 8: Use the Adam optimization algorithm to update the weight parameter θ of the Q network in the policy network (k) :

[0015]

[0016] Among them, β is the discount factor for updating the weight parameters.

[0017] Step 9: Repeat steps 3 to 8 until a stable block selection strategy is learned, with benefit u (k) converges to a stable value.

[0018] Preferably, in step 1, the DCT transformation module first converts the received carrier image into a grayscale image, and then performs block processing to divide the image into a number of 8×8 pixel blocks. A discrete cosine transform (DCT) is performed on each 8×8 image block to convert the pixel value in the spatial domain into the DCT coefficient in the frequency domain. The formula for DCT transformation is as follows:

[0019]

[0020] Where u and v are the pixel length and width of the carrier image, and α(u) and α(v) are normalization coefficients:

[0021]

[0022] The coefficients after DCT transformation are reordered, usually in a "zigzag" scanning order. The purpose of reordering is to concentrate low-frequency coefficients in the front and high-frequency coefficients in the back, which is convenient for subsequent steganographic operations. Through DCT transformation and reordering, the image is divided into 8×8 DCT blocks. Select n blocks as candidate blocks, and divide these N blocks into four parts (A, B, C, D). Part A and N A Modules, Part B N B Modules, Part C N C Modules, Part D N D modules, where N A +N B +N C +N D =N.

[0023] Preferably, in step 2, each policy network includes an input layer, a hidden layer and an output layer, the input layer receives the state information of the current part, and the output layer gives the probability distribution of block selection. Input layer: input the state information of the current part, including DCT coefficient matrix, energy distribution, image quality index, etc. Hidden layer: extract features through a multi-layer neural network, perform nonlinear transformation, and improve the expression ability of the model. Output layer: output the probability distribution of the selected block, which is used to determine the specific block selected in the current part. Initialize the weight parameter W of each policy network. Initialize the weight parameter of the Q network to θ, set the learning rate α and the discount factor γ. Among them, 0<α≤1, 0≤γ≤1; the learning rate is used to control the step size of each parameter update, which determines the amplitude of the adjustment of the weight parameter θ in each training iteration of the network; the discount factor γ represents the importance of future rewards and is used to convert future rewards into current values. Specifically, a larger γ value means that the algorithm pays more attention to future rewards, while a smaller γ value means that the algorithm pays more attention to immediate rewards.

[0024] Each layer of the policy network is responsible for selecting blocks within a specific part. Specifically: The first layer of the policy network is responsible for selecting N blocks in part A. a blocks, where 1≤N a ≤N A ; The second layer policy network is responsible for selecting N in part B b blocks, where 1≤N b ≤N B ; The third layer policy network is responsible for selecting N in part Cc blocks, where 1≤N c ≤N C ; The fourth layer policy network is responsible for selecting N in part D d blocks, where 1≤N d ≤N D Except for the first level, the state input of each level includes the DCT coefficient matrix, energy distribution, image quality index of the previous time slot, and the blocks selected by the previous level. The first level only contains the DCT coefficient matrix, energy distribution, and image quality index of the previous time slot.

[0025] Preferably, in step 3, each time slot k needs to construct a corresponding state vector for each part, and the specific process is as follows:

[0026] At time slot k, the DCT coefficient matrix is ​​obtained by the carrier image Energy distribution (k) , the image quality of the previous time slot, including the Peak Signal to Noise Ratio (PSNR) (k - 1) , Structural Similarity Index Metric (SSIM) m (k-1) and Mean-Squared-Error (MSE) (k-1) .

[0027] Each DCT coefficient matrix is ​​an 8×8 matrix. Each matrix is ​​a representation of each 8×8 image block in the frequency domain in step 1, with a value range of -1024 to 1024. The energy distribution matrix is ​​obtained by calculating the sum of the squares of all coefficients in each DCT coefficient matrix:

[0028]

[0029] Among them, d ij Represents the value of the i-th row and j-th column in the DCT coefficient matrix. Each DCT coefficient matrix corresponds to an energy value on the energy distribution matrix.

[0030] The peak signal-to-noise ratio is calculated by the following formula:

[0031]

[0032] Among them U max is the maximum pixel value of the carrier image.

[0033] The mean square error is calculated by the following formula:

[0034]

[0035] Where U(e, f) and V(e, f) represent the predicted pixel value and the true pixel value at the e row and f column of the image respectively, O represents the total number of rows of the image, and P represents the total number of columns of the image.

[0036] The structural similarity is calculated by the following formula:

[0037]

[0038] Among them, μ u and μ ij represents the average brightness of image u and image V, and represents the brightness variance of u and V, σ uv represents the brightness covariance of u and V. Two constants x = (h1·W) and y = (h2·W), where W = 2 位深度 -1, x and y are used here to avoid zero denominator.

[0039] Finally, the constructed state vector s (k) as follows:

[0040]

[0041] Preferably, in step 4, the policy set is defined as X={X A , X B , X C , X D}, where X A represents the set of actions for selecting modules in part A, where X B represents the set of actions for selecting modules in part B, where X C represents the set of actions for selecting modules in part C, where X D Represents the action set for selecting modules in part D. Specifically, the action set for each module contains the action numbers of all possible module selection methods in the current part. For example, in part A, we will select N A Choose N out of Part A modules a All optional ways of modules are numbered, and each number corresponds to a selection method of modules in part A, that is, each action number of module A, so X A The Chinese Communist Party elements, each of which is a number corresponding to an action of a selection module.

[0042] The state vector s of part A (k) Input to the first layer of the policy network. The first layer of the policy network calculates X based on the input state vector AThe Q value of each possible action in . Select an action according to the greedy strategy and get the action number selected in module A at the current moment Use s (k) and As the new state input vector of the second layer, it is input into the policy network of the second layer, from X B Select the action number selected by module B at the current moment Use s (k) , As the new state input vector of the third layer, it is input into the policy network of the third layer, from X C Select to get the action number selected by C module at the current moment Use s (k) , As the new state input vector of the fourth layer, it is input into the policy network of the fourth layer, from X D Select the action number selected by module D at the current moment Finally, according to the action number Select the corresponding part of the module and form the selection strategy a for the current time slot k (k) :

[0043]

[0044] After the module is selected, an encryption operation is performed. In each selected block, a set of DCT coefficients is selected for information embedding. The bits of the secret information are embedded into the selected DCT coefficients. A simple least significant bit (LSB) replacement method or other complex embedding methods can be used. Then an inverse DCT transform is performed on the modified DCT coefficient matrix to obtain an image block embedded with the secret information.

[0045] Preferably, in step 5, a steganographic operation is performed on the block selected in step 4. The DCT coefficients of the selected DCT block are modified to embed the required secret information. At the same time, it is ensured that the modified DCT coefficients meet the steganographic requirements while maintaining the image quality as much as possible. After performing the steganographic operation, the image quality index r after embedding is calculated (k) , m (k) , e (k) .

[0046] Finally calculate the reward u (k) :

[0047] u (k) =m (k) +c1r (k) -c2e (k)

[0048] Where c1 and c2 are the weight coefficients of PSNR and SSIM respectively.

[0049] Preferably, in step 6, the state s (k) 、Choose strategy a (k) 、Effect (k) Constructed as an empirical sequence f (k) ={s (k) , a (k) ,u (k)}, and store the experience sequence into the experience pool D = {f (1) , f (2) , …, f (k)}middle.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. Improved stealth: Through the design of a hierarchical reinforcement learning model, the present invention can adaptively select the best DCT block for steganographic operations in a complex image environment. Specifically, the reinforcement learning model dynamically adjusts the embedding strategy according to the DCT coefficient matrix, energy distribution, and image quality index of each block, thereby maximizing the selection of blocks that have the least impact on the overall visual quality of the image but the highest stealth for information embedding. This method can not only make the embedded information more concealed, but also effectively avoid being easily identified in conventional detection algorithms.

[0052] 2. Strong adaptability: Different from traditional steganographic methods with fixed rules or simple random selection, the present invention can achieve deep understanding of image content and adaptive strategy selection through reinforcement learning model. By independently analyzing image blocks through each layer of policy network and dynamically adjusting the block selection and information embedding method, the system can automatically select the optimal steganographic strategy when facing different image types. In addition, as the learning process proceeds, the weight parameters of the policy network are continuously updated, so that the system can gradually adapt to different image content characteristics and achieve efficient processing of complex image environments. DETAILED DESCRIPTION

[0053] In order to more clearly understand the technical content of the present invention, the following embodiments are given in detail.

[0054] An image DCT steganography optimization method based on reinforcement learning comprises the following steps:

[0055] Step 1: Receive a carrier image, input the carrier image into a pre-established DCT transform module, select 36 blocks as candidate blocks, and divide the 36 blocks into four parts, including 15 blocks in part A, 11 blocks in part B, 7 blocks in part C, and 3 blocks in part D;

[0056] Step 2: Define an independent policy network for each part. Each policy network contains an input layer, a hidden layer, and an output layer. Initialize the learning rate α to 0.5 and the discount factor δ to 0.7. Each layer of policy network is responsible for selecting blocks in a specific part. The first layer of policy network is responsible for selecting 2 blocks in part A; the second layer of policy network is responsible for selecting 3 blocks in part B; the third layer of policy network is responsible for selecting 2 blocks in part C; the fourth layer of policy network is responsible for selecting 1 block in part D;

[0057] Step 3: Calculate the DCT coefficient matrix based on the carrier image and energy distribution g (k) , g (k) is a vector containing 36 elements, corresponding to the energy of each matrix in the DCT coefficient matrix;

[0058] Based on the peak signal-to-noise ratio r at the previous moment (k-1) , structural similarity m (k-1) And the mean square error e (k-1) , jointly build

[0059]

[0060] In the SSIM calculation formula, the two constants x = (h1·W) and y = (h2·W), W = 2 位深度 -1, h1=0.01, h2=0.03, bit depth=8;

[0061] Step 4: Define the policy set as X = {X A , X B , X C , X D}, where X A There are C(15,2)=105 elements in total. Each element is a number, corresponding to an action of selecting a module. For example, number 1 represents selecting the first and second modules from 15 modules, number 2 represents selecting the first and third modules, and so on. There are 105 selection methods in total, so there are 105 elements. B There are C(11,3)=165 elements, X C There are C(7,2)=21 elements, X D There are C(3,1)=3 elements.

[0062] The state vector s of part A (k) Input to the first layer of the policy network. The first layer of the policy network calculates X based on the input state vector A The Q value of each possible action in . Select an action according to the greedy strategy and get the action number selected in module A at the current moment Use s(k) and As the new state input vector of the second layer, it is input into the policy network of the second layer, from X B Select the action number selected by module B at the current moment Use s (k) , As the new state input vector of the third layer, it is input into the policy network of the third layer, from X C Select to get the action number selected by C module at the current moment Use s (k) , As the new state input vector of the fourth layer, it is input into the policy network of the fourth layer, from X D Select the action number selected by module D at the current moment Finally, according to the action number Select the corresponding part of the module and form the selection strategy a for the current time slot k (k) :

[0063]

[0064] After selecting the module, the encryption operation is performed. In each selected block, a set of DCT coefficients is selected for information embedding. The bits of the secret information are embedded into the selected DCT coefficients. The embedding is performed using the LSB replacement method. Then the inverse DCT transform is performed on the modified DCT coefficient matrix to obtain the image block embedded with the secret information.

[0065] Step 5: After performing the steganographic operation, the quality index r of the embedded image is calculated based on the information between the carrier image and the original image. (k) , m (k) , e (k) .

[0066] Finally calculate the current reward u (k) :

[0067] u (k) =m (k) +c1r (k) -c2e (k)

[0068] Where c1 = 0.2; c2 = 5;

[0069] Step 6: Set the state s (k) 、Choose strategy a (k) 、Effect (k) Constructed as an empirical sequence f (k) ={s (k) , a (k) ,u (k)}, and store the experience sequence into the experience pool D = {f(1) ,f( 2) , …, f (k)}middle.

[0070] Step 7: Extract z samples from the buffer pool, traverse each sample, use the deep Q network algorithm to estimate the cumulative return, and use the Adam optimizer to update the Q network parameters.

[0071] Step 8: Update the weight parameter θ of the Q network using the Adam optimization algorithm (k)

[0072] Step 9: Repeat steps 3 to 8 until a stable block selection strategy is learned, with benefit u (k) converges to a stable value.

[0073] The above embodiments are only preferred embodiments of the present invention and cannot be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. An image DCT steganography optimization method based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Receive a carrier image, input the carrier image into a pre-established DCT transform module, select N blocks therein as candidate blocks, and divide the N blocks into four parts; Step 2: Define an independent policy network for each part. Each network is responsible for selecting a specific block in its corresponding part for steganography. The weight parameter of the Q network in each policy network is θ, and the learning rate α and discount factor γ are set. Step 3: Construct the state vector s for k time slots (k) ; Each time slot k constructs the corresponding state vector for each part. The specific process is: in time slot k, obtain its DCT coefficient matrix through the received carrier image Energy distribution (k) , the image quality of the previous time slot, including peak signal-to-noise ratio PSNR, structural similarity SSIM and mean square error MSE, where the peak signal-to-noise ratio PSNR is represented by r (k-1) , the structural similarity SSIM is represented by m (k-1) , mean square error MSE is expressed as e (k-1) ; Each DCT coefficient matrix is ​​an 8×8 matrix. Each matrix is ​​the representation of each 8×8 image block in the frequency domain in step 1, with a value range of -1024 to 1024. The energy distribution matrix is ​​obtained by calculating the square sum of all coefficients in each DCT coefficient matrix: Among them, d ij Represents the value of the i-th row and j-th column in the DCT coefficient matrix. Each DCT coefficient matrix corresponds to an energy value on the energy distribution matrix. The peak signal-to-noise ratio is calculated by the following formula: Among them U max is the maximum pixel value of the carrier image; The mean square error is calculated by the following formula: Where U(e,f) and V(e,f) represent the predicted pixel value and the true pixel value at row e and column f of the image respectively, O represents the total number of rows of the image, and P represents the total number of columns of the image; The structural similarity is calculated by the following formula: Among them, μ u and μ u represents the average brightness of image U and image V, and represents the brightness variance of U and V, σ uv Denotes the brightness covariance of U and V, two constants x = (h1·W) and y = (h2·W), where W = 2 位深度 -1, where x and y are used to avoid zero denominator; Finally, the constructed state vector s (k) as follows: Step 4: Each layer selects the state vector s constructed according to (k) , through the strategy set X, select the appropriate block and perform encryption operation; Step 5: After the steganographic operation is performed, the current reward u is calculated based on the information between the carrier image and the original image and the embedded image quality index. (k) ; Step 6: Store the current state, selected strategy, and calculated reward value into the experience pool; Step 7: Randomly sample Z experiences from the experience playback buffer pool D to form a batch sample, and construct sample B = {η (g(ζ)) } 1≤ζ≤Z ; Step 8: Use the Adam optimization algorithm to update the weight parameter θ of the Q network in the policy network (k) : Among them, β is the discount factor for weight parameter update; Step 9: Repeat steps 3 to 8 until a stable block selection strategy is learned, with benefit u (k) converges to a stable value.

2. According to the reinforcement learning-based image DCT steganography optimization method of claim 1, it is characterized in that: The step 4 is specifically as follows: the policy set is defined as X={X A ,X B ,X C ,X D }, where X A represents the set of actions for selecting modules in part A, where X B represents the set of actions for selecting modules in part B, where X C represents the set of actions for selecting modules in part C, where X D Represents the action set for selecting modules in part D. Specifically, the action set for each module contains the action numbers of all possible module selection methods in the current part; The state vector s of part A (k) Input to the first layer of the policy network, the first layer of the policy network calculates X according to the input state vector A The Q value of each possible action in , select an action according to the greedy strategy, and get the action number selected in module A at the current moment Use s (k) and As the new state input vector of the second layer, it is input into the policy network of the second layer, from X B Select the action number selected by module B at the current moment Use s (k) , As the new state input vector of the third layer, it is input into the policy network of the third layer, from X C Select to get the action number selected by C module at the current moment Use s (k) , As the new state input vector of the fourth layer, it is input into the policy network of the fourth layer, from X D Select the action number selected by module D at the current moment Finally, according to the action number Select the corresponding part of the module and form the selection strategy a for the current time slot k (k) : After the module is selected, the encryption operation is performed. In each selected block, a set of DCT coefficients is selected for information embedding. The bits of the secret information are embedded into the selected DCT coefficients using the least significant bit (LSB) replacement method. Then, the inverse DCT transform is performed on the modified DCT coefficient matrix to obtain the image block embedded with the secret information.

3. According to the image DCT steganography optimization method based on reinforcement learning in claim 1, it is characterized in that: The step 5 is specifically as follows: performing a steganographic operation on the block selected in the step 4, modifying the DCT coefficients of the selected DCT block to embed the required secret information, while ensuring that the modified DCT coefficients meet the steganographic requirements and keep the image quality as much as possible. After performing the steganographic operation, calculating the embedded image quality index r (k) ,m (k) ,e (k) ; Finally calculate the reward u (k) : you (k) =m (k) +c1r (k) -c2e (k) Where c1 and c2 are the weight coefficients of PSNR and SSIM respectively.

4. According to the reinforcement learning-based image DCT steganography optimization method of claim 1, it is characterized in that: The specific operation of step 6 is: (k) 、Choose strategy a (k) 、Effect (k) Constructed as an empirical sequence f (k) ={s (k) ,a (k) ,u (k) }, and store the experience sequence into the experience pool D = {f (1) ,f (2) ,…,f (k) }middle.

Images