Cross-modal steganography method and device, equipment and storage medium

Abstract

The invention provides a cross-modal steganography method and device, equipment and a storage medium, and the method comprises the steps: inputting a carrier image and a ciphertext image corresponding to ciphertext text data into a preset encoder, enabling a learnable wavelet transform network and a reversible neural network in the encoder to carry out the forward feature extraction of the carrier image and the ciphertext image, and obtaining a feature extraction result; obtaining a target carrier feature vector corresponding to the carrier image and a target ciphertext feature vector corresponding to the ciphertext image; and generating ciphertext cross-modal steganography result data corresponding to the ciphertext text data based on the target carrier feature vector and the target ciphertext feature vector for network transmission. According to the method and the device, the ciphertext text data of the text mode can be embedded into the carrier image of the image mode, a multi-mode steganography technology is realized, the steganography concealment and robustness of the ciphertext can be effectively improved, and the ciphertext network transmission security and reliability can be further improved.

Description

Technical Field

[0001] The present application relates to the field of information hiding technology, and in particular to a cross-modal steganography method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of information technology, information security issues have become increasingly prominent. Although traditional cryptographic methods can effectively protect the security of data during transmission, they are not flexible enough for the secret transmission and authentication of data, especially when the information needs to be embedded in public media content. Therefore, information hiding technology, especially image steganography, has received widespread attention as a means of covert communication. Image steganography allows secret information to be hidden in seemingly ordinary images, thereby achieving secret transmission of information. Digital watermarking technology, as an important branch of information hiding, is mainly used to protect the copyright of digital media content, authenticate the source, or embed additional information. Traditional watermark steganography methods usually embed watermarks into the frequency domain of the image to improve the invisibility of the watermark. However, their robustness under noise attacks is poor. Recently, deep learning-based methods have achieved better performance than traditional methods because they can better learn the deep features of watermarks and images.

[0003] However, existing watermark steganography technologies sacrifice the concealment of watermarks while ensuring robustness and cannot achieve cross-modal image steganography of ciphertext data. Traditional watermark steganography methods usually embed watermarks into the frequency domain of images, but when subjected to noise attacks, the watermarks will be destroyed and cannot be restored normally. In recent years, deep learning methods have made certain progress in the field of watermark steganography, but these methods often find it difficult to achieve satisfactory results in terms of concealment and robustness. In particular, when the encoding capacity is too strong, it becomes difficult to ensure the robustness of decoding, and robustness is the most important for watermarking methods. In other words, although the existing deep learning-based image steganography technology has played an important role in the fields of copyright protection and information security, its balance between robustness and invisibility, as well as its vulnerability to complex environments and malicious attacks, limit its potential for widespread application. Therefore, traditional technologies mainly face two challenges: imperceptibility and robustness, especially when facing various noise attacks. Summary of the invention

[0004] In view of this, embodiments of the present application provide a cross-modal steganography method, apparatus, device, and storage medium to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present application provides a cross-modal steganography method, comprising:

[0006] Inputting the carrier image and the ciphertext image corresponding to the ciphertext text data into a preset encoder respectively, so that the learnable wavelet transform network and the reversible neural network in the encoder respectively perform forward feature extraction on the carrier image and the ciphertext image to obtain a target carrier feature vector corresponding to the carrier image and a target ciphertext feature vector corresponding to the ciphertext image;

[0007] Based on the target carrier feature vector and the target ciphertext feature vector, ciphertext cross-modal steganography result data corresponding to the ciphertext text data is generated for network transmission.

[0008] In some embodiments of the present application, it also includes:

[0009] Receiving a noise image and extracting a noise carrier feature vector and a noise ciphertext feature vector corresponding to the noise image, wherein the noise image is formed after the ciphertext cross-modal steganography result data is transmitted over a network, and the ciphertext cross-modal steganography result data is generated in advance based on a target carrier feature vector corresponding to a carrier image output by the encoder and a target ciphertext feature vector corresponding to a ciphertext image of ciphertext text data;

[0010] The noise carrier feature vector and the noise ciphertext feature vector are respectively input into a decoder corresponding to the encoder, so that the learnable wavelet transform network and the reversible neural network in the decoder respectively perform inverse feature extraction on the noise carrier feature vector and the noise ciphertext feature vector, so as to obtain a loss carrier image corresponding to the noise carrier feature vector and a restored ciphertext image corresponding to the noise ciphertext feature vector; wherein the loss carrier image corresponding to the noise carrier feature vector is different from the carrier image used to generate the noise image to which the noise carrier feature vector belongs;

[0011] Information is extracted from the ciphertext image to obtain ciphertext text data corresponding to the restored ciphertext image.

[0012] In some embodiments of the present application, the encoder includes a forward transfer network corresponding to a learnable wavelet transform network, a forward transfer network corresponding to a reversible neural network, and a reverse transfer network corresponding to the learnable wavelet transform network, which are connected in sequence; correspondingly, the carrier image and the ciphertext image corresponding to the ciphertext text data are respectively input into a preset encoder, so that the learnable wavelet transform network and the reversible neural network in the encoder respectively perform forward feature extraction on the carrier image and the ciphertext image to obtain a target carrier feature vector corresponding to the carrier image and a target ciphertext feature vector corresponding to the ciphertext image, including:

[0013] The carrier image and the ciphertext image corresponding to the ciphertext text data are respectively input into a preset encoder, so that the forward transfer network corresponding to the learnable wavelet transform network in the encoder performs down-sampling feature extraction for multiple different frequency domains on the carrier image and the ciphertext image, and outputs the first multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively; then the forward transfer network corresponding to the reversible neural network in the encoder performs feature extraction for target operation on the first multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively based on the self-attention mechanism, and outputs the second multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively; then the reverse transfer network corresponding to the learnable wavelet transform network in the encoder performs up-sampling feature extraction for the same frequency domain on the second multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively, and outputs the target carrier feature vector corresponding to the carrier image and the target ciphertext feature vector corresponding to the ciphertext image.

[0014] In some embodiments of the present application, the decoder includes: a forward transfer network corresponding to the learnable wavelet transform network, a reverse transfer network corresponding to the reversible neural network, and a reverse transfer network corresponding to the learnable wavelet transform network, which are connected in sequence;

[0015] Correspondingly, the noise carrier feature vector and the noise ciphertext feature vector are respectively input into a decoder corresponding to the encoder, so that the learnable wavelet transform network and the reversible neural network in the decoder respectively perform inverse feature extraction on the noise carrier feature vector and the noise ciphertext feature vector to obtain a loss carrier image corresponding to the noise carrier feature vector and a restored ciphertext image corresponding to the noise ciphertext feature vector, including:

[0016] The noise carrier feature vector and the noise ciphertext feature vector are respectively input into the decoder, so that the forward transfer network of the learnable wavelet transform network in the decoder performs down-sampling feature extraction for multiple different frequency domains on the noise carrier feature vector and the noise ciphertext feature vector, and outputs the third multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively; then the reverse transfer network corresponding to the reversible neural network in the encoder performs feature restoration for the target operation on the third multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively based on the self-attention mechanism, and outputs the fourth multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively; then the reverse transfer network corresponding to the learnable wavelet transform network in the encoder performs up-sampling feature extraction for the same frequency domain on the fourth multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively, and outputs the loss carrier image corresponding to the noise ciphertext feature vector and the restored ciphertext image corresponding to the noise ciphertext feature vector.

[0017] In some embodiments of the present application, before the carrier image and the ciphertext image corresponding to the ciphertext text data are respectively input into a preset encoder, the method further includes:

[0018] The learnable wavelet transform network and the reversible neural network are trained based on a preset target loss function and training data, so that the learnable wavelet transform network and the reversible neural network constitute the encoder and the decoder respectively; wherein the training data is composed of various data samples, and each of the data samples contains a carrier image, a ciphertext image corresponding to the ciphertext data, and a noise image of the same size; the ciphertext image corresponding to the ciphertext data is generated by pre-processing the ciphertext data with watermark diffusion; the noise image is generated by pre-simulating a noise attack on the ciphertext cross-modal steganography result data of the ciphertext data corresponding to the noise image based on a plurality of preset noise types;

[0019] The learnable wavelet transform network is composed of a plurality of convolutional neural networks corresponding to different frequency domains for simulating Haar wavelet transform;

[0020] The reversible neural network is composed of multiple layers of deformable convolutional layers, and each layer of the deformable convolutional layers is associated based on a target operation, and the target operation includes: at least one of addition, subtraction, multiplication and division;

[0021] The deformable convolution layer is provided with a deformable attention dense block for executing a preset operation function, and the deformable attention dense block includes a connected self-attention block and a deformable dense block; wherein the deformable dense block includes: a deformable convolution layer and four convolution layers densely connected to the deformable convolution layer in sequence.

[0022] In some embodiments of the present application, the target loss function is composed of coding loss, image restoration loss, message decoding loss and contrastive learning loss, and weights corresponding to the coding loss, image restoration loss, message decoding loss and contrastive learning loss; the weights of the coding loss, image restoration loss and contrastive learning loss are positive values, and the weight of the message decoding loss is negative;

[0023] The coding loss is used to represent the pixel-level difference value and image loss value respectively corresponding to the target ciphertext feature vector extracted by the encoder and the ciphertext image, and the target carrier feature vector extracted by the encoder and the carrier image;

[0024] The image restoration loss is used to represent the pixel-level difference value between the loss carrier image output by the decoder and the carrier image;

[0025] The message decoding loss is used to represent the difference between the restored ciphertext image output by the decoder and the ciphertext image input to the encoder, calculated based on the mean square error;

[0026] The contrastive learning loss is used to represent the contrastive learning loss value between the encoder and the decoder obtained based on a joint contrastive learning unit; wherein the joint contrastive learning unit includes a connected encoding contrastive learning unit and a decoding contrastive learning unit.

[0027] In some embodiments of the present application, extracting information from the ciphertext image to obtain ciphertext text data corresponding to the restored ciphertext image includes:

[0028] The restored ciphertext image is input into a preset decoding optimization module based on an attention mechanism and a convolutional layer, so that the decoding optimization module outputs ciphertext text data corresponding to the restored ciphertext image.

[0029] Another aspect of the present application provides a cross-modal steganography device, comprising:

[0030] An encoding module, used for inputting the carrier image and the ciphertext image corresponding to the ciphertext text data into a preset encoder respectively, so that the learnable wavelet transform network and the reversible neural network in the encoder respectively perform forward feature extraction on the carrier image and the ciphertext image to obtain a target carrier feature vector corresponding to the carrier image and a target ciphertext feature vector corresponding to the ciphertext image;

[0031] The steganography module is used to generate ciphertext cross-modal steganography result data corresponding to the ciphertext text data based on the target carrier feature vector and the target ciphertext feature vector for network transmission.

[0032] A third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the cross-modal steganography method when executing the computer program.

[0033] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the cross-modal steganography method when executed by a processor.

[0034] A fifth aspect of the present application provides a computer program product, comprising a computer program and / or instructions, which implement the cross-modal steganography method when executed by a processor.

[0035] The cross-modal steganography method provided by the present application, by respectively inputting a carrier image and a ciphertext image corresponding to ciphertext text data into a preset encoder, so that a learnable wavelet transform network and a reversible neural network in the encoder respectively perform forward feature extraction on the carrier image and the ciphertext image to obtain a target carrier feature vector corresponding to the carrier image and a target ciphertext feature vector corresponding to the ciphertext image; based on the target carrier feature vector and the target ciphertext feature vector, ciphertext cross-modal steganography result data corresponding to the ciphertext text data is generated for network transmission, and the ciphertext text data of the text modality can be embedded into the carrier image of the image modality through a deep learning algorithm to realize multi-modal steganography technology, and by combining a reversible neural network with an end-to-end network, the concealment and robustness of the ciphertext steganography can be effectively improved, thereby not only improving the concealment of the ciphertext text data hiding, but also enhancing its resistance to various attacks, thereby realizing more secure and reliable information transmission.

[0036] Additional advantages, purposes, and features of the present application will be partially described in the following description, and will become partially apparent to those skilled in the art after studying the following, or may be learned from the practice of the present application. The purposes and other advantages of the present application can be achieved and obtained by the structures specifically pointed out in the specification and the drawings.

[0037] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0039] Figure 1 Schematic diagram of a first flow chart of a cross-modal steganography method in an embodiment of the present application.

[0040] Figure 2 Schematic diagram of a second flow chart of a cross-modal steganography method in an embodiment of the present application.

[0041] Figure 3 Schematic diagram of the architecture of an encoder in a cross-modal steganography method in an embodiment of the present application.

[0042] Figure 4 Schematic diagram of a third process of the cross-modal steganography method in one embodiment of the present application.

[0043] Figure 5 Schematic diagram of the architecture of a decoder in a cross-modal steganography method in an embodiment of the present application.

[0044] Figure 6 Schematic diagram of a fourth flow chart of the cross-modal steganography method in one embodiment of the present application.

[0045] Figure 7 Schematic diagram of the architecture of a learnable wavelet transform network in one embodiment of the present application.

[0046] FIG8( a ) is a schematic diagram showing an example of the architecture of a forward transfer network corresponding to the network architecture of a deformable attention reversible neural network DA-INN in one embodiment of the present application.

[0047] FIG8( b ) is a schematic diagram showing an example of the architecture of a reverse transfer network corresponding to the network architecture of the deformable attention reversible neural network DA-INN in one embodiment of the present application.

[0048] Fig. 9 A schematic diagram showing an example of the architecture of a deformable attention dense block DA-dense block in one embodiment of the present application.

[0049] Fig.10 Schematic diagram of the process of a cross-modal steganography method in an application example of the present application.

[0050] Fig.11This is a schematic diagram of noise results formed after noise simulation is performed on an encrypted image without noise attack (Identity) using some noise types in an application example of the present application.

[0051] Fig.12 Schematic diagram of the architecture of a joint contrastive learning unit in an application example of the present application.

[0052] Fig.13 Schematic diagram of the first structure of a cross-modal steganography device in one embodiment of the present application.

[0053] Fig.14 Schematic diagram of the second structure of the cross-modal steganography device in one embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the implementation modes and the accompanying drawings. Here, the illustrative implementation modes and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.

[0055] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, only the structures and / or processing steps closely related to the scheme according to the present application are shown in the accompanying drawings, while other details that are not very relevant to the present application are omitted.

[0056] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0057] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0058] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0059] Existing deep learning methods are divided into end-to-end methods and methods using invertible neural networks (INNs).

[0060] Typical end-to-end methods implement watermark embedding and recovery through end-to-end training. More appropriate constraints or optimization measures for such methods can often achieve stronger concealment and robustness. Some works use adversarial networks to optimize the encoding and decoding capabilities of the model. In addition, some works focus on watermarks in specific scenarios, such as decoding messages from wild videos, screen capture or multi-source image detection, and embedding watermarks into images of arbitrary resolution. They use different optimization methods to improve the performance of the model. However, in such methods, the decoded output is obtained by approximate likelihood inference rather than exact calculation, which makes the robustness of watermark extraction fragile. Usually, it is necessary to sacrifice imperceptibility in exchange for robustness. Therefore, the robustness and concealment of such methods are often unsatisfactory.

[0061] The reversible neural network INN is the key framework of the normalized flow model. Due to the characteristic of maintaining information losslessness, INN has become a valuable choice for watermark steganography tasks. In most neural network-based methods, such as the multi-scenario multi-task learning model HiNet, the combined reversible and irreversible mechanism CIN, and a novel invisible robust watermarking framework IRWArt, the encoding and decoding process uses a shared INN, which can extract watermarks more accurately compared to end-to-end methods. For the robustness problem, neural network-based methods are effective. They use a shared reversible neural network for watermark embedding and extraction, and decoding can be regarded as the inverse process of encoding, so they can robustly extract the watermark in an accurate form. However, they also have inherent limitations. First, it is difficult for digital watermarking methods based on neural networks to maintain robustness to complex noise (such as affine, rotation, etc.). In addition, there is a lack of dedicated optimization strategies to improve the encoder-decoder capabilities of the model. Therefore, there is still a lot of room for enhancing robustness and imperceptibility, especially when dealing with complex noise. Among them, INN (Invertible Neural Network) is a special neural network architecture, which is unique in that its forward propagation and back propagation processes are reversible. This means that through the same network structure and parameters, the input of the network can be completely restored from the output, and vice versa. This feature makes INN perform well in tasks that require accurate restoration of the input.

[0062] Among them, IRWArt is a novel invisible robust watermarking framework. In this architecture, watermark embedding and recovery are regarded as an inverse problem of a pair of image transformations, and can be achieved by implementing an invertible neural network INN through a reverse process and a forward process. In order to obtain high visual quality, the watermark is embedded in the high-frequency domain with minimal impact on the artwork, and the image reconstruction is supervised using the depth perception loss consistent with the human visual system HVS. At the same time, a quality enhancement processing module is constructed for distortions that may be caused by plagiarism.

[0063] CIN consists of a reversible part and an irreversible part, where the reversible part achieves high concealment and the irreversible part enhances robustness against strong noise attacks. In the reversible part, the diffusion and extraction module DEM and the fusion and splitting module FSM are used to embed and extract watermarks in a symmetrical manner. In the irreversible part, the attention-based non-reversible module NIAM and the noise-specific selection module NSM are introduced to solve the asymmetric extraction problem under strong noise attacks.

[0064] However, the disadvantage of existing watermark steganography technology is that it sacrifices the concealment of the watermark while ensuring robustness. Traditional watermark steganography methods usually embed the watermark into the frequency domain of the image, but when subjected to noise attacks, the watermark will be destroyed and cannot be restored normally. In recent years, deep learning methods have made certain progress in the field of watermark steganography, but these methods often find it difficult to achieve satisfactory results in terms of concealment and robustness. Especially when the encoding ability is too strong, it becomes difficult to ensure the robustness of decoding, and robustness is the most important for watermarking methods. In addition, although the watermark steganography method based on the reversible neural network INN achieves concealment while maintaining high robustness under most noises, it has poor robustness under complex noise (such as affine transformation, etc.), and lacks special optimization strategies to improve the encoding-decoding ability of the model. Therefore, the existing blind watermarking technology still has a lot of room for improvement in improving robustness and concealment.

[0065] Based on this, in order to realize cross-modal steganography of ciphertext and improve the concealment and robustness of ciphertext steganography, the embodiments of the present application respectively provide a cross-modal steganography method, a cross-modal steganography device for executing the cross-modal steganography method, a physical device, a computer-readable storage and a computer program product, and embed the ciphertext information (i.e., ciphertext text data) into the carrier image through a specific algorithm, while not significantly changing the visual effect of the image, ensuring that the embedded ciphertext text can withstand the test of various environmental changes or malicious analysis and maintain its recoverability. By combining the two different modal data of image and text, not only can the concealment of information hiding be improved, but also its resistance to various attacks can be enhanced, achieving more secure and highly robust information transmission.

[0066] The details are described in detail through the following examples.

[0067] Based on this, the embodiment of the present application provides a cross-modal steganography method that can be implemented by a cross-modal steganography device, see Figure 1 , the cross-modal steganography method specifically includes the following contents:

[0068] Step 100: Input the carrier image and the ciphertext image corresponding to the ciphertext text data into a preset encoder respectively, so that the learnable wavelet transform network and the reversible neural network in the encoder perform forward feature extraction on the carrier image and the ciphertext image respectively, so as to obtain a target carrier feature vector corresponding to the carrier image and a target ciphertext feature vector corresponding to the ciphertext image.

[0069] In one or more embodiments of the present application, a carrier image refers to image data that serves as a carrier for hiding ciphertext text data. Ciphertext text data refers to text data containing ciphertext information (i.e., secret information). The ciphertext image refers to image data formed after the ciphertext text data is converted into a picture format. In order to further improve the robustness and reliability of the encoder for forward feature extraction, the image size of the ciphertext image can be set to be the same as that of the carrier image.

[0070] It can be understood that the learnable wavelet transform network LWN (Deform-Attention Invertible Neural Network) is a learnable wavelet network with two processes, forward and reverse. When the forward process is executed, the model architecture of the learnable wavelet transform network can be used as the forward transfer network corresponding to the learnable wavelet transform network. When the reverse process is executed, the model architecture of the learnable wavelet transform network can be used as the reverse transfer network corresponding to the learnable wavelet transform network. The forward feature extraction in step 100 refers to the feature extraction process that can be performed using the forward transfer network corresponding to the learnable wavelet transform network LWN. The learnable wavelet transform network LWN can adaptively embed the watermark into a high-frequency area to which the human eye is insensitive to the watermark, thereby improving the invisibility and noise resistance of the watermark. It can be implemented based on the convolutional neural network CNN simulating the Haar transform, which is described in detail in the following embodiments.

[0071] In one or more embodiments of the present application, Haar transform (Haar Wavelet) refers to Haar wavelet transform, which is a basic form of discrete wavelet transform DWT, and is simple, fast and easy to implement. It helps extract different features in the signal by decomposing the signal into high-frequency and low-frequency components, and is widely used in signal processing, image compression and multi-resolution analysis, and is particularly suitable for capturing edges or mutation areas in images.

[0072] In addition, in order to further improve the effectiveness and reliability of cross-modal steganography of ciphertext, the currently received ciphertext text data and the carrier image can be preprocessed separately before step 100. Among them, the ciphertext text data can be watermarked and diffused to convert it from the original K-bit characters into an image format with the same size as the carrier image, so as to facilitate subsequent steganography operations.

[0073] In a specific example of preprocessing the carrier image, the carrier image may be first scaled to a uniform size, thereby standardizing the image size and facilitating subsequent operations; and then the carrier image may be divided into batches, thereby effectively utilizing the hardware performance of the server.

[0074] In a specific example of preprocessing ciphertext data, the ciphertext information is first copied into three copies, representing the three channels of R (red), G (green) and B (blue) of the image, and then the length of the ciphertext information is expanded through the fully connected layer. After passing through the fully connected layer, the ciphertext information becomes a string of length H*W (H and W represent the height and width of the image, respectively). Then, the expanded ciphertext information representing each channel is reshaped into an image of size (H, W, 1) (1 represents a single channel, and a color image usually has 3 channels). Finally, the three are spliced ​​together in the channel dimension to become a color image of size (H, W, 3) (hereinafter referred to as a ciphertext image) for subsequent steganographic operations.

[0075] Step 200: Generate ciphertext cross-modal steganographic result data corresponding to the ciphertext text data based on the target carrier feature vector and the target ciphertext feature vector for network transmission.

[0076] It can be understood that the ciphertext cross-modal steganography result data is the encrypted image corresponding to the ciphertext text data.

[0077] In step 200, the target carrier feature vector and the target ciphertext feature vector can be simply added together by pixel value to implement the steganographic operation. Specifically, since the target ciphertext feature vector has already achieved the lowest impact on the image generation method, it is sufficient to directly add the values. It can be understood that the target carrier feature vector refers to the image feature data obtained after forward feature extraction of the carrier image; the target ciphertext feature vector refers to the image feature data obtained after forward feature extraction of the ciphertext image.

[0078] From the above description, it can be seen that the cross-modal steganography method provided in the embodiment of the present application can embed the ciphertext text data of the text modality into the carrier image of the image modality through a deep learning algorithm, thereby realizing multimodal steganography technology. By combining a reversible neural network with an end-to-end network, the concealment and robustness of the ciphertext steganography can be effectively improved, thereby not only improving the concealment of the ciphertext text data, but also enhancing its resistance to various attacks, thereby realizing more secure and reliable information transmission.

[0079] In order to further realize cross-modal decoding of stego ciphertext and improve the accuracy and reliability of stego ciphertext decoding, in a cross-modal steganography method provided in an embodiment of the present application, see Figure 2, the cross-modal steganography method further specifically includes the following contents:

[0080] Step 300: Receive a noise image and extract a noise carrier feature vector and a noise ciphertext feature vector corresponding to the noise image, wherein the noise image is formed after the ciphertext cross-modal steganography result data is transmitted over the network, and the ciphertext cross-modal steganography result data is pre-generated based on a target carrier feature vector corresponding to a carrier image output by the encoder and a target ciphertext feature vector corresponding to a ciphertext image of ciphertext text data.

[0081] Step 400: Input the noise carrier feature vector and the noise ciphertext feature vector into the decoder corresponding to the encoder respectively, so that the learnable wavelet transform network and the reversible neural network in the decoder respectively perform inverse feature extraction on the noise carrier feature vector and the noise ciphertext feature vector, so as to obtain the loss carrier image corresponding to the noise carrier feature vector and the restored ciphertext image corresponding to the noise ciphertext feature vector; wherein the loss carrier image corresponding to the noise carrier feature vector is different from the carrier image used to generate the noise image to which the noise carrier feature vector belongs.

[0082] In one or more embodiments of the present application, the restored ciphertext image corresponds to the ciphertext image of the ciphertext text data, is the restored data corresponding to the ciphertext image, and can be referred to as the restored image, and the ciphertext image can be referred to as the original image.

[0083] Step 500: extract information from the ciphertext image to obtain ciphertext text data corresponding to the restored ciphertext image.

[0084] It can be understood that the steps 300 to 500 can be performed independently of the steps 100 to 200, or can be performed after the step 200, and can be specifically configured according to actual needs.

[0085] For example: in an actual application scenario, a cross-modal steganography device can be respectively set in different client devices A and client devices B. The cross-modal steganography device in client device A executes steps 100 to 200, and then the obtained ciphertext cross-modal steganography result data is transmitted in the network and received by client device B, and then the cross-modal steganography device in client device B executes steps 300 to 500.

[0086] For another example: in the testing phase of the cross-modal steganography method, the same server C can be used to execute steps 100 to 500. In order to improve the effectiveness of the test, it is necessary to perform noise simulation of the transmission of the ciphertext cross-modal steganography result data in the network between steps 200 and 300.

[0087] From the above description, it can be seen that the cross-modal steganography method provided in the embodiment of the present application can use a decoder that is also composed of the model parameters of the learnable wavelet transform network and the reversible neural network in the encoder to realize the decoding of the stego ciphertext, which can effectively improve the efficiency of cross-modal decoding of the stego ciphertext, and can effectively improve the accuracy and reliability of cross-modal decoding of the stego ciphertext.

[0088] In order to further enhance the imperceptibility and noise resistance of the watermark corresponding to the ciphertext data, in a cross-modal steganography method provided in an embodiment of the present application, see Figure 3 The encoder includes a forward transfer network corresponding to a learnable wavelet transform network, a forward transfer network corresponding to a reversible neural network, and a reverse transfer network corresponding to the learnable wavelet transform network, which are connected in sequence.

[0089] For the corresponding Figure 4 , step 100 of the cross-modal steganography method specifically includes the following contents:

[0090] Step 110: Input the carrier image and the ciphertext image corresponding to the ciphertext text data into a preset encoder respectively, so that the forward transfer network corresponding to the learnable wavelet transform network in the encoder performs down-sampling feature extraction for multiple different frequency domains on the carrier image and the ciphertext image respectively, and outputs the first multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively; then, the forward transfer network corresponding to the reversible neural network in the encoder performs feature extraction for target operation on the first multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively based on the self-attention mechanism, and outputs the second multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively; then, the reverse transfer network corresponding to the learnable wavelet transform network in the encoder performs up-sampling feature extraction for the same frequency domain on the second multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively, and outputs the target carrier feature vector corresponding to the carrier image and the target ciphertext feature vector corresponding to the ciphertext image.

[0091] In order to further realize cross-modal decoding of stego ciphertext and improve the accuracy and reliability of stego ciphertext decoding, in a cross-modal steganography method provided in an embodiment of the present application, see Figure 5 , the decoder comprises: a forward transfer network corresponding to the learnable wavelet transform network, a reverse transfer network corresponding to the reversible neural network and a reverse transfer network corresponding to the learnable wavelet transform network connected in sequence;

[0092] For the corresponding Figure 6 , step 300 in the cross-modal steganography method further specifically includes the following contents:

[0093] Step 310: Input the noise carrier feature vector and the noise ciphertext feature vector into the decoder respectively, so that the forward transfer network of the learnable wavelet transform network in the decoder performs down-sampling feature extraction for multiple different frequency domains on the noise carrier feature vector and the noise ciphertext feature vector respectively, and outputs the third multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively; then the reverse transfer network corresponding to the reversible neural network in the encoder performs feature restoration for the target operation on the third multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively based on the self-attention mechanism, and outputs the fourth multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively; then the reverse transfer network corresponding to the learnable wavelet transform network in the encoder performs up-sampling feature extraction for the same frequency domain on the fourth multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively, and outputs the loss carrier image corresponding to the noise ciphertext feature vector and the restored ciphertext image corresponding to the noise ciphertext feature vector.

[0094] In order to further improve the robustness and stealth of the encoder and decoder, in a cross-modal steganography method provided in an embodiment of the present application, see Figure 4 , the cross-modal steganography method further includes the following contents before step 100:

[0095] Step 010: training the learnable wavelet transform network and the reversible neural network based on a preset target loss function and training data, so that the learnable wavelet transform network and the reversible neural network constitute the encoder and the decoder respectively; wherein the training data is composed of various data samples, and each of the data samples contains a carrier image, a ciphertext image corresponding to the ciphertext data, and a noise image of the same size; the ciphertext image corresponding to the ciphertext data is generated by pre-processing the ciphertext data with watermark diffusion; the noise image is generated by pre-processing the ciphertext cross-modal steganography result data of the ciphertext data corresponding to the noise image with noise attack simulation based on multiple preset noise types;

[0096] The learnable wavelet transform network is composed of a plurality of convolutional neural networks corresponding one-to-one to different frequency domains and used to simulate Haar wavelet transform.

[0097] Specifically, the learnable wavelet transform network LWN has two processes, forward and reverse. This application uses a convolutional neural network to simulate the Haar transform process and implements a wavelet transform network that can adjust its own parameters. Here, this application selects the typical architecture of wavelet transform, Haar transform. The specific network architecture of LWN is as follows: Figure 7 As shown. A, V, H and D refer to the feature information in the four frequency domains of approximation, horizontal, vertical and diagonal respectively. Here, the application reduces the H and W of the ciphertext image and the carrier image by half, but splices the four feature information together to obtain a feature vector of size (H / 2, W / 2, 12). The core components of the learnable wavelet transform network LWN are Figure 7 The convolutional neural network (which can be written as DWT-CNN) used to simulate the Haar wavelet transform in the CNN is composed of a set of 2D convolution operations, which is designed to simulate the discrete wavelet transform. The difference between the learnable wavelet transform network LWN and the traditional discrete wavelet transform DWT is that the filter kernel of the traditional discrete wavelet transform DWT is fixed. In the learnable wavelet transform network LWN, the DWT is simulated by CNN. As a training process for watermark embedding and extraction, its convolution kernel is continuously learned and updated to select more appropriate high-frequency features to embed the watermark, thereby improving imperceptibility. When the input ciphertext image and carrier image are input into the forward process of DWT-CNN, it will be processed by four different convolutional neural networks DWT-CNN used to simulate the Haar wavelet transform, and finally a feature vector consisting of four different frequency domains will be obtained. Among them, I m I represents the image input to the learnable wavelet transform network, h is the height of the image input to the learnable wavelet transform network, and w is the width of the image input to the learnable wavelet transform network; out K represents the first multi-frequency domain feature vector output by the learnable wavelet transform network. w [i] represents a collection of convolution kernels (i.e., wavelet filters) used to simulate discrete wavelet transform (DWT). Kw[i] is learned and updated through a convolutional neural network (CNN) to dynamically optimize feature extraction in different frequency domains. This dynamic learning process is different from the traditional DWT where the filter is fixed.

[0098] The reversible neural network is composed of multiple layers of deformable convolutional layers, and each layer of the deformable convolutional layers is associated based on a target operation, and the target operation includes at least one of addition, subtraction, multiplication and division;

[0099] The deformable convolution layer is provided with a deformable attention dense block for executing a preset operation function, and the deformable attention dense block includes a connected self-attention block and a deformable dense block; wherein the deformable dense block includes: a deformable convolution layer and four convolution layers densely connected to the deformable convolution layer in sequence.

[0100] Specifically, the reversible neural network can adopt the deformable attention reversible neural network DA-INN (Deform-Attention Invertible Neural Network), which combines reversibility and deformable attention mechanism, and can further enhance the robustness of the network in dealing with complex noise attacks. The deformable attention reversible neural network DA-INN is a special reversible neural network architecture. It adopts deformable convolution to solve the low robustness of the reversible neural network in the face of deformation attacks and adopts the self-attention mechanism to better extract feature information. DA-INN consists of 16 layers of the same network architecture. In order to achieve complete reversibility of the operation, this application needs to adopt addition, subtraction, multiplication and division, which are easy to restore, in the feature extraction process within each layer. The forward transfer network corresponding to the network architecture of DA-INN is shown in Figure 8(a), and the reverse transfer network corresponding to the network architecture of DA-INN is shown in Figure 8(b).

[0101] The operation of each layer in DA-INN is composed of three feature extraction operations: F(), G(), and H(). The reversible feature extraction network is realized by addition, subtraction, multiplication, and division. Since the forward operation is parameterized by a reversible function, the tensor can be accurately restored by using shared parameters in the reverse process. Among them, F() represents the operation of processing the input data, which is mainly used to adjust certain information of the input image or feature; G() represents the nonlinear transformation of the input image or watermark feature, which is used to calculate a weight value, and the input is weighted point by point through an exponential function. Its main purpose is to adjust the scale of specific features; H() represents the compensatory operation of the input image or watermark feature. As an additional feature correction term, its role is to assist feature recovery and enhance the expression of specific frequencies. In general, F() is used for preliminary processing, G() adjusts the scale distribution of features, and H() is responsible for compensating and correcting features. The three work together to form the core functional module in DA-INN, supporting the embedding and extraction process of watermarks.

[0102] This application is for and Respectively represent the watermark input into the reversible neural network (i.e., the first multi-frequency domain feature vector corresponding to each ciphertext image) and the first multi-frequency domain feature vector corresponding to the carrier image; and the corresponding and They respectively represent the carrier tensor (i.e., the second multi-frequency domain feature vector corresponding to the carrier image) and the ciphertext information tensor (i.e., the second multi-frequency domain feature vector corresponding to the ciphertext image) after passing through the current layer.

[0103] Each operation function F(), G(), and H() in DA-INN is implemented through a dense block structure, which integrates deformable convolution and self-attention mechanism, called deformable attention dense block DA-denseblock. Fig. 9 , the deformable attention dense block DA-dense block of the present application consists of two parts: a self-attention block (SE-Block) and a deformable dense block (Deformable Dense Block). First, the feature map (i.e., the first multi-frequency domain feature vector) is input into the self-attention block to capture the importance of the channel. Then, the feature map output by the self-attention block is input into the deformable dense block (Deformable Dense Block), which contains five convolutional layers. The first of the five convolutional layers is a deformable convolution (DCN, Deformable Convolution), which densely connects four conventional convolutional layers, namely the first convolutional layer (Convolution Layer-1), the second convolutional layer (Convolution Layer-2), the third convolutional layer (Convolution Layer-3) and the fourth convolutional layer (Convolution Layer-4), and the feature channels are continuously expanded. The last convolutional layer reduces the feature map with a large number of channels back to the original number of channels. Among them, the attention module in the DA-denseblock assigns different attention weights to different channels, effectively processing the feature maps distinguished as high-frequency and low-frequency by LWN. In addition, the adaptive sampling positions of deformable dense blocks can enhance the robustness of INN in dealing with spatial transformations.

[0104] Among them, Deformable Convolution DCN (Deformable Convolutional Networks) is a method to enhance the capabilities of Convolutional Neural Networks (CNN). Traditional convolution operations may be limited by fixed sampling positions when processing objects with complex geometric deformations or different shapes, resulting in unsatisfactory feature extraction. DCN can effectively improve the performance of CNN in processing geometric deformations and complex scenes by introducing Deformable Convolution and Deformable RoI Pooling. SE-Attention (Squeeze-and-Excitation Attention) is a lightweight and efficient attention mechanism introduced in Squeeze-and-Excitation Networks (SE-Net). SE-Attention significantly improves the performance of Convolutional Neural Networks (CNN) by adaptively readjusting the weights of each channel, especially in tasks such as image classification, object detection and semantic segmentation.

[0105] In order to avoid restoring the original carrier image as much as possible to further prevent the model from being cracked, in a cross-modal steganography method provided in an embodiment of the present application, the target loss function in the cross-modal steganography method specifically includes the following contents:

[0106] The objective loss function is composed of the encoding loss L en , image restoration loss L re , message decoding loss L de and contrastive learning loss L con And the coding loss L en , image restoration loss L re , message decoding loss L de and contrastive learning loss L con The corresponding weights constitute the encoding loss L en , image restoration loss L re and contrastive learning loss L con The weight is positive, and the message decoding loss L de The weight of is negative;

[0107] The coding loss L en Used to represent the pixel-level difference value and image loss value respectively corresponding to the target ciphertext feature vector extracted by the encoder and the ciphertext image, and the target carrier feature vector extracted by the encoder and the carrier image;

[0108] The image restoration loss L reused to represent the pixel-level difference value between the loss carrier image output by the decoder and the carrier image;

[0109] The message decoding loss L de used to represent the difference value between the restored ciphertext image output by the decoder and the ciphertext image input to the encoder, calculated based on the mean square error;

[0110] The contrastive learning loss L con Used to represent the contrastive learning loss value between the encoder and the decoder obtained based on a joint contrastive learning unit; wherein the joint contrastive learning unit includes a connected encoding contrastive learning unit and a decoding contrastive learning unit.

[0111] The expression of the total loss function is as follows:

[0112] L total =λ1L en +λ2L re +λ3L de +λ4L con

[0113] Among them, the coding loss L en The function consists of two parts: L2 loss, which is used to evaluate the pixel-level difference in the image, and Lpips\cite, which measures the image loss from the perspective of human visual perception. λ1 to λ4 are the encoding losses L en , image restoration loss L re , message decoding loss L de and contrastive learning loss L con The corresponding weights.

[0114] The joint contrastive learning unit is used to perform a method for comprehensively evaluating the concealment and robustness of steganography during the training phase. It is a joint contrastive learning JCL (Joint Contrast Learning) method designed for watermarking tasks based on SimSiam. Contrast learning, as a self-supervised, label-free training strategy, is very suitable for the training process of blind watermarking methods. Traditional digital watermarking technology uses L2 loss or mean square error (MSE) loss for evaluation and training, which results in evaluation indicators based only on low-dimensional features. The JCL method evaluates the encoding and decoding performance of the model in a high-dimensional feature space. Joint contrastive learning JCL is a self-supervised contrastive learning strategy that can simultaneously optimize the encoding and decoding performance of watermarks in a high-dimensional feature space. SimSiam (Simple Siamese Network for Representation Learning) is a self-supervised learning method that focuses on representation learning of unlabeled data and aims to simplify the training process of contrastive learning. This method is used to solve the problem of how to effectively train neural networks to obtain useful feature representations without labels.

[0115] In order to further improve the robustness of DA-INN, in a cross-modal steganography method provided in an embodiment of the present application, see Figure 6 , step 500 in the cross-modal steganography method specifically includes the following contents:

[0116] Step 510: Input the restored ciphertext image into a preset decoding optimization module based on an attention mechanism and a convolutional layer, so that the decoding optimization module outputs ciphertext text data corresponding to the restored ciphertext image.

[0117] Specifically, the end-to-end watermarking method uses a decoder to restore information, which makes it highly robust to various attacks. However, the INN-based method only relies on the inherent reversible properties of the INN itself, resulting in low robustness in the face of complex noise. In order to improve the decoding ability of the INN-based method, the decoding optimization module DOM (Decoding Optimization Module) designed in this application, which is based on the attention mechanism and convolutional layer, is used to decode the ciphertext information from the restored ciphertext image. The decoding optimization module DOM combines the attention mechanism and the convolutional layer, which can improve the decoding ability of the model under complex noise during the decoding process.

[0118] Decoding optimization module DOM operation DOM (I RS ) can be expressed as follows:

[0119] DOM(I RS )=L M (SE(conv(I RS ))

[0120] Among them, L M is a fully connected layer (FC layer), SE() represents the attention mechanism, and conv() is a convolutional layer. It is worth noting that the DOM established in this application is applicable to all restored ciphertext images I RS , that is, it can not only effectively resist complex noise, but also improve the robustness against simple noise attacks. Therefore, this application combines the advantages of the end-to-end method in resisting noise attacks with the advantages of the precise calculation of the reversible neural network, and can greatly improve the robustness of the model through DOM.

[0121] In order to further illustrate the above embodiment, the present application also provides an application example of a cross-modal steganography method, specifically a robust image watermark steganography technology (Imperceptibility of the Watermark and Robustness against Noise attacks) based on cross-modal data fusion. Cross-modal data fusion refers to embedding ciphertext information into a carrier image through a specific algorithm, while not significantly changing the visual effect of the image, ensuring that the embedded ciphertext text can withstand the test of various environmental changes or malicious analysis and maintain its recoverability. By combining the two different modal data of image and text, not only can the concealment of information hiding be improved, but also its resistance to various attacks can be enhanced, thereby achieving more secure and highly robust information transmission. First, in order to enhance the imperceptibility of the watermark, the present application designs a learnable wavelet network (LWN) to adaptively embed the watermark into a high-frequency area where the human eye is insensitive to the watermark; secondly, the present application proposes a reversible neural network (DA-INN) based on deformation attention, which has the advantages of computational regression and combines the deformation-attention mechanism to enhance the model's anti-noise ability. In order to further improve the robustness of DA-INN, this application introduces DOM to make up for its shortcomings in optimization. In addition, this application establishes a joint contrast learning mechanism (JCL) based on Simsiam, through which the similarity between the learning image and the watermark image, as well as the similarity between the watermark and the decoded watermark, is compared and learned, thereby further improving the encoding and decoding capabilities of the model, and further enhancing the imperceptibility and robustness of the watermark. The so-called cross-modality is to combine the ciphertext whose original mode is text information with the stego carrier whose mode is an image through steganography. Under the premise of ensuring that the ciphertext information can still be successfully extracted after facing various noise attacks together with the stego carrier, the concealment of the fusion of the two is improved, so as to achieve the embedding of the ciphertext information when it cannot be seen that the carrier has been written with information.

[0122] See also Fig.10 , the application examples of the cross-modal steganography method specifically include the following contents:

[0123] Step 1: Preprocess the carrier image. Perform a watermark diffusion operation on the ciphertext information to convert it from the original K-bit characters to an image format with the same size as the carrier image, which is convenient for subsequent steganographic operations.

[0124] Specifically, a watermark diffusion operation is performed on the ciphertext information to convert it from the original K-bit characters into an image format with the same size as the carrier image, which facilitates subsequent steganographic operations.

[0125] Among them, the preprocessing operation of the carrier image includes:

[0126] (1) Scale the carrier image to a uniform size, thereby standardizing the image size and facilitating subsequent operations;

[0127] (2) Divide the carrier images into batches, thereby effectively utilizing the hardware performance of the server.

[0128] The watermark diffusion operation for the ciphertext information is as follows:

[0129] First, the ciphertext information is copied into three copies, representing the R, G, and B channels of the image respectively. Then, the length of the ciphertext information is expanded through the fully connected layer. After the fully connected layer, the ciphertext information becomes a string of length H*W (H and W represent the height and width of the image). Then, the expanded ciphertext information representing each channel is reshaped into an image of size (H, W, 1) (1 represents a single channel, and color images usually have 3 channels). Finally, the three are spliced ​​together in the channel dimension to become a color image of size (H, W, 3) (hereinafter referred to as the ciphertext image) for subsequent steganographic operations.

[0130] Step 2: Pass the ciphertext image and the carrier image through the forward process of the LWN network respectively to obtain the ciphertext image tensor and the carrier image tensor with different frequency domain information.

[0131] Specifically, the purpose of this step is to downsample the image size to (H / 2, W / 2) but the feature channels become 12, where every 3 feature channels form a group, representing LL, HL, LH, and HH four image features respectively.

[0132] Among them, the learnable wavelet transform network LWN has two processes, forward and reverse. This application uses a convolutional neural network to simulate the Haar transform process to realize a wavelet transform network that can adjust its own parameters. Here, this application selects the typical architecture of wavelet transform, Haar transform. For the specific network architecture of LWN, see Figure 7 .

[0133] A, V, H and D refer to the feature information in the four frequency domains of approximation, horizontal, vertical and diagonal respectively. Here, the application reduces the H and W of the ciphertext image and the carrier image by half, but splices the four feature information together to obtain a feature vector of size (H / 2, W / 2, 12). The core components of the learnable wavelet transform network LWN are Figure 7 The convolutional neural network (which can be written as DWT-CNN) used to simulate the Haar wavelet transform in the paper is composed of a set of 2D convolution operations, which is designed to simulate the discrete wavelet transform. The difference between the learnable wavelet transform network LWN and the traditional discrete wavelet transform DWT is that the filter kernel of the traditional discrete wavelet transform DWT is fixed. In the learnable wavelet transform network LWN, the DWT is simulated by CNN. As a training process for watermark embedding and extraction, its convolution kernel is constantly learned and updated, so as to select more appropriate high-frequency features to embed the watermark, thereby improving the imperceptibility. When the input ciphertext image and carrier image are input into the forward process of DWT-CNN, it will be processed by four different convolutional neural networks DWT-CNN used to simulate the Haar wavelet transform, and finally a feature vector consisting of four different frequency domains is obtained.

[0134] Step 3: The ciphertext image tensor after LWN and the carrier image tensor are input into the forward process of DA-INN for further feature extraction, so that the final ciphertext image tensor can be placed in the carrier image to be more concealed.

[0135] Referring to Figures 8(a) and 8(b), the Deformed Attention Reversible Neural Network DA-INN is a special reversible neural network architecture that uses deformable convolution to solve the low robustness of reversible neural networks in the face of deformation attacks and uses a self-attention mechanism to better extract feature information. DA-INN consists of 16 layers of the same network architecture. In order to achieve complete reversibility of the operation, this application needs to use addition, subtraction, multiplication and division, which are easy to restore, in the feature extraction process within each layer.

[0136] The operation of each layer in DA-INN is composed of three feature extraction operations: F(), G(), and H(), and the addition, subtraction, multiplication, and division methods are used to realize a reversible feature extraction network. Since the forward operation is parameterized by a reversible function, the tensor can be accurately restored using shared parameters in the reverse process.

[0137] This application is for and Respectively represent the watermark input into the reversible neural network (i.e., the first multi-frequency domain feature vector corresponding to each ciphertext image) and the first multi-frequency domain feature vector corresponding to the carrier image; and the corresponding and They respectively represent the carrier tensor (i.e., the second multi-frequency domain feature vector corresponding to the carrier image) and the ciphertext information tensor (i.e., the second multi-frequency domain feature vector corresponding to the ciphertext image) after passing through the current layer.

[0138] The forward operation process of DA-INN is as follows:

[0139]

[0140] Where ⊙ represents the Hadamard product.

[0141] The reverse operation process of DA-INN is as follows:

[0142]

[0143] Each operation function F(), G(), and H() in DA-INN is implemented through a dense block structure, which integrates deformable convolution and self-attention mechanism, called deformable attention dense block DA-denseblock. Fig. 9 , the deformable attention dense block DA-dense block of the present application consists of two parts: a self-attention block (SE-Block) and a deformable dense block (Deformable Dense Block). First, the feature map (i.e., the first multi-frequency domain feature vector) is input into the self-attention block to capture the importance of the channel. Then, the feature map output by the self-attention block is input into the deformable dense block (Deformable Dense Block), which contains five convolutional layers. The first of the five convolutional layers is a deformable convolution (DCN, Deformable Convolution), which is densely connected to four conventional convolutional layers, namely the first convolution layer (Convolution Layer-1), the second convolution layer (Convolution Layer-2), the third convolution layer (Convolution Layer-3) and the fourth convolution layer (Convolution Layer-4), and the feature channels are continuously expanded. The last convolution layer reduces the feature map with a large number of channels back to the original number of channels.

[0144] The attention module in the DA-dense block assigns different attention weights to different channels, effectively processing the feature maps distinguished as high-frequency and low-frequency by LWN. In addition, the adaptive sampling position of the deformable dense block enhances the robustness of INN in dealing with spatial transformations.

[0145] Step 4: The ciphertext image tensor and carrier image tensor after DA-INN further extracts image features are input into the reverse process of the LWN network, and the (H / 2, W / 2, 12) tensor is transformed back to the original image size of (H, W, 3) to obtain the final ciphertext image and carrier image.

[0146] This operation uses the same parameters as the LWN forward process to achieve lossless information transformation, transforming the 12-channel tensor back into a 3-channel image that is the same as the input image.

[0147] Step 5: Simply add the ciphertext image and the carrier image to achieve the steganographic operation. This is just a simple addition operation of the values ​​of the corresponding pixels. Since the ciphertext image has been generated in a way that has the lowest impact on the image, direct numerical addition is sufficient.

[0148] Step 6: The image is passed through a noise pool designed to simulate noise (i.e., noise) attacks that may occur in reality, so as to improve the robustness of the steganographic technology in the face of complex real-world environments.

[0149] In order to fully simulate the noise situation in the real world, this application finally selected the encrypted image without noise attack (Identity) (i.e., the ciphertext cross-modal steganography result data) and the following 12 types of noise, such as set N ALL :

[0150] N All ={Identity,GaussianNoise,GaussianBlur,Saltpepper,

[0151] Resize,Saturation,Hue,Contrast,Brightness,

[0152] Rotation,Affine,Dropout,Cropout}

[0153] Among them, GaussianNoise means adding random Gaussian noise to simulate random errors in signals; GaussianBlur means using Gaussian kernel to blur the image and reduce the clarity of the image; Saltpepper means adding random black and white noise to simulate interference during image transmission or storage; Resize means reducing or enlarging the image to test the robustness of the watermark to resolution changes; Saturation means adjusting the color saturation of the image to simulate the impact of image color changes on the watermark; Hue means modifying the hue of the image to test the robustness of the watermark to color shift; Contrast st means adjusting the brightness difference of the image to test the impact of contrast changes on the watermark; Brightness means adjusting the overall brightness of the image to test the impact of brightness changes on the watermark; Rotation means rotating the image at an angle to simulate the robustness of rotation interference to the watermark; Affine means a comprehensive attack including linear transformations such as translation, rotation, scaling, and shearing to test the performance of the watermark under complex geometric transformations; Dropout means randomly discarding some pixels in the watermark image to simulate the situation where image data is lost or blocked; Cropout means randomly cropping out part of the watermark image to simulate the situation where the image is partially intercepted.

[0154] The noise results of some noise types after noise simulation on the encrypted image without noise attack (Identity) are shown in Fig.11 , Cropout and Dropout are not in Fig.11 The reason is that these attacks involve replacing parts of the watermarked image with the corresponding parts of the original image, which has a great impact on the decoding of the watermark information but is difficult to detect visually.

[0155] Step 7: Copy the image attacked by noise and input it into the decoding process as redundant information.

[0156] Since the forward process of DA-INN has two input information and the forward and reverse processes share parameters, the present application uses the copied image attacked by noise as the second input to the reverse process in the reverse process.

[0157] Step 8: Put the two together through the forward process of LWN, the reverse process of DA-INN, and the reverse process of LWN in sequence to obtain the restored ciphertext image and carrier image.

[0158] The parameters of LWN and DA-INN in the reverse process are exactly the same as those in the forward process, and finally the ciphertext image and carrier image are restored.

[0159] Step 9: The restored ciphertext image is passed through DOM to extract the ciphertext information and obtain the final ciphertext information. In order to make the model more difficult to crack, the restored carrier image will be as far away from the original carrier image as possible.

[0160] The end-to-end watermarking method uses a decoder to recover information, which makes it highly robust against various attacks. However, the INN-based method only relies on the inherent reversible properties of the INN itself, resulting in low robustness in the face of complex noise. In order to improve the decoding capability of the INN-based method, the DOM module designed in this application is used to decode the ciphertext information from the restored ciphertext image. The operation of DOM can be expressed as follows:

[0161] DOM(I RS )=L M (SE(conv(I RS ))

[0162] Where L M is a fully connected layer (FC layer), SE() represents the attention mechanism, and conv() is a convolutional layer. It is worth noting that the DOM established in this application is applicable to all restored ciphertext images I RS , that is, it can not only effectively resist complex noise, but also improve the robustness against simple noise attacks. Therefore, this application combines the advantages of the end-to-end method in resisting noise attacks with the advantages of the precise calculation of the reversible neural network, and greatly improves the robustness of the model through DOM.

[0163] As for the restored carrier image, since the reversible neural network has the reversible characteristic, in order to prevent the plagiarist from cracking the model and obtaining the model of the reverse process so that the original image is restored, during the training process of the method, the weight of the image restoration loss function (image restoration loss) is set to a negative number, that is, the model will try not to restore the original image, and the final image will be far from the original image.

[0164] In particular, this application adopts a model optimization strategy combined with contrastive learning in the optimization strategy of model training and designs a rich loss function.

[0165] The overall loss function consists of four parts: encoding loss L en , image restoration loss L re , message decoding loss L de and contrastive learning loss L con These losses are designed to ensure that the encrypted image is close to the carrier image, that the ciphertext information matches the restored information, and that the restored image is distinct from the carrier image to prevent unauthorized reconstruction of the original image.

[0166] The expression of the total loss function is as follows:

[0167] L total =λ1L en +λ2L re +λ3L de +λ4L con

[0168] Coding loss L en The function consists of two parts: L2 loss, which is used to evaluate the pixel-level differences in the image, and Lpips\cite, which measures the image loss from the perspective of human visual perception. re The L2 loss between the support image and the restored image is used to evaluate the image reconstruction quality. de The function uses mean square error (MSE) to calculate the difference between the ciphertext information and the restored information. con The loss function uses the evaluation function in JCL during encoding and decoding.

[0169] The joint contrastive learning unit is used to perform a method for comprehensively evaluating the concealment and robustness of steganography during the training phase. It is a joint contrastive learning JCL (Joint Contrast Learning) method designed for watermarking tasks based on SimSiam. Contrast learning, as a self-supervised, label-free training strategy, is very suitable for the training process of blind watermarking methods. Traditional digital watermarking technology uses L2 loss or mean square error (MSE) loss for evaluation and training, which results in evaluation indicators based only on low-dimensional features. The JCL method evaluates the encoding and decoding performance of the model in a high-dimensional feature space. Fig.12 As shown in Figure 1, JCL consists of two parts: encoding contrast learning (JCL: encoding contrast) and decoding contrast learning (JCL: decoding contrast). The structure of the joint contrast learning unit JCL can be expressed as follows:

[0170] Fig.12 The JCL shown in the figure consists of two parts: encoding contrast learning and decoding process, which are used to compare the quality of encoding and decoding respectively. Fig.12 In the example, Ip indicates the image to be protected.

[0171] ), that is, the input image that is desired to be protected; Iw represents the watermarked image (WatermarkedImage), that is, the image data containing the steganographic watermark information; P-Feature represents the feature extracted from Ip; W-Feature represents the feature extracted from Iw; Z Ip Represents the projection result of P-Feature; Z IwRepresents the projection result of W-Feature; P Ip Indicates the prediction result of P-Feature; P Iw Represents the prediction result of W-Feature; NCS(Z Ip , P Iw ) indicates Z Ip and P Iw Negative Cosine Similarity (NCS) between Iw , P Ip ) indicates Z Iw and P Ip =The negative cosine similarity between them; E(Ip, Iw) represents the joint contrast loss of Ip and Iw, and combines the negative cosine similarity to evaluate the similarity of the projection and prediction results; Projection-Head represents the fully connected layer used to map features to a shared feature space to enhance the representation capability; Joint training represents joint training; Ms represents the original watermark (Secret Message), that is, the input watermark information that has not been steganographically processed; M RS Recovered SecretMessage represents the embedded and recovered watermark, i.e., the watermark data extracted from the watermark image Iw. S-Feature represents the feature extracted from Ms; RS-Feature represents the feature extracted from Ms. RS The features extracted from Z Ms Represents the projection result of S-Feature; Z MRS Represents the projection result of RS-Feature; P Ms Indicates the prediction result of S-Feature; P MRS Represents the prediction result of RS-Feature; NCS (Ms, P Ms ) indicates Ms and P Ms The negative cosine similarity between Ms , P MRS ) indicates Z Ms and P MRS The negative cosine similarity between RS ) indicates Ms and M RS The joint contrast loss is combined with negative cosine similarity to evaluate the similarity between watermark projection and prediction results.

[0172] The structure of the joint contrastive learning unit can be shown as follows:

[0173] Z T =ρ(T feature )

[0174] PT =γ(Z T )

[0175] Where T represents the corresponding tensor in the model of this application, T feature is the feature extracted from T. ρ() and γ() are the projection head and prediction head respectively. Z T and P T Respectively represent T feature The role of ρ() is to map features to a shared feature space, thereby enhancing the representation ability of the model. The role of γ() is to use the projected features to obtain the predicted representation of the image. In order to prevent the collapse solution, the projection Z T Does not participate in gradient updates. The i-th tensor for contrast enhancement must be input to the network to obtain the corresponding Z i and P i . Z i is the feature T of the i-th tensor through the projection head ρ() feature The projection result obtained after processing; P i It is the prediction representation generated by processing Zi through the prediction head γ().

[0176] It should also be noted that the robustness against affine transformation noise may be achieved by using other targeted network design structures such as STN networks, but compared with the DCN network, it does not solve the problems of CNN itself and cannot cope with more situations. In terms of optimization strategy, other contrastive learning design frameworks can also be used, but the purpose has not changed compared to this application, which is to use contrastive learning to obtain a more comprehensive model evaluation. In the implementation of INN, a variety of different designs can be used, but the purpose is to achieve complete recoverability to ensure that the watermark can be effectively steganographic and restored by the network during encoding and decoding.

[0177] That is to say, compared with the existing image steganography technology, the technology of the application example of this application realizes cross-modal data fusion, and the combination of text secret information and image carrier is obviously different from some existing technologies (such as IRWArt). And it achieves high concealment while maintaining robustness. INN-based methods such as (IRWArt and CIN) are difficult to achieve robustness when facing complex noises such as affine transformations. The method of the application example of this application combines DCN and designs DOM to achieve robustness in the face of stronger and more comprehensive noise. And it can still maintain high robustness even when the noise intensity exceeds the preset noise parameter by more than twice. The application example of this application adopts an optimization strategy based on contrastive learning, designs JCL, introduces the contrastive learning method into the watermark steganography technology, and effectively realizes a more effective watermark steganography model optimization strategy. As a technology with strong practicality, the computing resources and encoding and decoding speed required for digital watermarking technology are also important indicators. By adopting a method combining traditional watermarking algorithms with deep learning watermarking algorithms, the method of the application example of this application is also superior to other INN-based methods in terms of computing speed and required resources.

[0178] In summary, the present application can embed the ciphertext information of the text modality into the carrier image of the image modality through the algorithm of deep learning, realize the multimodal steganography technology, and due to the combination of reversible neural network and end-to-end network, the concealment and robustness of the model have reached an excellent level. Compared with other steganography techniques, the existing watermark steganography technology based on deep learning can be divided into three categories: end-to-end, based on generative adversarial network, and based on INN. Among them, both based on generative adversarial network and based on INN are to improve the concealment of watermarks, and the method based on INN has been proven to have excellent watermark concealment in recent years, but it is difficult to achieve sufficient robustness in the face of complex noise. Therefore, the present application combines the excellent robustness performance of the end-to-end algorithm in the algorithm based on INN, designs a decoding optimization module (DOM), and changes the ordinary convolution in INN to a deformed convolution and adds a self-attention mechanism to achieve the robustness of the information embedded in the INN while retaining the concealment of the information embedded in the INN. This application combines the traditional digital watermark steganography algorithm (based on Haar transform) with the deep learning digital watermark algorithm, designs the LWN network, and combines the effective separation method of high and low frequencies in the traditional watermark steganography algorithm into the deep learning watermark steganography algorithm, which not only improves the invisibility of the steganography technology, but also effectively reduces the model's demand for computing performance.

[0179] From the software level, the present application also provides a cross-modal steganography device for executing all or part of the cross-modal steganography method, see Fig.13 , the cross-modal steganography device specifically includes the following contents:

[0180] The encoding module 10 is used to input the carrier image and the ciphertext image corresponding to the ciphertext text data into a preset encoder respectively, so that the learnable wavelet transform network and the reversible neural network in the encoder respectively perform forward feature extraction on the carrier image and the ciphertext image to obtain a target carrier feature vector corresponding to the carrier image and a target ciphertext feature vector corresponding to the ciphertext image;

[0181] The steganography module 20 is used to generate ciphertext cross-modal steganography result data corresponding to the ciphertext text data based on the target carrier feature vector and the target ciphertext feature vector for network transmission.

[0182] The embodiment of the cross-modal steganography device provided in the present application can be specifically used to execute the processing flow of the embodiment of the cross-modal steganography method in the above-mentioned embodiment. Its functions are not repeated here, and reference can be made to the detailed description of the above-mentioned cross-modal steganography method embodiment.

[0183] The part of the cross-modal steganography device that performs cross-modal steganography of ciphertext can be completed in the server or in the client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor for specific processing of cross-modal steganography of ciphertext.

[0184] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0185] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0186] From the above description, it can be seen that the cross-modal steganography device provided in the embodiment of the present application can embed the ciphertext text data of the text modality into the carrier image of the image modality through a deep learning algorithm to realize multimodal steganography technology. By combining a reversible neural network with an end-to-end network, it can effectively improve the concealment and robustness of the ciphertext steganography, thereby not only improving the concealment of the ciphertext text data, but also enhancing its resistance to various attacks, thereby achieving more secure and reliable information transmission.

[0187] On this basis, in order to further realize cross-modal decoding of stego ciphertext and improve the accuracy and reliability of stego ciphertext decoding, in a cross-modal steganography device provided in an embodiment of the present application, see Fig.14 , the cross-modal steganography device further specifically includes the following contents:

[0188] The receiving module 30 is used to receive a noise image and extract a noise carrier feature vector and a noise ciphertext feature vector corresponding to the noise image, wherein the noise image is formed after the ciphertext cross-modal steganography result data is transmitted over the network, and the ciphertext cross-modal steganography result data is pre-generated based on a target carrier feature vector corresponding to a carrier image output by the encoder and a target ciphertext feature vector corresponding to a ciphertext image of ciphertext text data.

[0189] The decoding module 40 is used to input the noise carrier feature vector and the noise ciphertext feature vector into a decoder corresponding to the encoder respectively, so that the learnable wavelet transform network and the reversible neural network in the decoder respectively perform inverse feature extraction on the noise carrier feature vector and the noise ciphertext feature vector to obtain a loss carrier image corresponding to the noise carrier feature vector and a restored ciphertext image corresponding to the noise ciphertext feature vector; wherein the loss carrier image corresponding to the noise carrier feature vector is different from the carrier image used to generate the noise image to which the noise carrier feature vector belongs.

[0190] The restoration module 50 is used to extract information from the ciphertext image to obtain ciphertext text data corresponding to the restored ciphertext image.

[0191] The embodiment of the present application also provides an electronic device, which may include a processor, a memory, a receiver and a transmitter, wherein the processor is used to execute the cross-modal steganography method mentioned in the above embodiment, wherein the processor and the memory may be connected via a bus or other means, such as by bus connection. The receiver may be connected to the processor and the memory via wired or wireless means.

[0192] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0193] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the cross-modal steganography method in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, realizing the cross-modal steganography method in the above method embodiment.

[0194] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0195] The one or more modules are stored in the memory, and when executed by the processor, perform the cross-modal steganography method in the embodiment.

[0196] In some embodiments of the present application, the user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0197] As an implementation method, the functions of the receiver and the transmitter in the present application can be considered to be implemented through a transceiver circuit or a dedicated chip for transceiver, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general chip.

[0198] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiment of the present application, that is, to store the program code for implementing the functions of the processor, receiver, and transmitter in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0199] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned cross-modal steganography method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0200] An embodiment of the present application also provides a computer program product, including a computer program / instruction, which implements the steps of the aforementioned cross-modal steganography method when executed by a processor.

[0201] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0202] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0203] In this application, the features described and / or illustrated for one embodiment may be used in the same manner or in a similar manner in one or more other embodiments, and / or combined with or replace the features of other embodiments.

[0204] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A cross-modal steganography method, characterized in that: include: Inputting the carrier image and the ciphertext image corresponding to the ciphertext text data into a preset encoder respectively, so that the learnable wavelet transform network and the reversible neural network in the encoder respectively perform forward feature extraction on the carrier image and the ciphertext image to obtain a target carrier feature vector corresponding to the carrier image and a target ciphertext feature vector corresponding to the ciphertext image; Based on the target carrier feature vector and the target ciphertext feature vector, ciphertext cross-modal steganography result data corresponding to the ciphertext text data is generated for network transmission.

2. The cross-modal steganography method according to claim 1, characterized in that: Also includes: Receiving a noise image and extracting a noise carrier feature vector and a noise ciphertext feature vector corresponding to the noise image, wherein the noise image is formed after the ciphertext cross-modal steganography result data is transmitted over a network, and the ciphertext cross-modal steganography result data is generated in advance based on a target carrier feature vector corresponding to a carrier image output by the encoder and a target ciphertext feature vector corresponding to a ciphertext image of ciphertext text data; The noise carrier feature vector and the noise ciphertext feature vector are respectively input into a decoder corresponding to the encoder, so that the learnable wavelet transform network and the reversible neural network in the decoder respectively perform inverse feature extraction on the noise carrier feature vector and the noise ciphertext feature vector, so as to obtain a loss carrier image corresponding to the noise carrier feature vector and a restored ciphertext image corresponding to the noise ciphertext feature vector; wherein the loss carrier image corresponding to the noise carrier feature vector is different from the carrier image used to generate the noise image to which the noise carrier feature vector belongs; Information is extracted from the ciphertext image to obtain ciphertext text data corresponding to the restored ciphertext image.

3. The cross-modal steganography method according to claim 1, characterized in that: The encoder includes a forward transfer network corresponding to a learnable wavelet transform network, a forward transfer network corresponding to a reversible neural network, and a reverse transfer network corresponding to the learnable wavelet transform network, which are connected in sequence; Correspondingly, the carrier image and the ciphertext image corresponding to the ciphertext text data are respectively input into a preset encoder, so that the learnable wavelet transform network and the reversible neural network in the encoder respectively perform forward feature extraction on the carrier image and the ciphertext image to obtain a target carrier feature vector corresponding to the carrier image and a target ciphertext feature vector corresponding to the ciphertext image, including: The carrier image and the ciphertext image corresponding to the ciphertext text data are respectively input into a preset encoder, so that the forward transfer network corresponding to the learnable wavelet transform network in the encoder performs down-sampling feature extraction for multiple different frequency domains on the carrier image and the ciphertext image, and outputs the first multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively; then the forward transfer network corresponding to the reversible neural network in the encoder performs feature extraction for target operation on the first multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively based on the self-attention mechanism, and outputs the second multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively; then the reverse transfer network corresponding to the learnable wavelet transform network in the encoder performs up-sampling feature extraction for the same frequency domain on the second multi-frequency domain feature vectors corresponding to the carrier image and the ciphertext image respectively, and outputs the target carrier feature vector corresponding to the carrier image and the target ciphertext feature vector corresponding to the ciphertext image.

4. The cross-modal steganography method according to claim 2, characterized in that: The decoder comprises: a forward transfer network corresponding to the learnable wavelet transform network, a reverse transfer network corresponding to the reversible neural network and a reverse transfer network corresponding to the learnable wavelet transform network connected in sequence; Correspondingly, the noise carrier feature vector and the noise ciphertext feature vector are respectively input into a decoder corresponding to the encoder, so that the learnable wavelet transform network and the reversible neural network in the decoder respectively perform inverse feature extraction on the noise carrier feature vector and the noise ciphertext feature vector to obtain a loss carrier image corresponding to the noise carrier feature vector and a restored ciphertext image corresponding to the noise ciphertext feature vector, including: The noise carrier feature vector and the noise ciphertext feature vector are respectively input into the decoder, so that the forward transfer network of the learnable wavelet transform network in the decoder performs down-sampling feature extraction for multiple different frequency domains on the noise carrier feature vector and the noise ciphertext feature vector, and outputs the third multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively; then the reverse transfer network corresponding to the reversible neural network in the encoder performs feature restoration for the target operation on the third multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively based on the self-attention mechanism, and outputs the fourth multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively; then the reverse transfer network corresponding to the learnable wavelet transform network in the encoder performs up-sampling feature extraction for the same frequency domain on the fourth multi-frequency domain feature vectors corresponding to the noise carrier feature vector and the noise ciphertext feature vector respectively, and outputs the loss carrier image corresponding to the noise ciphertext feature vector and the restored ciphertext image corresponding to the noise ciphertext feature vector.

5. The cross-modal steganography method according to claim 2, characterized in that: Before the carrier image and the ciphertext image corresponding to the ciphertext text data are respectively input into a preset encoder, the method further includes: The learnable wavelet transform network and the reversible neural network are trained based on a preset target loss function and training data, so that the learnable wavelet transform network and the reversible neural network constitute the encoder and the decoder respectively; wherein the training data is composed of various data samples, and each of the data samples contains a carrier image, a ciphertext image corresponding to the ciphertext data, and a noise image of the same size; the ciphertext image corresponding to the ciphertext data is generated by pre-processing the ciphertext data with watermark diffusion; the noise image is generated by pre-simulating a noise attack on the ciphertext cross-modal steganography result data of the ciphertext data corresponding to the noise image based on a plurality of preset noise types; The learnable wavelet transform network is composed of a plurality of convolutional neural networks corresponding to different frequency domains for simulating Haar wavelet transform; The reversible neural network is composed of multiple layers of deformable convolutional layers, and each layer of the deformable convolutional layers is associated based on a target operation, and the target operation includes: at least one of addition, subtraction, multiplication and division; The deformable convolution layer is provided with a deformable attention dense block for executing a preset operation function, and the deformable attention dense block includes a connected self-attention block and a deformable dense block; wherein the deformable dense block includes: a deformable convolution layer and four convolution layers densely connected to the deformable convolution layer in sequence.

6. The cross-modal steganography method according to claim 5, characterized in that: The target loss function is composed of a coding loss, an image restoration loss, a message decoding loss and a contrastive learning loss, and weights corresponding to the coding loss, the image restoration loss, the message decoding loss and the contrastive learning loss; the weights of the coding loss, the image restoration loss and the contrastive learning loss are positive values, and the weight of the message decoding loss is a negative value; The coding loss is used to represent the pixel-level difference value and image loss value respectively corresponding to the target ciphertext feature vector extracted by the encoder and the ciphertext image, and the target carrier feature vector extracted by the encoder and the carrier image; The image restoration loss is used to represent the pixel-level difference value between the loss carrier image output by the decoder and the carrier image; The message decoding loss is used to represent the difference between the restored ciphertext image output by the decoder and the ciphertext image input to the encoder, calculated based on the mean square error; The contrastive learning loss is used to represent the contrastive learning loss value between the encoder and the decoder obtained based on a joint contrastive learning unit; wherein the joint contrastive learning unit includes a connected encoding contrastive learning unit and a decoding contrastive learning unit.

7. The cross-modal steganography method according to claim 2, characterized in that: The extracting information from the ciphertext image to obtain ciphertext text data corresponding to the restored ciphertext image includes: The restored ciphertext image is input into a preset decoding optimization module based on an attention mechanism and a convolutional layer, so that the decoding optimization module outputs ciphertext text data corresponding to the restored ciphertext image.

8. A cross-modal steganographic device, characterized in that: include: An encoding module, used for inputting the carrier image and the ciphertext image corresponding to the ciphertext text data into a preset encoder respectively, so that the learnable wavelet transform network and the reversible neural network in the encoder respectively perform forward feature extraction on the carrier image and the ciphertext image to obtain a target carrier feature vector corresponding to the carrier image and a target ciphertext feature vector corresponding to the ciphertext image; The steganography module is used to generate ciphertext cross-modal steganography result data corresponding to the ciphertext text data based on the target carrier feature vector and the target ciphertext feature vector for network transmission.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the cross-modal steganography method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cross-modal steganography method according to any one of claims 1 to 7 is implemented.

Images