Image storage method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image storage method and device, electronic equipment and a storage medium, and is applied to the technical field of images. According to the method provided by the embodiment of the invention, the to-be-stored image can be encoded through the encoder of the deep learning model, the classifier classifies the encoded first image feature, and the decoder decodes the sampled first potential spatial feature to obtain the first decoded image. Calculating a second loss value according to the first decoded image and the to-be-stored image, and calculating a first loss value according to the first classification result and the extraction code; and adjusting parameters of the deep learning model through the loss value to realize training of the deep learning model. According to the image storage method and device, storage of the to-be-stored image is realized through training of the deep learning model, the first coding information after coding of the to-be-stored image is stored in the potential space, a storage device cannot directly view the to-be-stored image, and the image can be read only by extracting the code, so that the image storage security is improved.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to an image storage method, device, electronic device and storage medium. Background Art

[0002] In the era of information development, facing huge amounts of data, there are usually higher requirements for the storage space of storage devices. For example, as people's requirements for image pixel quality increase, the image resolution also increases, and the storage capacity required for the image also increases. However, the storage space of storage devices is limited. When the number of images is large, the storage pressure of the storage device also increases.

[0003] In order to reduce the storage space occupied by images, in related image storage technologies, related compression tools are usually used to compress and encode the images to be stored. For example, an online compression tool is used to compress a picture to obtain a compressed image. However, for a compressed image, anyone can use a related compression tool to decompress it to obtain the image. Therefore, using commonly used related compression tools to compress and store images will result in the risk of image leakage. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image storage method, device, electronic device and storage medium to improve the security of image storage. The specific technical solution is as follows:

[0005] According to a first aspect of an embodiment of the present application, there is provided an image storage method, the method comprising:

[0006] Obtaining an image to be stored and an extraction code of the image to be stored;

[0007] Encoding the image to be stored using an encoder of a deep learning model to obtain first encoding information, and storing the first encoding information in a latent space, wherein the first encoding information includes a first image feature;

[0008] Using the classifier of the deep learning model to perform classification prediction on the first image feature to obtain a first classification result;

[0009] Calculating a first loss value of the deep learning model according to the extraction code and the first classification result;

[0010] Sampling the latent space to obtain a first latent space feature;

[0011] Decoding the extracted code and the first latent space feature using a decoder of the deep learning model to obtain a first decoded image;

[0012] Calculating a second loss value of the deep learning model according to the first decoded image and the image to be stored;

[0013] Adjust the parameters of the deep learning model according to the first loss value and the second loss value.

[0014] In a possible implementation, sampling the latent space to obtain a first latent space feature includes:

[0015] Sampling the latent space to obtain first sampling feature information, and calculating the mean and variance of each element in the first sampling feature information;

[0016] Obtain a preset matrix, and calculate the product of the variance and the preset matrix to obtain a first matrix, wherein the preset matrix obeys a normal distribution;

[0017] The mean is added to each element in the first matrix to obtain a first latent space feature.

[0018] In a possible implementation, the encoder of the deep learning model includes an image mapping module and an encoding module.

[0019] The encoder using the deep learning model encodes the image to be stored to obtain first encoding information, and stores the first encoding information in a latent space; comprising:

[0020] Using the image mapping module, a convolution operation is performed on the image to be stored to obtain a first coded image feature; wherein the first coded image feature includes image features after convolution of each image block of the image to be stored;

[0021] The first coded image feature is encoded by using the encoding module to obtain the first coded information, and the first coded information is stored in the latent space.

[0022] In a possible implementation, the encoding module includes a plurality of encoding submodules, each of which is connected in order; each of the encoding submodules includes a first normalization network, a first multi-head attention mechanism network, a second normalization network, and a first feedforward neural network, and the encoding module is used to encode the first encoded image feature to obtain the first encoded information, and the first encoded information is stored in the latent space, including:

[0023] For each encoding submodule, using the first normalization network of the encoding submodule, normalizing the input image feature to be encoded to obtain a second encoded image feature, wherein when the encoding submodule is the first encoding submodule, the image feature to be encoded is the first encoded image feature; otherwise, the image feature to be encoded is the seventh encoded image feature output by the previous encoding submodule;

[0024] Using the first multi-head attention mechanism network of the encoding submodule, feature extraction and encoding are performed on the second encoded image feature to obtain a third encoded image feature;

[0025] Performing a residual connection on the third coded image feature of the coding submodule and the image feature to be coded to obtain a fourth coded image feature;

[0026] The fourth encoded image feature input is normalized by using the second normalization network of the encoding submodule to obtain a fifth encoded image feature.

[0027] Using the first feedforward neural network of the encoding submodule to perform feature enhancement on the input fifth encoded image feature to obtain a sixth encoded image feature;

[0028] The sixth encoded image feature and the fourth encoded image feature are residually connected to obtain the seventh encoded image feature; wherein the first image feature is the seventh encoded image feature output by the last encoding submodule, and the first encoding information at least includes the first image feature.

[0029] In a possible implementation, the first feedforward neural network includes a first fully connected subnetwork, a first activation function subnetwork, a first regularization subnetwork, a second fully connected subnetwork, and a second regularization subnetwork.

[0030] The step of using the first feedforward neural network to perform feature enhancement on the input fifth encoded image feature to obtain a sixth encoded image feature includes:

[0031] Performing a linear transformation on the fifth encoded image feature through the first fully connected sub-network to obtain a first encoded image sub-feature;

[0032] Using the first activation function sub-network, performing a nonlinear transformation on the first encoded image sub-feature to obtain a second encoded image sub-feature;

[0033] Using the first regularization sub-network, regularize the second encoded image sub-feature to obtain a third encoded image sub-feature;

[0034] Using the second fully connected sub-network, linearly transform the third encoded image sub-feature to obtain a fourth encoded image sub-feature;

[0035] The fourth encoded image sub-feature is regularized by the second regularization sub-network to obtain the sixth encoded image feature.

[0036] In a possible implementation, the decoder of the deep learning model includes a decoding module and an image generation module.

[0037] The method of decoding the extraction code and the first latent space feature by using the decoder of the deep learning model to obtain a first decoded image includes:

[0038] Using the decoding module, the extraction code and the first latent space feature are decoded to obtain a second image feature; wherein the second image feature includes image features after decoding of multiple image blocks;

[0039] The image generation module is used to arrange and combine the image blocks in the second image feature to generate the first decoded image.

[0040] In a possible implementation, the decoding module includes a plurality of decoding submodules, each of which is connected in sequence; each of the decoding submodules includes a third normalized network, a first fully connected network, a masked multi-head attention mechanism network, a second fully connected network, and a second feedforward neural network.

[0041] The step of decoding the extraction code and the first latent space feature using the decoding module to obtain a second image feature includes:

[0042] For each decoding submodule, using the third normalization network of the decoding submodule, normalizing the feature to be decoded to obtain a first decoded image feature; wherein, when the decoding submodule is the first decoding submodule, the feature to be decoded is the first latent space feature; otherwise, the feature to be decoded is the seventh decoded image feature output by the previous decoding submodule;

[0043] Using a first fully connected network of the decoding submodule, linearly transform the first decoded image feature to obtain a second decoded image feature;

[0044] Decoding the second decoded image feature using the masked multi-head attention mechanism network of the decoding submodule to obtain a third decoded image feature;

[0045] Using the second fully connected network of the decoding submodule, linearly transform the third decoded image feature to obtain a fourth decoded image feature;

[0046] Performing a residual connection between the feature to be decoded and the fourth decoded image feature to obtain a fifth decoded image feature;

[0047] Using the second feedforward neural network of the decoding submodule to perform feature enhancement on the fifth decoded image feature to obtain a sixth decoded image feature;

[0048] The fifth decoded image feature is residually connected with the sixth decoded image feature to obtain the seventh decoded image feature; wherein the second image feature is the seventh decoded image feature output by the last decoding submodule.

[0049] In a possible implementation, adjusting the parameters of the deep learning model according to the first loss value and the second loss value includes:

[0050] According to the first loss value, adjusting parameters of the encoder and the classifier in the deep learning model;

[0051] According to the second loss value, adjust the parameters of the encoder and the decoder in the deep learning model.

[0052] In a possible implementation, the method further includes:

[0053] During the image reading process, obtaining the extraction code;

[0054] The extraction code is input into the deep learning model to obtain the read image.

[0055] In a second aspect of an embodiment of the present application, an image storage device is provided, the device comprising:

[0056] An image acquisition module, used to acquire an image to be stored and an extraction code of the image to be stored;

[0057] An image encoding module, configured to encode the image to be stored using an encoder of a deep learning model to obtain first encoding information, and store the first encoding information in a latent space, wherein the first encoding information includes a first image feature;

[0058] An image classification module, used to classify and predict the first image feature using the classifier of the deep learning model to obtain a first classification result;

[0059] A first loss value calculation module, used to calculate a first loss value of the deep learning model according to the extraction code and the first classification result;

[0060] A sampling module, used for sampling the latent space to obtain a first latent space feature;

[0061] An image decoding module, configured to decode the extraction code and the first latent space feature using a decoder of the deep learning model to obtain a first decoded image;

[0062] A second loss value calculation module, used to calculate a second loss value of the deep learning model according to the first decoded image and the image to be stored;

[0063] A parameter adjustment module is used to adjust the parameters of the deep learning model according to the first loss value and the second loss value.

[0064] In a possible implementation, the sampling module includes:

[0065] A sampling calculation submodule, specifically used to sample the latent space to obtain first sampling feature information, and calculate the mean and variance of each element in the first sampling feature information;

[0066] A matrix product submodule, specifically used to obtain a preset matrix, calculate the product of the variance and the preset matrix to obtain a first matrix, wherein the preset matrix obeys a normal distribution;

[0067] The mean superposition submodule is specifically used to add the mean to each element in the first matrix to obtain the first latent space feature.

[0068] In a possible implementation, the encoder of the deep learning model includes an image mapping module and an encoding module.

[0069] The image encoding module comprises:

[0070] An image convolution submodule, specifically used to use the image mapping module to perform a convolution operation on the image to be stored to obtain a first coded image feature; wherein the first coded image feature includes an image feature after convolution of each image block of the image to be stored;

[0071] The image encoding submodule is specifically used to use the encoding module to encode the first encoded image feature to obtain the first encoded information, and store the first encoded information in the latent space.

[0072] In a possible implementation, the encoding module includes a plurality of encoding submodules, each of which is connected in sequence; each of the encoding submodules includes a first normalization network, a first multi-head attention mechanism network, a second normalization network and a first feedforward neural network, and the image encoding submodule includes:

[0073] a first normalization unit, specifically configured to, for each encoding submodule, use a first normalization network of the encoding submodule to perform normalization processing on the input image feature to be encoded, so as to obtain a second encoded image feature, wherein when the encoding submodule is the first encoding submodule, the image feature to be encoded is the first encoded image feature; otherwise, the image feature to be encoded is the seventh encoded image feature output by the previous encoding submodule;

[0074] The encoding unit is specifically used to extract and encode the second encoded image features by using the first multi-head attention mechanism network of the encoding submodule to obtain third encoded image features;

[0075] A first residual connection unit, specifically used to perform residual connection on the third coded image feature of the coding submodule and the to-be-coded image feature to obtain a fourth coded image feature;

[0076] The second normalization unit is specifically used to use the second normalization network of the encoding submodule to perform normalization processing on the input fourth encoded image feature to obtain a fifth encoded image feature,

[0077] A feature enhancement unit, specifically configured to perform feature enhancement on the input fifth encoded image feature using the first feedforward neural network of the encoding submodule to obtain a sixth encoded image feature;

[0078] The second residual connection unit is specifically used to perform residual connection on the sixth encoded image feature and the fourth encoded image feature to obtain the seventh encoded image feature; wherein the first image feature is the seventh encoded image feature output by the last encoding submodule, and the first encoding information at least includes the first image feature.

[0079] In a possible implementation, the first feedforward neural network includes a first fully connected subnetwork, a first activation function subnetwork, a first regularization subnetwork, a second fully connected subnetwork, and a second regularization subnetwork.

[0080] The feature enhancement unit comprises:

[0081] A first linear transformation subunit, specifically configured to perform a linear transformation on the fifth encoded image feature through the first fully connected subnetwork to obtain a first encoded image sub-feature;

[0082] A first nonlinear transformation subunit is specifically used to perform a nonlinear transformation on the first encoded image sub-feature using the first activation function sub-network to obtain a second encoded image sub-feature;

[0083] a first regularization processing subunit, specifically configured to use the first regularization subnetwork to perform regularization processing on the second encoded image sub-feature to obtain a third encoded image sub-feature;

[0084] A second linear transformation subunit is specifically used to use the second fully connected sub-network to perform a linear transformation on the third encoded image sub-feature to obtain a fourth encoded image sub-feature;

[0085] The second regularization processing subunit is specifically used to perform regularization processing on the fourth encoded image sub-feature through the second regularization sub-network to obtain the sixth encoded image feature.

[0086] In a possible implementation, the decoder of the deep learning model includes a decoding module and an image generation module.

[0087] The image decoding module comprises:

[0088] An image decoding submodule, specifically configured to use the decoding module to decode the extraction code and the first latent space feature to obtain a second image feature; wherein the second image feature includes image features after decoding of a plurality of image blocks;

[0089] The image generation submodule is specifically used to use the image generation module to arrange and combine the image blocks in the second image feature to generate the first decoded image.

[0090] In a possible implementation, the decoding module includes a plurality of decoding submodules, each of which is connected in sequence; each of the decoding submodules includes a third normalized network, a first fully connected network, a masked multi-head attention mechanism network, a second fully connected network, and a second feedforward neural network.

[0091] The image decoding submodule comprises:

[0092] a third normalization unit, specifically configured to perform normalization processing on the feature to be decoded by using the third normalization network of the decoding submodule for each decoding submodule, so as to obtain a first decoded image feature; wherein, when the decoding submodule is the first decoding submodule, the feature to be decoded is the first latent space feature; otherwise, the feature to be decoded is the seventh decoded image feature output by the previous decoding submodule;

[0093] A first linear transformation unit, specifically configured to use a first fully connected network of the decoding submodule to perform a linear transformation on the first decoded image feature to obtain a second decoded image feature;

[0094] An image decoding unit, specifically configured to decode the second decoded image feature using a masked multi-head attention mechanism network of the decoding submodule to obtain a third decoded image feature;

[0095] A second linear transformation unit, specifically configured to use the second fully connected network of the decoding submodule to perform a linear transformation on the third decoded image feature to obtain a fourth decoded image feature;

[0096] A third residual connection unit is specifically used to perform a residual connection on the feature to be decoded and the fourth decoded image feature to obtain a fifth decoded image feature;

[0097] A second feature enhancement unit, specifically configured to perform feature enhancement on the fifth decoded image feature using the second feedforward neural network of the decoding submodule to obtain a sixth decoded image feature;

[0098] The fourth residual connection unit is specifically used to perform a residual connection on the fifth decoded image feature and the sixth decoded image feature to obtain the seventh decoded image feature; wherein the second image feature is the seventh decoded image feature output by the last decoding submodule.

[0099] In a possible implementation, the parameter adjustment module includes:

[0100] A first parameter adjustment submodule, specifically used to adjust parameters of an encoder and a classifier in the deep learning model according to the first loss value;

[0101] The second parameter adjustment submodule is specifically used to adjust the parameters of the encoder and the decoder in the deep learning model according to the second loss value.

[0102] In a possible implementation, the device further includes:

[0103] An extraction code acquisition module, used to acquire the extraction code during the image reading process;

[0104] An image reading module is used to input the extraction code into the deep learning model to obtain a read image.

[0105] Another aspect of the embodiments of the present application further provides an electronic device, including:

[0106] Memory, used to store computer programs;

[0107] The processor is used to implement any image storage method described in the embodiments of the present application when executing the program stored in the memory.

[0108] In another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned image storage methods is implemented.

[0109] An embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned image storage methods.

[0110] Beneficial effects of the embodiments of the present application:

[0111] The embodiment of the present application provides an image storage method, device, electronic device and storage medium, which can encode the image to be stored through the encoder of the deep learning model, store the encoded first encoding information in the latent space of the deep learning model, classify the encoded first image features through the classifier of the deep learning model, establish the corresponding relationship between the extraction code and the first image features, and decode the sampled first latent space features through the decoder of the deep learning model to obtain the first decoded image. Thus, the second loss value is calculated based on the first decoded image and the image to be stored, and the first loss value is calculated based on the first classification result and the extraction code; the parameters of the deep learning model are adjusted through the first loss value and the second loss value to realize the training of the deep learning model.

[0112] Compared with the prior art, the embodiment of the present application realizes the storage of the image to be stored by training the deep learning model, and stores the first encoding information after encoding the image to be stored in the potential space of the deep learning model. The image to be stored cannot be visually seen on the relevant storage device, thereby improving the security of image storage. In addition, when acquiring an image, the embodiment of the present application needs to input the extraction code into the trained deep learning model to obtain the image corresponding to the extraction code, further ensuring the storage security of the image data.

[0113] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0115] Figure 1 A schematic diagram of a flow chart of an image storage method provided in an embodiment of the present application;

[0116] Figure 2A schematic diagram of sampling a latent space provided in an embodiment of the present application;

[0117] Figure 3 A schematic diagram of a network structure of an encoder provided in an embodiment of the present application;

[0118] Figure 4 A schematic diagram of a network structure of the encoding submodule provided in an embodiment of the present application;

[0119] Figure 5 A schematic diagram of a network structure of a first feedforward neural network provided in an embodiment of the present application;

[0120] Figure 6 A schematic diagram of a network structure of a decoder provided in an embodiment of the present application;

[0121] Figure 7 A schematic diagram of a network structure of a decoding submodule provided in an embodiment of the present application;

[0122] Figure 8 A schematic diagram of a network structure of a deep learning model provided in an embodiment of the present application;

[0123] Fig. 9 A schematic diagram of the structure of an image storage device provided in an embodiment of the present application;

[0124] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0125] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0126] In image storage technology, in order to reduce the storage space occupied by images, relevant compression tools are usually used to compress and encode the images to be stored. For example, an online compression tool is used to compress a picture to obtain a compressed image. However, for a compressed image, anyone can use the relevant compression tool to decompress it to obtain the image. Therefore, using commonly used relevant compression tools to compress and store images will cause the risk of image leakage.

[0127] In order to solve at least one of the above problems, a first aspect of an embodiment of the present application provides an image storage method, the method comprising: Figure 1 Steps shown:

[0128] Step S101: obtaining an image to be stored and an extraction code of the image to be stored.

[0129] The image to be stored can be an RGB (Red / Green / Blue) color image or a grayscale image. In the embodiment of the present application, there is no restriction on the resolution, shape, size and other parameters of the image to be stored. The extraction code is used to indicate the category of the image to be stored, which is a unique identifier of the image to be stored, and can be the id (identity document) information, name or text content used to describe the content of the image to be stored.

[0130] In practical applications, the method of the embodiments of the present application can be implemented through a terminal device, which can be a computer, server, or other electronic device that can be used for deep learning model training.

[0131] Step S102: Encode the image to be stored using an encoder of the deep learning model to obtain first encoding information, and store the first encoding information in a latent space.

[0132] Among them, the deep learning model is a neural network model including modules such as an encoder, a classifier and a decoder. In an example, the deep learning model of an embodiment of the present application is a transformer (natural language processing model) neural network. The encoder in the deep learning model has a multi-layer coding structure, and each coding structure is connected in order. Each layer of the coding structure encodes the input image or image features to be stored to obtain a coded image feature. The first coding information may include the coding information output by each layer of the coding structure, or may only include the coding information output by part of the coding structure. Among them, the first image feature is the coding information output by the last layer of the coding structure, and the first coding information should at least include the first image feature.

[0133] Step S103: Use the classifier of the deep learning model to perform classification prediction on the first image feature to obtain a first classification result.

[0134] The first classification result is information indicating the category of the image to be stored. In order to intuitively represent the predicted probability of the category of the image to be stored, the Softmax function can be selected to classify the first image feature. Through the Softmax function, the original output of the deep learning model can be indexed and normalized so that it is mapped to a probability value, and the sum of the probabilities of each classification result is 1. In practical applications, the classifier can be connected to the last layer of the encoder structure so that the first image feature is input into the classifier for classification.

[0135] Step S104: Calculate a first loss value of the deep learning model according to the extracted code and the first classification result.

[0136] The first loss value is the image category loss. The first classification result can be used as the prediction result, and the extracted code can be used as the true value to calculate the first loss value by cross entropy. In an example, the first loss value can be calculated by the following formula:

[0137]

[0138] Among them, C represents the vector dimension of the true label; x represents the input data, x i Represents the i-th element of the true label, y i represents the probability that the deep learning model predicts that x belongs to the i-th category.

[0139] In this embodiment, the input data is the first image feature, and the true label is the extraction code of the image to be stored. When the first loss value is smaller, it means that the probability distribution of the first classification result predicted by the model is closer to the probability distribution of the actual extraction code, and the accuracy of the classifier prediction of the deep learning model is higher.

[0140] Step S105: sampling the latent space to obtain a first latent space feature.

[0141] In order to obtain the image information of the image to be stored and simplify the expression of image features, the latent space can be sampled to obtain low-dimensional features to facilitate the subsequent processing of image feature data. In one example, the latent space can be sampled by VAE (Variational AutoEncoder), and the image features of the image to be stored are mapped to the latent space using the VAE encoder, and the distribution parameters of the latent space are calculated. According to the distribution parameters, the latent space is randomly sampled to obtain a latent vector, which is the first latent space feature in the embodiment of the present application.

[0142] Step S106: Use the decoder of the deep learning model to decode the extracted code and the first latent space feature to obtain a first decoded image.

[0143] Similarly, the decoder may also include a multi-layer decoding structure, each decoding structure is sequentially connected in order, and each decoding structure works together to restore the first latent space feature from a vector to an image. In an example, a mapping relationship between a feature vector and an image is learned through a deep learning model, and a first decoded image is reconstructed based on the mapping relationship and the first latent space feature.

[0144] Step S107: Calculate a second loss value of the deep learning model based on the first decoded image and the image to be stored.

[0145] The second loss value is the image loss, which can be calculated by calculating the mean square error (MSE) or the cross entropy loss. The cross entropy loss can be calculated in the same way as step S104, and the mean square error can be calculated by the following formula:

[0146] MSE=(A i -A , i ) 2

[0147] Among them, A i Indicates the image to be stored, A , i Represents the first decoded image.

[0148] Step S108: Adjust the parameters of the deep learning model according to the first loss value and the second loss value.

[0149] Among them, the first loss value is used to adjust the parameters of the encoder and classifier in the deep learning model; the second loss value is used to adjust the parameters of the encoder and decoder in the deep learning model. In practical applications, the encoder, classifier and decoder are trained by the back propagation algorithm, and the parameters of the network are adjusted by reverse calculation through the calculated loss value. The adjustment of the deep learning model parameters needs to meet the requirement that when the image to be stored and the extraction code of the image to be stored are input into the adjusted deep learning model, the recalculated first loss value and second loss value are both less than a preset value, which is used to ensure that the difference between the first decoded image and the image to be stored is within a tolerable difference range.

[0150] In one example, according to the first loss value, the parameters of the encoder and classifier in the deep learning model are adjusted, that is, the weights and biases of the encoder and classifier are adjusted to reduce the first loss value; according to the second loss value, the parameters of the encoder and decoder in the deep learning model are adjusted, that is, the weights and biases of the encoder and decoder are adjusted to reduce the second loss value. This process is iterated continuously until the first loss value is less than the first loss threshold and the second loss value is less than the second loss threshold, and the training of the deep learning model is completed. Among them, the weight is used to determine the propagation mode of the image feature data in the deep learning model to ensure that the deep learning model learns the effective representation of the image to be stored; the bias is used to make the deep learning model better adapt to the distribution of the feature data.

[0151] By applying the method of the embodiment of the present application, the image to be stored can be encoded by the encoder of the deep learning model, the encoded first encoding information can be stored in the latent space of the deep learning model, the encoded first image features can be classified by the classifier of the deep learning model, the corresponding relationship between the extraction code and the first image features can be established, and the first latent space features after sampling can be decoded by the decoder of the deep learning model to obtain the first decoded image. Thus, the second loss value is calculated according to the first decoded image and the image to be stored, and the first loss value is calculated according to the first classification result and the extraction code; the parameters of the deep learning model are adjusted by the first loss value and the second loss value to realize the training of the deep learning model. Compared with the prior art, the embodiment of the present application realizes the storage of the image to be stored by training the deep learning model, and the first encoding information after encoding the image to be stored is stored in the latent space of the deep learning model. The image to be stored cannot be intuitively seen on the storage device, and the image cannot be restored by deciphering the latent space, thereby improving the security of image storage. And when acquiring an image, the embodiment of the present application needs to input the extraction code into the trained deep learning model to obtain the image corresponding to the extraction code, which further ensures the storage security of the image data.

[0152] In a possible implementation, step S105 can implement sampling of the latent space by the following steps:

[0153] Step 1: Sample the latent space to obtain first sampling feature information, and calculate the mean and variance of each element in the first sampling feature information.

[0154] Taking the first image feature as an example, when sampling, random sampling can be performed according to the distribution of the first image feature, so that the sampled first sampling feature information is consistent with the distribution of the first image feature, and the dimension of the first sampling feature is lower than that of the first image feature, thereby simplifying the image feature data. In an example, the mean and variance of each element in the first image feature can be calculated to obtain the probability distribution of the first image feature, random sampling is performed according to the probability distribution to obtain the first sampling feature information, and the mean and variance of each element in the first sampling feature information are calculated.

[0155] Step 2: Obtain a preset matrix, and calculate the product of the variance and the preset matrix to obtain a first matrix.

[0156] Among them, the preset matrix obeys the normal distribution. Figure 2 As shown, the preset matrix is ​​a vector ε sampled by the autoencoder VAE from a standard normal distribution N(0, I). By multiplying the variance σ of the first sampling feature information with the preset matrix ε, the generated preset matrix can be associated with the first sampling feature information, which facilitates subsequent data processing.

[0157] Step 3: Add the mean value to each element in the first matrix to obtain the first latent space feature.

[0158] like Figure 2 As shown, the encoder of the deep learning model stores the first encoded information in the latent space, samples the latent space to obtain the first sampled feature information, and calculates the mean μ and variance σ of each element in the first sampled feature information. Sample ε that conforms to the normal distribution from N(0, I), and obtain the first matrix through εxσ. Then the first latent space feature z=μ+εxσ. The first latent space feature is input into the decoder for decoding to obtain the first decoded image.

[0159] By applying the method of the embodiment of the present application, the latent space can be sampled according to the probability distribution, so that the first latent space feature has similarity with the first encoded information while reducing the dimension and simplifying the data, thereby ensuring the accuracy of the image data. And by sampling a noise ε from a standard normal distribution, and then using μ and σ to adjust the noise, the final first latent space feature is obtained, which can ensure that randomness is introduced into the deep learning model while allowing the gradient of the deep learning model to be back-transmitted, thereby allowing the deep learning model to learn and optimize during the training process, thereby completing the storage of image data.

[0160] In a possible implementation, the encoder of the deep learning model includes an image mapping module and an encoding module, and the encoder in step S102 can implement encoding and storage of the image to be stored by the following steps:

[0161] Step A: Using the image mapping module, a convolution operation is performed on the image to be stored to obtain a first encoded image feature.

[0162] The first encoded image feature includes the image features of each image block of the image to be stored. In practical applications, the image mapping module can divide the image to be stored into patches, and then for each image block, flatten the image block so that it changes from a two-dimensional image to a one-dimensional vector, thereby mapping the image into a feature vector. The feature vectors of each image block are concatenated and combined to obtain the first encoded image feature. In practical applications, the block division and feature mapping of the image to be stored can be directly realized through convolution operations.

[0163] Step B: Encode the first encoded image feature using the encoding module to obtain first encoded information, and store the first encoded information in the latent space.

[0164] When the encoding module encodes the first encoded image features, important feature information of the first encoded image can be extracted through the attention mechanism network to achieve effective representation of the image to be stored. In practical applications, the encoding module includes multiple encoding submodules, each encoding submodule encodes the first encoded image features, and the first encoding information may include image feature information extracted by each encoding submodule.

[0165] like Figure 3 As shown, the encoder 31 of the deep learning network includes an image mapping module 311 and an encoding module 312; the encoding module includes a plurality of encoding submodules 3121, each encoding submodule is connected in order, and the last encoding submodule is connected to the classifier. The image mapping module convolves the image to be stored, converts it from an image to a first encoded image feature represented by a feature vector, and inputs it to the first encoding submodule. The first encoding submodule encodes the first encoded image feature, and inputs the encoded image feature to the next encoding submodule, and so on, until the last encoding submodule outputs the first image feature. Each encoding submodule can also store the image features output by itself in the latent space.

[0166] By applying the method of the embodiment of the present application, the image to be stored can be converted from a two-dimensional image into a vector through an image mapping module, and the first encoded image features can be encoded through the encoding module to obtain the feature information of the image to be stored, so as to store the feature information, that is, the first encoded information, in the latent space of the deep learning model, thereby improving the security of image data storage.

[0167] In a possible implementation, the encoding module includes a plurality of encoding submodules, and the encoding submodules are sequentially connected; each encoding submodule includes a first normalized network, a first multi-head attention mechanism network, a second normalized network, and a first feedforward neural network, then step B can be implemented by the following steps:

[0168] Step a: For each encoding submodule, use the first normalization network of the encoding submodule to normalize the input image features to be encoded to obtain second encoded image features.

[0169] Wherein, when the encoding submodule is the first encoding submodule, the image feature to be encoded is the first encoding image feature; otherwise, the image feature to be encoded is the seventh encoding image feature output by the previous encoding submodule;

[0170] Step b: Use the first multi-head attention mechanism network of the encoding submodule to extract and encode the second encoded image features to obtain the third encoded image features.

[0171] Step c: Perform a residual connection on the third coded image feature of the coding submodule and the image feature to be coded to obtain a fourth coded image feature.

[0172] Step d: using the second normalization network of the encoding submodule, normalize the input fourth encoded image feature to obtain the fifth encoded image feature.

[0173] Step e: Use the first feedforward neural network of the encoding submodule to perform feature enhancement on the input fifth encoded image feature to obtain the sixth encoded image feature.

[0174] Step f: Perform a residual connection on the sixth coded image feature and the fourth coded image feature to obtain a seventh coded image feature.

[0175] The first image feature is the seventh coded image feature output by the last coding submodule, and the first coding information at least includes the first image feature.

[0176] like Figure 4 The network structure of one of the encoding submodules in the encoding module is shown. The network structure includes a first normalized network, a first multi-head attention mechanism network, a second normalized network and a first feedforward neural network. Among them, the first normalized network and the second normalized network have the same structure, which is a Layer Norm (layer normalization) network layer. In an example, the first normalized network or the second normalized network calculates the mean and standard deviation of each element in the input image feature to be encoded or the fourth encoded image feature, and normalizes each eigenvalue of the image feature according to the mean and standard deviation, and represents the normalized eigenvalue according to the learnable parameters (scaling factor, offset factor) to obtain the normalized second encoded image feature or the fifth encoded image feature.

[0177] The first multi-head attention mechanism network (Multi-Head Attention) converts the input second encoded image features into a vector set of Q (Query), K (Key), and V (Value) through a linear transformation matrix, calculates the attention score of each eigenvalue according to Q, K, and V of the eigenvalue, normalizes and weighted sums the attention scores, and obtains the third encoded image features representing important image information.

[0178] The first feedforward neural network (FNN) is composed of multiple network layers, each of which is connected in a weighted manner, and the image feature data flows from the input layer to the output layer in a single direction, wherein each network layer continuously extracts and weights the image feature data, thereby achieving feature enhancement and retaining important image feature information. In an example, the feedforward neural network of an embodiment of the present application is an MLP (Multilayer Perceptron) Block.

[0179] In practical applications, the encoding submodule also includes a regularization network, which is used to discard image features in a certain proportion to prevent overfitting of the deep learning model during training. The regularization network can be regularized by a drop out method or a drop path method.

[0180] like Figure 4 As shown, if the encoding submodule is the first encoding submodule, the image feature input to the first normalization network is the first encoded image feature, otherwise, the image feature input to the first normalization network is the image feature output by the previous encoding submodule, that is, the seventh encoded image feature in this embodiment. The first normalization network normalizes the input image feature to be encoded to obtain the second encoded image feature; the second encoded image feature is subjected to feature extraction and encoding by the first multi-head attention mechanism network to obtain the third encoded image feature. In one example, after the second encoded image passes through the first multi-head attention mechanism network, it is also subjected to regularization by the first regularization network to obtain the third encoded image feature. The third encoded image feature and the image feature to be encoded are residually connected to obtain the fourth encoded image feature. Through the residual connection, the image feature to be encoded can directly skip the output connection between the multi-layer network layer and the network layer, thereby avoiding the problem of information loss, gradient disappearance or gradient explosion during model training.

[0181] The fourth coded image is normalized by the second normalization network to obtain the fifth coded image, and then the feature is enhanced by the first feedforward neural network to obtain the sixth coded image. Similarly, after the feature enhancement, the image features can also be regularized to obtain the sixth coded image. The fourth coded image feature and the sixth coded image feature are residually connected to obtain the seventh coded image feature, and the seventh coded image feature is continuously input into the next coding submodule. When the coding submodule is the last coding submodule, the seventh coded image feature output by it is the feature in the first figure above. The first coding information can also include the seventh coded image feature output by each coding submodule.

[0182] By applying the method of the embodiment of the present application, the first encoded image features are encoded by connecting multiple encoding sub-modules in sequence, so that important feature information of the image to be stored can be obtained, and the encoded first encoded information is stored. The image to be stored cannot be directly viewed on the storage device, thereby improving the security of image data storage.

[0183] In practical applications, the first feedforward neural network includes a first fully connected subnetwork, a first activation function subnetwork, a first regularized subnetwork, a second fully connected subnetwork, and a second regularized subnetwork. The first feedforward neural network can perform feature enhancement through the following steps:

[0184] Step 1: Perform a linear transformation on the fifth encoded image feature through the first fully connected sub-network to obtain the first encoded image sub-feature.

[0185] Step 2: Using the first activation function sub-network, perform a nonlinear transformation on the first encoded image sub-feature to obtain the second encoded image sub-feature.

[0186] Step 3: Use the first regularization sub-network to regularize the second encoded image sub-feature to obtain the third encoded image sub-feature.

[0187] Step 4: Use the second fully connected sub-network to perform a linear transformation on the third encoded image sub-feature to obtain the fourth encoded image sub-feature.

[0188] Step 5: Regularize the fourth encoded image sub-feature through the second regularization sub-network to obtain the sixth encoded image feature.

[0189] like Figure 5 As shown, the first fully connected subnetwork or the second fully connected subnetwork is a Linear layer, which is mainly used to multiply the input image feature vector by the weight matrix, add the bias vector, and perform a linear transformation on the fifth encoded image feature or the third encoded image sub-feature. The first activation function subnetwork is an activation function layer, and the activation function can be a ReLU activation function, a GELU activation function, or an SiLU activation function. In one example, the activation function is a GELU activation function, which is used to add nonlinear elements to the deep learning model so that the first activation function subnetwork performs a nonlinear transformation on the first encoded image sub-feature, so that the deep learning model can handle nonlinear relationships.

[0190] The first regularization subnetwork or the second regularization subnetwork can be regularized by the same or different method as the first regularization network. In one example, the drop-out method is used to regularize the sub-features in the second encoded image or the sub-features in the fourth encoded image.

[0191] By applying the method of the embodiment of the present application, after the fifth encoded image feature passes through the sub-networks of the first feedforward neural network, overfitting of the deep learning network can be avoided through multiple regularization networks, and linear and nonlinear transformations are performed through the fully connected sub-network and the activation function sub-network, so that the sixth encoded image feature after feature enhancement can adapt to more complex linear or nonlinear environments.

[0192] In one possible implementation, the decoder of the deep learning model includes a decoding module and an image generation module.

[0193] The deep learning model can obtain the first decoded image through the following steps:

[0194] Step I: Use the decoding module to decode the extracted code and the first latent space feature to obtain the second image feature.

[0195] The decoding module decodes the extraction code, and obtains the first latent space feature corresponding to the extraction code from the latent space according to the decoded extraction code, thereby decoding the first latent space feature. Since the encoder divides the image to be stored into image blocks when encoding the image to be stored, and encodes each image block, when the decoding module decodes the first latent space feature, the second image feature obtained by decoding also includes the image features after decoding each image block.

[0196] Step II: Using the image generation module, arrange and combine the image blocks in the second image feature to generate a first decoded image.

[0197] When encoding the image to be stored, the image mapping module of the encoder maps the image to be stored from a two-dimensional image to a feature vector. Similarly, when decoding the first latent space feature, the image generation module of the decoder can reconstruct the first latent space feature into a first decoded image according to the mapping relationship. In one example, the image features of each image block in the second image feature can be reconstructed into an image according to the mapping relationship between the image and the feature vector, and then the images corresponding to each image block are spliced ​​according to the positional relationship of each image block to obtain the first decoded image.

[0198] like Figure 6 As shown, the decoder 61 of the deep learning network includes an image generation module 611 and a decoding module 612; the decoding module includes a plurality of decoding submodules 6121, and each encoding submodule is sequentially connected. The first decoding submodule decodes the extraction code and the first latent space feature, and inputs the output decoded image feature to the next decoding submodule to continue the decoding operation until the last decoding submodule obtains the output second image feature. The image generation module reconstructs the second image feature into the first decoded image according to the mapping relationship between the feature vector and the image.

[0199] By applying the method of the embodiment of the present application, the extraction code and the first latent space feature can be decoded by the decoding module of the deep learning model to obtain the second image feature, and the second image feature can be reconstructed into the first decoded image by the image generation module. In this way, in the process of acquiring the image, it is necessary to input the extraction code into the deep learning model to obtain the image to be acquired, thereby improving the security of image storage.

[0200] In a possible implementation, the decoding module includes a plurality of decoding submodules, each of which is sequentially connected in order; each decoding submodule includes a third normalized network, a first fully connected network, a masked multi-head attention mechanism network, a second fully connected network, and a second feedforward neural network.

[0201] Then the decoding submodule in the decoding module can obtain the second image feature through the following steps:

[0202] Step (1): For each decoding submodule, use the third normalization network of the decoding submodule to normalize the features to be decoded to obtain the first decoded image features.

[0203] Among them, when the decoding submodule is the first decoding submodule, the feature to be decoded is the first latent space feature; otherwise, the feature to be decoded is the seventh decoded image feature output by the previous decoding submodule.

[0204] Step (2): using the first fully connected network of the decoding submodule, linearly transform the first decoded image feature to obtain the second decoded image feature.

[0205] Step (3): Use the masked multi-head attention mechanism network of the decoding submodule to decode the second decoded image feature to obtain the third decoded image feature.

[0206] Step (4): using the second fully connected network of the decoding submodule, linearly transform the third decoded image feature to obtain a fourth decoded image feature.

[0207] Step (5): Perform a residual connection between the feature to be decoded and the fourth decoded image feature to obtain the fifth decoded image feature.

[0208] Step (6): Use the second feedforward neural network of the decoding submodule to perform feature enhancement on the fifth decoded image feature to obtain the sixth decoded image feature.

[0209] Step (7): Perform a residual connection on the fifth decoded image feature and the sixth decoded image feature to obtain a seventh decoded image feature; wherein the second image feature is the seventh decoded image feature output by the last decoding submodule.

[0210] like Figure 7As shown, the third normalization network is an RMSNorm (root mean square layer normalization) network layer, which is used to calculate the root mean square of each eigenvalue of the input feature to be decoded, and normalize each eigenvalue according to the root mean square to obtain the first decoded image feature.

[0211] The first fully connected network is a linear layer, which performs a linear transformation on each feature value of the first decoded image feature to obtain the second decoded image feature. The mask tensor MASK is introduced in the masked multi-head attention mechanism network, which can shield or ignore the information at certain positions of the second decoded image feature, so as to only focus on the effective information part. Figure 7 As shown, for each eigenvalue of the second decoded image feature, the Q (Query), K (Key) and V (Value) of the eigenvalue are calculated, Q and K are encoded in the RoPE (Rotary Position Embedding) manner, and the attention score is calculated in combination with the mask tensor. The weights are normalized by the SoftMax function, and the normalized weights are weighted summed in combination with the Value to obtain the third decoded image feature output by the masked multi-head attention mechanism network.

[0212] After the second fully connected network (Linear layer), the third decoded image feature is linearly transformed to obtain a fourth decoded image feature, and the fourth decoded image feature is residually connected with the feature to be decoded input into the decoding submodule to obtain a fifth decoded image feature.

[0213] The second feedforward neural network is composed of multiple fully connected layers, normalization layers and activation functions, performs linear transformation and nonlinear transformation on the input fifth decoded image feature to obtain the sixth decoded image feature, and performs residual connection between the fifth decoded image feature and the sixth decoded image feature to obtain the seventh decoded image feature.

[0214] like Figure 7 As shown, the second feedforward neural network includes a first normalization layer (RMSNorm), a first fully connected layer (Linear), a second fully connected layer (Linear), a first activation function layer (SiLU) and a third fully connected layer (Linear). The fifth decoded image feature is normalized by the first normalization layer to obtain a first decoded sub-image feature; the first fully connected layer and the second fully connected layer respectively perform linear transformation on the first decoded sub-image feature to obtain a second decoded sub-image feature and a third decoded sub-image feature. The first activation function layer performs a nonlinear transformation on the third decoded sub-image feature to obtain a fourth decoded sub-image feature, and the fourth decoded sub-image feature is superimposed on the second decoded sub-image feature to obtain a fifth decoded sub-image feature. Finally, the third fully connected layer performs a linear transformation on the fifth decoded sub-image feature to obtain a sixth decoded image feature.

[0215] By applying the method of the embodiment of the present application, the first latent space feature and the extraction code can be decoded by sequentially connecting multiple decoding submodules. Through multiple decoding and superposition, the accuracy of generating the first decoded image can be improved.

[0216] In a possible implementation, the image to be read may be obtained by the following steps:

[0217] Step (a): During the image reading process, the extraction code is obtained.

[0218] Step (b): Input the extracted code into the deep learning model to obtain the read image.

[0219] After the deep learning model is trained, during the image reading process, the extraction code is input into the trained deep learning model. After the trained deep learning model obtains the extraction code, it uses the decoder to decode the extraction code, and obtains the first latent space feature corresponding to the extraction code from the latent space according to the decoded extraction code. Thus, the first latent space feature is decoded by each decoding submodule of the decoder to obtain the decoded image feature of the image to be read, and then the decoded image feature is restored into a two-dimensional picture through the image generation module of the decoder to obtain the read image.

[0220] By applying the method of the embodiment of the present application, people need to input an extraction code into the deep learning model during the process of acquiring an image. The decoder of the deep learning model can then obtain the first latent space feature corresponding to the extraction code based on the extraction code, decode the first latent space feature, and obtain the read image, thereby improving the security of image storage.

[0221] In a possible implementation manner, the following example is used to specifically illustrate the image storage method of the embodiment of the present application:

[0222] like Figure 8 As shown, the original image and the extraction code are input into the deep learning model, wherein the original image is the image to be stored. The image mapping module (Embeded Patches) performs a convolution operation on the original image to convert the original image into a first encoded image feature, and the first encoded image feature is encoded by multiple encoding submodules (Encoder Block) to obtain a first image feature.

[0223] The first image feature is input into the classifier of the deep learning model to obtain the first classification result, and the classifier includes AVG (Average Pooling Layer), FC (fully connected layer) and CELoss (Cross-Entropy Loss). The first loss value of the first classification result and the extracted code is calculated by the cross entropy loss function, and according to the first loss value, the parameters of the deep learning model encoder and the classifier are adjusted by reverse calculation.

[0224] After the encoder obtains the image features, it stores the image features in the latent space of the deep learning model. The stored image features are the first coding information, which at least includes the first image features output by the last coding submodule, and may also include image features output by other coding submodules. The latent space is sampled by the variational autoencoder to obtain the first sampling feature, and the mean μ and variance σ of each element in the first sampling feature information are calculated; the preset matrix ε is obtained, which obeys the normal distribution N(0, I), and the first latent space feature is calculated by the formula z=μ+εxσ.

[0225] Each decoding submodule (Decoder Block) of the deep learning network decodes the extraction code and the first latent space feature to obtain the second image feature, and reconstructs the second image feature into the first decoded image through the image generation module (unpachifide). The second loss value of the first decoded image and the original image is calculated by the cross entropy loss function CELoss, and the parameters of the encoder and decoder of the deep learning model are adjusted by reverse calculation to finally realize the training of the deep learning model. When the deep learning model training is completed, the original image is also stored in the latent space of the deep learning model in the form of the first encoding information.

[0226] By applying the method of the embodiment of the present application, the image to be stored can be encoded by the encoder of the deep learning model, the first encoded information can be stored in the latent space of the deep learning model, the first image feature after encoding can be classified by the classifier of the deep learning model, the corresponding relationship between the extraction code and the first image feature can be established, and the first latent space feature after sampling can be decoded by the decoder of the deep learning model to obtain the first decoded image. Thus, the second loss value is calculated according to the first decoded image and the image to be stored, and the first loss value is calculated according to the first classification result and the extraction code; the parameters of the deep learning model are adjusted by the first loss value and the second loss value to realize the training of the deep learning model. Compared with the prior art, the embodiment of the present application realizes the storage of the image to be stored by training the deep learning model, and the first encoded information after encoding the image to be stored is stored in the latent space of the deep learning model. The image to be stored cannot be intuitively seen on the storage device, thereby improving the security of image storage. And when acquiring an image, the embodiment of the present application also needs to input the extraction code into the trained deep learning model to obtain the image corresponding to the extraction code, which further ensures the storage security of the image data.

[0227] In a second aspect of the embodiments of the present application, an image storage device is provided, such as Fig. 9 As shown, the device comprises:

[0228] The image acquisition module 901 is used to acquire the image to be stored and the extraction code of the image to be stored.

[0229] The image encoding module 902 is used to encode the image to be stored using an encoder of a deep learning model to obtain first encoding information, and store the first encoding information in a latent space, wherein the first encoding information includes a first image feature.

[0230] The image classification module 903 is used to use the classifier of the deep learning model to classify and predict the first image feature to obtain a first classification result.

[0231] The first loss value calculation module 904 is used to calculate the first loss value of the deep learning model according to the extraction code and the first classification result.

[0232] The sampling module 905 is used to sample the latent space to obtain a first latent space feature.

[0233] The image decoding module 906 is used to decode the extracted code and the first latent space feature using a decoder of the deep learning model to obtain a first decoded image.

[0234] The second loss value calculation module 907 is used to calculate the second loss value of the deep learning model according to the first decoded image and the image to be stored.

[0235] The parameter adjustment module 908 is used to adjust the parameters of the deep learning model according to the first loss value and the second loss value.

[0236] In a possible implementation, the sampling module includes:

[0237] The sampling calculation submodule is specifically used to sample the latent space to obtain the first sampling feature information, and calculate the mean and variance of each element in the first sampling feature information.

[0238] The matrix product submodule is specifically used to obtain a preset matrix, calculate the product of the variance and the preset matrix to obtain a first matrix, wherein the preset matrix obeys a normal distribution.

[0239] The mean superposition submodule is specifically used to add the mean to each element in the first matrix to obtain the first latent space feature.

[0240] In one possible implementation, the encoder of the deep learning model includes an image mapping module and an encoding module.

[0241] Image encoding module, including:

[0242] The image convolution submodule is specifically used to use the image mapping module to perform a convolution operation on the image to be stored to obtain a first encoded image feature; wherein the first encoded image feature includes the image feature after convolution of each image block of the image to be stored.

[0243] The image encoding submodule is specifically used to encode the first encoded image feature using the encoding module to obtain first encoded information, and store the first encoded information in the latent space.

[0244] In a possible implementation, the encoding module includes a plurality of encoding submodules, each of which is connected in sequence; each encoding submodule includes a first normalization network, a first multi-head attention mechanism network, a second normalization network and a first feedforward neural network, and the image encoding submodule includes:

[0245] The first normalization unit is specifically used to normalize the input image feature to be encoded for each encoding submodule using the first normalization network of the encoding submodule to obtain a second encoded image feature, wherein when the encoding submodule is the first encoding submodule, the image feature to be encoded is the first encoded image feature; otherwise, the image feature to be encoded is the seventh encoded image feature output by the previous encoding submodule.

[0246] The encoding unit is specifically used to utilize the first multi-head attention mechanism network of the encoding submodule to extract and encode the second encoded image features to obtain the third encoded image features.

[0247] The first residual connection unit is specifically used to perform residual connection on the third encoded image feature of the encoding submodule and the image feature to be encoded to obtain a fourth encoded image feature.

[0248] The second normalization unit is specifically used to utilize the second normalization network of the encoding submodule to perform normalization processing on the input fourth encoded image feature to obtain the fifth encoded image feature.

[0249] The feature enhancement unit is specifically used to utilize the first feedforward neural network of the encoding submodule to perform feature enhancement on the input fifth encoded image feature to obtain the sixth encoded image feature.

[0250] The second residual connection unit is specifically used to perform residual connection on the sixth encoded image feature and the fourth encoded image feature to obtain the seventh encoded image feature; wherein the first image feature is the seventh encoded image feature output by the last encoding submodule, and the first encoding information at least includes the first image feature.

[0251] In a possible implementation, the first feedforward neural network includes a first fully connected subnetwork, a first activation function subnetwork, a first regularized subnetwork, a second fully connected subnetwork, and a second regularized subnetwork.

[0252] Feature enhancement unit, including:

[0253] The first linear transformation subunit is specifically used to perform a linear transformation on the fifth encoded image feature through a first fully connected sub-network to obtain a first encoded image sub-feature.

[0254] The first nonlinear transformation subunit is specifically used to use the first activation function subnetwork to perform a nonlinear transformation on the first encoded image sub-feature to obtain the second encoded image sub-feature.

[0255] The first regularization processing subunit is specifically used to use the first regularization subnetwork to perform regularization processing on the second encoded image sub-feature to obtain a third encoded image sub-feature.

[0256] The second linear transformation subunit is specifically used to use the second fully connected sub-network to perform a linear transformation on the third encoded image sub-feature to obtain a fourth encoded image sub-feature.

[0257] The second regularization processing subunit is specifically used to perform regularization processing on the fourth encoded image sub-feature through the second regularization sub-network to obtain a sixth encoded image feature.

[0258] In one possible implementation, the decoder of the deep learning model includes a decoding module and an image generation module.

[0259] Image decoding module, including:

[0260] The image decoding submodule is specifically used to use the decoding module to decode the extraction code and the first latent space feature to obtain a second image feature; wherein the second image feature includes image features after decoding of multiple image blocks.

[0261] The image generation submodule is specifically used to use the image generation module to arrange and combine the image blocks in the second image feature to generate a first decoded image.

[0262] In a possible implementation, the decoding module includes a plurality of decoding submodules, each of which is sequentially connected in order; each decoding submodule includes a third normalized network, a first fully connected network, a masked multi-head attention mechanism network, a second fully connected network, and a second feedforward neural network.

[0263] Image decoding submodule, including:

[0264] The third normalization unit is specifically used to normalize the feature to be decoded for each decoding submodule using the third normalization network of the decoding submodule to obtain a first decoded image feature; wherein, when the decoding submodule is the first decoding submodule, the feature to be decoded is the first latent space feature; otherwise, the feature to be decoded is the seventh decoded image feature output by the previous decoding submodule.

[0265] The first linear transformation unit is specifically used to use the first fully connected network of the decoding submodule to perform a linear transformation on the first decoded image feature to obtain a second decoded image feature.

[0266] The image decoding unit is specifically used to use the masked multi-head attention mechanism network of the decoding submodule to decode the second decoded image feature to obtain the third decoded image feature.

[0267] The second linear transformation unit is specifically used to use the second fully connected network of the decoding submodule to perform a linear transformation on the third decoded image feature to obtain a fourth decoded image feature.

[0268] The third residual connection unit is specifically used to perform a residual connection on the feature to be decoded and the fourth decoded image feature to obtain a fifth decoded image feature.

[0269] The second feature enhancement unit is specifically used to utilize the second feedforward neural network of the decoding submodule to perform feature enhancement on the fifth decoded image feature to obtain the sixth decoded image feature.

[0270] The fourth residual connection unit is specifically used to perform a residual connection on the fifth decoded image feature and the sixth decoded image feature to obtain a seventh decoded image feature; wherein the second image feature is the seventh decoded image feature output by the last decoding submodule.

[0271] In a possible implementation, the parameter adjustment module includes:

[0272] The first parameter adjustment submodule is specifically used to adjust the parameters of the encoder and the classifier in the deep learning model according to the first loss value.

[0273] The second parameter adjustment submodule is specifically used to adjust the parameters of the encoder and the decoder in the deep learning model according to the second loss value.

[0274] In a possible implementation, the device further includes:

[0275] The extraction code acquisition module is used to obtain the extraction code during the image reading process.

[0276] The image reading module is used to input the extraction code into the deep learning model to obtain the read image.

[0277] The device of the embodiment of the present application can encode the image to be stored by the encoder of the deep learning model, store the first encoded information in the latent space of the deep learning model, classify the encoded first image features by the classifier of the deep learning model, establish the corresponding relationship between the extraction code and the first image features, and decode the sampled first latent space features by the decoder of the deep learning model to obtain the first decoded image. Thus, the second loss value is calculated according to the first decoded image and the image to be stored, and the first loss value is calculated according to the first classification result and the extraction code; the parameters of the deep learning model are adjusted by the first loss value and the second loss value to realize the training of the deep learning model. Compared with the prior art, the embodiment of the present application realizes the storage of the image to be stored by training the deep learning model, and stores the first encoded information of the image to be stored in the latent space of the deep learning model. The image to be stored cannot be intuitively seen on the storage device, thereby improving the security of image storage. And when acquiring an image, the embodiment of the present application also needs to input the extraction code into the trained deep learning model to obtain the image corresponding to the extraction code, which further ensures the storage security of the image data.

[0278] The present application also provides an electronic device, such as Fig.10 As shown, including:

[0279] Memory 1001, used for storing computer programs;

[0280] The processor 1002 is used to implement the following steps when executing the program stored in the memory 1001:

[0281] Obtaining the image to be stored and the extraction code of the image to be stored;

[0282] Encoding the image to be stored using an encoder of the deep learning model to obtain first encoding information, and storing the first encoding information in a latent space, wherein the first encoding information includes a first image feature;

[0283] Using a classifier of a deep learning model to perform classification prediction on the first image feature to obtain a first classification result;

[0284] Calculate a first loss value of the deep learning model according to the extracted code and the first classification result;

[0285] Sampling the latent space to obtain a first latent space feature;

[0286] Decoding the extracted code and the first latent space feature using a decoder of the deep learning model to obtain a first decoded image;

[0287] Calculating a second loss value of the deep learning model according to the first decoded image and the image to be stored;

[0288] According to the first loss value and the second loss value, the parameters of the deep learning model are adjusted.

[0289] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor xx2, the communication interface, and the memory xx1 communicate with each other via the communication bus.

[0290] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0291] The communication interface is used for communication between the above electronic device and other devices.

[0292] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0293] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0294] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image storage methods are implemented.

[0295] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any image storage method in the above embodiments.

[0296] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0297] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0298] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0299] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image storage method, characterized in that: The method comprises: Obtaining an image to be stored and an extraction code of the image to be stored; Encoding the image to be stored using an encoder of a deep learning model to obtain first encoding information, and storing the first encoding information in a latent space, wherein the first encoding information includes a first image feature; Using the classifier of the deep learning model to perform classification prediction on the first image feature to obtain a first classification result; Calculating a first loss value of the deep learning model according to the extraction code and the first classification result; Sampling the latent space to obtain a first latent space feature; Decoding the extracted code and the first latent space feature using a decoder of the deep learning model to obtain a first decoded image; Calculating a second loss value of the deep learning model according to the first decoded image and the image to be stored; Adjust the parameters of the deep learning model according to the first loss value and the second loss value.

2. The method according to claim 1, characterized in that The sampling of the latent space to obtain a first latent space feature includes: Sampling the latent space to obtain first sampling feature information, and calculating the mean and variance of each element in the first sampling feature information; Obtain a preset matrix, and calculate the product of the variance and the preset matrix to obtain a first matrix, wherein the preset matrix obeys a normal distribution; The mean is added to each element in the first matrix to obtain a first latent space feature.

3. The method according to claim 1, characterized in that: The encoder of the deep learning model includes an image mapping module and an encoding module, The encoder using the deep learning model encodes the image to be stored to obtain first encoding information, and stores the first encoding information in a latent space; comprising: Using the image mapping module, a convolution operation is performed on the image to be stored to obtain a first coded image feature; wherein the first coded image feature includes image features after convolution of each image block of the image to be stored; The first coded image feature is encoded by using the encoding module to obtain the first coded information, and the first coded information is stored in the latent space.

4. The method according to claim 3, characterized in that The encoding module includes a plurality of encoding submodules, each of which is connected in order; each of the encoding submodules includes a first normalization network, a first multi-head attention mechanism network, a second normalization network and a first feedforward neural network, and the encoding module is used to encode the first encoded image feature to obtain the first encoded information, and the first encoded information is stored in the latent space, including: For each encoding submodule, using the first normalization network of the encoding submodule, normalizing the input image feature to be encoded to obtain a second encoded image feature, wherein when the encoding submodule is the first encoding submodule, the image feature to be encoded is the first encoded image feature; otherwise, the image feature to be encoded is the seventh encoded image feature output by the previous encoding submodule; Using the first multi-head attention mechanism network of the encoding submodule, feature extraction and encoding are performed on the second encoded image feature to obtain a third encoded image feature; Performing a residual connection on the third coded image feature of the coding submodule and the image feature to be coded to obtain a fourth coded image feature; Using the second normalization network of the encoding submodule, normalizing the input fourth encoded image feature to obtain a fifth encoded image feature; Using the first feedforward neural network of the encoding submodule to perform feature enhancement on the input fifth encoded image feature to obtain a sixth encoded image feature; The sixth encoded image feature and the fourth encoded image feature are residually connected to obtain the seventh encoded image feature; wherein the first image feature is the seventh encoded image feature output by the last encoding submodule, and the first encoding information at least includes the first image feature.

5. The method according to claim 4, characterized in that The first feedforward neural network includes a first fully connected sub-network, a first activation function sub-network, a first regularization sub-network, a second fully connected sub-network and a second regularization sub-network, The step of using the first feedforward neural network to perform feature enhancement on the input fifth encoded image feature to obtain a sixth encoded image feature includes: Performing a linear transformation on the fifth encoded image feature through the first fully connected sub-network to obtain a first encoded image sub-feature; Using the first activation function sub-network, performing a nonlinear transformation on the first encoded image sub-feature to obtain a second encoded image sub-feature; Using the first regularization sub-network, regularize the second encoded image sub-feature to obtain a third encoded image sub-feature; Using the second fully connected sub-network, linearly transform the third encoded image sub-feature to obtain a fourth encoded image sub-feature; The fourth encoded image sub-feature is regularized by the second regularization sub-network to obtain the sixth encoded image feature.

6. The method according to claim 1, characterized in that The decoder of the deep learning model includes a decoding module and an image generation module. The method of decoding the extraction code and the first latent space feature by using the decoder of the deep learning model to obtain a first decoded image includes: Using the decoding module, the extraction code and the first latent space feature are decoded to obtain a second image feature; wherein the second image feature includes image features after decoding of multiple image blocks; The image generation module is used to arrange and combine the image blocks in the second image feature to generate the first decoded image.

7. The method according to claim 6, characterized in that The decoding module includes a plurality of decoding submodules, each of which is connected in sequence; each of the decoding submodules includes a third normalized network, a first fully connected network, a masked multi-head attention mechanism network, a second fully connected network, and a second feedforward neural network, The step of decoding the extraction code and the first latent space feature using the decoding module to obtain a second image feature includes: For each decoding submodule, using the third normalization network of the decoding submodule, normalizing the feature to be decoded to obtain a first decoded image feature; wherein, when the decoding submodule is the first decoding submodule, the feature to be decoded is the first latent space feature; otherwise, the feature to be decoded is the seventh decoded image feature output by the previous decoding submodule; Using a first fully connected network of the decoding submodule, linearly transform the first decoded image feature to obtain a second decoded image feature; Decoding the second decoded image feature using the masked multi-head attention mechanism network of the decoding submodule to obtain a third decoded image feature; Using the second fully connected network of the decoding submodule, linearly transform the third decoded image feature to obtain a fourth decoded image feature; Performing a residual connection between the feature to be decoded and the fourth decoded image feature to obtain a fifth decoded image feature; Using the second feedforward neural network of the decoding submodule to perform feature enhancement on the fifth decoded image feature to obtain a sixth decoded image feature; The fifth decoded image feature is residually connected with the sixth decoded image feature to obtain the seventh decoded image feature; wherein the second image feature is the seventh decoded image feature output by the last decoding submodule.

8. The method according to claim 1, characterized in that The adjusting the parameters of the deep learning model according to the first loss value and the second loss value includes: According to the first loss value, adjusting parameters of the encoder and the classifier in the deep learning model; According to the second loss value, adjust the parameters of the encoder and the decoder in the deep learning model.

9. The method according to claim 1, characterized in that: The method further comprises: During the image reading process, obtaining the extraction code; The extraction code is input into the deep learning model to obtain the read image.

10. An image storage device, characterized in that: The device comprises: An image acquisition module, used to acquire an image to be stored and an extraction code of the image to be stored; An image encoding module, configured to encode the image to be stored using an encoder of a deep learning model to obtain first encoding information, and store the first encoding information in a latent space, wherein the first encoding information includes a first image feature; An image classification module, used to classify and predict the first image feature using the classifier of the deep learning model to obtain a first classification result; A first loss value calculation module, used to calculate a first loss value of the deep learning model according to the extraction code and the first classification result; A sampling module, used for sampling the latent space to obtain a first latent space feature; An image decoding module, configured to decode the extraction code and the first latent space feature using a decoder of the deep learning model to obtain a first decoded image; A second loss value calculation module, used to calculate a second loss value of the deep learning model according to the first decoded image and the image to be stored; A parameter adjustment module is used to adjust the parameters of the deep learning model according to the first loss value and the second loss value.

11. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-9 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images