Image set compression with knowledge redundancy

Abstract

A computer-implemented method for scalable compression of digital images. The method comprises the following steps of: extracting semantic information from an image on a semantic layer; extracting structural information from the image on the structural layer; extracting signal information from the image on the signal layer; and compressing each of the semantic information, the structure information, and the signal information into a bitstream. Therefore, a novel scalable cross-mode image compression technique is provided, which has a wide range of novel functionality, making a codec available for various applications from semantic understanding to signal-level reconstruction, and the like.

Description

Technical Field

[0001] The present invention relates to compression of digital images, and in particular to compression of image sets based on deep learning. Background Art

[0002] The widespread use of handheld cameras and mobile phones has made it more convenient than ever to take and share photos, which has also led to a massive increase in the number of digital images taken and shared in recent years. This trend has led to an explosive growth in the number of image collections, posing huge challenges to their storage, backup, and maintenance. Therefore, the development of efficient image collection compression algorithms is an urgent need.

[0003] The current popular compression scheme is to compress a single image. However, this approach only reduces the redundancy within the image, with the goal of achieving a compact representation of each individual image. However, in the case of image set compression, the redundancy that exists is far more than the redundancy within the image, which indicates that different images in the image set have similarities at multiple granularities. This shared information can be further exploited by removing the redundancy between images. This has prompted people to develop compression strategies by developing pseudo video coding methods, because a series of images in a set can form a pseudo video. However, traditional video-based solutions rely on existing compression schemes, which are usually designed to eliminate temporal redundancy through signal-level comparison. However, this strategy may not be suitable for image sets because the images in the image set are not acquired in the same way as the video (for example, the images in the image set are captured from different scenes with different backgrounds, camera perspectives, and settings). Therefore, the traditional method of removing redundancy at the signal level is inefficient and cumbersome. Summary of the invention

[0004] The present invention provides, in one aspect, a computer-implemented method for compressing a digital image set, the image set comprising a plurality of original images. The method comprises the following steps: decomposing each of the plurality of original images into three parts: common knowledge, a unique mask, and a unique texture; and compressing each of the common knowledge, the unique mask, and the unique texture into a bitstream, respectively. The common knowledge represents shared features between the plurality of original images.

[0005] Preferably, the common knowledge is extracted from a master reference image among the plurality of original images.

[0006] More preferably, the main reference image is determined by the following steps: determining a unique mask for each of a plurality of original images; multiplying each of the plurality of original images by the unique mask corresponding thereto to obtain a plurality of object images; establishing a directed graph based on the similarity of features extracted from the plurality of object images; processing the directed graph using a minimum spanning tree algorithm; and taking the node with the largest number of edges in the spanning tree as the main reference image.

[0007] In an exemplary embodiment, the step of compressing the common knowledge further includes using a trained reversible neural module to align the common knowledge with multiple images at a spatial level to eliminate redundancy in the common knowledge.

[0008] In an exemplary embodiment, the above-mentioned decomposition step further comprises: 2 -Net saliency detection method and alpha matting technique to extract a unique mask from each of multiple original images.

[0009] In an exemplary embodiment, the compression step further includes: forming the unique masks of the plurality of original images into a pseudo mask video in a predetermined order; and compressing the pseudo mask video using a video codec to achieve compression of the unique mask.

[0010] Preferably, the predetermined order is a depth-first search order.

[0011] In an exemplary embodiment, the compressing step further comprises compressing the unique texture given the public knowledge and the unique mask.

[0012] Preferably, the step of compressing the unique texture also includes: using a trained reversible neural module to align common knowledge with multiple images at the spatial level to eliminate redundancy in the common knowledge; using the aligned common knowledge and a unique mask of each of the multiple images to enhance the texture of the region of interest in each of the multiple images; and performing a reversible activation transformation on each of the multiple images.

[0013] Preferably, the alignment step is a coarse to fine alignment.

[0014] More preferably, the aligning step comprises performing a homography transformation on each of the plurality of images; and performing a progressive optical flow estimation on each of the plurality of images.

[0015] In one exemplary embodiment, the reversible neural module is trained by using a training pair; the training pair includes an original object image in a training image set and another decoded object image in the same training image set.

[0016] In another aspect, the present invention provides a computer-implemented method for reconstructing a digital image set from a plurality of bitstreams including a common knowledge stream, a unique mask stream, and a unique texture stream. The method comprises the following steps: decoding common knowledge from the common knowledge stream; decoding a plurality of unique masks corresponding to a plurality of images in the digital image set from the unique mask stream; decoding a plurality of unique textures corresponding to a plurality of images in the digital image set from the unique texture stream; and reconstructing the digital image set using the common knowledge, the plurality of unique masks, and the plurality of unique textures.

[0017] Preferably, the step of decoding a plurality of unique textures further comprises decoding the unique textures given the decoded common knowledge and the unique mask.

[0018] Preferably, the method further comprises using a post-processing model to enhance the perceived quality of the reconstructed digital image set using the common knowledge of the alignment as a guide.

[0019] In an exemplary embodiment, the step of enhancing the perceptual quality of the reconstructed digital image set also includes: using a trained reversible neural module to align the decoded common knowledge with multiple images in the reconstructed digital image set at the spatial level to eliminate redundancy in the common knowledge; mapping the aligned common knowledge to a mapped image consistent with the spatial information of the multiple images; using a convolutional layer to extract features of the mapped image; and inputting the features into a polarized self-attention module.

[0020] Preferably, the alignment step is a coarse to fine alignment.

[0021] More preferably, the aligning step comprises: performing a homography transformation on each of the plurality of images; and performing a progressive optical flow estimation on each of the plurality of images.

[0022] In one exemplary implementation, the post-processing model is based on the Restormer model.

[0023] On the other hand, the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the above-mentioned computer-implemented method for compressing a digital image set.

[0024] On the other hand, the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the above-mentioned method for reconstructing a digital image set from multiple bit streams including a common knowledge stream, a unique mask stream and a unique texture stream.

[0025] Therefore, an embodiment of the present invention proposes a solution for eliminating redundancy in an image set at the knowledge level, wherein the knowledge characterizes the high-level relationship and correlation between images, specifically, each image in the image set is decomposed into three components: common knowledge, a unique mask, and a unique texture. The common knowledge shared by all images is used to reduce redundancy between images. For each image, a unique mask is extracted to form a video sequence, which is then compressed by a video codec based on a feature-based minimum spanning tree and a depth-first search algorithm. Given the common knowledge and the unique mask, the unique texture is compressed. The present invention is efficient, flexible, and scalable, and can effectively eliminate the set redundancy between images (even if there is only a low correlation between the images) and the redundancy within a single image. Experimental results show that the solution proposed in the embodiment of the present invention achieves better performance in terms of signal quality, perceptual quality, and high accuracy of downstream tasks.

[0026] The above summary is neither intended to define the claimed invention (which is recited by the claims) nor to limit the scope of the invention in any way. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The above and further features of the present invention will be readily apparent from the following description of embodiments. The description of these embodiments is given by way of example only in conjunction with the accompanying drawings, in which:

[0028] Figure 1 A schematic diagram of the structure of an image set compression and decompression scheme according to a first embodiment of the present invention is shown.

[0029] Figure 2 The specific steps of compressing the unique mask in the scheme of FIG. 1 are shown.

[0030] Figure 3 Shows Figure 1 The structure and training steps of the coarse-to-fine alignment module in our scheme.

[0031] Figure 4 Shows Figure 1 The knowledge in the scheme guides the main framework of the texture compression and decompression process.

[0032] Figure 5 Shows Figure 1 The structure of the mask feature extraction module and the down-conversion module in the scheme.

[0033] Figure 6 Shows Figure 1 The structure of the pseudo-inverse knowledge attention module in the scheme.

[0034] Figure 7 Shows Figure 1The structure of the reversible texture attention module in our scheme.

[0035] Figure 8 Shows Figure 1 The framework of the knowledge-guided post-processing model in the scheme.

[0036] Figure 9a is based on Figure 1 A neural network prototype of our scheme is used in experiments, using the CUB-200-2011 dataset as input, and a comparison graph of PSNR with the existing technology.

[0037] Figure 9b yes Figure 9a Comparison graph of DISTS of the neural network prototype in the experiment, using the CUB-200-2011 dataset as input compared with the existing technology.

[0038] Fig.9c yes Figure 9a Comparison graph of the accuracy (%) of the neural network prototype in the experiment, using the CUB-200-2011 dataset as input compared with the existing technology.

[0039] Figure 9d yes Figure 9a A graph comparing the PSNR of the neural network prototype in an experiment, using the Oxford Flower Dataset as input, compared with the existing technology.

[0040] Fig.9e yes Figure 9a Comparison graph of DISTS of the neural network prototype in experiments, using the Oxford Flower Dataset as input, compared with the state of the art.

[0041] Figure 9f yes Figure 9a Comparison graph of the accuracy (%) of the neural network prototype in the experiment, using the Oxford Flower Dataset as input, compared with the existing technology.

[0042] Fig.10 Yes Figure 9a Visual comparison of different methods including neural network prototypes on the CUB-200-2011 and Oxford Flower datasets.

[0043] In the specification and drawings, like numerals represent like parts in the several embodiments described. DETAILED DESCRIPTION

[0044] Figure 1 is a demonstration of an image set compression and decompression scheme according to an embodiment of the present invention. Specifically, Figure 1The digital image set 20 in the video image processing apparatus includes a plurality of original images. The degree of association between these images can be arbitrary: they can be unrelated to each other or have a certain degree of association. For example, the image set 20 can contain images with different backgrounds, and their inter-frame correlation is relatively low. Each of the plurality of original images in the image set 20 is decomposed into three parts before compression: common knowledge 22a, unique mask 24a and unique texture 26a, and these three parts are respectively encoded / compressed into three bit streams for network transmission, namely common knowledge stream 22b, unique mask stream 24b and unique texture stream 26b. Figure 1 The trapezoidal symbols on the left and right of the common knowledge stream 22b, the unique mask stream 24b and the unique texture stream 26b in FIG. 2 represent the encoder and the decoder, respectively. Figure 1 Medium The decoded unique mask 24a is Figure 1 Medium It represents the i-th unique mask In addition, the decoded unique texture 26a is Figure 1 Medium It represents the i-th unique texture The decoded version of . Then it is input into the post-processing module 28, and the final reconstructed image is obtained after processing. It represents the final reconstructed version of the i-th image. All images The combination becomes a reconstructed version of the digital image set 20.

[0045] The common knowledge 22a represents the shared features among the multiple original images, and is therefore unique to the image set 20. A typical example of a shared feature is Figure 1 The multiple images in the image set 20 shown depict a bird photographed in different positions and postures, and the common knowledge 22a is the representation of the bird itself in the feature space. The fact that the entropy of the image signal is greater than or equal to the conditional entropy of the common knowledge proves that Figure 1 The feasibility of the image set compression method shown in FIG. 2 is a schematic diagram showing a method for compressing an image set 20. As will be described in detail later, the common knowledge 22a shared by all images in the image set 20 reduces redundancy between images, thereby improving compression performance. On the other hand, the unique mask 24a corresponds one-to-one to each image in the image set 20. The unique mask 24a represents the layout of the image to avoid misalignment and areas of particular interest. Finally, the unique texture 26a also corresponds one-to-one to each image in the image set 20. The unique texture 26a is compressed with the common knowledge 22a and the unique mask 24a as conditions, which further reduces the representation cost of representing a specific image.

[0046] The following will introduce the extraction / compression methods of the public knowledge 22a, the unique mask 24a and the unique texture 26a. In general, the public knowledge 22a and the unique mask 24a are compressed independently, while the unique texture 26a is compressed under the condition of given public knowledge 22a and unique mask 24a.

[0047] First, the common knowledge 22 a may be extracted from a main reference object image among a plurality of images in the image set 20 . Figure 2 The workflow for determining the master reference object image and the unique mask I m Specifically, first, in step 30, a unique mask I of each of the multiple images in the image set 20 is extracted. m (i.e., unique mask 24a, and the specific extraction method is described in detail below), and in step 32, each image is multiplied by its corresponding unique mask I m To obtain multiple object images I o Then, based on the images from these objects I o The similarity of the features extracted from the images is used to build a directed graph 34, and the minimum spanning tree (MST) algorithm is applied to obtain the best prediction order. Specifically, the node with the largest number of edges in the MST is considered to be the root node 36, i.e., the main reference object image. Finally, the public knowledge F can be extracted from the main reference object image. c (Right now Figure 1 22a) and compressed using the HiFi-VRIC model [4].

[0048] Let's look at the only mask I m In order to effectively compress the unique mask I m They are usually formed into video sequences by the above-mentioned MST of object images, coupled with a depth-first search (DFS) algorithm, and then compressed by any existing video codec. Specifically, Figure 2 As shown, first, each of the plurality of images in the image set 20 as described above is subjected to a unique mask I in step 30. m In a preferred embodiment, U 2 -Net saliency detection method

[21] and alpha matting technique are used to extract masks from images

[10] , which can robustly detect important regions in each image [4]. Subsequently, the saliency mask can separate objects in the foreground and background of the image. In this way, through the Hadamard product, that is, As described above, in step 32, a mask image can be obtained. and object image Where index i represents the i-th image in the image set.

[0049] Mask image There is a strong spatial and image-level correlation, so one of the most effective compression methods is to form a video and compress it through a video codec. For video coding, obtaining an ideal prediction path can reduce the overall rate-distortion cost. Therefore, it is necessary to arrange the images into a video according to the prediction path, such as Figure 2 In order to determine the ideal prediction order, the directed graph 34 is introduced, which can represent the correlation between multiple images in the image set 20. o , E), each node Represents an object image, each edge e i,j ∈E represents the cost between the i-th and j-th object images. According to graph optimization theory

[24] , the directed subgraph of the MST of G has the minimum overall rate-distortion coding cost. However, finding an ideal MST generation method with a total rate-distortion cost is an NP-hard problem. The cost is usually estimated by a prediction metric between two surrounding images (e.g., root mean square error RMSE or structural similarity SSIM)

[22] .

[0050] Here, the preferred embodiment of the present invention proposes a prediction measure based on VGG (Visual Geometry Group, University of Oxford), using the VGG model to convert the image into a compact representation, i.e., a feature vector of size 512×1. The distance between two nodes is represented by the MSE cost. The proposed method can be used to implement a feature-based prediction tree that minimizes the total feature distance

[13] . The tree is scanned in DFS order. Then, the random mask images can be organized into a high-correlation video with YUV400 format. Preferably, the VVC test model (VTM) [1] with QP=58 is adopted. Afterwards, the unique mask I m according to Figure 2 The DFS prediction order 40 shown is further formed into a pseudo mask video 42 in step 38. The pseudo mask video 42 is then compressed using an existing video codec (eg, VVC) to become a unique mask bitstream 24b.

[0051] Now let's look at the unique texture 26a. As mentioned above, the multiple unique textures 26a in the image set 20 are obtained by and unique mask The compression is carried out under the guidance of and Public knowledge F c and the i-th unique mask Here, the i-th image in the image set 20 is defined as In order to make public knowledgec Aligned with the input image (i.e., multiple original images in the image set 20), a preferred embodiment of the present invention proposes a coarse-to-fine alignment strategy. In addition, a preferred embodiment of the present invention also incorporates reversible neural modules, including a pseudo-inverse knowledge attention module (IKA) and a reversible texture attention module (ITA), into the compression model of the unique texture [27, 4] to solve the problem of information loss. In the decoding process of the bitstream, in order to obtain a better human perception effect, a preferred embodiment of the present invention proposes a post-processing model that uses the common knowledge of alignment as a guide to enhance the perceptual quality of the image. The above-mentioned various modules / models will be described in detail below.

[0052] Align modules from coarse to fine

[0053] First, let’s look at the coarse-to-fine alignment module that implements the above coarse-to-fine alignment strategy. Figure 3 In general, video compression schemes rely on an assumption that the foreground and background have high correlation between frames, so inter-frame prediction can effectively reduce redundancy and improve compression performance. However, the situation is different for image sets, because image sets usually contain images with different backgrounds, resulting in relatively low inter-frame correlation. Nevertheless, it is worth noting that each image inherently contains a certain degree of shared information that can be predicted by decoding common knowledge. In order to achieve the goal of aligning the shared information in each image at the spatial level, a coarse-to-fine alignment method is proposed. Specifically, Figure 3 The coarse-to-fine alignment module includes a homography transformation module 44 and a progressive optical flow estimation module 46, and Figure 3 shows the training process of the coarse-to-fine alignment module. Figure 3 middle, is the original object image in the image set, and is another decoded object image in the same image set. We randomly select the i-th and j-th images as training pairs for training a more robust model. During training, the i-th original object image and the jth decoded object image Input to the homography transformation module 44 for global transformation, such as Figure 3 This results in the image output by the homography transformation module 44 and A rough alignment is achieved. Then, the image With image is used as input to the progressive optical flow estimation module 46, which predicts a more refined object image after the (k-1)th iteration In this process, after the first iteration, the input image More detailed object images replace.

[0054] The above-mentioned homography transformation module 44 is intended to mitigate geometric distortions caused by camera and position variations in capturing images. Images are captured from a range of viewpoints and different camera settings. Inconsistencies in light quality, such as mode variations, lead to challenges in image prediction. Therefore, homography prediction is the first step from a global perspective. In order to achieve a balance between stability and geometric deformation performance, a six-degree-of-freedom (6-DOFs) homography, also known as an affine transformation

[18] , is selected in a preferred embodiment. However, due to possible changes in camera settings and image deformation, matching image key points and constructing homographies can be extremely challenging in complex scenes. To address this challenge, a preferred embodiment of the present invention draws on the "homography flow" representation

[16] and estimates the homography matrix for each coordinate. It estimates the position-dependent homography matrix T p , to minimize the deformation error between the ground control pixel p and its corresponding pixel p′.

[0055]

[0056] Among them, x p and x p， are the pixel coordinates, T p ∈R 3×3 The affine transformation at pixel point p is defined. Here,

[0057]

[0058] Where G is a 2×2 geometry matrix that describes scale, rotation, and shear changes. t is a 2×1 translation vector, and 0 1×2 is a 1×2 matrix of zeros.

[0059] Although simple homography can effectively reduce geometric distortion, inter-frame prediction suffers from smaller local deformations, such as local displacements and holes. In order to address the limitations of homography, a preferred embodiment of the present invention introduces a progressive optical flow estimation model, which uses FlowNet 2.0

[12] to estimate local optical flow, which is implemented by the above-mentioned progressive optical flow estimation module 46. Optical flow estimation is a basic step in image prediction, which requires precise positioning of each pixel and identification of the correspondence between two input images. Although there are many advanced optical flow estimation algorithms, due to its stability and reliability, the above-mentioned preferred embodiment chooses to adopt an iterative strategy based on FlowNet 2.0 to improve fine image prediction, which can solve the optical flow estimation problem as a supervised learning task. Specifically, for the progressive optical flow estimation module 46, based on the previous distorted image The distorted image after the Kth iteration is repeatedly input into FlowNet 2.0 for more accurate motion compensation, where K represents the Kth iteration. Ultimately, the method produces the final predicted object image It has improved prediction accuracy.

[0060] The training of the homography transformation module 44 and the progressive optical flow estimation module 46 is carried out in two stages. The optimization loss of the two stages is shown in the following formula (3). In the first stage, the homography transformation module 44 is trained. In the second stage, the homography transformation module 44 and the progressive optical flow estimation module 46 are optimized simultaneously under the constraints of formula (3).

[0061]

[0062] Among them, λ h and λ k is the weight. In one example, λ h =10,λ k =10×(k+1). After the training is completed, the homography transformation module 44 and the progressive optical flow estimation module 46 are applied to the knowledge-guided unique texture compression module and the post-processing module, which will be described in detail below.

[0063] Knowledge-guided unique texture compression module

[0064] In a traditional autoencoder, information loss occurs when an image is converted to a low-dimensional latent space, and some information is irreversibly discarded before quantization. In order to solve the information loss problem, a preferred embodiment of the present invention utilizes a method based on an INN (invertible neural network) model, which Figure 4It is shown that this is a mathematically reversible way to help preserve information. INN is a generative model that simplifies complex distributions and can accurately and efficiently estimate probability density. Integrating the concept of INN into the encoding-decoding process may be a feasible solution to the information loss problem. Such input-output bijection and strictly reversible properties are suitable for image compression. In addition, INN demonstrates fine variable rate control capabilities while better maintaining image fidelity.

[0065] Figure 4 The figure shows the process of texture compression (hereinafter referred to as the forward process, the main data flow in the forward process follows the arrows marked with v, such as Figure 4 , 6 -7), and also shows the process of texture decompression (hereinafter referred to as reverse processing, the main data flow in the reverse processing follows the arrows marked with O, such as Figure 4 and Figure 6 In the forward process, the i-th unique mask The decoded version of The mask feature extraction module 48 is input to generate a plurality of mask features m1, m2, m3 and m4. These m1, m2, m3 and m4 are respectively input to a plurality of ITA modules 50, which are serially connected in the forward direction (including the reverse direction). (i.e., the unique texture 26a in FIG. 1) is input to the IKA module 52, which in turn receives the public knowledge F c The decoded version of and the i-th unique mask The decoded version of As input and guidance. Figure 4 Two seemingly independent IKA modules 52 are shown in FIG. 5 , but in fact, the encoder and decoder of the image set can be completed by only one IKA module 52 because the encoder and decoder can be the same neural network. Figure 4 The operation of the two IKA modules 52 shown in FIG. 1 is shown in FIG. 2 . In the forward process, the output of the IKA module 52 is input to the reversible block 54, followed by the ITA module 50 and the IAT module 56. There are a total of four such "reversible block 54-ITA module 50-IAT module 56" combinations, and these four combinations are connected in series. The output of the last IAT module 56 in the forward direction is provided to the channel attention squeeze module 58. The output of the channel attention squeeze module 58 is the texture potential representation y, which is then passed through the quantizer 60 to obtain the quantized potential representation The potential representation of this quantization Finally, it is encoded into a unique texture stream 26b by an entropy encoder 62. During the encoding process, the entropy encoder 62 receives the output μ, σ from the context entropy model 66, and the input of the context entropy model 66 is the texture potential representation y mentioned above.

[0066] Figure 5 In its upper part it shows Figure 4 The structure of the mask feature extraction module 48 in the upper part is shown in the figure, and the structure of the down conversion module in the mask feature extraction module 48 is shown in the lower part. Specifically, the decoded version of the only mask as the input of the mask feature extraction module 48 is The convolution operation 68 and the down-conversion operation 70 are performed four times in succession. After each convolution operation 68 and the down-conversion operation 70, a mask feature is generated. After four such operations, the mask feature extraction module 48 extracts a multi-scale mask feature m through the convolution operation and the down-conversion operation. i (i={1, 2, 3, 4}). In each down-conversion module, the input of the module is sequentially subjected to a first convolution operation 71, a maximum pooling operation 72, a second convolution operation 73, and a rectified linear unit (ReLU) operation 74 as an activation function. The data processed by these four steps is then accumulated (as a shortcut connection) with the data output by the first convolution operation 71 as the output of the down-conversion module. After the down-conversion operation, the feature resolution is reduced by two times. In an example, the first convolution operation 71 of the down-conversion module is a convolution operation with a step size and a kernel size of 3. The maximum pooling operation 72 is a maximum pooling operation of size 2. The second convolution operation 73 is a convolution operation with a step size of 1 and a kernel size of 3. It should be noted that the above-mentioned feature m i It is available and shared in both the forward process (encoder) and the backward process (decoder).

[0067] In a preferred embodiment, the above-mentioned IKA module 52 and ITA module 50 are based on the HiFi-VRIC model [4]. The IKA module 52 effectively reduces the redundancy of public knowledge, while the ITA module 50 pays special attention to the texture reconstruction of the region of interest. Figure 6 Shown, the only texture As the input of this module, u is obtained through the dense block (DenseBlock) module 76 t , then u t is input to the coarse-to-fine alignment module 78 as described above. The input to the coarse-to-fine alignment module 78 also includes the i-th unique mask The decoded version of and public knowledge c The decoded version of The latter undergoes a 3×3 convolution operation 77 and then a dense block module 76 to obtain f d , which is then input to the coarse-to-fine alignment module 78. Note that in the reverse process, the public knowledge F c The decoded version of The processing is similar, but the final output is It is then input to the coarse to fine alignment module 78. The output of the coarse to fine alignment module 78 is u w , then u w with u t The sum is obtained to obtain u, which is the output of the IKA module 52.

[0068] Due to the public knowledge of decoding and unique texture has redundancy, so public knowledge can be distorted to make it Spatial alignment is performed to eliminate redundancy in public knowledge. In order to solve the problem of information loss, it is hoped to build a reversible module. Head. Then forward processing (u=u t +u w ) and reverse processing is reversible. From the input Extracted from . w includes public knowledge and is the output of CFA(*), where CFA(*) is the coarse-to-fine alignment module mentioned above. In fact, because And from Figure 6 It can be seen from the structure that Therefore, the constraints are relaxed and a pseudo-inverse process is constructed, that is, constraining u in the optimization w Close to In addition, the f extracted from different dense block modules 76 is relaxed d and , because we only need to satisfy constraints.

[0069] Let's look at the ITA module 50. Its structure is Figure 7 As described above, the ITA module 50 enhances the encoding of the region of interest because it assigns more weight to the texture reconstruction of the region of interest. For the ITA module 50, it is applied before the invertible activation transform (IAT) module 56 in the HiFi-VRIC model. Figure 7 The top The labeled arrows illustrate the forward transformation of the ITA module 50. The input of the ITA module 50 includes image features Mask Features Characteristics of Kazuichi These texture attention transforms are applied to the features f via formula (4) i , to recursively generate features f i+1 , as shown below:

[0070]

[0071] The symbol ⊙ represents the Hadamard product. Figure 7 The arrow marked with O at the bottom illustrates the reverse transformation of the ITA module 50. This reverse transformation is described as follows:

[0072]

[0073] The symbol represents element-wise division. Here, m i have the same value in the encoder and decoder so that the feature f i and f i+1 The training loss is as follows:

[0074]

[0075] in, and are the quantized latent space representation, auxiliary information, input image and decoded image respectively. T = C × H × W, where C, H and W are the input image I s The number of channels, height and width. Λ can be calculated directly from the quality level L through a monotonically increasing function: Λ = V(L), where V: V(L)=θ×e τ×L , θ=0.0012, τ=4.382, from The resizing process is done by direct channel copying [4]. In addition, α and λ have a high positive correlation with the quality level L. Empirically, α i =0.5*λ i Furthermore, an exemplary embodiment of the present invention compresses the common knowledge 22a in the image set by means of the codec proposed in [4], while the unique textures 22a are compressed by means of the codec specifically described above. (Right now Figure 1 Since the common knowledge 22a is shared by all images in the image set 20, its quality is better than higher.

[0076] Knowledge-guided post-processing module

[0077] In order to improve the perceived quality of the image, for the decoding process, a preferred embodiment of the present invention proposes a knowledge-guided post-processing model

[12] based on the Restormer model

[29] , which can be used, for example, Figure 1 The post-processing model uses the decoded public knowledge as a guide to improve the visual quality of the final image. Specifically, after performing coarse-to-fine alignment, the public knowledge is mapped to the image I that is consistent with the spatial information of the picture. w Then, the features of these common knowledge are extracted using convolutional layers and concatenated and input into the polarized self-attention (PSA) module. The polarized self-attention module is designed to maintain high internal resolution in channel and spatial attention calculations as described in

[15] . The feature extraction process is as follows Figure 8 The decoded image as input It passes through a 3×3 convolutional layer and then enters the transformation block. It is concatenated with the features processed by the transformation block and then passes through a 1×1 convolutional layer to adjust the number of channels. It is extracted from the edge by the Sobel filter. The extracted features are combined with the features after the up-conversion (Up) module and the features after the transformation block. In order to ensure the visual quality of the image, a preferred embodiment of the present invention introduces a new loss function, namely the deep image structure and texture similarity (DISTS) loss [8] to further enhance the input image and the image enhanced by the post-processing module (I p ). By combining and The intrinsic similarity between the input image and the generated image can be greatly improved, thus promoting the formation of concept representation.

[0078]

[0079] in, and They are mean square error loss, multi-scale structural similarity loss and DISTS loss. m ,λ s and λ d is a weight parameter used to balance each part. In one example, we empirically set λ m =500,λ s =2 and λ d =10 is set as the weight parameter.

[0080] Therefore, in the above combination Figure 1-8 After introducing the image set compression and decompression scheme according to the embodiment of the present invention, it can be seen that the compression (encoding) and decompression (decoding) processes are basically the opposite of each other, and the models and modules used are basically the same. In particular, Figure 4 The process of unique texture compression shown in is completely reversible, and its inversion is the process of unique texture decompression. Similarly, Figure 6 The IKA module in Figure 7 The respective processing processes of the ITA modules in are also completely reversible, the forward processing can be used for encoding, and the reverse processing can be used for decoding. In the case where the forward processing is described in detail above, the reverse processing (decoding) is not described for the sake of brevity, and the post-processing module unique to the decoding process has been described in detail above.

[0081] Next, based on the above Figure 1-8 An experiment is designed to introduce the image set compression and decompression scheme according to an embodiment of the present invention. The experiment uses two datasets for training and testing: the CUB-200-2011 dataset

[25] and the Oxford Flower dataset

[19] . The CUB-200-2011 dataset contains 8855 training images and 2933 test images, while the Oxford Flower dataset contains 2040 training images and 6149 test images. All training images are combined to optimize the model and are resized and center-cropped to 256×256. The quality evaluation measures of the experiment include using the peak signal-to-noise ratio (PSNR) to evaluate the performance of knowledge-guided unique texture compression. PSNR is a widely used metric in compression tasks to estimate the fidelity of pixels. However, considering that PSNR cannot reflect the visual quality well [14, 30, 8], DISTS [8] is further used in the experiment as a quality evaluation metric after the post-processing module. In particular, the lower the DISTS value, the better the quality. The encoding bit rate is evaluated as bits per pixel (bpp). In the experiments, the deep learning network was implemented in the PyTorch framework and trained on an NVIDIA GeForce RTX 3090 GPU.

[0082] In order to verify the effectiveness of the scheme involved, the performance of the following image compression schemes is compared. The comparison results are shown in Figures 9a-9fIn the figure, a neural network prototype of the scheme of an embodiment of the present invention according to FIG. 1 is referred to as “the present invention” in the figure and in the following description. Some state-of-the-art models are selected for comparison, including VVC, the method of Ballé et al. [3], the method of Minnen et al.

[17] , and the method of Cai et al. [4]. Here, the VVC test model (VTM-15.2) is adopted, and the quantization parameters QPs = {47, 42, 37, 32, 27, 22} are set under the full internal (AI) setting, and higher QP values ​​correspond to higher compression rates. In addition, the images are organized into a pseudo video through the prediction order of the MST and DFS algorithms, and compressed under the low latency B (LDB) configuration, named VTM-LDB (MST). For the method of Ballé et al. [3] and the method of Minnen et al.

[17] , their training and testing strategies follow the strategies provided by CompressAI (https: / / github.com / InterDigitalInc / CompressAI). For the method of Cai et al. [4], the training and testing strategies follow the official code of HiFi-VRIC (https: / / github.com / CaiShilv / HiFi-VRIC).

[0083] In the comparison of signal quality, all images in the test set are compressed using different quality factors. Figure 9a and 9d The performance comparison of rate-distortion (RD) is shown, and PSNR is used as the quality evaluation index to compare the signal quality. It can be seen that the scheme of the present invention can achieve good performance in terms of signal quality, especially at high coding rates.

[0084] Considering that PSNR cannot reflect the visual quality well, DISTS is used as the quality evaluation index to compare the perceptual quality in the comparison of visual quality. Figure 9b and 9e shows the performance comparison of DISTS. The results produced by the knowledge-guided post-processing model can achieve a good effect. The visualization results are shown in Fig.10 As shown in Figure 2. The images in the same column come from the same model. Fig.10 In the present invention, " represents the result of unique texture compression, while "the present invention (I p )” represents the output of knowledge-guided unique texture compression. The numerical value below each image is bpp / PSNR / DISTS, where higher PSNR values ​​represent better signal quality and lower DISTS values ​​mean better perceptual quality. With the knowledge-guided post-processing model, the images show better perceptual effects, showing more textures in birds and flowers. Visual comparison shows that the post-processing module promotes the reconstruction of common knowledge.

[0085] To evaluate the effectiveness of our proposed framework for downstream tasks, a classification method is employed, which shows high accuracy on original images in the testing dataset. c The results shown in 9f and 9f use the accuracy (%) as the evaluation index. The results of the present invention show a higher accuracy, especially at low bit rates, indicating that the method of the present invention has robustness and high-quality performance.

[0086] Ablation studies

[0087] To evaluate the performance of each component, some ablation studies are performed on each module.

[0088] Coarse-to-fine alignment module: This module consists of two components, including the global homography transformation and local progressive optical flow estimation The results of the ablation study on the impact of these two components are shown in Table 1, where Δ represents the rate of increase of PSNR as the iteration increases. Since the rate of increase is found to be very slow, the number of iterations is set to 3 to maintain a balance between quality and speed.

[0089]

[0090] Table 1. Ablation study on the coarse-to-fine arrangement module.

[0091] Knowledge-guided unique texture compression: In order to verify the efficiency of the common knowledge selection strategy, a master reference object image is randomly selected and common knowledge is extracted from it. The results in Table 2 reflect that the object image selected by MST can improve the results of BDBR (Bjontegaard bitrate difference) in terms of PSNR. At the same time, the present invention is compared with the "present invention (low-quality object image)" model, in which the common knowledge extracted from the low-quality main reference object image is used as conditional information, and the results are shown in Table 2. The scheme proposed in the present invention can achieve better performance because the common knowledge extracted from the higher-quality object image can promote image compression. In addition, the impact of mask compression of different qualities is tested. The mask only depicts important areas and has no texture, so that the quality of the mask will only show a small impact if the important areas can be correctly depicted. The pseudo-mask video is compressed by VTM with QP=58, because a higher QP can save bit rate. In addition, it is also explored. and The conditional information needs to be compressed, and costs some bits, but it brings promising improvements to compression.

[0092]

[0093] Table 2. Ablation study in the knowledge-guided unique texture compression model.

[0094] Knowledge-guided post-processing model: To verify and Some experiments were conducted on them, as shown in Table 3. The results show that and can improve the final performance, especially for downstream tasks. The efficiency of each component in the proposed approach is evaluated and the results show that they can benefit both the perceived quality and downstream tasks.

[0095]

[0096] Table 3. Ablation study on the knowledge-guided post-processing model.

[0097] all in all, Figures 1 to 8 The disclosed embodiment of the present invention discloses a knowledge-guided image set compression scheme based on INN for eliminating redundancy of image sets. The scheme effectively utilizes knowledge redundancy with sharable feature-level representation. In particular, the scheme extracts common knowledge and unique masks to promote compression of image sets. An efficient common knowledge extraction strategy is proposed, and in terms of unique masks, a pseudo mask video is formed using MST and DFS algorithms to improve the mask compression rate. Due to the lack of alignment between common knowledge and input images at the spatial level, a coarse-to-fine alignment module is developed. In addition, in order to achieve a balance between signal and perceptual quality, a knowledge-guided post-processing model is developed, which is conditioned on the aligned common knowledge. Together with the post-processing model, the scheme of the present invention can improve the perceptual quality of decoded images and improve the performance of downstream tasks, giving priority to signal quality without increasing the number of bits, while also improving human perception. In addition, a knowledge-guided unique texture compression model conditioned on common knowledge and unique masks is proposed, including a pseudo-inverse knowledge attention module (IKA) and a reversible texture attention module (ITA). These modules effectively eliminate the redundancy of public knowledge, mitigate information loss, and improve the signal quality of the decoded image.

[0098] Therefore, exemplary embodiments have been fully described. Although specific embodiments have been mentioned in the description, it is clear to those skilled in the art that the present invention can be implemented by variations of these specific details. Therefore, the present invention should not be interpreted as being limited to the embodiments described herein.

[0099] Although embodiments of the present invention have been described and described in detail in the accompanying drawings and the foregoing description, they should be considered illustrative rather than restrictive, and should be understood to only show and describe exemplary embodiments without limiting the scope of the present invention in any way. It is understood that any feature described herein can be used in any embodiment. The illustrative embodiments do not exclude each other or other embodiments not described herein. Therefore, the present invention also provides an embodiment comprising a combination of one or more of the illustrative embodiments described above. Without departing from the spirit and scope of the present invention, the present invention may be modified and varied, and therefore, only the limitations indicated by the appended claims should be applied.

[0100] Those skilled in the art may further recognize that, according to the description of the embodiments of the present disclosure, the units and steps of the algorithm may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly describe the interchangeability of hardware and software, in the above description, the composition and steps of the embodiments are generally described in terms of function. Whether these functions are performed by hardware or software depends on the design limitations of the specific application and technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, and such implementation should not be construed as departing from the scope of the present disclosure.

[0101] The steps of the method or algorithm described in the embodiments of the present disclosure can be implemented directly by hardware, software modules executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically controlled programmable ROM, an electrically controlled erasable and programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0102] It should be noted that the description of the aforementioned electronic device embodiment may be the same as the description of the aforementioned method embodiment, and the device embodiment has the same beneficial effects as the method embodiment. Therefore, the details may not be described again here. For technical details not disclosed in the embodiments of the electronic device disclosed in the present invention, those skilled in the art can understand them according to the method embodiments disclosed in the present invention.

[0103] In several embodiments of the present disclosure, it should be understood that the disclosed apparatus and method can be implemented in other ways. The device embodiments described above are exemplary only. All functional modules or units in the embodiments of the present disclosure can be fully integrated in one processing unit, or each unit can be used as a separate unit. Two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a combination of hardware and software functional units.

[0104] It should be understood by those skilled in the art that all or part of the steps of implementing the above method embodiments may be implemented by hardware associated with a computer instruction program. The instruction program may be stored in a computer-readable storage medium, and when executed, the processor performs the steps of the above method embodiments. The aforementioned storage medium may include various types of storage media, such as a removable storage device, a read-only memory (ROM), a random access memory (RAM), a disk, or any medium storing program code.

[0105] In addition, when the above-mentioned integrated unit of the present disclosure is sold or used as an independent product in the form of a software function module, the integrated unit can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution provided by the embodiments of the present disclosure can be basically or partially embodied in the form of a software product stored in a storage medium. The storage medium stores instructions, which are executed by a computer device (which can be a personal computer, a server, a network device, etc.) to implement all or part of the embodiments of the present disclosure. The above-mentioned storage medium may include various media capable of storing program codes, such as a removable storage device, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0106] When the logic is implemented in software, it can be written in an appropriate language, such as but not limited to C# or C++, and can be stored on or transmitted through a computer-readable storage medium (e.g., not a transient signal), such as a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM) or other optical storage, such as a digital versatile disk (DVD), magnetic disk storage or other magnetic storage devices, including removable USB drives, etc.

[0107] References

[0108] Each of the following references (and relevant appendices and / or supplements) is expressly incorporated herein by reference in its entirety.

[0109] [1]VVC software vtm-15.2.https: / / vcgit.hhi.

[0110] fraunhofer.de / jvet / VVCSoftware_VTM / - / tags / VTM-15.2.Online; accessed February 2022.4,6

[0111] [2]Johannes Ball′e,Valero Laparra,and Eero P.Simoncelli.Densitymodeling of images using a generalized normalization transformation.In ICLR,20l6.2

[0112] [3]Johannes Ball′e,David Minnen,Saurabh Singh,Sung Jin Hwang,and NickJohnston.Variational image compression with a scale hyperprior.In ICLR,2018.2,6

[0113] [4]Shilv Cai,Zhijun Zhang,Liqun Chen,Luxin Yan,Sheng Zhong,and XuZou.High-fidelity variable-rate image compression via invertible activationtransformation.arXiv preprint arXiv:2209.05054,2022.2,3,4,5,6

[0114] [5]Si-Yuan Cao,Jianxin Hu,Zehua Sheng,and Hui-Liang Shen.Iterativedeep homography estimation.In CVPR.pages 1879-1888,June 2022.2

[0115] [6]Thomas M Cover and Joy A Thomas.Information theory andstatistics.Elementsof information theory,1(1):279–335,1991.1

[0116] [7]Daniel DeTone,Tomasz Malisiewicz,and Andrew Rabinovich.Deepimagehomography estimation.arXiv preprint arXiv:1606.03798,2016.2

[0117] [8]Keyan Ding,Kede Ma,Shiqi Wang,and Eero Simoncelli.Imagequalityassessment:Unifying structure and texture similarity.IEEE Transactionson PatternAnalysis and Machine Intelligence,PP:1–1,12 2020.6,7

[0118] [9]Alexey Dosovitskiy,Philipp Fischer,Eddy Ilg,Philip Hausser,CanerHazirbas,Vladimir Golkov,Patrick van der Smagt,Daniel Cremers,and ThomasBrox.Flownet:Learning optical flow with convolutional networks.In ICCV,December 2015.2

[0119]

[10] Thomas Germer,Tobias Uelwer,Stefan Conrad,and StefanHarmeling.Pymatting:A python library for alpha matting.Journal of Open SourceSoftware.3

[0120]

[11] Shu Zhang Aseem Agarwala Hoang Le,Feng Liu.Deep homographyestimationfor dynamic scenes.In CVPR,2020.2

[0121]

[12] Eddy Ilg,Nikolaus Mayer,Tonmoy Saikia,Margret Keuper,AlexeyDosovitskiy,and Thomas Brox.Flownet 2.0:Evolution of optical flow estimationwith deepnetworks.In CVPR,July 2017.3,4

[0122]

[13] Joseph B Kruskal.On the shortest spanning subtree of a graph andthe travelingsalesman problem.Proceedings of the American Mathematicalsociety,7(1):48–50,1956.4

[0123]

[14] Yang Li,Shiqi Wang,Xinfeng Zhang,Shanshe Wang,Siwei Ma,and YueWang.Quality assessment of end-to-end learned image compression:The benchmarkandobjective measure.In Proceedings of the 29th ACM International ConferenceonMultimedia,pages 4297–4305,2021.6,7

[0124]

[15] Huajun Liu,Fuqiang Liu,Xinyi Fan,and Dong Huang.Polarized self-attention:towards high-quality pixel-wise regression.arXiv preprint arXiv:2107.00782,2021.6

[16] Shuaicheng Liu,Yuhang Lu,Hai Jiang,Nianjin Ye,ChuanWang,and BingZeng.Unsupervised global and local homography estimation withmotion basislearning.IEEE Transactions on Pattern Analysis and MachineIntelligence,2022.4

[0125]

[17] David Minnen,Johannes Ball′e,and George D Toderici.Jointautoregressive andhierarchical priors for learned image compression.InAdvances in Neural InformationProcessing Systems,volume 31.Curran Associates,Inc.,2018.6

[0126]

[18] Ty Nguyen,Steven W Chen,Shreyas S Shivakumar,Camillo Jose Taylor,andVijay Kumar.Unsupervised deep homography:A fast and robusthomographyestimation model.IEEE Robotics and Automation Letters,3(3):23462353,2018.4

[0127]

[19] Maria-Elena Nilsback and Andrew Zisserman.Automated flowerclassificationover a large number of classes.In 2008 Sixth Indian Conferenceon Computer Vision,Graphics&Image Processing,pages 722–729.IEEE,2008.6

[0128]

[20] Farzan Erlik Nowruzi,Robert Laganiere,and NathalieJapkowicz.Homographyestimation from image pairs with hierarchicalconvolutional networks.In ICCVW,pages 904–911,2017.2

[0129]

[21] Xuebin Qin,Zichen Zhang,Chenyang Huang,Masood Dehghan,OsmarRZaiane,and Martin Jagersand.U2-net:Going deeper with nested u-structureforsalient object detection.Pattern recognition,106:107404,2020.3

[0130]

[22] Zhongbo Shi,Xiaoyan Sun,and Feng Wu.Feature-based image setcompression.In ICME,pages 1–6,2013.1,2,3

[0131]

[23] Tushar Shinde.Efficient image set compression.In 2019 IEEEInternationalConference on Image Processing(ICIP),pages 3016–3017.IEEE,2019.1

[0132]

[24] William Thomas Tutte and William Thomas Tutte.Graph theory,volume21.Cambridge university press,2001.3

[0133]

[25] Catherine Wah,Steve Branson,Peter Welinder,Pietro Perona,andSergeBelongie.The caltech-ucsd birds-200-2011 dataset.2011.6

[0134]

[26] Jin Wang,Yunhui Shi,Yinsen Xing,Nam Ling,and Baocai Yin.Deepcorrelatedimage set compression based on distributed source coding and multi-scale fusion.InDCC,pages 192–201,2022.2

[0135]

[27] Yueqi Xie,Ka Leong Cheng,and Qifeng Chen.Enhanced invertibleencoding forlearned image compression.In Proceedings of the 29th ACMInternationalConference on Multimedia,pages 162–170,2021.2,3

[0136]

[28] Nianjin Ye,Chuan Wang,Haoqiang Fan,and Shuaicheng Liu.Motionbasislearning for unsupervised deep homography estimation with subspaceprojection.InICCV,pages 13117–13125,October 2021.2

[0137]

[29] Syed Waqas Zamir,Aditya Arora,Salman Khan,Munawar Hayat,FahadShahbaz Khan,and Ming-Hsuan Yang.Restormer:Efficient transformerforhigh-resolution image restoration.In CVPR,pages 5728–5739,June 2022.6

[0138]

[30] Richard Zhang,Phillip Isola,Alexei A Efros,Eli Shechtman,andOliver Wang.The unreasonable effectiveness of deep features as a perceptualmetric.In CVPR,pages586–595,2018.6,7

Claims

1. A computer-implemented method for compressing a digital image set, the image set comprising a plurality of original images; characterized in that: The method comprises the following steps: Decomposing each of the plurality of original images into three parts: common knowledge, a unique mask, and a unique texture; wherein the common knowledge represents shared features between the plurality of original images; and Each of the common knowledge, the unique mask and the unique texture is compressed into a bit stream respectively.

2. The method of claim 1, wherein the common knowledge is extracted from a main reference image among the plurality of original images.

3. The method of claim 2, wherein the primary reference image is determined by: determining the unique mask for each of the plurality of original images; multiplying each of the plurality of original images by the unique mask corresponding thereto to obtain a plurality of object images; Building a directed graph based on similarities of features extracted from the plurality of object images; Processing the directed graph using a minimum spanning tree algorithm; as well as The node with the largest number of edges in the spanning tree is taken as the main reference image.

4. The method according to claim 1, wherein the step of compressing the public knowledge further comprises: The common knowledge is spatially aligned with the multiple images using a trained reversible neural module to eliminate redundancy in the common knowledge.

5. The method of claim 1, wherein the decomposing step further comprises: The unique mask is extracted from each of the plurality of original images by using a U2-Net saliency detection method and an alpha matting technique.

6. The method of claim 1, wherein the compressing step further comprises: forming the unique masks of the plurality of original images into a pseudo mask video in a predetermined order; as well as The pseudo mask video is compressed using a video encoder to achieve compression of the unique mask. The method of claim 6 , wherein the predetermined order is a depth-first search order.

8. The method of claim 1, wherein the compressing step further comprises: Given the common knowledge and the unique mask, the unique texture is compressed.

9. The method of claim 8, wherein the step of compressing the unique texture further comprises: Using a trained reversible neural module, aligning the common knowledge with the plurality of images at a spatial level to eliminate redundancy in the common knowledge; performing texture enhancement of a region of interest on each of the plurality of images using the aligned common knowledge and the unique mask for each of the plurality of images; as well as A reversible activation transformation is performed on each of the plurality of images.

10. The method of claim 4 or 9, wherein the alignment step is a coarse-to-fine alignment.

11. The method of claim 9, wherein the aligning step further comprises: performing a homography transform on each of the plurality of images; as well as Progressive optical flow estimation is performed on each of the plurality of images.

12. The method of claim 9, wherein the reversible neural module is trained by using a training pair; the training pair comprises an original object image in a training image set, and another decoded object image in the same training image set.

13. A computer-implemented method for reconstructing a set of digital images from a plurality of bitstreams comprising a common knowledge stream, a unique mask stream, and a unique texture stream; the method comprising the steps of: decoding public knowledge from the public knowledge stream; decoding a plurality of unique masks corresponding to a plurality of images in the set of digital images from the unique mask stream; decoding a plurality of unique textures corresponding to a plurality of images in the digital image set from the unique texture stream; The set of digital images is reconstructed using the common knowledge, the plurality of unique masks, and the plurality of unique textures.

14. The method of claim 13, wherein the step of decoding the plurality of unique textures further comprises: The unique texture is decoded given the decoded common knowledge and the unique mask.

15. The method of claim 13, further comprising using a post-processing model to enhance the perceptual quality of the reconstructed set of digital images using the common knowledge of alignment as a guide.

16. The method of claim 15, wherein the step of enhancing the perceived quality of the reconstructed set of digital images further comprises: Using the trained reversible neural module, spatially aligning the decoded common knowledge with the reconstructed plurality of images in the digital image set to eliminate redundancy in the common knowledge; Mapping the aligned common knowledge to a mapping image consistent with spatial information of the plurality of images; Extracting features of the mapped image using a convolutional layer; as well as The features are input into the polarized self-attention module.

17. The method of claim 16, wherein the aligning step is a coarse to fine alignment.

18. The method of claim 17, wherein the aligning step comprises: performing a homography transform on each of the plurality of images; as well as Progressive optical flow estimation is performed on each of the plurality of images.

19. The method of claim 15, wherein the post-processing model is based on a Restormer model.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the method according to claim 1.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the method of claim 13.

Images