Image compression, reconstruction method, electronic device, storage medium, and program product

Abstract

This application provides an image compression and reconstruction method, electronic device, storage medium, and program product, relating to the field of image processing technology. The method includes extracting structural features and image features from a target image. Structural features characterize the structural information of image elements within the target image, while image features characterize the image information of those elements. Based on these structural and image features, implicit features for compression are extracted from the target image. Compared to features obtained by directly extracting features from the target image, these implicit features are more representative. Furthermore, the bitstream obtained after compressing the target image based on these implicit features retains key image information, further improving the accuracy of subsequent image reconstruction.

Description

Technical Field

[0001] This application relates to the field of image compression and reconstruction technology, and in particular to an image compression and reconstruction method, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Images need to be compressed before transmission and storage. When stored on edge devices or transmitted over narrow bandwidth, limited storage space or insufficient network bandwidth may prevent the transmission of uncompressed or poorly compressed images. Therefore, images need to be compressed to an extremely low bit rate to meet the aforementioned transmission requirements.

[0003] However, there are concerns that compressing images at extremely low bit rates can cause the compressed images to lose key information, resulting in lower accuracy when reconstructing them. Summary of the Invention

[0004] This application provides an image compression and reconstruction method, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] In a first aspect, embodiments of this application provide an image compression method, comprising: extracting structural features and image features from a target image, wherein the structural features are used to characterize the structural information of image elements in the target image, and the image features are used to characterize the image information of the image elements; determining implicit features for image compression based on the structural features and the image features; and compressing the target image based on the implicit features to obtain a bitstream.

[0006] Secondly, embodiments of this application provide an image reconstruction method, comprising: acquiring a bitstream of a target image and structural features of the target image, wherein the bitstream is obtained by compressing the target image based on implicit features, the implicit features are determined based on the structural features and image features of the target image, the structural features are used to characterize the structural information of image elements in the target image, and the image features are used to characterize the image information of the image elements; decoding the bitstream to obtain the implicit features; and generating a reconstructed image of the target image based on the structural features and the implicit features.

[0007] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0009] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the methods described in the embodiments of this application.

[0010] In the technical solution of this application embodiment, structural features and image features are extracted from the target image. Since structural features are used to characterize the structural information of image elements in the target image, they can enrich the semantic expression of the target image. Image features are used to characterize the image information of image elements. Based on this, implicit features for compression can be extracted from the target image according to the structural features and image features. Compared with the features obtained by directly extracting image features from the target image, the implicit features have stronger semantic expressiveness. Furthermore, the bitstream obtained after compressing the target image based on the implicit features will retain the key semantic information of the image, thereby improving the accuracy of subsequent image reconstruction.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0013] Figure 1 A system architecture diagram of an exemplary scenario for the image compression and reconstruction method provided in the embodiments of this application;

[0014] Figure 2 A flowchart illustrating the image compression method provided in this application embodiment;

[0015] Figure 3 A schematic diagram of the target image and the structural features corresponding to the target image provided in the embodiments of this application;

[0016] Figure 4A , Figure 4B , Figure 4C and Figure 4D A schematic diagram of a module combining structural features and image features is shown;

[0017] Figure 5 A flowchart illustrating the image compression method provided in this application embodiment;

[0018] Figure 6 This is a schematic diagram of semantic adaptation in the embodiments of this application;

[0019] Figure 7 This is a schematic diagram illustrating compression and reconstruction in an embodiment of this application;

[0020] Figure 8 This is a schematic diagram of the compression and decoding method according to an embodiment of this application;

[0021] Figure 9 This is a schematic diagram of the functional modules in an embodiment of this application;

[0022] Figure 10 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0023] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0024] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0025] Images do indeed need to be compressed before transmission. When stored on edge devices or transmitted over narrow bandwidth, due to limited storage space or insufficient network bandwidth, it is impossible to transmit uncompressed original images. Therefore, images need to be compressed to an extremely low bit rate to meet these transmission requirements.

[0026] However, a major problem with ultra-low bitrate compression in related technologies is the significant loss of information in the compressed image. This loss leads to a substantial decrease in image quality when reconstructing from the compressed image, manifesting as blurred image details, color distortion, blockiness, or increased noise. These issues make processing and analyzing images compressed with ultra-low bitrates extremely challenging in applications with high image quality requirements, such as medical image analysis and security monitoring.

[0027] In view of this, embodiments of this application propose an image compression and reconstruction method to solve the above problems. To more clearly illustrate the image compression and reconstruction method provided in the embodiments of this application, application scenarios that can be used to implement this method are first introduced.

[0028] The technical solution of this application can be applied to scenarios that require image compression and high-quality details in the reconstructed image, such as high-definition image transmission, high-definition streaming media playback or real-time communication, and medical image analysis. Figure 1 The diagram shown is a system architecture diagram of an exemplary scenario of the image compression and reconstruction method provided in this application. The encoding end extracts implicit features from the target image and quantizes and encodes these implicit features to obtain a bitstream; the decoding end decodes and dequantizes the bitstream to obtain implicit features, and reconstructs the implicit features based on pre-acquired structural features to obtain the reconstructed image.

[0029] Among them, structural features characterize the structural information of image elements in the target image, which can minimize the distortion of the reconstructed target image at low bit rates, thus becoming an effective reference guide in the compression and reconstruction process of the target image.

[0030] For example, the encoding and decoding ends can be applications, services, instances, software modules, virtual machines (VMs), containers, or cloud servers, or hardware devices (such as servers or terminal devices) or hardware chips with data processing capabilities. Terminal devices can be fixed terminals, such as personal computers, or mobile terminals, such as mobile phones or tablets. Furthermore, the encoding and decoding ends can be deployed on the same side. For example, they can be deployed as software modules in the same terminal device, or they can be deployed as cloud products in the same cloud server or cluster.

[0031] It should be noted that the application scenarios or examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions. In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0032] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0033] Figure 2 This is a flowchart illustrating an image compression method provided in an embodiment of this application. This method can be applied to an image compression apparatus, which can be deployed at an encoding end. Figure 2 As shown, the image compression method includes:

[0034] Step S201: Extract structural features and image features from the target image.

[0035] Among them, structural features are used to characterize the structural information of image elements in the target image, while image features are used to characterize the image information of image elements.

[0036] The execution entity in this embodiment can be an encoding end, wherein the target image is the original image to be compressed. Image features of the target image can be extracted by vectorizing the target image. Specifically, the target image is transformed from a high-dimensional pixel space to a low-dimensional feature vector space. Preprocessing can be performed on the target image before vectorization.

[0037] For example, if the target image is in color, it is usually first converted to grayscale to reduce the dimensionality and complexity of the data. The target image is then scaled and cropped as needed to ensure it has an appropriate size for subsequent processing. Filters (such as Gaussian filters) are used to remove noise from the target image, improving image quality. Vectorization can involve extracting global or local features of the target image. Global features extract overall characteristics of the target image, such as color histograms and shape features; these semantic features describe the global attributes of the target image. Local features focus on local regions of the target image, extracting features such as edges, corners, and textures; these features are more sensitive to the details and local changes of the target image.

[0038] After feature extraction, the features are transformed into vectors. Specifically, several vectorization methods can be used: Scale-Invariant Feature Transform (SIFT) generates feature vectors by calculating the histograms of keypoints and their oriented gradients in the image. Speeded-Up Robust Features (SURF) is an accelerated version of SIFT, using integral images and box filters to speed up computation. Alternatively, Local Binary Patterns (LBP) can be used to compare the grayscale values ​​of each pixel in the image with its neighboring pixels to generate binary patterns for vectorizing the target image.

[0039] In some embodiments, image features can also be extracted using a Latent Compression Module (LCM) to extract latent features of the target image, which are then used as image features. LCM effectively reduces redundancy in the latent representation and provides guidance for subsequent denoising and diffusion processes, thereby improving the quality of image encoding and reconstruction. The LCM network architecture is based on a Variational Autoencoder (VAE), forming a system that includes a latent transform network, a super-prior network, and a channel-wise context model. Gaussian analysis transforms the features of the target image into an implicit compact representation y. The module corresponding to the Gaussian analysis transform consists of multiple residual blocks and downsampling convolutional layers, aiming to extract deeper feature information, making the subsequent encoding process more efficient.

[0040] Structural features carry geometric information about the target image, and extracting these features plays a crucial role in the encoding process. For example, a normal map can be extracted from the target image and used as a structural feature. The normal map represents the spatial data corresponding to the target image; for instance, if the target image is a map, such as roads and buildings, the normal map is represented by geometric shapes and coordinates (lines and polygons representing spatial locations). Figure 3 The left side of the image shows the initial target image, and the right side shows the normal map corresponding to the target image. It can be seen that the normal map provides structural information about the image elements in the target image, and can be used as a guide during the compression and decompression process to improve the accuracy of target image compression.

[0041] In one implementation, extracting a normal map from a target image may include: inputting the target image into a Normal Bayesian Active Energy Model (Normal BAE) to obtain a normal map output by NormalBAE, wherein the Normal Bayesian Active Energy Model is used to obtain the normal map through multiple rounds of normal prediction.

[0042] In other words, normal maps can be extracted using deep learning methods with Normal BAE. For example, the RGB feature map of the target image with size H×W is first input into the Normal BAE model for upsampling. Interpolation techniques are then used to restore the 1 / 8 resolution RGB feature map to a higher resolution. That is, the deep learning network in the Normal BAE model performs initial coarse-grained normal prediction starting from the 1 / 8 resolution RGB feature map. For example, a 3×3 convolutional layer in the Normal BAE model is used to predict the initial normal map, with the output channel number set to 4 to facilitate subsequent fine-tuning. After the coarse-grained initial normal map is generated, it enters three pixel-level fine-tuning modules for pixel-level fine-tuning prediction. Through further convolutional operations, the normal information of each pixel is refined, gradually increasing the resolution of the normal map and enhancing the accuracy of the prediction. Each fine-tuning module takes a lower-resolution feature map and a coarse prediction as input, and outputs a high-resolution normal map with size H / n×W / n. Through this series of steps, the Normal BAE model can accurately generate detailed normal maps, improving its semantic representation characteristics.

[0043] Furthermore, the structural features of the target image can also be extracted through edge detection, shape analysis, etc., to obtain structural features such as lines, shapes, and textures. This application does not limit the structural features of the target image; those skilled in the art can determine them according to the actual situation.

[0044] Step S202: Determine the implicit features for image compression based on structural features and image features.

[0045] In one example, techniques such as support vector machines and convolutional neural networks can be used to further analyze and process the extracted structural and image features to determine implicit features that are highly correlated with the compression effect of the target image. These implicit features play an important role in the image compression process.

[0046] In another example, implicit features can be obtained by processing structural and image features using the spatially-adaptive positional encoding and decoder residual blocks (SPADEResBlock) and residual blocks (ResBlock) of the Gaussian analysis module (which will be described in detail below).

[0047] Step S203: Compress the target image based on implicit features to obtain the bitstream.

[0048] Specifically, compression can be performed in the following ways. For example, a VAE (Visual Image Array) can be used to map the target image into a latent space, and then the image can be reconstructed using a decoder. In the latent space, the target image is represented as a probability distribution, typically a multivariate Gaussian distribution. The bitstream can be a latent variable in this latent space.

[0049] Compression can also be achieved using Generative Adversarial Networks (GANs), which consist of a generator and a discriminator. The generator learns the distribution of the reconstructed image, while the discriminator evaluates the difference between the reconstructed image and the real target image. During compression, the generator encodes the target image into a compressed representation and then reconstructs the image from this representation, where the compressed representation is the bitstream.

[0050] In the technical solution of this application embodiment, structural features and image features are extracted from the target image. Since structural features are used to characterize the structural information of image elements in the target image, they can enrich the semantic expression of the target image. Image features are used to characterize the image information of image elements. Based on this, implicit features for compression can be extracted from the target image according to the structural features and image features. Compared with the features obtained by directly extracting image features from the target image, the implicit features have stronger semantic expressiveness. Furthermore, the bitstream obtained after compressing the target image based on the implicit features will retain the key semantic information of the image, thereby improving the accuracy of subsequent image reconstruction.

[0051] In one embodiment, in step S202 above, determining the implicit features for image compression based on structural features and image features may include: concatenating structural features and image features to obtain concatenated features; performing convolution processing on the concatenated features to obtain a dimension factor and a bias term; and multiplying the concatenated features by the dimension factor and then adding the bias term to obtain the implicit features.

[0052] For example, structural features and image features are both corresponding feature matrices, and the dimensions of the corresponding feature matrices are consistent. If the dimensions are inconsistent, the features of the corresponding dimensions can be aligned, and blank positions can be padded with zeros to achieve dimensional consistency. The extracted image features and structural features are concatenated. The two are stacked along the feature dimension to form a new feature vector or feature map. In this embodiment, the image features can be added to the structural features of the corresponding dimension to obtain the concatenated features, or the two feature vectors can be directly filled with rows or columns. Compared with image features, concatenated features can reflect the structural information of the target image, thereby improving the representativeness of subsequent implicit features and improving the accuracy of image reconstruction. Before performing convolution processing on the concatenated features, one or more convolutional layers need to be designed to process the concatenated features. Convolutional layers typically contain multiple convolutional kernels, each of which can capture different patterns in the concatenated features.

[0053] For example, structural features and image features are concatenated to obtain concatenated features; convolution is performed on the concatenated features to obtain dimensionality factors and bias terms; the concatenated features are multiplied by the dimensionality factors and then added to the bias terms to obtain implicit features, which can be implemented using SPADE ResBlock and ResBlock. Figure 4A The diagram shown is a structural schematic of SPADE ResBlock. Figure 4B For the above Figure 4A A schematic diagram of the structure of a basic block. Figure 4C The diagram shows the structure of a Spatially-Adaptive Positional Encoding Block (SPADE Block). Figure 4D This is a schematic diagram of the structure of a residual block (ResBlock).

[0054] The SPADE ResBlock consists of two SPADE Blocks and two Basic Blocks. The SPADE Block first concatenates image features and structural features to obtain the concatenated features. Within the SPADE Block, the Basic Block performs convolutional processing (ConV 3*3) on image elements to obtain image features. These image features are then concatenated with the structural features along their corresponding dimensions to obtain the concatenated features. During the concatenation process, BatchNorm performs batch normalization on the data from each convolutional layer and before concatenation to improve network performance.

[0055] Subsequently, the concatenated features undergo convolutional processing (ConV 3*3) in the Basic Block to generate a dimension factor γ and a bias term β. These two are then multiplied and added element-wise to obtain implicit features. These implicit features are normalized using LayerNorm in the Basic Block, effectively incorporating structural features into the encoding of image features. This ensures that the encoded features retain the corresponding structural information. LayerReLU is used to alleviate the gradient vanishing problem in deep network training, improving the network's convergence speed and performance. The SPADE ResBlock module, by combining residual connections and structural feature guidance, achieves a higher level of feature representation. Implicit feature extraction via SPADE ResBlock allows for better capture of complex image structures during compression encoding.

[0056] The ResBlock is located between the convolutional layer and the SPADE ResBlock, adding residual connections between them. The ResBlock typically consists of two Basic Blocks, each containing a 3×3 convolutional layer (ConV 3*3), a normalization layer (Layer Norm), and a Rectified Linear Unit (Layer ReLU) activation layer. This design effectively alleviates the vanishing gradient problem in deep network training through residual connections, improving the network's convergence speed and performance.

[0057] In one implementation, step S203 above, encoding the target image based on implicit features to obtain a bitstream, includes: determining the statistical attributes of the implicit features based on the feature distribution of the implicit features, wherein the statistical attributes include at least one of probability density function, mean, variance, distribution shape, standard deviation, median, and quartiles; converting the image features into discrete feature vectors based on the statistical attributes; and encoding the discrete feature vectors to obtain a bitstream.

[0058] For example, implicit features can be subjected to distribution analysis to determine their probability density function, mean, variance, distribution shape, standard deviation, median, quartiles and other statistical properties, which will be used in the subsequent discretization process.

[0059] Specifically, image features are discretized based on the statistical properties of implicit features. Exemplary discretization methods may include equal-width discretization (dividing the feature value range into equally wide intervals), equal-frequency discretization (dividing feature values ​​into intervals containing the same number of samples), and clustering-based methods (such as K-means clustering). In this process, each image feature is mapped to a discrete interval or category, resulting in a discrete feature vector. All discretized features are then combined into a single feature vector. This feature vector will be used in the subsequent encoding process.

[0060] Furthermore, an appropriate encoder is selected as needed to convert the discrete feature vectors into a bitstream. For example, the encoding methods used by the encoder may include entropy coding, Huffman coding, arithmetic coding, run-length encoding, dictionary coding, etc. The choice of encoder can be based on factors such as the statistical properties of the feature vectors, compression requirements, and decoding complexity. The encoding process involves steps such as quantizing the feature vectors, selecting encoding symbols, and optimizing the bitstream length. The final encoded bitstream contains a discrete representation of the image features. The bitstream can be used for storage, transmission, or further processing.

[0061] Through the encoding process, the originally lengthy continuous feature data can be transformed into more compact discrete feature vectors, and further compressed into a bitstream, thereby significantly reducing data storage space. The compressed bitstream is easier to store and manage, reducing storage costs while improving the efficiency and speed of data access. Discrete features are highly robust to outliers because outliers are usually classified into a discrete interval or category without significantly impacting the overall model.

[0062] Corresponding to the image compression method described above, this application also discloses an image reconstruction method, such as... Figure 5 The diagram shown is a flowchart of an image reconstruction method provided in an embodiment of this application. This method can be applied to an image reconstruction apparatus, which can be deployed at the decoding end. Figure 5 As shown, the image compression method includes:

[0063] Step S501: Obtain the bitstream of the target image and the structural features of the target image.

[0064] The bitstream is obtained by compressing the target image based on implicit features. The implicit features are determined based on structural features and image features of the target image. Structural features are used to characterize the structural information of the image elements in the target image, and image features are used to characterize the image information of the image elements.

[0065] Step S502: Decode the bitstream to obtain implicit features.

[0066] For example, when decoding the bitstream, a decoding method corresponding to the encoding method is used.

[0067] Step S503: Generate a reconstructed image of the target image based on structural features and implicit features.

[0068] For example, when performing image reconstruction at the decoding end, it is necessary to obtain the bitstream and structural features of the target image. If the decoding end and the encoding end are on the same device, they can be directly obtained from the storage addresses corresponding to the stored bitstream and structural features. If the decoding end and the encoding end are located on different devices, when image reconstruction is required, the decoding end obtains the bitstream and structural features from the encoding end. Specifically, this can be achieved through network transmission or file transfer. The encoding end sends the compressed bitstream and related structural feature data to the decoding end via a network (such as Ethernet, Wi-Fi, 4G / 5G, etc.). The decoding end receives this data through a network interface and performs corresponding decoding and image reconstruction, or the encoding end saves the bitstream and structural feature data to a local file or cloud storage. The decoding end obtains these files through file sharing or manual download. The decoding end reads the data from the files and performs decoding and image reconstruction.

[0069] Before reconstructing an image, structural features can be preprocessed to ensure that structural features and implicit features are consistent in format, dimension, or aspect ratio. Preprocessing involves steps such as feature scaling, normalization, or feature selection.

[0070] Fusion of implicit and structural features can be achieved through features concatenating, weighting, or mapping. The goal of fusion is to combine them into a unified feature representation for subsequent image reconstruction. Incorporating structural features provides a more comprehensive description of the target image, leading to more accurate reconstruction and improved image quality. Based on the fused feature representation, an image reconstruction algorithm is used to generate the reconstructed image. This algorithm can be based on deep learning methods (such as Generative Adversarial Networks (GANs) or Convolutional Neural Networks (CNNs)) or traditional image processing algorithms (such as interpolation and filtering).

[0071] In the technical solution of this application embodiment, the code stream of the target object and the structural features of the target object are obtained. Since the implicit features are determined based on the structural features and the image features of the target image, where the structural features are the structural information of the image elements in the target image, which can enrich the semantic expression of the target image, and the image features are used to characterize the image information of the image elements, the implicit features obtained by decoding the code stream include the key semantic information of the target image. Further combining the structural features with the implicit features to perform image reconstruction can improve the accuracy of the reconstructed image to a certain extent.

[0072] In some embodiments, step S502 above includes: decoding the bitstream to obtain a discrete feature vector; determining the statistical properties of the implicit features based on the feature distribution of the implicit features, wherein the statistical properties include at least one of probability density function, mean, variance, distribution shape, standard deviation, median, and quartiles; and converting the discrete feature vector into implicit features based on the statistical properties.

[0073] For example, an appropriate decoder is used to receive and read the bitstream. Based on the bitstream's encoding format (e.g., H.264, H.265), the decoder is initialized and relevant parameters are configured. Further, the decoder performs decoding operations on the bitstream, extracting discrete feature vectors. These vectors may be compressed, encoded, or encrypted; the decoding process requires a reverse operation to recover their original form. Further, based on the feature distribution of the implicit features, statistical attributes are determined, such as probability density function, mean, variance, distribution shape, standard deviation, median, quartiles, etc. These statistical attributes will be used in the subsequent feature transformation process. Based on the determined statistical attributes, the discrete feature vectors are transformed into implicit features. For example, the transformation process may involve feature scaling, normalization, classification mapping, etc., to ensure that the transformed features are consistent with the implicit features in terms of value range, distribution shape, etc.

[0074] In this embodiment, corresponding to the encoding process in image compression, the decoder in the image reconstruction process decodes the bitstream after receiving it. Then, inverse quantization and inverse Gaussian transform are performed to obtain initial latent features. During decoding, structural features (e.g., normal maps) are also used as auxiliary information input to improve the accuracy and quality of the reconstruction process. The decoder is structurally similar to the encoder; the inverse Gaussian transform module also consists of multiple residual blocks and upsampling convolutional layers to facilitate the recovery of image structure from latent features.

[0075] In an optional embodiment, the structural features and implicit features have the same dimension. Step S503 includes: for any dimension of implicit features, determining the corresponding dimension of reconstruction features based on the implicit features and the corresponding dimension of structural features; and reconstructing the reconstruction features of each dimension using a diffusion model to obtain a reconstructed image.

[0076] For example, for any implicit feature in any dimension, its corresponding structural features need to be obtained first. Structural features can include information such as spatial location, shape, and texture in that dimension. This is achieved through feature concatenation, feature mapping, or feature fusion algorithms. The fused features will contain comprehensive information from both implicit and structural features. Based on the fused features, the reconstructed features for the corresponding dimension are determined. The reconstructed features may involve enhancing, refining, or adjusting the original features to better reflect the details and structure of the image in that dimension.

[0077] Diffusion models include those based on partial differential equations (PDEs) or deep learning-based diffusion models. Reconstructed features are input into the diffusion model, and details and structure of the reconstructed image in each dimension are gradually generated through iterative solutions to PDEs or optimization of loss functions. Diffusion models can simulate physical diffusion or image generation processes, thus producing image content with continuity and consistency. This ensures that the reconstructed image remains smooth and continuous across all dimensions, avoiding breaks or inconsistencies. The reconstructed images from each dimension are then synthesized to obtain a complete reconstructed image. Post-processing of the reconstructed image, such as denoising, contrast enhancement, and brightness adjustment, improves image quality and visual appeal.

[0078] In the process of obtaining reconstructed features from structural features and implicit features in the embodiments of this application, structural features can also be transformed into semantic features. For example, a semantic encoder is used to convert the normal map into a semantic feature map (the process of convolution), and the semantic feature map is fused with the implicit features to obtain reconstructed features. The reconstructed features are then used for image reconstruction.

[0079] In an optional embodiment, the structural feature is a normal map. For any dimension of implicit feature, the reconstructed feature of the corresponding dimension is determined based on the implicit feature and the structural feature of the corresponding dimension, including: converting the normal map into a semantic feature map; converting the semantic feature map into a semantic feature with the same dimension as the implicit feature; and for any dimension, adding the implicit feature to the semantic feature of the corresponding dimension to obtain the reconstructed feature of the corresponding dimension.

[0080] Specifically, such as Figure 6The diagram illustrates the process of semantically encoding the normal map and obtaining reconstructed features. A semantic encoder converts the normal map into a semantic feature map (a process involving multiple convolutions), and then transforms it into multi-dimensional semantic features fmis for use as explicit conditional features in subsequent diffusion processes. First, the semantic feature map is downsampled to match the dimension of the implicit features. Next, this downsampled semantic feature map is concatenated with the implicit features to form the input to the first residual block (ResBlock), thereby generating the dimensional semantic features fm1s. In subsequent dimensionality processing, the corresponding dimensional semantic features fmis are extracted through a convolutional layer (Conv k5s2) and a combination of two residual blocks.

[0081] It is worth noting that the generated multi-dimensional semantic features fms = {fm1s, fm2s, fm3s, fm4s, ...} have the same dimensions as the implicit features fnc = {f1, f2, f3, f4, ...}. In each dimension, the semantic feature fmis is added to the corresponding fnc. This method, by combining explicit guidance information with a denoising diffusion process, ensures that the reconstructed image maintains structural consistency with the original image even at extremely low bit rates.

[0082] During image generation, sharp edge details are effectively produced. This process not only ensures high-quality visual effects for the target image but also preserves important structural features during reconstruction. The generation and processing of each feature aims to better integrate the image's structural information with its latent features, thereby achieving efficient image generation.

[0083] The following section provides a comprehensive overview of the image compression and image reconstruction methods described above. For example... Figure 7 The diagram shows the structure of image compression and image reconstruction. Figure 7 The process involves two stages. The first stage involves the latent representation (image features) of the LCM compressed image, while simultaneously using structural features as guidance to extract implicit features from the image features. Specifically, the target image is mapped into the latent representation and further compressed using a VAE to maximize bit rate savings. The latent representation (implicit features) generated in this stage provides implicit guidance for the model during the subsequent diffusion process. In the second stage, structural features serve as explicit guidance for the diffusion process. Specifically, a semantic adaptation module can be used to transform structural features into multi-dimensional semantic features, providing explicit guidance for the diffusion process. By combining implicit and explicit guidance, the accuracy of the reconstructed image is improved.

[0084] like Figure 8 The diagram shown is a structural schematic of an LCM. Figure 8The encoding and decoding process is described. In the LCM encoding process, the target image needs to undergo three convolutional layers, and structural features (normal maps) are used as guidance during each convolutional layer, for example, utilizing... Figure 4A The SPADE ResBlock shown integrates information from structural features to enhance the effectiveness of latent transform coding.

[0085] In the reconstruction process using diffusion models, VAE-based LCM often only retains blurry outlines, while clear texture details are generated by the diffusion model. In fact, stable diffusion models perform diffusion processing within the latent space, resulting in relatively small spatial resolution. Although diffusion models have the ability to diffuse and complete image content, to improve the completion effect, implicit and explicit guidance mechanisms must be introduced during the denoising diffusion process.

[0086] Specifically, the output of LCM already provides most of the details of the original image, and therefore can be used as implicit guidance. The mapping of structural features provides additional structural information, which can be used as explicit guidance. Implicit guidance means that implicit variables are connected as additional inputs in each step of the denoising U-shaped network (UN), thus effectively utilizing the detailed information in the LCM output.

[0087] In the final output stage of the denoising diffusion process, the model's output is under implicit guidance, directing it towards the original latent representation. This means that despite the extremely low input bit rate, the denoising model can still optimize towards restoring the target image, thereby achieving the highest possible image quality in the output. This method fully utilizes the local details provided by LCM and the additional structural information from structural feature mapping, enabling the generation of visually appealing images even at extremely low bit rates.

[0088] Figure 9A block diagram of a decoding system combining structural features and a diffusion module is shown, which can be used for the aforementioned image compression and image reconstruction. The system includes: a structural feature acquisition module for performing the process of extracting structural features as described above; a latent compression module for performing the process of determining implicit features based on structural features and image features as described above; a semantic adaptation module for performing the process of converting structural features (normal maps) into semantic feature maps as described above; a variational autoencoder module for performing the image feature extraction process in the latent compression module as described above; a SPADE Block module for performing the process of concatenating structural features and image features to obtain implicit features in the latent compression module as described above; an arithmetic encoding module for performing the process of compressing and encoding implicit features to obtain a bitstream as described above; an arithmetic decoding module for performing the process of decoding the bitstream as described above; a feature distribution module for performing the process of determining the statistical properties of the implicit features based on the feature distribution of the implicit features as described above; a quantization module for performing the process of converting the image features into discrete feature vectors based on the statistical properties as described above; an inverse quantization module for performing the process of converting the discrete feature vectors into the implicit features as described above based on the statistical properties as described above; and a hyperprior part module for more accurately determining the statistical properties of the implicit features as described above in the feature distribution module. Each module operates independently yet interacts with the others to collectively accomplish the image compression task. The specific implementation process is as follows: Figure 2 and Figure 5 As shown.

[0089] Furthermore, the models used in the image compression and reconstruction processes described above can be pre-trained or gradually optimized during use. The following loss function can be used to adjust both the training process optimization and the fine-tuning during use.

[0090] L=λ rate L rate +λ ld L ld +λ diff L diff

[0091] Where L is the loss value, L rate For rate loss, L ld For potential distortion loss L ld L diff λ represents the diffusion loss. rate , λ ld and λ diff These are the corresponding weights.

[0092] During the optimization process, a two-stage training strategy can be adopted to optimize the compression module and the diffusion model separately.

[0093] In the first stage, rate loss and potential distortion loss are introduced to optimize the latent compression module (LCM), and the loss function is as follows:

[0094] L 1st =L rate +λL ld

[0095] Here, λ is a Lagrange multiplier used to control the expected trade-off between rate and potential distortion.

[0096] The second stage involves fully training the objective function, which includes three loss terms.

[0097] L=λ rate L rate +λ ld L ld +λ diff L diff

[0098] In summary, the introduction of structural features provides guidance for both image compression and reconstruction processes, thereby preserving important details in image reconstruction.

[0099] Corresponding to the application scenarios and methods of the image compression method provided in the embodiments of this application, the embodiments of this application also provide an image compression apparatus. This image compression apparatus may include: a feature extraction module, used to extract structural features and image features from a target image, wherein the structural features are used to characterize the structural information of image elements in the target image, and the image features are used to characterize the image information of the image elements; an implicit feature extraction module, used to determine implicit features for image compression based on the structural features and the image features; and an encoding module, used to encode the target image based on the implicit features to obtain a bitstream.

[0100] In one implementation, the implicit feature extraction module is specifically used to: concatenate the structural features with the image features to obtain concatenated features; perform convolution processing on the concatenated features to obtain a dimension factor and a bias term; multiply the concatenated features by the dimension factor and then add the result to the bias term to obtain the implicit features.

[0101] In one implementation, the implicit feature extraction module is specifically used to: extract the normal map from the target image to obtain the structural features.

[0102] In one implementation, the implicit feature extraction module is specifically used to: input the target image into the normal Bayesian active energy model to obtain the normal map output by the normal Bayesian active energy model, wherein the normal Bayesian active energy model is used to obtain the normal map through multiple rounds of normal prediction.

[0103] In one implementation, the encoding module is specifically used to: determine the statistical properties of the implicit features based on the feature distribution of the implicit features, convert the image features into discrete feature vectors based on the statistical properties, and encode the discrete feature vectors to obtain the bitstream.

[0104] Corresponding to the application scenarios and methods of the image reconstruction method provided in the embodiments of this application, the embodiments of this application also provide an image reconstruction apparatus. This image reconstruction apparatus may include: a bitstream acquisition module, used to acquire the bitstream of a target image and the structural features of the target image, wherein the bitstream is obtained by compressing the target image based on implicit features, the implicit features are determined according to the structural features and the image features of the target image, the structural features are used to characterize the structural information of image elements in the target image, and the image features are used to characterize the image information of the image elements; a decoding module, used to decode the bitstream to obtain the implicit features; and a reconstruction module, used to generate a reconstructed image of the target image based on the structural features and the implicit features.

[0105] In one embodiment, the decoding module is specifically used to: decode the bitstream to obtain a discrete feature vector; determine statistical attributes based on the feature distribution of the pre-acquired implicit features; and convert the discrete feature vector into the implicit features based on the statistical attributes.

[0106] In one embodiment, the structural features and the implicit features have the same dimension, and the reconstruction module is specifically used to: for any dimension of implicit features, determine the corresponding dimension of reconstruction features based on the implicit features and the corresponding dimension of structural features; and reconstruct the reconstruction features of each dimension using a diffusion model to obtain the reconstructed image.

[0107] In one embodiment, the reconstruction module is further configured to: generate semantic features of the structural features; and add the implicit features to the semantic features of the structural features of the corresponding dimension to obtain the reconstructed features.

[0108] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0109] Figure 10 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 10As shown, the electronic device includes a memory 1001 and a processor 1002. The memory 1001 stores a computer program that can run on the processor 1002. When the processor 1002 executes the computer program, it implements the method described in the above embodiments. The number of memories 1001 and processors 1002 can be one or more. In a specific implementation, the electronic device may also include a communication interface 1003 for communicating with external devices and performing data exchange and transmission.

[0110] In practical implementation, if the memory 1001, processor 1002, and communication interface 1003 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0111] Optionally, in a specific implementation, if the memory 1001, processor 1002 and communication interface 1003 are integrated on a single chip, the memory 1001, processor 1002 and communication interface 1003 can communicate with each other through an internal interface.

[0112] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0113] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0114] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0115] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0116] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0117] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0118] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0119] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0120] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0121] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0122] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0123] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0124] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0125] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image compression method, comprising: Structural features and image features are extracted from the target image, wherein the structural features are used to characterize the structural information of the image elements in the target image, and the image features are used to characterize the image information of the image elements; Based on the structural features and the image features, implicit features for image compression are determined; The target image is compressed based on the implicit features to obtain a bitstream.

2. The method according to claim 1, wherein, The step of determining the implicit features for image compression based on the structural features and the image features includes: The structural features are concatenated with the image features to obtain the concatenated features; The concatenated features are convolved to obtain the dimension factor and bias term; The implicit feature is obtained by multiplying the concatenated feature by the dimension factor and then adding it to the bias term.

3. The method according to claim 2, wherein, Extracting the structural features from the target image includes: Extract the normal map from the target image and use the normal map as the structural feature.

4. The method according to claim 3, wherein, Extracting the normal map from the target image includes: The target image is input into the Normal Bayesian Active Energy Model to obtain the normal map output by the Normal Bayesian Active Energy Model, wherein the Normal Bayesian Active Energy Model is used to obtain the corresponding normal map through multiple rounds of normal prediction of the image.

5. The method according to any one of claims 1 to 4, wherein, The step of compressing the target image based on the implicit features to obtain a bitstream includes: Based on the feature distribution of the implicit feature, the statistical attributes of the implicit feature are determined, and the statistical attributes include at least one of probability density function, mean, variance, distribution shape, standard deviation, median, and quartiles. Based on the statistical properties, the image features are converted into discrete feature vectors; The discrete feature vectors are encoded to obtain the bitstream.

6. An image reconstruction method, comprising: The code stream of the target image and the structural features of the target image are obtained, wherein the code stream is obtained by compressing the target image based on implicit features, the implicit features are determined according to the structural features and the image features of the target image, the structural features are used to characterize the structural information of the image elements in the target image, and the image features are used to characterize the image information of the image elements; The implicit feature is obtained by decoding the bitstream; Based on the structural features and the implicit features, a reconstructed image of the target image is generated.

7. The method according to claim 6, wherein, Decoding the bitstream to obtain the implicit feature includes: The bitstream is decoded to obtain discrete feature vectors; Based on the feature distribution of the implicit feature, the statistical attributes of the implicit feature are determined, and the statistical attributes include at least one of probability density function, mean, variance, distribution shape, standard deviation, median, and quartiles. The discrete feature vector is transformed into the implicit feature based on the statistical properties.

8. The method according to claim 6, wherein, The structural features and the implicit features have the same dimension. Generating a reconstructed image of the target image based on the structural features and the implicit features includes: For any implicit feature in any dimension, the reconstructed feature of the corresponding dimension is determined based on the implicit feature and the structural feature of the corresponding dimension. The reconstructed image is obtained by reconstructing the features of each dimension using a diffusion model.

9. The method according to claim 8, wherein, The structural feature is a normal map. For any dimension of implicit features, the corresponding dimension's reconstructed features are determined based on the implicit features and the corresponding dimension's structural features, including: Convert the normal map into a semantic feature map; The semantic feature map is transformed into semantic features with the same dimension as the implicit features; For any dimension, the implicit feature is added to the semantic feature of the corresponding dimension to obtain the reconstructed feature of the corresponding dimension.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 9.

11. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 9.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images