An image reconstruction method, device, storage medium and computer program product

Abstract

Embodiments of the present application provide an image reconstruction method, device, storage medium and computer program product. The method comprises: obtaining high-frequency sub-band coefficients and low-frequency sub-band coefficients corresponding to a first image; determining a first high-frequency coefficient block corresponding to the high-frequency sub-band coefficients and a first low-frequency approximate image block corresponding to the low-frequency sub-band coefficients; wherein the first high-frequency coefficient block comprises a high-frequency coefficient block corresponding to each layer in a three-layer wavelet transform; determining a reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determining an adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein the adaptive sampling rate is used to adjust a sampling rate of the high-frequency coefficient block of each layer in a corresponding direction; determining a reconstructed high-frequency coefficient block based on the adaptive sampling rate, and determining a compressed reconstructed image based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image, thereby improving the quality of the compressed reconstructed image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image reconstruction method, apparatus, storage medium, and computer program product. Background Technology

[0002] In the field of image processing and signal restoration, efficiently reconstructing high-quality images at low sampling rates is an important research direction. Multiscale Block Compressed Sensing (MS-BCS) is a commonly used technique for signal acquisition and reconstruction. This method achieves efficient compression and restoration of high-dimensional data through a non-uniform sampling strategy, making it suitable for image processing scenarios in resource-constrained environments. Meanwhile, deep learning techniques are also widely applied to image reconstruction tasks. By extracting image features and optimizing the restoration process through neural network structures, they improve reconstruction quality.

[0003] In related technologies, some solutions attempt to combine MS-BCS with deep learning to leverage the advantages of both to improve image reconstruction results. For example, some methods introduce the sparse representation concept of MS-BCS during deep learning training to reduce the amount of training data; or use deep models within the MS-BCS framework to optimize sampling strategies and reconstruction algorithms. However, these methods typically do not fully consider the multi-scale characteristics of images and the adaptive processing of high-frequency information, resulting in fixed sampling rates across different frequency bands, making it difficult to dynamically adjust according to image content, thus affecting the overall reconstruction accuracy and efficiency.

[0004] In summary, the main problem with related technologies is that in the process of image reconstruction, a uniform sampling rate is often used to process all frequency components, ignoring the differences in local image structure. This can easily lead to loss of detail or redundant sampling, affecting image quality and increasing computational burden. Summary of the Invention

[0005] This application provides an image reconstruction method, apparatus, storage medium, and computer program product that can improve sampling efficiency and accuracy.

[0006] The technical solution of this application embodiment is implemented as follows: In a first aspect, embodiments of this application provide an image reconstruction method, the method comprising: Obtain the high-frequency subband coefficients and low-frequency subband coefficients corresponding to the first image; and determine the first high-frequency coefficient block corresponding to the high-frequency subband coefficients and the first low-frequency approximate image block corresponding to the low-frequency subband coefficients; wherein, the first high-frequency coefficient block includes the high-frequency coefficient block corresponding to each of the three-layer wavelet transform; The reconstructed low-frequency approximate image is determined based on the first low-frequency approximate image block, and an adaptive sampling rate is determined based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein, the adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction; The reconstructed high-frequency coefficient block is determined based on the adaptive sampling rate, and the compressed reconstructed image is determined based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image.

[0007] Secondly, embodiments of this application provide an image reconstruction device, the image reconstruction device comprising: a processor and a memory; wherein, The memory is used to store computer programs that can run on the processor; The processor is configured to execute the image reconstruction method described above when running the computer program.

[0008] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer program code, which, when executed by a computer, implements the image reconstruction method described above.

[0009] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the image reconstruction method described above.

[0010] This application provides an image reconstruction method, device, storage medium, and computer program product. The method includes: acquiring high-frequency subband coefficients and low-frequency subband coefficients corresponding to a first image; and determining a first high-frequency coefficient block corresponding to the high-frequency subband coefficients and a first low-frequency approximate image block corresponding to the low-frequency subband coefficients; wherein the first high-frequency coefficient block includes high-frequency coefficient blocks corresponding to each layer in a three-layer wavelet transform; determining a reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determining an adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein the adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient blocks of each layer in the corresponding direction; determining the reconstructed high-frequency coefficient block based on the adaptive sampling rate, and determining a compressed reconstructed image based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image. Therefore, the embodiments of this application can determine the reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determine the adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block. The adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction. That is, the embodiments of this application can adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction, thereby ensuring that more samples are collected in key areas and sampling is reduced in non-key areas, thereby improving sampling efficiency and accuracy, while avoiding the problems of oversampling and inefficient sampling, and reducing unnecessary computation. Furthermore, the reconstructed high-frequency coefficient block can be determined based on the adaptive sampling rate, and the compressed reconstructed image can be determined based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image, which greatly improves the quality of the compressed reconstructed image. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the image reconstruction method proposed in the embodiments of this application. Figure 1 ; Figure 2 This is a schematic diagram of the GAN network structure proposed in the embodiments of this application; Figure 3 This is a schematic diagram of the image reconstruction method proposed in the embodiments of this application. Figure 2 ; Figure 4 This is a schematic diagram of the three-layer wavelet decomposition image proposed in the embodiments of this application; Figure 5 This is a visual saliency diagram proposed in the embodiments of this application; Figure 6 This is a schematic diagram of the composition and structure of the image reconstruction device proposed in the embodiments of this application. Figure 1 ; Figure 7 This is a schematic diagram of the composition and structure of the image reconstruction device proposed in the embodiments of this application. Figure 2 . Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the accompanying drawings.

[0013] In the current fields of signal processing and machine learning, MS-BCS and deep learning are two major research hotspots. MS-BCS, with its ability to efficiently recover signals at low sampling rates, has shown great potential in processing high-dimensional data, compression storage, and signal processing. Deep learning, on the other hand, has achieved remarkable results in image prediction, image restoration, and other fields by learning representations and predictions through multi-layer neural networks.

[0014] With the continuous development of technology, combining MS-BCS with deep learning aims to generate innovative results in multiple fields. The advantage of this combination is that MS-BCS can reduce the amount of data and improve training efficiency during deep learning training; at the same time, deep learning can learn better sampling strategies and recovery algorithms within MS-BCS.

[0015] However, the integration of MS-BCS with deep learning still faces some challenges. First, designing a suitable sampling matrix to ensure sufficient signal information is acquired at low sampling rates while minimizing the impact of noise and redundant data is a pressing issue. Furthermore, traditional MS-BCS does not consider prior information about the image itself during the sampling process. Therefore, improving the quality of compressed reconstruction while maintaining the original reconstruction algorithm and sampling methods usually requires increasing the sampling rate. However, increasing the sampling rate makes hardware implementation more complex and difficult, hindering the full realization of MS-BCS's advantages. Second, effectively integrating the ideas of MS-BCS into deep learning models to improve their generalization ability and robustness is also a key focus of current research.

[0016] Furthermore, Generative Adversarial Networks (GANs) struggle to fully recover features when compressing and reconstructing images with complex textures. This often results in overly smooth compressed and reconstructed images that fail to accurately capture texture details, presenting certain limitations. Traditional MS-BCS applies a uniform sampling rate to all sub-blocks within the same scale, ignoring differences in texture complexity between sub-blocks, leading to wasted sampling resources or loss of crucial texture details. Moreover, GANs learn generative models based on the statistical characteristics of the input data. When combining MS-BCS with GANs for compression and reconstruction, an unreasonable sampling rate allocation will affect the statistical characteristics of the input data, impacting the iterative learning of the generator and discriminator networks, and consequently causing unnatural or distorted texture details in the reconstructed image. MS-BCS does not process low-frequency approximation images, resulting in block artifacts at the edges of these approximation images, affecting the quality of the final compressed and reconstructed image.

[0017] To address the problem that current methods use a uniform sampling rate to process all frequency components, ignoring local structural differences in the image and easily leading to loss of detail or redundant sampling, thus affecting image quality and increasing computational burden, this application provides an image reconstruction method, device, storage medium, and computer program product. The method includes: acquiring high-frequency subband coefficients and low-frequency subband coefficients corresponding to a first image; and determining a first high-frequency coefficient block corresponding to the high-frequency subband coefficients and a first low-frequency approximate image block corresponding to the low-frequency subband coefficients; wherein the first high-frequency coefficient block includes high-frequency coefficient blocks corresponding to each of the three-layer wavelet transform; determining a reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determining an adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein the adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient blocks of each layer in the corresponding direction; determining the reconstructed high-frequency coefficient block based on the adaptive sampling rate, and determining a compressed reconstructed image based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image. Therefore, the embodiments of this application can determine the reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determine the adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block. The adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction. That is, the embodiments of this application can adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction, thereby ensuring that more samples are collected in key areas and sampling is reduced in non-key areas, thereby improving sampling efficiency and accuracy, while avoiding the problems of oversampling and inefficient sampling, and reducing unnecessary computation. Furthermore, the reconstructed high-frequency coefficient block can be determined based on the adaptive sampling rate, and the compressed reconstructed image can be determined based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image, which greatly improves the quality of the compressed reconstructed image.

[0018] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0019] This application provides an image reconstruction method. Figure 1 This is a schematic diagram of the image reconstruction method proposed in the embodiments of this application. Figure 1 ,like Figure 1 As shown, the image reconstruction method may include the following steps: Step 101: Obtain the high-frequency subband coefficients and low-frequency subband coefficients corresponding to the first image; and determine the first high-frequency coefficient block corresponding to the high-frequency subband coefficients and the first low-frequency approximate image block corresponding to the low-frequency subband coefficients; wherein, the first high-frequency coefficient block includes the high-frequency coefficient block corresponding to each layer in the three-layer wavelet transform.

[0020] In the embodiments of this application, the image reconstruction device can obtain the high-frequency subband coefficients and low-frequency subband coefficients corresponding to the first image; and determine the first high-frequency coefficient block corresponding to the high-frequency subband coefficients and the first low-frequency approximate image block corresponding to the low-frequency subband coefficients.

[0021] It should be noted that, in the embodiments of this application, the image reconstruction device can be any terminal or device with storage and communication functions. For example, the image reconstruction device can be a personal computer (PC). This application does not specifically limit the type of image reconstruction device.

[0022] It should be noted that, in the embodiments of this application, when the image reconstruction device obtains the high-frequency subband coefficients and low-frequency subband coefficients corresponding to the first image, the first image can undergo three layers of wavelet transform to obtain the high-frequency subband coefficients and low-frequency approximation coefficients at different scales.

[0023] It should be noted that, in the embodiments of this application, the three-level wavelet transform can decompose the first image into low-frequency approximation coefficients (containing the overall image structure) and high-frequency detail coefficients (containing information such as edges and textures) at multiple scales. The three-level wavelet transform means that the first image is decomposed three times, and each decomposition produces a finer-grained low-frequency approximation image and high-frequency sub-bands in three directions (horizontal, vertical, and diagonal directions).

[0024] It should be noted that, in the embodiments of this application, each high-frequency subband coefficient can be further divided into several high-frequency coefficient blocks, and the low-frequency approximation image is also divided into several low-frequency approximation image blocks (i.e., the first low-frequency approximation image block). This is to facilitate subsequent processing. For example, for a first image of size 256×256, after performing a three-level decomposition using 9 / 7 orthogonal wavelet transform, the low-frequency approximate image size of the third level is 32×32, while each high-frequency subband coefficient is divided into multiple 256 high-frequency coefficient blocks (i.e., the first high-frequency coefficient block).

[0025] Step 102: Determine the reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determine the adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein, the adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction.

[0026] In the embodiments of this application, after the image reconstruction device acquires the high-frequency subband coefficients and low-frequency subband coefficients corresponding to the first image, and determines the first high-frequency coefficient block corresponding to the high-frequency subband coefficients and the first low-frequency approximate image block corresponding to the low-frequency subband coefficients, it can determine the reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determine the adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block.

[0027] It should be noted that, in the embodiments of this application, a reconstructed low-frequency approximation image is first generated using a first low-frequency approximation image block. Since the low-frequency approximation image contains the main structural information of the first image, its reconstruction process is generally simpler. However, to reduce block artifacts, visual saliency value (VS) is used to classify the first low-frequency approximation image block. Subsequently, based on the first low-frequency approximation image block and the first high-frequency coefficient block, an adaptive sampling rate for each high-frequency coefficient block is calculated. The adaptive sampling rate refers to dynamically adjusting the sampling density of each high-frequency coefficient block according to the content of the first image to improve sampling efficiency and avoid resource waste. In the embodiments of this application, the adaptive sampling strategy combines the visual saliency and directional features of the first image, increasing the sampling density in key areas and decreasing the sampling density in non-key areas, thereby achieving more efficient image compression.

[0028] Optionally, in embodiments of this application, when determining the adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block, the image reconstruction device may first determine the visual saliency value corresponding to the first low-frequency approximate image block; then, it may determine the first sampling rate based on the visual saliency value and the absolute value of the wavelet coefficients corresponding to the first high-frequency coefficient block; wherein, the first sampling rate includes the total sampling rate of the i-th high-frequency coefficient block in each layer, where i is a positive integer; furthermore, it may determine the first direction information corresponding to the first low-frequency approximate image block, and determine the adaptive sampling rate based on the first direction information and the first sampling rate; wherein, the first direction information includes one or more of the vertical direction, diagonal direction, and horizontal direction.

[0029] It should be noted that, in the embodiments of this application, the visual saliency value is a quantitative indicator used to measure the attractiveness of a certain region in an image to the human eye. The higher the visual saliency value, the more easily the region corresponding to the visual saliency value, i.e., the first low-frequency approximate image patch, attracts human attention. In the embodiments of this application, by calculating the visual saliency value of the first low-frequency approximate image patch, regions with high visual importance in the image can be identified, thereby providing a basis for subsequent sampling rate allocation.

[0030] It should be noted that, in the embodiments of this application, the absolute value of wavelet coefficients may include the absolute value of wavelet coefficients of the i-th high-frequency coefficient block in each layer. The absolute value of wavelet coefficients reflects the energy distribution of high-frequency subbands in the image. Regions with higher energy usually contain more detailed information, such as edges and textures.

[0031] It should be noted that, in the embodiments of this application, the first direction information may refer to the distribution of the main gradient directions in the image patch. Common directions include the vertical direction, the horizontal direction, and the diagonal direction. Direction information is crucial for optimizing the sampling strategy because signal changes in different directions may have different frequency characteristics.

[0032] It should be noted that, in the embodiments of this application, a multi-dimensional adaptive sampling rate allocation mechanism is constructed by introducing visual saliency values, the absolute values ​​of wavelet coefficients corresponding to the first high-frequency coefficient block, and first direction information. This allows for more accurate identification of key regions in the image, enabling reasonable allocation of sampling resources, avoiding oversampling or undersampling, and ultimately improving image compression efficiency while maintaining high-quality reconstruction results.

[0033] For example, in an embodiment of this application, when determining the visual saliency value corresponding to the first low-frequency approximate image block, the image reconstruction device can first calculate a visual saliency map based on the low-frequency approximate image, and then calculate the visual saliency value of the salient image block (i.e., the first low-frequency approximate image block).

[0034] For example, in an embodiment of this application, the image reconstruction device can perform deDC processing on a low-frequency approximation image to obtain a deDC-reduced image. The calculation process is shown in the following formula (1). Then, the first matrix can be obtained by performing a Discrete Cosine Transform (DCT) on the deDC-reduced image. The calculation process is shown in the following formula (2).

[0035] (1) in, Represents a low-frequency approximation image. This represents the mean of the low-frequency approximation image. This represents the image after DC removal.

[0036] (2) in, This represents the image after DC removal. This represents the signal after DCT transformation (i.e., the first matrix). , Indicates image size, This represents the compensation coefficient, ensuring that the matrix after DCT transformation is an orthogonal matrix.

[0037] For example, in an embodiment of this application, the first matrix is ​​symbolically transformed as shown in the following formula (3).

[0038] (3) in, This represents the first matrix after the sign transformation.

[0039] For example, in an embodiment of this application, the first matrix after symbol transformation... Performing an Inverse Discrete Cosine Transformation (IDCT) yields the transformed... As shown in the following formula (4), and for the converted After taking the absolute value, the visual saliency map is obtained by passing it through a Gaussian low-pass filter, as shown in the following formula (5).

[0040] (4) Where cos represents the cosine function.

[0041] (5) in, Represents a visual saliency map. This represents a Gaussian low-pass filter. Indicates the converted Take the absolute value. , This indicates the absolute value operation.

[0042] Furthermore, in the embodiments of this application, the visual saliency value of the salient image patch (i.e., the first low-frequency approximate image patch) corresponding to the visual saliency map is calculated using the following formula (6). (6) in, Indicates visual saliency value, Indicates the block size, Indicates image size, This represents the sign transformation value of the pixel at position j in the i-th image block of the visual saliency map.

[0043] Optionally, in embodiments of this application, when determining the first sampling rate based on the visual saliency value and the absolute value of the wavelet coefficients corresponding to the first high-frequency coefficient block, the image reconstruction device can perform weighted processing on the visual saliency value and the absolute value of the wavelet coefficients to obtain a pre-allocation factor; then, the first sampling rate can be determined based on the pre-allocation factor, the second sampling rate, and the third sampling rate; wherein, the second sampling rate includes a preset total sampling rate range, and the third sampling rate includes the lower limit value of the total sampling rate range.

[0044] It should be noted that, in the embodiments of this application, the second sampling rate is a preset sampling rate range, such as [0.1, 0.5], representing the maximum and minimum allowed sampling ratios. The sampling rate range corresponding to the second sampling rate can be flexibly adjusted according to factors such as hardware capabilities and transmission bandwidth to ensure the feasibility and stability of operation. This application does not specifically limit the total sampling rate range included in the second sampling rate.

[0045] It should be noted that, in the embodiments of this application, the third sampling rate is the lower limit of the total sampling rate range, i.e., the minimum sampling rate, used to prevent a severe degradation in image quality due to excessively low sampling. The third sampling rate serves as a safety threshold, ensuring that even in extreme cases, a basically acceptable image reconstruction effect can still be obtained.

[0046] For example, in an embodiment of this application, when the image reconstruction device performs weighted processing on the visual saliency value and the absolute value of the wavelet coefficients to obtain the pre-allocation factor, it can first calculate the absolute value of the coefficients (i.e., the absolute value of the wavelet coefficients) of the high-frequency coefficient block numbered i in each high-frequency coefficient sub-band of each layer, as shown in the following formula (7). Then, the visual saliency value and the absolute value of the wavelet coefficients can be weighted and combined to calculate the sampling rate pre-allocation factor. The calculation process is shown in the following formula (8).

[0047] (7) in, Represents the absolute value of the wavelet coefficients. This indicates the position of the i-th high-frequency coefficient block in each layer. The coefficient value, where l is the layer number and i represents the coefficient block number. This represents the subband size of the high-frequency coefficients in the l-th layer.

[0048] (8) Indicates the pre-allocation factor. This represents the weighting factor, which can be set to 0.75. Indicates the visual saliency value. This represents the absolute value of the wavelet coefficients.

[0049] It should be noted that, in the embodiments of this application, there is a synergistic relationship between the visual saliency value and the absolute value of the wavelet coefficients corresponding to the first high-frequency coefficient block. The visual saliency value focuses on the perceptual characteristics of the human visual system, while the absolute value of the wavelet coefficients corresponding to the first high-frequency coefficient block reflects the structural information of the image itself. Combining the visual saliency value with the absolute value of the wavelet coefficients corresponding to the first high-frequency coefficient block allows for the simultaneous consideration of subjective perception and objective structure, thereby obtaining a more scientific and reasonable sampling rate allocation strategy.

[0050] Optionally, in the embodiments of this application, when the image reconstruction device determines the first sampling rate based on the pre-allocated factor, the second sampling rate, and the third sampling rate, it assumes that the second sampling rate includes a pre-set sampling rate range, such as [0.1, 0.5], and sets the lower bound of the sampling rate to... The total sampling rate of the high-frequency coefficient block at position i in each layer can be calculated using the following formula (9). (i.e., the first sampling rate).

[0051] (9) in, Indicates the second sampling rate. Indicates the third sampling rate. Indicates the first sampling rate. This represents the pre-allocation factor.

[0052] It should be noted that, in the embodiments of this application, by weighting the visual saliency value and the absolute value of the wavelet coefficients to generate a pre-assignment factor, and combining it with a preset sampling rate range and lower limit, the total sampling rate of the high-frequency coefficient block at position i in each layer is dynamically determined. This enables an adaptive sampling strategy for image content, improves image compression efficiency and reconstruction quality, and provides higher quality input data for double difference GAN, thereby improving the training effect and output accuracy of double difference GAN.

[0053] It should be noted that, in the embodiments of this application, after the image reconstruction device determines the first sampling rate based on the visual saliency value and the absolute value of the wavelet coefficients corresponding to the first high-frequency coefficient block, it can determine the adaptive sampling rate based on the first direction information corresponding to the first low-frequency approximate image block and the first sampling rate; wherein, the first direction information includes one or more of the vertical direction, diagonal direction, and horizontal direction.

[0054] For example, in an embodiment of this application, when determining the first direction information corresponding to the first low-frequency approximate image block, the image reconstruction device can perform... The gradient matrix G of (i.e., the first low-frequency approximate image block) is subjected to singular value decomposition, as shown in the following formula (10). (10) in, Indicates size is Let S be an orthogonal matrix and V be an orthogonal matrix. and Representing the gradient field and The magnitude of energy in the principal direction, and Representing the gradient field and The main direction, , , , These represent the feature vector values.

[0055] Optionally, in embodiments of this application, the image reconstruction device can determine directional salience, For directional significance parameters, when hour, There is no obvious directionality, when Time indicates It has a clear directionality; among them, This is a directional empirical threshold. For clearly directional... The principal direction angle can be calculated using the following formula (11). (i.e., first direction information).

[0056] (11) in, This represents the arctangent function.

[0057] Optionally, in embodiments of this application, when the image reconstruction device determines the adaptive sampling rate based on the first direction information and the first sampling rate, it may determine the fourth sampling rate based on the first sampling rate; wherein the fourth sampling rate includes the sampling rate of the i-th high-frequency coefficient block in each layer; and then the adaptive sampling rate may be determined based on the first direction information, the fourth sampling rate, and a preset coefficient value; wherein the preset coefficient value is associated with the first direction information.

[0058] For example, in an embodiment of this application, when the image reconstruction device determines the fourth sampling rate based on the first sampling rate, it can... As the total sampling rate of the high-frequency coefficient block at position i in each layer, calculate the sampling rate corresponding to each layer. (i.e., the fourth sampling rate), and the sampling rate of the low-frequency subband. The calculation formula is shown in formula (12) below.

[0059] (12) For example, in the embodiments of this application, when the image reconstruction device determines the adaptive sampling rate based on the first direction information, the fourth sampling rate, and the preset coefficient value, it can combine... The main direction (i.e., the first direction information) is calculated. Adaptive sampling rate of (i.e., the first high-frequency coefficient block) Assuming that Taking the vertical direction as an example, the high-frequency coefficient blocks at the corresponding positions of each layer should be sampled more on the sub-bands of the corresponding main direction and less on the sub-bands of other directions, as shown in the following formula (13).

[0060] (13) in, This indicates the sampling rate with the main direction being vertical. Indicates the sampling rate in the diagonal direction. Indicates the sampling rate in the horizontal direction. This represents the sampling rate corresponding to each layer (i.e., the fourth sampling rate). , This indicates the preset coefficient value.

[0061] It should be noted that, in the embodiments of this application, the preset coefficient values ​​are a set of parameters pre-set based on the first direction information, used to adjust the generation logic of the adaptive sampling rate. The preset coefficient values ​​reflect the directional characteristics of the image structure, such as the information intensity in the vertical, horizontal, or diagonal directions. By combining the preset coefficient values ​​with the first direction information and introducing a fourth sampling rate as a basis, the final sampling rate of each sub-block can be dynamically adjusted, resulting in higher sampling density in key directions and appropriately lower sampling density in non-key directions, thereby optimizing the overall sampling strategy.

[0062] In other words, in the embodiments of this application, by introducing a fourth sampling rate and a preset coefficient value, an adaptive sampling rate adjustment mechanism based on image content characteristics is realized, which not only improves the intelligence of the sampling process, but also provides higher quality data support for subsequent GAN model training.

[0063] Step 103: Determine the reconstructed high-frequency coefficient block based on the adaptive sampling rate, and determine the compressed reconstructed image based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image.

[0064] In the embodiments of this application, when the image reconstruction device determines the reconstructed high-frequency coefficient block based on the adaptive sampling rate, it can determine the measurement matrix based on the adaptive sampling rate; wherein, the measurement matrix is ​​used for dimensionality reduction observation; then, it can determine the first measurement value corresponding to the first high-frequency coefficient block based on the measurement matrix, and generate the reconstructed high-frequency coefficient block based on the initial reconstruction result corresponding to the first measurement value, the generator, and the discriminator; wherein, the generator is trained based on a first loss function, the first loss function including a second loss function and a third loss function, the second loss function is used to measure the difference between the edge information of the reconstructed image and the edge information of the original image, and the third loss function is used to reflect the difference between the pixels of the reconstructed image and the pixels of the original image.

[0065] It should be noted that, in the embodiments of this application, the measurement matrix can be used to project high-dimensional signal or image data into a low-dimensional space to reduce the amount of data while retaining key information. The measurement matrix can be composed of random numbers, such as a Gaussian distributed random matrix, the size of which depends on the sampling rate and the signal dimension. In the embodiments of this application, the measurement matrix can be adaptively designed based on the visual saliency and directional features of the image, thereby collecting more samples in key areas and reducing the sampling density in non-key areas, thus improving sampling efficiency and accuracy. This can improve the quality and efficiency of subsequent reconstruction without increasing the computational burden.

[0066] It should be noted that, in the embodiments of this application, the initial reconstruction result refers to the image estimate obtained by initially reconstructing the measured values ​​using a compressed sensing algorithm (such as the Landweber iterative algorithm). The initial reconstruction result provides the input basis for subsequent deep learning models and helps guide the generator to produce results that are closer to the real image.

[0067] It should be noted that, in the embodiments of this application, the generator is part of the GAN model and is responsible for generating high-quality image output from low-dimensional measurement vectors. The goal of the generator is to make the generated image as realistic as possible.

[0068] It should be noted that in the embodiments of this application, the discriminator is another part of the GAN model, responsible for determining whether the input image comes from a real dataset or was generated by the generator. The discriminator continuously optimizes the generator's output through adversarial training, enabling the generator to produce higher-quality images. For example, in practical applications, the generator might be a Convolutional Neural Network (CNN), while the discriminator might also be a CNN structure. The generator and discriminator are trained alternately until convergence. The synergistic effect of the generator and discriminator can achieve high-quality reconstruction of high-frequency coefficient blocks.

[0069] For example, in an embodiment of this application, when the image reconstruction device determines the first measurement value corresponding to the first high-frequency coefficient block based on the measurement matrix, it can select a value of size [missing value]. Gaussian random matrix; combined with ,calculate The number of measurements is used to generate a measurement matrix. (i.e., measurement matrix) dimensionality reduction observation, to obtain Measured values (i.e., the first measurement value).

[0070] It should be noted that, in the embodiments of this application, when the image reconstruction device generates the reconstructed high-frequency coefficient block based on the initial reconstruction result corresponding to the first measurement value, the generator, and the discriminator, Figure 2 This is a schematic diagram of the GAN network structure proposed in the embodiments of this application, such as... Figure 2 As shown, the initial reconstruction result can be calculated using the following formula (14). Then, the initial reconstruction result can be input into the generator, and the generator's output result and the original high-frequency coefficient block can be combined. The first high-frequency coefficient block (i.e., the first high-frequency coefficient block) is used as the input of the discriminator for judgment, thereby obtaining the final reconstructed high-frequency coefficient block.

[0071] (14) in, This indicates the initial reconstruction result. Represents the transpose of a Gaussian random matrix. This indicates the first measured value.

[0072] It should be noted that in the embodiments of this application, the generator hopes that the probability of the output high-frequency coefficient block being judged as true by the discriminator is as high as possible, and its loss function can be the following formula (15).

[0073] (15) in, This represents the generator loss weight parameters. The network mapping of the generator, This represents the reconstructed image output by the generator.

[0074] It should be noted that, in the embodiments of this application, considering that GAN models have difficulty fully recovering these features when reconstructing images with complex texture structures, the embodiments of this application... Add total variation loss function (i.e., the second loss function) and the residual loss function (i.e., the third loss function) is used to calculate and construct the DD-GAN loss function, where It can effectively measure the pixel edge activity within a local area of ​​an image. If the reconstructed image can preserve the edge information of the original image well, then its total variation value should be close to that of the original image. This includes the reconstructed image residual and the measurement vector residual, which reflect the differences at each pixel level within the overall range between the reconstructed image and the original image. Incorporating both into the loss function as constraints allows the reconstructed image to more closely approximate the original image; among them, and The calculations are shown in formulas (16) and (17) below, respectively.

[0075] (16) in, Indicates the size of the image block.

[0076] (17) in, Represents the overall residual of the reconstructed image. This represents the measurement vector residual.

[0077] It should be noted that, in the embodiments of this application, the total loss function of the generator (i.e., the first loss function) can be as shown in the following formula (18).

[0078] (18) It should be noted that, in the embodiments of this application, for the discriminator, when the input is the original high-frequency coefficients... When the input is the generator output, the higher the discrimination probability, the better; when the input is the generator output... When the probability of discrimination is as small as possible, as shown in the following formula (19).

[0079] (19) Furthermore, by rewriting the minimum value in the above formula (19) as the maximum value, the loss function of the discriminator can be obtained. As shown in the following formula (20).

[0080] (20) It should be noted that, in the embodiments of this application, the generator and discriminator network parameters are iteratively trained, and the loss function is calculated to determine whether the DD-GAN network has converged. That is, during the iterative training process of the network, if... and When the values ​​of the two networks steadily decrease towards a certain value, the DD-GAN network is considered to have converged. Afterwards, the convergence can be determined. The input is fed into the converged DD-GAN network, and the generator outputs the final reconstructed high-frequency coefficient block.

[0081] It should be noted that, in the embodiments of this application, after determining the first low-frequency approximate image block corresponding to the low-frequency sub-band coefficient, the image reconstruction device can determine the reconstructed low-frequency approximate image based on the first low-frequency approximate image block.

[0082] Optionally, in embodiments of this application, when the image reconstruction device determines the reconstructed low-frequency approximate image based on the first low-frequency approximate image block, it can perform segmentation processing on the first low-frequency approximate image block based on the visual saliency value corresponding to the first low-frequency approximate image block and a preset clustering algorithm to obtain a segmentation result; wherein, the segmentation result includes non-salient blocks, transition blocks, and salient blocks; then, blind deconvolution processing can be performed on the non-salient blocks to obtain reconstructed non-salient blocks, and adaptive weighted filtering processing can be performed on the transition blocks and salient blocks to obtain filtered low-frequency approximate image blocks; then, the reconstructed low-frequency approximate image can be determined based on the reconstructed non-salient blocks and the filtered low-frequency approximate image blocks.

[0083] It should be noted that, in the embodiments of this application, the preset clustering algorithm can be a machine learning method that classifies image patches based on their visual saliency values, thereby dividing the image patches into different categories. In the embodiments of this application, the K-means++ clustering algorithm can be used as one implementation of the preset clustering algorithm, or other clustering algorithms can be used. This application does not specifically limit the type of preset clustering algorithm.

[0084] For example, in the embodiments of this application, the K-means++ clustering algorithm divides image patches into three categories: salient patches, transitional patches, and non-salient patches, based on their salience from high to low. This classification method can effectively distinguish different structural regions in the image, thereby facilitating the application of different processing methods for different types of image patches.

[0085] Optionally, in embodiments of this application, when the image reconstruction device performs blind deconvolution processing on non-salient blocks to obtain reconstructed non-salient blocks, the non-salient blocks can be regarded as a blurring degradation problem, and its degradation model is shown in the following formula (21), and the gradient of the original image is calculated. Then, fuzzy kernel The following formulas (22) and (23) can be used alternately to determine the solution. Then, a blind deconvolution restoration minimization model is established, and the solution of the blind deconvolution restoration minimization model is the reconstructed non-salient block.

[0086] (twenty one) in, Indicates a non-significant block. This represents the image patch corresponding to the original image. Indicates the fuzzy kernel. Indicates noise.

[0087] (twenty two) (twenty three) in, =0.002, =1 is the regularization parameter.

[0088] Optionally, in embodiments of this application, the blind deconvolution restoration minimization model is shown in the following formula (24).

[0089] (twenty four) in, This represents the weighting coefficient, whose function is to ensure that the noise mean is 0 during the iterative recovery process. It is typically set to a value of [value missing]. .

[0090] For example, in the embodiments of this application, when the image reconstruction device performs adaptive weighted filtering on the transition block and the salient block to obtain the filtered low-frequency approximate image block, it can be calculated by the following formulas (25)-(27).

[0091] (25) in, Indicates the pixel to be filtered, with Generate a size of centered at The filter template, for The eight neighboring pixels, express pixel values, Represented as The filtering results express The pixel value.

[0092] (26) in, , =0.001, =0.5 measures the degree of nonlinearity. Represents local variance. express The overall variance of (i.e., the first low-frequency approximation image).

[0093] (27) in, , express pixel values, express The pixel value.

[0094] It should be noted that, in the embodiments of this application, in pixels During the filtering process, the symmetrical pixel difference and weighting factor They are negatively correlated; the larger the difference, the more likely the pixels are to be negatively correlated. The closer to the edge, the more... The smaller the size, the better it is for preserving edge details. If If the pixel values ​​of (i.e., low-frequency approximate image patches) are the same, then it becomes a linear mean filter, which plays a smoothing role.

[0095] It should be noted that, in the embodiments of this application, by dividing image blocks based on visual saliency values ​​and preset clustering algorithms, computational resources and processing strategies can be allocated more rationally, improving the overall efficiency and quality of image compression and reconstruction. Differential processing can be performed based on the complexity of image content, avoiding a one-size-fits-all approach, thereby enhancing the naturalness and realism of the reconstructed image.

[0096] In other words, in the embodiments of this application, the image reconstruction device can first classify image blocks using visual saliency values ​​and a preset clustering algorithm to clarify the importance and characteristics of each image block; then, based on the identified different categories, it selects appropriate image processing methods (such as blind deconvolution processing or adaptive weighted filtering processing) to achieve a balance between detail preservation and efficiency optimization; subsequently, the processed image blocks can be integrated to generate a high-quality low-frequency approximation image. That is, the hierarchical processing method of this application, which clusters image blocks based on visual saliency values ​​and adopts different processing methods for different categories of image blocks, not only improves the intelligence level of image processing but also enhances scalability and adaptability, making it suitable for various image application scenarios.

[0097] It should be noted that, in the embodiments of this application, after obtaining the reconstructed low-frequency approximate image and the reconstructed high-frequency coefficient block, the image reconstruction device can determine the compressed reconstructed image based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image.

[0098] Optionally, in embodiments of this application, when determining a compressed reconstructed image based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image, the image reconstruction device can perform a stitching process on the reconstructed high-frequency coefficient block to obtain the stitched high-frequency coefficients; then, it can perform an inverse transform process on the high-frequency coefficients to obtain an edge texture image; and then, it can perform a normalization superposition process on the edge texture image and the reconstructed low-frequency approximate image to obtain a compressed reconstructed image.

[0099] It should be noted that, in the embodiments of this application, normalization overlay processing refers to numerically standardizing the edge texture image and the reconstructed low-frequency approximation image, and then overlaying them according to a certain ratio to synthesize the final compressed reconstructed image. The low-frequency approximation image mainly reflects the overall structure of the original image, while the edge texture image contains detailed information of the original image. Therefore, normalization overlay processing can ensure the coordination of brightness and contrast between the low-frequency approximation image and the edge texture image, avoiding problems such as brightness imbalance or over-enhancement in the compressed reconstructed image. For example, the low-frequency approximation image and the edge texture image can be added in a linear weighted manner, and normalization processing can be performed according to the maximum pixel value, so that the grayscale range of the compressed reconstructed image is between [0, 1], thereby ensuring that the visual effect of the compressed reconstructed image is natural and realistic.

[0100] In summary, this application embodiment achieves an optimized iterative training strategy by integrating the total variation loss function (i.e., the second loss function) and the residual loss function (i.e., the third loss function) into the loss function of the GAN generator network. This strategy prioritizes both the overall texture features (i.e., residuals) and local texture features (i.e., total variation) of the image during reconstruction. This significantly improves the naturalness of the reconstructed image details and the overall reconstruction quality. Secondly, this application embodiment also proposes an adaptive sampling strategy (i.e., adaptive sampling rate). Combining the image's VS and orientation features, it densely samples in key regions of the image while reducing sampling density in non-key regions, significantly improving sampling efficiency and accuracy and effectively avoiding oversampling and undersampling. This strategy also provides the generator with initial input information that better reflects the statistical characteristics of the original image data, helping the generator understand and utilize this adaptive sampling data during iterative training to generate more accurate and high-quality images. Furthermore, for low-frequency approximate images, this application embodiment employs a VS-based clustering method to divide them into salient blocks, transitional blocks, and non-salient blocks. Blind deconvolution and adaptive weighted filters are applied to different categories to reduce image blockiness and further improve the reconstruction quality.

[0101] This application provides an image reconstruction method, which includes: acquiring high-frequency subband coefficients and low-frequency subband coefficients corresponding to a first image; and determining a first high-frequency coefficient block corresponding to the high-frequency subband coefficients and a first low-frequency approximate image block corresponding to the low-frequency subband coefficients; wherein the first high-frequency coefficient block includes high-frequency coefficient blocks corresponding to each layer in a three-layer wavelet transform; determining a reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determining an adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein the adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient blocks of each layer in the corresponding direction; determining the reconstructed high-frequency coefficient block based on the adaptive sampling rate, and determining a compressed reconstructed image based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image. Therefore, the embodiments of this application can determine the reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determine the adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block. The adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction. That is, the embodiments of this application can adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction, thereby ensuring that more samples are collected in key areas and sampling is reduced in non-key areas, thereby improving sampling efficiency and accuracy, while avoiding the problems of oversampling and inefficient sampling, and reducing unnecessary computation. Furthermore, the reconstructed high-frequency coefficient block can be determined based on the adaptive sampling rate, and the compressed reconstructed image can be determined based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image, which greatly improves the quality of the compressed reconstructed image.

[0102] Based on the above embodiments, another embodiment of this application provides an image reconstruction method. Figure 3 This is a schematic diagram of the image reconstruction method proposed in the embodiments of this application. Figure 2 ,like Figure 3As shown, the image reconstruction method can include the following: First, the original image is subjected to three-level wavelet decomposition to obtain high-frequency sub-bands and low-frequency approximate images. For the low-frequency approximate images, a block-based strategy is adopted, clustering the image blocks into non-salient blocks, transitional blocks, and salient blocks based on their VS (visual representation). Depending on the VS, blind deconvolution or adaptive weighted filters are used for reconstruction to mitigate the block artifacts of the low-frequency approximate images. High-frequency sub-bands at each scale are also divided proportionally, and the sampling rate is adaptively allocated based on the VS, wavelet coefficient values, and principal direction of the corresponding low-frequency approximate image blocks. Subsequently, a measurement matrix of appropriate size is constructed for measurement, and the obtained measurement values ​​are used for initial reconstruction as input to the DD-GAN generator. A discriminator makes a judgment, and the DD-GAN loss function is constructed by combining the total variation loss function and residual loss function of the output high-frequency coefficient blocks for iterative training to obtain the high-frequency coefficient block reconstruction results. By stitching together all high-frequency coefficient blocks, the edge detail images obtained by the inverse transformation of the high-frequency sub-band coefficients are normalized and superimposed with the reconstructed low-frequency approximate images to obtain the final compressed reconstructed image.

[0103] For example, in an embodiment of this application, the image reconstruction method may include the following steps: Step 1, reconstructing an image of size... The image (not limited to this value) is decomposed into three levels using 9 / 7 orthogonal wavelet transform, with each level containing three high-frequency sub-band coefficients. ;in This represents the wavelet decomposition level. Finally, only the low-frequency subband coefficients of the third level are inversely transformed to obtain the low-frequency approximate image. Step 2: Approximate the low-frequency image. The image is uniformly divided into non-overlapping low-frequency approximation patches of size 32×32 (not limited to this value). (i.e., the first low-frequency approximate image block). Figure 4 This is a schematic diagram of the three-layer wavelet decomposition image proposed in the embodiments of this application, as shown below. Figure 4 As shown, for the high-frequency sub-band coefficients of the same layer, based on the structural characteristics of each layer of wavelet transform, a partitioning method of the same size is adopted to obtain 256 non-overlapping high-frequency coefficient blocks. (i.e., the first high-frequency coefficient block), where For the number of floors, For the first High-frequency coefficient subband of the layer High-frequency coefficient subband The block number in the middle. Block size With each layer Block size The relationship can be Step 3: Calculate the visual saliency map of the low-frequency approximation image and the saliency of each low-frequency approximation image patch. (i.e., visual salience value), Corresponding positions in each high-frequency coefficient sub-band of each layer absolute value of coefficient (i.e., the weighted combination of the absolute values ​​of wavelet coefficients) Sampling rate pre-allocation is performed to obtain the position of each high-frequency coefficient sub-band in each layer. The total sampling rate of the coefficient block (i.e., the first sampling rate), the specific process is as follows: Step 3-1: DC removal processing of low-frequency approximate image As shown in formula (1) above; then by... Perform a DCT transformation to obtain the first matrix, as shown in formula (2) above. Perform a sign transformation on the first matrix, and then perform a sign transformation on the first matrix. Perform IDCT transformation to obtain the transformed... As shown in formula (4) above, and for the converted After taking the absolute value, the visual saliency map is obtained by passing it through a Gaussian low-pass filter, as shown in the above formula (5); Figure 5 This is a visual saliency diagram proposed in the embodiments of this application; the salient image block (i.e., the first low-frequency approximate image block) corresponding to the visual saliency diagram is calculated using the above formula (6). (i.e., visual saliency value); Step 3-2: Calculate the numbered subbands of each high-frequency coefficient in each layer. The absolute value of the coefficients of the high-frequency coefficient block (i.e., the absolute value of the wavelet coefficients), where For each level The positions of the high-frequency coefficient blocks are as follows: Coefficient value; Number of floors; Number the coefficient blocks. ; For the first Layer high-frequency coefficient subband block size; Step 3-3: ... and The sampling rate pre-allocation factor is calculated by weighted combination. As shown in formula (8) above; Step 3-4: Assume the total sampling rate (second sampling rate) is set to =[0.1, 0.5] (not limited to these values), setting the lower bound of the sampling rate (i.e., the third sampling rate) to . =0.1, calculate the position in each layer as The total sampling rate of the high-frequency coefficient block (i.e., the first sampling rate), the calculation formula is as shown in formula (9) above; Step 4, calculate (i.e., the main direction of the first low-frequency approximate image patch) and based on the corresponding position Based on the directionality, the sampling rate is redistributed. Adaptive sampling rate ,according to The size of the construction is Measurement Matrix Dimensional reduction observation, among which It is a Gaussian random matrix; the specific process is as follows, step 4-1: It can be used to... The gradient matrix G is subjected to singular value decomposition, as shown in the following formula (10); Step 4-2: Determine directional salience, For directional significance parameters, when hour, There is no obvious directionality, when Time indicates It has a clear directionality; among them, This is a directional empirical threshold. For clearly directional... The principal direction angle can be calculated using the formula (11) above. (i.e., first direction information); Step 4-4: ... As the location of each layer The total sampling rate of the high-frequency coefficient block is used to calculate the sampling rate corresponding to each layer. (i.e., the fourth sampling rate) and the sampling rate of the low-frequency subband. The calculation formula is shown in formula (12) above; Steps 4-5: Combine The main direction, calculation Adaptive sampling rate ;by Taking the vertical direction as the main direction, the high-frequency coefficient blocks at the corresponding positions of each layer should be sampled more on the sub-bands of the corresponding main direction and less on the sub-bands of other directions, as shown in the above formula (13); Step 4-6: Measurement matrix Select size as Gaussian random matrix; combined with ,calculate The number of measurements is used to generate a measurement matrix. Dimensional reduction observation yields Measured values (i.e., the first measured value).

[0104] Furthermore, in the embodiments of this application, the image reconstruction method may further include the following step: Step 5, converting each... Measured values Initial reconstruction results (i.e., the first measurement value) This is used as input to the DD-GAN generator, and the generator output is combined with the original high-frequency coefficient block. The image is used as input to the discriminator for judgment, and the reconstructed image output by the generator is used for further analysis. Total variation loss function and residual loss function The loss functions of the generator and discriminator are added to the generator's loss function, and the generator and discriminator of DD-GAN are iteratively trained to obtain the final reconstructed high-frequency coefficient blocks. The specific process is as follows: Step 5-1: First, based on the idea of ​​finding initial values ​​in the smooth projection Landweber algorithm in the MS-BCS algorithm, the initial reconstruction results of each high-frequency coefficient block are calculated. As shown in formula (14) above; Step 5-2: Construct the DD-GAN model, and generate discriminant training through iterative generation of the generator and discriminator, so that the reconstructed image output by the generator can be reconstructed. Total variation and residual The loss function of the generator is added to the loss function to form the DD-GAN loss function. Training continues until the generator and discriminator networks converge, and the final high-frequency coefficient reconstruction result is obtained. The specific process is as follows: The generator hopes that the probability of the output high-frequency coefficient block being judged as true by the discriminator is as high as possible. Its loss function can be the above formula (15); the total loss function of the DD-GAN generator (i.e. the first loss function) can be as shown in the above formula (18); for the discriminator, when the input is the original high-frequency coefficients When the input is the generator output, the higher the discrimination probability, the better; when the input is the generator output... At this time, the discrimination probability should be as small as possible, as shown in the above formula (19); the loss function of the discriminator is then obtained. As shown in formula (20) above; then iteratively train the generator and discriminator network parameters, calculate the loss function to determine whether the DD-GAN network has converged, that is, during the network iterative training process, if and When the values ​​of the two networks steadily decrease towards a certain value, the DD-GAN network is considered to have converged. Afterwards, the convergence can be determined. The input is fed into the converged DD-GAN network, and the generator outputs the final reconstructed high-frequency coefficients. If convergence fails, the process returns to step 5-2 iteration; step 6, based on... The significance of the differences was determined using the K-means++ clustering algorithm. The image is divided into non-salient blocks, transition blocks, and salient blocks. To address the blurring degradation problem in non-salient blocks, blind deconvolution is used to reconstruct edges and eliminate block artifacts. For transition and salient blocks, an adaptive weighted filter is used to optimize their texture details. The specific process is as follows: Step 6-1: Using each image block... As input parameters, the K-means++ clustering algorithm is used to sort the clusters according to their visual saliency from high to low. Clustering is divided into salient blocks, transitional blocks, and non-salient blocks; Step 6-2: Treat the non-salient blocks as a blur degradation problem and use blind deconvolution for processing: its degradation model is shown in the above formula (21), and the gradient of the original image is calculated. Then, fuzzy kernel The solution can be obtained by alternately solving formulas (22) and (23) above. Then, a blind deconvolution restoration minimization model is established. The solution of the blind deconvolution restoration minimization model is the reconstructed non-salient block. The blind deconvolution restoration minimization model is shown in the above formula (24). Step 6-3: When performing adaptive weighted filtering on the transition block and the salient block to obtain the filtered low-frequency approximate image block, it can be calculated by the above formula (25)-formula (27). Step 7: After splicing the high-frequency coefficient blocks reconstructed from the above layers, the complete high-frequency coefficients are obtained. The edge texture image obtained after inverse transformation is obtained. Compared with the reconstructed low-frequency approximation image The final compressed and reconstructed image is obtained by normalizing and stacking the images.

[0105] In summary, this application embodiment achieves an optimized iterative training strategy by integrating the total variation loss function (i.e., the second loss function) and the residual loss function (i.e., the third loss function) into the loss function of the GAN generator network. This strategy prioritizes both the overall texture features (i.e., residuals) and local texture features (i.e., total variation) of the image during reconstruction. This significantly improves the naturalness of the reconstructed image details and the overall reconstruction quality. Secondly, this application embodiment also proposes an adaptive sampling strategy (i.e., adaptive sampling rate). Combining the image's VS and orientation features, it densely samples in key regions of the image while reducing sampling density in non-key regions, significantly improving sampling efficiency and accuracy and effectively avoiding oversampling and undersampling. This strategy also provides the generator with initial input information that better reflects the statistical characteristics of the original image data, helping the generator understand and utilize this adaptive sampling data during iterative training to generate more accurate and high-quality images. Furthermore, for low-frequency approximate images, this application embodiment employs a VS-based clustering method to divide them into salient blocks, transitional blocks, and non-salient blocks. Blind deconvolution and adaptive weighted filters are applied to different categories to reduce image blockiness and further improve the reconstruction quality.

[0106] This application provides an image reconstruction method, which includes: acquiring high-frequency subband coefficients and low-frequency subband coefficients corresponding to a first image; and determining a first high-frequency coefficient block corresponding to the high-frequency subband coefficients and a first low-frequency approximate image block corresponding to the low-frequency subband coefficients; wherein the first high-frequency coefficient block includes high-frequency coefficient blocks corresponding to each layer in a three-layer wavelet transform; determining a reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determining an adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein the adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient blocks of each layer in the corresponding direction; determining the reconstructed high-frequency coefficient block based on the adaptive sampling rate, and determining a compressed reconstructed image based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image. Therefore, the embodiments of this application can determine the reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determine the adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block. The adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction. That is, the embodiments of this application can adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction, thereby ensuring that more samples are collected in key areas and sampling is reduced in non-key areas, thereby improving sampling efficiency and accuracy, while avoiding the problems of oversampling and inefficient sampling, and reducing unnecessary computation. Furthermore, the reconstructed high-frequency coefficient block can be determined based on the adaptive sampling rate, and the compressed reconstructed image can be determined based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image, which greatly improves the quality of the compressed reconstructed image.

[0107] Based on the above embodiments, this application provides an image reconstruction device. Figure 6 Schematic diagram of the composition structure of an image reconstruction device Figure 1 ,like Figure 6 As shown, the image reconstruction device 10 includes: an acquisition unit 11 and a determination unit 12; wherein, The acquisition unit 11 acquires the high-frequency subband coefficients and low-frequency subband coefficients corresponding to the first image; and determines the first high-frequency coefficient block corresponding to the high-frequency subband coefficients and the first low-frequency approximate image block corresponding to the low-frequency subband coefficients; wherein, the first high-frequency coefficient block includes the high-frequency coefficient block corresponding to each of the three-layer wavelet transform; The determining unit 12 is configured to determine the reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and to determine an adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein the adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction; and to determine the reconstructed high-frequency coefficient block based on the adaptive sampling rate, and to determine the compressed reconstructed image based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image.

[0108] In the embodiments of this application, further, Figure 7 Schematic diagram of the composition structure of an image reconstruction device Figure 2 ,like Figure 7 As shown, the image reconstruction device 10 proposed in this application embodiment may further include a processor 13, a memory 14 storing instructions executable by the processor 13, and further, the image reconstruction device 10 may further include a communication interface 15 and a bus 16 for connecting the processor 13, the memory 14 and the communication interface 15.

[0109] In the embodiments of this application, the processor 13 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other types, and this application embodiment does not specifically limit this. The image reconstruction device 10 may also include a memory 14, which can be connected to the processor 13. The memory 14 is used to store executable program code, which includes computer operation instructions. The memory 14 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0110] In embodiments of this application, bus 16 is used to connect communication interface 15, processor 13 and memory 14 and the mutual communication between these devices.

[0111] In embodiments of this application, memory 14 is used to store instructions and data.

[0112] Further, in an embodiment of this application, the processor 13 is configured to acquire high-frequency subband coefficients and low-frequency subband coefficients corresponding to a first image; and determine a first high-frequency coefficient block corresponding to the high-frequency subband coefficients and a first low-frequency approximate image block corresponding to the low-frequency subband coefficients; wherein, the first high-frequency coefficient block includes high-frequency coefficient blocks corresponding to each layer of the three-layer wavelet transform; determine a reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determine an adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein, the adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient blocks of each layer in the corresponding direction; determine a reconstructed high-frequency coefficient block based on the adaptive sampling rate, and determine a compressed reconstructed image based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image.

[0113] In practical applications, the aforementioned memory 14 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 13.

[0114] This application provides an image reconstruction device that acquires high-frequency subband coefficients and low-frequency subband coefficients corresponding to a first image; and determines a first high-frequency coefficient block corresponding to the high-frequency subband coefficients and a first low-frequency approximate image block corresponding to the low-frequency subband coefficients; wherein the first high-frequency coefficient block includes high-frequency coefficient blocks corresponding to each layer in a three-layer wavelet transform; a reconstructed low-frequency approximate image is determined based on the first low-frequency approximate image block, and an adaptive sampling rate is determined based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein the adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient blocks of each layer in the corresponding direction; a reconstructed high-frequency coefficient block is determined based on the adaptive sampling rate, and a compressed reconstructed image is determined based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image. Therefore, the embodiments of this application can determine the reconstructed low-frequency approximate image based on the first low-frequency approximate image block, and determine the adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block. The adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction. That is, the embodiments of this application can adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction, thereby ensuring that more samples are collected in key areas and sampling is reduced in non-key areas, thereby improving sampling efficiency and accuracy, while avoiding the problems of oversampling and inefficient sampling, and reducing unnecessary computation. Furthermore, the reconstructed high-frequency coefficient block can be determined based on the adaptive sampling rate, and the compressed reconstructed image can be determined based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image, which greatly improves the quality of the compressed reconstructed image.

[0115] This application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the image reconstruction method described above.

[0116] Specifically, the program instructions corresponding to an image reconstruction method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to an image reconstruction method in the storage media are read or executed by an electronic device, the following steps are included: Obtain the high-frequency subband coefficients and low-frequency subband coefficients corresponding to the first image; and determine the first high-frequency coefficient block corresponding to the high-frequency subband coefficients and the first low-frequency approximate image block corresponding to the low-frequency subband coefficients; wherein, the first high-frequency coefficient block includes the high-frequency coefficient block corresponding to each of the three-layer wavelet transform; The reconstructed low-frequency approximate image is determined based on the first low-frequency approximate image block, and an adaptive sampling rate is determined based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein, the adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction; The reconstructed high-frequency coefficient block is determined based on the adaptive sampling rate, and the compressed reconstructed image is determined based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image.

[0117] This application also provides a computer program product, including a computer program that can be executed by the processor 13 of the image reconstruction device 10 to perform the steps described in any of the foregoing methods.

[0118] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0119] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0120] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0122] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. An image reconstruction method, characterized in that, The method includes: Obtain the high-frequency subband coefficients and low-frequency subband coefficients corresponding to the first image; and determine the first high-frequency coefficient block corresponding to the high-frequency subband coefficients and the first low-frequency approximate image block corresponding to the low-frequency subband coefficients; wherein, the first high-frequency coefficient block includes the high-frequency coefficient block corresponding to each of the three-layer wavelet transform; The reconstructed low-frequency approximate image is determined based on the first low-frequency approximate image block, and an adaptive sampling rate is determined based on the first low-frequency approximate image block and the first high-frequency coefficient block; wherein, the adaptive sampling rate is used to adjust the sampling rate of the high-frequency coefficient block of each layer in the corresponding direction; The reconstructed high-frequency coefficient block is determined based on the adaptive sampling rate, and the compressed reconstructed image is determined based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximate image.

2. The method according to claim 1, characterized in that, The step of determining the adaptive sampling rate based on the first low-frequency approximate image block and the first high-frequency coefficient block includes: Determine the visual saliency value corresponding to the first low-frequency approximate image block; The first sampling rate is determined based on the visual saliency value and the absolute value of the wavelet coefficients corresponding to the first high-frequency coefficient block; wherein, the first sampling rate includes the total sampling rate of the i-th high-frequency coefficient block in each layer, and i is a positive integer; The first direction information corresponding to the first low-frequency approximate image block is determined, and the adaptive sampling rate is determined based on the first direction information and the first sampling rate; wherein, the first direction information includes one or more of the vertical direction, diagonal direction, and horizontal direction.

3. The method according to claim 2, characterized in that, The determination of the first sampling rate based on the visual saliency value and the absolute value of the wavelet coefficients corresponding to the first high-frequency coefficient block includes: The visual saliency value and the absolute value of the wavelet coefficients are weighted to obtain the pre-assignment factor; The first sampling rate is determined based on the pre-allocated factor, the second sampling rate, and the third sampling rate; wherein the second sampling rate includes a preset total sampling rate range, and the third sampling rate includes the lower limit of the total sampling rate range.

4. The method according to claim 2, characterized in that, Determining the adaptive sampling rate based on the first direction information and the first sampling rate includes: A fourth sampling rate is determined based on the first sampling rate; wherein, the fourth sampling rate includes the sampling rate of the i-th high-frequency coefficient block in each layer; The adaptive sampling rate is determined based on the first direction information, the fourth sampling rate, and a preset coefficient value; wherein the preset coefficient value is associated with the first direction information.

5. The method according to claim 1, characterized in that, The process of determining the reconstructed high-frequency coefficient block based on the adaptive sampling rate includes: The measurement matrix is ​​determined based on the adaptive sampling rate; wherein the measurement matrix is ​​used for dimensionality reduction observation. Based on the measurement matrix, a first measurement value corresponding to the first high-frequency coefficient block is determined, and the reconstructed high-frequency coefficient block is generated based on the initial reconstruction result corresponding to the first measurement value, the generator, and the discriminator; wherein, The generator is trained based on a first loss function, which includes a second loss function and a third loss function. The second loss function measures the difference between the edge information of the reconstructed image and the edge information of the original image, and the third loss function reflects the difference between the pixels of the reconstructed image and the pixels of the original image.

6. The method according to claim 1 or 2, characterized in that, The step of determining the reconstructed low-frequency approximation image based on the first low-frequency approximation image block includes: Based on the visual saliency value corresponding to the first low-frequency approximate image block, a preset clustering algorithm is used to divide the first low-frequency approximate image block to obtain the division result; wherein, the division result includes non-salient blocks, transition blocks and salient blocks; Blind deconvolution is performed on the non-salient blocks to obtain reconstructed non-salient blocks, and adaptive weighted filtering is performed on the transition blocks and the salient blocks to obtain filtered low-frequency approximate image blocks; The reconstructed low-frequency approximation image is determined based on the reconstructed non-salient blocks and the filtered low-frequency approximation image blocks.

7. The method according to claim 1, characterized in that, The step of determining the compressed reconstructed image based on the reconstructed high-frequency coefficient block and the reconstructed low-frequency approximation image includes: The reconstructed high-frequency coefficient blocks are spliced ​​together to obtain spliced ​​high-frequency coefficients; The high-frequency coefficients are subjected to inverse transform processing to obtain an edge texture image; The edge texture image and the reconstructed low-frequency approximation image are normalized and superimposed to obtain the compressed reconstructed image.

8. An image reconstruction device, characterized in that, The image reconstruction device includes: a processor and a memory; wherein... The memory is used to store computer programs that can run on the processor; The processor is configured to perform the method as described in any one of claims 1-7 when running the computer program.

9. A computer-readable storage medium, characterized in that, The storage medium stores computer program code, which, when executed by a computer, performs the method described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1-7.

Images