Image lossy compression method based on depth image prior

Abstract

The application discloses a kind of based on depth image priori image loss compression method, method includes: constructing a convolutional neural network, the network structure of the convolutional neural network is used as the prior information of image, using the network parameter of the convolutional neural network as the low-dimensional representation of image;Iterative calculation is carried out by gradient descent optimizer, the fitting processing of image is completed, and the optimal network parameter set is obtained;According to the optimal network parameter set as the data after image compression, then realize by input tensor to the convolutional neural network, output corresponding image, realize the loss compression of image.The embodiment of the application can maintain the generality under the premise, the recovered image can still maintain higher image quality at high compression rate, and can be widely applied in image processing technical field.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a lossy image compression method based on depth image priors. Background Technology

[0002] Among lossy image compression techniques, the JPEG series of algorithms is the most widely used. The JPEG algorithm primarily employs a joint coding method combining predictive coding (DPCM), discrete cosine transform (DCT), and entropy coding to remove redundant image and color data. This allows for image compression within a small storage space, achieving relatively good image quality with minimal disk space. However, at extremely high compression ratios (extremely low bpp), significant image information loss occurs, potentially resulting in jagged edges, color blocks, and visible fine squares from the segmentation process, leading to lower image quality after decompression.

[0003] With the development of deep learning technology, deep learning-based image compression algorithms have emerged in academia. These algorithms typically employ a convolutional neural network with an encoder-decoder structure. The first part of the network acts as the encoder, taking an image as input and outputting a set of low-dimensional encoded data to compress the image. The second part acts as the decoder, taking the encoded data as input and reconstructing the image to decompress it. A well-designed network requires extensive pre-training with a large amount of image data. After training, the network can maintain high image quality even at extremely high compression rates. However, the training data is insufficient to cover all image types, and compression performance may be poor for image types not included in the training data. Furthermore, both the compression and decompression ends need the same trained network parameters to properly reconstruct the compressed data, making widespread public distribution of the compressed data difficult. Therefore, deep learning-based image compression algorithms have relatively low versatility.

[0004] JPEG algorithms are generally used, but at extremely high compression ratios (extremely low bpp), there is a significant loss of image information, resulting in low quality of the image recovered after decompression.

[0005] Deep learning-based algorithms can maintain high image quality even at extremely high compression rates, but the compression effect may be poor for image types not included in the training data. In addition, the need to distribute network parameters in advance makes it difficult to publicly distribute the compressed data. Both of these factors result in the low versatility of such algorithms. Summary of the Invention

[0006] The main objective of this invention is to propose a lossy image compression method based on depth image priors, which can maintain high image quality even at high compression rates while maintaining universality.

[0007] To achieve the above objectives, one aspect of this invention proposes a lossy image compression method based on depth image priors, comprising the following steps:

[0008] Construct a convolutional neural network, use the network structure of the convolutional neural network as prior information of the image, and use the network parameters of the convolutional neural network as a low-dimensional representation of the image;

[0009] The gradient descent optimizer performs iterative calculations to fit the image and obtain the optimal set of network parameters.

[0010] By using the optimal set of network parameters as the compressed image data, lossy compression of the image is achieved by inputting a tensor into the convolutional neural network and outputting the corresponding image.

[0011] In some embodiments, the expression for the process of outputting a corresponding image after inputting a tensor into the convolutional neural network is:

[0012]

[0013] The tensor input to the convolutional neural network is: The output image is H0×W0 and H×W represent the input and output space sizes, respectively, and k0 and k... out Let f represent the number of input and output channels, respectively, and λ represent all network parameters. If the tensor Z0 is fixed, then f can be considered a function of λ. The network structure itself serves as prior information for the image, and the network parameters λ serve as parameters for the image, thereby enabling different images to be displayed. It is represented by network parameters θ; when the number of parameters in θ is smaller than that in the image When the spatial domain parameters are large, lossy compression of the image is achieved through this representation.

[0014] In some embodiments, the network structure of the convolutional neural network consists of a set of progressively upsampling structures, and the reconstructed image is finally output by the output layer.

[0015] The upsampling layer consists of a 1×1 convolution, an upsampling operator, a linear rectified activation function, and channel normalization.

[0016] The upsampling operator is implemented using bilinear interpolation or bicubic interpolation. The output tensor of the i-th upsampling layer is represented as:

[0017] Zi =cn(relu(U i Z i-1 θ i ))

[0018] Where cn(·) represents channel normalization, U i Indicates the upsampling operator. Z represents the weights of the 1×1 convolution kernel in the i-th layer. i-1 Represents the input tensor of the i-th layer;

[0019] The output layer consists of a 1×1 convolutional kernel and a sigmoid activation function, and its output is represented as follows:

[0020]

[0021] Where d indicates that the network uses a d-layer upsampling structure, This represents the weights of the 1×1 convolutional kernel in the output layer;

[0022] When the number of channels k of the output tensor of each upsampling layer i When k = k, the total number of network parameters is determined by the following formula:

[0023] N = dk 2 +2dk+kk out

[0024] Where 2dk is the sum of the parameter values ​​of the two parameters in all channel normalization operations, and k out k is the number of channels in the output image; when the image is grayscale, k is the number of channels. out =1; k = 1 when the image is a color image out =3.

[0025] In some embodiments, for small-sized images, the compression and restoration process of the convolutional neural network includes the following steps:

[0026] In the initial state, the convolutional neural network does not need to be trained, and its network parameters are randomly initialized.

[0027] In image compression, the optimal network parameters that can approximately represent the original image I are obtained by solving an optimization model. The expression for the optimization model is: in,‖·‖ p It represents the L1 norm or L2 norm; f(Z0; θ) represents the expression of the convolutional neural network;

[0028] The network parameters are iteratively updated using the gradient descent optimizer during neural network training until the optimal network parameters are obtained.

[0029] Save optimal network parameters Achieve lossy compression of small-sized images;

[0030] During image restoration, the optimal network parameters saved during compression are read. And perform one forward propagation computation of the network. The restored image is obtained.

[0031] In some embodiments, for large images, the compression and restoration process of the convolutional neural network includes the following steps:

[0032] The original large image is divided into blocks, and then each block is processed separately.

[0033] When compressing an image, the block size is defined as a square M×M;

[0034] Record the original image size H×W; perform mirror filling on the right and bottom edges of the original image to fill the image to size H′×W′, such that H′ and W′ are both integer multiples of M;

[0035] The image is divided into several blocks, forming a block set B;

[0036] Perform a small-size image compression process on each block to obtain a set Θ of the optimal network parameters for each block;

[0037] Save the collection Θ to achieve lossy compression of large images;

[0038] When restoring the image, each set of parameters in the network parameter set Θ is used for forward propagation to obtain the image block set B;

[0039] Arrange the blocks of image block set B sequentially into an image of size H′×W′; and crop it according to the original image size H×W to obtain the restored image.

[0040] In some embodiments, the method further includes:

[0041] The image compression and restoration stage includes the following steps:

[0042] After completing the image compression calculation process, the network parameters are converted from the original single-precision floating-point type to half-precision floating-point type before being stored.

[0043] Another aspect of this invention provides a lossy image compression system based on depth image priors, comprising:

[0044] The first module is used to construct a convolutional neural network, using the network structure of the convolutional neural network as prior information of the image, and using the network parameters of the convolutional neural network as a low-dimensional representation of the image.

[0045] The second module is used to perform iterative calculations through the gradient descent optimizer to complete the fitting process of the image and obtain the optimal set of network parameters.

[0046] The third module is used to compress the image by using the optimal set of network parameters as the data, and then outputting the corresponding image after inputting a tensor into the convolutional neural network, thereby achieving lossy compression of the image.

[0047] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0048] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0049] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0050] The embodiments of this invention include at least the following beneficial effects: This invention provides a lossy image compression method based on depth image priors. This scheme constructs a convolutional neural network (CNN), using the network structure of the CNN as prior information for the image and the network parameters of the CNN as a low-dimensional representation of the image. Iterative calculations are performed using a gradient descent optimizer to fit the image and obtain the optimal set of network parameters. Based on the optimal set of network parameters, the compressed image data is used to achieve lossy image compression by inputting tensors into the CNN and outputting the corresponding image. The embodiments of this invention can maintain high image quality even at high compression rates while maintaining versatility. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0052] Figure 2This is a flowchart of the overall steps provided in the embodiments of the present invention;

[0053] Figure 3 This is a schematic diagram of the network structure of the convolutional neural network provided in an embodiment of the present invention;

[0054] Figure 4 This is a flowchart of small-size image compression provided in an embodiment of the present invention;

[0055] Figure 5 This is a flowchart of the small-size image restoration process provided in an embodiment of the present invention;

[0056] Figure 6 This is the large-size image compression process provided in the embodiments of the present invention;

[0057] Figure 7 This is a flowchart of the large-size image restoration process provided in an embodiment of the present invention;

[0058] Figure 8 This is an example diagram of the compression result of a small-sized image provided in an embodiment of the present invention;

[0059] Figure 9 This is an example diagram of the large-size image segmentation process provided in an embodiment of the present invention;

[0060] Figure 10 This is an example diagram of the compression result of a large-size image provided in an embodiment of the present invention;

[0061] Figure 11 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0063] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0064] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0066] Before providing a detailed description of the embodiments of the present invention, some related technologies involved in the embodiments of the present invention will be described first, as follows:

[0067] Grayscale image: An image in which each pixel has only one sampled color, typically displayed as grayscale ranging from the darkest black to the brightest white. The most common grayscale images use 8-bit unsigned integers to represent the grayscale value of each pixel, in which case the grayscale value of each pixel ranges from [0, 255].

[0068] Color images: Each color image consists of three color channels: red (R), green (G), and blue (B). Similar to grayscale images, each color channel uses a grayscale value to represent the brightness of its respective color. In common color images, each channel uses an 8-bit unsigned integer to represent the grayscale value, with a value range of [0, 255]. Each pixel occupies a total of 24 bits of binary data.

[0069] Floating-point numbers: Computers generally use floating-point format to store and process decimals, representing them in scientific notation. In the IEEE 754-2019 standard, a floating-point number consists of three parts: a sign bit, an exponent bit, and a mantissa bit. Depending on the binary length used, common floating-point numbers include double-precision floating-point numbers (FP64), single-precision floating-point numbers (FP32), and half-precision floating-point numbers (FP16), occupying 64, 32, and 16 bits of binary length respectively. The number of bits occupied by the three parts are 1:11:52, 1:8:23, and 1:5:10 respectively.

[0070] Bits per pixel (bpp) is a unit describing the number of binary bits used to represent color information for each pixel in a digital image. For uncompressed images, a 24-bit RGB color image occupies 24 bpp, and an 8-bit grayscale image occupies 8 bpp. For compressed images, the bpp value is the average number of bits required per pixel after compression.

[0071] Lossless compression: Compression is achieved by utilizing the statistical redundancy of data, which can completely restore the original data without causing any distortion.

[0072] Lossy compression: It takes advantage of the fact that humans are not sensitive to certain frequency components in images, allowing some information to be lost during the compression process. Although the original data cannot be completely recovered, the lost part has little impact on understanding the original image.

[0073] The image lossy compression method based on depth image prior provided in this invention relates to the field of image processing technology. This method can be applied to terminals, servers, or software running on either a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the image lossy compression method based on depth image prior, but is not limited to the above forms.

[0074] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0075] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0076] Server 101 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0077] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0078] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0079] Exemplary based on Figure 1The implementation environment shown in this embodiment of the invention provides a lossy image compression method based on depth image prior. The following description uses the application of this lossy image compression method based on depth image prior in server 101 as an example. It can be understood that this method can also be applied in terminal 102.

[0080] Reference Figure 2 , Figure 2 The flowchart illustrates a depth-based image prior-based lossy compression method for servers, provided in this embodiment of the invention. The method can be executed by any of the aforementioned computer devices (including servers or terminals). See also... Figure 2 The method may include the following steps:

[0081] Construct a convolutional neural network, use the network structure of the convolutional neural network as prior information of the image, and use the network parameters of the convolutional neural network as a low-dimensional representation of the image;

[0082] The gradient descent optimizer performs iterative calculations to fit the image and obtain the optimal set of network parameters.

[0083] By using the optimal set of network parameters as the compressed image data, lossy compression of the image is achieved by inputting a tensor into the convolutional neural network and outputting the corresponding image.

[0084] In some embodiments, the expression for the process of outputting a corresponding image after inputting a tensor into the convolutional neural network is:

[0085]

[0086] The tensor input to the convolutional neural network is: The output image is H0×W0 and H×W represent the input and output space sizes, respectively, and k0 and k... out Let f represent the number of input and output channels, respectively, and θ represent all network parameters. If the tensor Z0 is fixed, then f can be considered a function of θ. The network structure itself serves as prior information for the image, and the network parameters θ serve as parameters for the image, thereby enabling different images to... It is represented by network parameters θ; when the number of parameters in θ is smaller than that in the image When the spatial domain parameters are large, lossy compression of the image is achieved through this representation.

[0087] In some embodiments, the network structure of the convolutional neural network consists of a set of progressively upsampling structures, and the reconstructed image is finally output by the output layer.

[0088] The upsampling layer consists of a 1×1 convolution, an upsampling operator, a linear rectified activation function, and channel normalization.

[0089] The upsampling operator is implemented using bilinear interpolation or bicubic interpolation. The output tensor of the i-th upsampling layer is represented as:

[0090] Z i =cn(relu(U i Z i-1 θ i ))

[0091] Where cn(·) represents channel normalization, U i Indicates the upsampling operator. Z represents the weights of the 1×1 convolution kernel in the i-th layer. i-1 Represents the input tensor of the i-th layer;

[0092] The output layer consists of a 1×1 convolutional kernel and a sigmoid activation function, and its output is represented as follows:

[0093]

[0094] Where d indicates that the network uses a d-layer upsampling structure, This represents the weights of the 1×1 convolutional kernel in the output layer;

[0095] When the number of channels k of the output tensor of each upsampling layer i When k = k, the total number of network parameters is determined by the following formula:

[0096] N = dk 2 +2dk+kk out

[0097] Where 2dk is the sum of the parameter values ​​of the two parameters in all channel normalization operations, and k out k is the number of channels in the output image; when the image is grayscale, k is the number of channels. out =1; k = 1 when the image is a color image out =3.

[0098] In some embodiments, for small-sized images, the compression and restoration process of the convolutional neural network includes the following steps:

[0099] In the initial state, the convolutional neural network does not need to be trained, and its network parameters are randomly initialized.

[0100] In image compression, the optimal network parameters that can approximately represent the original image I are obtained by solving an optimization model. The expression for the optimization model is: in,‖·‖p It represents the L1 norm or L2 norm; f(Z0; θ) represents the expression of the convolutional neural network;

[0101] The network parameters are iteratively updated using the gradient descent optimizer during neural network training until the optimal network parameters are obtained.

[0102] Save optimal network parameters Achieve lossy compression of small-sized images;

[0103] During image restoration, the optimal network parameters saved during compression are read. And perform one forward propagation computation of the network. The restored image is obtained.

[0104] In some embodiments, for large images, the compression and restoration process of the convolutional neural network includes the following steps:

[0105] The original large image is divided into blocks, and then each block is processed separately.

[0106] When compressing an image, the block size is defined as a square M×M;

[0107] Record the original image size H×W; perform mirror filling on the right and bottom edges of the original image to fill the image to size H′×W′, such that H′ and W′ are both integer multiples of M;

[0108] The image is divided into several blocks, forming a block set B;

[0109] Perform a small-size image compression process on each block to obtain a set Θ of the optimal network parameters for each block;

[0110] Save the collection Θ to achieve lossy compression of large images;

[0111] When restoring the image, each set of parameters in the network parameter set Θ is used for forward propagation to obtain the image block set B;

[0112] Arrange the blocks of image block set B sequentially into an image of size H′×E′; and crop it according to the original image size H×W to obtain the restored image.

[0113] In some embodiments, the method further includes:

[0114] The image compression and restoration stage includes the following steps:

[0115] After completing the image compression calculation process, the network parameters are converted from the original single-precision floating-point type to half-precision floating-point type before being stored.

[0116] The specific implementation process of the embodiments of the present invention will be described in detail below with reference to the accompanying drawings and taking a specific application scenario as an example:

[0117] 1. Low-dimensional representation of images:

[0118] Given a convolutional neural network f, input a tensor to the network. Output an image This process can be represented as

[0119]

[0120] Where H0×W0 and H×W represent the input and output space sizes, respectively, and k0 and k out Let Z0 represent the number of input and output channels, respectively, and θ represent all the network parameters. If the tensor Z0 is fixed, f can be viewed as a function of θ. The network structure itself serves as prior information for the image, and the network parameters θ serve as the image parameters, thus enabling different images to... It can be represented by network parameters θ, when the number of parameters of θ is smaller than that of the image. When the spatial domain parameters are defined, lossy compression of the image can be achieved through this representation.

[0121] 2. Network structure of convolutional neural networks:

[0122] This invention uses a low-parameter convolutional network structure, consisting of a series of progressively upsampling structures, with the reconstructed image output by the final output layer. The network structure is as follows: Figure 3 As shown. The upsampling layer consists of a 1×1 convolution, an upsampling operator, a rectified linear unit (ReLU) activation function, and channel normalization (CN). The upsampling operator can be either bilinear interpolation or bicubic interpolation. The output tensor of the i-th upsampling layer can be expressed as...

[0123] Z i =cn(relu(U i Z i-1 θ i ))#(2)

[0124] Where cn(·) represents channel normalization, U i Indicates the upsampling operator. Z represents the weights of the 1×1 convolution kernel in the i-th layer. i-1Let represent the input tensor of the i-th layer. The output layer consists of a 1×1 convolutional kernel and a sigmoid activation function, and its output can be represented as .

[0125]

[0126] Where d indicates that the network uses a d-layer upsampling structure, This represents the weights of the 1×1 convolutional kernel in the output layer.

[0127] When the number of channels k of the output tensor of each upsampling layer i When k = k, the total number of network parameters can be determined by the following formula:

[0128] N = dk 2 +2dk+kk out #(4)

[0129] Where 2dk is the sum of the parameter values ​​of the two parameters in all channel normalization operations, and k out k is the number of channels in the output image; when the image is grayscale, k is the number of channels. out =1, k = 1 when it is a color image out =3.

[0130] 3. Compression and restoration of small-sized images:

[0131] In the initial state, the network does not require prior training, and the network parameters θ are randomly initialized. The goal of this invention in image compression is to make the network output image... To approximate the original image I as closely as possible, i.e., to fit the original image I through the network f(Z0; θ), it is necessary to solve the optimization model to obtain the optimal network parameters that can approximately represent the original image I. The optimization model can be expressed as follows:

[0132]

[0133] in,‖·‖ p The network parameters are either L1 norm (p=1) or L2 norm (p=2). The network parameters are iteratively updated using the gradient descent optimizer, a common feature in neural network training, until the optimal network parameters are obtained. because The number of parameters is less than that of the original image, therefore... Lossy compression of the image is achieved upon saving. During image restoration, only the optimal network parameters saved during compression need to be read. And perform one forward propagation computation of the network. The image can then be output. The complete compression process is as follows: Figure 4 As shown, the recovery process is as follows: Figure 5 As shown.

[0134] 4. Compression and restoration of large-size images:

[0135] In order to compress large images and to unify the calculation process and reuse parameter settings when compressing images of different sizes, the original image needs to be divided into blocks first, and then each block needs to be processed separately.

[0136] When compressing an image, the block size is defined as a square M×M. First, the dimensions of the original image H×W are recorded; then, the right and bottom edges of the original image are mirrored to fill the image to a size H′×W′, ensuring that H′ and W′ are both integer multiples of M; the image is divided into several blocks, forming a block set B; the compression process for small-sized images is performed on each block, resulting in a set Θ of optimal network parameters for each block; finally, Θ is saved. The compression process for large-sized images is as follows... Figure 6 As shown.

[0137] During image restoration, each set of parameters in the network parameter set Θ is used for forward propagation to obtain a set of image blocks B. The blocks in image block set B are then arranged sequentially into an image of size H′×W′. Finally, the image is cropped according to the original image size H×W to obtain the restored image. The restoration process for large images is as follows: Figure 7 As shown.

[0138] 5. Storage of compressed data:

[0139] In the image compression and restoration stages, to maintain the accuracy of forward propagation and gradient calculation, all parameters are calculated using single-precision floating-point type (fp32, each value occupies 32 bits). However, the original image is generally stored using 8-bit unsigned integer type (uint8). If the parameters are stored using fp32 type after compression, the space occupied may be larger than that of the original image, rendering the compression meaningless. To balance accuracy and space usage, after completing the aforementioned image compression calculation process, the network parameters need to be converted from the original single-precision floating-point type to half-precision floating-point type (fp16, each value occupies 16 bits) before storage. For example, for a 256×256 pixel color image block, when k=32 and d=5 are selected, the space occupied for storing the network parameters is (5×32... 2 (2×5×32+3×32)×16=88576 bits, corresponding to a bpp of 85576÷(256×256)≈1.352. It should be noted that because large images require padding at the edges to an integer multiple of the image block size, and this padding is invalid data, the actual effective data bpp for large images may be slightly larger.

[0140] The technical effects of the method in this embodiment of the invention will be explained below using experimental data as an example:

[0141] In this embodiment of the invention, all results are set with the following parameters: the upsampling operator uses bicubic interpolation, the input tensor Z0 is randomly generated and fixed, follows a uniform distribution U[-0.2,0.2], the optimization model uses a smooth L1 norm, the gradient optimizer uses the Adam optimizer, the initial learning rate is 1e-2 and gradually decreases to 1e-4, and the number of iterations is fixed at 20,000.

[0142] 1. Small-sized images:

[0143] This example demonstrates the compression result of a 256×256 pixel color image. The algorithm was tested under k=24, k=32, and k=64, with corresponding bpp values ​​of 0.779, 1.352, and 5.203, respectively. The original and compressed image results are shown below. Figure 8 As shown.

[0144] 2. Large-size images:

[0145] This example demonstrates the compression result of a 1020×678 pixel image, with a block size of 256×256. The algorithm is tested under k=24, k=32, and k=64 conditions, yielding effective bitpage ratios of 0.886, 1.537, and 5.917 respectively. The image block division operation is as follows: Figure 9 As shown, the original image and the compressed image results are as follows: Figure 10 As shown.

[0146] In summary, this invention proposes an image compression algorithm based on deep image priors. It utilizes some deep learning techniques by constructing a low-parameter neural network. Without prior training, the network structure serves as prior information about the image, and the network parameters are used as a low-dimensional representation of the image. After iterative calculation and fitting of the image using a gradient descent optimizer, the optimal network parameters are used as the compressed image data. This invention maintains high image quality even at high compression rates while maintaining versatility.

[0147] Another aspect of this invention provides a lossy image compression system based on depth image priors, comprising:

[0148] The first module is used to construct a convolutional neural network, using the network structure of the convolutional neural network as prior information of the image, and using the network parameters of the convolutional neural network as a low-dimensional representation of the image.

[0149] The second module is used to perform iterative calculations through the gradient descent optimizer to complete the fitting process of the image and obtain the optimal set of network parameters.

[0150] The third module is used to compress the image by using the optimal set of network parameters as the data, and then outputting the corresponding image after inputting a tensor into the convolutional neural network, thereby achieving lossy compression of the image.

[0151] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0152] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned lossy image compression method based on depth image priors. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0153] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0154] Please see Figure 11 , Figure 11 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0155] The processor 1101 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0156] The memory 1102 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1102 and is called and executed by the processor 1101 to execute the image lossy compression method based on depth image prior of the embodiments of this invention.

[0157] Input / output interface 1103 is used to implement information input and output;

[0158] The communication interface 1104 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0159] Bus 1105 transmits information between various components of the device (e.g., processor 1101, memory 1102, input / output interface 1103, and communication interface 1104);

[0160] The processor 1101, memory 1102, input / output interface 1103 and communication interface 1104 are connected to each other within the device via bus 1105.

[0161] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned lossy image compression method based on depth image priors.

[0162] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0163] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0164] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0165] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0166] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0168] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0169] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0170] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0171] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0172] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0174] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A method of image lossy compression based on depth image priors, characterized in that, The method comprises the following steps: constructing a convolutional neural network, taking the network structure of the convolutional neural network as prior information of an image, and taking the network parameters of the convolutional neural network as a low-dimensional representation of the image; performing iterative calculation through a gradient descent optimizer to complete fitting processing of the image and obtain an optimal network parameter set; taking the optimal network parameter set as compressed data of the image, and then outputting a corresponding image after inputting a tensor into the convolutional neural network, thereby realizing lossy compression of the image; the network structure of the convolutional neural network is composed of a group of layer-by-layer progressive up-sampling structures, and a reconstructed image is output by an output layer; the up-sampling layer is composed of a 1*1 convolution, an up-sampling operator, a linear rectifier activation function and channel normalization; The up-sampling operator is implemented using bilinear or bicubic interpolation, the first The output tensor of the layer up-sampling layer is represented as: wherein, denotes channel normalization, denotes an up-sampling operator, denotes the weights of the 1 x 1 convolutional kernel of the layer, denotes the input tensor of the layer; the output layer is composed of a 1*1 convolution kernel and a Sigmoid activation function, and the output is represented as: wherein, represents that the network uses a layer upsampling structure, represents the weight of the 1x1 convolution kernel of the output layer; When the number of channels of the output tensor of each layer upsampling layer is The total parameter amount of the network is determined by the following formula: wherein, is the sum of the parameter quantities of the two parameters in the normalization operation for all channels, is the number of channels of the output image, when the image is a grayscale image ; when the image is a color image .

2. The method of claim 1, wherein, the expression of the process of outputting a corresponding image after inputting a tensor into the convolutional neural network is: where the input tensor of the convolutional neural network is , the output image is , and represent the spatial size of the input and output respectively, and represent the channel number of the input and output respectively, represents all the parameters of the network; if the tensor is fixed, then is regarded as a function of , taking the network structure itself as the prior information of the image, and the network parameters as the parameters of the image, so that different images are represented by the network parameters ; when the parameter amount of is less than the spatial domain parameter amount of the image , then the lossy compression of the image is realized through this representation.

3. The image lossy compression method based on deep image prior according to claim 1, characterized in that for small-size images, the compression and recovery process of the convolutional neural network comprises the following steps: in the initial state, the convolutional neural network does not need to be trained, and the network parameters of the convolutional neural network are randomly initialized; In image compression, an optimization model is solved to obtain an approximate representation of the original image. Optimal network parameters The expression for the optimization model is: ,in, It is either the L1 norm or the L2 norm; An expression representing a convolutional neural network; iteratively updating the network parameters by using a gradient descent optimizer in the neural network training until optimal network parameters are obtained ; Saving optimal network parameters Implementing lossy compression of small-sized images; In the image restoration, the optimal network parameters saved in compression are read And perform a forward propagation calculation of the network , Get the restored image.

4. The image lossy compression method based on deep image prior according to claim 1, characterized in that for large-size images, the compression and recovery process of the convolutional neural network comprises the following steps: the original large-size image is divided into blocks, and each block is processed separately; In compressing an image, the block size is defined as a square ; The size of the original image is recorded ; the image is padded to size , with mirror padding on the right and bottom of the original image , such that both are integer multiples of Divide the image into a plurality of blocks to form a block set ; The compression processing flow of the small size image is performed on all blocks respectively, to obtain a set of optimal network parameters of each block ; saving a collection enabling lossy compression of large size images; In restoring the image, forward propagation is performed using each group of parameters in the network parameter set to obtain a set of image blocks , respectively. arranging the set of image blocks in sequence of blocks of the original image; and cropping the restored image according to the original image size ​​ 5. The method of claim 1, wherein, the method further comprises: in the image compression and recovery stage, the following steps are included: after completing the image compression calculation process, the network parameters are converted from single-precision floating-point type to half-precision floating-point type and then stored.

6. A system for lossy compression of images based on depth image priors, characterized in that, comprise: a first module for constructing a convolutional neural network, taking the network structure of the convolutional neural network as prior information of an image, and taking the network parameters of the convolutional neural network as a low-dimensional representation of the image; a second module for performing iterative calculation through a gradient descent optimizer to complete fitting processing of the image and obtain an optimal network parameter set; a third module for taking the optimal network parameter set as compressed data of the image, and then outputting a corresponding image after inputting a tensor into the convolutional neural network, thereby realizing lossy compression of the image; the network structure of the convolutional neural network is composed of a group of layer-by-layer progressive up-sampling structures, and a reconstructed image is output by an output layer; the up-sampling layer is composed of a 1*1 convolution, an up-sampling operator, a linear rectifier activation function and channel normalization; The up-sampling operator is implemented using bilinear or bicubic interpolation, the first The output tensor of the layer up-sampling layer is represented as: wherein, denotes a channel normalization, denotes an up-sampling operator, denotes the weights of the 1 x 1 convolutional kernel of the layer, denotes the input tensor of the layer; the output layer is composed of a 1*1 convolution kernel and a Sigmoid activation function, and the output is represented as: wherein, represents that the network uses a layer up-sampling structure, represents the weight of the 1x1 convolution kernel of the output layer; When the number of channels of the output tensor of each layer upsampling layer is The total parameter amount of the network is determined by the following formula: wherein, is the sum of the parameter quantities of the two parameters in the normalization operation for all channels, is the number of channels of the output image, when the image is a grayscale image ; when the image is a color image .

7. An electronic device, comprising: comprise a processor and a memory; the memory is used to store a program; the processor executes the program to realize the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by a processor to realize the method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program, which when executed by the processor, implements the method as claimed in any one of claims 1 to 5.

Images