Image encoding and decoding method and device for eliminating spatial redundancy
By setting up an adjacent block attention model in the image encoding and decoding model, the spatial redundancy of the image is captured and eliminated, the problem of insufficient image compression performance in the prior art is solved, and a more efficient image compression effect is achieved.
Patent Information
- Application Number
- CN202210521907.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-13
- Publication Date
- 2025-03-07
- Estimated Expiration
- 2042-05-13
AI Technical Summary
Existing deep learning-based image codec models are not effective in eliminating spatial redundancy, resulting in image compression performance not yet reaching the optimal level.
The adjacent block attention model is respectively set up in the encoding and transformation network of the image encoding and decoding transformation network of the image encoding and decoding transformation model, through which the spatial redundant information is captured from the image features, and the image features that eliminate spatial redundant information are output.
By introducing information from adjacent locations, expanding the receptive field can better capture and eliminate the redundancy of the image in space, thereby improving the image compression effect.
Smart Images
Figure CN115086680B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image coding and decoding, and in particular to an image coding and decoding method and device for eliminating spatial redundancy. Background Art
[0002] At present, the image coding and decoding model based on deep learning in academia has surpassed the effects of traditional H266 encoding, H265 encoding, WEBP encoding, and JPEG encoding in terms of compression performance based on PSNR (peak signal-to-noise ratio), MS-SSIM (multi-level structural similarity) and subjective human eye. The end-to-end image coding and decoding model based on deep learning has great potential.
[0003] Currently, two main attention models are used in deep learning-based image encoding and decoding models. The first attention model is a local model, which only performs 3*3 or 5*5 convolution on its own position. However, when the convolution kernel continues to increase, such as 7*7, the features will tend to be averaged, which is not conducive to the retention of spatial information because the convolution itself is a spatial multiplication and accumulation method. The second attention model is a non-local model, but it needs to traverse the entire image information during calculation, the improvement effect is limited, and there is a large degree of computing power waste; Overall, the two existing attention models are not effective in eliminating spatial redundancy. Therefore, the compression performance of the existing image encoding and decoding models needs to be improved. Summary of the invention
[0004] The purpose of the present invention is to provide an image encoding and decoding method and device for eliminating spatial redundancy, which can effectively improve image compression performance.
[0005] The objective of the present invention is achieved through the following technical solutions:
[0006] An image encoding and decoding method for eliminating spatial redundancy, comprising:
[0007] The adjacent block attention models are respectively set in the encoding transformation network and the decoding transformation network in the image encoding and decoding model;
[0008] When the input original image enters the encoding transformation network, the image features are extracted through the convolution layer, and then the adjacent block attention model inside the encoding transformation network captures the spatial redundant information from the image features, and outputs the image features that eliminate the spatial redundant information;
[0009] When the image features with spatial redundant information eliminated enter the decoding transformation network, they are processed by the residual block, and the image features output by the residual block are processed by the adjacent block attention model inside the decoding transformation network, and then processed by the deconvolution layer to output the reconstructed image.
[0010] An image coding and decoding device for eliminating spatial redundancy, wherein an image coding and decoding model is provided inside the device. The image coding and decoding model sets an adjacent block attention model based on the aforementioned method and realizes coding and decoding of an input original image.
[0011] A processing device, comprising: one or more processors; a memory for storing one or more programs;
[0012] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.
[0013] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.
[0014] It can be seen from the technical solution provided by the present invention that by setting an adjacent block attention model, information of adjacent (up, down, left, and right) positions can be introduced when processing image features, thereby expanding the receptive field and better capturing and eliminating the redundancy of the image in space, thereby improving the image compression effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0016] Figure 1 A flowchart of an image encoding and decoding method for eliminating spatial redundancy provided by an embodiment of the present invention;
[0017] Figure 2 A schematic diagram of a neighboring block attention model provided by an embodiment of the present invention;
[0018] Figure 3 A schematic diagram of an NP-UNIT provided in an embodiment of the present invention;
[0019] Figure 4 A schematic diagram of a residual convolution module provided by an embodiment of the present invention;
[0020] Figure 5 A schematic diagram of the overall framework of an image encoding and decoding model with a neighboring block attention model provided in an embodiment of the present invention;
[0021] Figure 6 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0022] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.
[0023] First, the terms that may be used in this article are explained as follows:
[0024] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.
[0025] The following is a detailed description of an image encoding and decoding method and device for eliminating spatial redundancy provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals in the field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer shall be followed.
[0026] Embodiment 1
[0027] An embodiment of the present invention provides an image encoding and decoding method for eliminating spatial redundancy. In this method, a new attention module, called Neighbour-Patch-Attention (NPA), is proposed on the current end-to-end image compression algorithm framework. It can better eliminate spatial redundancy and improve image compression effect. Figure 1 The main process of the present invention is shown, comprising the following steps:
[0028] Step 1: Set adjacent block attention models in the encoding transformation network and decoding transformation network in the image encoding and decoding model respectively.
[0029] Step 2: When the original image is input into the encoding transformation network, the image features are extracted through the convolution layer, and then the adjacent block attention model inside the encoding transformation network captures the spatial redundant information from the image features, and outputs the image features that eliminate the spatial redundant information.
[0030] Step 3: When the image features that eliminate spatial redundant information enter the decoding transformation network, they are processed by the residual block, and the image features output by the residual block are processed by the adjacent block attention model inside the decoding transformation network to improve the receptive field range, and then processed by the deconvolution layer to output the reconstructed image.
[0031] The neighboring block attention model provided by the embodiment of the present invention further reduces spatial redundancy by introducing adjacent position information, and is a spatial attention model that is very suitable for a deep learning end-to-end image compression framework. Figure 2 The main structure of the neighboring block attention model is shown, which includes: two branches, the first branch is provided with NP-UNIT (Neighbour-Patch-Unit), multiple residual convolution modules (RES131) and a single convolution layer (Conv) in sequence, and the second branch is provided with multiple residual convolution modules in sequence. The main processing flow includes: the input image features enter the two branches for processing respectively, in the first branch, the input image features are processed by NP-UNIT and multiple residual convolution modules and a convolution layer with a sigmoid activation function, and the point-by-point scale is output. In the second branch, the multi-input image features are processed by multiple residual convolution modules. The outputs of the two branches are multiplied point by point to obtain the attention residual of the neighboring block attention model; the attention residual is added point by point to the input image features, and the image features with spatial redundant information eliminated are output.
[0032] like Figure 3 As shown in FIG. 1 , the main structure of NP-UNIT is shown. Convolutional layers are respectively arranged at the entrance and the exit of NP-UNIT. A concatenation layer is arranged in the middle part between the entrance and the exit, and the concatenation layer is connected to the input of the convolutional layer at the entrance. The processing flow is as follows: multiple convolutional layers (for example, Figure 3 The four convolutional layers shown in FIG. 1 and FIG. 2 perform convolution operations on the input centered at four adjacent positions in the image features of the C channel in turn, such as Figure 3As shown in the right part, c represents the center position, N1 to N4 represent four adjacent positions up, down, left, and right. The output of the convolution layer at the entrance (the number of output channels is 1 / 4 of the number of input channels) is concatenated with the input image features in the channel dimension through the concatenation layer. The concatenated features are reduced to C in the channel dimension through the convolution layer at the exit; where C is a positive integer. The above processing principle is: convolution processing is performed on the four positions N1 to N4 through multiple convolution layers set at the entrance, and the center position c is not processed. After that, the convolution processing results of the positions N1 to N4 are concatenated with the original input. The original input here is equivalent to the convolution processing of c. Through NP-UNIT, the model can introduce information about adjacent (up, down, left, and right) positions, and the relative position relationship is also implicitly introduced in concatenate (channel concatenation). In general, the receptive field is expanded, and the redundancy of the image in space can be better captured and eliminated.
[0033] like Figure 4 As shown in the figure, the main structure of the residual convolution module includes: a plurality of convolution layers arranged in sequence; wherein the original input of the residual convolution module and the output of the last convolution layer are added point by point as the output of the residual convolution module.
[0034] It should be noted that Figure 2 The number of residual convolution modules shown, Figure 3 and Figure 4 The number of convolutional layers shown in the figure is for example only and does not constitute a limitation. In actual applications, users can set the number of residual convolution modules and each convolutional layer according to their needs or experience. Figure 2 to Figure 4 Conv1*1 (or 3*3) in each convolutional layer indicates that the size of the convolution kernel is 1*1 (or 3*3), the C (or C / 2, C / 4) in the middle indicates the channel dimension of the convolution operation output, and the leaky-relu at the end indicates the activation function.
[0035] In an embodiment of the present invention, the encoding transformation network includes: multiple convolutional layers and multiple adjacent block attention models; wherein, a convolutional layer is arranged at the entrance of the encoding transformation network, and an adjacent block attention model is arranged at the exit; and multiple convolutional layers and at least one adjacent block attention model are arranged in the middle part between the entrance and the exit.
[0036] In an embodiment of the present invention, the decoding transformation network includes: multiple residual blocks, multiple adjacent block attention models and multiple deconvolution layers; wherein, multiple residual blocks and adjacent block attention models are sequentially arranged at the entrance of the decoding transformation network, and a deconvolution layer is arranged at the exit; multiple deconvolution layers and at least one adjacent block attention model are arranged in the middle part between the entrance and the exit.
[0037] Figure 5The overall framework of the image encoding and decoding model with the adjacent block attention model is shown, wherein the encoding transformation network provides an example including four convolutional layers and two adjacent block attention models, and the decoding transformation network provides an example including three residual blocks (Resblock), two adjacent block attention models and four deconvolution layers (Deconv); similarly, 5*5 represents the size of the convolution kernel (deconvolution kernel), and 3*3 represents the residual block size; 192, 3 represents the channel dimension of the output, S2 represents the step size of 2, and GDN is the activation function. The dotted box on the right is the hyper-priori model (hyper-prioi), which includes a context prediction model (not shown), a hyper-priori encoder (hyper analysis), a hyper-priori decoder (hypersynthesis), and an entropy model (including AE and AD), AE and AD represent the entropy encoder and entropy decoder (performing lossless entropy encoding and lossless entropy decoding), LMM represents the Laplace mixture model, ROUND represents the quantization operation, and bit stream represents the bit rate estimation result; considering that the hyper-priori model can adopt the existing model structure, its workflow can also refer to conventional technology, so it will not be repeated. The following is combined with Figure 5 Introduce the overall processing flow, including:
[0038] Step S1: Input the original image (dimensions: H, W, 3), where H and W represent the height and width of the image respectively, and 3 represents the number of image channels (for example, RGB channels). The image features that eliminate spatial redundant information are output through the encoding transformation network (dimensions: H / 16, W / 16, C, where C is the number of channels of the image features).
[0039] Step S2: The image features with spatial redundant information eliminated enter the super-prior probability model and are divided into two paths. The two paths quantize the image features respectively, estimate the probability of the quantized features, and estimate the entropy respectively (arithmetic coding is used to obtain the actual bit stream in actual encoding and decoding).
[0040] Step S3: The entropy-decoded features output by the super-prior probability model enter the decoding transformation network, and the decoding transformation network generates a decoded reconstructed image (dimension: H, W, 3).
[0041] It should be noted that the processing of the adjacent block attention model involved in the encoding transformation network and the decoding transformation network has been described in detail before, so it will not be repeated here; at the same time, the specific processing flow involved after the image features enter the super prior probability model can also refer to conventional technology, so it will not be repeated here. In addition, since the super prior probability model performs lossless entropy encoding and lossless entropy decoding, the features after entropy decoding are equivalent to the image features output by the encoding transformation network that eliminate spatial redundant information.
[0042] See also Figure 5 ,In the training phase, the rate-distortion loss function is constructed according to the information obtained in the above processing flow, and the model parameters are updated through back propagation through the rate-distortion loss RD-LOSS.
[0043] The rate-distortion loss function is expressed as:
[0044] RD-LOSS=λ*D-LOSS+R-LOSS
[0045] Among them, D-LOSS represents distortion loss (Mse / Mae / Lpips / GAN, etc.), R-LOSS represents rate loss, and λ is a hyperparameter that controls distortion loss and rate loss.
[0046] The distortion loss D-LOSS can be calculated by calculating the difference between the original image and the reconstructed image output by the decoding transformation network, expressed as:
[0047]
[0048] Among them, E represents expectation, x i and Represent the pixel value of the i-th pixel in the original image and the reconstructed image respectively.
[0049] The rate loss R-LOSS is calculated by the two-way bit rate estimation results of the super prior probability model, which is expressed as:
[0050] R-LOSS=-E(log2p(Q(y i )))-E(log2p(Q(z j )))
[0051] Among them, y represents the output of the coding transformation network, z represents the output of the hyper-prior encoder (hyper analysis), Q represents the quantization function, p represents the probability function, and the subscripts i and j represent the information sequence numbers in the bit rate estimation result.
[0052] Embodiment 2
[0053] The present invention also provides an image coding and decoding device for eliminating spatial redundancy, wherein an image coding and decoding model is provided inside the device, wherein the image coding and decoding model sets an adjacent block attention model based on the method provided in the aforementioned embodiment 1, and implements coding and decoding of an input original image through the method provided in the aforementioned embodiment 1. Figure 5 Relevant examples are provided.
[0054] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0055] Embodiment 3
[0056] The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided by the aforementioned embodiments.
[0057] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.
[0058] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:
[0059] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;
[0060] The output device may be a display terminal;
[0061] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.
[0062] Embodiment 4
[0063] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.
[0064] In the embodiment of the present invention, the readable storage medium is a computer-readable storage medium and can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.
[0065] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. An image encoding and decoding method for eliminating spatial redundancy, characterized in that: include: The adjacent block attention models are respectively set in the encoding transformation network and the decoding transformation network in the image encoding and decoding model; When the input original image enters the encoding transformation network, the image features are extracted through the convolution layer, and then the adjacent block attention model inside the encoding transformation network captures the spatial redundant information from the image features, and outputs the image features that eliminate the spatial redundant information; When the image features with spatial redundant information eliminated enter the decoding transformation network, they are processed by the residual block, and the image features output by the residual block are processed by the adjacent block attention model inside the decoding transformation network, and then processed by the deconvolution layer to output the reconstructed image; The adjacent block attention model includes: two branches, the first branch is provided with NP-UNIT, multiple residual convolution modules and a single convolution layer in sequence, and the second branch is provided with multiple residual convolution modules in sequence; NP-UNIT represents an adjacent block processing unit; the input image features enter the two branches for processing respectively, and the outputs of the two branches are multiplied point by point to obtain the attention residual of the adjacent block attention model; the attention residual is added point by point to the input image features, and the image features that eliminate spatial redundant information are output; A convolution layer is respectively arranged at the entrance and the exit of the NP-UNIT, a splicing layer is arranged in the middle part between the entrance and the exit, and the splicing layer is connected to the input of the convolution layer at the entrance; a plurality of convolution layers are arranged at the entrance of the NP-UNIT, and convolution operations are performed on the input centered at four adjacent upper, lower, left and right positions in the image features of the C channel in turn, and the output of the convolution layer at the entrance is spliced with the input image features in the channel dimension through the splicing layer, and the splicing features are reduced to C in the channel dimension through the convolution layer at the exit; wherein C is a positive integer.
2. The image encoding and decoding method for eliminating spatial redundancy according to claim 1, characterized in that: The encoding transformation network includes: multiple convolutional layers and multiple adjacent block attention models; wherein, a convolutional layer is arranged at the entrance of the encoding transformation network, and an adjacent block attention model is arranged at the exit; and multiple convolutional layers and at least one adjacent block attention model are arranged in the middle part between the entrance and the exit.
3. The image encoding and decoding method for eliminating spatial redundancy according to claim 1, characterized in that: The decoding transformation network includes: multiple residual blocks, multiple adjacent block attention models and multiple deconvolution layers; wherein, multiple residual blocks and adjacent block attention models are sequentially arranged at the entrance of the decoding transformation network, and a deconvolution layer is arranged at the exit; multiple deconvolution layers and at least one adjacent block attention model are arranged in the middle part between the entrance and the exit.
4. The image encoding and decoding method for eliminating spatial redundancy according to claim 1, characterized in that: Each residual convolution module includes: a plurality of convolution layers arranged in sequence; wherein the original input of the residual convolution module and the output of the last convolution layer are added point by point as the output of the residual convolution module.
5. An image encoding and decoding device for eliminating spatial redundancy, characterized in that: An image encoding and decoding model is provided inside the device. The image encoding and decoding model sets an adjacent block attention model based on the method described in any one of claims 1 to 4, and realizes encoding and decoding of the input original image.
6. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.
7. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.