Screen content image compression method and device based on content adaptation

Abstract

The present invention discloses a screen content image compression method and device based on content adaptation, including a main encoder, a super prior encoder, a super prior decoder, an entropy encoder, and a main decoder. The method and device are used to extract the original image structure information, and then perform affine transformation on the intermediate image features in the main encoder, the super prior encoder, and the main decoder through feature transformation, thereby introducing structural prior to the image features to achieve the purpose of structural enhancement. At the same time, the text position information is obtained through text detection, and the original gradient is adjusted using the text area mask to enhance the text quality in the image compression, thereby adjusting the external prior and the loss function. Compared with the prior art, the present invention fully considers the situation that the screen content image contains a large amount of text, improves the screen content image compression performance, and effectively removes structural redundancy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image coding, and in particular relates to a method and device for compressing screen content images based on content adaptation. Background Art

[0002] Screen content images are an important part of daily communications, mainly including text, graphics, and natural images captured by cameras. Among them, computer-generated content such as text and graphics accounts for more than 90% of screen content.

[0003] Compared with natural images, screen content images have sharp edges, single color tonality, high contrast, and highly differentiated regional complexity. Due to these unique features, the properties of screen content images are very different from those of natural images. Traditional compression algorithms designed for natural images are difficult to effectively apply to screen content images, resulting in the loss of a large amount of high-frequency information during the compression process, resulting in sharp edges and blurred text, which seriously reduces the quality of reconstructed screen content. Traditional algorithms usually only optimize prediction algorithms and rarely involve deep learning compression algorithm optimization. At the same time, they only optimize the overall characteristics of screen content images, lack attention to text areas, and cannot maintain high text quality. Summary of the invention

[0004] In order to solve the shortcomings of the prior art, achieve the purpose of improving the compression performance of screen content images and improving the quality of reconstructed screen content, the present invention adopts the following technical solutions:

[0005] A screen content image compression device based on content adaptation comprises a main encoder, a super a priori encoder, a super a priori decoder and a main decoder connected in sequence. The device also comprises a condition module and a feature transformation module. The original image structure information extracted by the condition module is used to perform structural enhancement adjustment on the image features in the main encoder, the super a priori encoder and the main decoder through the feature transformation module. The latent features output by the main encoder are passed through the super a priori encoder, and the super a priori information output is passed through the super a priori decoder, and then combined with the latent features to obtain quantized latent features. The quantized latent features are combined with the image structure information recovered from the super a priori information, and the restored image is generated through the main decoder.

[0006] Furthermore, the feature transformation module is a nonlinear enhanced spatial feature transformation module, which generates affine transformation parameters based on the output of the conditional module, and performs spatial affine transformation on the image features according to the affine transformation parameters to obtain structurally enhanced image features.

[0007] Furthermore, the device also includes an entropy module, which encodes and decodes the quantized super-a priori information and inputs it into a super-a priori decoder, and then encodes and decodes the quantized latent features based on the output of the super-a priori decoder to obtain the quantized latent features.

[0008] Furthermore, the entropy encoder includes a feature entropy module and a Gaussian entropy module. The super-prior decoder obtains the output of the feature entropy module, generates latent features of the mean and variance of the Gaussian distribution, and combines the latent features output by the main encoder to obtain the probability density of the latent features of the main encoder through the Gaussian entropy module. The latent features output by the main encoder are quantized and encoded and decoded in combination with the probability density to obtain the quantized latent features.

[0009] Furthermore, the device also includes a text detector to locate the text in the original image. The original text detection method will fuse and relocate the overlapping original positioning frames. The present invention converts the obtained positioning information into a text area mask, adjusts the image structure prior based on the text area mask, modifies the original image gradient, and is used for extracting the original image structure information, thereby improving the attention of the text area.

[0010] A method for compressing screen content images based on content adaptation comprises the following steps:

[0011] Step S1: extracting the original image structure information, after feature transformation, to perform structural enhancement and adjustment on the image features in the main coding, main decoding and super a priori coding;

[0012] Step S2: Perform super-prior encoding on the latent features after the main encoding to obtain super-prior information;

[0013] Step S3: after the super-prior information is decoded, it is combined with the latent features to generate quantized latent features;

[0014] Step S4: The quantized latent features are combined with the image structure information recovered from the super prior information, and the recovered image is generated through main decoding.

[0015] Furthermore, in step S3, the quantized super-prior information is quantized and encoded, and then super-prior decoded to generate latent features of the mean and variance of the Gaussian distribution. Combined with the latent features of the main coding output, the probability density of the main coding latent features is obtained through the Gaussian entropy model. The latent features of the main coding output are quantized and encoded and decoded through the probability density to obtain the quantized latent features.

[0016] Furthermore, the feature transformation in step S1 is a nonlinear enhanced spatial feature transformation, and affine transformation parameters are generated based on the extracted original image structure information. The image features are spatially affine transformed according to the affine transformation parameters to obtain structurally enhanced image features:

[0017] SFT(F,Ψ)=γ×F+β

[0018] Among them, F represents the intermediate features of the image, γ and β represent a pair of affine transformation parameters, and Ψ represents the original image structure information as an external prior feature.

[0019] Furthermore, in step S1, the text in the original image is located by text detection. The original text detection method will fuse and relocate the overlapping original positioning frame. The present invention converts the obtained positioning information into a text area mask, adjusts the image structure prior based on the text area mask, modifies the original image gradient, and is used for extracting the original image structure information, thereby improving the attention of the text area:

[0020]

[0021] Among them, M represents the text area mask, g represents the original image gradient, g′ represents the modified image gradient, and ε represents an empirical constant, which can be set according to the desired network attention to the text area.

[0022] Furthermore, spatially adaptive variable bit rate compression is performed by minimizing the bit rate R and distortion D:

[0023] Λ T ·D+R

[0024] Where Λ=(λ i ) i＝1:N represents the vector of Lagrange multipliers of the constant scalar change of the text region mask M, N represents the number of potential features (i.e., the number of image pixels), and each element λ i By the corresponding m i Determined by a monotonically increasing function mapping, with m i The change in pixel x i The weight of the distortion term λ i Corresponding changes, through the constructed loss function, make the image compression based on M explicit spatial bit allocation, the spatial adaptive loss function is as follows:

[0025]

[0026] Among them, x i ′ represents the corresponding pixel in the restored image, represents the quantized potential features, express The probability under condition M.

[0027] The advantages and beneficial effects of the present invention are:

[0028] The content-adaptive screen content image compression method and device of the present invention respectively perform affine transformation on intermediate features in the main encoder, main decoder and super encoder, introduce structural priors into end-to-end image compression, extract features through a conditional network, and guide potential features to perform spatial transformation; at the same time, a text detection method is used to obtain text position information to better process text areas, and the original gradient is adjusted using a text area mask to enhance the text quality in image compression, thereby adjusting the external prior and loss function; the above improvements are acceptable for the increase in coding complexity, and achieve improved compression performance. Compared with the existing end-to-end image compression algorithm, structural redundancy is effectively removed, and higher screen content image compression performance is achieved; compared with the traditional method that does not consider the situation where the screen content image contains a large amount of text, the present invention allocates more bits to the text area and retains more text semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a network overall framework diagram of the device in the embodiment of the present invention.

[0030] Figure 2a is the binary mask map of the existing SFT region of interest.

[0031] Figure 2b is an existing continuous mask map of interest for the task with values ​​in [0,1].

[0032] Figure 2c It is a gradient external prior information map extracted from the image in an embodiment of the present invention.

[0033] Figure 3 It is a schematic diagram of the structure of the nonlinear enhanced spatial feature affine transformation module in an embodiment of the present invention.

[0034] Figure 4a It is a priori graph of structural information before text enhancement in an embodiment of the present invention.

[0035] Figure 4b It is a priori graph of structural information after text enhancement in an embodiment of the present invention.

[0036] Figure 5 is a flow chart of a method in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0038] like Figure 1As shown, a screen content image compression device based on content adaptation adopts a gradient-guided spatial affine transformation algorithm and text area enhancement, including a main encoder GA, a main decoder GS, a super prior encoder HA, a super prior decoder HS, an entropy coding module, etc.

[0039] The Laplace gradient operator is used to obtain the gradient of the original image, and the text area mask of the original image is obtained using the text detection method. The text area mask is weighted on the original image gradient to obtain the structural prior and input into the network.

[0040] In the main encoder, main decoder, and super-prior encoder, the present invention uses the convolution module conv to extract the original potential features of the image, and uses the proposed conditional module CB to extract the structural information in the external prior, and inputs the structural information into the network backbone through the optimized nonlinear enhanced spatial feature transform block NESFT to adjust the original potential features to obtain structurally enhanced potential features. The present invention inputs the image and structural prior information into the main encoder ga to obtain a highly compressed potential feature y, and inputs y into the super-prior encoder ha to obtain a super-prior z. Then z is quantized Q, arithmetic encoded AE, and arithmetic decoded AD to obtain a quantized super-prior at the decoding end. The latent features of the mean and variance of the Gaussian distribution are obtained through the super prior decoder hs, and are input into the Gaussian Entropy Model together with the latent feature y to obtain the probability density of the latent feature y. We quantize the latent feature y and use the obtained probability density for arithmetic encoding and decoding to obtain the quantized latent feature at the decoding end. we will Input to the network composed of transposed convolution block TConv and convolution block Conv to recover the structural information and They are input together into the main decoder gs to restore the image.

[0041] 1. Network structure optimization: The present invention retains the structural information of the screen content image as much as possible, introduces gradient information, uses the conditional network to use external priors to generate appropriate features, and uses the spatial adaptive feature transformation module to adjust the features.

[0042] Compared with natural images, screen content images have sharper edges, large text areas, more monotonous colors, and flatter flat areas. At the same time, there are significant differences between screen content images and natural images in terms of regional complexity and smoothness, including large flat areas and complex areas and the interweaving of the two. In addition, the transition between different areas of the image is drastic and the edges are sharp. However, natural images have smooth edges and rich colors. The partial loss of structural information and edge blur have relatively little effect on the compression and reconstruction of natural images. Therefore, the algorithms designed for natural images do not emphasize the retention of structural information, and the algorithms perform poorly on screen content. In the prior art, Ma et al. proposed a structure-preserving super-resolution method SPSR, which propagates the gradient information of the image as a structural prior to the RGB branch, helping the network to pay more attention to the geometric structure of the image, generating clearer super-resolution images while retaining finer geometric structures. Song et al. proposed deep image compression based on regions of interest, allocating more bits to regions of interest, which shows that features and their bit allocation can be adjusted through external prior information in deep image compression.

[0043] Inspired by this, the present invention uses the gradient information of the image as a structural prior, designs a conditional network that uses an external prior to generate appropriate features Ψ, and uses a spatial adaptive feature transformation module to adjust the features, thereby providing the image with a global complex area focusing capability and improving the compression capability of a large range of simple areas. The spatial affine transformation module generates a pair of parameters (γ, β) based on the output of the conditional network. The features are spatially affine transformed according to this pair of parameters. The feature transformation formula is as follows:

[0044] SFT(F,Ψ)=γ×F+β (1)

[0045] Where F represents the intermediate feature. The present invention does not explicitly transmit the structural prior, so the decoder cannot obtain the original structural prior. The present invention extracts the super prior latent features containing spatial information from the super encoder. The spatial structure features are recovered from the image as the input of the conditional network at the decoding end.

[0046] like Figure 2a to Figure 2c As shown in the figure, the external priors used in the early SFT (Spatial Feature Transformation) in the application of intelligent image compression, the binary mask of the region of interest ( Figure 2a ), the task region of interest mask external prior with a value of [0,1] used by Song et al. ( Figure 2b ), the gradient external prior information extracted from the image using the Laplacian operator adopted by the present invention ( Figure 2c). The complexity of the external prior information used in the present invention, whether in terms of value or spatial structure, far exceeds the previous ROI mask, so it is necessary to improve the SFT and conditional network to adapt to the complex external prior information. The existing SFT and conditional network show good feature extraction capabilities and can handle task-aware ROI masks well, but when faced with more complex external priors, their feature extraction capabilities are still insufficient and cannot well extract spatial transformation coefficients from the external prior for feature transformation.

[0047] The present invention proposes a nonlinear enhanced SFT and adjusts the structure of the conditional network to adapt to more complex external priors. The specific structure is as follows Figure 3 As shown, the external structure prior inputs a conditional block (CB, Condition Block) composed of a 1x1 convolution block Conv1x1, a 3x3 convolution block Conv3x3 and a GELU activation function, outputs a conditional feature Condition, and the conditional feature Condition inputs a nonlinear enhanced spatial feature transform block (NESFT, Nolinear-enhanced SFT) composed of Conv1x1, Conv3x3 and GELU, outputs a pair of affine transformation parameters (γ, β), and performs the transformation shown in the above formula (1) on the input image feature Feature. The present invention enhances the ability of the network to process complex priors by inserting Conv1x1 and RELU activation functions at appropriate positions, eliminates reconstruction artifacts caused by insufficient nonlinear capabilities of the original SFT and conditional networks, and improves the quality of the reconstructed image.

[0048] 2. Text area enhancement: Introduce the text area mask obtained by text detection, and adjust the image structure prior and loss function based on the text position prior.

[0049] Screen content images contain a large number of text regions, while natural images contain little or no text. Therefore, previous compression algorithms for natural images did not take into account the particularity of text content, resulting in the loss of text semantics. However, text usually carries a lot of information, and humans are very sensitive to text information. Therefore, it is necessary to design algorithms specifically for text regions. The structural prior of the image contains rich text prior information, but there are some areas in the screen content image where the sharp edge gradient is greater than the blurred part in the text area, which causes the network to pay more attention to the sharp edges than the text area.

[0050] Therefore, the present invention introduces text detection and adjusts it, inputs the image into the text detector, and obtains the original text positioning frame. The original text detection method will fuse and reposition the overlapping original positioning frames. The present invention does not fuse and reposition the overlapping original positioning frames, but directly converts them into a text area mask M = [m i]i=1:N, to obtain higher text coverage, N represents the number of image pixels, and then adjust the image structure prior based on the text area mask M, the formula is as follows:

[0051]

[0052] Among them, g represents the original image gradient, g′ represents the modified image gradient, and ε represents an empirical constant, which can be set according to the desired network attention to the text area. According to experience, we set it to 2. The structural information before and after adjustment is as follows Figure 4a , Figure 4b As shown, Figure 4b The contrast between the text area and the natural image area is higher than Figure 4a , thereby highlighting the text area and achieving the effect of text area enhancement.

[0053] In order to improve the network's ability to compress text content, the present invention adjusts the network loss function. The goal of lossy image compression is to achieve a trade-off between minimizing the bit rate R and the distortion D. The present invention minimizes Λ T ·D+R to achieve spatial adaptive variable bit rate compression, where Λ=(λ i ) i＝1:N , is the vector of Lagrange multipliers of the constant scalar change of the text region mask M, where N is the number of potential features (i.e., the number of image pixels in the above text), and each element λ i By the corresponding m i Determined by a monotonically increasing function mapping, m i The larger the corresponding pixel x i The weight of the distortion term λ i The larger the loss function, the larger the space bit allocation is. The loss function we designed makes the compression process explicit according to M. The proposed spatial adaptive loss function is as follows:

[0054]

[0055] Among them, x i ′ represents the corresponding pixel in the reconstructed image, represents the quantized potential features, express The probability under condition M.

[0056] like Figure 5 As shown, a method for compressing screen content images based on content adaptation comprises the following steps:

[0057] Step S1: extracting the original image structure information, after feature transformation, to perform structural enhancement and adjustment on the image features in the main coding, main decoding and super a priori coding;

[0058] Step S2: Perform super-prior encoding on the latent features after the main encoding to obtain super-prior information;

[0059] Step S3: after the super-prior information is decoded, it is combined with the latent features to generate quantized latent features;

[0060] Step S4: The quantized latent features are combined with the image structure information recovered from the super prior information, and the recovered image is generated through main decoding.

[0061] The implementation method of this part is similar to that of the above method embodiment and will not be repeated here.

[0062] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A screen content image compression device based on content adaptation, comprising a main encoder, a super a priori encoder, a super a priori decoder and a main decoder connected in sequence, characterized in that: The device also includes a condition module and a feature transformation module, using the original image structure information extracted by the condition module, The image features in the main encoder, the super prior encoder and the main decoder are structurally enhanced and adjusted by the feature transformation module. The latent features output by the main encoder are passed through the super prior encoder, and the super prior information output is then decoded by the super prior. The super prior information is then combined with the latent features to obtain quantized latent features. The quantized latent features are combined with the image structure information recovered from the super prior information, and the restored image is generated by the main decoder. The device also includes a text detector, which locates the text in the original image, converts the obtained location information into a text area mask, adjusts the image structure prior based on the text area mask, modifies the original image gradient, and is used to extract the original image structure information: Where M represents the text region mask, g represents the original image gradient, g′ represents the modified image gradient, and ε represents the empirical constant; Spatially adaptive variable bit rate compression by minimizing bit rate R and distortion D: L T ·D+R Where Λ=(λ i ) i＝1：N represents the vector of Lagrange multipliers that are changed by the text region mask M to a constant scalar, N represents the number of potential features, and increases with m i The change in pixel x i The weight of the distortion term λ i Corresponding changes, through the constructed loss function, make the image compression based on M explicit spatial bit allocation, the spatial adaptive loss function is as follows: Among them, x i ′ represents the corresponding pixel in the restored image, represents the quantized potential features, express The probability under condition M.

2. The content-adaptive screen content image compression device according to claim 1, characterized in that: The feature transformation module is a nonlinear enhanced spatial feature transformation module, which generates affine transformation parameters based on the output of the conditional module, performs spatial affine transformation on the image features according to the affine transformation parameters, and obtains structurally enhanced image features.

3. The content-adaptive screen content image compression device according to claim 1, characterized in that: The device also includes an entropy module, which encodes and decodes the quantized super-a priori information and inputs it into a super-a priori decoder, and then encodes and decodes the quantized latent features based on the output of the super-a priori decoder to obtain the quantized latent features.

4. The content-adaptive screen content image compression device according to claim 3, characterized in that: The entropy encoder includes a feature entropy module and a Gaussian entropy module. The super priori decoder obtains the output of the feature entropy module, generates potential features of the mean and variance of the Gaussian distribution, and combines the potential features output by the main encoder. Through the Gaussian entropy module, the probability density of the potential features of the main encoder is obtained, the potential features output by the main encoder are quantized and encoded and decoded in combination with the probability density to obtain the quantized potential features.

5. A method for compressing screen content images based on content adaptation, characterized in that The steps include: Step S1: extracting the original image structure information, after feature transformation, to perform structural enhancement and adjustment on the image features in the main coding, main decoding and super a priori coding; The text in the original image is located through text detection, and the obtained location information is converted into a text area mask. Based on the text area mask, the image structure prior is adjusted and the original image gradient is modified to extract the original image structure information: Where M represents the text region mask, g represents the original image gradient, g′ represents the modified image gradient, and ε represents the empirical constant; Spatially adaptive variable bit rate compression by minimizing bit rate R and distortion D: L T ·D+R Where Λ=(λ i ) i＝1：N represents the vector of Lagrange multipliers that are changed by the text region mask M to a constant scalar, N represents the number of potential features, and increases with m i The change in pixel x i The weight of the distortion term λ i Corresponding changes, through the constructed loss function, make the image compression based on M explicit spatial bit allocation, the spatial adaptive loss function is as follows: Among them, x i ′ represents the corresponding pixel in the restored image, represents the quantized potential features, express The probability under condition M; Step S2: Perform super-prior encoding on the latent features after the main encoding to obtain super-prior information; Step S3: after the super-prior information is decoded, it is combined with the latent features to generate quantized latent features; Step S4: The quantized latent features are combined with the image structure information recovered from the super prior information, and the recovered image is generated through main decoding.

6. The method for compressing screen content images based on content adaptation according to claim 5, characterized in that: In step S3, the quantized super-prior information is quantized and encoded and decoded, and then super-prior decoded to generate latent features of the mean and variance of the Gaussian distribution. The latent features of the main coding output are combined and the probability density of the main coding latent features is obtained through the Gaussian entropy model. The latent features of the main coding output are quantized and encoded and decoded through the probability density to obtain the quantized latent features.

7. The method for compressing screen content images based on content adaptation according to claim 5, characterized in that: The steps The feature transformation in S1 is a nonlinear enhanced spatial feature transformation. Affine transformation parameters are generated based on the extracted original image structure information. The image features are spatially affine transformed according to the affine transformation parameters to obtain structurally enhanced image features: SFT(F,Ψ)=γ×F+β Among them, F represents the image feature, γ, β represent a pair of affine transformation parameters, and Ψ represents the original image structure information as an external prior feature.

Images