Image characterization and compression method and system based on 2DGS

Abstract

The invention provides an image representation and compression method and system based on 2DGS, and the method comprises the steps: carrying out the point cloud initialization of a to-be-represented image through employing an image block initialization method, and determining an initial point cloud; determining a plurality of two-dimensional Gaussian distributions according to the initial point cloud; performing image fitting on the plurality of two-dimensional Gaussian distributions, and determining a rendered image represented by the two-dimensional Gaussian distributions; optimizing parameters of the rendered image and the plurality of two-dimensional Gaussian distributions by adopting color loss functions of the rendered image and the image to be represented; performing quantization processing on the plurality of parameters of the two-dimensional Gaussian distribution, and determining the quantized parameters of the two-dimensional Gaussian distribution; and carrying out optimization processing on the quantized parameters of the two-dimensional Gaussian distribution by adopting a color loss function and the loss of the residual vector quantization processing, and determining a compressed image. According to the method and the device, the performance of 2DGS in image representation and image compression is improved, high-quality representation and efficient compression of the image are realized, and the rate distortion performance is optimized.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology. Specifically, it relates to an image representation and compression method and system based on 2DGS. Background Art

[0002] Image representation is a fundamental task in signal processing and computer vision. Traditional image representation methods, including grid graphics, wavelet transform, and discrete cosine transform, have been widely applied in technical fields ranging from image compression to visual task analysis. However, these techniques encounter significant obstacles when dealing with large-scale datasets and in the pursuit of efficient storage solutions.

[0003] The emergence of implicit neural representations (INRs) marks a major paradigm shift in image representation technology. Generally, implicit neural representations INRs adopt a compact neural network to derive an implicit continuous mapping from input coordinates to corresponding output values. This enables implicit neural representations INRs to capture and retain image details more efficiently, thus demonstrating significant advantages in applications such as image compression, deblurring, and super-resolution. However, most state-of-the-art implicit neural representations INRs methods rely on a large high-dimensional multi-layer perceptron (MLP) network to accurately represent high-resolution images. This dependence leads to extended training time, increased GPU memory requirements, and slower decoding speed. Although recent methods have introduced a combination of multi-resolution feature grids and compact MLPs to accelerate training and inference, they still require sufficient GPU memory to support their fast training and inference, which is difficult to meet in resource-constrained situations. Therefore, these challenges severely impede the deployment of implicit neural representations INRs in practical scenarios.

[0004] Recently, GaussianImage proposed using 2DGS for image representation and calculating the color of each pixel using an accumulative summation algorithm, which greatly accelerates the training and inference speed. However, GaussianImage adopts a simple global random initialization strategy when initializing the two-dimensional point cloud and does not design a corresponding initialization algorithm according to the image. During the image fitting process, its rendering quality highly depends on the distribution of the initial points, and the global random initialization limits the performance of 2DGS in image representation. In addition, during the image fitting process, GaussianImage uses the tanh activation function with a relatively steep gradient for the position parameters, resulting in a large number of points being optimized to the edge regions, thus significantly reducing the performance of image representation. Therefore, GaussianImage cannot fully exploit the performance of 2DGS in image representation. There is a significant redundant space for the position parameters, and GaussianImage does not perform any quantization processing on the position parameters, so its compression performance is not optimal either. Summary of the Invention

[0005] In view of the deficiencies in the prior art, an object of the present disclosure is to provide a method and system for image characterization and compression based on 2DGS.

[0006] To achieve the above object, according to one aspect of the present disclosure, there is provided a method for image characterization and compression based on 2DGS, including:

[0007] Performing point cloud initialization on the image to be characterized by using an image block initialization method to determine an initial point cloud;

[0008] Determining a plurality of two-dimensional Gaussian distributions according to the initial point cloud;

[0009] Performing image fitting on the plurality of two-dimensional Gaussian distributions to determine a rendered image characterized by the two-dimensional Gaussian distributions;

[0010] Optimizing the parameters of the plurality of two-dimensional Gaussian distributions of the rendered image by using the color loss function of the rendered image and the image to be characterized to determine the optimized rendered image and the parameters of the optimized two-dimensional Gaussian distributions;

[0011] Performing quantization processing on the parameters of the optimized two-dimensional Gaussian distributions to determine the quantized parameters of the two-dimensional Gaussian distributions;

[0012] Optimizing the parameters of the quantized two-dimensional Gaussian distributions according to the color loss function of the rendered image and the image to be characterized and the loss of residual vector quantization processing in the quantization processing to determine the compressed image.

[0013] Optionally, the performing point cloud initialization on the image to be characterized by using an image block initialization method to determine an initial point cloud includes:

[0014] Dividing the image to be characterized into image blocks of a preset size of pixels;

[0015] Determining the complexity of each image block according to the variance of each image block;

[0016] Sorting the image blocks in descending order according to the complexity of each image block to determine the sorted image blocks;

[0017] Evenly dividing the sorted image blocks into three categories and allocating the initialization points according to a preset ratio to the three categories to determine the initialization points allocated to each category;

[0018] Evenly distributing the initialization points allocated to each category to the image blocks corresponding to each category to determine the initial point cloud.

[0019] Optionally, the attribute definition of each of the two-dimensional Gaussian distributions includes a position parameter, a covariance parameter, and a color coefficient.

[0020] Optionally, the image fitting of the multiple two-dimensional Gaussian distributions to determine a rendered image represented by the two-dimensional Gaussian distributions includes:

[0021] Construct a new activation function for the position parameters of the multiple two-dimensional Gaussian distributions;

[0022] Activate the position parameters of the multiple two-dimensional Gaussian distributions using the new activation function for the position parameters of the multiple two-dimensional Gaussian distributions to determine the activated position parameters of the multiple two-dimensional Gaussian distributions;

[0023] Perform image fitting on the multiple two-dimensional Gaussian distributions according to the activated position parameters, the covariance parameter, and the color coefficient of the multiple two-dimensional Gaussian distributions to determine the rendered image represented by the two-dimensional Gaussian distributions;

[0024] Determine each pixel value using an accumulation summation algorithm according to the color coefficient of the two-dimensional Gaussian distribution corresponding to each pixel value of the rendered image;

[0025] Determine the color loss function between the rendered image and the image to be represented according to each pixel value of the rendered image and each pixel value of the image to be represented.

[0026] Optionally, the quantization process of the parameters of the optimized two-dimensional Gaussian distribution to determine the parameters of the quantized two-dimensional Gaussian distribution includes:

[0027] Perform a first integer quantization process on the position parameter in the parameters of the optimized two-dimensional Gaussian distribution according to a preset first quantization bit number to determine the position parameter after the first integer quantization process:

[0028] Perform a second integer quantization process on the covariance parameter in the parameters of the optimized two-dimensional Gaussian distribution according to a preset second quantization bit number to determine the covariance parameter after the second integer quantization process;

[0029] Perform a residual vector quantization process on the color coefficient in the parameters of the optimized two-dimensional Gaussian distribution to determine the color coefficient after the residual vector quantization process.

[0030] Optionally, the optimization process of the parameters of the quantized two-dimensional Gaussian distribution according to the color loss function between the rendered image and the image to be represented and the loss of the residual vector quantization process in the quantization process to determine a compressed image includes:

[0031] Construct a new loss function based on the color loss function of the rendered image and the image to be characterized, as well as the loss of residual vector quantization in the quantization process;

[0032] Optimize the parameters of the quantized two-dimensional Gaussian distribution using the new loss function to determine the compressed image.

[0033] According to a second aspect of the present disclosure, there is provided an image characterization and compression system based on 2DGS, including:

[0034] An initialization module for performing point cloud initialization on the image to be characterized using an image block initialization method to determine an initial point cloud;

[0035] A two-dimensional Gaussian distribution determination module for determining a plurality of two-dimensional Gaussian distributions based on the initial point cloud;

[0036] An image fitting module for fitting the plurality of two-dimensional Gaussian distributions to determine a rendered image represented by the two-dimensional Gaussian distribution;

[0037] An image characterization module for optimizing the parameters of the plurality of two-dimensional Gaussian distributions of the rendered image using the color loss function of the rendered image and the image to be characterized to determine the optimized rendered image and the parameters of the optimized two-dimensional Gaussian distribution;

[0038] A quantization module for performing quantization processing on the parameters of the optimized two-dimensional Gaussian distribution to determine the parameters of the quantized two-dimensional Gaussian distribution;

[0039] An image compression module for optimizing the parameters of the quantized two-dimensional Gaussian distribution based on the color loss function of the rendered image and the image to be characterized, as well as the loss of residual vector quantization in the quantization process, to determine the compressed image. According to a third aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method provided in the first aspect of the present disclosure.

[0040] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0041] A memory having stored thereon a computer program;

[0042] A processor for executing the computer program in the memory to implement the steps of the method provided in the first aspect of the present disclosure.

[0043] Compared with the prior art, the embodiments of the present disclosure have at least one of the following beneficial effects:

[0044] Through the above technical solutions, an initialization algorithm corresponding to the image is designed, and the point cloud initialization of the image to be characterized is performed by using the image block initialization method, so as to improve the image quality of the rendered image in the image fitting process and improve the performance of the two-dimensional Gaussian distribution in the image characterization; the parameters of the optimized two-dimensional Gaussian distribution are quantized to further improve the compression performance of the two-dimensional Gaussian distribution and optimize the rate-distortion performance, so as to achieve a better compression efficiency ratio.

[0045] In the embodiment of the present disclosure, during the image fitting process, a new activation function for the position parameters of multiple two-dimensional Gaussian distributions is constructed, which significantly improves the performance of the two-dimensional Gaussian distribution in the image characterization. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] By reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings, other features, objects, and advantages of the present disclosure will become more apparent:

[0047] Figure 1 It is a schematic flowchart of a method for image characterization and compression based on 2DGS shown according to an exemplary embodiment.

[0048] Figure 2 It is a schematic process diagram of a method for image characterization based on 2DGS shown according to an exemplary embodiment.

[0049] Figure 3 It is a schematic process diagram of a method for image compression based on 2DGS shown according to an exemplary embodiment.

[0050] Figure 4 It is a schematic diagram of the comparison of the PSNR index results of the method provided by the present disclosure and the existing baseline method on the Kodak dataset shown according to an exemplary embodiment.

[0051] Figure 5 It is a schematic diagram of the comparison of the MS-SSIM index of the method provided by the present disclosure and the existing baseline method on the Kodak dataset shown according to an exemplary embodiment.

[0052] Figure 6 It is a block diagram of a system for image characterization and compression based on 2DGS shown according to an exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] The present disclosure will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present disclosure, but do not limit the present disclosure in any form. It should be noted that those of ordinary skill in the art can make several modifications and improvements without departing from the concept of the present disclosure. These all belong to the protection scope of the present disclosure.

[0054] Figure 1 It is a schematic flowchart of a 2DGS-based image representation and compression method shown according to an exemplary embodiment. Figure 2 It is a schematic process diagram of a 2DGS-based image representation method shown according to an exemplary embodiment. Figure 3 It is a schematic process diagram of a 2DGS-based image compression method shown according to an exemplary embodiment.

[0055] As Figure 1 shown, the present disclosure provides a 2DGS-based image representation and compression method, including S11 to S16. Among them, as Figure 1 , Figure 2 shown, steps S11 to S14 are 2DGS-based image representation methods, as Figure 1 , Figure 3 shown, steps S15 to S16 are 2DGS-based image compression methods.

[0056] S11, adopt an image block initialization method to perform point cloud initialization on the image to be represented, and determine the initial point cloud.

[0057] S12, according to the initial point cloud, determine multiple two-dimensional Gaussian distributions.

[0058] Among them, the two-dimensional Gaussian distribution is defined by three attributes. The attribute definition of each two-dimensional Gaussian distribution includes a position parameter, a covariance parameter, and a color coefficient, that is, the parameters of the two-dimensional Gaussian distribution include a position parameter, a covariance parameter, and a color coefficient.

[0059] S13, perform image fitting on multiple two-dimensional Gaussian distributions, and determine a rendered image represented by the two-dimensional Gaussian distribution.

[0060] Among them, the image to be represented is represented by multiple two-dimensional Gaussian distributions.

[0061] S14, use the color loss function of the rendered image and the image to be represented to optimize the parameters of the multiple two-dimensional Gaussian distributions of the rendered image, and determine the optimized rendered image and the parameters of the optimized two-dimensional Gaussian distribution.

[0062] Among them, the parameters of the optimized two-dimensional Gaussian distribution include the optimized position parameter, the optimized covariance parameter, and the optimized color coefficient.

[0063] S15, perform quantization processing on the parameters of the optimized two-dimensional Gaussian distribution, and determine the quantized parameters of the two-dimensional Gaussian distribution.

[0064] Among them, the quantization processing process includes first integer quantization processing, second integer quantization processing, and residual vector quantization processing.

[0065] S16. Optimize the parameters of the quantized two-dimensional Gaussian distribution based on the color loss function of the rendered image and the target image to be represented, as well as the loss in the residual vector quantization process in quantization processing, and determine the compressed image.

[0066] Through the above technical solutions, design an initialization algorithm corresponding to the image, use the image block initialization method to perform point cloud initialization on the target image to be represented, improve the image quality of the rendered image in the image fitting process, and improve the performance of the two-dimensional Gaussian distribution in image representation; perform quantization processing on the parameters of the optimized two-dimensional Gaussian distribution, perform the first integer quantization processing on the position parameters, perform the second integer quantization processing on the covariance parameters, and perform residual vector quantization processing on the color coefficients, further improve the compression performance of the two-dimensional Gaussian distribution, optimize the rate-distortion performance, and achieve a better compression efficiency ratio.

[0067] In a possible embodiment, S11. Use the image block initialization method to perform point cloud initialization on the target image to be represented and determine the initial point cloud, including: S111 to S115.

[0068] S111. Divide the target image to be represented into image blocks of a preset pixel size.

[0069] Among them, the preset pixel size can be 4×4 pixels.

[0070] S112. Determine the complexity of each image block according to the variance of each image block.

[0071] Among them, use the variance of each image block as the complexity of each image block.

[0072] S113. Sort the image blocks in descending order according to the complexity of each image block and determine the image blocks sorted in descending order.

[0073] S114. Divide the image blocks sorted in descending order into three categories on average, and allocate the initialization points according to a preset ratio to the three categories to determine the number of initialization points allocated to each category.

[0074] Among them, the preset ratio can be 6:2:1.

[0075] S115. Average the number of initialization points allocated to each category and allocate them to the image blocks corresponding to each category to determine the initial point cloud.

[0076] As an example, when the number of initialization points allocated to this category can be divisible by the number of image blocks in the corresponding category, average the number of initialization points allocated to each category and allocate them to the image blocks in the corresponding category.

[0077] As another example, when the initialized number of points assigned to the category cannot be divided evenly by the number of image patches in the corresponding category, sort them in descending order of complexity, and sequentially assign the remaining initialized points to the image patches of the corresponding category from the highest to the lowest complexity.

[0078] By designing a point cloud initialization algorithm corresponding to the image, improve the quality of the rendered image in the subsequent image fitting process, and break through the performance limitations of global random initialization in the image representation of 2DGS.

[0079] In a possible implementation, the attribute definition of each two-dimensional Gaussian distribution includes a position parameter, a covariance parameter, and a color coefficient.

[0080] In a possible embodiment, S13. For image fitting of multiple two-dimensional Gaussian distributions to determine a rendered image represented by the two-dimensional Gaussian distribution may include S131 to S135.

[0081] S131. Construct a new activation function for the position parameters of multiple two-dimensional Gaussian distributions.

[0082] S132. Activate the position parameters of multiple two-dimensional Gaussian distributions using the new activation function for the position parameters of multiple two-dimensional Gaussian distributions to determine the activated position parameters of multiple two-dimensional Gaussian distributions.

[0083] Based on steps S131 to S132, construct a new activation function for the position parameter x and activate the position parameter using the new activation function. The specific process is as follows:

[0084]

[0085] Among them, represents the activated position parameter, and x represents the position parameter of the two-dimensional Gaussian distribution.

[0086] The activated position parameter can be used to calculate the pixel value of the final rendered image. By constructing a new activation function for the position parameter of the two-dimensional Gaussian distribution in the image fitting process, the performance of the two-dimensional Gaussian distribution in image representation is significantly improved.

[0087] S133. Perform image fitting on multiple two-dimensional Gaussian distributions according to the new activation function for the position parameters of multiple two-dimensional Gaussian distributions to determine a rendered image represented by the two-dimensional Gaussian distribution.

[0088] S134. According to the color coefficient of the two-dimensional Gaussian distribution corresponding to each pixel value of the rendered image, use the cumulative summation algorithm to determine each pixel value.

[0089] Among them, each pixel value in the rendered image is determined using the following formula:

[0090]

[0091] Among them, represents the i-th pixel value in the rendered image, N represents the number of two-dimensional Gaussian distributions corresponding to the i-th pixel value in the rendered image, and c n represents the color coefficient of the two-dimensional Gaussian distribution;

[0092]

[0093] Among them, d n represents the displacement distance between the center of the two-dimensional Gaussian distribution and the pixel center, and Σ represents the covariance parameter.

[0094] S135. According to each pixel value of the rendered image and each pixel value of the image to be characterized, determine the color loss function of the rendered image and the image to be characterized.

[0095] Among them, the color loss function of the rendered image and the image to be characterized adopts the L2 loss between the rendered image and the image to be characterized.

[0096] The color loss function of the rendered image and the image to be characterized is expressed as:

[0097]

[0098] Among them, L train represents the L2 loss between the rendered image and the image to be characterized, C i represents the i-th pixel value of the image to be characterized, represents the i-th pixel value in the rendered image, and I represents the total number of pixels in the image to be characterized.

[0099] In a possible embodiment, S14. Use the color loss function of the rendered image and the image to be characterized to optimize the parameters of multiple two-dimensional Gaussian distributions of the rendered image, and determine the optimized rendered image and the parameters of the optimized two-dimensional Gaussian distributions, including:

[0100] Based on the color loss function of the rendered image and the image to be characterized, optimize the parameters of the rendered image and the multiple two-dimensional Gaussian distributions corresponding to the rendered image, that is, optimize the position parameters, optimize the covariance parameters, and optimize the color coefficients, and determine the optimized rendered image, the optimized position parameters, the optimized covariance parameters, and the optimized color coefficients.

[0101] Such as Figure 3As shown, a method for image representation and compression based on 2DGS provided by the present disclosure can also be used as an ultra-high-speed image codec for image compression. In a possible embodiment, in S15, the parameters of the optimized two-dimensional Gaussian distribution are quantized to determine the quantized parameters of the two-dimensional Gaussian distribution, including S151 to S153.

[0102] S151, perform a first integer quantization process on the position parameters in the parameters of the optimized two-dimensional Gaussian distribution according to a preset first quantization bit number to determine the position parameters after the first integer quantization process.

[0103] Among them, the preset first quantization bit number is 12 bits, that is, perform 12-bit integer quantization on the optimized position parameters.

[0104] S152, perform a second integer quantization process on the covariance parameters in the parameters of the optimized two-dimensional Gaussian distribution according to a preset second quantization bit number to determine the covariance parameters after the second integer quantization process.

[0105] Among them, the preset second quantization bit number is 7 bits, that is, perform 7-bit integer quantization on the optimized covariance parameters.

[0106] In the present disclosure, performing 12-bit integer quantization on the optimized position parameters and performing 7-bit integer quantization on the optimized covariance parameters can adopt the following formula:

[0107]

[0108] Among them, represents the parameter after the integer quantization process, represents the quantized data, s represents the scaling coefficient, Z represents the zero-point offset, b represents the quantization bit number, represents the rounding operation.

[0109] The scaling coefficient s and the zero-point offset Z are both learnable parameters. The initial value of the scaling coefficient s is 0, and the initial value of the zero-point offset Z is v max represents the maximum value of the parameter, v min represents the minimum value of the parameter.

[0110] clamp(·) is defined as:

[0111]

[0112] Among them, x represents the input value, a represents the lower limit value, and c represents the upper limit value.

[0113] S153, perform residual vector quantization on the color coefficient in the parameters of the optimized two-dimensional Gaussian distribution to determine the color coefficient after the residual vector quantization process.

[0114] Among them, the residual vector quantization is a cascaded two-layer vector quantization, the codebook size of each layer is 8, and the color coefficient is calculated according to the values in the codebook, specifically as follows:

[0115]

[0116] Among them, represents the color vector output after the quantization of the m-th layer, that is, the color coefficient after the residual vector quantization process, represents the codebook of the m-th layer, and i m ∈{0,…,7}N represents the codebook index of the m-th layer, represents the vector corresponding to the index i in the codebook C.

[0117] The present disclosure further improves the compression performance of 2DGS and optimizes the rate-distortion performance by applying 12-bit integer quantization, 7-bit integer quantization, and residual vector quantization to the position parameter, covariance parameter, and color coefficient respectively.

[0118] In a possible embodiment, S16, optimize the parameters of the quantized two-dimensional Gaussian distribution according to the color loss function of the rendered image and the image to be characterized and the loss of the residual vector quantization process in the quantization process to determine the compressed image, which may include S161 to S162.

[0119] S161, construct a new loss function according to the color loss function of the rendered image and the image to be characterized and the loss of the residual vector quantization process in the quantization process.

[0120] Among them, the new loss function is the weighted sum of the color loss function of the rendered image and the image to be characterized and the loss of the residual vector quantization process.

[0121] The new loss function is:

[0122]

[0123] Among them, L tune represents the new loss function, represents the L2 loss between the rendered image and the image to be characterized, L c represents the loss of the residual vector quantization process, λ represents the weight of the residual vector quantization loss, N represents the number of two-dimensional Gaussian distributions corresponding to the i-th pixel value in the rendered image, and c n represents the color coefficient of the two-dimensional Gaussian distribution, represents the color vector output after the quantization of the (k - 1)-th layer, Ck The codebook of the k-th layer is denoted as, and sg[·] represents the stop-gradient operation.

[0124] S162. A new loss function is used to optimize the parameters of the quantized two-dimensional Gaussian distribution to determine the compressed image.

[0125] Among them, a new loss function is used to finely tune and optimize the parameters of the quantized two-dimensional Gaussian distribution.

[0126] In a possible embodiment, the Kodak dataset is used as the test sequence. The Kodak dataset includes 24 images. The rendering performance of a 2DGS-based image representation and compression method provided by the present disclosure, as well as the SIREN, WIRE, I-NGP, NeuRBF, 3DGS, and GaussianImage methods, is compared. The evaluation metrics are: peak signal-to-noise ratio PSNR, multi-scale structural similarity parameter MS-SSIM, training time Training Time, rendering image speed FPS, GPU memory capacity required for optimization GPU Mem, and number of parameters Params. Among them, the larger the value of the peak signal-to-noise ratio PSNR, the higher the quality of the rendered image; the closer the value of the multi-scale structural similarity parameter MS-SSIM is to 1, the higher the quality of the rendered image; the smaller the value of the training time Training Time, the better the rendering performance; the larger the rendering image speed FPS, the better the rendering performance; the smaller the value of the GPU memory capacity required for optimization GPU Mem, the better the rendering performance; and the smaller the value of the number of parameters Params, the better the rendering performance. The comparison results of the rendering performance metrics are shown in Table 1:

[0127]

[0128]

[0129] Table 1

[0130] As shown in Table 1, a 2DGS-based image representation and compression method provided by the present disclosure achieves the best in rendering quality and is second only to the GaussianImage method in terms of training time and rendering speed.

[0131] Figure 4 It is a schematic diagram comparing the PSNR index results of a method provided by the present disclosure and existing baseline methods on the Kodak dataset according to an exemplary embodiment. Figure 5 It is a schematic diagram comparing the MS-SSIM index of a method provided by the present disclosure and existing baseline methods on the Kodak dataset according to an exemplary embodiment.

[0132] Such as Figure 4 、Figure 5 As shown, the rate-distortion performance of a 2DGS-based image representation and compression method provided by the present disclosure is compared with that of traditional encoders JPEG, JPEG2000, the GaussianImage codec based on implicit neural representation, and COIN using the baseline method on the Kodak dataset.

[0133] As Figure 4 , Figure 5 shown, compared with GaussianImage, the embodiment of the method provided by the present disclosure saves a BD-rate of -31.5821% and a BD-PSNR of 1.0194 dB; compared with COIN, the embodiment of the method provided by the present disclosure saves a BD-rate of -34.5708% and a BD-PSNR of 1.2171 dB. Thus, the rate-distortion performance of the method provided by the present disclosure on the Kodak dataset is superior to that of GaussianImage and COIN.

[0134] In addition, for further comparison, the method provided by the present disclosure and the existing GaussianImage and COIN were used to process multiple images, and it can be obtained that the method provided by the present disclosure can achieve a higher-quality image representation while consuming less storage space.

[0135] Using a 2DGS-based image representation and compression method provided by the present disclosure can give full play to the advantages of high-quality reconstruction of 2DGS. When using 2DGS to represent images, the image rendering speed can reach 1500 FPS. Moreover, it can also be used as an efficient image codec to compress images, achieving higher rate-distortion performance compared with GaussianImage and COIN, thereby realizing high-quality image representation and efficient image compression.

[0136] Figure 6 is a block diagram of a 2DGS-based image representation and compression system shown according to an exemplary embodiment.

[0137] Based on the same concept, as Figure 6 shown, the present disclosure also provides a 2DGS-based image representation and compression system 100, including: an initialization module 110, a two-dimensional Gaussian distribution determination module 120, an image fitting module 130, an image representation module 140, a quantization module 150, and an image compression module 160.

[0138] The initialization module 110 is used to perform point cloud initialization on the image to be represented by using an image block initialization method to determine the initial point cloud;

[0139] The two-dimensional Gaussian distribution determination module 120 is used to determine a plurality of two-dimensional Gaussian distributions according to the initial point cloud;

[0140] An image fitting module 130 is configured to perform image fitting on multiple two-dimensional Gaussian distributions to determine a rendered image represented by a two-dimensional Gaussian distribution.

[0141] An image characterization module 140 is configured to optimize parameters of multiple two-dimensional Gaussian distributions of a rendered image by using a color loss function of the rendered image and an image to be characterized, and determine an optimized rendered image and parameters of the optimized two-dimensional Gaussian distribution.

[0142] A quantization module 150 is configured to perform quantization processing on parameters of the optimized two-dimensional Gaussian distribution to determine quantized parameters of the two-dimensional Gaussian distribution.

[0143] An image compression module 160 is configured to optimize parameters of the quantized two-dimensional Gaussian distribution according to a color loss function of the rendered image and an image to be characterized and a loss of residual vector quantization processing in the quantization processing, and determine a compressed image.

[0144] Through the above technical solutions, an initialization algorithm corresponding to an image is designed, and a point cloud initialization is performed on an image to be characterized by using an image block initialization method, so as to improve the image quality of the rendered image in the image fitting process and improve the performance of the two-dimensional Gaussian distribution in image characterization; quantization processing is performed on parameters of the optimized two-dimensional Gaussian distribution to further improve the compression performance of the two-dimensional Gaussian distribution, optimize the rate-distortion performance, and achieve a better compression efficiency ratio.

[0145] Regarding the embodiments of the above system, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0146] Based on the same concept as above, in another embodiment of the present disclosure, an electronic device is further provided, including a memory, a processor, and a computer program stored on the memory and capable of running on the processor. When the processor executes the program, it is used to execute an image characterization and compression method based on 2DGS.

[0147] Optionally, a memory for storing programs; the memory may include volatile memory (e.g., random-access memory, such as static random-access memory (SRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), etc.); the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules for implementing the above methods), computer instructions, etc. The above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0148] The above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0149] A processor for executing the computer programs stored in the memory to implement each step in the method related to the above embodiments. For details, reference can be made to the relevant descriptions in the previous method embodiments.

[0150] The processor and the memory can be in an independent structure or an integrated structure. When the processor and the memory are in an independent structure, the memory and the processor can be coupled and connected through a bus.

[0151] In the embodiments of the present disclosure, a non-transitory computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by the processor, the steps of a method for 2DGS-based image representation and compression in any of the above embodiments are implemented.

[0152] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0153] This disclosure is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to produce a machine, such that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0154] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0155] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0156] Although the preferred embodiments of the disclosure have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications that fall within the scope of the disclosure.

[0157] Obviously, those skilled in the art can make various changes and variations to the disclosure without departing from the spirit and scope of the disclosure. Thus, if these modifications and variations of the disclosure fall within the scope of the claims of the disclosure and their equivalent technologies, the disclosure is also intended to include these modifications and variations.

Claims

1. An image representation and compression method based on 2DGS, characterized in that, Comprising: Performing point cloud initialization on the image to be characterized by using an image block initialization method to determine an initial point cloud; Determining a plurality of two-dimensional Gaussian distributions according to the initial point cloud; Performing image fitting on the plurality of two-dimensional Gaussian distributions to determine a rendered image represented by the two-dimensional Gaussian distributions; Optimizing parameters of the plurality of two-dimensional Gaussian distributions of the rendered image by using a color loss function of the rendered image and the image to be characterized to determine an optimized rendered image and parameters of the optimized two-dimensional Gaussian distributions; Performing quantization processing on the parameters of the optimized two-dimensional Gaussian distributions to determine quantized parameters of the two-dimensional Gaussian distributions; Optimizing the quantized parameters of the two-dimensional Gaussian distributions according to the color loss function of the rendered image and the image to be characterized and the loss of residual vector quantization processing in the quantization processing to determine a compressed image.

2. The method according to claim 1, wherein The performing point cloud initialization on the image to be characterized by using an image block initialization method to determine an initial point cloud includes: Dividing the image to be characterized into image blocks of preset size pixels; Determining the complexity of each image block according to the variance of each image block; Performing descending order sorting on the image blocks according to the complexity of each image block to determine the image blocks after descending order sorting; Evenly dividing the image blocks after descending order sorting into three categories and allocating the initialization points according to a preset ratio to the three categories to determine the initialization points allocated to each category; Evenly distributing the initialization points allocated to each category to the image blocks corresponding to each category to determine the initial point cloud.

3. The method according to claim 1, wherein The attribute definition of each two-dimensional Gaussian distribution includes a position parameter, a covariance parameter, and a color coefficient; The performing image fitting on the plurality of two-dimensional Gaussian distributions to determine a rendered image represented by the two-dimensional Gaussian distributions includes: Constructing a new activation function for the position parameters of the plurality of two-dimensional Gaussian distributions; Activating the position parameters of the plurality of two-dimensional Gaussian distributions by using the new activation function for the position parameters of the plurality of two-dimensional Gaussian distributions to determine the activated position parameters of the plurality of two-dimensional Gaussian distributions; Performing image fitting on the plurality of two-dimensional Gaussian distributions according to the activated position parameters of the plurality of two-dimensional Gaussian distributions, the covariance parameter, and the color coefficient to determine the rendered image represented by the two-dimensional Gaussian distributions; Determining each pixel value by using an accumulation summation algorithm according to the color coefficient of the two-dimensional Gaussian distribution corresponding to each pixel value of the rendered image; Determining a color loss function of the rendered image and the image to be characterized according to each pixel value of the rendered image and each pixel value of the image to be characterized.

4. The method according to claim 3, characterized in that, The constructing a new activation function for the position parameters of the plurality of two-dimensional Gaussian distributions includes: Among them, represents the position parameter after activation, and x represents the position parameter of the two-dimensional Gaussian distribution; The determining each pixel value by using an accumulation summation algorithm according to the color coefficient of the two-dimensional Gaussian distribution corresponding to each pixel value of the rendered image includes: Among them, represents the i-th pixel value in the rendered image, N represents the number of two-dimensional Gaussian distributions corresponding to the i-th pixel value in the rendered image, and c n represents the color coefficient of the two-dimensional Gaussian distribution; where d n represents the displacement distance between the center of the two-dimensional Gaussian distribution and the pixel center, and Σ represents the covariance parameter; The determining a color loss function of the rendered image and the image to be characterized according to each pixel value of the rendered image and each pixel of the image to be characterized includes: Among them, L train represents the L2 loss between the rendered image and the image to be characterized, and C i represents the i-th pixel value of the image to be characterized, represents the i-th pixel value in the rendered image, and I represents the total number of pixels of the image to be characterized.

5. The method according to claim 2, wherein Quantifying the parameters of the optimized two-dimensional Gaussian distribution to determine the parameters of the quantized two-dimensional Gaussian distribution, including: Performing a first integer quantization process on the position parameters in the parameters of the optimized two-dimensional Gaussian distribution according to a preset first quantization bit number to determine the position parameters after the first integer quantization process: Performing a second integer quantization process on the covariance parameters in the parameters of the optimized two-dimensional Gaussian distribution according to a preset second quantization bit number to determine the covariance parameters after the second integer quantization process; Performing a residual vector quantization process on the color coefficients in the parameters of the optimized two-dimensional Gaussian distribution to determine the color coefficients after the residual vector quantization process.

6. The method according to claim 1, characterized in that, Optimizing the parameters of the quantized two-dimensional Gaussian distribution according to the color loss function of the rendered image and the to-be-represented image and the loss of the residual vector quantization process in the quantization process to determine the compressed image, including: Constructing a new loss function according to the color loss function of the rendered image and the to-be-represented image and the loss of the residual vector quantization process in the quantization process; Using the new loss function to optimize the parameters of the quantized two-dimensional Gaussian distribution to determine the compressed image.

7. The method according to claim 6, wherein Constructing the new loss function according to the color loss function of the rendered image and the to-be-represented image and the loss of the residual vector quantization process in the quantization process, including: Among them, L tune represents the new loss function, represents the L2 loss between the rendered image and the image to be characterized, L c represents the loss of the residual vector quantization process, λ represents the weight of the loss of the residual vector quantization process, N represents the number of two-dimensional Gaussian distributions corresponding to the i-th pixel value in the rendered image, c n represents the color coefficient of the two-dimensional Gaussian distribution, represents the color vector output after quantization through k - 1 layers, C k represents the codebook of the k-th layer, and sg[·] represents the stop gradient operation.

8. An image representation and compression system based on 2DGS, characterized in that, Including: An initialization module for performing point cloud initialization on the to-be-represented image by using an image block initialization method to determine the initial point cloud; A two-dimensional Gaussian distribution determination module for determining a plurality of two-dimensional Gaussian distributions according to the initial point cloud; An image fitting module for performing image fitting on the plurality of two-dimensional Gaussian distributions to determine the rendered image represented by the two-dimensional Gaussian distribution; An image representation module for optimizing the parameters of the plurality of two-dimensional Gaussian distributions of the rendered image by using the color loss function of the rendered image and the to-be-represented image to determine the optimized rendered image and the parameters of the optimized two-dimensional Gaussian distribution; A quantization module for quantifying the parameters of the optimized two-dimensional Gaussian distribution to determine the parameters of the quantized two-dimensional Gaussian distribution; An image compression module for optimizing the parameters of the quantized two-dimensional Gaussian distribution according to the color loss function of the rendered image and the to-be-represented image and the loss of the residual vector quantization process in the quantization process to determine the compressed image.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

10. An electronic device, characterized in that, Including: A memory storing a computer program thereon; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-7.

Images