Voxel grid decoding method, voxel grid coding method and voxel grid decoding device

Abstract

The invention provides a voxel grid decoding method, a voxel grid coding method and a voxel grid coding device. The voxel grid decoding method comprises the following steps: acquiring decoding metadata, a pruning mask, a coarse residual error and a reference voxel index from a code stream; constructing an empty voxel grid based on the decoding metadata; for any sub-voxel in the empty voxel grid, screening out a voxel which is not pruned from the plurality of sub-voxels through the pruning mask of the sub-voxel; determining a corresponding reference voxel for each unpruned voxel according to the reference voxel index; for any one non-pruned voxel in the plurality of non-pruned voxels, determining the sum of the reference voxel corresponding to the non-pruned voxel and the coarse residual as a coarse reconstruction value of the non-pruned voxel; and performing inverse quantization on the coarse reconstruction value of each non-pruned voxel by using a coarse quantization step to obtain a first voxel value of the non-pruned voxel. The embodiment of the invention is used for compressing the voxel grid, and a higher compression ratio can be realized by fully utilizing spatial correlation.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a voxel grid decoding method, a voxel grid encoding method and a device. Background Art

[0002] Neural Radiance Field (NeRF) technology can reconstruct a three-dimensional scene based on sparse views and then render the image from any viewpoint, but NeRF technology has the problem of high computational complexity.

[0003] At present, although voxel-based radiation fields can effectively alleviate the problem of computational complexity, this technology will generate a large number of additional parameters, resulting in a heavy computational burden.

[0004] Therefore, there is an urgent need for a data processing method for compressing voxel radiation fields to solve the above technical problems. Summary of the invention

[0005] In view of this, the present disclosure proposes a voxel grid decoding method, a voxel grid encoding method and a device to solve the problem in the related art that the voxel radiation field will generate a large number of additional parameters, resulting in a heavy computational burden.

[0006] The first aspect of the present disclosure provides a method for decoding a voxel grid, including:

[0007] Get decoding metadata, pruning mask, coarse residual and reference voxel index from the bitstream;

[0008] constructing an empty voxel grid based on the decoded metadata; the decoded metadata including the size, resolution and coarse quantization step of the voxel grid; the voxel grid including a plurality of sub-voxels;

[0009] For any sub-voxel in the empty voxel grid, filter out unpruned voxels from the multiple sub-voxels by using the pruning mask of the sub-voxel;

[0010] Determine a corresponding reference voxel for each unpruned voxel according to the reference voxel index;

[0011] For any unpruned voxel among the plurality of unpruned voxels, determining the sum of the reference voxel and the rough residual corresponding to the unpruned voxel as the rough reconstruction value of the unpruned voxel;

[0012] The coarse reconstruction value of each of the unpruned voxels is dequantized using the coarse quantization step size to obtain a first voxel value of the unpruned voxel.

[0013] The disclosed embodiment constructs an empty voxel grid by decoding metadata, and processes each sub-voxel in the empty voxel grid using a pruning mask, a coarse residual, and a reference voxel index, thereby making full use of spatial correlation to achieve a higher compression ratio.

[0014] In the embodiment of the present disclosure, filtering out unpruned voxels from the plurality of sub-voxels by using the pruning mask of the sub-voxel includes:

[0015] If the pruning mask of the sub-voxel is a first value, determining the sub-voxel as a pruned voxel;

[0016] If the pruning mask of the sub-voxel is a second value, the sub-voxel is determined to be an unpruned voxel.

[0017] In the embodiment of the present disclosure, the reference voxel index includes index values ​​of multiple unpruned voxels and a reference voxel corresponding to each index value; and determining a corresponding reference voxel for each unpruned voxel according to the reference voxel index includes:

[0018] For any unpruned voxel among the plurality of unpruned voxels, a reference voxel corresponding to the index value of the unpruned voxel is determined based on the reference voxel index.

[0019] In the embodiment of the present disclosure, after obtaining the first voxel value of the unpruned voxel, the method further includes:

[0020] Select key voxels from multiple unpruned voxels;

[0021] For any key voxel among the multiple key voxels, dequantize the fine residual of the key voxel using a fine quantization step to obtain a reconstruction residual;

[0022] The sum of the first voxel value and the reconstruction residual is used as the fine reconstruction value of the key voxel.

[0023] In the disclosed embodiment, the key voxels are screened out from a plurality of unpruned voxels, including:

[0024] Obtaining a key voxel mask from the bitstream;

[0025] For any unpruned voxel, if the key voxel mask of the unpruned voxel is a preset mask value, the unpruned voxel is determined to be a key voxel.

[0026] A second aspect of the present disclosure provides a method for encoding a voxel grid, including:

[0027] Generate a first voxel grid according to images from multiple perspectives; the first voxel grid is used to represent the three-dimensional structure of the current scene; each first voxel in the first voxel grid is used to represent radiation information at different positions;

[0028] Pruning pruned voxels in the first voxel grid to obtain a second voxel grid; the pruned voxels refer to first voxels whose importance scores are less than a first importance score threshold, and any importance score is used to represent a contribution value of the first voxel to an image rendering result;

[0029] For any second voxel in the second voxel grid, select a reference voxel corresponding to the second voxel, the reference voxel being a second voxel in the second voxel grid that is adjacent to the second voxel and has the smallest absolute error sum;

[0030] The second voxel grid is encoded based on a plurality of second voxels and a plurality of reference voxels corresponding one-to-one to the plurality of second voxels.

[0031] In an embodiment of the present disclosure, encoding the second voxel grid based on a plurality of second voxels and a plurality of reference voxels corresponding one-to-one to the plurality of second voxels includes:

[0032] Calculating a second absolute error sum between each second voxel and its corresponding reference voxel to obtain a plurality of second absolute error sums corresponding one-to-one to the plurality of second voxels;

[0033] taking an average value of the plurality of second absolute error sums as a first weighting coefficient, and using the first weighting coefficient to perform weighted processing on the first loss function of the first voxel grid to obtain a second loss function;

[0034] The second voxel grid is adjusted by the second loss function to obtain a third voxel grid; the third voxel grid includes a plurality of third voxels;

[0035] quantizing the plurality of third voxels respectively using a coarse quantization step size to obtain a plurality of quantized third voxels;

[0036] Subtract each quantized third voxel from its corresponding reference voxel to obtain the corresponding rough residual;

[0037] According to the coarse residual corresponding to each quantized third voxel, dequantize each quantized third voxel to obtain a coarse reconstructed voxel grid corresponding to the third voxel grid; the coarse reconstructed voxel grid includes a plurality of fourth voxels;

[0038] Screening out a plurality of key voxels from the plurality of fourth voxels; the key voxels refer to fourth voxels whose importance scores are greater than a second importance score threshold among the plurality of fourth voxels;

[0039] For any one of the plurality of key voxels, determining a target second voxel having the same position as the key voxel from the second voxel grid;

[0040] Calculating a third absolute error sum between the key voxel and the target second voxel to obtain a plurality of third absolute error sums corresponding to the plurality of key voxels one by one;

[0041] Taking an average value of the plurality of third absolute error sums as a second weighting coefficient, and using the second weighting coefficient to perform weighted processing on the first loss function to obtain a third loss function;

[0042] Adjusting a plurality of key voxels in the coarsely reconstructed voxel grid by using the third loss function to obtain a fourth voxel grid; the fourth voxel grid includes a plurality of adjusted key voxels;

[0043] For any one of the plurality of adjusted key voxels, subtract the adjusted key voxel from a second voxel at the same position in the second voxel grid;

[0044] Quantize the subtraction result to obtain a fine residual;

[0045] The coarse residual, the fine residual, the reference voxel index, the pruned voxel mask and the key voxel mask are written into a bitstream; the pruned voxel mask is used to determine the position of the pruned voxel in the empty voxel grid, and the key voxel mask is used to determine the position of the key voxel in the empty voxel grid.

[0046] The third aspect of the present disclosure provides a voxel grid decoding device, including:

[0047] A data acquisition module, used to obtain decoding metadata, pruning mask, coarse residual and reference voxel index from the bitstream;

[0048] An empty voxel grid construction module, used to construct an empty voxel grid based on the decoded metadata; the decoded metadata includes the size, resolution and coarse quantization step of the voxel grid; the voxel grid includes a plurality of sub-voxels;

[0049] An unpruned voxel screening module, configured to screen out unpruned voxels from the plurality of sub-voxels according to the pruning mask of any sub-voxel in the empty voxel grid;

[0050] A reference voxel determination module, configured to determine a corresponding reference voxel for each unpruned voxel according to the reference voxel index;

[0051] a coarse reconstruction value determination module, configured to determine, for any unpruned voxel among a plurality of unpruned voxels, a sum of a reference voxel and a coarse residual corresponding to the unpruned voxel as a coarse reconstruction value of the unpruned voxel;

[0052] The dequantization module is used to dequantize the coarse reconstruction value of each of the unpruned voxels using the coarse quantization step size to obtain a first voxel value of the unpruned voxel.

[0053] A fourth aspect of the present disclosure provides a voxel grid encoding device, including:

[0054] A first voxel grid generation module is used to generate a first voxel grid according to images of multiple perspectives; the first voxel grid is used to represent the three-dimensional structure of the current scene; each first voxel in the first voxel grid is used to represent radiation information at different positions;

[0055] A pruning module, configured to prune pruned voxels in the first voxel grid to obtain a second voxel grid; the pruned voxels refer to first voxels whose importance scores are less than a first importance score threshold, and any importance score is used to characterize a contribution value of the first voxel to an image rendering result;

[0056] a reference voxel determination module, configured to select, for any second voxel in the second voxel grid, a reference voxel corresponding to the second voxel, wherein the reference voxel refers to a second voxel in the second voxel grid that is adjacent to the second voxel and has the smallest absolute error sum;

[0057] The encoding module is used to encode the second voxel grid based on the plurality of second voxels and a plurality of reference voxels corresponding one-to-one to the plurality of second voxels.

[0058] An embodiment of the fifth aspect of the present disclosure provides an electronic device, which includes a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the voxel grid decoding method described in the first aspect or the voxel grid encoding method described in the second aspect by executing the computer instructions.

[0059] An embodiment of the sixth aspect of the present disclosure provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the voxel grid decoding method described in the first aspect or the voxel grid encoding method described in the second aspect.

[0060] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will become apparent from the following description or be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] By reading the detailed description of the preferred embodiment below, various other advantages and benefits will become clear to those of ordinary skill in the art. The accompanying drawings are only used for the purpose of illustrating the preferred embodiment and are not considered to be limitations of the present disclosure. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings.

[0062] In the attached picture:

[0063] Figure 1 A schematic flow chart of a voxel grid decoding method provided by an embodiment of the present disclosure is shown;

[0064] Figure 2 A schematic diagram of a process of decoding and encoding a voxel grid provided by an embodiment of the present disclosure is shown;

[0065] Figure 3 A schematic flow chart of a voxel grid encoding method provided by an embodiment of the present disclosure is shown;

[0066] Figure 4 A schematic structural diagram of a voxel grid decoding device provided by an embodiment of the present disclosure is shown;

[0067] Figure 5 A schematic structural diagram of a voxel grid encoding device provided by an embodiment of the present disclosure is shown;

[0068] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown;

[0069] Figure 7 A schematic diagram of a storage medium provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0070] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0071] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in the present disclosure should have the common meanings understood by those skilled in the art to which the present disclosure belongs.

[0072] The following describes the technical scenarios involved in the embodiments of the present disclosure.

[0073] Generating models representing three-dimensional (3D) scenes and publishing them as photos on the Internet will bring profound changes to entertainment and future media. With the development of virtual reality technology, this issue has attracted more and more attention.

[0074] Neural Radiance Field (NeRF) technology can reconstruct 3D scenes from sparse views and then render realistic pictures from arbitrary viewpoints. Early neural radiance fields used neural grids for implicit 3D scene modeling. However, these methods suffer from high computational complexity, resulting in heavy training and rendering burdens. Recent research has established a promising direction to accelerate training and rendering by learning voxel-based radiance field representations. Voxel representations have great potential in alleviating computational complexity challenges. However, the explicit voxel grid EVG representation generates additional huge parameters, which are much larger than the neural grid representation. The huge number of parameters of the voxel grid brings heavy storage and computational burdens, hindering further development and practical applications.

[0075] To address these challenges, recent studies have applied data compression methods to voxel-based neural radiance fields, including importance pruning, frequency domain transforms, and vector quantization. While effective, these methods rarely take into account the spatial correlation of voxel data. In addition, due to the complexity of modeling the encoding bit rate in these coding frameworks, these methods rarely consider compression issues when training voxel grids.

[0076] In view of the above problems, the present invention proposes a decoding method of a voxel grid for compressing a voxel radiation field and a coding method of a voxel grid. The method can make full use of spatial correlation to achieve a higher compression ratio. In addition, the present invention proposes a rate-distortion loss function to jointly optimize rendering distortion and encoding bit rate.

[0077] According to an embodiment of the present disclosure, an embodiment of a method for decoding a voxel grid is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0078] In this embodiment, a voxel grid decoding method is provided, which is applied to the decoding side. Figure 1 is a flowchart of a method for decoding a voxel grid according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:

[0079] Step S101 , obtaining decoding metadata, pruning mask, coarse residual and reference voxel index from a bitstream.

[0080] Among them, the decoding metadata includes but is not limited to the size (i.e., length, width, height), resolution, coarse quantization step and fine quantization step of the empty voxel grid, and the empty voxel grid can be constructed by decoding the metadata; the pruning mask can also be understood as a pruning placeholder code, and the unpruned voxels can be filtered out from multiple sub-voxels of the empty voxel grid through the pruning mask; the reference voxel index includes multiple index values ​​of unpruned voxels, and the reference voxel corresponding to each index value, and the reference voxel index can be used to determine the corresponding reference voxel for each unpruned voxel.

[0081] Step S102: construct an empty voxel grid based on the decoded metadata.

[0082] Specifically, an empty voxel grid including a plurality of sub-voxels may be constructed according to the size, resolution and coarse quantization step of the empty voxel grid.

[0083] More specifically, after constructing an empty voxel grid, each sub-voxel in the empty voxel grid can be processed by a pruning mask, a coarse residual and a reference voxel index to obtain a first voxel grid; the first voxel grid is used to characterize the three-dimensional structure of the current scene; each first voxel in the first voxel grid is used to characterize radiation information at different positions.

[0084] Step S103 : for any sub-voxel in the empty voxel grid, filter out unpruned voxels from the multiple sub-voxels using the pruning mask of the sub-voxel.

[0085] In some specific embodiments, the above step S103 includes step S1031-step S1032:

[0086] Step S1031: if the pruning mask of the sub-voxel is the first value, determining the sub-voxel as a pruned voxel;

[0087] Step S1032: If the pruning mask of the sub-voxel is the second value, determine that the sub-voxel is an unpruned voxel.

[0088] In the above steps S1031 to S1032, the first value and the second value can be set arbitrarily and are not specifically limited here. For example, when the pruning mask of a sub-voxel is 0, the sub-voxel is a pruned voxel; conversely, when the pruning mask of the sub-voxel is 1, the sub-voxel is an unpruned voxel.

[0089] Step S104: determining a corresponding reference voxel for each unpruned voxel according to the reference voxel index.

[0090] In some specific embodiments, the above step S104 includes step S1041:

[0091] Step S1041 : for any unpruned voxel among the plurality of unpruned voxels, based on the reference voxel index, determining a reference voxel corresponding to the index value of the unpruned voxel.

[0092] The above step S1041 is described with an example:

[0093] The reference voxel index includes A1--B1, A2--B2, ..., An--Bn, wherein A1--B1 represents the index value A1 of the unpruned voxel 1 and the reference voxel B1 corresponding to A1. When the index value of the unpruned voxel is A2, the corresponding reference voxel B2 can be determined by the above reference voxel index.

[0094] Step S105 : for any unpruned voxel among the plurality of unpruned voxels, determine the sum of the reference voxel and the rough residual corresponding to the unpruned voxel as the rough reconstruction value of the unpruned voxel.

[0095] Step S106 : Dequantize the coarse reconstruction value of each of the unpruned voxels using the coarse quantization step size to obtain a first voxel value of the unpruned voxel.

[0096] The size of the coarse quantization step can be set according to actual conditions and is not specifically limited here.

[0097] In the above steps S105-S106, the embodiment of the present disclosure determines the coarse reconstruction value of the unpruned voxel by referring to the sum of the voxel and the coarse residual, and then dequantizes it, which helps to well restore the first voxel grid generated by images from multiple perspectives, and can ensure the accuracy and completeness of the voxel grid restoration.

[0098] In some specific embodiments, after the above step S106, the method further includes:

[0099] Step S107, selecting key voxels from a plurality of unpruned voxels.

[0100] In some specific embodiments, the above step S107 includes step S1071-step S1072:

[0101] Step S1071, obtaining a key voxel mask from the bitstream.

[0102] Step S1072: for any unpruned voxel, if the key voxel mask of the unpruned voxel is a preset mask value, the unpruned voxel is determined to be a key voxel.

[0103] In step S1071-step S1072, whether the unpruned voxel is a key voxel can be determined by whether the key voxel mask of the unpruned voxel is a preset mask value. For example, when the key voxel mask of the unpruned voxel is 1, the unpruned voxel is a key voxel; conversely, when the key voxel mask of the unpruned voxel is 0, the unpruned voxel is a non-key voxel.

[0104] Step S108 : for any key voxel among the multiple key voxels, use the fine quantization step size to dequantize the fine residual of the key voxel to obtain a reconstructed residual.

[0105] The size of the fine quantization step can be set according to actual conditions and is not specifically limited here.

[0106] Step S109: taking the sum of the first voxel value and the reconstruction residual as the fine reconstruction value of the key voxel.

[0107] Specifically, the embodiment of the present disclosure dequantizes the fine residual of the key voxel by using a fine quantization step to obtain a reconstruction residual; then the sum of the first voxel value and the reconstruction residual is used as the fine reconstruction value of the key voxel, which helps to better restore the first voxel grid generated by images from multiple perspectives, greatly ensuring the accuracy and completeness of the voxel grid restoration.

[0108] In some specific embodiments, the present disclosure provides a method for decoding a voxel grid, the process of which is as follows: Figure 2 The "blank grid → pruning → selection of reference index → ​​coarse reconstruction → fine reconstruction" in the spatial prediction coding framework is used to obtain the first voxel grid used to characterize the three-dimensional structure of the current scene. This decoding method can well alleviate the problem of high computational complexity.

[0109] According to an embodiment of the present disclosure, an embodiment of a method for encoding a voxel grid is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0110] In this embodiment, a voxel grid encoding method is provided, which is applied to the encoding side. Figure 3 is a flowchart of a method for encoding a voxel grid according to an embodiment of the present disclosure, such as Figure 3 As shown, the process includes the following steps:

[0111] Step S301 : generating a first voxel grid according to images of multiple viewing angles.

[0112] Specifically, the first voxel grid can be understood as a voxel radiation field, which is used to characterize the three-dimensional structure of the current scene. Each first voxel in the first voxel grid is used to characterize radiation information at different positions, including but not limited to color information, density information, illumination information, transparency information, reflectivity information, and other physical property information, so as to reconstruct the geometric shape of the scene.

[0113] In some specific embodiments, the first voxel grid may be generated through steps S3011 to S3013:

[0114] Step S3011, randomly initialize the voxel grid and neural grid.

[0115] Step S3012, according to the input camera posture, several rays are emitted through each image pixel position, and uniform sampling is performed on the rays. For each sampling point, linear interpolation is performed on the voxel grid to obtain the color feature value and volume density value. The color feature value is input into the neural grid to calculate the color value, and the corresponding image is rendered according to the volume rendering formula.

[0116]

[0117] 0=alpha(0i,0i)=1-exp(-0i0i)

[0118]

[0119] Among them, α i represents the probability that the light stops at point i, T i represents the loss of light transmitted from point i to the camera, σ i is the volume density value at point i, δ i is the distance between point i and the adjacent sampling point, c bg Indicates the background color, c i Represents the color at point i.

[0120] Step S3013, calculate the loss function according to the distortion between the rendered image and the real image (such as peak signal-to-noise ratio PSNR and / or structural similarity SSIM), and then calculate the gradient of the loss function with respect to the color feature value, volume density value, and neural grid parameter, and use the stochastic gradient descent method to iterate multiple times to obtain the final voxel model.

[0121] Step S302: Prune the pruned voxels in the first voxel grid to obtain a second voxel grid.

[0122] Specifically, the pruned voxel refers to the first voxel whose importance score is less than the first importance score threshold, and any importance score is used to represent the contribution value of the first voxel to the image rendering result. Among them, since the volume density determines the visibility of a specific voxel, the volume density value of the voxel can be used as the importance score.

[0123] In some specific embodiments, before the above step S302, the method further includes step S3021:

[0124] Step S3021: Filter out pruned voxels and key voxels from the plurality of first voxels.

[0125] Specifically, the pruned voxel is used to represent the first voxel whose importance score is less than the first importance score threshold, and the key voxel is used to represent the first voxel whose importance score is greater than the second importance score threshold, and the second importance score threshold is greater than the first importance score threshold. The first importance score threshold and the second importance score threshold can be set by the user according to actual conditions and are not specifically limited here.

[0126] More specifically, in step S3021, a corresponding pruning mask may be set for the pruned voxel, so that in the decoding stage, whether the voxel is a pruned voxel can be determined according to the pruning mask, for example, if the pruning mask of the voxel is a first value, the voxel is determined to be a pruned voxel. A corresponding key voxel mask may also be set for the key voxel, so that in the decoding stage, whether the voxel is a key voxel can be determined according to the key voxel mask, for example, if the key voxel mask of the voxel is a preset value, the voxel is determined to be a key voxel.

[0127] Step S303: for any second voxel in the second voxel grid, select a reference voxel corresponding to the second voxel.

[0128] Specifically, the reference voxel refers to a second voxel in the second voxel grid that is adjacent to the second voxel and has the smallest absolute error sum.

[0129] In some specific embodiments, the above step S303 includes steps S3031 to S3033:

[0130] Step S3031 : for any second voxel in the second voxel grid, calculate a first sum of absolute errors between the second voxel and each encoded adjacent second voxel.

[0131] Specifically, each second voxel can be used as a high-dimensional vector, and the first absolute difference (Sum of Absolute Difference, SAD) between the second voxel and each encoded adjacent second voxel can be understood as: the SAD of two high-dimensional vectors. The specific calculation method of SAD can be determined by existing technology and is not specifically limited here.

[0132] Step S3032: determine the encoded adjacent second voxel with the smallest sum of the first absolute error as the reference voxel corresponding to the second voxel.

[0133] Step S3033: construct a reference voxel index based on the multiple second voxels and the reference voxel corresponding to each second voxel.

[0134] Specifically, the reference voxel index includes index values ​​of multiple second voxels and adjacent positions corresponding to each index value; for example: when the index value of second voxel 1 is a first value, the voxel at the first adjacent position is selected as the reference voxel of second voxel 1; when the index value of second voxel 1 is a second value, the voxel at the second adjacent position is selected as the reference voxel of second voxel 1.

[0135] Step S304 : encoding the second voxel grid based on the plurality of second voxels and a plurality of reference voxels corresponding one-to-one to the plurality of second voxels.

[0136] Specifically, in the process of encoding the second voxel grid, progressive encoding is mainly adopted, that is, the second voxel grid is encoded at multiple levels, for example Figure 2 As shown, fine tuning of rate-distortion optimization--coarse quantization--fine tuning of rate-distortion optimization--fine quantization.

[0137] The disclosed embodiment can remove voxels with little impact on image rendering by pruning the pruned voxels in the first voxel grid, thereby improving the compression rate without significantly affecting the visual quality; preferably, the disclosed embodiment selects reference voxels corresponding to the second voxels, encodes the second voxel grid based on multiple second voxels and multiple reference voxels corresponding one-to-one to the multiple second voxels, thereby making full use of spatial correlation and achieving a higher compression ratio.

[0138] In some specific embodiments, the above step S304 further includes steps a1 to a3:

[0139] Step a1, calculating the second absolute error sum between each second voxel and its corresponding reference voxel, and obtaining a plurality of second absolute error sums corresponding one-to-one to the plurality of second voxels.

[0140] Step a2: taking the average value of the plurality of second absolute error sums as a first weighting coefficient, and using the first weighting coefficient to perform weighted processing on the first loss function of the first voxel grid to obtain a second loss function.

[0141] Step a3: adjusting the second voxel grid by using the second loss function to obtain a third voxel grid; the third voxel grid includes a plurality of third voxels.

[0142] Specifically, the second loss function can be used as a rate-true loss function for fine-tuning the second voxel grid.

[0143] The disclosed embodiment uses the average value of multiple second absolute error sums as the first weighting coefficient, and uses the first weighting coefficient to perform weighted processing on the first loss function to obtain a second loss function; and adjusts the second voxel grid through the second loss function to obtain a third voxel grid, which can achieve the following technical effects, namely: the fine-tuned grid can more effectively represent key information during compression, thereby improving the compression rate while maintaining visual quality; through weighted processing, it can ensure that more visually important features are retained during the compression process, reducing the impact on visual quality.

[0144] In some specific embodiments, after the above step a3, the method further includes:

[0145] Step b1, quantizing the plurality of third voxels respectively using a coarse quantization step size to obtain a plurality of quantized third voxels.

[0146] Specifically, the quantization method generally uses uniform quantization, and the coarse quantization step size is a larger quantization step size, which can be set according to actual conditions and is not specifically limited here.

[0147] Step b2: subtract each quantized third voxel from its corresponding reference voxel to obtain a corresponding rough residual.

[0148] Step b3, according to the coarse residual corresponding to each quantized third voxel, dequantize each quantized third voxel to obtain a coarse reconstructed voxel grid corresponding to the third voxel grid; the coarse reconstructed voxel grid includes a plurality of fourth voxels.

[0149] The disclosed embodiment obtains a corresponding coarse residual by subtracting each quantized third voxel from its corresponding reference voxel; and dequantizes each quantized third voxel according to the coarse residual corresponding to each quantized third voxel to obtain a coarse reconstructed voxel grid corresponding to the third voxel grid, so that only the voxels with large changes can be finely processed, and the compression ratio of the grid can be improved without sacrificing too much visual quality.

[0150] In some specific embodiments, after the above step b3, the method further includes steps c1 to c5:

[0151] Step c1, selecting a plurality of key voxels from the plurality of fourth voxels.

[0152] Specifically, the key voxel refers to a fourth voxel among multiple fourth voxels whose importance score is greater than the second importance score threshold.

[0153] Step c2: for any one of the plurality of key voxels, determine a target second voxel having the same position as the key voxel from the second voxel grid.

[0154] Step c3, calculating the third absolute error sum between the key voxel and the target second voxel, and obtaining a plurality of third absolute error sums corresponding to the plurality of key voxels one by one.

[0155] Step c4: taking the average value of the plurality of third absolute error sums as a second weighting coefficient, and using the second weighting coefficient to perform weighted processing on the first loss function to obtain a third loss function.

[0156] Step c5, adjusting the multiple key voxels in the coarsely reconstructed voxel grid by using the third loss function to obtain a fourth voxel grid; the fourth voxel grid includes the multiple adjusted key voxels.

[0157] Specifically, the third loss function can be used as a rate-distortion loss function to fine-tune all key voxels.

[0158] The disclosed embodiment can improve the compression efficiency of the grid by screening key voxels and focusing on the compression of these voxels; and because only the screened key voxels need to be processed in detail, the overall calculation amount is reduced and the calculation efficiency is improved.

[0159] In some specific embodiments, after step c5, the method further comprises:

[0160] Step d1: for any adjusted key voxel among the plurality of adjusted key voxels, subtract the adjusted key voxel from a second voxel at the same position in the second voxel grid.

[0161] Step d2, quantizing the subtraction result to obtain a fine residual.

[0162] Specifically, in the quantization process, a fine quantization step size may be used for uniform quantization, wherein the fine quantization step size may be a smaller quantization step size, and may be set according to actual conditions, and is not specifically limited here.

[0163] The disclosed embodiment can retain the detailed changes of key voxels during the adjustment process by calculating fine residuals, thereby avoiding the loss of important data during the mesh compression process; during the decompression and reconstruction process, the use of fine residuals can reduce reconstruction errors and improve the quality of the final reconstructed mesh.

[0164] In some specific embodiments, after step d2, the method further comprises:

[0165] The coarse residual, the fine residual, the reference voxel index, the pruned voxel mask and the key voxel mask are written into a bitstream.

[0166] The pruned voxel mask is used to determine the position of the pruned voxel in the empty voxel grid, and the key voxel mask is used to determine the position of the key voxel in the empty voxel grid.

[0167] Corresponding to the implementation of the above voxel grid decoding method, the embodiment of the present disclosure also provides a voxel grid decoding device, which is used to execute the voxel grid encoding method described in the above embodiment. Figure 4 As shown, the decoding device of the voxel grid includes:

[0168] A data acquisition module, used to obtain decoding metadata, pruning mask, coarse residual and reference voxel index from the bitstream;

[0169] An empty voxel grid construction module, used to construct an empty voxel grid based on the decoded metadata; the decoded metadata includes the size, resolution and coarse quantization step of the voxel grid; the voxel grid includes a plurality of sub-voxels;

[0170] An unpruned voxel screening module, configured to screen out unpruned voxels from the plurality of sub-voxels according to the pruning mask of any sub-voxel in the empty voxel grid;

[0171] A reference voxel determination module, configured to determine a corresponding reference voxel for each unpruned voxel according to the reference voxel index;

[0172] a coarse reconstruction value determination module, configured to determine, for any unpruned voxel among a plurality of unpruned voxels, a sum of a reference voxel and a coarse residual corresponding to the unpruned voxel as a coarse reconstruction value of the unpruned voxel;

[0173] The dequantization module is used to dequantize the coarse reconstruction value of each of the unpruned voxels using the coarse quantization step size to obtain a first voxel value of the unpruned voxel.

[0174] Optionally, the unpruned voxel screening module is further used to: determine that the sub-voxel is a pruned voxel if the pruning mask of the sub-voxel is a first value; and determine that the sub-voxel is an unpruned voxel if the pruning mask of the sub-voxel is a second value.

[0175] Optionally, the reference voxel index includes index values ​​of multiple unpruned voxels and a reference voxel corresponding to each index value; the reference voxel determination module is further used to: for any one of the multiple unpruned voxels, determine the reference voxel corresponding to the index value of the unpruned voxel based on the reference voxel index.

[0176] Optionally, the device also includes: a fine reconstruction module, which is used to screen out key voxels from multiple unpruned voxels after obtaining the first voxel value of the unpruned voxel; for any one of the multiple key voxels, use a fine quantization step to dequantize the fine residual of the key voxel to obtain a reconstruction residual; and use the sum of the first voxel value and the reconstruction residual as the fine reconstruction value of the key voxel.

[0177] Optionally, the fine reconstruction module is further used to: obtain a key voxel mask from the bitstream; for any unpruned voxel, if the key voxel mask of the unpruned voxel is a preset mask value, determine that the unpruned voxel is a key voxel.

[0178] The voxel grid decoding device provided in the above-mentioned embodiment of the present disclosure and the voxel grid decoding method provided in the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0179] Corresponding to the implementation of the above voxel grid encoding method, the embodiment of the present disclosure further provides a voxel grid encoding device, which is used to execute the voxel grid encoding method described in the above embodiment. Figure 5 As shown, the encoding device of the voxel grid includes:

[0180] A first voxel grid generation module is used to generate a first voxel grid according to images of multiple perspectives; the first voxel grid is used to represent the three-dimensional structure of the current scene; each first voxel in the first voxel grid is used to represent radiation information at different positions;

[0181] A pruning module, configured to prune pruned voxels in the first voxel grid to obtain a second voxel grid; the pruned voxels refer to first voxels whose importance scores are less than a first importance score threshold, and any importance score is used to characterize a contribution value of the first voxel to an image rendering result;

[0182] a reference voxel determination module, configured to select, for any second voxel in the second voxel grid, a reference voxel corresponding to the second voxel, wherein the reference voxel refers to a second voxel in the second voxel grid that is adjacent to the second voxel and has the smallest absolute error sum;

[0183] The encoding module is used to encode the second voxel grid based on the plurality of second voxels and a plurality of reference voxels corresponding one-to-one to the plurality of second voxels.

[0184] Optionally, the device also includes: a voxel screening module, used to screen out pruned voxels and key voxels from the multiple first voxels before pruning the pruned voxels in the first voxel grid; the pruned voxels are used to represent first voxels whose importance scores are less than a first importance score threshold, and the key voxels are used to represent first voxels whose importance scores are greater than a second importance score threshold, and the second importance score threshold is greater than the first importance score threshold.

[0185] Optionally, the reference voxel determination module is also used to: calculate, for any second voxel in the second voxel grid, the sum of the first absolute errors between the second voxel and each encoded adjacent second voxel; determine the encoded adjacent second voxel with the smallest sum of the first absolute errors as the reference voxel corresponding to the second voxel; construct a reference voxel index based on multiple second voxels and the reference voxel corresponding to each second voxel; the reference voxel index includes multiple reference voxels corresponding one-to-one to the multiple second voxels.

[0186] Optionally, the encoding module is also used to: calculate the second absolute error sum between each second voxel and its corresponding reference voxel to obtain multiple second absolute error sums corresponding to the multiple second voxels one by one; use the average value of the multiple second absolute error sums as the first weighting coefficient, and use the first weighting coefficient to weight the first loss function of the first voxel grid to obtain a second loss function; adjust the second voxel grid by the second loss function to obtain a third voxel grid; the third voxel grid includes multiple third voxels; use a coarse quantization step to quantize the multiple third voxels respectively to obtain multiple quantized third voxels; subtract each quantized third voxel from its corresponding reference voxel to obtain the corresponding coarse residual; according to the coarse residual corresponding to each quantized third voxel, dequantize each quantized third voxel to obtain a coarse reconstructed voxel grid corresponding to the third voxel grid; the coarse reconstructed voxel grid includes multiple fourth voxels; screen out multiple key points from the multiple fourth voxels voxel; the key voxel refers to a fourth voxel whose importance score is greater than the second importance score threshold among multiple fourth voxels; for any one of the multiple key voxels, determine a target second voxel with the same position as the key voxel from the second voxel grid; calculate the third absolute error sum between the key voxel and the target second voxel to obtain multiple third absolute error sums corresponding to the multiple key voxels one by one; use the average value of the multiple third absolute error sums as the second weighting coefficient, and use the second weighting coefficient to weight the first loss function to obtain a third loss function; adjust the multiple key voxels in the coarsely reconstructed voxel grid by the third loss function to obtain a fourth voxel grid; the fourth voxel grid includes multiple adjusted key voxels; for any one of the multiple adjusted key voxels, subtract the adjusted key voxel from the second voxel at the same position in the second voxel grid; quantize the subtraction result to obtain a fine residual.

[0187] Optionally, the device also includes a data writing module for writing the coarse residual, the fine residual, the reference voxel index, the pruned voxel mask and the key voxel mask into the code stream; the pruned voxel mask is used to determine the position of the pruned voxel in the empty voxel grid, and the key voxel mask is used to determine the position of the key voxel in the empty voxel grid.

[0188] The voxel grid encoding device provided in the above-mentioned embodiment of the present disclosure and the voxel grid encoding method provided in the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0189] The present disclosure also provides an electronic device to perform the above-mentioned voxel grid encoding method or voxel grid decoding method. Figure 6 , which shows a schematic diagram of an electronic device provided by some embodiments of the present disclosure. Figure 6 As shown, the electronic device 6 includes: a processor 600, a memory 601, a bus 602 and a communication interface 603, wherein the processor 600, the communication interface 603 and the memory 601 are connected via the bus 602; the memory 601 stores a computer program that can be run on the processor 600, and when the processor 600 runs the computer program, it executes the voxel grid encoding method or the voxel grid decoding method provided in the aforementioned embodiment of the present disclosure.

[0190] The memory 601 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 603 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0191] The bus 602 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 601 is used to store a program, and the processor 600 executes the program after receiving an execution instruction. The voxel grid encoding method or the voxel grid decoding method disclosed in the above-mentioned embodiment may be applied to the processor 600, or implemented by the processor 600.

[0192] The processor 600 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 600. The above processor 600 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 601, and the processor 600 reads the information in the memory 601 and completes the steps of the above method in combination with its hardware.

[0193] The electronic device provided by the embodiment of the present disclosure and the voxel grid encoding method or voxel grid decoding method provided by the embodiment of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.

[0194] The present disclosure also provides a computer-readable storage medium corresponding to the voxel grid encoding method or voxel grid decoding method provided in the above embodiments. Figure 7 The computer-readable storage medium shown is a CD 30 on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the voxel grid encoding method or the voxel grid decoding method provided by any of the aforementioned embodiments.

[0195] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0196] The computer-readable storage medium provided by the above-mentioned embodiments of the present disclosure and the voxel grid encoding method or voxel grid decoding method provided by the embodiments of the present disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0197] It should be noted that:

[0198] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present disclosure can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0199] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present disclosure, the various features of the present disclosure are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed disclosure requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims below, the inventive aspects lie in less than all the features of the single embodiment disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present disclosure.

[0200] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of this disclosure and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0201] The above is only a preferred specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for decoding a voxel grid, characterized in that: The method comprises: Get decoding metadata, pruning mask, coarse residual and reference voxel index from the bitstream; constructing an empty voxel grid based on the decoded metadata; the decoded metadata including the size, resolution and coarse quantization step of the voxel grid; the voxel grid including a plurality of sub-voxels; For any sub-voxel in the empty voxel grid, filter out unpruned voxels from the multiple sub-voxels by using the pruning mask of the sub-voxel; Determine a corresponding reference voxel for each unpruned voxel according to the reference voxel index; For any unpruned voxel among the plurality of unpruned voxels, determining the sum of the reference voxel and the rough residual corresponding to the unpruned voxel as the rough reconstruction value of the unpruned voxel; The coarse reconstruction value of each of the unpruned voxels is dequantized using the coarse quantization step size to obtain a first voxel value of the unpruned voxel.

2. The method according to claim 1, characterized in that Filtering out unpruned voxels from the plurality of sub-voxels by using the pruning mask of the sub-voxel comprises: If the pruning mask of the sub-voxel is a first value, determining the sub-voxel as a pruned voxel; If the pruning mask of the sub-voxel is a second value, the sub-voxel is determined to be an unpruned voxel.

3. The method according to claim 1 or 2, characterized in that: The reference voxel index includes index values ​​of a plurality of unpruned voxels and a reference voxel corresponding to each index value; According to the reference voxel index, determining a corresponding reference voxel for each unpruned voxel includes: For any unpruned voxel among the plurality of unpruned voxels, a reference voxel corresponding to the index value of the unpruned voxel is determined based on the reference voxel index.

4. The method according to claim 1 or 2, characterized in that: After obtaining the first voxel value of the unpruned voxel, the method further includes: Select key voxels from multiple unpruned voxels; For any key voxel among the multiple key voxels, dequantize the fine residual of the key voxel using a fine quantization step to obtain a reconstruction residual; The sum of the first voxel value and the reconstruction residual is used as the fine reconstruction value of the key voxel.

5. The method according to claim 4, characterized in that The key voxels are screened out from multiple unpruned voxels, including: Obtaining a key voxel mask from the bitstream; For any unpruned voxel, if the key voxel mask of the unpruned voxel is a preset mask value, the unpruned voxel is determined to be a key voxel.

6. A method for encoding a voxel grid, characterized in that: The method comprises: Generate a first voxel grid according to images from multiple perspectives; the first voxel grid is used to represent the three-dimensional structure of the current scene; each first voxel in the first voxel grid is used to represent radiation information at different positions; Pruning pruned voxels in the first voxel grid to obtain a second voxel grid; the pruned voxels refer to first voxels whose importance scores are less than a first importance score threshold, and any importance score is used to represent a contribution value of the first voxel to an image rendering result; For any second voxel in the second voxel grid, select a reference voxel corresponding to the second voxel, the reference voxel being a second voxel in the second voxel grid that is adjacent to the second voxel and has the smallest absolute error sum; The second voxel grid is encoded based on a plurality of second voxels and a plurality of reference voxels corresponding one-to-one to the plurality of second voxels.

7. The method according to claim 6, characterized in that Encoding the second voxel grid based on a plurality of second voxels and a plurality of reference voxels corresponding one-to-one to the plurality of second voxels comprises: Calculating a second absolute error sum between each second voxel and its corresponding reference voxel to obtain a plurality of second absolute error sums corresponding one-to-one to the plurality of second voxels; taking an average value of the plurality of second absolute error sums as a first weighting coefficient, and using the first weighting coefficient to perform weighted processing on the first loss function of the first voxel grid to obtain a second loss function; The second voxel grid is adjusted by the second loss function to obtain a third voxel grid; the third voxel grid includes a plurality of third voxels; quantizing the plurality of third voxels respectively using a coarse quantization step size to obtain a plurality of quantized third voxels; Subtract each quantized third voxel from its corresponding reference voxel to obtain the corresponding rough residual; According to the coarse residual corresponding to each quantized third voxel, dequantize each quantized third voxel to obtain a coarse reconstructed voxel grid corresponding to the third voxel grid; the coarse reconstructed voxel grid includes a plurality of fourth voxels; Screening out a plurality of key voxels from the plurality of fourth voxels; the key voxels refer to fourth voxels whose importance scores are greater than a second importance score threshold among the plurality of fourth voxels; For any one of the plurality of key voxels, determining a target second voxel having the same position as the key voxel from the second voxel grid; Calculating a third absolute error sum between the key voxel and the target second voxel to obtain a plurality of third absolute error sums corresponding to the plurality of key voxels one by one; Taking an average value of the plurality of third absolute error sums as a second weighting coefficient, and using the second weighting coefficient to perform weighted processing on the first loss function to obtain a third loss function; Adjusting a plurality of key voxels in the coarsely reconstructed voxel grid by using the third loss function to obtain a fourth voxel grid; the fourth voxel grid includes a plurality of adjusted key voxels; For any one of the plurality of adjusted key voxels, subtract the adjusted key voxel from a second voxel at the same position in the second voxel grid; The subtraction result is quantized to obtain a fine residual; The coarse residual, the fine residual, the reference voxel index, the pruned voxel mask and the key voxel mask are written into a bitstream; the pruned voxel mask is used to determine the position of the pruned voxel in the empty voxel grid, and the key voxel mask is used to determine the position of the key voxel in the empty voxel grid.

8. A voxel grid decoding device, characterized in that: The device comprises: A data acquisition module, used to obtain decoding metadata, pruning mask, coarse residual and reference voxel index from the bitstream; An empty voxel grid construction module, used to construct an empty voxel grid based on the decoded metadata; the decoded metadata includes the size, resolution and coarse quantization step of the voxel grid; the voxel grid includes a plurality of sub-voxels; An unpruned voxel screening module, configured to screen out unpruned voxels from the plurality of sub-voxels according to the pruning mask of any sub-voxel in the empty voxel grid; A reference voxel determination module, configured to determine a corresponding reference voxel for each unpruned voxel according to the reference voxel index; a coarse reconstruction value determination module, configured to determine, for any unpruned voxel among a plurality of unpruned voxels, a sum of a reference voxel and a coarse residual corresponding to the unpruned voxel as a coarse reconstruction value of the unpruned voxel; The dequantization module is used to dequantize the coarse reconstruction value of each of the unpruned voxels using the coarse quantization step size to obtain a first voxel value of the unpruned voxel.

9. A voxel grid encoding device, characterized in that The device comprises: A first voxel grid generation module is used to generate a first voxel grid according to images of multiple perspectives; the first voxel grid is used to represent the three-dimensional structure of the current scene; each first voxel in the first voxel grid is used to represent radiation information at different positions; A pruning module, configured to prune pruned voxels in the first voxel grid to obtain a second voxel grid; the pruned voxels refer to first voxels whose importance scores are less than a first importance score threshold, and any importance score is used to characterize a contribution value of the first voxel to an image rendering result; a reference voxel determination module, configured to select, for any second voxel in the second voxel grid, a reference voxel corresponding to the second voxel, wherein the reference voxel refers to a second voxel in the second voxel grid that is adjacent to the second voxel and has the smallest absolute error sum; The encoding module is used to encode the second voxel grid based on the plurality of second voxels and a plurality of reference voxels corresponding one-to-one to the plurality of second voxels.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

Images