Depth map and normal map file compression method and system in multi-view three-dimensional reconstruction

By separating the integer and decimal parts of the depth map and normal map and using JPEG or PNG compression technology, the problem of storage space occupation in multi-view three-dimensional reconstruction is solved, and efficient compression of the depth map and normal map is achieved, reducing the Disk capacity requirements improve the practicality of the technology.

CN118984374APending Publication Date: 2024-11-19BEIJING MANMAN STAR TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411100078.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In multi-view three-dimensional reconstruction technology, the storage space of depth maps and normal maps occupies a huge amount of storage space, and the existing technology fails to effectively compress, resulting in high requirements for disk capacity and cost.

Method used

By separating the integer and decimal parts of the depth map and the normal map, and compressing them using JPEG or PNG compression technology, the specific steps include directly compressing the integer parts of the depth map, discretizing the decimal parts and then compressing them. , the components of the normal graph are discretized and compressed, and a quantization table and encoding algorithm are combined to improve compression efficiency.

Benefits of technology

The storage space size of depth maps and normal maps is significantly reduced, the disk capacity requirements are reduced, the practicality of multi-view 3D reconstruction technology is improved, and the compression effect is verified through experiments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984374A_ABST
    Figure CN118984374A_ABST
Patent Text Reader

Abstract

The invention provides a depth map and normal map file compression method and system in multi-view three-dimensional reconstruction. Comprising the following steps: reading a depth map and a normal map, and separating an integer part and a decimal part of the depth map; for an integer part of the depth map, directly adopting a JPEG (Joint Photographic Experts Group) or PNG compression technology to compress; for the decimal part of the depth map, discretizing the decimal part by adopting an 8-bit discretization or 16-bit discretization method, and then compressing the discretized two-dimensional map by adopting a JPEG (Joint Photographic Experts Group) or PNG (Portable Network Gateway) compression technology; discretizing three components of a normal vector in the normal diagram by adopting an 8-bit discretization method; and compressing a two-dimensional diagram formed by the three discretized components by adopting a JPEG (Joint Photographic Experts Group) or PNG (Portable Network Gateway) compression technology. According to the method, the storage space size of the depth map and the normal map is remarkably reduced, the requirement for the capacity of a disk is lowered, the disk cost is reduced, the practicability of a multi-view three-dimensional reconstruction technology is improved, and the effect of the method is verified through experiments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a multi-view Figure 3 Method and system for compressing depth map and normal map files in 3D reconstruction. Background Art

[0002] In Multiview Figure 3 In the 3D reconstruction technology, dense reconstruction is an important step in obtaining 3D mesh results. The strategy of dense reconstruction is to estimate the depth of each image through the depth map estimation technology, obtain the depth map of the image, and store it on disk. The depth map is a two-dimensional image of the same size or scaled proportionally to the image. Unlike the image, the depth map stores the depth value of each pixel. In order to ensure the accuracy of the calculation, the depth map is used in the multi-view. Figure 3 In 3D reconstruction technology, depth values ​​are basically stored as floating point numbers, which are represented by 32 bits or 4 bytes in computer language. Therefore, a depth map of the same size as the original image occupies 4 times the image width times the image height bytes. Taking a 20-megapixel image as an example, a depth map occupies 76.29 megabytes of space, and multi-view reconstruction usually has hundreds to tens of thousands of images. This space is very huge, which places high demands on disk capacity and cost.

[0003] Simultaneous and deep Figure 1 Also stored are the normal map and depth Figure 1 It is also a two-dimensional image of the same size or scaled proportionally to the image. The normal map stores the three-dimensional normal vector of each pixel. Figure 3 In 3D reconstruction technology, normal vector values ​​are basically stored as 3-channel floating-point numbers, which are represented by three 32-bit or 12 bytes in computer language. Therefore, a normal map of the same size as the original image occupies 12 times the image width times the image height bytes. Taking a 20-megapixel image as an example, a normal map occupies 228.87 megabytes of space.

[0004] However, many existing 3D reconstruction methods do not specifically compress depth maps and normal maps, resulting in a waste of storage space. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention proposes a multi-view Figure 3 A method for compressing depth map and normal map files in dimensional reconstruction comprises the following steps:

[0006] Step 1, read the depth map and normal map, and separate the integer part and the decimal part of the depth map;

[0007] Step 2: For the integer part of the depth map, directly compress it using JPEG or PNG compression technology;

[0008] Step 3: For the decimal part of the depth map, firstly discretize it by using 8-bit discretization or 16-bit discretization method, and then compress the discretized two-dimensional map by using JPEG or PNG compression technology;

[0009] Step 4, discretize the three components of the normal vector in the normal map using an 8-bit discretization method;

[0010] Step 5: compress the two-dimensional graph composed of the three discretized components using JPEG or PNG compression technology.

[0011] Furthermore, the specific implementation of step 2 is as follows:

[0012] First, count the range of the depth map. For depth maps with a range less than 255, offset the depth value to 0-255 and store it in JPEG or PNG format. For depth maps with a range of 255-65535, offset the depth value to 0-65535 and store it in PNG format. For depth maps with a range exceeding 65535, no compression is performed.

[0013] Furthermore, the depth value offset method is to subtract the minimum depth value in the entire depth map from the pixel depth value.

[0014] Furthermore, the specific calculation formula of the 8-bit discretization method is:

[0015] ds = floor(df*255)

[0016] Among them, ds is the discretized value, df is the decimal part of the depth value, and floor is the floor function.

[0017] Furthermore, the specific calculation formula of the 16-bit discretization method is:

[0018] ds = floor(df*65535)

[0019] Among them, ds is the discretized value, df is the decimal part of the depth value, and floor is the floor function.

[0020] Furthermore, the three components in step 4 satisfy:

[0021] (Nx) 2 +(Ny) 2 +(Nz) 2 =1

[0022] Among them, Nx, Ny, and Nz are the components of the normal vector in the X, Y, and Z directions respectively. Therefore, only two components need to be discretized in practice, and the third component is calculated from the values ​​of the first two components according to the above formula.

[0023] Furthermore, the specific implementation process of JPEG compression technology is as follows:

[0024] First, convert the image from RGB color space to YCbCr color space. In YCbCr color space, Y represents luminance, while Cb and Cr represent chrominance, which represent the deviation of blue and red respectively. The formula for color space conversion is:

[0025] Y=0.299R+0.587G+0.114B

[0026] Cb=-0.1687R-0.3313G+0.5B+128

[0027] Cr=0.5R-0.4187G-0.0813B+128

[0028] Then the chrominance signals Cb and Cr are downsampled, the image is divided into several 8*8 pixel blocks, and each block is subjected to discrete cosine transform DCT to obtain the DCT coefficient; the formula for DCT transformation is:

[0029]

[0030] F(u,v) is the transformed coefficient, f(x,y) is the original pixel value, C(u) and C(v) are normalized coefficients, x and y refer to the spatial coordinates of the pixel in the image block, u and v are frequency variables used to represent the frequency block after the transformation;

[0031] The DCT coefficients are divided element by element by the quantization table, and the result is rounded to the nearest integer to obtain the quantization matrix. Finally, the quantization matrix is ​​further compressed through the existing encoding method, and all the processed data and necessary file header information are assembled into a JPEG format file.

[0032] Furthermore, the design process of the quantization table is as follows:

[0033] First, the entire shooting scene is classified according to the content of the captured image, and a unique quantization table is designed for each scene based on the empirical values ​​of depth map and normal map compression. In the JPEG compression process, the corresponding quantization table is selected according to the judged shooting scene. Feedback is continuously collected during the compression process, and this data is used to improve scene classification and quantization table design.

[0034] Furthermore, existing encoding algorithms include run-length coding, Huffman coding or arithmetic coding.

[0035] The present invention also provides a multi-view Figure 3 File compression system for depth and normal maps in 3D reconstruction, including:

[0036] one or more processors;

[0037] A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the above solution.

[0038] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0039] The present invention significantly reduces the storage space size of the depth map and the normal map, reduces the capacity requirement of the disk, reduces the disk cost, and improves the multi-view Figure 3 The practicability of the three-dimensional reconstruction technology is investigated, and the effect of the present invention is verified through experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is an example of the original image.

[0041] Figure 2 This is an example of a raw image depth map.

[0042] Figure 3 This is an example of a normal map of an original image.

[0043] Figure 4 This is a schematic diagram showing that the integer part of the depth value is the same.

[0044] Figure 5 This is an example image where the integer part of the depth value in the local window is exactly the same.

[0045] Figure 6 The following are example graphs showing exactly the same discrete values ​​for the normal vector components.

[0046] Figure 7 An example image of the integer portion of the depth value stored in JPEG or PNG format.

[0047] Figure 8 It is a two-dimensional graph composed of three components after the normal map is discretized.

[0048] Fig. 9 The figure is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0050] The idea of ​​the present invention is to store the integer part and the decimal part of the depth map and the normal map as two-dimensional images with a local high repetition value respectively, and compress the space of the two-dimensional image by using PNG and JPEG compression algorithms.

[0051] Theoretical explanation of the present invention:

[0052] In image processing technology, PNG and JPEG compression algorithms can greatly compress the space of two-dimensional images. The former is lossless compression, and the latter is lossy compression. They have one thing in common, which is to use Huffman coding to encode and store pixel values. Huffman coding is to assign a shorter binary code to characters with a high probability of occurrence, and a longer binary code to characters with a low probability of occurrence, so that the average encoding length of the characters is the shortest. Therefore, the more repetitive values ​​a two-dimensional image has, the higher the compression rate. For example, a pure black or pure white image can get a very high compression rate. Therefore, the depth and normal map is decomposed into a two-dimensional map with a high local repetitive value, and then applied to the PNG or JPEG compression algorithm, the volume of the image can be effectively compressed.

[0053] For floating-point depth maps, the probability of repeated depth values ​​in some parts is very small, so directly compressing with PNG or JPEG compression technology cannot achieve an ideal compression ratio. However, there is a high probability that the integer part of the depth will be repeated locally, because the depth represents the distance of the target from the camera in the Z direction of the camera coordinate system. If the target is a plane in a certain part, then the probability that the integer distance value from the local part to the camera is the same is much greater than the probability of the same floating-point depth value. Assuming that a certain plane is strictly perpendicular to the optical axis, then its depth value will be exactly the same, such as Figure 4 shown.

[0054] In actual cases, when reconstructing urban buildings from multiple drone views, there is a high probability that the integer part of the depth value of a certain building surface in the local window is exactly the same, such as Figure 5 shown.

[0055] Therefore, the integer part and the decimal part of the depth map are separated, and the integer part is rearranged into a two-dimensional image, which is compressed using JPEG and PNG to effectively reduce the volume.

[0056] Similarly, in actual cases, in multi-view reconstruction of urban buildings, the discrete values ​​of the normal vector components on the same building surface have a high probability of repeated values. This feature effectively increases the compression ratio.

[0057] The following is a multi-view video provided by the present invention. Figure 3 The processing flow of the file compression method for depth map and normal map in 3D reconstruction:

[0058] Step 1, read the depth map and normal map, and separate the integer part and the decimal part of the depth map;

[0059] Step 2: For the integer part of the depth map, directly compress it using JPEG or PNG compression technology;

[0060] Since JPEG compression only supports 8-bit, PNG compression supports 8-bit and 16-bit images. Therefore, the range of the depth map is first counted. For depth maps with a range less than 255, the depth value is offset to 0-255 and stored in JPEG or PNG format (offset method: pixel depth value minus the minimum depth value of the entire image); for depth maps with a range of 255-65535, the depth value is offset to 0-65535 and stored in PNG format; for depth ranges exceeding 65535, no compression is performed.

[0061] Step 3: For the decimal part of the depth map, firstly discretize it by using 8-bit discretization or 16-bit discretization method, and then compress the discretized two-dimensional map by using JPEG or PNG compression technology;

[0062] For the decimal part of the depth map, its value range is 0 to 1. For different accuracy requirements, appropriate value compression strategies are used. For centimeter-level precision 3D reconstruction, the 0 to 1 value is discretized to 0 to 255. The depth accuracy after discretization is 1 / 255 = 0.0039 meters, which is sufficient for centimeter-level reconstruction. The specific formula is:

[0063] ds = floor(df*255)

[0064] Among them, ds is the discretized value, df is the decimal part of the depth value, and floor is the floor function.

[0065] For 3D reconstruction with millimeter level or higher accuracy, the value of 0 to 1 is discretized to 0 to 65535. The depth accuracy after discretization is 1 / 65535 = 0.000015 meters, which is high enough. The specific formula is:

[0066] ds = floor(df*65535)

[0067] Step 4, discretize the three components of the normal map using an 8-bit discretization method;

[0068] Since the normal map does not actually participate in the coordinate calculation of the point cloud, but only participates in the calculation of the point cloud orientation, the accuracy requirement is significantly lower than that of the depth map. Therefore, the three components of the normal are discretized to 8 bits respectively. The depth accuracy after discretization is 1 / 255=0.0039 meters. The discretization formula is the same as the discretization formula of the decimal part of the depth map.

[0069] Step 5: compress the two-dimensional graph composed of the three discretized components using JPEG or PNG compression technology.

[0070] In addition, the three components of the normal vector satisfy:

[0071] (Nx) 2 +(Ny) 2 +(Nz) 2 =1

[0072] Among them, Nx, Ny, and Nz are the components of the normal vector in the X, Y, and Z directions respectively. Therefore, only two components need to be discretized in practice, and the third component can be calculated from the values ​​of the first two components according to this formula.

[0073] The specific implementation process of JPEG compression technology is as follows:

[0074] First, JPEG usually converts the image from RGB color space to YCbCr color space. In YCbCr color space, Y stands for luminance, while Cb and Cr stand for chrominance, which represent the deviation of blue and red respectively. This conversion allows the subsequent color information to be compressed more effectively because the human eye is more sensitive to luminance than chrominance.

[0075] The formula for color space conversion is:

[0076] Y=0.299R+0.587G+0.114B

[0077] Cb=-0.1687R-0.3313G+0.5B+128

[0078] Cr=0.5R-0.4187G-0.0813B+128

[0079] The chrominance signals (Cb and Cr) are then downsampled because the human eye is less sensitive to color details. The amount of data can be further reduced by reducing the resolution of the chrominance components.

[0080] The image is then divided into several 8*8 pixel blocks, and each block is subjected to a discrete cosine transform (DCT). DCT helps convert the image from the spatial domain (pixel level) to the frequency domain, and the result is 64 DCT coefficients for each 8*8 block, which represent the spatial frequency content of the image block. The main purpose of DCT is to find low-frequency and high-frequency information in the image block.

[0081] The formula for DCT transformation is:

[0082]

[0083] Where x and y refer to the spatial coordinates of the pixel in the image block. u and v are frequency variables used to represent the coordinates in the transformed frequency block (that is, the block after DCT processing). In DCT, u and v represent the horizontal and vertical positions of the frequency. F(u,v) is the transformed coefficient, f(x,y) is the original pixel value, and C(u) and C(v) are the normalized coefficients.

[0084] After applying the DCT, the next step is quantization. In this step, the DCT coefficients are divided element-by-element by the quantization table, and the result is rounded to the nearest integer to obtain the quantization matrix. The quantization table controls the degree to which different frequency components are retained. High-frequency components can usually be quantized more coarsely because the human eye is less sensitive to them. The quantization matrix controls the compression rate and image quality. High-frequency coefficients are usually compressed more sharply.

[0085] The quantization table is designed as follows: First, the entire shooting scene is classified according to the content of the captured image (such as city, countryside, wilderness, etc.), and a unique quantization table is designed for each scene based on the past experience of compressing depth maps and normal maps. In the JPEG compression process, the corresponding quantization table is selected according to the judged shooting scene and applied in the DCT transformation and quantization steps to improve compression efficiency. Feedback is continuously collected during the compression process, and this data is used to improve scene classification and quantization table design.

[0086] Finally, the quantization matrix is ​​further compressed by run-length coding, Huffman coding or arithmetic coding. The most commonly used is Huffman coding, which is a variable-length coding technology based on the frequency of symbol occurrence. All processed data and necessary file header information are assembled into a JPEG format file.

[0087] The effect of the present invention is illustrated below by the experimental data in Table 1:

[0088] Table 1 Compression results of depth map and normal map (volume unit Mb)

[0089]

[0090] In the above experiment, the integer part of the depth map is compressed using PNG, and the decimal part is discretized using 8 bits and compressed using PNG; each component of the normal map is discretized using 8 bits and compressed using JPEG. The experimental results show that both the depth map and the normal map can achieve a very high compression ratio, effectively reducing the size of the results.

[0091] On the other hand, the embodiment of the present invention also provides a multi-view Figure 3 File compression system for depth and normal maps in 3D reconstruction, including:

[0092] one or more processors;

[0093] A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the above solution.

[0094] The above description is only a preferred specific implementation manner of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention should be included in the scope of protection of the invention.

Claims

1. A file compression method for depth maps and normal maps in multi-view 3D reconstruction. It is characterized in that The steps include: Step 1, read the depth map and normal map, and separate the integer part and the decimal part of the depth map; Step 2: For the integer part of the depth map, directly compress it using JPEG or PNG compression technology; Step 3: For the decimal part of the depth map, firstly discretize it by using 8-bit discretization or 16-bit discretization method, and then compress the discretized two-dimensional map by using JPEG or PNG compression technology; Step 4, discretize the three components of the normal vector in the normal map using an 8-bit discretization method; Step 5: compress the two-dimensional graph composed of the three discretized components using JPEG or PNG compression technology.

2. A method for compressing files of depth maps and normal maps in multi-view 3D reconstruction as claimed in claim 1, Features: The specific implementation of step 2 is as follows: First, count the range of the depth map. For depth maps with a range less than 255, offset the depth value to 0-255 and store it in JPEG or PNG format. For depth maps with a range of 255-65535, offset the depth value to 0-65535 and store it in PNG format. For depth maps with a range exceeding 65535, no compression is performed.

3. The method for compressing depth maps and normal maps in multi-view 3D reconstruction according to claim 2, Features: The depth value is offset by subtracting the minimum depth value in the entire depth map from the pixel depth value.

4. The method for compressing depth maps and normal maps in multi-view 3D reconstruction according to claim 1, Features: The specific calculation formula of the 8-bit discretization method is: ds = floor(df*255) Among them, ds is the discretized value, df is the decimal part of the depth value, and floor is the floor function.

5. The method for compressing files of depth map and normal map in multi-view 3D reconstruction according to claim 1, Features: The specific calculation formula of the 16-bit discretization method is: ds = floor(df*65535) Among them, ds is the discretized value, df is the decimal part of the depth value, and floor is the floor function.

6. The method for compressing files of depth map and normal map in multi-view 3D reconstruction according to claim 1, Features: The three components in step 4 satisfy: (Nx) 2 +(Or) 2 +(Nz) 2 1 Among them, Nx, Ny, and Nz are the components of the normal vector in the X, Y, and Z directions respectively. Therefore, only two components need to be discretized in practice, and the third component is calculated from the values ​​of the first two components according to the above formula.

7. The method for compressing files of depth maps and normal maps in multi-view 3D reconstruction according to claim 1, Features: The specific implementation process of JPEG compression technology is as follows: First, convert the image from RGB color space to YCbCr color space. In YCbCr color space, Y represents luminance, while Cb and Cr represent chrominance, which represent the deviation of blue and red respectively. The formula for color space conversion is: Y=0.299R+0.587G+0.114B Cb=-0.1687R-0.3313G+0.5B+128 Cr=0.5R-0.4187G-0.0813B+128 Then the chrominance signals Cb and Cr are downsampled, the image is divided into several 8*8 pixel blocks, and each block is subjected to discrete cosine transform DCT to obtain the DCT coefficient; the formula for DCT transformation is: F(u,v) is the transformed coefficient, f(x,y) is the original pixel value, C(u) and C(v) are normalized coefficients, x and y refer to the spatial coordinates of the pixel in the image block, u and v are frequency variables used to represent the frequency block after the transformation; The DCT coefficients are divided element by element by the quantization table, and the result is rounded to the nearest integer to obtain the quantization matrix. Finally, the quantization matrix is ​​further compressed through the existing encoding method, and all the processed data and necessary file header information are assembled into a JPEG format file.

8. The method for compressing files of depth map and normal map in multi-view 3D reconstruction according to claim 1, Features: The design process of the quantization table is as follows: First, the entire shooting scene is classified according to the content of the captured image, and a unique quantization table is designed for each scene based on the empirical values ​​of depth map and normal map compression. In the JPEG compression process, the corresponding quantization table is selected according to the judged shooting scene. Feedback is continuously collected during the compression process, and this data is used to improve scene classification and quantization table design.

9. The method for compressing files of depth map and normal map in multi-view 3D reconstruction according to claim 1, Features: Existing encoding algorithms include run-length encoding, Huffman encoding or arithmetic coding.

10. A file compression system for depth maps and normal maps in multi-view 3D reconstruction. It is characterized in that include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 9.