Projection transformation method from hexahedron panoramic data to spherical panoramic data

Abstract

The invention provides a projection transformation method for hexahedral panoramic data to spherical panoramic data, and the method achieves the seamless splicing of panoramic images of six surfaces of a cube and the mapping of the panoramic images to a spherical coordinate system through the determination of a direction vector, the normalization of the vector, the conversion of spherical coordinates, the projection to an equidistant histogram, and an illumination and color correction algorithm. The problems of splicing gaps, deformation, texture distortion and the like in a traditional method are solved. The method is not only widely applied to the fields of virtual reality, augmented reality, panoramic video production and the like, but also greatly expands the application scenes, including but not limited to panoramic roaming, streetscape roaming experience, integrated use of specific roaming plug-ins, seamless embedding on a map platform and the like, and has wide application prospects. Therefore, the display effect of the panoramic data and the immersive experience of the user are remarkably improved.

Description

Technical Field

[0001] With the rapid development of virtual reality and augmented reality technologies, panoramic images are being used more and more widely. A cubic panoramic image is usually composed of six faces, each of which is a rectangular image, while a spherical panoramic image is a complete spherical image. The conversion process requires mapping the six faces of the cube onto a spherical surface. Traditional panoramic data is usually stored and displayed in spherical form, while hexahedral panoramic data is more popular in certain scenarios due to its easy collection and processing characteristics. However, how to effectively convert hexahedral panoramic data into spherical panoramic data has become a technical bottleneck restricting its widespread application. The present invention proposes an innovative solution to achieve accurate projection transformation of hexahedral panoramic data into spherical panoramic data. Background Art

[0002] Existing algorithms and technologies for projecting hexahedral panoramic data into spherical panoramic data present significant challenges. First, projection inaccuracy is a major issue. Due to inaccurate mathematical models and pixel mapping errors, the projected image content can be deformed, stretched, or compressed, impacting the viewing experience. Second, stitching misalignment is another pressing issue. Improper boundary processing and image content discrepancies can lead to noticeable boundaries or discontinuities in the stitched area.

[0003] Existing methods also suffer from high computational complexity and image quality loss, limiting their widespread use in practical applications. Therefore, there is an urgent need to develop a more accurate and efficient projection transformation algorithm to overcome the shortcomings of existing technologies, improve the accuracy and quality of the conversion from hexahedral panoramic data to spherical panoramic data, and provide users with a better panoramic viewing experience. Summary of the Invention

[0004] In order to overcome the above-mentioned deficiencies, the present invention provides a method for projecting and transforming hexahedral panoramic data into spherical panoramic data.

[0005] In order to achieve the above object, the present invention provides a method for projecting hexahedral panoramic data into spherical panoramic data, comprising the following steps:

[0006] Step 1: Data preprocessing to obtain panoramic image data containing six faces of the cube (front, back, left, right, top, and bottom).

[0007] Step 2: Projection mapping model is constructed. This model is based on the principle of spherical geometry to ensure that the image of each cube face can be accurately mapped to the corresponding area on the spherical coordinate system.

[0008] Step 3, texture mapping and stitching, projects the panoramic images of each face of the cube onto the spherical coordinate system according to the mapping model, and uses the edge fusion algorithm to process the stitching edges to ensure seamless connection.

[0009] Step 4: spherical image optimization. The stitched spherical panoramic image is subjected to color correction using a polynomial regression method, brightness balance using a gamma correction algorithm, and detail enhancement using a bilateral filtering algorithm to improve image quality.

[0010] Step 5: Output and storage: Output the optimized spherical panoramic image to a standard JPEG or PNG image format with a panoramic aspect ratio of 2:1 for easy subsequent application.

[0011] Based on the above, the main operations of data preprocessing in step 1 include:

[0012] The six face data of the cube are stored in a single folder and named according to the six pictures (front, back, left, right, up, down) according to the photo position.

[0013] Based on the above, the projection mapping model is constructed in step 2. During the mapping process, the spherical coordinates of each cube face are converted. The main process includes:

[0014] Step 2-1, determine the direction vector: For each pixel (x, y) on the cube, determine a 3D direction vector V = (X, Y, Z) based on the face it is on.

[0015] Step 2-2, normalize the vector: Normalize the above vector to obtain a unit vector V_normalized, which represents the direction from the center of the sphere to the position of the pixel on the sphere.

[0016] Step 2-3, Spherical Coordinate Conversion: Calculate spherical coordinates using normalized vectors where θ is the zenith angle (polar angle), is the azimuth (longitude angle).

[0017] Steps 2-4, using projection to the equidistant histogram, relocate each pixel of the cube to the corresponding position in the target spherical image and the position projected to the equidistant histogram based on spherical coordinates, and then perform spherical projection transformation based on spherical coordinates. Relocate each pixel to its corresponding position in the target spherical image. In this projection, the latitude θ and longitude Directly mapped to the y and x coordinates on the two-dimensional plane. The UV coordinate formula is as follows:

[0018]

[0019] v=θ / π

[0020] Based on the above, in step 3, based on the mapping model in step 2, the main process of texture mapping and stitching includes:

[0021] Step 3-1: Define texture coordinates. For each face of the cube, define a two-dimensional texture coordinate system (u, v), where u and v represent the horizontal and vertical coordinates of the texture image, respectively. For each point (x, y, z) on the cube, calculate the corresponding texture coordinates (u, v) based on its face and position.

[0022] Step 3-2, apply the interpolation algorithm: The points on the cube surface may not be integer coordinates, so a bilinear interpolation algorithm is needed to calculate the texture coordinates corresponding to these points. The formula of the bilinear interpolation algorithm is:

[0023] f(x,y)=lerp(t,lerp(s,u 00 ,u 10 ),lerp(s,u 01 ,u 11 ))

[0024] Among them, u 00 ,u 01 ,u 10 ,u 11 are the four pixel values around the sampling point, and s and t are the interpolation weights of the sampling point in the horizontal and vertical directions.

[0025] Step 3-3, project to the spherical coordinate system, project the texture image of each face of the cube onto the spherical coordinate system. Convert the point (x, y, z) in the three-dimensional space to the point in the spherical coordinate system The conversion formula is as follows:

[0026] θ=arctan2(y,x)

[0027]

[0028] Among them, arctan2 is the four-quadrant inverse tangent function, and arccos is the inverse cosine function.

[0029] Steps 3-4 are used to stitch the images together. In the spherical coordinate system, the stitching edges between adjacent cube faces are determined. These edges are typically the intersections of two adjacent faces in the spherical coordinate system. The weighted average method of the edge fusion algorithm is used to process the stitching edges to reduce or eliminate stitching artifacts. The formula for the weighted average method is:

[0030] I fused (x,y)=αI1(x,y)+(1-α)I2(x,y)

[0031] Among them, I fused(x, y) is the pixel value after fusion, I1(x, y) and I2(x, y) are the pixel values on adjacent surfaces, and a is the weighting coefficient, which is usually determined according to the distance from the stitching edge.

[0032] Based on the above, in step 4, spherical image optimization is performed based on the image stitched in step 3. The stitched spherical panoramic image is subjected to color correction using the polynomial regression method, brightness balance using the gamma correction algorithm, and detail enhancement using the bilateral filtering algorithm to improve image quality. The main process includes:

[0033] Step 4-1, color correction: First, apply the polynomial regression method to adjust the overall hue of the image to ensure color consistency between different parts. Select multiple point pairs from the original image and the target reference image as training samples. Use each channel value in the RGB color space as an input feature. Use the collected sample data to train a polynomial regression model. Use the trained model to predict and adjust the color values of all pixels in the original image. Assuming we want to correct the R channel (similar to the G and B channels), we can build a polynomial regression model in the following form:

[0034]

[0035] where a i is the coefficient obtained by fitting using methods such as the least squares method, and n is the order of the polynomial. For each color channel (R, G, B), we need to train a separate polynomial regression model.

[0036] Step 4-2, brightness balance: Then use gamma correction to adjust the brightness distribution of the image to make the image look more natural. Select the appropriate gamma value γ according to the specific situation of the image. Transform each pixel value in the image according to the gamma function. The formula for gamma correction is:

[0037]

[0038] Here V in Represents the brightness value of the input image (usually normalized to the [0,1] interval), V out Represents the brightness value of the output image, and γ is the gamma value, which determines the shape of the curve. When γ < 1, the image will become brighter; when γ > 1, the image will become darker.

[0039] Step 4-3, detail enhancement: Finally, bilateral filtering is used to preserve edge information while reducing noise and improving the visual quality of the image. A spatial distance weight and a color difference weight are calculated for each pixel in the neighborhood. The two weights are combined to perform a weighted average of the pixels in the neighborhood to generate a new pixel value. The output pixel of the bilateral filter depends on the neighborhood of the current convolved pixel. i and j are the coordinates of the current convolved pixel, and k and l are the coordinates of the neighborhood pixels. The bilateral filtering formula is as follows:

[0040]

[0041] The weighting coefficient w is determined by the domain kernel and the range kernel, and is the product of them. The domain kernel is a Gaussian kernel, as shown in the following formula:

[0042]

[0043] The range kernel is a method used to "infer" whether it is an edge point, as shown in the following formula:

[0044]

[0045] The size of the range kernel depends on the difference between the grayscale value of the convolved pixel and the grayscale value of the neighboring pixels. Large grayscale changes at the edge generate smaller weights, while areas with similar grayscale values to the convolved pixel generate larger weights. Multiplying these together yields the weighting coefficient w, as shown in the following formula:

[0046]

[0047] Based on the above, in step 5, output and storage: output the optimized spherical panoramic image to a standard panoramic aspect ratio of 2:1 JPEG or PNG image format, and use a panoramic viewer to verify whether the generated image is displayed correctly and confirm that there is no obvious distortion or other problems for subsequent use.

[0048] Compared with the existing projection transformation method, the present invention has the following advantages:

[0049] 1. By constructing a projection mapping model based on spherical geometry, we ensure that the image of each cube face is accurately mapped to the corresponding area on the spherical coordinate system. This approach avoids the deformation and distortion problems common in traditional methods and ensures the geometric consistency and accuracy of the converted image.

[0050] 2. An edge blending algorithm is used to process the stitching edges, ensuring smooth and natural transitions between different surfaces, eliminating the obvious stitching lines or color inconsistencies that may occur with traditional methods. This not only improves the visual effect, but also enhances the realism and immersion of the user experience.

[0051] 3. Full-process integrated design provides a complete solution from data preprocessing, projection mapping, texture mapping and stitching to image optimization and final output storage, covering all key links from input to output.

[0052] 4. Flexible application expansion: This method is not only applicable to the conversion of static images, but can also be extended to real-time processing of dynamic video streams, leaving room for future technological development.

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a framework flow chart of the method of the present invention;

[0055] Figure 2 This is a schematic diagram of an improved projection transformation method of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] In order to make the method of the present invention more clear, the method of the present invention is described in detail below with reference to the accompanying drawings and examples.

[0058] like Figure 1-2 As shown, this embodiment provides a method for projecting hexahedral panoramic data into spherical panoramic data, including the following steps:

[0059] Step 1: Data preprocessing: store the six face data of the cube in a single folder and name them according to the six pictures (front, back, left, right, up, down) according to the shooting position.

[0060] Step 2: Projection mapping model construction. During the mapping process, the spherical coordinates of each cube face are converted. The main process includes:

[0061] Step 2-1, determine the direction vector: For each pixel (x, y) on the cube, determine a 3D direction vector V = (X, Y, Z) based on the face it is on.

[0062] Step 2-2, normalize the vector: Normalize the above vector to obtain a unit vector V_normalized, which represents the direction from the center of the sphere to the position of the pixel on the sphere.

[0063] Step 2-3, Spherical Coordinate Conversion: Calculate spherical coordinates using normalized vectors where θ is the zenith angle (polar angle), is the azimuth (longitude angle).

[0064] Steps 2-4, using projection to the equidistant histogram, relocate each pixel of the cube to the corresponding position in the target spherical image and the position projected to the equidistant histogram based on spherical coordinates, and then perform spherical projection transformation based on spherical coordinates. Relocate each pixel to its corresponding position in the target spherical image. In this projection, the latitude θ and longitude Directly mapped to the y and x coordinates on the two-dimensional plane. The UV coordinate formula is as follows:

[0065]

[0066] v=θ / π

[0067] Based on the above, in step 3, based on the mapping model in step 2, the main process of texture mapping and stitching includes:

[0068] Step 3-1: Define texture coordinates. For each face of the cube, define a two-dimensional texture coordinate system (u, v), where u and v represent the horizontal and vertical coordinates of the texture image, respectively. For each point (x, y, z) on the cube, calculate the corresponding texture coordinates (u, v) based on its face and position.

[0069] Step 3-2, apply the interpolation algorithm: The points on the cube surface may not be integer coordinates, so a bilinear interpolation algorithm is needed to calculate the texture coordinates corresponding to these points. The formula of the bilinear interpolation algorithm is:

[0070] f(x,y)=lerp(t,lerp(s,u 00 ,u 10 ),lerp(s,u 01 ,u 11 ))

[0071] Among them, u 00 ,u 01 ,u 10 ,u 11 are the four pixel values around the sampling point, and s and t are the interpolation weights of the sampling point in the horizontal and vertical directions.

[0072] Step 3-3, project to the spherical coordinate system, project the texture image of each face of the cube onto the spherical coordinate system. Convert the point (x, y, z) in the three-dimensional space to the point in the spherical coordinate system The conversion formula is as follows:

[0073] θ=arctan2(y,x)

[0074]

[0075] Among them, arctan2 is the four-quadrant inverse tangent function, and arccos is the inverse cosine function.

[0076] Steps 3-4 are used to stitch the images together. In the spherical coordinate system, the stitching edges between adjacent cube faces are determined. These edges are typically the intersections of two adjacent faces in the spherical coordinate system. The weighted average method of the edge fusion algorithm is used to process the stitching edges to reduce or eliminate stitching artifacts. The formula for the weighted average method is:

[0077] I fused (x,y)=αI1(x,y)+(1-α)I2(x,y)

[0078] Among them, I fused (x, y) is the pixel value after fusion, I1(x, y) and I2(x, y) are the pixel values on adjacent faces, and a is the weighting coefficient, which is usually determined based on the distance from the stitching edge.

[0079] Step 4: Use the polynomial regression method to perform color correction on the stitched spherical panoramic image, the gamma correction algorithm to balance brightness, and the bilateral filtering algorithm to enhance details to improve image quality. The main process includes:

[0080] Step 4-1, color correction: First, apply the polynomial regression method to adjust the overall hue of the image to ensure color consistency between different parts. Select multiple point pairs from the original image and the target reference image as training samples. Use each channel value in the RGB color space as an input feature. Use the collected sample data to train a polynomial regression model. Use the trained model to predict and adjust the color values of all pixels in the original image. Assuming we want to correct the R channel (similar to the G and B channels), we can build a polynomial regression model in the following form:

[0081]

[0082] where a i is the coefficient obtained by fitting using methods such as the least squares method, and n is the order of the polynomial. For each color channel (R, G, B), we need to train a separate polynomial regression model.

[0083] Step 4-2, brightness balance: Then use gamma correction to adjust the brightness distribution of the image to make the image look more natural. Select the appropriate gamma value γ according to the specific situation of the image. Transform each pixel value in the image according to the gamma function. The formula for gamma correction is:

[0084]

[0085] Here V in Represents the brightness value of the input image (usually normalized to the [0,1] interval), V out Represents the brightness value of the output image, and γ is the gamma value, which determines the shape of the curve. When γ < 1, the image will become brighter; when γ > 1, the image will become darker.

[0086] Step 4-3, detail enhancement: Finally, bilateral filtering is used to preserve edge information while reducing noise and improving the visual quality of the image. A spatial distance weight and a color difference weight are calculated for each pixel in the neighborhood. The two weights are combined to perform a weighted average of the pixels in the neighborhood to generate a new pixel value. The output pixel of the bilateral filter depends on the neighborhood of the current convolved pixel. i and j are the coordinates of the current convolved pixel, and k and l are the coordinates of the neighborhood pixels. The bilateral filtering formula is as follows:

[0087]

[0088] The weighting coefficient w is determined by the domain kernel and the range kernel, and is the product of them. The domain kernel is a Gaussian kernel, as shown in the following formula:

[0089]

[0090] The range kernel is a method used to "infer" whether it is an edge point, as shown in the following formula:

[0091]

[0092] The size of the range kernel depends on the difference between the grayscale value of the convolved pixel and the grayscale value of the neighboring pixels. Large grayscale changes at the edge generate smaller weights, while areas with similar grayscale values to the convolved pixel generate larger weights. Multiplying these together yields the weighting coefficient w, as shown in the following formula:

[0093]

[0094] Step 5: Output and storage: Output the optimized spherical panoramic image to a JPEG or PNG image format with a standard panoramic aspect ratio of 2:1. Use a panoramic viewer to verify whether the generated image is displayed correctly and confirm that there is no obvious distortion or other problems for subsequent use.

[0095] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for projecting hexahedral panoramic data into spherical panoramic data, characterized in that The following steps are involved: Step 1: Data preprocessing to obtain panoramic image data containing six faces of the cube (front, back, left, right, top, and bottom); Step 2: Projection mapping model construction. This model is based on the principles of spherical geometry to ensure that the image of each cube face can be accurately mapped to the corresponding area on the spherical coordinate system. Step 3: Texture mapping and stitching: Project the panoramic images of each face of the cube onto the spherical coordinate system according to the mapping model, and use the edge fusion algorithm to process the stitching edges to ensure seamless connection; Step 4: spherical image optimization: the stitched spherical panoramic image is subjected to color correction using a polynomial regression method, brightness balance using a gamma correction algorithm, and detail enhancement using a bilateral filtering algorithm to improve image quality. Step 5: Output and storage: Output the optimized spherical panoramic image to a standard JPEG or PNG image format with a panoramic aspect ratio of 2:1 for easy subsequent application.

2. The method according to claim 1, characterized in that In the projection mapping model construction step, an equidistant columnar projection method is adopted.

3. The method according to claim 1, characterized in that In the texture mapping and stitching steps, the edge fusion algorithm adopts a weighted average method.

4. The method according to claim 1, wherein In the output and storage steps, multiple panoramic formats are supported for output, including at least one of JPEG or PNG formats.

5. A computer program product for implementing the method according to any one of claims 1 to 4, comprising computer program code stored on a computer-readable storage medium, wherein when the program code is run on a computer, the method according to any one of claims 1 to 4 is executed.

Images