A VR three-dimensional virtual scene rendering method and system

Abstract

The application relates to the technical field of virtual scene rendering, in particular to a VR three-dimensional virtual scene rendering method and system, which comprises the following steps: determining the reconstruction weight of each subband image by analyzing the similarity of color characteristic values of each pixel point in all color channels between images in each subband of two color spaces; obtaining a color guide image by reconstructing the subband images based on the reconstruction weight; determining a color guide value by combining the difference between each pixel point in the preset window of each pixel point in the color guide image and the average distribution of all pixel points, and combining the difference between the pixel value of each pixel point and any pixel point in the preset window of the pixel point; and repairing a scene depth map by combining a filtering algorithm. The application solves the problems of depth information loss, edge blur and detail loss in the traditional depth map repair method in a high-reflective or transparent object scene, and improves the rendering quality of the VR three-dimensional virtual scene.

Description

Technical Field

[0001] This application relates to the field of virtual scene rendering technology, specifically to a VR three-dimensional virtual scene rendering method and system. Background Technology

[0002] Virtual reality (VR) technology serves as a bridge connecting the real and virtual worlds, finding widespread application in entertainment, education, healthcare, and other fields, providing users with unprecedented immersive experiences. In creating a more realistic and fluid virtual environment within a VR system, the image rendering engine is a key technology for achieving this goal.

[0003] The core goal of 3D virtual scene rendering is to highly reproduce a realistic visual experience, which relies on high-quality 3D scene data. However, actual 3D scene images often suffer from depth information errors due to highly reflective or transparent objects such as glass and metal, leading to jitter and jagged edges in the rendering and affecting visual realism. Traditional depth map restoration methods are prone to edge blurring and detail loss in complex scenes, making it difficult to balance smoothness and structural fidelity, thus reducing the rendering quality of VR 3D virtual scenes. Summary of the Invention

[0004] To address the aforementioned technical problems, the purpose of this application is to provide a VR 3D virtual scene rendering method and system, the specific technical solution of which is as follows:

[0005] In a first aspect, embodiments of this application provide a VR three-dimensional virtual scene rendering method, which includes the following steps:

[0006] Obtain the scene depth map of the target scene, and convert the scene depth map to two color spaces, namely RGB space and HSV space;

[0007] For scene depth maps in two color spaces, wavelet transform algorithm is used to obtain the images in each sub-band and the wavelet coefficients of each pixel in the images in any color channel. In each color space, based on the average distribution and dispersion of the wavelet coefficients of all pixels in the neighborhood of each pixel in the image of each sub-band, the color feature value of each pixel in the image of each sub-band in each color space in any color channel is determined. By analyzing the similarity of the color feature values ​​of each pixel in all color channels between the images of each sub-band in the two color spaces, the color consistency of each pixel in the images of each sub-band in the two color spaces is determined.

[0008] Based on the distribution of color consistency of all pixels in each sub-band image under the two color spaces, the overall color consistency of each sub-band image under the two color spaces is determined, so as to determine the reconstruction weight of each sub-band image; based on the reconstruction weight, the inverse wavelet transform algorithm is used to reconstruct the color guide map of all sub-band images in RGB space.

[0009] Based on the difference between each pixel in the preset window of each pixel in the color guide map and the average distribution of all pixels, the color kernel width of each pixel is determined, and combined with the difference of pixel value between each pixel and any pixel in its preset window, the color guide value between each pixel and any pixel in its preset window is determined.

[0010] The scene depth map is repaired based on the color guidance value using a filtering algorithm for scene rendering.

[0011] Preferably, the color feature value of each pixel in the image of each sub-band under each color space in any color channel is: the ratio of the mean of the wavelet coefficients of all pixels in the neighborhood of each pixel in the image of each sub-band under each color space to the degree of dispersion in any channel.

[0012] Preferably, the color consistency of each pixel in the image under each sub-band in the two color spaces is the cosine similarity of the color feature values ​​of each pixel in all color channels between the images under each sub-band in the two color spaces.

[0013] Preferably, the overall color consistency of each sub-band image in the two color spaces is the average of the color consistency of all pixels in each sub-band image in the two color spaces.

[0014] Preferably, the reconstruction weight of each sub-band image is the ratio of the overall color consistency of each sub-band image in the two color spaces to the sum of the overall color consistency of all sub-band images in the two color spaces.

[0015] Preferably, the step of reconstructing the color guide map by using the inverse wavelet transform algorithm based on the reconstruction weights for all sub-band images in the RGB space includes:

[0016] All sub-band images in the RGB space are used as input to the inverse wavelet transform algorithm. The sub-band images are fused according to the reconstruction weights, and the output image is used as a color guide map.

[0017] Preferably, the expression for the color kernel width of each pixel is:

[0018] In the formula, This represents the width of the color kernel at pixel i in the color guide map; This represents the pixel value of the j-th pixel within a preset window of pixel i in the color guide map; This represents the average pixel value of all pixels within a preset window of pixel i in the color guide map; This indicates the number of all pixels within the preset window of pixel i in the color guide map.

[0019] Preferably, the expression for the color guiding value between each pixel and any pixel in its neighborhood is: ; This represents the color guidance value between pixel i in the color guide map and the j-th pixel j within its preset window; This represents the pixel value of pixel i in the color guide map; This represents the pixel value of the j-th pixel within a preset window of pixel i in the color guide map; represents the color kernel width of pixel i in the color guide diagram; exp[ ] represents an exponential function with the natural constant as the base.

[0020] Preferably, the step of repairing the scene depth map using a filtering algorithm based on the color guidance value includes:

[0021] The scene depth map is used as the input to the filtering algorithm. The color guide value between each pixel and any pixel within a preset window is used as the color guide term in the filtering algorithm. The filtering window in the filtering algorithm is set to the same size as the preset window. The output image is used as the repaired scene depth map.

[0022] Secondly, embodiments of this application also provide a VR three-dimensional virtual scene rendering system, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement any of the above-described VR three-dimensional virtual scene rendering methods.

[0023] This application has at least the following beneficial effects:

[0024] This application first performs multi-scale decomposition of depth maps in RGB and HSV color spaces using wavelet transform. It then calculates color feature values ​​by combining the mean and dispersion of the neighborhood wavelet coefficients and evaluates color consistency using the cosine similarity of the feature vectors from the two color spaces. This effectively filters reliable color information, suppresses noise interference, preserves details in edge regions, and optimizes noise in smooth regions, thus improving the accuracy and reliability of depth map restoration. Second, this application calculates the overall color consistency of each sub-band image in RGB and HSV color spaces and uses its mean as a global reliability index to adaptively determine the reconstruction weights of each sub-band. Based on these reconstruction weights, it employs inverse wavelet transform to perform weighted fusion of all sub-band images in the RGB space, generating high-quality color images. The guiding image helps improve the reliability and edge preservation capabilities of depth map restoration. Furthermore, this application calculates the color kernel width based on the distribution differences of pixel values ​​in the neighborhood of each pixel in the color guiding image, and determines the color guiding value by combining the color differences between pixels. This allows for adaptive adjustment of weights during the filtering process, effectively preserving edge details while smoothing noise and filling holes, thus improving the precision and visual consistency of depth map restoration. Finally, this application improves the bilateral filtering algorithm based on the color guiding value, achieving high-quality restoration of the scene depth map. This effectively suppresses noise interference, preserves edge details, and repairs depth holes and outliers, thereby providing more realistic, stable, and precise depth information for 3D virtual scene rendering and improving the rendering quality of VR 3D virtual scenes. Attached Figure Description

[0025] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating the steps of a VR 3D virtual scene rendering method provided in one embodiment of this application;

[0027] Figure 2 This is a flowchart of a scene depth map denoising and enhancement process provided in one embodiment of this application. Detailed Implementation

[0028] To further illustrate the technical means and effects adopted by this application to achieve the intended purpose of the invention, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a VR three-dimensional virtual scene rendering method and system proposed in this application. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0030] The following description, in conjunction with the accompanying drawings, details the specific scheme of a VR three-dimensional virtual scene rendering method and system provided in this application.

[0031] Please see Figure 1 The diagram illustrates a flowchart of a VR 3D virtual scene rendering method according to an embodiment of this application. The method includes the following steps:

[0032] Step S1: Obtain the scene depth map of the target scene and convert the scene depth map to two color spaces, namely RGB space and HSV space.

[0033] In RGB-D images, RGB reflects color information and D reflects depth information. Although they are different modal information, they work together to record and present the same scene or object during the imaging process. Depth information intuitively shows the spatial position of objects in the scene, while RGB can indirectly reflect the spatial distribution of objects based on features such as non-alternative colors and textures. Using the color of the object as a guide to repair missing or erroneous depth information can effectively improve the effect of filtering algorithms on the repair of image depth information.

[0034] Therefore, this embodiment uses a 3D ToF camera to acquire a scene depth map of the target scene, i.e., an RGB-D image, which contains optical color and depth information in the target scene. The RGB channel information in the scene depth map is then obtained to obtain an image in RGB space.

[0035] The acquired scene depth map includes omnidirectional scene images centered on the camera device, including horizontal and vertical 360° panoramic views.

[0036] Furthermore, since the three channels in the RGB color space simultaneously mix color and brightness information, the expression of color information is not independent and is easily affected by optical fibers, resulting in shadows and inaccurate restoration of depth images. In contrast, the HSV color space divides information into three relatively independent dimensions: hue, saturation, and brightness, which can better describe the characteristics of color. Therefore, this embodiment converts the scene depth map in the RGB color space to the HSV color space. The method of converting the RGB image to the HSV color space is a well-known technique, and the specific conversion process will not be described in detail.

[0037] Thus, the scene depth map was obtained in both RGB and HSV spaces.

[0038] Step S2: Analyze the multi-scale features of the depth map under the two color spaces using wavelet transform, calculate the color consistency to determine the reconstruction weight of each sub-band, generate a color guide map, and adaptively calculate the color kernel width and color guide value based on the color guide map for the repair and enhancement of the scene depth map.

[0039] Traditional depth map restoration methods, such as bilateral filtering algorithms, rely solely on the spatial distance and depth values ​​of pixels in the RGB-D image to filter and enhance the image when repairing depth information loss or deviation caused by highly reflective or transparent objects in RGB-D images. However, due to the complexity of scene features and uneven lighting conditions, traditional depth map restoration methods struggle to effectively preserve object edges and details, resulting in a decrease in the quality of the restored image and affecting the visual realism of the final 3D virtual scene.

[0040] To address the aforementioned issues, this embodiment analyzes the similarity of wavelet components in each channel of the target scene image in RGB and HSV spaces, constructs an adaptively reconstructed all-red color map, and generates a color guide map, thereby more accurately reflecting the true color information of objects. Finally, based on the color guide map, the color guide depth is calculated, and an improved bilateral filtering algorithm is applied to achieve depth restoration and enhancement of the scene image. The flowchart of the scene depth map denoising and enhancement process provided in this embodiment is as follows: Figure 2 As shown, the specific steps for denoising and enhancing the scene depth map are as follows:

[0041] S2.1 For scene depth maps in two color spaces, wavelet transform algorithm is used to obtain the images in each sub-band and the wavelet coefficients of each pixel in the image in any color channel. In each color space, based on the average distribution and dispersion of the wavelet coefficients of all pixels in the neighborhood of each pixel in the image in each sub-band, the color feature value of each pixel in the image in each sub-band in each color space in any color channel is determined. By analyzing the similarity of the color feature values ​​of each pixel in all color channels between the images in each sub-band of the two color spaces, the color consistency of each pixel in the image in each sub-band in the two color spaces is determined.

[0042] In the RGB color space, color and brightness information are mixed across all three channels, resulting in a lack of independent color representation. This makes the color susceptible to lighting conditions, leading to shadows and reducing the accuracy of scene depth map restoration. While the HSV color space separates information into three independent dimensions—hue, saturation, and brightness—and better describes color characteristics, color information can still deviate from the true surface features of objects when lighting changes, such as when shadows or highly reflective objects appear in the target scene, making it difficult to effectively extract hue information. Therefore, relying solely on either the RGB or HSV color space as a color guide cannot guarantee the stability of depth information restoration.

[0043] Therefore, based on the above analysis, this embodiment uses wavelet transform algorithm to obtain the images of scene depth maps in each sub-band and the wavelet coefficients of each pixel in the image in any color channel for each sub-band under the two color spaces. In each color space, based on the average distribution and dispersion of the wavelet coefficients of all pixels in the neighborhood of each pixel in the image of each sub-band under any color channel, the color feature value of each pixel in the image of each sub-band under each color space is determined. By analyzing the similarity of the color feature values ​​of each pixel in all color channels between the images of each sub-band under the two color spaces, the color consistency of each pixel in the image of each sub-band under the two color spaces is determined, so as to filter out reliable color information and suppress the interference of unreliable color information. In areas with drastic color changes, such as object edges, details are preserved, and noise is smoothed in areas with uniform color, so as to improve the accuracy of scene depth map restoration and edge preservation capability. The specific process is as follows:

[0044] First, this embodiment addresses scene depth maps in two color spaces, namely RGB and HSV. The scene depth maps in both color spaces are used as inputs to a discrete wavelet transform algorithm. The wavelet basis function is the db2 wavelet basis, the decomposition level is set to K, and the number of sub-bands is set to F. When decomposing the scene depth maps in both color spaces, the corresponding frequency bands of the sub-bands are the same. The output consists of wavelet components in F sub-bands, which in this embodiment correspond to the image in F sub-bands. And output the wavelet coefficients of each pixel in the image under each sub-band, where, These are the approximate component, horizontal detail component, vertical detail component, and diagonal detail component, respectively.

[0045] It should be noted that the values ​​of the decomposition level K and the number of sub-bands F are set manually. In this embodiment, the value of the decomposition level K is 1 and the value of the number of sub-bands F is 4. In actual applications, as other implementation methods, implementers can also set them according to specific circumstances. This embodiment does not impose any special restrictions.

[0046] Among them, the discrete wavelet transform algorithm is a well-known technique. The specific process of using it to decompose an image into multiple wavelet components, that is, the specific process of decomposing an image into multiple sub-bands, will not be described in detail.

[0047] Furthermore, in each color space, this embodiment determines the color feature value of each pixel in each sub-band of the image in any color channel based on the average distribution and dispersion of the wavelet coefficients of all pixels in the neighborhood of each pixel in any sub-band under any channel. Specifically:

[0048] In this embodiment, the ratio of the mean to the dispersion of the wavelet coefficients of all pixels in the neighborhood of each pixel in the image of each sub-band under each color space is used as the color feature value of each pixel in the image of each sub-band under each color space in any color channel.

[0049] It should be noted that the method for constructing the neighborhood of a pixel is as follows: with each pixel as the center, an M×M window is used as the neighborhood of each pixel. The value of M is set manually. In this embodiment, the value of M is 3. In actual applications, as other implementation methods, implementers can also set it according to specific circumstances. This embodiment does not impose any special restrictions.

[0050] It should be noted that there are many methods to measure the dispersion of a set of data. In this embodiment, the variance of the wavelet coefficients of all pixels in the neighborhood of each pixel in the image of each sub-band in any channel is used as the dispersion of the wavelet coefficients of all pixels in the neighborhood of each pixel in the image of each sub-band in any channel. In practical applications, as other implementation methods, implementers may also use other methods such as standard deviation or coefficient of variation to measure the dispersion of data, depending on the specific circumstances. This embodiment does not impose any special restrictions.

[0051] Based on the color feature value of each pixel in the image under each sub-band in any color channel, it can be understood that the color feature value is used to measure the stability of the color features in the neighborhood of the pixel. If the mean value of the wavelet coefficients of all pixels in the neighborhood of the current pixel is larger and the dispersion is smaller, it means that the wavelet coefficients in the neighborhood of the current pixel are concentrated and have high intensity. This indicates that the color information in the neighborhood of the current pixel is stable and less affected by noise, etc. The larger the corresponding color feature value, the higher the weight will be given in the subsequent enhancement and restoration process.

[0052] Conversely, if the mean value of the wavelet coefficients of all pixels in the neighborhood of the current pixel is smaller and the dispersion is greater in any channel, it indicates that the wavelet coefficients in the neighborhood of the current pixel are scattered and have low intensity. This indicates that the color information in the neighborhood of the current pixel is unstable and is subject to more interference from noise, etc. The smaller the corresponding color feature value, the lower the weight will be given in the subsequent enhancement and repair process.

[0053] Furthermore, this embodiment determines the color consistency of each pixel in the image under each sub-band in the two color spaces by analyzing the similarity of color feature values ​​of all color channels of each pixel in the images of each sub-band. Specifically:

[0054] In this embodiment, the cosine similarity of the color feature values ​​of each pixel in all color channels of the images under each sub-band of the two color spaces is used as the color consistency of each pixel in the images under each sub-band in the two color spaces.

[0055] The method for calculating cosine similarity is a well-known technique, and its specific calculation process will not be elaborated here.

[0056] To further clarify the calculation process for color consistency, the expression for calculating color consistency is provided below:

[0057] In the RGB space, the color feature values ​​of each pixel in the image under each sub-band under all channels are used to form an RGB color feature vector. Similarly, in the HSV space, the color feature values ​​of each pixel in the image under each sub-band under all channels are used to form an HSV color feature vector.

[0058] Color consistency of pixel m in the image under subband n in two color spaces The expression is: In the formula, , These represent the RGB color feature vector and HSV color feature vector of pixel m in the image under subband n, respectively. This represents the cosine similarity function.

[0059] Based on the color consistency of each pixel in all channels of the image under each sub-band, it can be understood that color consistency reflects the consistency of color features in the local range of the scene depth map in RGB and HSV spaces. If the cosine similarity of the color feature values ​​of the current pixel in all color channels between the two color spaces is greater, it indicates that the color information is consistent in RGB and HSV spaces, indicating that the reliability of the color information of the current pixel is higher, which means that the possibility of interference is smaller.

[0060] Conversely, if the cosine similarity of the color feature values ​​of all color channels of the current pixel is smaller between the two color spaces, it indicates that the color information is inconsistent in the RGB and HSV spaces, which means that the reliability of the color information of the current pixel is low, and that the possibility of interference is greater.

[0061] Thus, this embodiment decomposes the depth map in RGB and HSV color spaces using wavelet transform at multiple scales, calculates color feature values ​​by combining the mean and dispersion of the neighborhood wavelet coefficients, and evaluates color consistency by using the cosine similarity of the feature vectors of the two color spaces. This effectively filters reliable color information, suppresses noise interference, preserves details in edge areas, and optimizes noise in smooth areas, which helps to improve the accuracy and reliability of depth map restoration.

[0062] S2.2 Based on the distribution of color consistency of all pixels in each sub-band image under the two color spaces, determine the overall color consistency of each sub-band image under the two color spaces, and determine the reconstruction weight of each sub-band image; based on the reconstruction weight, use the inverse wavelet transform algorithm to reconstruct all sub-band images in RGB space to obtain the color guide map.

[0063] Furthermore, based on the color consistency obtained in step S2.1, this embodiment determines the overall color consistency of each sub-band image in the two color spaces based on the distribution of color consistency of all pixels in each sub-band image in the two color spaces, so as to determine the reconstruction weight of each sub-band image, specifically as follows:

[0064] In this embodiment, the average color consistency of all pixels in each sub-band image under the two color spaces is taken as the overall color consistency of each sub-band image under the two color spaces.

[0065] Based on the overall color consistency of each sub-band image in the two color spaces, it can be understood that the overall color consistency reflects the overall consistency of color information in the RGB space and the HSV space, and is used to characterize the global reliability of the sub-band color information. If the average consistency of all pixels in the current sub-band image in the two color spaces is larger, the corresponding overall color consistency is larger, indicating that the current sub-band color information is generally stable and the color information reliability is higher. When reconstructing the color guide map, the sub-band image should be given a higher weight.

[0066] Conversely, if the mean of the consistency of all pixels in the current sub-band image under the two color spaces is smaller, the overall color consistency is smaller, indicating that the color information of the current sub-band is unstable and the reliability of the color information is lower. When reconstructing the color guide map, the sub-band image should be given a lower weight.

[0067] Furthermore, the overall color consistency of each sub-band image in both color spaces is compared with the sum of the overall color consistency of all sub-band images in both color spaces, and this result is used as the reconstruction weight of each sub-band image.

[0068] Based on the reconstruction weight of each sub-band image, it can be understood that the reconstruction weight measures the importance of the sub-band in the reconstruction of the color guide map. If the overall color consistency of the current sub-band image is greater, that is, the reconstruction weight is greater, it means that the color information of the current sub-band is more reliable, and the current sub-band contributes more when reconstructing the color guide map. The color guide map should rely more on the color information of the current sub-band.

[0069] Conversely, if the overall color consistency of the current sub-band image is smaller, that is, the reconstruction weight is smaller, it indicates that the reliability of the color information of the current sub-band is low. When reconstructing the color guide map, the current sub-band contributes less, and the color guide map should reduce its dependence on the color information of the current sub-band.

[0070] Furthermore, in this embodiment, based on the aforementioned reconstruction weights, an inverse wavelet transform algorithm is used to reconstruct the color guide map from all sub-band images in the RGB space, specifically as follows:

[0071] In this embodiment, all sub-band images in the RGB space are used as input to the inverse wavelet transform algorithm. The sub-band images are fused according to the reconstruction weights, and the output image is used as a color guide map.

[0072] The inverse wavelet transform algorithm is a well-known technique, and the specific process of using the inverse wavelet transform algorithm to fuse sub-band images will not be elaborated here.

[0073] Thus, this embodiment calculates the overall color consistency of each sub-band image in the RGB and HSV color spaces and uses the average value as a global reliability index to adaptively determine the reconstruction weight of each sub-band. Based on the reconstruction weight, inverse wavelet transform is used to perform weighted fusion on all sub-band images in the RGB space to generate a high-quality color guide map, which helps to improve the reliability of depth map restoration and edge preservation capabilities.

[0074] S2.3 Based on the difference between each pixel in the preset window of each pixel in the color guide map and the average distribution of all pixels, determine the color kernel width of each pixel, and combine the difference of pixel value between each pixel and any pixel in the preset window to determine the color guide value between each pixel and any pixel in the preset window.

[0075] Based on the color guide map obtained in step S2.3, and based on the difference between each pixel within a preset window in the color guide map and the average distribution of all pixels, the color kernel width of each pixel is determined, specifically as follows:

[0076] As one implementation method, in this embodiment, the color kernel width of pixel i in the color guide map... The expression is: In the formula, This represents the pixel value of the j-th pixel within a preset window of pixel i in the color guide map; This represents the average pixel value of all pixels within a preset window of pixel i in the color guide map; This indicates the number of all pixels within the preset window of pixel i in the color guide map.

[0077] It should be noted that, in this embodiment, since the color guide image is an RGB image, the pixel value of each pixel in the color guide image is the average value of the pixel's components in the R, G, and B color channels.

[0078] Furthermore, in this embodiment, the color kernel width of each pixel, combined with the difference in pixel value between each pixel and any pixel within its preset window, determines the color guiding value between each pixel and any pixel within its preset window, specifically as follows:

[0079] As one implementation method, in this embodiment, the color guidance value between pixel i in the color guidance map and the j-th pixel j within its preset window is... The expression is: ; This represents the pixel value of pixel i in the color guide map; This represents the pixel value of the j-th pixel within a preset window of pixel i in the color guide map; represents the color kernel width of pixel i in the color guide diagram; exp[ ] represents an exponential function with the natural constant as the base.

[0080] Based on the color guidance value between each pixel and the j-th pixel within its preset window, it can be understood that the color guidance value measures the similarity of the color values ​​of two pixels. The smaller the difference between pixel i and the j-th pixel within its preset window, the more similar the colors of pixel i and the j-th pixel are, and the larger the corresponding color guidance value. This enhances the influence of pixel i and its neighboring pixels during filtering, which helps to smooth noise and has a higher weight when filling depth holes. At the same time, if the color kernel width of pixel i in the color guidance map is larger, even if the color difference between pixel i and its neighboring pixels is greater, the corresponding color guidance value is still larger, which helps to smooth noise.

[0081] Conversely, the greater the difference between pixel i and the j-th pixel in the preset window, the less similar the colors of pixel i and the j-th pixel are. The corresponding color guiding value is smaller, which reduces the influence of pixel i and its neighboring pixels during filtering, helps to preserve image edges and avoid over-smoothing. At the same time, if the color kernel width of pixel i in the color guiding map is smaller, the weight of pixel i will drop sharply if the color difference between pixel i and its neighboring pixels is slightly dissimilar. The corresponding color guiding value is still smaller, which helps to preserve edge details and avoid over-smoothing.

[0082] Thus, this embodiment calculates the color kernel width based on the distribution differences of pixel values ​​in the neighborhood of each pixel in the color guide map, and determines the color guide value by combining the color differences between pixels. In this way, the weights are adaptively adjusted during the filtering process, effectively preserving edge details while smoothing noise and filling holes, which helps to improve the precision and visual consistency of depth map restoration.

[0083] S2.4 Based on the color guidance value, a filtering algorithm is used to repair the scene depth map for scene rendering.

[0084] Based on the color guidance value, the scene depth map is enhanced and repaired, and then scene rendering is performed based on the repaired scene map, specifically as follows:

[0085] In this embodiment, the scene depth map is used as the input to the filtering algorithm. The color guide value between each pixel and any pixel within a preset window is used as the color guide term in the filtering algorithm. The filtering window in the filtering algorithm is set to the same size as the preset window. The kernel widths of the spatial term and the depth term are set to u and v, respectively. The output image is used as the repaired scene depth map.

[0086] It should be noted that in this embodiment, a bilateral filtering algorithm is used to enhance and denoise the scene depth map. The kernel width v of both the spatial term u and the depth term is 0.5. In actual applications, as other implementation methods, implementers can also set their own values ​​according to specific circumstances. This embodiment does not impose any special restrictions.

[0087] Among them, the bilateral filtering algorithm is a well-known technology, and the specific process of using it to denoise, enhance and repair the scene depth map will not be described in detail.

[0088] Furthermore, in this embodiment, a rasterization method is used to render the denoised and enhanced scene depth map to obtain a more realistic 3D virtual scene. The rasterization rendering method is a well-known technology, and the specific process of rendering the image using it will not be described in detail.

[0089] Thus, this embodiment achieves high-quality restoration of scene depth maps by constructing a color guide map and improving the bilateral filtering algorithm based on adaptive color kernel width and color guide value. It effectively suppresses noise interference, preserves edge details, and repairs depth holes and outliers, thereby providing more realistic, stable, and detailed depth information for 3D virtual scene rendering and significantly improving the rendering quality of VR 3D virtual scenes.

[0090] Based on the same inventive concept as the above method, this application embodiment also provides a VR three-dimensional virtual scene rendering system, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-described VR three-dimensional virtual scene rendering methods.

[0091] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments of this specification have been described above. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0092] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0093] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A VR 3D virtual scene rendering method, characterized in that, The method includes the following steps: Obtain the scene depth map of the target scene, and convert the scene depth map to two color spaces, namely RGB space and HSV space; For scene depth maps in two color spaces, wavelet transform algorithm is used to obtain the images in each sub-band and the wavelet coefficients of each pixel in the images in any color channel. In each color space, based on the average distribution and dispersion of the wavelet coefficients of all pixels in the neighborhood of each pixel in the image of each sub-band, the color feature value of each pixel in the image of each sub-band in each color space in any color channel is determined. By analyzing the similarity of the color feature values ​​of each pixel in all color channels between the images of each sub-band in the two color spaces, the color consistency of each pixel in the images of each sub-band in the two color spaces is determined. Based on the distribution of color consistency of all pixels in each sub-band image under the two color spaces, the overall color consistency of each sub-band image under the two color spaces is determined, so as to determine the reconstruction weight of each sub-band image; based on the reconstruction weight, the inverse wavelet transform algorithm is used to reconstruct the color guide map of all sub-band images in RGB space. Based on the difference between each pixel in the preset window of each pixel in the color guide map and the average distribution of all pixels, the color kernel width of each pixel is determined, and combined with the difference of pixel value between each pixel and any pixel in its preset window, the color guide value between each pixel and any pixel in its preset window is determined. The scene depth map is repaired based on the color guidance value using a filtering algorithm for scene rendering.

2. The VR three-dimensional virtual scene rendering method as described in claim 1, characterized in that, The color feature value of each pixel in the image of each sub-band under each color space in any color channel is: the ratio of the mean to the dispersion of the wavelet coefficients of all pixels in the neighborhood of each pixel in the image of each sub-band under each color space in any channel.

3. The VR three-dimensional virtual scene rendering method as described in claim 1, characterized in that, The color consistency of each pixel in the image under each sub-band in the two color spaces is the cosine similarity of the color feature values ​​of each pixel in all color channels between the images under each sub-band in the two color spaces.

4. The VR three-dimensional virtual scene rendering method as described in claim 1, characterized in that, The overall color consistency of each sub-band image in the two color spaces is the average of the color consistency of all pixels in each sub-band image in the two color spaces.

5. The VR three-dimensional virtual scene rendering method as described in claim 1, characterized in that, The reconstruction weight of each sub-band image is the ratio of the overall color consistency of each sub-band image in the two color spaces to the sum of the overall color consistency of all sub-band images in the two color spaces.

6. The VR three-dimensional virtual scene rendering method as described in claim 1, characterized in that, The process of reconstructing a color guide map by using an inverse wavelet transform algorithm based on the reconstruction weights for all sub-band images in the RGB space includes: All sub-band images in the RGB space are used as input to the inverse wavelet transform algorithm. The sub-band images are fused according to the reconstruction weights, and the output image is used as a color guide map.

7. The VR three-dimensional virtual scene rendering method as described in claim 1, characterized in that, The expression for the color kernel width of each pixel is: In the formula, This represents the width of the color kernel at pixel i in the color guide map; This represents the pixel value of the j-th pixel within a preset window of pixel i in the color guide map; This represents the average pixel value of all pixels within a preset window of pixel i in the color guide map; This indicates the number of all pixels within the preset window of pixel i in the color guide map.

8. The VR three-dimensional virtual scene rendering method as described in claim 1, characterized in that, The expression for the color guidance value between each pixel and any pixel in its neighborhood is: ; This represents the color guidance value between pixel i in the color guide map and the j-th pixel j within its preset window; This represents the pixel value of pixel i in the color guide map; This represents the pixel value of the j-th pixel within a preset window of pixel i in the color guide map; represents the color kernel width of pixel i in the color guide diagram; exp[ ] represents an exponential function with the natural constant as the base.

9. A VR three-dimensional virtual scene rendering method as described in claim 1, characterized in that, The process of repairing the scene depth map using a filtering algorithm based on the color guidance value includes: The scene depth map is used as the input to the filtering algorithm. The color guide value between each pixel and any pixel within a preset window is used as the color guide term in the filtering algorithm. The filtering window in the filtering algorithm is set to the same size as the preset window. The output image is used as the repaired scene depth map.

10. A VR three-dimensional virtual scene rendering system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements a VR three-dimensional virtual scene rendering method as described in any one of claims 1-9.

Images