Optimization method and system for multi-layer semitransparent material in virtual reality scene

Abstract

The invention discloses a method and a system for optimizing multilayer semitransparent materials in a virtual reality scene, and relates to the technical field of virtual reality rendering. Comprising the following steps: traversing a material group in a virtual reality scene, and screening out picture materials needing to be processed in the same group; classifying and identifying the materials in each group, and distinguishing static materials from dynamic materials; for the static material, executing adjacent graph merging processing, and merging adjacent sub-graphs into one graph; performing adjacent graph axis alignment processing on the dynamic material, and enabling a connecting line and orientations of central points of the two pictures to completely coincide; and outputting the processed dynamic material, the static combined graph and the isolated material, integrating, and packaging and outputting according to a target platform. According to the method, the purposes of simplifying the scene and optimizing the frame rate are achieved by combining the proximity graphs, the OIT problem is avoided, the experience quality of a user is guaranteed, and the method has good universality.

Description

Technical Field

[0001] This invention relates to the field of virtual reality rendering technology, and more specifically to an optimization method and system for multi-layered semi-transparent materials in a virtual reality scene. Background Technology

[0002] Currently, game engines are increasingly used in the XR field, with their scene rendering function serving as a core support, driving the continuous development of virtual reality experiences. However, the rendering of semi-transparent objects remains a technical challenge that the industry has not yet fully resolved. In rendering opaque objects, the depth buffer algorithm (z-buffer algorithm) has become a mature solution. By initializing the depth buffer to infinity and recording the nearest depth of each pixel position, new pixels only update the buffer and write to the color buffer when the depth is better. This efficiently handles object occlusion relationships and ensures accurate rendering results. However, semi-transparent objects, because the occluded parts still need to be visible, cannot use the z-buffer algorithm. Current game engines generally use the painter's algorithm for semi-transparent rendering—that is, sorting all semi-transparent objects from back to front before drawing, and then rendering them sequentially according to the sorting results. This process is called "order-dependent semi-transparent blending." At the same time, various order-independent semi-transparent blending (OIT) techniques have emerged in the industry. These techniques attempt to improve the semi-transparent rendering effect from different angles. Among them, per-pixel linked lists, because they can guarantee pixel-level accuracy, have been used by some creative tools for auxiliary rendering during the scene creation stage.

[0003] However, existing technologies have many insurmountable flaws: the painter algorithm not only consumes a lot of CPU computing power during the sorting process, but can only achieve object-level sorting and cannot achieve pixel-by-pixel sorting. More importantly, its sorting results will change with the camera position, causing the occlusion order of semi-transparent objects to jump. When there are a lot of semi-transparent images in the scene, the image stability is extremely poor, which seriously affects the VR experience. At the same time, the inherent properties of semi-transparent objects determine that multi-layer semi-transparent rendering will inevitably have the phenomenon of overdraw, repeatedly drawing a large number of invalid pixels, resulting in a waste of computing power. Various OIT technologies also have significant shortcomings: deep stripping and template routing are extremely slow, requiring multiple renderings and having very low practicality; while per-pixel linked lists offer high precision, their performance is insufficient in VR device scenarios, failing to meet the smoothness requirements of VR devices; adaptive transparency and weighted blending OIT, although faster, use approximate calculations, lacking accuracy and universality, unable to reproduce the effects of professional image editing software, and still suffer from inconsistent colors when viewed from different angles in the same area; pixel synchronization, as a relatively new and advanced algorithm, not only relies on specific Intel hardware and graphics drivers but also requires deep modifications to the underlying rendering logic, and the closed nature of mainstream engines like Unity makes it difficult to adapt. In addition, VR devices themselves have limited hardware capabilities but need to render high-resolution images for both eyes simultaneously, with extremely high frame rate requirements to prevent user dizziness. At the same time, functions such as spatial positioning and eye tracking further consume hardware resources, making rendering optimization even more urgent; and creative tools cannot restrict user creative behavior like game development teams do, with users often stacking large amounts of semi-transparent images to achieve specific visual effects, which undoubtedly exacerbates the difficulty of adapting existing technologies.

[0004] Therefore, developing a virtual reality multi-layer semi-transparent material optimization solution that can ensure the quality of semi-transparent rendering, avoid occlusion jumps and overdraw issues, adapt to the performance limitations of VR devices, take into account universality and ease of use, and meet the needs of large-scale creative scenarios is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention provides an optimization method for multi-layered semi-transparent materials in a virtual reality scene to overcome or at least partially solve the above problems.

[0006] In a first aspect, embodiments of the present invention provide an optimization method for multi-layered semi-transparent materials in a virtual reality scene, comprising the following steps:

[0007] Material Grouping: Traverse the material groups in the virtual reality scene and filter out the image materials that need to be processed within the same group; Material Classification: Classify and identify materials within each group, distinguishing between static and dynamic materials; Material processing: For static materials, perform adjacent image merging processing, merging adjacent sub-images into one image; for dynamic materials, perform adjacent image axis alignment processing, ensuring that the line connecting the center points of two images completely overlaps in orientation. Scene output: Output processed dynamic materials, static merged images, and isolated materials, which are then packaged and output according to the target platform.

[0008] Furthermore, in the material processing steps, the neighboring image merging process includes the following steps: Coplanarity and Connectivity Determination: Perform coplanarity determination and connectivity test on the static materials in the same group in turn, filter out the coplanar materials that are clustered together and output multiple image sets; Image collection sorting: The static materials within each image collection are sorted bidirectionally according to the design logic priority; Image parameter calculation: Calculate the pixel density of all static materials in each image set and the minimum area bounding rectangle of the final image size in turn, and determine the resolution of the final image based on the pixel density and the minimum area bounding rectangle of the final image size; Merge Baking: Create a blank canvas according to the final resolution, and write the image data of all sub-images and their corresponding position, rotation, and scaling parameters to the canvas based on the sorting results using the Unity graphics API to complete the baking. Loop and Output: Determine if all material groups have been processed; if not, return to the image collection sorting step and execute subsequent steps in sequence; if processed, replace the original aggregated static sub-image in the corresponding material group with the baked static merged image to complete the optimization.

[0009] Furthermore, the determination of coplanar materials specifically involves: calculating the included angle of normals between two material faces in the calculation space, wherein the included angle of normals... When the absolute value of the difference between the angle and 0 degrees or 180 degrees is less than a preset angle threshold, and the spacing between the materials is less than a preset distance threshold, they are determined to be coplanar. The included angle of the normals The formula used for the calculation is as follows:

[0010] Here, a and b are the vectors corresponding to the two images, and |a| and |b| are the moduli of a and b, respectively.

[0011] Furthermore, the connectivity determination specifically involves: extracting the Alpha channel of the coplanar material and performing contour extraction, and constructing a rough mesh corresponding to the contour for each coplanar material; Traverse all triangles of the rough mesh and check for overlap with the mesh triangles of other coplanar materials; if there is overlap, determine that the two coplanar materials are connected.

[0012] Furthermore, in the image parameter calculation, the formula used for pixel density calculation is as follows:

[0013] Where pixelwidth is the pixel width, pixelheight is the pixel height, scalex is the scaling factor X, and scaley is the scaling factor Y.

[0014] Furthermore, to maximize image pixel utilization, the calculation steps for the final image size in the parameter calculation are as follows: Vertex collection: Obtain the minimum area bounding rectangles corresponding to all subgraphs to be merged, calculate the four vertices of each rectangle, and form a vertex set S containing 4n vertices, where n is the number of subgraphs to be merged; Calculate the convex hull: Process the point set S using the convex hull algorithm to obtain the convex hull geometry corresponding to the point set S; Solving for the minimum area bounding rectangle: Based on the geometric property that the minimum area bounding rectangle has at least one edge that coincides with the edge of the convex hull, traverse all edges of the convex hull, filter and record the rectangle with the smallest area, which is the minimum area bounding rectangle that can enclose all subgraphs.

[0015] Furthermore, during the merge baking process, both different sides of the aggregated coplanar materials are baked to ensure the accuracy of rendering from different viewing angles.

[0016] Furthermore, in the material processing step, the nearest image axis alignment means that the nearest image of the dynamic material and the face of the dynamic material are completely parallel, and their axes are aligned.

[0017] Furthermore, the adjacent graph axis alignment includes the following steps: Analyze the dynamic materials and determine whether the dynamic materials have neighboring images. If they do, use the dynamic materials as alignment targets. The mesh of the neighboring image is enlarged so that the mesh axis of the neighboring image is aligned with the axis of the alignment target, and the texture carried by the mesh is enlarged synchronously. By passing the scaling factor of the mesh to a custom shader, the textures of the adjacent images are displayed at a reduced scale, and the excess parts of the expanded mesh are left blank and invisible.

[0018] Secondly, embodiments of the present invention provide an optimization system for multi-layered semi-transparent materials in a virtual reality scene, the system comprising: Material grouping module: Used to traverse material groups in a virtual reality scene and filter out the image materials that need to be processed within the same group; Material Classification Module: Used to classify and identify materials within each group, distinguishing between static and dynamic materials; Material processing module: For static materials, it performs a neighboring image merging process, merging adjacent sub-images into one image; for dynamic materials, it performs a neighboring image axis alignment process, ensuring that the line connecting the center points of two images completely overlaps in orientation. Scene output module: Outputs processed dynamic materials, static merged images, and isolated materials, which are then packaged and output according to the target platform.

[0019] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following: This invention merges and bakes static materials after determining their coplanarity and connectivity, eliminating viewpoint-dependent jumps caused by the object-level sorting in the painter's algorithm; dynamic materials are aligned using an axis center method, ensuring that the lines connecting the dynamic materials to the center points of nearby images and their orientations are completely aligned, thus ensuring a stable rendering order under all viewpoints, avoiding color deviations in OIT technology, and meeting the high requirements of VR immersive experience for image stability.

[0020] This invention significantly reduces the number of objects in a scene by merging static materials, thereby reducing CPU sorting overhead and overdraw caused by repeated rendering of multiple layers of semi-transparent materials, saving computing power. The solution does not require high-end hardware or complex OIT algorithms, and is compatible with performance-intensive devices such as VR headsets and AR glasses, ensuring smooth operation even in high-resolution, high-frame-rate scenarios for both eyes.

[0021] This invention merges static materials to determine the resolution based on the maximum pixel density within the set, avoiding loss of detail; it calculates the minimum area bounding rectangle to maximize pixel utilization and reduce redundant consumption; it preserves the position, rotation, and scaling parameters of the sub-images during merging and synthesizes both the front and back sides to ensure visual accuracy from different perspectives; it supports users to freely stack materials without restricting creative guidelines, balancing flexibility and image quality.

[0022] This invention covers mainstream static and dynamic graphic material types; it optimizes the process by embedding the packaging step, supports multi-platform output, adapts to different terminals, and can handle large-scale VR creation scenarios.

[0023] This invention automates operations such as mesh enlargement and texture scaling using the axis alignment method, replacing manual alignment. The overall solution requires no user intervention and automatically completes material optimization and scene integration without changing creative habits, reducing operating costs and improving the efficiency of VR content production. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0025] Figure 1 This is a flowchart of the z-buffer algorithm provided in an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram of the rendering order provided in an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of the critical surface positioning provided in an embodiment of the present invention.

[0028] Figure 4 This is a rendering effect of a two-color semi-transparent plane overlay provided in an embodiment of the present invention.

[0029] Figure 5 This is a schematic diagram of the image rendering process provided in an embodiment of the present invention.

[0030] Figure 6 This is a scene diagram provided in an embodiment of the present invention.

[0031] Figure 7 This is a schematic diagram of the floor composition provided in an embodiment of the present invention.

[0032] Figure 8 This is an overall flowchart provided in an embodiment of the present invention.

[0033] Figure 9 This is a flowchart of static material processing provided in an embodiment of the present invention.

[0034] Figure 10 This is a coplanarity determination diagram provided in an embodiment of the present invention.

[0035] Figure 11 This is a schematic diagram of clustering and disconnection provided in an embodiment of the present invention.

[0036] Figure 12 This is a schematic diagram of contour extraction provided in an embodiment of the present invention.

[0037] Figure 13 This is a schematic diagram of axis alignment provided in an embodiment of the present invention.

[0038] Figure 14 This is a schematic diagram of the axis alignment process provided in an embodiment of the present invention.

[0039] Figure 15 This is a system flowchart provided in an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] Example 1: Game engines have become increasingly powerful, evolving from simple to complex, and from single-platform to multi-platform, with their influence extending beyond the gaming industry to play a significant role in the XR field. However, regardless of the game engine, scene rendering, as a core function, always presents some unresolved issues.

[0042] Most people outside the industry would assume that rendering a large number of glasses is as natural as air. However, rendering semi-transparent objects is actually a surprisingly challenging problem. The root of the problem lies in the combination of depth testing and color blending, which contrasts sharply with rendering opaque objects. For opaque objects, the core logic is "black and white"—only the nearest surface is visible. The z-buffer algorithm was proposed. Simply put, we initialize the depth buffer to infinity, and each pixel only records the nearest depth at that location. If a newly inserted pixel is closer in depth than the current one, the value is updated, and the new pixel value is written to the color buffer.

[0043] Please refer to Figure 1 This is a flowchart of the z-buffer algorithm. It begins by rendering a single object, converting 3D geometry into pixels on the screen through rasterization, and calculating color and depth values ​​for each pixel. The process then moves to the core depth testing stage, where the existing depth value at the current pixel location is read from the depth buffer and compared with the depth value of the new pixel. If the new pixel's depth value does not meet the condition (i.e., it's farther from the camera), its color information is discarded. If the test is passed (i.e., it's closer to the camera), the new pixel's color is written to the color buffer, and the depth value at that location in the depth buffer is updated simultaneously. The purpose of the entire process is to ensure that only pixels closer to the viewer are retained and displayed in the final image, thus correctly resolving object occlusion issues in 3D scenes. The z-buffer algorithm handles object occlusion relationships well, and the final result is correct regardless of whether opaque objects are rendered from near to far or far to near, making it a core solution for rendering opaque objects.

[0044] However, rendering semi-transparent objects is much more complex; please refer to [the documentation / reference]. Figure 2Different rendering orders result in different colors. Because occluded objects can still be seen, they cannot be simply discarded. Therefore, the z-buffer algorithm cannot be used.

[0045] Currently, game engines generally use the painter's algorithm to handle the rendering of semi-transparent objects. Its core logic is to perform a global sort of all semi-transparent objects from back to front before rendering the scene, and then render them sequentially according to the sorting results. However, this approach has significant limitations: the sorting process of semi-transparent objects itself consumes a lot of CPU computing power, and due to the limitations of the algorithm, it can only achieve object-level sorting, not pixel-level precise sorting. This means that the algorithm can only solve about 50% of the semi-transparent rendering problems. After all, for semi-transparent objects, the rendering order directly determines the final color blending effect, and even a small sorting deviation can lead to an unexpected visual presentation.

[0046] The painter's algorithm also exhibits a phenomenon that is difficult for the average person to understand intuitively; please refer to [the relevant documentation / reference]. Figure 3 The black dot represents the center point of the image, and the two cameras on the left and right represent two viewing angles. It can be observed that the sorting result is calculated based on the camera viewpoint. When the camera position changes, the sorting result of the semi-transparent objects will change. It appears that the occlusion order between objects jumps as the camera position changes.

[0047] Please refer to Figure 4 Viewed from the left, the green image is closer to the camera; viewed from the right, the red image is closer. This shows that the center point order of the image changes depending on the viewing position, thus altering the semi-transparent blending effect. The critical plane lies between the center points of the two images, perpendicular to the line connecting them. To the left of the critical plane, the green image is closer; to the right, the red image is closer. Moving near the critical plane makes the abrupt transition easily observable.

[0048] Such rendering issues occur frequently when there are many semi-transparent images in the scene, resulting in very unstable and confusing visuals that reduce the quality of the VR experience.

[0049] However, the challenges of semi-transparent rendering do not end there. Another core issue is overdraw. Since the computing power of the device is limited, in order to avoid waste, the ideal rendering situation is that each pixel on the screen is calculated only once.

[0050] Please refer to Figure 5The rendering process is shown from left to right. Assuming there are three walls, the furthest red wall is rendered first. Then the second green wall is rendered, at which point most of the first red wall is obscured; although the result is correct, most pixels are covered. Finally, the third blue wall is rendered, again obscuring most of the second green wall. During this process, pixels in the central area of ​​the screen are calculated three times, but only the last calculation is visible. The first two calculations consume significant computing power but contribute nothing to the final image.

[0051] If the rendering order is reversed, and nearby objects are rendered first, the pixels of later objects will not pass through the z-buffer algorithm and will not be output to the screen. Therefore, although the final image is the same regardless of the rendering order, the latter saves computing power.

[0052] However, the inherent properties of semi-transparent objects mean that all pixels of all objects must be calculated to get the correct image. Therefore, no matter how the order is adjusted, multiple layers of semi-transparent objects cannot avoid overdraw.

[0053] Moreover, current VR headsets and AR glasses lack powerful hardware capabilities. Furthermore, VR glasses need to render images for both eyes simultaneously, requiring a higher resolution than typical mobile phone screens; for example, the Pico 4U boasts a single-eye resolution of 2160. 2160, both eyes are 4320 2160, but still insufficient for the human eye. Dual-lens 8K means 4 times the pixels and 4 times the rendering pressure; to avoid dizziness, VR applications also require a higher frame rate than mobile phones; the glasses must also constantly perform spatial positioning calculations, and many new devices also need to support eye tracking.

[0054] It's fair to say that VR glasses themselves consume a lot of resources, and coupled with the immersive VR experience, the human eye is more sensitive to image imperfections and frame rate fluctuations. Therefore, the need for optimization in VR glasses is far more urgent than in smartphones with comparable hardware.

[0055] This optimization challenge is further exacerbated by the usage scenarios, where users may pile up a large number of images to achieve the desired visual effect. A common production scenario is... Figure 6 As shown, the floor of the room appears to be a single piece, but it is actually composed of many small images stacked together, such as... Figure 7 As shown, each quadrilateral wireframe represents an image.

[0056] To solve the core challenges of semi-transparent rendering, a number of well-known OITs have emerged in the industry over the past two decades.

[0057] Table 1. Specific parameters of OIT technology

[0058] Among them, deep stripping and template routing are the slowest, requiring multiple renderings.

[0059] Per-pixel linked lists represent the closest to ideal rendering solution and are one of the core technologies used in the MRCreator editing tool. The key advantage of this technology is its ability to guarantee the accuracy of pixel rendering, but its rendering speed still has room for improvement. On hardware devices equipped with an RTX 1060 graphics card, smooth rendering can be achieved at 720p resolution. However, when experiencing this on a PC platform via SteamVR streaming, even on hardware devices equipped with an RTX 3080 graphics card, rendering only an empty scene results in a frame rate dropping below 72fps. Therefore, this technology cannot be directly applied to VR devices and can only be used as an auxiliary means in the scene creation stage.

[0060] Adaptive transparency and weighted blending (OIT) are faster than pixel-by-pixel linked lists, but they are both approximate calculations. They lack accuracy and universality in complex scenes, cannot reproduce the effects in Photoshop, and the same area will still appear to have different colors when viewed from different angles, thus failing to achieve the goal of improving the VR experience.

[0061] Pixel synchronization is a newly introduced advanced algorithm, but it requires relatively new hardware; only Intel hardware and graphics drivers support this feature. Another issue is that the algorithm requires significant modifications to the underlying rendering logic, and Unity, being a closed system that is difficult to modify, is not open source.

[0062] In summary, all existing sequence-independent semi-transparent blending technologies have irreconcilable shortcomings. They either cannot keep up with the high-performance requirements of VR devices in terms of speed, or they have defects in rendering accuracy and universality, or they are limited by hardware compatibility or engine closedness and cannot be implemented. They cannot simultaneously meet the comprehensive requirements of rendering quality, running efficiency and adaptability in VR scenarios.

[0063] Please refer to Figure 8 To address the aforementioned problems, this invention discloses an optimization method for multi-layered semi-transparent materials in virtual reality scenes, comprising the following steps: (1) Material grouping: Traverse the material groups in the virtual reality scene and filter out the image materials that need to be processed within the same group.

[0064] Once the scene is created, the packaging process is triggered. All materials in the scene are traversed and grouped according to the same creative level. Only materials within the same group are processed subsequently. The boundaries of material processing are clearly defined to avoid cross-interference between materials from different creative levels, ensuring a targeted and focused optimization process. This reduces the judgment and computational overhead of irrelevant materials, improving optimization efficiency. It aligns with the creative logic, ensuring visual continuity after optimization of materials at the same level and avoiding visual inconsistencies caused by cross-level optimization.

[0065] (2) Material Classification: The materials within each group are classified and identified, distinguishing between static and dynamic materials. Static materials are images without dynamic frames or viewpoint switching logic, while dynamic materials include videos, GIFs, and 3D images. Optimization schemes are matched differently based on the characteristics of the materials to avoid compatibility issues caused by uniform processing. The scheme accurately covers the mainstream planar material types in VR scenes, ensuring that both static and dynamic materials receive appropriate optimization strategies. Clear material definitions reduce classification ambiguity and improve the accuracy and stability of the scheme implementation.

[0066] (3) Material processing: 1) For static assets, perform proximity image merging to combine adjacent sub-images into one image. This optimization process first determines the coplanarity and connectivity of static assets in the same group, filtering out coplanar assets to form multiple image sets. Then, the assets in each set are bidirectionally sorted according to design logic priority. The final image resolution is then determined by calculating pixel density and the minimum area bounding rectangle. Subsequently, a canvas is created according to the resolution, and the Unity graphics API is used to write all sub-image data and related parameters according to the sorting results to complete the baking. Finally, all asset groups are processed in a loop. After all are completed, the original clustered sub-images are replaced with the baked static merged image to achieve optimized output.

[0067] Static preprocessing addresses the pain points of semi-transparent rendering while optimizing VR device performance. It eliminates viewpoint-dependent rendering jumps caused by traditional painter algorithms, ensuring the stability of the image in the VR immersive experience. It significantly reduces the number of objects in the scene, lowers CPU sorting overhead and overdraw phenomenon of multi-layer semi-transparent materials, saves computing power, and adapts to the high-resolution, high-frame-rate performance requirements of VR devices. The resolution is determined by the maximum pixel density and the minimum area bounding rectangle, preserving material details and visual consistency, and supporting users to freely stack and create, balancing creative flexibility and image quality.

[0068] 2) Perform adjacent image axis alignment processing on dynamic materials, making the line connecting the center points of the two images completely coincide with their orientation; count the dynamic materials and determine whether there are adjacent images. If there are, use the dynamic material as the alignment target; expand the adjacent image grid so that its axis is aligned with the axis of the alignment target, and simultaneously enlarge the texture carried by the grid; pass the grid expansion ratio through a custom shader, shrink the adjacent image texture according to the ratio, and leave the excess part of the expanded grid empty and invisible, thus completing the adjacent image axis alignment.

[0069] Addressing the characteristics of dynamic materials, this technology avoids the rendering defects of OIT technology, achieving low-cost and highly stable rendering optimization. It ensures the stable rendering order of dynamic materials and adjacent images under all viewing angles, completely resolving color deviation and jump issues caused by changes in viewing angle. It automates operations such as mesh enlargement and texture scaling, replacing tedious manual alignment and reducing user operation costs. The processing does not change the visual effect and does not rely on high-end hardware or complex algorithms, making it compatible with performance-intensive VR devices and highly practical.

[0070] (4) Scene Output: Replace the original corresponding sub-images with static merged images, retain the processed dynamic materials, static merged images, and isolated materials, and package and output them according to the target platform after integration. Complete the scene integration after material optimization to achieve the final delivery of multi-platform adaptation, ensure the integrity of the optimized scene, and retain the functions and visual effects of all effective materials; support output to multiple target platforms such as PC, PCVR, Android phones, and Pico all-in-one machines, adapt to the usage needs of different terminals, and highlight universality and scalability; optimize the process and embed it into the packaging process, without the need for additional user intervention, and improve the production and delivery efficiency of VR architectural roaming content.

[0071] This invention achieves efficient optimization of semi-transparent materials in VR scenes through a complete processing flow for static and dynamic materials: First, scene materials are traversed and grouped according to the same creative level, clarifying processing boundaries to avoid cross-interference, reduce invalid calculations, and ensure visual continuity at the same level; then, each group of materials is classified and identified as static or dynamic, ensuring that materials with different characteristics are matched with optimization strategies, improving the accuracy of the solution execution. For static materials, materials are filtered and grouped into sets based on coplanarity and connectivity, and after bidirectional sorting, the resolution is determined by combining the maximum pixel density and the minimum area bounding rectangle. The materials are then baked into a merged image using the Unity graphics API and replaced with the original materials. This eliminates the rendering jumps of traditional algorithms, preserves image details and creative flexibility, and simplifies the scene by merging adjacent images, reduces the number of objects and overdraw phenomenon, optimizes the frame rate, and adapts to the performance requirements of VR devices. For dynamic materials, an automated axis alignment method is used to expand the adjacent image mesh and adjust the texture display through a custom shader, ensuring stable rendering under all perspectives, avoiding the defects of OIT technology, and reducing operating costs. The final integrated and optimized materials support multi-target platform packaging and output. This invention avoids OIT problems, ensures the quality of user experience, and has low equipment requirements, making it highly versatile and adaptable to increasingly large-scale creative scenarios in the future. Without requiring additional user intervention or changing creative habits, it balances VR experience stability, image quality, and device compatibility, significantly improving the production and delivery efficiency of VR architectural walkthrough content. Its practicality, versatility, and scalability are outstanding.

[0072] The above technical solution will be further explained in detail below with reference to specific embodiments of static materials: To address the core requirement of VR devices to simultaneously balance image quality and performance in scene rendering, this invention focuses on the practical problems of existing semi-transparent rendering technologies, such as rendering jumps, high overdraw consumption, and insufficient adaptability, and proposes a specialized optimization scheme for semi-transparent static materials in VR scenes—nearest neighbor image merging technology.

[0073] Compared to existing OIT technologies, the technical solution of this invention falls under the category of static preprocessing. Specifically, this invention merges adjacent sub-images that meet preset conditions in a VR merged image scene, integrating multiple adjacent sub-images into a single merged image. This fundamentally solves the rendering jump problem in the rendering of semi-transparent materials, and maximizes the integrity of image details and visual presentation quality. Simultaneously, this merging process significantly reduces the number of objects in the scene, thereby reducing the computational overhead of object sorting in the merged image CPU, effectively reducing the overdraw phenomenon caused by multiple layers of semi-transparent materials, and significantly reducing device computing power consumption. Therefore, it is particularly suitable for performance-intensive terminal devices such as VR headsets and AR glasses, and can well adapt to the application requirements of such devices for high resolution, high frame rate, and high basic resource consumption for both eyes. Please refer to [reference needed]. Figure 9 The specific nearest neighbor graph merging includes the following steps: S1: Packaging and Collecting Materials: Once the scene is completed, the packaging process is triggered, and the collection step is entered to collect all static material images to be processed.

[0074] Once a user completes the creation of a VR scene, the packaging process of this invention can be triggered to generate an independent running program for the user to immerse themselves in the scene content. The MRCreator tool adapted for this invention supports multi-target platform packaging output: for PC and PCVR platforms, a compressed file is generated after packaging; after decompressing the compressed file, the user can double-click the executable program (.exe) to launch and view the scene; for Android phones and Pico all-in-one devices, an installation package (.apk) is generated after packaging; after installing the package on the corresponding terminal device, the user can launch the application and enter the scene to view it.

[0075] S2: Coplanarity and Connectivity Determination: Perform coplanarity determination and connectivity test on the static materials in the same group in turn, filter out the coplanar materials that are clustered together and output multiple image sets.

[0076] The nearest neighbor image merging program is integrated into the VR scene packaging process, and has a preset judgment rule: only images belonging to the same creation level will participate in coplanarity and connectivity judgment. For each creation level, the program first traverses and filters all images under that level, and then performs coplanarity and proximity judgments sequentially according to preset logic. The coplanarity judgment satisfies two conditions: co-direction and proximity. The co-orientation determination refers to the fact that two images have the same orientation. The determination method is as follows: each image corresponds to a normal perpendicular to its own center. The angle θ between the corresponding normals of the two images in space is calculated, and this angle θ represents the orientation angle between the two images. When the angle θ meets the requirements, the two images are determined to be co-oriented. For example... Figure 10 To illustrate, since the normal only indicates direction and not position, it can be moved to the origin, and the position before and after the movement is equivalent.

[0077] The formula used for the calculation is as follows:

[0078] Where a and b are the vectors corresponding to the two images, and |a| and |b| are the moduli of a and b, respectively. When the absolute value of the difference between the included angle of the normal and 0 degrees or 180 degrees is less than a preset angle threshold (e.g., 0.1°), and the spacing between the materials is less than a preset distance threshold (e.g., 0.1cm), they are determined to be coplanar.

[0079] The proximity determination process involves considering two images already identified as co-oriented. If the spatial distance between them is ≤0.1cm, they are considered proximity. This process may result in multiple coplanar images, such as the floor and ceiling, which, although both are horizontally oriented, are not proximity.

[0080] like Figure 11 As shown, for the set of coplanar images obtained by the coplanarity determination, a connectivity test needs to be performed to divide the spatially distant independent clusters. The connectivity determination is based on whether the grid boundaries of the images are in contact with each other. If the grid boundaries of multiple images are in contact with each other, they are determined to be the same cluster; otherwise, they are divided into different clusters.

[0081] Through the connectivity test described above, a set of coplanar images can be decomposed into multiple independent image clusters. For example, a floor composed of multiple clusters of far-away sub-images will be divided into multiple independent clusters. Subsequently, each independent cluster will be merged to ensure the rationality and optimization effect of the merged image.

[0082] like Figure 12 As shown, given the semi-transparent nature and irregular regions of PNG images, existing connectivity determination methods based on rectangular grids are prone to misjudgment. That is, the rectangular grid may appear connected, but the visual content may actually be separated. To solve this problem, we use an outline grid instead of rectangles for determination, specifically including the following steps: S201: Contour Extraction and Coarse Mesh Construction: Extract the Alpha channel of each PNG image, and use the Teh-Chin chain approximation algorithm in the OpenCV library or the built-in contour extraction algorithm in Unity to obtain the contour information of the visually effective area of ​​each image; based on the contour information, construct a coarse mesh for each image that roughly matches the visually effective area.

[0083] S202: Triangle overlap detection: Traverse all triangle units in the above coarse grid, and perform overlap detection on each triangle unit in the coarse grid corresponding to the image to be judged, so as to accurately determine the actual connectivity between the images.

[0084] By replacing rectangular grids with outline grids as described above, the actual visual connectivity of PNG images can be accurately identified, avoiding misjudgments of connectivity caused by the semi-transparent nature of images or irregular areas, thus ensuring the accuracy of subsequent aggregation, grouping, and merging processes.

[0085] S3: Image Collection Sorting: Sort the static materials in each image collection bidirectionally according to the design logic priority.

[0086] After completing the above preparations, we will perform baking and merging processing on each individual image cluster in sequence, with bidirectional sorting being the first step in the merging process.

[0087] Since the rendering order of semi-transparent materials directly determines the accuracy of the visual presentation, a clear merging and sorting rule must be determined for each image group. This sorting rule supports a two-way sorting mode: with... Figure 7 Taking the floor-type scene material shown as an example, it can be sorted in the reverse order of top to bottom or bottom to top. Both sorting methods can satisfy the correctness of the rendering logic. However, in actual application scenarios, it is usually difficult for tourists to observe the area under the floor. Therefore, one sorting direction corresponds to the visual presentation effect expected by the design, that is, the design is correct. For scene materials such as guide panels that allow users to move and observe freely, both the front and back sides of the material may be observed. In this case, bidirectional sorting can ensure that the visual presentation of both the front and back sides of the material meets the design requirements and ensures the consistency of the rendering effect under different viewing angles.

[0088] S4: Image parameter calculation: Calculate the pixel density of all static materials in each image set and the minimum area bounding rectangle of the final image size in sequence, and determine the resolution of the final image based on the pixel density and the minimum area bounding rectangle of the final image size.

[0089] In VR scene creation, creators often struggle to ensure uniform resolution of the images placed. In practice, scenarios frequently involve using low-resolution images as a large background and high-resolution smaller images to refine details. To guarantee the quality of the composite image after merging multiple images, this invention uses the highest resolution among the images as a benchmark to determine the parameters of the final composite image. Specifically, this is measured using a pixel density index, with units of pixels per square centimeter (pixel / cm²).

[0090] Let the resolution of a single image be (pixelwidth, pixelheight), and the scaling parameters of the image in a 3D scene be (scalex, scaley).

[0091] Then the pixel density of the image Calculate using the following formula:

[0092] Where pixelwidth is the pixel width, pixelheight is the pixel height, scalex is the scaling factor X, and scaley is the scaling factor Y.

[0093] After calculating the pixel density of all images within the same image cluster using the above formula, the maximum pixel density value is selected as the pixel density standard of the final image after merging the cluster. This ensures that the merged image can retain the detail information of the high-definition image to the greatest extent and guarantees the visual presentation quality of the synthesized image.

[0094] To maximize image pixel utilization and avoid pixel waste, it is necessary to calculate the minimum area rectangle that can completely enclose all subgraphs within the same image set, which serves as the basic outline of the final merged image. Since the geometric essence of each subgraph is a rectangle, and each rectangle can be uniquely represented by four vertices, the problem can be simplified to: finding the minimum area rectangle that can enclose the point set formed by the vertices of all subgraphs.

[0095] This invention employs the rotating caliper method to solve for the minimum area bounding rectangle mentioned above. The specific algorithm steps are as follows: Vertex collection: For each sub-image rectangle within the same image cluster, obtain its geometric parameters, including center coordinates (cx, cy), width w, height h, and rotation angle θ. Calculate the coordinates of the four vertices of each sub-image rectangle based on these geometric parameters. All vertices of the sub-images together form a vertex set S, with a total of 4n vertices in set S, where n is the number of sub-images within the cluster. Calculate the convex hull: The convex hull algorithm is used to process the vertex set S to obtain the convex hull geometry that can enclose all vertices. This convex hull provides the core geometric foundation for the subsequent solution of the minimum area bounding rectangle. Solution for rotating calipers: Based on the geometric properties of convex hulls, namely that the minimum area bounding rectangle has at least one edge that coincides with a certain edge of the convex hull, we traverse all edges of the convex hull, construct the corresponding bounding rectangle for each edge and calculate its area, and filter and record the bounding rectangle with the smallest area, which is the target minimum area bounding rectangle.

[0096] This solution can accurately obtain the minimum area rectangle enclosing all sub-images, minimizing invalid pixel areas and improving pixel utilization while ensuring that all sub-images are completely included. Furthermore, the algorithm has sufficient execution efficiency in practical applications to meet the real-time requirements of VR scene packaging optimization. Combining the previously determined highest pixel density, the resolution parameters of the final merged image can be calculated by multiplying the area of ​​the minimum area rectangle enclosing the highest pixel density.

[0097] S5: Merge Baking: Create a blank canvas according to the final resolution. Based on the sorting results, write the image data of all sub-images and their corresponding position, rotation, and scaling parameters to the canvas using the Unity graphics API to complete the baking.

[0098] Based on the final image resolution determined earlier, a blank image of the corresponding size is created as a baking "canvas". Then, according to the preset bidirectional sorting result, the image data of all sub-images in the same image cluster, as well as the position coordinates, rotation angle and scaling parameters of each sub-image on the canvas, are written into the blank canvas in sequence. The baking process is then performed through the Unity graphics API to complete the integration of all sub-images into a single large image.

[0099] For both the front and back sides of the image cluster, the program will perform the baking and compositing operation described above to ensure that the visual presentation of the front and back sides of the merged large image meets the design requirements from different viewing angles, thus ensuring the correctness and consistency of the double-sided rendering effect of the image.

[0100] S6: Loop and Output: Finally, determine whether all material groups have been processed; if not, return to the image collection sorting step and execute the subsequent steps in sequence; if processed, replace the original aggregated static sub-image in the corresponding material group with the baked static merged image to complete the optimization.

[0101] In VR scenarios, besides the aforementioned bakeable static 2D materials, there are also dynamic materials such as videos, GIFs, and 3D images. 3D images achieve stereoscopic visual effects through left-right eye parallax. Because these dynamic materials have dynamic frame playback or perspective switching logic, the baking and merging schemes used for static materials cannot be applied. For example, creators often overlay semi-transparent images in front of videos to simulate screen reflections; baking would destroy the original functionality of the dynamic materials. Therefore, this invention provides a non-baking optimization scheme for axis alignment to solve the OIT (Optical In-Time) problem for this type of dynamic material.

[0102] The following detailed description of specific embodiments of dynamic materials further illustrates the above technical solution: against Figure 3 The invention illustrates the rendering and sorting principle of semi-transparent materials. It provides a material arrangement scheme that ensures the stability of rendering results from all viewing directions. For example... Figure 13 As shown, when two materials to be processed simultaneously meet the dual conditions of "the center point line coincides" and "the orientation is completely consistent", their corresponding critical surface will be located between the two materials. At this time, no matter how the viewing angle changes, all viewing directions are on the same side of the critical surface, thus keeping the rendering sorting result of the materials constant and effectively avoiding the occurrence of sorting jump problems.

[0103] like Figure 14 As shown, after adopting this arrangement scheme, the green image appears closer in all forward viewing angles, and the red image appears closer in all backward viewing angles, with no rendering jumps throughout.

[0104] However, this arrangement has strict application limitations, requiring the two images to be perfectly parallel and their axes precisely aligned. Manual operation is not only cumbersome and complex but also has poor adaptability. To balance image quality, device speed, and ease of use, this invention performs automated optimization of the arrangement to improve its adaptability. The specific implementation steps are as follows: S400: Analyze all dynamic assets and determine if there are any adjacent images. If so, treat the dynamic asset as the alignment target.

[0105] S500: Expand the mesh of all adjacent images and align it with the axis of the dynamic material. At this time, the texture carried by the mesh will also be enlarged synchronously.

[0106] S600: Using a custom shader, the image texture is scaled down proportionally by passing in the mesh enlargement ratio, while the rest remains completely invisible. Although the image mesh has been enlarged, certain rendering techniques ensure consistent visual effects before and after processing.

[0107] While this axis alignment optimization solution does not simplify VR scenes, it can ensure the correct rendering of dynamic materials and corresponding semi-transparent materials with low computing power consumption and implementation cost, effectively avoiding OIT problems. At the same time, it eliminates the need for users to manually perform material alignment operations, significantly reducing the complexity of user creation operations and saving creative effort.

[0108] Example 2: like Figure 15 As shown, based on the same inventive concept, this embodiment of the invention also provides an optimization system for multi-layered semi-transparent materials in a virtual reality scene, the system comprising: Material grouping module: Used to traverse material groups in a virtual reality scene and filter out the image materials that need to be processed within the same group; Material Classification Module: Used to classify and identify materials within each group, distinguishing between static and dynamic materials; Material processing module: For static materials, it performs a neighboring image merging process, merging adjacent sub-images into one image; for dynamic materials, it performs a neighboring image axis alignment process, ensuring that the line connecting the center points of two images completely overlaps in orientation. Scene output module: Outputs processed dynamic materials, static merged images, and isolated materials, which are then packaged and output according to the target platform.

[0109] Since these devices and the principles they address are similar to the optimization methods for multi-layered semi-transparent materials in the aforementioned virtual reality scenarios, the implementation of these devices can refer to the implementation of the aforementioned methods, and the repetitions will not be repeated.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0111] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for optimizing multi-layered semi-transparent materials in a virtual reality scene, characterized in that, Includes the following steps: Material Grouping: Traverse the material groups in the virtual reality scene and filter out the image materials that need to be processed within the same group; Material Classification: Classify and identify materials within each group, distinguishing between static and dynamic materials; Material processing: For static materials, perform adjacent image merging processing, merging adjacent sub-images into one image; for dynamic materials, perform adjacent image axis alignment processing, ensuring that the line connecting the center points of two images completely overlaps in orientation. Scene output: Output processed dynamic materials, static merged images, and isolated materials, which are then packaged and output according to the target platform.

2. The method as described in claim 1, characterized in that, The material processing steps include the following steps for merging adjacent images: Coplanarity and Connectivity Determination: Perform coplanarity determination and connectivity test on the static materials in the same group in turn, filter out the coplanar materials that are clustered together and output multiple image sets; Image collection sorting: The static materials within each image collection are sorted bidirectionally according to the design logic priority; Image parameter calculation: Calculate the pixel density of all static materials in each image set and the minimum area bounding rectangle of the final image size in turn, and determine the resolution of the final image based on the pixel density and the minimum area bounding rectangle of the final image size; Merge Baking: Create a blank canvas according to the final resolution, and write the image data of all sub-images and their corresponding position, rotation, and scaling parameters to the canvas based on the sorting results using the Unity graphics API to complete the baking. Loop and Output: Determine if all material groups have been processed; if not, return to the image collection sorting step and execute subsequent steps in sequence; if processed, replace the original aggregated static sub-image in the corresponding material group with the baked static merged image to complete the optimization.

3. The method as described in claim 2, characterized in that, The determination of coplanar materials specifically involves: The angle between the normals of two material faces in computational space is calculated. When the absolute value of the difference between the angle and 0 degrees or 180 degrees is less than a preset angle threshold, and the spacing between the materials is less than a preset distance threshold, they are determined to be coplanar. The included angle of the normals The formula used for the calculation is as follows: Here, a and b are the vectors corresponding to the two images, and |a| and |b| are the moduli of a and b, respectively.

4. The method as described in claim 2, characterized in that, The connectivity determination specifically refers to: Extract the Alpha channel of the coplanar material and perform contour extraction, and construct a rough mesh corresponding to the contour for each coplanar material; Traverse all triangles of the rough mesh and check for overlap with the mesh triangles of other coplanar materials; if there is overlap, determine that the two coplanar materials are connected.

5. The method as described in claim 2, characterized in that, The formula used for calculating pixel density in the image parameter calculation is as follows: Where pixelwidth is the pixel width, pixelheight is the pixel height, scalex is the scaling factor X, and scaley is the scaling factor Y.

6. The method as described in claim 2, characterized in that, The specific steps for calculating the final image size in the parameter calculation are as follows: Vertex collection: Obtain the minimum area bounding rectangles corresponding to all subgraphs to be merged, calculate the four vertices of each rectangle, and form a vertex set S containing 4n vertices, where n is the number of subgraphs to be merged; Calculate the convex hull: Process the point set S using the convex hull algorithm to obtain the convex hull geometry corresponding to the point set S; Solving for the minimum area bounding rectangle: Based on the geometric property that the minimum area bounding rectangle has at least one edge that coincides with the edge of the convex hull, traverse all edges of the convex hull, filter and record the rectangle with the smallest area, which is the minimum area bounding rectangle that can enclose all subgraphs.

7. The method as described in claim 2, characterized in that, During the merge baking process, both different sides of the aggregated coplanar material are baked to ensure the accuracy of rendering from different viewing angles.

8. The method as described in claim 1, characterized in that, In the material processing step, the nearest image axis alignment means that the nearest image of the dynamic material and the face of the dynamic material are completely parallel and their axes are aligned.

9. The method as described in claim 8, characterized in that, The alignment of adjacent graph axes includes the following steps: Analyze the dynamic materials and determine whether the dynamic materials have neighboring images. If they do, use the dynamic materials as alignment targets. The mesh of the neighboring image is enlarged so that the mesh axis of the neighboring image is aligned with the axis of the alignment target, and the texture carried by the mesh is enlarged synchronously. By passing the scaling ratio of the mesh to a custom shader, the textures of the adjacent images are scaled down proportionally, and the excess parts of the expanded mesh are left blank and invisible.

10. An optimization system for multi-layered semi-transparent materials in a virtual reality scene, characterized in that, The system comprises: Material grouping module: Used to traverse material groups in a virtual reality scene and filter out the image materials that need to be processed within the same group; Material Classification Module: Used to classify and identify materials within each group, distinguishing between static and dynamic materials; Material processing module: For static materials, it performs a neighboring image merging process, merging adjacent sub-images into one image; for dynamic materials, it performs a neighboring image axis alignment process, ensuring that the line connecting the center points of two images completely overlaps in orientation. Scene output module: Outputs processed dynamic materials, static merged images, and isolated materials, which are then packaged and output according to the target platform.

Images