A lightweight and fine-grained multi-threaded scheduling method and its application for Web3D

Abstract

The present invention discloses a Web3D lightweight fine-grained multi-thread scheduling method and application, and relates to the technical field of online real-time visualization of web pages in ultra-large-scale scenes. A server backend progressively calculates a first batch of visible sets, a second batch of visible sets and a potential visible set based on a visible association scene graph. When the first batch of visible sets is calculated, the server backend transmits it to a child thread, and the child thread loads and parses it, and the main thread renders it. When the second batch of visible sets is calculated, the server backend transmits it to a child thread, and the child thread loads and parses it, and the main thread renders it. When the potential visible set is calculated, the server backend transmits it to a child thread, and the child thread loads and parses it. Through visible association scene graph construction, progressive calculation and multi-threaded loading, parsing and rendering, the technical bottleneck of online display of ultra-large-scale scenes on the Web side can be broken through, and the method is suitable for online real-time visualization of web pages in ultra-large-scale scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of online real-time visualization of web pages in ultra-large-scale scenes, and in particular to a Web3D lightweight fine-grained multi-threaded scheduling method and application for ultra-large-scale scenes. Background Art

[0002] With the rapid rise of the concept of the metaverse in the market, the continuous development of digital technology has promoted the construction of a 3D immersive world, and further promoted the possibility of the metaverse becoming the next generation of the Internet. New concepts and new formats, such as the metaverse and digital assets, have quickly exploded in the global market and become the focus of much attention. In this context, Web3D technology, as a network extension of 3D technology, has significantly enriched the way information is expressed and presented on the Web.

[0003] Web3D technology not only increases user interactive operations and immersive experience, but also greatly improves the quality of user experience in obtaining network information by improving the display of objects. This technology provides strong support for the realization of the metaverse and brings the digital virtual world into the user's field of vision. Web3D technology provides a lightweight and convenient three-dimensional object representation method, providing powerful technical means and more detailed data supplements for the implementation of the metaverse, thus promoting the rise of the Web3D metaverse.

[0004] However, with the continuous expansion of application scenarios and the rapid increase in the scale of the metaverse, existing Web3D technologies face many challenges when facing ultra-large-scale scenarios. The factors affecting the application of the Web3D metaverse can be summarized as follows:

[0005] First, the amount of data is huge. In Web3D projects, there are a large number of components in the scenes that need to be observed, and the models in large-scale scenes bring a huge amount of data. These components are repetitive and similar, and can be transmitted once on the server side and then referenced multiple times on the Web side, thereby significantly reducing the amount of data without losing data accuracy. Therefore, the first task is to analyze and extract the features of the scene, calculate simple occlusion bodies, and then store them with fine-grained instantiation information. This step is a necessary prerequisite for reducing the "data volume" of the scene.

[0006] Second, the scene loads slowly. In large-scale scenes, the granularity of data access is too coarse. For example, the data volume of a building model file usually reaches hundreds of megabytes, and the current Internet bandwidth (even 5G) seems a bit "struggling". Therefore, it is necessary to propose an index scheduling scheme suitable for ultra-large-scale scenes, fine-grained, real-time scheduling, management of the scene, and finally fine-grained progressive transmission. This is the key factor in determining the success of ultra-large-scale scene display.

[0007] Third, the rendering efficiency is low. For Web3D, web browsers have limited computing power and cache space, and large scenes may place a heavy burden on the browser. When the number of triangles that need to be rendered reaches millions, ordinary browsers may become stuck, and may even cause the renderer to crash. Therefore, a suitable culling algorithm is needed to determine the parts that need to be rendered and which ones can be culled before rendering. This culling algorithm is implemented through scene graphs and model simplification, which determines the user's perception and experience during scene roaming.

[0008] Fourth, it is difficult to coordinate the three ends. In the process of online visualization, the visible component set calculated by the server backend needs to be efficiently transmitted to the web frontend. Therefore, it is necessary to establish a real-time communication mechanism to ensure that the server backend and the front-end browser (i.e., the web frontend) can instantly transmit roaming information and visible component sets through the network middle end to meet the real-time requirements. A multi-threaded loading mechanism is used to coordinate the loading and parsing of the front-end browser and the server backend to improve the efficiency of online visualization.

[0009] Based on this, there is an urgent need for a lightweight and fine-grained Web3D multi-threaded scheduling technology for ultra-large-scale scenarios. Summary of the invention

[0010] The purpose of the present invention is to provide a Web3D lightweight fine-grained multi-threaded scheduling method and application, which is suitable for online real-time visualization of web pages in ultra-large-scale scenarios and improves user experience.

[0011] To achieve the above object, the present invention provides the following solutions:

[0012] A lightweight fine-grained multi-threaded scheduling method for Web3D, comprising:

[0013] The web front end obtains the camera data of the user when roaming the scene to be rendered, and transmits the camera data to the server back end through the main thread of the network middle end; the camera data includes the camera coordinates and the camera observation direction;

[0014] The server backend uses the camera data as input, and based on the visible associated scene graph corresponding to the scene to be rendered, progressively calculates the first batch of visible sets, the second batch of visible sets and the potential visible sets according to the rendering priority; the visible associated scene graph includes a scene layer and a visible associated layer, the scene layer includes a primary voxel space and a secondary voxel space, the primary voxel space includes tile model data of tiles in the scene to be rendered and component model data, occluder information and instantiation information of components with a filling density higher than a preset value, the secondary voxel space includes component model data, occluder information and instantiation information of components with a filling density not higher than a preset value in the scene to be rendered, and the visible associated layer includes a core component and a visible associated component corresponding to the core component;

[0015] When the first batch of visible sets is calculated, the server backend transmits the first batch of visible sets to a child thread at the network end, the child thread sequentially loads and parses the first components in the first batch of visible sets, and transmits the reconstruction information of the first components obtained by parsing to the main thread; the main thread renders the first components based on the reconstruction information of the first components;

[0016] When the second batch of visible sets is calculated, the server backend transmits the second batch of visible sets to the child thread at the network end, the child thread sequentially loads and parses the second components in the second batch of visible sets, and transmits the reconstruction information of the second components obtained by parsing to the main thread; the main thread renders the second components based on the reconstruction information of the second components;

[0017] When the potential visible set is calculated, the server backend transmits the potential visible set to the child thread in the network middle end, and the child thread sequentially loads and parses the potential components in the potential visible set, and transmits the reconstruction information of the potential components obtained by parsing to the main thread.

[0018] In some embodiments, before the server backend uses the camera data as input and progressively calculates the first batch of visible sets, the second batch of visible sets and the potential visible sets according to the rendering priority based on the visible associated scene graph corresponding to the scene to be rendered, it also includes: using a lightweight fine-grained scene graph construction method to construct the visible associated scene graph of the scene to be rendered, specifically including:

[0019] Performing spatial multi-granularity voxelization on the scene to be rendered to obtain a primary voxel space and a secondary voxel space;

[0020] Adding tiles in the scene to be rendered to the primary voxel space; calculating the filling density of each component in the scene to be rendered, adding components with a filling density higher than a preset value to the primary voxel space, and adding components with a filling density not higher than a preset value to the secondary voxel space;

[0021] For each component in the scene to be rendered, principal component analysis is performed on the component to obtain a characteristic value of the component, similar components similar to the component are determined based on the characteristic value, and a spatial transformation matrix between the component and the similar components is calculated to obtain instantiation information of the component; sampling, edge connection, curve simplification and triangulation are performed on the component in sequence to obtain occluder information of the component;

[0022] A visibility analysis is performed on each component in the scene to be rendered, and a core component and visible associated components corresponding to the core component are selected according to the visibility analysis results of all the components to obtain a visible associated layer.

[0023] In some embodiments, the server backend uses the camera data as input, and based on the visible associated scene graph corresponding to the scene to be rendered, progressively calculates the first batch of visible sets, the second batch of visible sets, and the potential visible sets according to the rendering priority, specifically including:

[0024] The server backend establishes a viewing cone based on the camera data;

[0025] The server backend determines the primary voxel space and the secondary voxel space intersecting with the view frustum based on the scene layer, selects tiles in the primary voxel space intersecting with the view frustum as the first component, selects the core component in the secondary voxel space intersecting with the view frustum and located in the view frustum as the visible core component, takes the visible core component as the first component, and all the first components form a first batch visible set;

[0026] The server backend determines the visible associated component corresponding to each of the visible core components based on the visible associated layer, determines whether the visible associated component corresponding to the visible core component is visible based on the occluder information of the visible core component, and uses the visible visible associated component as the second component, and all the second components form a second batch of visible sets;

[0027] The server backend selects components whose distance from the camera coordinates is less than a first preset distance as potential components based on the scene layer; the server backend determines, based on the scene layer, extended coordinates extending from the camera coordinates by a second preset distance along the camera observation direction, establishes an extended viewing cone according to the extended coordinates and the camera observation direction, determines, based on the scene layer, a primary voxel space and a secondary voxel space intersecting with the extended viewing cone, selects tiles in the primary voxel space intersecting with the extended viewing cone and core components in the secondary voxel space intersecting with the extended viewing cone and located within the extended viewing cone as potential components, and all of the potential components constitute a potential visible set.

[0028] In some embodiments, the child thread sequentially loads and parses the first component in the first batch of visible sets, specifically including:

[0029] The child thread sequentially loads the first component in the first batch of visible sets into the component loading queue, and determines whether the queue length of the component loading queue is less than a preset threshold. If so, return to the step of "the child thread sequentially loads the first component in the first batch of visible sets into the component loading queue".

[0030] During the loading process, the subthread sequentially loads the first component in the component loading queue, and deletes the first component from the component loading queue after the loading is completed;

[0031] The sub-thread parses the loaded first component to obtain reconstruction information of the first component, where the reconstruction information includes vertices, indexes, normals, colors, and texture maps.

[0032] In some embodiments, before the server backend transmits the first batch of visible sets to the child thread of the network middle end, the method further includes: determining whether the first component in the first batch of visible sets is a potential component obtained by historical calculation; if so, deleting the first component in the first batch of visible sets, and the main thread rendering the first component based on the reconstruction information of the potential component obtained by historical calculation;

[0033] Before the server backend transmits the second batch of visible sets to the child thread at the network end, it also includes: determining whether the second component in the second batch of visible sets is a potential component obtained by historical calculation; if so, deleting the second component in the second batch of visible sets, and the main thread renders the second component based on the reconstruction information of the potential component obtained by historical calculation.

[0034] In some embodiments, after the first batch of visible sets, the second batch of visible sets, and the potential visible sets are obtained by calculation, the method further includes:

[0035] Setting the state of the component to be loaded being loaded as a cache state to be loaded; the component to be loaded includes a first component, a second component and a potential component;

[0036] After loading is completed, if the component to be loaded belongs to the first batch visible set or the second batch visible set, the state of the component to be loaded is updated to the current visible state and recorded as a visible component; if the component to be loaded belongs to the potential component set, the state of the component to be loaded is updated to the potential visible state and recorded as a potential visible component;

[0037] During the user's roaming process, if the potentially visible component becomes the visible component, the state of the potentially visible component is updated to the current visible state and recorded as a visible component; if the potentially visible component becomes an invisible component, the state of the potentially visible component is updated to a reserved cache state and recorded as a reserved cache component; if the visible component becomes an invisible component, it is determined whether the visible component exists in the potentially visible set, if so, the state of the visible component is updated to a potentially visible state and recorded as a potentially visible component, if not, the state of the visible component is updated to a reserved cache state and recorded as a reserved cache component;

[0038] Determine whether the cache capacity upper limit of the web page front end is reached; if so, update the status of some of the retained cache components to a cache status to be cleaned according to the cleaning priority, record them as components to be cleaned, and the web page front end cleans the components to be cleaned.

[0039] In some embodiments, the cleaning priority is determined based on component visibility and component file size of the retained cache component.

[0040] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-mentioned Web3D lightweight fine-grained multi-threaded scheduling method.

[0041] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned Web3D lightweight fine-grained multi-threaded scheduling method.

[0042] A computer program product includes a computer program, which, when executed by a processor, implements the steps of the above-mentioned Web3D lightweight fine-grained multi-threaded scheduling method.

[0043] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0044] The present invention provides a Web3D lightweight fine-grained multi-thread scheduling method and application. The webpage front end obtains the camera data of the user when roaming the scene to be rendered, and transmits the camera data to the server back end through the main thread of the network middle end. The server back end uses the camera data as input, and based on the visible associated scene graph corresponding to the scene to be rendered, progressively calculates the first batch of visible sets, the second batch of visible sets and the potential visible sets according to the rendering priority. When the first batch of visible sets is calculated, the server back end transmits the first batch of visible sets to the child thread of the network middle end, the child thread loads and parses, and the main thread renders. When the second batch of visible sets is calculated, the server back end transmits the second batch of visible sets to the child thread of the network middle end. The visible set is transmitted to the child thread at the middle of the network, the child thread loads and parses, and the main thread renders. When the potential visible set is calculated, the server backend transmits the potential visible set to the child thread at the middle of the network, and the child thread loads and parses. Through the visible association scene graph construction, progressive calculation and multi-threaded loading, parsing and rendering, the technical bottleneck of online display of ultra-large-scale scenes on the Web side can be broken: the server backend visible set picking is too slow, the network mid-end transmission is too congested, the web front-end online rendering is too stuck, and the organic coordination and unified scheduling of the three ends is too difficult. It solves the pain points that have long plagued Web3D online rendering of ultra-large-scale scenes, and is suitable for real-time online visualization of web pages for ultra-large-scale scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A method flow chart of a Web3D lightweight fine-grained multi-threaded scheduling method provided in Example 1 of the present invention.

[0047] Figure 2 A timing diagram of a Web3D lightweight fine-grained multi-threaded scheduling method provided in Example 1 of the present invention.

[0048] Figure 3 A schematic diagram of the structure of a visible association scene graph provided in Example 1 of the present invention.

[0049] Figure 4 A schematic diagram of a method flow of a lightweight fine-grained scene graph construction method provided in Example 1 of the present invention.

[0050] Figure 5 A schematic diagram of a method flow of a visible set progressive picking calculation method provided in Embodiment 1 of the present invention.

[0051] Figure 6 A method flow chart of a multi-threaded model loading, parsing and scheduling method provided in Example 1 of the present invention.

[0052] Figure 7 A schematic diagram of a web page front-end cache management state machine provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The purpose of the present invention is to provide a Web3D lightweight fine-grained multi-threaded scheduling method and application for ultra-large-scale scenarios, aiming to improve the performance and efficiency of the system and ensure that users have a smoother and more realistic experience in the metaverse.

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Example 1

[0057] like Figure 1 and Figure 2 As shown, a Web3D lightweight fine-grained multi-threaded scheduling method in this embodiment includes:

[0058] S1: The web front end obtains the camera data of the user when roaming the scene to be rendered, and transmits the camera data to the server back end through the main thread of the network middle end; the camera data includes the camera coordinates and the camera observation direction.

[0059] When the user is roaming in real time, for example, when the user views the various locations of the scene to be rendered in real time by dragging on the web page front end, the web page front end obtains the user's roaming information, that is, obtains the camera data when the user roams the scene to be rendered. The specific location of the scene to be rendered viewed by the user can be determined through the camera data.

[0060] In this embodiment, the scene to be rendered may be a super-large-scale scene, specifically a super-large-scale three-dimensional scene.

[0061] S2: The server backend takes the camera data as input, and based on the visible associated scene graph corresponding to the scene to be rendered, progressively calculates the first batch of visible sets, the second batch of visible sets and the potential visible sets according to the rendering priority; the visible associated scene graph includes a scene layer and a visible associated layer, the scene layer includes a primary voxel space and a secondary voxel space, the primary voxel space includes tile model data of tiles in the scene to be rendered and component model data, occluder information and instantiation information of components with a filling density higher than a preset value, the secondary voxel space includes component model data, occluder information and instantiation information of components in the scene to be rendered with a filling density not higher than a preset value, and the visible associated layer includes a core component and a visible associated component corresponding to the core component.

[0062] like Figure 3 and Figure 4 As shown, this embodiment pre-adopts a lightweight fine-grained scene graph construction method to construct a visible associated scene graph based on a kd tree for the scene to be rendered, specifically including:

[0063] (1) Multi-granularity voxelization: The scene to be rendered is spatially voxelized to obtain a primary voxel space and a secondary voxel space.

[0064] This embodiment first reads and parses the source data file of the scene to be rendered, then performs duplicate checking preprocessing on instantiated components, and then performs multi-granularity voxelization.

[0065] The voxelization preprocessing (i.e., spatial multi-granularity voxelization) of the scene model to be rendered means: using a spatial plane perpendicular to the longest side of the subspace to perform 2-dimensional voxelization on the scene and its subspaces. N , 2 N+3 The tiles in the scene to be rendered are voxelized and located to determine the voxel space to which the tiles belong. The voxelized component indexing is then performed to obtain 2 N with 2 N+3 subspaces, that is, 2 N First-level voxelized scene units (i.e., first-level voxel space), 2 N+3 A second-level voxelized scene unit (i.e., second-level voxel space).

[0066] When dividing the scene space, first instantiate the entire scene to be rendered, check for duplicates, calculate the scene bounding box, calculate the length of the bounding box on the x, y, and z axes, and divide the scene space into two parts in the space perpendicular to the longest axis to obtain two subspaces. Repeat the above steps N times for the subspaces to obtain 2 N subspaces, which are used as the first-level voxel layer (i.e., first-level voxel space) of the scene; after the division is completed, these subspaces are divided three times to obtain 2 N+3Subspaces are used as the secondary voxel layer (i.e., secondary voxel space) of the scene. In this way, the scene to be rendered is divided into several voxel spaces, and the shape of the voxel space is close to a cube, which is convenient for subsequent calculation and scheduling.

[0067] This embodiment can also construct a multi-granularity voxel index tree structure based on the multi-granularity voxel partition space (ie, the primary voxel space and the secondary voxel space), and the multi-granularity voxel index tree structure is used to represent the association relationship between the primary voxel space and the secondary voxel space.

[0068] (2) Constructing a scene graph: adding tiles in the scene to be rendered to the primary voxel space; calculating the filling density of each component in the scene to be rendered, adding components with a filling density higher than a preset value to the primary voxel space, and adding components with a filling density not higher than the preset value to the secondary voxel space.

[0069] This embodiment calculates the instantiation bounding box and filling density of each component, regards the center of the instantiation bounding box of the component as the component coordinate, and places the components into corresponding scene unit levels according to the component coordinates and the filling density, that is, adds them to the voxel space of the corresponding granularity. Specifically, terrain tiles (i.e., tiles) and components with high filling density (i.e., components with filling density higher than a preset value) are placed in the first-level voxel space, and components with low filling density (i.e., components with filling density not higher than a preset value) are placed in the second-level voxel space, until all components are added to the scene graph.

[0070] In order to meet the user's demand for fullness of the picture during scene roaming, it is necessary to fill the components in the scene into the primary voxel space and the secondary voxel space respectively according to certain rules. Therefore, this embodiment uses packing density (PD) to describe the visual contribution of a component in the scene. The packing density PD of component i is i The following conditions are met:

[0071] ① Surface area S of the component bounding box i The larger the size, the easier it is to see the component in the scene, so the filling density PD i The bigger.

[0072] ② Component instantiation quantity IN i The more the component is, the easier it is to observe in the scene, so the filling density PD i The larger the component is, the principal component analysis is performed on the component to obtain the eigenvalue of the component, and similar components similar to the component are determined based on the eigenvalue. The number of component instantiations refers to the number of similar components similar to the component.

[0073] Based on the above conditions, the calculation formula for filling density is as follows:

[0074]

[0075] Among them, w1 and w2 represent weights. When instantiating components, the relationship between visibility and the number of instantiations is not linear, so weights are needed to make it more accurate. S represents the surface area of ​​the bounding box of the entire scene, so that the filling density range does not change significantly with the size of the scene itself, which is convenient for standardized processing.

[0076] (3) Component lightweighting: For each component in the scene to be rendered, perform principal component analysis on the component and calculate the eigenvalue of the component. Based on the eigenvalue, similar components similar to the component are determined, that is, components with similar features are found, and the spatial transformation matrix between the component and similar components is calculated to obtain the instantiation information of the component. The components are sampled, edge connected, curve simplified, and triangulated in turn to obtain the occlusion body information of the component.

[0077] When calculating the occluder information of a component, the component is sampled on the x, y, and z axes respectively to obtain the maximum sampling surface of the component, the edge of the maximum sampling surface is connected to obtain an edge ring, the edge ring is curve simplified to obtain a simplified curve, the simplified curve is triangulated to obtain the occluder information of the component.

[0078] (4) Constructing a visible association graph: Perform visibility analysis on each component in the scene to be rendered, select the core component and the visible associated components corresponding to the core component based on the visibility analysis results of all components, and construct a visible association graph between the core component and the corresponding visible associated components to obtain the visible association layer of the inner body subspace of the scene to be rendered.

[0079] In order to quickly pick up entities in massive scenes and realize online scheduling based on visibility correlation, it is necessary to judge and calculate the visibility before measuring the visibility correlation. This embodiment selects sampling points and performs visibility sampling on the scene, calculates the visibility and visibility weight of each component, and then calculates the visibility correlation and constructs the visibility correlation layer.

[0080] In order to measure the contribution of components in the field of view during scene sampling and calculation, this embodiment proposes the concept of visibility. The visibility of component i is Vis_Degree i Calculated by the following formula:

[0081]

[0082] Among them, pixel_count i Indicates the number of pixels of the component in the render target buffer (i.e., the rendering field of view, or the field of view for observing the scene to be rendered), W and H represent the length and width of the render target buffer respectively. The visible component index and visibility of each sampling point are saved in a list for the construction of the visible association graph.

[0083] In order to calculate the visual correlation between two components, this embodiment proposes the concept of visible correlation degree. The visible correlation degree (VCD) of component i and component j is i,j The calculation formula is:

[0084]

[0085] Among them, Distance i,j Represents the distance between components i and j. This formula is summed over all sampling points.

[0086] Based on the concepts of visibility and visibility correlation, the construction method of the visibility correlation layer specifically includes:

[0087] 1) Visibility calculation: Calculate the bounding box for the entire scene, and evenly select several points within the bounding box. For each point, calculate the number of components within a certain range of the point. If the number of components exceeds the threshold, this point is used as a sampling point for subsequent sampling, i.e., visibility calculation operation. For each sampling point, the camera is directed to the front, back, left, right, top, and bottom directions to calculate the visibility of the components, and the visible component list of the sampling point is recorded. The visible component list includes the visible component index and visibility of all visible components corresponding to the sampling point.

[0088] 2) Core component selection: After the sampling calculation of all sampling points is completed, analysis is performed. For each secondary voxel space, assuming that there are a total of N visible components in the secondary voxel space, the one with the highest visibility in the secondary voxel space is selected. A visible component is used as the core component.

[0089] 3) Construct visible association layer: Select the core components in descending order of their visibility, and select the layer closest to the core component. Visible components as core components The visible association components are calculated, and the visible association degree between the core component and each visible association component is calculated. After the visible association degree is calculated, the core component is connected to the visible association component, and the visible association degree is recorded as the weight value of the edge connecting the core component and the visible association component to construct a visible association layer.

[0090] (5) Exporting scene graph: Exporting a lightweight fine-grained scene graph (i.e., a visible associated scene graph) containing fine-grained component data. The visible associated scene graph forms a multi-granularity voxel index tree structure, and the component mesh model data (i.e., tile model data and component model data), occlusion body information, and instantiation information are stored in the voxel space.

[0091] like Figure 5 As shown, when the user is roaming in real time, the server backend calculates the current visible component set with a progressive algorithm based on the visible associated scene graph and the user's roaming information, that is, a scene graph-based visible set progressive picking calculation method is provided, which sequentially calculates the first batch of visible sets, the second batch of visible sets and the potential visible sets.

[0092] The server backend uses the camera data as input, and based on the visible associated scene graph corresponding to the scene to be rendered, progressively calculates the first batch visible set, the second batch visible set, and the potential visible set according to the rendering priority, including:

[0093] (1) Scene graph loading: Load the visible associated scene graph information (i.e., scene data, including scene graph files, bounding boxes, and instantiation information) on the server backend, read the occluder information, and reconstruct the scene to be rendered in a tree structure on the server backend based on the visible associated scene graph information. That is, construct a voxelized scene on the server backend to facilitate subsequent loading scheduling calculations.

[0094] (2) Picking the first batch: The server backend establishes a viewing cone based on the camera data, that is, the viewing cone is established based on the coordinates of the user during roaming and the camera projection matrix, world matrix and other information. If the voxel space of the visible associated scene graph intersects with the viewing cone, then calculate whether the core component or tile in the voxel space is within the viewing cone, and recursively calculate the voxel space within the field of view until all the core components and tiles in the viewing cone are picked up to form a rough visible set (i.e., the first batch of visible sets). Specifically, the server backend determines the primary voxel space and secondary voxel space that intersect with the viewing cone based on the scene layer, selects the tile in the primary voxel space that intersects with the viewing cone as the first component, selects the core component in the secondary voxel space that intersects with the viewing cone and is located in the viewing cone as the visible core component, and takes the visible core component as the first component. All first components form the first batch of visible sets.

[0095] (3) Picking up the second batch: The server backend determines the visible associated components corresponding to each visible core component based on the visible associated layer, that is, according to the picked visible core component, the visible associated components associated with it are searched in the visible associated scene graph, and whether the visible associated components corresponding to the visible core component are visible is judged based on the occluder information of the visible core component. Specifically, the visible association degree of the visible associated component is divided by the distance between the visible associated component and the camera, and the visible associated components are sorted from near to far. The occluder information of the visible core component and the sorting result of the visible associated components are used to construct a hierarchical Z buffer, and the hierarchical Z buffer is used to perform occlusion culling based on soft rasterization to judge the visibility of the visible associated components. The visible visible associated components are used as the second components, and all the second components constitute an accurate visible set (i.e., the second batch visible set).

[0096] (4) The third batch of picking: According to the voxel space in the visible associated scene graph, with the user's camera position as the center, pick up components within a certain distance, and extend a distance in the camera observation direction, and then pick up the first batch again, and use the two sets of components picked up as potential visible sets. Specifically, the server backend selects components whose distance from the camera coordinates is less than the first preset distance as potential components based on the scene layer; the server backend determines the extended coordinates extending from the camera coordinates by a second preset distance along the camera observation direction based on the scene layer, establishes an extended view cone based on the extended coordinates and the camera observation direction, determines the primary voxel space and the secondary voxel space intersecting with the extended view cone based on the scene layer, selects tiles in the primary voxel space intersecting with the extended view cone and core components in the secondary voxel space intersecting with the extended view cone located in the extended view cone as potential components, and all potential components constitute a potential visible set.

[0097] After obtaining the first batch of visible sets, this embodiment can transmit the first batch of visible sets to the web page front end without waiting for the second batch of visible sets and potential visible sets. After obtaining the second batch of visible sets, this embodiment can transmit the second batch of visible sets to the web page front end without waiting for the potential visible sets. After obtaining the potential visible sets, the potential visible sets can be transmitted to the web page front end.

[0098] S3: When the first batch of visible sets is calculated, the server backend transmits the first batch of visible sets to the child thread in the middle of the network, the child thread sequentially loads and parses the first component in the first batch of visible sets, and transmits the reconstruction information of the first component obtained by parsing to the main thread; the main thread renders the first component based on the reconstruction information of the first component.

[0099] like Figure 6 As shown, according to the obtained visible set (i.e., the first batch visible set, the second batch visible set, and the potential visible set), the network middle end performs multi-threaded (i.e., main thread and sub-thread) model loading and parsing, the web page front end performs multi-threaded collaborative operation and communication, and performs lightweight rendering on the visible set according to the result of visibility calculation, that is, a multi-threaded model loading and parsing scheduling method is provided, which specifically includes:

[0100] (1) Starting multithreading: When a user opens the front end of a web page, the main thread starts the loading and parsing sub-thread based on Web Worker, and enables the sub-thread to run. While the user is browsing the web page (i.e., during the user's roaming process), the sub-thread keeps running and communicates with the main thread in real time.

[0101] (2) Bandwidth detection loading: The visible set to be loaded is continuously loaded into the component loading queue and maintained during the loading process. If one of the components (i.e., the first component, the second component, and the potential component) is loaded, it is deleted from the component loading queue. When the queue length of the component loading queue is less than the threshold, the visible set to be loaded is loaded into the component loading queue again until the visible set to be loaded is loaded.

[0102] (3) Component parsing and reconstruction: After the child thread completes loading the component, it parses the component model (i.e., component model data) to obtain its vertices, indexes, normals, colors, texture maps, and other information, and transmits this information to the main thread, which then reconstructs the component model, i.e., renders it.

[0103] In this embodiment, the main thread transmits real-time camera data to the server backend, and the server backend starts to execute visible set calculation. After the server backend calculates the first batch of visible sets, it transmits the first batch of visible set information to the web frontend. The web frontend loads and parses the first batch of visible set data in the child thread, and passes the parsed three-dimensional data to the main thread, including geometric meshes, material information, and instantiation information. The main thread performs lightweight rendering. After the server backend calculates the second batch of visible sets, it continues to transmit it to the web frontend. The web frontend loads and parses the second batch of visible set data in the child thread, and passes the parsed three-dimensional data to the main thread, and the main thread performs lightweight rendering.

[0104] The child thread then sequentially loads and parses the first component in the first batch of visible sets, specifically including: the child thread sequentially loads the first component in the first batch of visible sets into the component loading queue, and determines whether the queue length of the component loading queue is less than a preset threshold. If so, returns to the step of "sequentially loading the first component in the first batch of visible sets into the component loading queue"; during the loading process, the child thread sequentially loads the first component in the component loading queue, and deletes the first component from the component loading queue after loading is completed; the child thread parses the loaded first component to obtain reconstruction information of the first component, and the reconstruction information includes vertices, indices, normals, colors and texture maps.

[0105] S4: When the second batch of visible sets is calculated, the server backend transmits the second batch of visible sets to the child thread in the middle of the network, the child thread sequentially loads and parses the second components in the second batch of visible sets, and transmits the reconstruction information of the second components obtained by the parsing to the main thread; the main thread renders the second component based on the reconstruction information of the second component.

[0106] Similarly, the child thread sequentially loads and parses the second component in the second batch of visible sets, specifically including: the child thread sequentially loads the second component in the second batch of visible sets into the component loading queue, and determines whether the queue length of the component loading queue is less than a preset threshold. If so, returns to the step of "sequentially loading the second component in the second batch of visible sets into the component loading queue"; during the loading process, the child thread sequentially loads the second component in the component loading queue, and deletes the second component from the component loading queue after loading is completed; the child thread parses the loaded second component to obtain reconstruction information of the second component, and the reconstruction information includes vertices, indices, normals, colors and texture maps.

[0107] S5: When the potential visible set is calculated, the server backend transmits the potential visible set to the child thread in the middle of the network. The child thread sequentially loads and parses the potential components in the potential visible set, and transmits the reconstruction information of the potential components obtained by parsing to the main thread.

[0108] Similarly, the child thread sequentially loads and parses the potential components in the potential visible set, specifically including: the child thread sequentially loads the potential components in the potential visible set into the component loading queue, determines whether the queue length of the component loading queue is less than a preset threshold, and if so, returns to the step of "sequentially loading the potential components in the potential visible set into the component loading queue"; during the loading process, the child thread sequentially loads the potential components in the component loading queue, and deletes the potential components from the component loading queue after the loading is completed; the child thread parses the loaded potential components to obtain reconstruction information of the potential components, and the reconstruction information includes vertices, indices, normals, colors and texture maps.

[0109] Before the server backend transmits the first batch of visible sets to the child thread in the network middle end, the method of this embodiment also includes: determining whether the first component in the first batch of visible sets is a potential component obtained by historical calculation; if so, deleting the first component in the first batch of visible sets, and the main thread rendering the first component based on the reconstruction information of the potential component obtained by historical calculation.

[0110] Similarly, before the server backend transmits the second batch of visible sets to the child thread at the middle end of the network, the method of this embodiment also includes: determining whether the second component in the second batch of visible sets is a potential component obtained by historical calculation; if so, deleting the second component in the second batch of visible sets, and the main thread renders the second component based on the reconstruction information of the potential component obtained by historical calculation.

[0111] This embodiment performs cache management on components in a web page front-end scene based on the result of visibility calculation, and further combines information such as component retention time and instantiation information, that is, provides a web page rendering and cache management method, which specifically includes:

[0112] (1) Instanced rendering: After the component model is rebuilt in the main thread, an instanced mesh model is constructed based on its instantiation information. If it belongs to the rough visible set or the precise visible set, it is set to visible; if it belongs to the potential visible set, it is set to invisible. The instanced mesh model is added to the scene, and its visibility is updated in real time during the user's roaming process.

[0113] (2) Recording component information: After a component is loaded on the front end of a web page, the component's existence time and component visibility in the front end scene of the web page are recorded in real time. The component cleaning priority is calculated based on the component visibility and the component file size. The components to be cleaned in the component list composed of all components to be cleaned are sorted from high to low according to the component cleaning priority to facilitate subsequent cache cleaning.

[0114] (3) Cache cleaning: Calculate the number of components in the cache in real time. When the number of components cached in the scene exceeds a certain number, delete the components that exceed the number in the component list. First, set the component to invisible and remove it from the scene, then delete its instantiation information group, and finally delete its index in the component list.

[0115] In order to facilitate web page cache management and data transmission, based on the analysis of cache data status in web page nodes and the requirements of cache update algorithm, the component cache data in the node cache space is divided into five states:

[0116] (1) Cache state to be loaded q0: indicates component cache data being loaded.

[0117] (2) Current visible cache state q1: represents the component cache data visible in the current scene.

[0118] (3) Potentially visible cache state q2: represents preloaded component cache data that is potentially visible in the future.

[0119] (4) Retain cache state q3: indicates component cache data that is not visible at present or in the future and does not exceed the cache capacity threshold.

[0120] (5) Cache status to be cleaned up q4: indicates that the component cache data is not visible now and in the future, exceeds the cache capacity threshold, and is about to be cleaned up.

[0121] As the user navigates through the scene, the component cache data is converted between different states. The conversion process is as follows: Figure 7 The detailed description of the process is as follows:

[0122] (1) The user enters the scene, obtains the current visible set (i.e., the first batch visible set and the second batch visible set) or the potentially visible set, and starts loading component resources. At this time, the component state being loaded is the pending cache state. After the loading is completed, the component cache data in the current visible set is the current visible cache state, and the component cache data in the potentially visible set is the potentially visible cache state.

[0123] (2) As the user moves, some components in the potentially visible set become visible components and are observed by the user. The cache data of these components is converted from the potentially visible cache state to the currently visible cache state.

[0124] (3) Some components that have been observed become completely invisible as the user moves. At this time, if the component exists in the potential visible set, it is converted from the current visible cache state to the potential visible cache state, otherwise it is converted to the retained cache state.

[0125] (4) For the cache status retaining component, after calculating the cleaning priority, the part of the resources that exceeds the cache capacity of the web page node is converted to the cache status to be cleaned until it is cleaned up.

[0126] (5) In the cache management process of tile data, when tile data is in a reserved cache state, the invisible time of the tile data is recorded, and tile data of different LoD levels are gradually cleaned up according to the invisible time until the tile data is completely cleaned up.

[0127] After the first batch of visible sets, the second batch of visible sets and the potential visible sets are calculated, the method of this embodiment also includes: setting the state of the component to be loaded being loaded to the cache state to be loaded, and the components to be loaded include the first component, the second component and the potential component; after the loading is completed, if the component to be loaded belongs to the first batch of visible sets or the second batch of visible sets, the state of the component to be loaded is updated to the current visible cache state and recorded as a visible component; if the component to be loaded belongs to the potential component set, the state of the component to be loaded is updated to the potential visible cache state and recorded as a potential visible component; during the user's roaming process, if the potential visible component becomes a visible component, the state of the potential visible component is updated to the current visible cache state and recorded as a visible component. Component; if a potentially visible component becomes an invisible component, the state of the potentially visible component is updated to a retained cache state and recorded as a retained cache component; if a visible component becomes an invisible component, it is determined whether the visible component exists in the potentially visible set. If so, the state of the visible component is updated to a potentially visible cache state and recorded as a potentially visible component. If not, the state of the visible component is updated to a retained cache state and recorded as a retained cache component; it is determined whether the cache capacity upper limit of the web page front end is reached; if so, the state of some retained cache components is updated to a to-be-cleaned cache state according to the cleaning priority and recorded as a to-be-cleaned component. The web page front end cleans the to-be-cleaned components, and the cleaning priority is determined according to the component visibility and component file size of the retained cache components.

[0128] This embodiment provides a Web3D lightweight fine-grained multi-threaded scheduling method for ultra-large-scale scenes, which belongs to the field of Internet three-dimensional engines and key technologies related to the metaverse, and breaks through the technical bottleneck of online display of ultra-large-scale scenes on the Web: the visible set picking of the server backend is too slow, the transmission in the middle of the network is too congested, the online rendering of the web page frontend is too stuck, and the organic coordinated and unified scheduling of the three ends is too difficult. It solves the pain points that have long plagued Web3D online rendering of ultra-large-scale scenes, and studies a complete set of ultra-large-scale, high-efficiency, low-cost, and low-energy lightweight fine-grained multi-threaded solutions from progressive visible set picking scheduling of ultra-large-scale scenes on the server backend, to progressive fine-grained loading scheduling in the middle of the network, and then to multi-threaded parsing, rendering, and cache scheduling of online ultra-large-scale scenes on the web page frontend. This has a great role in helping and enabling the large-scale application of Web3D in large-scale digital twin factories, smart cities, MMORPGs, three-dimensional Internet, and the metaverse.

[0129] This embodiment provides a Web3D lightweight fine-grained multi-threaded scheduling method for ultra-large-scale scenes. Combined with the spatial characteristics of the entire scene, the ultra-large-scale scene is organized in the form of a scene graph, and combined with visibility culling and multi-threading methods, a lightweight fine-grained online computing, transmission, parsing, rendering, and cache management scheduling scheme for ultra-large-scale scenes is formed, and a Web3D ultra-large-scale scene online visualization platform is provided. Compared with the prior art, the Web3D lightweight fine-grained intelligent scheduling method provided by this embodiment has the following advantages: first, automation, that is, automatically dividing ultra-large-scale scenes and organizing them in scene units, and users can have a corresponding in-depth professional understanding of the data according to their own professional situation; second, fine-grained, that is, based on the characteristics of spatial culling of visualization applications, the scenes with complex structures are uniformly fine-grained and expressed as hierarchical granularity, further supporting the application to timely schedule and access data of appropriate granularity according to actual conditions; third, lightweight, considering the front-end performance and user hardware conditions, complex calculations are placed on the back end to reduce the front-end pressure as much as possible.

[0130] Example 2

[0131] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a Web3D lightweight fine-grained multi-threaded scheduling method described in Example 1.

[0132] Example 3

[0133] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a Web3D lightweight fine-grained multi-threaded scheduling method described in Example 1.

[0134] Example 4

[0135] A computer program product includes a computer program, which, when executed by a processor, implements the steps of a Web3D lightweight fine-grained multi-thread scheduling method described in Example 1.

[0136] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A lightweight and fine-grained multi-threaded scheduling method for Web3D, characterized in that: include: The web front end obtains the camera data of the user when roaming the scene to be rendered, and transmits the camera data to the server back end through the main thread of the network middle end; the camera data includes the camera coordinates and the camera observation direction; The server backend uses the camera data as input, and based on the visible associated scene graph corresponding to the scene to be rendered, progressively calculates the first batch of visible sets, the second batch of visible sets and the potential visible sets according to the rendering priority; the visible associated scene graph includes a scene layer and a visible associated layer, the scene layer includes a primary voxel space and a secondary voxel space, the primary voxel space includes tile model data of tiles in the scene to be rendered and component model data, occluder information and instantiation information of components with a filling density higher than a preset value, the secondary voxel space includes component model data, occluder information and instantiation information of components with a filling density not higher than a preset value in the scene to be rendered, and the visible associated layer includes a core component and a visible associated component corresponding to the core component; When the first batch of visible sets is calculated, the server backend transmits the first batch of visible sets to a child thread at the network end, the child thread sequentially loads and parses the first components in the first batch of visible sets, and transmits the reconstruction information of the first components obtained by parsing to the main thread; the main thread renders the first components based on the reconstruction information of the first components; When the second batch of visible sets is calculated, the server backend transmits the second batch of visible sets to the child thread at the network end, the child thread sequentially loads and parses the second components in the second batch of visible sets, and transmits the reconstruction information of the second components obtained by parsing to the main thread; the main thread renders the second components based on the reconstruction information of the second components; When the potential visible set is calculated, the server backend transmits the potential visible set to the child thread at the network end, and the child thread sequentially loads and parses the potential components in the potential visible set, and transmits the reconstruction information of the potential components obtained by parsing to the main thread; The server backend uses the camera data as input, and based on the visible associated scene graph corresponding to the scene to be rendered, progressively calculates the first batch visible set, the second batch visible set, and the potential visible set according to the rendering priority, specifically including: The server backend establishes a viewing cone based on the camera data; The server backend determines the primary voxel space and the secondary voxel space intersecting with the view frustum based on the scene layer, selects tiles in the primary voxel space intersecting with the view frustum as the first component, selects the core component in the secondary voxel space intersecting with the view frustum and located in the view frustum as the visible core component, takes the visible core component as the first component, and all the first components form a first batch visible set; The server backend determines the visible associated component corresponding to each of the visible core components based on the visible associated layer, determines whether the visible associated component corresponding to the visible core component is visible based on the occluder information of the visible core component, and uses the visible visible associated component as the second component, and all the second components form a second batch of visible sets; The server backend selects components whose distance from the camera coordinates is less than a first preset distance as potential components based on the scene layer; the server backend determines, based on the scene layer, extended coordinates extending from the camera coordinates by a second preset distance along the camera observation direction, establishes an extended viewing cone according to the extended coordinates and the camera observation direction, determines, based on the scene layer, a primary voxel space and a secondary voxel space intersecting with the extended viewing cone, selects tiles in the primary voxel space intersecting with the extended viewing cone and core components in the secondary voxel space intersecting with the extended viewing cone and located within the extended viewing cone as potential components, and all of the potential components constitute a potential visible set.

2. According to claim 1, a lightweight fine-grained multi-threaded scheduling method for Web3D is characterized in that: Before the server backend uses the camera data as input and progressively calculates the first batch of visible sets, the second batch of visible sets and the potential visible sets according to the rendering priority based on the visible associated scene graph corresponding to the scene to be rendered, it also includes: constructing the visible associated scene graph of the scene to be rendered by using a lightweight fine-grained scene graph construction method, specifically including: Performing spatial multi-granularity voxelization on the scene to be rendered to obtain a primary voxel space and a secondary voxel space; Adding tiles in the scene to be rendered to the primary voxel space; calculating the filling density of each component in the scene to be rendered, adding components with a filling density higher than a preset value to the primary voxel space, and adding components with a filling density not higher than a preset value to the secondary voxel space; For each component in the scene to be rendered, principal component analysis is performed on the component to obtain a characteristic value of the component, similar components similar to the component are determined based on the characteristic value, and a spatial transformation matrix between the component and the similar components is calculated to obtain instantiation information of the component; sampling, edge connection, curve simplification and triangulation are performed on the component in sequence to obtain occluder information of the component; A visibility analysis is performed on each component in the scene to be rendered, and a core component and visible associated components corresponding to the core component are selected according to the visibility analysis results of all the components to obtain a visible associated layer.

3. A Web3D lightweight fine-grained multi-threaded scheduling method according to claim 1, characterized in that: The child thread sequentially loads and parses the first component of the first batch of visible sets, specifically including: The child thread sequentially loads the first component in the first batch of visible sets into the component loading queue, and determines whether the queue length of the component loading queue is less than a preset threshold. If so, return to the step of "the child thread sequentially loads the first component in the first batch of visible sets into the component loading queue". During the loading process, the subthread sequentially loads the first component in the component loading queue, and deletes the first component from the component loading queue after the loading is completed; The sub-thread parses the loaded first component to obtain reconstruction information of the first component, where the reconstruction information includes vertices, indexes, normals, colors, and texture maps.

4. A Web3D lightweight fine-grained multi-threaded scheduling method according to claim 1, characterized in that: Before the server backend transmits the first batch of visible sets to the sub-thread at the network middle end, the method further includes: determining whether the first component in the first batch of visible sets is a potential component obtained by historical calculation; if so, deleting the first component from the first batch of visible sets, and the main thread rendering the first component based on the reconstruction information of the potential component obtained by historical calculation; Before the server backend transmits the second batch of visible sets to the child thread at the network end, it also includes: determining whether the second component in the second batch of visible sets is a potential component obtained by historical calculation; if so, deleting the second component in the second batch of visible sets, and the main thread renders the second component based on the reconstruction information of the potential component obtained by historical calculation.

5. A Web3D lightweight fine-grained multi-threaded scheduling method according to claim 4, characterized in that: After calculating the first batch of visible sets, the second batch of visible sets and the potential visible sets, it also includes: Setting the state of the component to be loaded being loaded as a cache state to be loaded; the component to be loaded includes a first component, a second component and a potential component; After loading is completed, if the component to be loaded belongs to the first batch visible set or the second batch visible set, the state of the component to be loaded is updated to the current visible cache state and recorded as a visible component; if the component to be loaded belongs to the potential component set, the state of the component to be loaded is updated to the potential visible cache state and recorded as a potential visible component; During the user's roaming process, if the potentially visible component becomes the visible component, the state of the potentially visible component is updated to the current visible cache state and recorded as a visible component; if the potentially visible component becomes an invisible component, the state of the potentially visible component is updated to the reserved cache state and recorded as a reserved cache component; if the visible component becomes an invisible component, it is determined whether the visible component exists in the potentially visible set, if so, the state of the visible component is updated to the potentially visible cache state and recorded as a potentially visible component, if not, the state of the visible component is updated to the reserved cache state and recorded as a reserved cache component; Determine whether the cache capacity upper limit of the web page front end is reached; if so, update the status of some of the retained cache components to a cache status to be cleaned according to the cleaning priority, record them as components to be cleaned, and the web page front end cleans the components to be cleaned.

6. A Web3D lightweight fine-grained multi-threaded scheduling method according to claim 5, characterized in that: The cleaning priority is determined according to the component visibility and the component file size of the retained cache component.

7. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a Web3D lightweight fine-grained multi-threaded scheduling method as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a Web3D lightweight fine-grained multi-thread scheduling method described in any one of claims 1 to 6 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of a Web3D lightweight fine-grained multi-thread scheduling method described in any one of claims 1 to 6 are implemented.

Images