An immersive image transmission method and system for AR cinemas

Abstract

This invention discloses an immersive image transmission method and system for AR cinemas, relating to the field of AR image transmission technology. The method includes optimizing a stereoscopic unit set and a NeRF radiant field unit set using a variable-weight joint optimization method; predicting the fusion probability of stereoscopic units and NeRF radiant field units using a spatiotemporal Transformer fusion method; performing unit fusion based on the fusion probability; and mapping fused and unfused units onto a 3D mesh structure to generate a stereoscopic-NeRF hybrid mesh. The variable-weight joint optimization method dynamically calculates optimization weights, prioritizing optimization of high-interaction areas, improving resource allocation efficiency, and enhancing the detail expression accuracy of special effects areas. The spatiotemporal Transformer fusion method is introduced, utilizing a learnable weight matrix to predict the fusion probability. Spatiotemporal feature modeling enhances the semantic association between stereoscopic units and NeRF units, eliminating light field breaks and geometric distortions, and significantly improving the rendering quality of AR cinema scenes in dynamic special effects areas.

Description

Technical Field

[0001] This invention relates to the field of AR image transmission technology, and in particular to an immersive image transmission method and system for AR cinemas. Background Technology

[0002] In recent years, the application of augmented reality (AR) technology in immersive cinemas has significantly promoted the development of real-time interaction and high-quality rendering technologies for virtual scenes. Existing technologies primarily rely on point cloud data processing, geometric modeling, and light field rendering to construct virtual scenes. Among these, point cloud data, through the discretization of stereoscopic units and combined with Neural Radiation Field (NeRF) technology to generate continuous light field information, has become a core method for AR cinema scene construction. However, existing technologies have significant shortcomings in the optimization and fusion of stereoscopic units and NeRF units, limiting the rendering quality and real-time interactive experience of dynamic special effects areas (such as flames and magical light effects). Existing technologies typically employ fixed-weight optimization methods (such as standard gradient descent or mean aggregation) to adjust the parameters of stereoscopic units and NeRF units, ignoring the differences in interaction frequency and complexity among different units. This results in the optimization process failing to dynamically adapt to the error distribution in high-interaction areas (such as special effects areas), leading to uneven resource allocation. Static areas (such as walls) may be over-optimized, while dynamic areas may be over-optimized. The existing methods suffer from several drawbacks. First, they lack sufficient detail representation of the domain. Second, existing fusion methods often rely on simple feature stitching or weighted averaging (e.g., feature fusion based on Euclidean distance), failing to adequately consider the semantic relationships between stereoscopic units (discrete geometric structures) and NeRF units (continuous light field information). This results in unnatural transitions in dynamic effect areas, leading to light field information breaks or geometric distortions. Third, when mapping optimized units to 3D mesh structures, existing techniques typically employ uniform mesh partitioning or static weighted mapping, neglecting the differences in interaction frequency and complexity across different areas of the scene. This results in insufficient mesh resolution in high-interaction areas (e.g., effect areas), making it difficult to capture complex dynamic light field changes, while low-interaction areas (e.g., walls) may be allocated excessive computational resources, reducing overall efficiency. Fourth, existing methods lack hierarchical partitioning mechanisms based on spectral clustering and dynamic weight allocation (e.g., focal layer, transition layer, background layer), failing to achieve adaptive mesh optimization based on interaction frequency and complexity, thus affecting the rendering quality of stereoscopic-NeRF hybrid meshes. Summary of the Invention

[0003] In view of the aforementioned existing problems, the present invention is proposed.

[0004] Therefore, the present invention provides an immersive image transmission method and system for AR cinemas, which solves the limitations of existing unit optimization methods, the inadequacy of unit fusion strategies, and the inefficiency of grid mapping and hierarchical allocation.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] In a first aspect, the present invention provides an immersive image transmission method for AR cinemas, comprising: loading AR cinema virtual scene data through a cloud server to extract a point cloud dataset from it; and using a weighted K-means algorithm to set each point in the point cloud dataset into a stereoscopic unit set.

[0007] Based on the stereo unit set, a pre-trained GNN model is used to generate a dynamic point cloud subset. An octree partitioning method is used to generate an initial grid from the point cloud subset. The complexity features of each grid region are calculated, and a NeRF radiation field unit set is generated based on the complexity features.

[0008] The stereo view unit set and the NeRF radiation field unit set are optimized by a variable weight joint optimization method. The spatiotemporal Transformer fusion method is used to predict the fusion probability of stereo view units and NeRF radiation field units. Unit fusion is performed according to the fusion probability. The fused and unfused units are mapped to a 3D mesh structure to generate a stereo-NeRF hybrid mesh.

[0009] Based on the collection of real-time interactive data and the priority of stereo-NeRF hybrid grid computing units, the data within the unit is configured into a priority queue using the QUIC protocol, and the data is transmitted efficiently according to the QUIC priority queue.

[0010] As a preferred embodiment of the immersive image transmission method for AR cinemas described in this invention, the step of loading AR cinema virtual scene data through a cloud server and extracting a point cloud dataset from it, and using a weighted K-means algorithm to group each point in the point cloud dataset into a stereoscopic unit set, refers to loading the AR cinema virtual scene through the cloud server and extracting the point cloud dataset from it. }, each point For each point cloud point Calculate interaction weights A point is randomly selected from the point cloud dataset P as the first center of the stereoscopic unit. The weighted K-means++ algorithm is used to generate the first... Individual view unit center, calculate each point The minimum weighted distance to the center of the selected stereoscopic unit is determined based on the weighted distance and interaction weight. Calculate the probability that each point is selected as the new center. Using weighted random sampling, based on probability Choose a point from set P as the new center. Repeatedly iterate to create new centers The generation step obtains the set of stereoscopic unit centers, and then points... Assigned to the stereoscopic unit closest to the center point stereoscopic unit Form a set And calculate the point density within the cell. .

[0011] As a preferred embodiment of the immersive image transmission method for AR cinemas described in this invention, the following steps are described: A dynamic point cloud subset is generated using a pre-trained GNN model based on a stereoscopic unit set; an initial grid is generated from the point cloud subset using an octree partitioning method; the complexity features of each grid region are calculated; and NeRF radiation field units are generated based on the complexity features. Construct a point cloud map, with each stereoscopic unit center point As a node, the node feature vector is The k-nearest neighbor method is used to construct edges in the point cloud graph based on the Euclidean spatial distance between the centers of the stereo vision units. A pre-trained GNN model is then used to process each node in the point cloud graph. Predicting dynamic tags Generate a dynamic point cloud subset based on the dynamic effect area. For dynamic point cloud subsets Perform octree partitioning to generate the initial mesh. Calculate the complexity features of each grid region. Principal component analysis was used to calculate the value of each grid cell. Color characteristics [ Using feature binding to combine complexity features and color features [ Bind to the center coordinates of the grid To form a spatial point set K-means clustering was used to cluster the point set, with K being the number of clusters and each cluster representing a NeRF unit. ,Will The elements are set together to obtain the NeRF element set. .

[0012] As a preferred embodiment of the immersive image transmission method for AR cinemas described in this invention, the following steps are taken: The stereoscopic unit set and the NeRF radiation field unit set are optimized using a variable weight joint optimization method; the spatiotemporal Transformer fusion method is used to predict the fusion probability of stereoscopic units and NeRF radiation field units; unit fusion is performed based on the fusion probability; the fused and unfused units are mapped onto a three-dimensional mesh structure to generate a stereoscopic-NeRF hybrid mesh; and the node feature vectors are calculated using a mean aggregation method. Average values ​​of each dimension, generating Unit feature vector The Gaussian kernel similarity method is used to calculate and Similarity of unit feature vectors Based on feature vector similarity Construct a similarity matrix Construct the Laplace matrix Spectral clustering is applied to the Laplacian matrix. The data is clustered into three layers. eigenvalue decomposition divides the units in set A into three natural clusters. A high interaction frequency threshold H, a medium interaction frequency threshold C, and a complexity threshold O are set. The clusters are then determined based on the interaction frequency of the units. and complexity Assign a hierarchy to each category;

[0013] Based on interaction frequency and complexity Calculate the average interaction frequency and complexity of each layer, and dynamically allocate weights. The stereoscopic unit set and the NeRF radiation field unit set are combined into set A. The unit set, the computation set Optimize weights for each unit and The error and regularization term of the stereo elements and NeRF radiation field elements are minimized by a variable weight joint optimization method. The Adam optimizer is used to iteratively adjust the element parameters to generate the optimized elements. and ;

[0014] Predicting the fusion probability of stereo cells and NeRF radiation field cells using the spatiotemporal Transformer fusion method. Set a fusion probability threshold T. If the fusion probability... If the value is greater than the threshold T, then the stereoscopic unit and the NeRF radiation field unit will be fused. If the fusion probability is... If the value is less than the threshold T, the stereoscopic unit and the NeRF radiation field unit will not be fused, and the original features will be preserved. The fused and unfused units will be mapped onto the three-dimensional mesh structure to generate a stereoscopic-NeRF hybrid mesh.

[0015] As a preferred embodiment of the immersive image transmission method for AR cinemas described in this invention, wherein: the step of prioritizing the collection of real-time interactive data and the stereoscopic-NeRF hybrid grid computing unit refers to the collection of real-time interactive data and the units in the hybrid grid, and the computing unit priority. Set the unit priority threshold ,like Then it is a high-priority unit. Otherwise, it is a low-priority unit. .

[0016] As a preferred embodiment of the immersive image transmission method for AR cinemas described in this invention, the following steps are taken: configuring a priority queue for data within a unit using the QUIC protocol refers to configuring the TLS 1.3 protocol; using session parameters in TLS to initialize the QUIC connection; and calculating the priority of the data stream based on the hierarchical unit characteristics, combined with interaction changes, compressed data volume, network quality, and device performance. Prioritize the data The input QUIC implementation library serves as the basis for priority mapping configuration. The QUIC library performs pairing based on TLS configuration and real-time network performance, binding high-priority data streams to high-interaction units and allocating low-priority QUIC data streams to fill the remaining bandwidth, thereby optimizing throughput.

[0017] As a preferred embodiment of the immersive image transmission method for AR cinemas described in this invention, the efficient transmission of data based on a QUIC priority queue refers to compressing high-priority and low-priority data in the QUIC priority queue using Brotli and Gzip algorithms respectively, encapsulating them into QUIC data packets and marking their priorities. The QUIC protocol dynamically allocates bandwidth based on BBR congestion control and priority scheduling, prioritizing the transmission of high-priority data streams, and adaptively adjusting according to network quality and interaction rate. During transmission, the client receives data through QUIC, prioritizes decoding high-priority data streams to render the focus area in real time, and combines it with Unity to render the scene using a stereoscopic-NeRF mesh. The client receives QUIC data, prioritizes decoding high-priority data streams, maps them to a stereoscopic-NeRF hybrid mesh, and merges them in Unity to generate an AR scene. The rendering resolution is adjusted dynamically based on device performance through resolution scaling. Simultaneously, the Prometheus tool is used to monitor latency, packet loss rate, and frame rate, providing real-time feedback to optimize priority and scheduling strategies.

[0018] In a second aspect, the present invention provides an immersive image transmission system for AR cinemas, including a data loading and point cloud extraction module for loading AR scene data from a cloud server and extracting point cloud datasets.

[0019] The stereo unit generation module is used to generate stereo unit sets, providing the discrete geometric structure of the point cloud and supporting dynamic region partitioning;

[0020] The NeRF radiation field element generation module is used to generate the initial mesh and NeRF element set, supporting fine-grained representation of special effect regions;

[0021] The element optimization and fusion module is used to optimize element parameters, fuse stereo and NeRF radiation field elements, and generate high-quality hybrid meshes.

[0022] The unit and data stream priority configuration transmission module is used to assign priorities to units and data streams, optimize transmission efficiency, and ensure that high-priority data streams arrive first.

[0023] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein the computer program, when executed by the processor, implements any step of the immersive image transmission method for AR cinema as described in the first aspect of the present invention.

[0024] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the immersive image transmission method for AR cinema as described in the first aspect of the present invention.

[0025] The beneficial effects of this invention are as follows: First, by adopting a variable weight joint optimization method, the optimization weights are dynamically calculated and combined with the Adam optimizer to iteratively adjust parameters, thereby prioritizing the optimization of high-interaction areas, improving resource allocation efficiency, and enhancing the detail expression accuracy of special effects areas. Second, addressing the problem of insufficient fusion strategies in existing technologies, this solution introduces a spatiotemporal Transformer fusion method, using a learnable weight matrix to predict the fusion probability, enhancing the semantic association between stereoscopic units and NeRF units through spatiotemporal feature modeling, eliminating light field breaks and geometric distortion, and improving the visual consistency of the fused mesh. Third, addressing the inefficient mesh mapping and layer allocation problem, the solution uses spectral clustering and dynamic weight allocation methods to divide the focus layer, transition layer, and background layer based on interaction frequency and complexity, dynamically adjusting the mesh resolution, significantly improving the rendering quality, real-time interactive experience, and computational efficiency of AR cinema scenes in dynamic special effects areas, providing users with a smooth and immersive viewing experience, while maintaining efficient cloud-to-client transmission performance in complex interactive scenarios. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of an immersive image transmission method for AR cinemas in Example 1.

[0028] Figure 2 This is a schematic diagram of the structure of an immersive image transmission system for AR cinemas in Example 1.

[0029] Figure 3This is a flowchart of the stereoscopic-NeRF hybrid mesh generation process for an immersive image transmission method used in AR cinemas, as shown in Example 1.

[0030] Figure 4 This is a QUIC priority transmission flowchart of an immersive image transmission method for AR cinemas in Example 1. Detailed Implementation

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0033] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0034] Example 1, referring to Figures 1 to 4 This is the first embodiment of the present invention, which provides an immersive image transmission method for AR cinemas, including the following steps:

[0035] S1. Load AR cinema virtual scene data from the cloud server and extract point cloud datasets. Use the weighted K-means algorithm to set each point in the point cloud dataset into a stereoscopic unit set.

[0036] Based on the stereo unit set, a pre-trained GNN model is used to generate a dynamic point cloud subset. An octree partitioning method is used to generate an initial grid from the point cloud subset. The complexity features of each grid region are calculated, and a NeRF radiation field unit set is generated based on the complexity features.

[0037] Specifically, point cloud datasets are extracted from AR cinema virtual scene data loaded onto a cloud server. A weighted K-means algorithm is then used to group each point in the point cloud dataset into a stereoscopic unit set. This involves loading the AR cinema virtual scene onto the cloud server, including 3D models (OBJ format), 4K textures (PNG format), and PBR lighting effects (material parameters), and then extracting the point cloud dataset from it. }, each point ( For 3D positioning, RGB color (RGB color represents the visual attributes of a point and is not involved in distance calculation). (representing the i-th point in the point cloud set P), for each point cloud point... Calculate interaction weights :

[0038] ,

[0039] in, For the f-th focal coordinate, the viewpoint focal coordinates (provided by the eye tracker) are converted into 3D spatial coordinates using the viewpoint matrix. , The Gaussian kernel width is used to control the range of the focal region. Cross-validation is employed to select the optimal kernel width by evaluating the impact of different Gaussian kernel widths on model performance. β is the interaction frequency factor, which is automatically adjusted by optimizing the objective function (such as the accuracy of the interaction effect). The interaction frequency at point t is obtained through multi-sensor data fusion.

[0040] Randomly select a point from the point cloud dataset P as the first center of the stereoscopic unit. The weighted K-means++ algorithm is used to generate the first... Calculate the center of each individual visual unit (n=2,3,…,N) for each point. To the center of the selected stereoscopic unit The minimum weighted distance, based on the weighted distance and interaction weights. Calculate the probability that each point is selected as the new center. :

[0041] ,

[0042] in, Let j be the j-th point in the point cloud set P. Both are dimensional vectors, but their structures are different. same, for The corresponding weights are used to calculate the weighted distance of each point in the denominator. Given the shortest distance from a point to the center of the nth individual visual unit, high-weighted points (such as special effects areas) are more likely to be selected as the center, ensuring a dense distribution of centers;

[0043] Using weighted random sampling, based on probability Choose a point from set P as the new center. (By making the probability value proportional to the interaction weight and the square of the minimum weighted distance, points with high interaction weights (such as water flow and fire effect areas) that are far from the existing center have a higher probability. The probabilities of all points are normalized to form a probability distribution and accumulated into an interval [0, 1]. A random number uniformly distributed within [0, 1] is generated, and the corresponding point is selected based on the probability interval that the random number falls into.) As the new center Ensure that the new centers are distributed in high-interaction areas (such as special effects areas) and not too densely (avoid overlap by using minimum weighted distance), and repeatedly iterate to create new centers. The generation process (calculating distance, probability, sampling, and generating new centers each time) yields a set of stereoscopic unit centers. ;

[0044] Point Assigned to the stereoscopic unit closest to the center point :

[0045] ,

[0046] in, This is a mathematical operator that indicates finding the stereo unit that minimizes the weighted distance among all stereo units.

[0047] stereoscopic unit Form a set The stereoscopic unit set provides the discrete geometric structure of the point cloud and calculates the point density within the unit. :

[0048] ,

[0049] in, For unit Number of points in the inner point cloud , referring to the center Center of the sphere, radius of Volume of a sphere , refers to the maximum distance from a point within a cell to the center.

[0050] By combining interaction weights, this method prioritizes points in high-interaction areas as stereoscopic unit centers, resulting in a denser center distribution and improved accuracy in geometric structure representation. The K-means++ algorithm, through weighted random sampling and minimum weighted distance probability, ensures that the new centers are distributed in high-interaction areas and avoids overlap, significantly improving the ability to capture details in special effects areas. Compared with traditional K-means clustering, the consistency of unit partitioning in dynamic areas is improved, and the calculation of point density within units further optimizes resource allocation, reduces the computational overhead of static areas, and significantly enhances the visual realism and immersive interactive experience of dynamic special effects areas in AR cinema scenes.

[0051] Furthermore, based on the stereo unit set, a pre-trained GNN model is used to generate dynamic point cloud subsets. An octree partitioning method is used to generate an initial grid from the point cloud subsets. The complexity features of each grid region are calculated, and NeRF radiation field units are generated based on these complexity features. Construct a point cloud map, with each stereoscopic unit center point As a node, the node feature vector is The node feature vectors are used as the feature vectors of the stereo vision units. A k-nearest neighbor (e.g., k-NN, k=8) method is used to construct edges in the point cloud graph based on the Euclidean spatial distance (the straight-line distance between 3D coordinates) between the centers of the stereo vision units. A pre-trained GNN model is then used to process each node in the point cloud graph. Predicting dynamic tags :

[0052] ,

[0053] in, The weights are obtained by optimizing a pre-trained GNN model. For aggregate functions, ,in, The edge weights are calculated using a Gaussian kernel function. The set of neighboring nodes of node n (determined by the k-NN method (k=8), including those of the center point) (the k other center points with the smallest Euclidean distance in three-dimensional space) For the first Individual node characteristics The sigmoid function maps the output of the GNN model to [0, 1], and the dynamic labels are... , The threshold value is determined using cross-validation. For indicator functions, if ,but This is indicated as a dynamic effect area (such as fire, magical light effects, etc.). =1 stereoscopic unit The set is defined as follows: the region's extent is determined by the geometric boundaries of the point cloud points within the cell; otherwise... Represented as a static area (such as a wall);

[0054] According to the dynamic special effects area ( =1), generate a dynamic point cloud subset:

[0055] ,

[0056] For dynamic point cloud subsets Perform octree partitioning to generate the initial mesh. (Input the point cloud data into an octree, determine the spatial boundary, construct a bounding box containing all points, recursively divide the bounding box into eight sub-cubes to form an octree structure, each sub-cube corresponding to a node. If the number of points in a node exceeds a preset threshold, further subdivision is performed; otherwise, the node becomes a leaf node, and the point cloud data in the leaf node generates the initial mesh.) Calculate the complexity features of each mesh region:

[0057] ,

[0058] in, For grid volume, , Let the side length of the cube grid be . For the first Each grid interaction factor For the first The complexity of each quantized region;

[0059] Each grid cell was calculated using principal component analysis. Color characteristics [ Using feature binding to combine complexity features and color features [ ] Bound to the center of the mesh (referring to the mesh area) The geometric center of a cube, represented as a three-dimensional coordinate point. , forming a spatial point set { The point set is clustered using K-means clustering, and the number of clusters K is set using the silhouette coefficient method (e.g., 3, representing low, medium, and high complexity ranges). Each cluster represents a NeRF unit. ,Will The elements are set together to obtain the NeRF element set. The NeRF radiation field unit provides continuous scene light field information (such as color and density).

[0060] To address the issues of insufficient accuracy in identifying dynamic effect regions, low efficiency in mesh partitioning, and lack of continuous light field representation in NeRF unit generation in existing AR cinema image transmission technologies, this paper proposes a method for generating dynamic point cloud subsets and constructing NeRF units based on a pre-trained GNN model and octree partitioning. The method predicts dynamic labels using a pre-trained GNN model, calculates edge weights using the k-nearest neighbor method and Gaussian kernel function, accurately identifies dynamic effect regions, and improves the accuracy of dynamic point cloud subset generation. An initial mesh is generated using octree partitioning, and recursive subdivision ensures mesh resolution in highly interactive regions, reducing computational overhead in static regions. By calculating and binding color and complexity features, combined with K-means clustering, a NeRF unit set is generated, providing continuous scene light field information, improving rendering quality, and significantly enhancing the detail representation and rendering consistency of dynamic effect regions in AR cinema scenes.

[0061] S2. Optimize the stereo view unit set and the NeRF radiation field unit set through the variable weight joint optimization method. Use the spatiotemporal Transformer fusion method to predict the fusion probability of stereo view units and NeRF radiation field units. Perform unit fusion according to the fusion probability. Map the fused and unfused units to the three-dimensional mesh structure to generate a stereo-NeRF hybrid mesh.

[0062] Based on the collection of real-time interactive data and the priority of stereo-NeRF hybrid grid computing units;

[0063] Specifically, the stereo view unit set and the NeRF radiation field unit set are optimized using a variable weight joint optimization method. The spatiotemporal Transformer fusion method is used to predict the fusion probability of stereo view units and NeRF radiation field units. Unit fusion is performed based on the fusion probability, and the fused and unfused units are mapped onto a 3D mesh structure to generate a stereo-NeRF hybrid mesh. The node feature vectors are calculated using the mean aggregation method. Average values ​​of each dimension, generating Unit feature vector :

[0064] ,

[0065] Calculate using Gaussian kernel similarity method and Similarity of unit feature vectors :

[0066] ,

[0067] in, denoted as Euclidean distance between eigenvectors. For the first The similarity kernel width is obtained through cross-validation. The similarity of the eigenvectors of the nth individual visual unit and the jth NeRF radiation field unit;

[0068] Based on feature vector similarity Construct a similarity matrix Construct the Laplace matrix :

[0069] ,

[0070] ,

[0071] in, The ( )th degree matrix D , ) elements, These represent the row and column indices of the degree matrix D, respectively.

[0072] Applying spectral clustering to the Laplacian matrix The data is clustered into three layers. Eigenvalue decomposition divides the units in set A into three natural clusters. The elbow rule is used to set layering thresholds: a high interaction frequency threshold H, a medium interaction frequency threshold C, and a complexity threshold O. The clustering is then determined based on the interaction frequency of the units. (The interaction frequency of each unit is obtained by kernel density weighting) ) and complexity The hierarchical levels are assigned to each category as follows:

[0073] The focus layer (highly interactive areas such as special effects) is and O unit;

[0074] The transition layer (medium interaction area) is H unit;

[0075] The background layer (low-interaction areas such as walls) is Units;

[0076] Based on interaction frequency and complexity Calculate the average interaction frequency and complexity of each layer, and dynamically allocate weights. (The average interaction frequency of all units in each layer is calculated using a dynamic weighting method) The average complexity of all units in each layer is calculated using a clustering weighted method. , each layer and The scores are summed to obtain the dynamic importance score for that layer. The scores of each layer are then divided by the sum of the scores of all layers to obtain the normalized weights. The focus layer, due to its highest interaction frequency and complexity, is assigned the highest weight, followed by the transition layers, and the background layer has the lowest weight. The weights are normalized (totaling to 1).

[0077] Set A is formed by combining the stereoscopic element set and the NeRF radiation field element set. The set of elements (N being the total number of stereoscopic elements and NeRF radiation field elements) is used to calculate the set. Optimize the weights for each unit:

[0078] ,

[0079] ,

[0080] in, This refers to the error term in the stereoscopic unit, specifically the point cloud reprojection error, which is obtained through the point cloud reprojection error calculation method. This is the error term in the NeRF unit, referring to the rendering color error, obtained through the radiation field rendering error integration method, δ= The values ​​are small positive numbers, obtained using a numerical stability setting method to prevent the denominator from being zero. The interaction factor is obtained through hyperparameter tuning methods. Optimize the weights for the nth view cell. Let j be the j-th NeRF radiation field element;

[0081] The error and regularization term of the stereo elements and NeRF radiation field elements are minimized by a variable-weight joint optimization method. The element parameters are then adjusted iteratively using the Adam optimizer to generate the optimized elements.

[0082] ,

[0083] ,

[0084] in, λ is the weighted mean, obtained through mean aggregation, and λ is the regularization weight (tuned through cross-validation). For the nth optimized stereoscopic unit, For the j-th optimized NeRF radiation field element;

[0085] Predicting the fusion probability of stereoscopic cells and NeRF radiation field cells using the spatiotemporal Transformer fusion method:

[0086] ,

[0087] in, Let be the fusion probability of the nth individual visual unit and the jth NeRF radiation field unit. The learnable weight matrix is ​​obtained by training by minimizing the cross-entropy loss;

[0088] The fusion probability threshold T is set by using cosine similarity. If the value is greater than the threshold T, the stereo unit will be fused with the NeRF radiation field unit;

[0089] If the fusion probability If the value is less than the threshold T, the stereo unit and the NeRF radiation field unit will not be fused, and the original features will be preserved. The fused and unfused units will be mapped to the three-dimensional mesh structure. (The fused units (weighted average of point cloud and radiation field parameters) and the unfused units (original features) will be mapped to the three-dimensional mesh by weighted mapping. If multiple units fall into the same mesh, the features will be fused by weighted averaging.) This will generate a stereo-NeRF hybrid mesh.

[0090] To address the issues of insufficient dynamic adaptability of unit optimization, lack of semantic fusion, inefficient mesh allocation, and high computational complexity in existing AR cinema image transmission technologies, this paper proposes a variable-weight joint optimization method. This method utilizes point cloud reprojection errors and interaction factors to dynamically adjust optimization weights, prioritizing the enhancement of detail in high-interaction areas, thus improving rendering quality and resource allocation efficiency. A spatiotemporal Transformer fusion method is employed, using a learnable weight matrix to predict fusion probabilities. This optimizes mesh resolution allocation for units fusing high-interaction areas, achieving semantic consistency between stereoscopic units and NeRF units, enhancing detail in special effects areas, eliminating light field breaks, and improving visual consistency. Furthermore, spectral clustering and dynamic weight allocation are used to divide the focus layer, transition layer, and background layer based on interaction frequency and complexity, optimizing mesh resolution allocation, enhancing detail in high-interaction areas, and reducing computational overhead in static areas. Finally, by optimizing the similarity matrix construction and using a weighted mapping method to generate a stereoscopic-NeRF hybrid mesh, this method ensures a significant improvement in the realism of dynamic special effects rendering and the real-time interactive experience in AR cinema scenes.

[0091] Furthermore, based on the collection of real-time interactive data and the priority of computing units in the stereo-NeRF hybrid mesh, the priority of computing units is determined by the collection of real-time interactive data, including interaction frequency, sensor data, network status, device performance, and cells in the hybrid mesh:

[0092] ,

[0093] in, For the first Unit priority weight, For the first Each focal point is obtained through eye tracking. The maximum interaction frequency is obtained using the maximum pooling method. The distance from the stereo center to the focal point is obtained through geometric calculation. ∈[0,1] represents network state factors, such as bandwidth, latency, and packet loss rate, which quantify the real-time performance of the network. ∈[0,1] represents the device performance factor, which is obtained by weighted averaging of frame rate, bandwidth, computing power, and memory.

[0094] Setting unit priority thresholds using the analytic hierarchy process. ,like Then it is a high-priority unit. Otherwise, it is a low-priority unit. .

[0095] To address the issues of lack of dynamic adaptability in data priority allocation and inability to effectively handle real-time interaction changes and network fluctuations in existing AR cinema image transmission technologies, this paper proposes a method that integrates interaction frequency, viewpoint focus, geometric distance, network state factors, and device performance factors. It utilizes the analytic hierarchy process (AHP) to calculate unit priority weights, enabling priority processing of high-interaction areas and improving resource allocation efficiency. Compared to traditional static priority rules, this method dynamically adapts to changes in network bandwidth, latency, and device performance, ensuring reduced transmission latency for high-priority units and improved rendering frame rate stability. Furthermore, by employing max pooling and eye-tracking methods to accurately identify user interaction focus and optimize data scheduling in special effects areas, this method significantly enhances the immersive interactive experience of AR cinema scenes, improves overall smoothness, and provides efficient support for high-quality viewing in complex interactive environments.

[0096] S3. Use the QUIC protocol to configure a priority queue for the data within the unit, and transmit the data efficiently according to the QUIC priority queue;

[0097] Specifically, the priority queue configuration within the cell is achieved using the QUIC protocol, which involves configuring the TLS 1.3 protocol. Session parameters (such as session keys) from TLS are used to initialize the QUIC connection, ensuring that the priority queue configuration is executed over a secure QUIC connection. Based on the layered cell characteristics, combined with interaction variations, compressed data volume, network quality, and device performance, the priority of the data stream is calculated.

[0098] ,

[0099] Where σ represents the Sigmoid function, , and The weights are dynamically adjusted (obtained through adaptive weight optimization). For the first Each mesh feature vector (a feature vector assigned to each 3D mesh cell during the stereo-NeRF hybrid mesh generation process, such as coordinates, color, interaction factor, density, and complexity, generated through a weighted mapping method). For the first The compressed data volume (obtained through data stream compression and statistical methods). No. Network quality parameters (such as bandwidth fluctuation and latency jitter; bandwidth fluctuation is calculated by measuring the standard deviation of the time series using continuous bandwidth testing (such as iPerf); latency jitter is extracted from the latency variance using network analysis tools (such as Wireshark) For the first Each device performance factor;

[0100] Prioritize the data The input QUIC implementation library serves as the basis for priority mapping configuration. The QUIC library performs pairing based on TLS configuration and real-time network performance (such as bandwidth and latency), binding high-interaction units (focus layer) to high-priority data streams, and assigning low-interaction units (transition layer and background layer) to QUIC low-priority data streams to fill the remaining bandwidth, thus optimizing throughput.

[0101] To address the issues of static data transmission priority allocation and inability to adapt to network fluctuations and interactive changes in existing AR cinema image transmission technologies, this paper combines hierarchical unit characteristics, interactive changes, compressed data volume, network quality, and device performance. It employs the Sigmoid function and dynamically adjusted weights to calculate data stream priorities, prioritizing data streams in high-interaction areas and reducing transmission latency. Secure QUIC connections are initialized using the TLS 1.3 protocol, and QUIC is used to dynamically pair high-priority data streams, optimizing throughput. Data streams in low-interaction areas fill remaining bandwidth, improving resource utilization. Compared to traditional TCP protocols or static scheduling, BBR congestion control and real-time network performance feedback adaptively adjust bandwidth allocation, ensuring improved interactive smoothness in AR cinema scenes under complex network environments and significantly enhancing the immersive viewing experience.

[0102] Furthermore, efficient data transmission based on the QUIC priority queue involves compressing high-priority and low-priority data using Brotli and Gzip algorithms respectively, encapsulating them into QUIC data packets, and marking their priorities. The QUIC protocol dynamically allocates bandwidth based on BBR congestion control and priority scheduling, prioritizing the transmission of high-priority data streams and adaptively adjusting according to network quality and interaction rate. During transmission, the client receives data via QUIC, prioritizing the decoding of high-priority data streams for real-time rendering of the focus area. This is combined with Unity's fused stereo-NeRF mesh rendering scene (in the Unity engine, the stereo-NeRF rendering pipeline maps decoded data to a stereo-NeRF hybrid mesh, using stereo point clouds). The AR scene is generated by fusing Open3D rendering and NeRF light field rendering (PyTorch), and the rendering resolution is determined based on the device performance k. The client receives QUIC data, prioritizes decoding high-priority data streams, and maps them to a stereo-NeRF hybrid mesh (the stereo point cloud is rendered by Open3D, and the NeRF light field is rendered by PyTorch). The AR scene is then generated in Unity by fusing the data. The rendering resolution is dynamically adjusted (range 0.5x-1.0x) based on the device performance (CPU / GPU computing power collected by the sensor). At the same time, the latency, packet loss rate, and frame rate are monitored by the Prometheus tool, and the optimization priorities and scheduling strategies are fed back in real time to form a closed-loop optimization to ensure a smooth experience in interactive scenes.

[0103] To address the issues of high transmission latency, lack of adaptive priority scheduling, and limited rendering efficiency in existing AR cinema image transmission technologies, this paper proposes a method that uses Brotli and Gzip algorithms to compress high-priority and low-priority data respectively. Combined with BBR congestion control and priority scheduling of the QUIC protocol, bandwidth is dynamically allocated, prioritizing the transmission of data streams in high-interaction areas. This reduces transmission latency and increases throughput. By utilizing the Unity stereoscopic-NeRF rendering pipeline, data is mapped to a stereoscopic-NeRF hybrid mesh in real time, improving the detail accuracy of rendering focal areas. Dynamic resolution scaling is used to adapt to device performance, improving frame rate stability and significantly enhancing smoothness on low-end devices. The Prometheus tool monitors latency, packet loss rate, and frame rate in real time, forming a closed-loop optimization mechanism that dynamically adjusts priorities and scheduling strategies to improve interactive smoothness and significantly enhance the immersive viewing experience in AR cinema scenes.

[0104] This embodiment also provides an immersive image transmission system for AR cinemas, including:

[0105] The data loading and point cloud extraction module is used to load AR scene data from the cloud server and extract point cloud datasets.

[0106] The stereo unit generation module is used to generate stereo unit sets, providing the discrete geometric structure of the point cloud and supporting dynamic region partitioning;

[0107] The NeRF radiation field element generation module is used to generate the initial mesh and NeRF element set, supporting fine-grained representation of special effect regions;

[0108] The element optimization and fusion module is used to optimize element parameters, fuse stereo and NeRF radiation field elements, and generate high-quality hybrid meshes.

[0109] The unit and data stream priority configuration transmission module is used to assign priorities to units and data streams, optimize transmission efficiency, and ensure that high-priority data streams arrive first.

[0110] This embodiment also provides a computer device applicable to an immersive image transmission method for AR cinemas, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the immersive image transmission method for AR cinemas as proposed in the above embodiment.

[0111] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0112] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the immersive image transmission method and system for AR cinemas as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0113] In summary, this invention employs a variable-weight joint optimization method, dynamically calculating and optimizing weights, and iteratively adjusting parameters using the Adam optimizer to prioritize optimization of high-interaction areas, thereby improving resource allocation efficiency and enhancing the detail accuracy of special effects areas. Secondly, addressing the insufficient fusion strategies in existing technologies, this solution introduces a spatiotemporal Transformer fusion method. It utilizes a learnable weight matrix to predict fusion probabilities, enhances the semantic association between stereoscopic units and NeRF units through spatiotemporal feature modeling, eliminates light field breaks and geometric distortions, and improves the visual consistency of the fused mesh. Finally, addressing the inefficiency of mesh mapping and layer allocation, it uses spectral clustering and dynamic weight allocation to divide the focus layer, transition layer, and background layer based on interaction frequency and complexity, dynamically adjusting the mesh resolution. This significantly improves the rendering quality, real-time interactive experience, and computational efficiency of AR cinema scenes in dynamic special effects areas, providing users with a smooth and immersive viewing experience while maintaining efficient cloud-to-client transmission performance in complex interactive scenarios.

[0114] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An immersive image transmission method for AR cinemas, characterized in that: include, The point cloud dataset is extracted from the AR cinema virtual scene data loaded by the cloud server, and the weighted K-means algorithm is used to combine each point in the point cloud dataset into a stereoscopic unit set. Based on the stereo unit set, a pre-trained GNN model is used to generate a dynamic point cloud subset. An octree partitioning method is used to generate an initial grid from the point cloud subset. The complexity features of each grid region are calculated, and a NeRF radiation field unit set is generated based on the complexity features. The stereoscopic unit set and the NeRF radiation field unit set are optimized using a variable-weight joint optimization method, whereby the variable-weight joint optimization method refers to calculating the node eigenvectors through mean aggregation. Average values ​​of each dimension, generating Unit feature vector The Gaussian kernel similarity method is used to calculate and Similarity of unit feature vectors Based on feature vector similarity Construct a similarity matrix Construct the Laplace matrix Spectral clustering is applied to the Laplacian matrix. The data is clustered into three layers. eigenvalue decomposition divides the units in set A into three natural clusters. A high interaction frequency threshold H, a medium interaction frequency threshold C, and a complexity threshold O are set. The clusters are then determined based on the interaction frequency of the units. and complexity Assign a hierarchy to each category; Based on interaction frequency and complexity Calculate the average interaction frequency and complexity of each layer, and dynamically allocate weights. The stereoscopic unit set and the NeRF radiation field unit set are combined into a set A, A = { , The set of elements of the set |n=1,…,N,j=1,…,N is used to compute the set. Optimize weights for each unit and The error and regularization term of the stereo elements and NeRF radiation field elements are minimized by a variable weight joint optimization method. The Adam optimizer is used to iteratively adjust the element parameters to generate the optimized elements. and ; The spatiotemporal Transformer fusion method is used to predict the fusion probability of stereoscopic cells and NeRF radiation field cells. Cell fusion is performed based on the fusion probability, and fused and unfused cells are mapped to a 3D mesh structure to generate a stereoscopic-NeRF hybrid mesh. Based on the collection of real-time interactive data and the priority of stereo-NeRF hybrid grid computing units, the data within the unit is configured into a priority queue using the QUIC protocol, and the data is transmitted efficiently according to the QUIC priority queue.

2. The immersive image transmission method for AR cinemas as described in claim 1, characterized in that: The process involves loading AR cinema virtual scene data onto a cloud server, extracting a point cloud dataset from it, and then using a weighted K-means algorithm to group each point in the point cloud dataset into a stereoscopic unit set. This process is repeated in the original text. Each point For each point cloud point Calculate interaction weights A point is randomly selected from the point cloud dataset P as the first center of the stereoscopic unit. The weighted K-means++ algorithm is used to generate the first... Individual view unit center, calculate each point The minimum weighted distance to the center of the selected stereoscopic unit is determined based on the weighted distance and interaction weight. Calculate the probability that each point is selected as the new center. Using weighted random sampling, based on probability Choose a point from set P as the new center. Repeatedly iterate to create new centers The generation step obtains the set of stereoscopic unit centers, and then points... Assigned to the stereoscopic unit closest to the center point stereoscopic unit Form a set And calculate the point density within the cell. .

3. The immersive image transmission method for AR cinemas as described in claim 2, characterized in that: The process involves generating dynamic point cloud subsets based on a stereoscopic unit set using a pre-trained GNN model, creating an initial grid from the point cloud subsets using an octree partitioning method, calculating the complexity features of each grid region, and generating NeRF radiation field units based on these complexity features. Construct a point cloud map, with each stereoscopic unit center point As a node, the node feature vector is The k-nearest neighbor method is used to construct edges in the point cloud graph based on the Euclidean spatial distance between the centers of the stereo vision units. A pre-trained GNN model is then used to process each node in the point cloud graph. Predicting dynamic tags Generate a dynamic point cloud subset based on the dynamic effect area. For dynamic point cloud subsets Perform octree partitioning to generate the initial mesh. Calculate the complexity features of each grid region. Principal component analysis was used to calculate the value of each grid cell. Color characteristics Using feature binding to combine complexity features and color features Bind to the center coordinates of the grid , forming a spatial point set K-means clustering was used to cluster the point set, with K being the number of clusters and each cluster representing a NeRF unit. ,Will The elements are set together to obtain the NeRF element set. .

4. The immersive image transmission method for AR cinemas as described in claim 3, characterized in that: The method optimizes the stereocell set and the NeRF radiation field cell set using a variable-weight joint optimization method. It then uses a spatiotemporal Transformer fusion method to predict the fusion probability of stereocells and NeRF radiation field cells, performs cell fusion based on this probability, and maps fused and unfused cells onto a 3D mesh structure to generate a stereo-NeRF hybrid mesh. The method also uses the spatiotemporal Transformer fusion method to predict the fusion probability of stereocells and NeRF radiation field cells. Set a fusion probability threshold T. If the fusion probability... If the value is greater than the threshold T, then the stereoscopic unit and the NeRF radiation field unit will be fused. If the fusion probability is... If the value is less than the threshold T, the stereoscopic unit and the NeRF radiation field unit will not be fused, and the original features will be preserved. The fused and unfused units will be mapped onto the three-dimensional mesh structure to generate a stereoscopic-NeRF hybrid mesh.

5. The immersive image transmission method for AR cinemas as described in claim 4, characterized in that: The phrase "based on the collection of real-time interactive data and the priority of computational units in the stereo-NeRF hybrid mesh" refers to the collection of real-time interactive data and the priority of computational units within the hybrid mesh. Set the unit priority threshold ,like Then it is a high-priority unit. Otherwise, it is a low-priority unit. .

6. The immersive image transmission method for AR cinemas as described in claim 5, characterized in that: The step of configuring a priority queue for data within a unit using the QUIC protocol refers to configuring the TLS 1.3 protocol, using session parameters in TLS to initialize the QUIC connection, and calculating the priority of the data stream based on the layered unit characteristics, combined with interaction changes, compressed data volume, network quality, and device performance. Prioritize the data The input QUIC implementation library serves as the basis for priority mapping configuration. The QUIC library performs pairing based on TLS configuration and real-time network performance, binding high-priority data streams to high-interaction units and allocating low-priority QUIC data streams to fill the remaining bandwidth, thereby optimizing throughput.

7. The immersive image transmission method for AR cinemas as described in claim 6, characterized in that: The efficient data transmission based on the QUIC priority queue refers to compressing high-priority and low-priority data in the QUIC priority queue using Brotli and Gzip algorithms respectively, encapsulating them into QUIC data packets and marking their priorities. The QUIC protocol dynamically allocates bandwidth based on BBR congestion control and priority scheduling, prioritizing the transmission of high-priority data streams, and adaptively adjusting according to network quality and interaction rate. During transmission, the client receives data through QUIC, prioritizes decoding high-priority data streams to render the focus area in real time, and combines it with Unity's fused stereo-NeRF mesh rendering scene. The client receives QUIC data, prioritizes decoding high-priority data streams, maps them to the stereo-NeRF hybrid mesh, and merges them in Unity to generate an AR scene. The rendering resolution is adjusted dynamically based on device performance through resolution scaling. At the same time, the Prometheus tool is used to monitor latency, packet loss rate, and frame rate, providing real-time feedback to optimize priority and scheduling strategies.

8. An immersive image transmission system for AR cinemas, based on the immersive image transmission method for AR cinemas according to any one of claims 1 to 7, characterized in that: include, The data loading and point cloud extraction module is used to load AR scene data from the cloud server and extract point cloud datasets. The stereo unit generation module is used to generate stereo unit sets, providing the discrete geometric structure of the point cloud and supporting dynamic region partitioning; The NeRF radiation field element generation module is used to generate the initial mesh and NeRF element set, supporting fine-grained representation of special effect regions; The element optimization and fusion module is used to optimize element parameters, fuse stereo and NeRF radiation field elements, and generate high-quality hybrid meshes. The unit and data stream priority configuration transmission module is used to assign priorities to units and data streams, optimize transmission efficiency, and ensure that high-priority data streams arrive first.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the immersive image transmission method for AR cinema as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the immersive image transmission method for AR cinema as described in any one of claims 1 to 7.

Images