Augmented reality 3D modeling system based on light field distribution

Abstract

This invention relates to the field of augmented reality technology, and more specifically, to an augmented reality 3D modeling system based on light field distribution. The system includes: a high-dimensional light field data acquisition unit for deploying a spherical array camera system to generate high-dimensional light field data; a data analysis and calculation unit for establishing a light field gradient tensor based on the high-dimensional light field data and correcting the light field gradient tensor using propagation characteristics to characterize spatial light intensity changes; and a region partitioning and rendering unit for calculating light field gradient discontinuities based on the corrected light field gradient tensor, partitioning high-curvature regions, and performing real-time rendering based on the partitioning results. This invention can accurately depict optical phenomena in complex scenes, such as reflection, refraction, and scattering, while optimizing detail capture in high-curvature regions and reducing redundant calculations in smooth regions.

Description

Technical Field

[0001] This invention belongs to the field of augmented reality technology, specifically relating to an augmented reality 3D modeling system based on light field distribution. Background Technology

[0002] Augmented reality (AR) technology, an advanced technology that overlays and merges virtual information with real-world scenes, has been widely applied in entertainment, education, healthcare, and industry in recent years. The core of an AR system lies in the perception of real-world scenes and the high-precision overlay of virtual information, with 3D modeling technology being a crucial element. To achieve high-quality AR effects, accurate 3D reconstruction of the real-world scene is essential, combined with the object's geometry, surface material, and lighting conditions to present a natural fusion between virtual objects and the real-world scene in real time. While existing technologies have made significant progress in 3D modeling and light field processing, many technical bottlenecks remain in the modeling efficiency of complex scenes, the reproduction of optical properties, and real-time rendering effects.

[0003] Multi-view geometry is an important method in traditional 3D modeling. Its core idea is to capture scene images using multi-view cameras and then calculate the 3D point cloud of the target object using parallax. Typical multi-view geometry methods include structured light, photometric stereo, and binocular stereo vision. These methods perform well in modeling simple objects, but have significant limitations when dealing with complex scenes. For example, multi-view geometry methods struggle to obtain accurate depth information for objects with transparent or reflective materials. Furthermore, these methods lack the ability to model the characteristics of light propagation, thus failing to realistically reproduce the appearance of the target object in complex optical environments. Light field cameras can capture the spatial distribution and direction information of light in a single shot, providing a completely new solution for 3D modeling. Unlike traditional cameras that only record two-dimensional images, light field cameras can capture four-dimensional light field data (i.e., the spatial position and direction of light rays), allowing for 3D modeling through light field refocusing technology in post-processing. The advantage of this type of method is that it can directly acquire multi-view data of the scene, avoiding the tedious operation of multiple shots required in multi-view geometry. However, due to the limitations of spatial resolution, light field cameras typically acquire light field data with low accuracy, especially in high-curvature regions and complex material scenes, resulting in unsatisfactory modeling accuracy and realism. In recent years, with the rapid development of deep learning technology, machine learning-based 3D modeling methods have gradually emerged. For example, by utilizing neural networks to perform deep learning on multi-view images, the 3D structure and surface texture of target objects can be predicted. These methods perform well when handling objects with regular structures (such as buildings and furniture), but their modeling effects are poor for complex materials, dynamic lighting, and translucent or reflective objects. Furthermore, these methods typically require large amounts of training data and rely on expensive computing resources, making them difficult to promote in real-time augmented reality applications. Summary of the Invention

[0004] The main objective of this invention is to provide an augmented reality 3D modeling system based on light field distribution. By combining spatial and angular dimensional information of light, this invention can accurately depict optical phenomena in complex scenes, such as reflection, refraction, and scattering, while optimizing detail capture in high-curvature areas and reducing redundant calculations in smooth areas. Its beneficial effects include improving the realism and accuracy of 3D modeling, optimizing modeling efficiency, supporting real-time updates in dynamic scenes, and achieving a natural integration of virtual objects with real-world scenes.

[0005] To solve the above problems, the technical solution of the present invention is implemented as follows:

[0006] An augmented reality 3D modeling system based on light field distribution, the system comprising:

[0007] The high-dimensional light field data acquisition unit is used to deploy a spherical array camera system. Each camera sampling point includes a main camera and four auxiliary cameras. The main camera acquires high-resolution spatial light field information, and the auxiliary cameras acquire HDR light field information with different exposure parameters to form high-dimensional light field data.

[0008] The data analysis and computation unit is used to perform optical field interpolation using anisotropic kernel functions based on high-dimensional optical field data to obtain interpolated high-dimensional optical field data; to construct a hierarchical optical field propagation model and solve the propagation characteristics of light in different media based on the interpolated high-dimensional optical field data; to establish an optical field gradient tensor and use propagation characteristics to correct the optical field gradient tensor to characterize changes in spatial light intensity.

[0009] The region partitioning and rendering unit is used to calculate the discontinuities of the light field gradient based on the corrected light field gradient tensor. Through the singular value decomposition of the light field gradient tensor, it identifies the contours and surfaces of the target object, constructs an adaptive octree structure, partitions high curvature regions, and performs real-time rendering based on the partitioning results.

[0010] Furthermore, the execution process of the high-dimensional light field data acquisition unit specifically includes: uniformly arranging 12 camera sampling points on a sphere centered on the target object, with the sphere radius adaptively adjusted according to the size of the target object, and the sphere radius being larger than the maximum diameter of the target object; each camera sampling point includes one 4K resolution main camera and four 2K resolution auxiliary cameras arranged along the cross direction; using structured light projection to calibrate the internal camera group and obtain the pose relationship between each camera; performing global calibration of the entire spherical array camera system through a spherical calibration plate to establish a unified spherical coordinate system; the four auxiliary cameras adopt different exposure parameters, namely -2EV, -1EV, +1EV, and +2EV; the maximum aperture value of the main camera lens is 1.4; synchronously triggering all cameras with a sampling frequency of 30fps; performing HDR synthesis on the data acquired by the four auxiliary cameras at each camera sampling point to obtain HDR data, using an adaptive weighting algorithm to reduce the influence of overexposed and underexposed areas, and achieving a linear HDR light field through camera response curve correction; and fusing the high-resolution data acquired by the main camera with the HDR data to construct high-dimensional light field data.

[0011] Furthermore, the high-dimensional light field data is a vector whose vector elements include: spatial position coordinates, orientation angle, wavelength, and exposure parameters; the spatial position coordinates include: X-axis coordinates, Y-axis coordinates, and Z-axis coordinates; the orientation angle includes: the angle between the camera position and the Z-axis in the spherical coordinate system, and the angle between the camera and the X-axis in the XY plane.

[0012] Furthermore, the data analysis and computation unit, based on high-dimensional light field data, performs light field interpolation using anisotropic kernel functions. Specifically, this includes: spatially resampling the high-dimensional light field data to ensure uniform data distribution in the spherical sampling space; normalizing the high-dimensional light field data at each camera sampling point to eliminate systematic errors between different camera sampling points (this process is achieved through iterative least squares) until the high-dimensional light field data at all camera sampling points meet a preset consistency threshold, resulting in interpolated high-dimensional light field data; based on the surface of the target object... The shape and size of the anisotropic kernel function are adaptively designed based on local geometric features. In high-frequency detail regions corresponding to the edges of the target object, a first anisotropic kernel function is used to support the domain to preserve detail features. In smooth regions, a second anisotropic kernel function is used to support the domain to enhance data continuity. The shape and size of the first and second anisotropic kernel functions are controlled by the covariance matrix, which is determined by the principal direction of the local light field gradient tensor. At the same time, anisotropic Gaussian weights are introduced to ensure a smooth transition of the interpolation results in both spatial and angular dimensions.

[0013] Furthermore, the data analysis and computation unit constructs a hierarchical light field propagation model to describe the propagation characteristics of light in different media. Specifically, this process includes: constructing the hierarchical light field propagation model; firstly, at the coarsest level, ray tracing technology is used to simulate the propagation characteristics of light in a homogeneous medium; then, the model is refined layer by layer, considering the non-homogeneity and anisotropy of the medium in each layer; combining the interpolated high-dimensional light field data, the propagation characteristics of light in different media are obtained by solving the light field transmission model. This is a three-dimensional vector whose elements include: reflection characteristic parameters, refraction characteristic parameters, and scattering characteristic parameters. This process requires iterative calculation until the difference in the light field between adjacent levels is less than a preset threshold; the reflection characteristic parameter is reflectivity; the refraction characteristic parameter is refractive index; and the scattering characteristic parameter is scattering rate.

[0014] Furthermore, the data analysis and computation unit establishes an optical field gradient tensor and uses propagation characteristics to correct the optical field gradient tensor. The process of characterizing spatial light intensity changes specifically includes: constructing a fourth-order optical field gradient tensor by calculating the partial derivatives of the optical field propagation model in the spatial and angular dimensions; multiplying the fourth-order optical field gradient tensor by the three-dimensional vector corresponding to the propagation characteristics to obtain the propagation characteristic-corrected optical field gradient tensor, which is used to characterize the local variation characteristics of the optical field. The eigenvalues ​​and eigenvectors of this fourth-order optical field gradient tensor reveal the main direction and intensity of the optical field changes.

[0015] Furthermore, the region segmentation and rendering unit, based on the light field gradient tensor, calculates the light field gradient discontinuities. The process of identifying the target object's contour and surface through singular value decomposition of the light field gradient tensor specifically includes: normalizing each component of the fourth-order light field gradient tensor, then setting a global threshold. If the gradient amplitude of a camera sampling point exceeds the global threshold, it is determined that there is a discontinuity in the contour or material boundary, and the camera sampling point is considered a discontinuity point. Near the detected discontinuities, singular value decomposition is performed on the light field gradient tensor. The decomposed eigenvalues ​​and eigenvectors reveal the main direction and intensity of the gradient change. When the largest singular value is greater than other singular values, and the sum of the differences with other singular values ​​exceeds the set first judgment threshold, it indicates that there is a directional change at the camera sampling point, corresponding to features on the surface or contour. If multiple singular values ​​are similar, it indicates that the area around the camera sampling point is smooth or has isotropic characteristics.

[0016] Furthermore, the region partitioning and rendering unit constructs an adaptive octree structure. The specific process for partitioning high-curvature regions includes: after obtaining the overall bounding box of the target object, an octagon is built at the coarsest level to enclose the entire target object; within each octagon, a set number of camera sampling points are maintained; for each octagon, the variance or mean curvature of its internal light field gradient tensor is calculated; if the variance or mean curvature exceeds a set second judgment threshold, it indicates that there is a surface change in this region, and it needs to be further subdivided into smaller octagons; the smaller subdivided octagons continue to undergo the same judgment until the subdivision stopping condition is met. Subdivision stops when the octagon size reaches the resolution limit or the light field gradient tensor is in a smooth state. This adaptive octagon structure can not only effectively partition the target object's region in space, but also reduce oversampling and calculation in smooth regions and increase sampling and calculation in high-curvature regions.

[0017] Furthermore, local extremum analysis is performed on the light field gradient tensor contained in each smallest octagon to find the local maximum or minimum light intensity points in the spatial and angular dimensions, as well as the points where the direction of the light field gradient tensor changes. Due to the differences in the reflection or refraction properties of different materials, their light field gradient tensors will exhibit different extremum distributions at specific angles. By analyzing the extremum distribution, the criteria for determining material boundaries are obtained. For the same octagon, if two or more distinct extremum distributions of the light field gradient tensor are observed, it is determined that there is a material transition zone within it. Further, combined with the propagation characteristics of light in different media, it is identified which extrema belong to the same material region. When the covariance of the extremum distributions of the light field gradient tensors corresponding to the camera sampling points in multiple octagons is within the set third judgment threshold, they are merged into the same material region. Conversely, if the differences in the extremum distributions are large, they are determined to be adjacent different material regions, forming a meshed description of the surface material partitions of the target object.

[0018] The augmented reality 3D modeling system based on light field distribution of this invention has the following beneficial effects: The light field gradient tensor analysis and material partitioning technology employed in this invention solve the problem of identifying and partitioning complex material boundaries, improving the system's adaptability to multi-material target objects. In real-world scenarios, target objects are typically composed of multiple different materials, and their optical properties exhibit significant differences at boundaries and transition regions. This invention, through local extremum analysis of the light field gradient tensor combined with light propagation characteristics, can accurately determine the location of material boundaries and distinguish material properties within transition regions. Furthermore, this invention, through covariance analysis of light field data from different regions, can identify and merge regions of the same material, while dividing regions with significant differences into adjacent meshes of different materials. This high-precision material partitioning capability not only presents more realistic material representations in augmented reality but also provides a reliable basis for the lighting interaction between virtual objects and regions of different materials. The adaptive octree structure of this invention optimizes the efficiency and precision of 3D modeling, achieving efficient computation and optimized resource allocation in complex scenes. Existing modeling methods typically use uniform resolution for spatial partitioning, which can lead to the loss of geometric details in high-curvature regions or waste of computational resources in smooth regions. This invention subdivides and dynamically adjusts the spatial region of the target object layer by layer, enabling the system to increase sampling density in high-curvature regions and reduce subdivision depth in regions with smooth light field gradient changes. This adaptive subdivision strategy not only improves the system's ability to capture the geometric features of complex object surfaces but also significantly reduces the demand for computing resources, ensuring the efficient operation of augmented reality systems in real-time scenarios. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the system flow of an augmented reality 3D modeling system based on light field distribution provided in an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] Example 1, Reference Figure 1 An augmented reality 3D modeling system based on light field distribution, the system comprising:

[0022] The high-dimensional light field data acquisition unit is used to deploy a spherical array camera system. Each camera sampling point includes a main camera and four auxiliary cameras. The main camera acquires high-resolution spatial light field information, and the auxiliary cameras acquire HDR light field information with different exposure parameters to form high-dimensional light field data.

[0023] In this process, each sampling point is equipped not only with a high-resolution main camera to capture rich image information, but also with several auxiliary cameras to record HDR data under different exposure parameters, thus encompassing details in both dark and bright areas that are difficult to capture with traditional single-exposure acquisition. Since the light field contains not only spatial distribution information but also rich variations in the angle and intensity of light, the camera orientation and exposure combinations must be flexibly adjusted based on the spherical array to ensure accurate observation of shadow areas, bright areas, and translucent materials in the target environment. During actual acquisition, the main camera often plays a primary role in image resolution and fine structure fidelity, while the auxiliary cameras, through varying exposure times or sensitivity parameters, gradually expand the dynamic range to cover lighting conditions ranging from extremely dark to extremely bright. This high-dimensional light field data formed by multi-exposure fusion not only provides high-precision sampling of light direction and spatial position but also lays a reliable foundation for subsequent light field interpolation, light propagation analysis, and gradient tensor correction. In the augmented reality 3D modeling process emphasized in this invention, the scene information output by the high-dimensional light field data acquisition unit under multiple angles, resolutions, and exposure parameters can effectively cope with complex lighting changes and material differences in the real environment, because these changes directly affect the correct identification of the curvature of the object surface, edge contours, and refraction behavior of transparent media.

[0024] Relying solely on traditional multi-view geometry or single light field cameras for imaging often results in problems such as loss of detail in dark areas, overexposure in bright areas, and blurred boundaries of transparent objects due to insufficient exposure range or limited viewing angle. However, by deploying multiple cameras on a spherical acquisition platform and setting differentiated exposure schemes, the high-dimensional light field data acquisition unit can simultaneously capture the clearest details and the most realistic light intensity distribution. This allows the system to perform more accurate and realistic calculations based on diverse light field information during the post-processing stage for 3D modeling. It is worth noting that this spherical array combination is not merely a simple stacking of numbers, but rather fully considers the uniform arrangement of cameras in polar or spherical coordinates, ensuring that each direction has approximately the same angular resolution and coverage depth. With this layout, the acquisition unit can complete high-density sampling of the scene in a relatively short time and fuse image information from different cameras into a unified high-dimensional light field dataset, thus avoiding interpolation difficulties caused by missing viewing angles or resolution imbalances. Especially in augmented reality applications, when systems need to dynamically fuse complex environments in real time, extremely high precision is required for the complete distribution of light and the appearance information of objects at different angles. This high-dimensional light field acquisition unit meets these stringent requirements by utilizing a multi-camera, multi-exposure, and multi-viewpoint stereoscopic layout. By combining the high-resolution images obtained by the main camera with the HDR synthesis results from the auxiliary camera, the system can more accurately infer the brightness values ​​of missing sampling points in the subsequent light field interpolation stage. It also effectively characterizes the propagation trajectory of light on semi-transparent media or reflective surfaces in the layered light field propagation model, and provides a solid data foundation for the construction and correction of the light field gradient tensor.

[0025] The data analysis and computation unit is used to perform optical field interpolation using anisotropic kernel functions based on high-dimensional optical field data to obtain interpolated high-dimensional optical field data; to construct a hierarchical optical field propagation model and solve the propagation characteristics of light in different media based on the interpolated high-dimensional optical field data; to establish an optical field gradient tensor and use propagation characteristics to correct the optical field gradient tensor to characterize changes in spatial light intensity.

[0026] This unit needs to effectively integrate high-dimensional light field data acquired by a spherical array camera system through multiple exposures and perspectives. This high-dimensional data often exhibits some degree of sampling inhomogeneity or resolution differences. Direct, coarse stitching can lead to breaks, overlaps, or obvious missing areas in the light intensity distribution. Therefore, this unit introduces anisotropic kernel functions to interpolate the light field data, ensuring that key high-frequency details in the scene are preserved as much as possible when processing missing information and smoothing transitions. Compared to traditional isotropic smoothing kernel functions, anisotropic kernel functions can identify the gradient characteristics of light field changes in different directions, preventing blurring or fuzziness at object edges or sharp surfaces, thus maintaining accurate shape and texture recognition in augmented reality environments. After interpolation, this unit uses a layered light field propagation model to further analyze the refraction, scattering, and attenuation mechanisms of light in different media, providing a physical basis for subsequent light field gradient construction. This layered propagation model is not a simple empirical brightness compensation, but rather incorporates potential elements such as air, glass, water vapor, or other transparent and semi-transparent media into the same analytical framework of light field propagation paths. Through iterative calculations or prediction models, it accurately describes the refraction shift of light at the boundaries of the media and the attenuation changes within semi-transparent materials, allowing the interpolated light field to be corrected in a way that more conforms to physical laws. Its significance lies in the fact that augmented reality systems need to accurately overlay digital information onto the real environment. If the light propagation process lacks rigorous characterization, it can cause positioning deviations of virtual objects in transparent or reflective scenes, making the AR effect abrupt or distorted.

[0027] After completing the layered propagation calculation, the data analysis and computation unit models and corrects the light field gradient. Specifically, it first calculates the gradient of spatial and angular components on the interpolated light field, forming a high-dimensional gradient tensor, and incorporates the changes in light intensity in all directions into the components of this tensor. Since direct gradient calculation is often affected by noise and artifacts in complex media or regions with rapid brightness changes, this unit uses the light refraction and scattering distribution derived from the layered propagation model to accurately correct the gradient tensor. This eliminates unreasonable gradient responses in minute local changes, thus more reliably reflecting brightness transitions and object surface orientations in real-world scenes. It is noteworthy that this correction process is not simply filtering and denoising; it iteratively simulates the actual propagation behavior of light at the interface of the medium to determine which gradient information represents real physical boundary changes and which may simply be errors caused by inconsistent imaging or exposure. Only when the light field gradient tensor is sufficiently corrected can the system effectively identify the actual surface contours, high-curvature areas, and even minute concave and convex structures of objects, thus laying an accurate geometric and lighting foundation for augmented reality modeling and rendering. At the same time, after completing gradient correction, the unit will also pass the interpolation and propagation results to the real-time rendering module as needed, so as to achieve dynamic updates in complex interactive situations.

[0028] The region partitioning and rendering unit is used to calculate the discontinuities of the light field gradient based on the corrected light field gradient tensor. Through the singular value decomposition of the light field gradient tensor, it identifies the contours and surfaces of the target object, constructs an adaptive octree structure, partitions high curvature regions, and performs real-time rendering based on the partitioning results.

[0029] After correction using a layered light field propagation model and anisotropic interpolation, the light field gradient tensor can accurately pinpoint abrupt changes in light intensity and direction. These changes typically correspond to object surface contours, boundaries, or refractive and reflective regions. To better distinguish which changes are true edges, the system uses singular value decomposition (SVD) on the light field gradient tensor to extract the rates of change in the principal and secondary directions. It identifies locations with significantly larger singular values ​​and marks them as points potentially containing clear boundaries or high curvature features. Since many objects in augmented reality scenes may have multi-layered surfaces or complex textures, especially under transparent or translucent materials, the regions corresponding to light field gradient discontinuities are not simply two-dimensional curves but often exhibit layered or overlapping surface features. Therefore, SVD can distinguish significant edges from ordinary transition regions from multiple dimensions. Next, the system constructs an adaptive octree structure based on the identified distribution of high curvature points. "Adaptive" means it doesn't uniformly divide the entire space, but rather uses a finer division depth for areas with high curvature or rich edge details, thus capturing every subtle bump and inflection point on the object's surface. In relatively smooth or low-curvature areas, the division depth is reduced to conserve storage and computational resources. Since real-world environments often contain both large, flat surfaces (such as floors or walls) and objects with complex shapes (such as leaves, utensils, sculptures, etc.), the adaptive octree allows the system to apply differentiated sampling and data storage strategies to different areas. This ensures that the geometric and lighting details of complex parts are not overlooked, while also maintaining high system efficiency, preventing it from being slowed down by redundant data when presenting large scenes. After region partitioning, the rendering unit combines these spatial blocks, organized by an octree structure, with corrected light field information. It then uses the physical properties of the medium, such as refraction and scattering, provided by the previously layered propagation model for rendering calculations. This not only maintains a high degree of consistency with the real environment in the surface reflection and shadow representation of conventional opaque objects, but also accurately reflects refraction paths and attenuation changes in transparent or translucent materials, thus maximizing the reproduction of the visual characteristics of various complex materials in the scene. In interactive augmented reality scenarios, users often need to observe the mutual occlusion, light changes, and even dynamic compositing between virtual objects and the real environment in real time. When the light field gradient indicates that the object's surface may be slightly moving or deformed, the system can also re-partition the local area according to the adaptive octree update strategy and quickly synchronize the changed information to the rendering pipeline. In this way, whether relying on a head-mounted device or a handheld terminal, users can experience lighting and reflection effects similar to those of real objects during movement or operation, significantly enhancing the immersion and credibility of augmented reality applications.It's worth noting that this unit isn't just for static scenes. If lighting conditions or objects in the scene change, the system recalculates the light field gradient to locate newly appearing or disappearing edges and high-curvature areas. It then re-triggers a partial update of the adaptive octree, automatically incorporating these changes into the final output image during rendering. This ensures accurate simulation of realistic optical phenomena even in dynamic natural environments. Furthermore, when the system needs to zoom in on objects locally or render only specific areas from certain perspectives, the hierarchical indexing structure provided by the adaptive octree allows the renderer to acquire key local information at higher resolution, while less important areas remain at lower subdivision levels, thus reducing the computational burden on the graphics processing unit.

[0030] Example 2: The execution process of the high-dimensional light field data acquisition unit specifically includes: uniformly arranging 12 camera sampling points on a sphere centered on the target object, with the sphere radius adaptively adjusted according to the size of the target object, and the sphere radius being larger than the maximum diameter of the target object; each camera sampling point includes one 4K resolution main camera and four 2K resolution auxiliary cameras arranged along the cross direction; using structured light projection for internal calibration of the camera group to obtain the pose relationship between each camera; performing global calibration of the entire spherical array camera system through a spherical calibration plate to establish a unified spherical coordinate system; the four auxiliary cameras adopt different exposure parameters, namely -2EV, -1EV, +1EV, and +2EV; the maximum aperture value of the main camera lens is 1.4; synchronously triggering all cameras at a sampling frequency of 30fps; performing HDR synthesis on the data acquired by the four auxiliary cameras at each camera sampling point to obtain HDR data, using an adaptive weighting algorithm to reduce the influence of overexposed and underexposed areas, and achieving a linear HDR light field through camera response curve correction; fusing the high-resolution data acquired by the main camera with the HDR data to construct high-dimensional light field data.

[0031] Specifically, in actual execution, the main camera is responsible for acquiring high-resolution images. These images preserve the spatial geometric features of the target object, such as sharp edges, fine surface textures, and local details in complex shapes. Since the main camera typically operates with a single exposure parameter, while its acquired data has spatial resolution advantages, it may lose some light intensity information in scenes with complex lighting conditions due to overexposure or underexposure. To address this issue, the auxiliary camera uses multiple exposure parameters to record the low-light, high-light, and mid-brightness areas of the scene separately, and then uses HDR synthesis to generate light intensity distribution data covering the full dynamic range. In this way, the data acquired by the main and auxiliary cameras each perform their respective functions: one provides high-precision geometric information, and the other fills in the gaps in optical information. To achieve effective fusion of the two, the system first needs to align the resolution of the main camera with the HDR data. This step typically involves coordinate transformation and pixel interpolation. Through structured cursor calibration and global calibration of the spherical array, the pose relationship between the main and auxiliary cameras has been accurately determined, thus allowing geometric transformations to map the coordinate system of the HDR data onto the coordinate system of the main camera. Although the auxiliary camera has a lower resolution than the main camera, HDR data can be resampled at the main camera's resolution using optical mapping and interpolation techniques, ensuring spatial consistency with the main camera's data. At this point, each pixel in the HDR data can be correlated with a high-resolution pixel in the main camera, laying the foundation for fusion. Next, the system uses an adaptive weighting algorithm to combine the high-resolution data from the main camera with the HDR data in terms of light intensity. The core idea of ​​this adaptive weighting algorithm is to assign different weights to the fusion of the main camera and HDR data based on the scene's lighting distribution and the exposure characteristics of pixel regions. In areas with near-normal light intensity, the main camera's data is often more reliable because its high resolution more clearly reflects geometric edges and texture details. In highlight or low-light areas, the dynamic range advantage of HDR data is more significant; therefore, the weight of HDR data is appropriately increased in these areas. This adaptive adjustment of weights ensures that the fusion result maintains the fineness of the spatial structure while encompassing complete light intensity information. Simultaneously, the physical consistency of the data must also be considered during the fusion process. Since HDR data, after being corrected by the camera response curve, has been converted into a linear light field representation, its light intensity values ​​can directly reflect the true brightness distribution of the scene. However, the data from the main camera may still have some non-linear distortion. Therefore, the system performs corresponding light intensity correction on the main camera data to ensure that it is consistent with the HDR data in optical characteristics. In this way, the fused high-dimensional light field data not only achieves the spatial resolution of the main camera but also demonstrates the advantages of HDR data in dynamic range, ultimately achieving a light field expression that is both refined and realistic.

[0032] Each camera sampling point consists of one 4K resolution main camera and four 2K resolution auxiliary cameras. The main camera is located at the core of the sampling point and is primarily responsible for acquiring high-resolution image data to preserve key details and geometric features of the target scene. The four auxiliary cameras are arranged in a cross shape to supplement the light field information of the target scene from different angles. The auxiliary cameras acquire images with different exposure parameters, set to -2EV, -1EV, +1EV, and +2EV respectively. This high dynamic range (HDR) sampling method can simultaneously record the light intensity distribution of the target object in both dark and bright areas, solving the problem of loss of detail in dark areas or overexposure in bright areas under single exposure conditions. The collaborative work of the main and auxiliary cameras fully combines the advantages of spatial resolution and dynamic range, so that the entire light field data has both delicate spatial structure and accurately reflects the real scene under complex lighting conditions. Before data acquisition, the system needs to complete the calibration of the camera group and the unified calibration of the global spherical array. By using structured light projection to calibrate the camera group within each camera sampling point, the pose relationship between the main camera and the auxiliary cameras can be accurately obtained, providing geometric constraints for subsequent data fusion. Global calibration is performed using a spherical calibration plate, which is placed within the sampling range of the spherical array. By recording the projected positions of known points on the calibration plate in each camera, a unified spherical coordinate system is established for the entire spherical array camera system. This ensures that all sampling points are processed and fused within a consistent coordinate system. This calibration process not only improves the accuracy of data processing but also effectively solves the data misalignment problem that may occur in multi-camera systems due to relative position errors.

[0033] Example 3: The high-dimensional light field data is a vector, whose vector elements include: spatial position coordinates, orientation angle, wavelength and exposure parameters; the spatial position coordinates include: X-axis coordinates, Y-axis coordinates and Z-axis coordinates; the orientation angle includes: the angle between the camera position and the Z-axis in the spherical coordinate system, and the angle between the camera and the X-axis in the XY plane.

[0034] Specifically, the spatial position coordinates in the vector consist of three-dimensional coordinates along the X, Y, and Z axes. These three-dimensional coordinates define the specific position of the light ray within the entire scene, forming the spatial basis for constructing the light field distribution. Each position coordinate corresponds not only to the spatial location of the light ray's origin or propagation but also directly relates to the geometric structure of the target object in augmented reality 3D modeling. Through this coordinate information, the system can accurately define the boundary position, surface texture, and volumetric properties of the target object, providing fine spatial resolution for the 3D model. This design makes high-dimensional light field data applicable not only to simple planar scenes but also to irregular objects in complex 3D space, ensuring that the augmented reality system can adapt to various application scenarios. Secondly, the introduction of the orientation angle provides a detailed description of the light ray propagation direction. In spherical coordinates, the orientation angle is defined as two parts: the angle between the camera position and the Z-axis (usually called the elevation angle) and the angle between the camera in the XY plane and the X-axis (usually called the azimuth angle). These two orientation angles together define the propagation direction of the light ray, and combined with the spatial position, they can accurately describe the geometric path of the light ray. The design of the orientation angle is particularly important because the core essence of the light field lies in not only recording the position of the light ray but also recording its propagation direction in space. By using the orientation angle, the system can capture the differences in the viewing angle of light, providing data support for multi-view 3D modeling and light field interpolation. In augmented reality applications, this orientation information can help the system simulate complex light interactions, such as reflection, refraction, or occlusion, thereby achieving a more natural integration between virtual objects and the real environment.

[0035] Example 4: The data analysis and calculation unit, based on high-dimensional light field data, uses anisotropic kernel functions for light field interpolation. Specifically, this includes: spatially resampling the high-dimensional light field data to ensure uniform data distribution in the spherical sampling space; normalizing the high-dimensional light field data at each camera sampling point to eliminate systematic errors between different camera sampling points; this process is achieved through iterative least squares until the high-dimensional light field data at all camera sampling points meet a preset consistency threshold, resulting in interpolated high-dimensional light field data; based on the surface of the target object... The shape and size of the anisotropic kernel function are adaptively designed based on local geometric features. In high-frequency detail regions corresponding to the edges of the target object, a first anisotropic kernel function is used to support the domain to preserve detail features. In smooth regions, a second anisotropic kernel function is used to support the domain to enhance data continuity. The shape and size of the first and second anisotropic kernel functions are controlled by the covariance matrix, which is determined by the principal direction of the local light field gradient tensor. At the same time, anisotropic Gaussian weights are introduced to ensure a smooth transition of the interpolation results in both spatial and angular dimensions.

[0036] Specifically, in Example 4: the data analysis and calculation unit, based on high-dimensional light field data, performs light field interpolation using anisotropic kernel functions. This process includes: spatially resampling the high-dimensional light field data to ensure uniform data distribution in the spherical sampling space; normalizing the high-dimensional light field data at each camera sampling point to eliminate systematic errors between different camera sampling points; this process is achieved through iterative least squares until the high-dimensional light field data at all camera sampling points meet a preset consistency threshold, resulting in interpolated high-dimensional light field data; based on the target object surface... The system adaptively designs the shape and size of the anisotropic kernel function based on the local geometric features of the surface. In high-frequency detail regions corresponding to the edges of the target object, a first anisotropic kernel function is used to support the domain to preserve detail features; in smooth regions, a second anisotropic kernel function is used to support the domain to enhance data continuity. The shape and size of the first and second anisotropic kernel functions are controlled by the covariance matrix, which is determined by the principal direction of the local light field gradient tensor. Simultaneously, anisotropic Gaussian weights are introduced to ensure a smooth transition of the interpolation results in both spatial and angular dimensions. After completing spatial resampling and normalization, the system begins light field interpolation based on the anisotropic kernel function. The core of this stage lies in designing a kernel function that adapts to the local geometric features of the target object to achieve differentiated data interpolation in different regions. The surface geometric features of the target object are one of the most important pieces of information in the light field data, and these features are typically manifested as changes in surface curvature and high-frequency details at the edges. During the interpolation process, the system adaptively adjusts the shape and size of the kernel function based on these geometric features to ensure that the interpolation result faithfully preserves the structural information of the target object.

[0037] In the edge regions of the target object, due to the large changes in the light field gradient, the distribution of light direction and intensity exhibits obvious high-frequency characteristics. To preserve these details, the system employs a first type of anisotropic kernel function. Its support domain has strong directionality, enabling interpolation along directions with smaller gradient changes, while limiting the interpolation range along directions with larger gradient changes, thus preventing the smoothing of details. The shape and size of this kernel function are controlled by a covariance matrix, the calculation of which depends on the principal direction of the local light field gradient tensor. The principal direction of the gradient tensor reflects the main trend of light field data variation in space. Based on this information, the covariance matrix dynamically adjusts the support domain of the kernel function, allowing the interpolation process to adapt to the geometric characteristics of the target object's edge. In the smooth regions of the target object, the gradient changes in the light field data are smaller, and the distribution of light direction and intensity is more uniform. Here, the system employs a second type of anisotropic kernel function. Its support domain is closer to isotropic, enabling data interpolation over a wider range, thereby enhancing the data continuity in smooth regions. This kernel function design effectively reduces potential interpolation errors in smooth regions while improving the overall consistency of the light field distribution. In smooth regions, the eigenvalues ​​of the covariance matrix are relatively balanced, and the kernel function's shape is closer to an ellipse or circle, thus providing more uniform interpolation results in both spatial and angular dimensions. Furthermore, to further improve the smoothness and accuracy of interpolation, the system introduces anisotropic Gaussian weights during the interpolation process. Gaussian weights dynamically adjust the contribution value of interpolation points based on the spatial location and angular dimension of the light field data, ensuring a smooth transition of the interpolation results in both space and angle. Specifically, Gaussian weights assign higher weights to sampling points closer to the current interpolation point, while adjusting the weight distribution according to the angular characteristics of the light field data, ensuring physical consistency of the interpolation results across different dimensions. By introducing Gaussian weights, the interpolation process not only better adapts to local light field characteristics but also effectively suppresses the impact of noise on the interpolation results. After interpolation, the system generates processed high-dimensional light field data, which possesses a uniform spatial distribution and preserves the geometric details and optical properties of the target object, laying a solid foundation for subsequent light propagation modeling and light field rendering. In augmented reality 3D modeling systems, this light field interpolation method based on anisotropic kernel functions not only improves the accuracy of modeling and rendering, but also provides higher quality light field data support for the natural integration of virtual objects with real scenes.

[0038] Example 5: The data analysis and calculation unit constructs a layered light field propagation model to describe the propagation characteristics of light in different media. Specifically, this process includes: constructing a layered light field propagation model; firstly, at the coarsest level, ray tracing technology is used to simulate the propagation characteristics of light in a homogeneous medium; then, the model is refined layer by layer, considering the non-homogeneity and anisotropy of the medium in each layer; combining the interpolated high-dimensional light field data, the propagation characteristics of light in different media are obtained by solving the light field transmission model. This is a three-dimensional vector whose elements include: reflection characteristic parameters, refraction characteristic parameters, and scattering characteristic parameters. This process requires iterative calculation until the difference in the light field between adjacent levels is less than a preset threshold; the reflection characteristic parameter is reflectivity; the refraction characteristic parameter is refractive index; and the scattering characteristic parameter is scattering rate.

[0039] Specifically, this process begins at the coarsest level. The system first assumes the medium is homogeneous and uses ray tracing to simulate the basic propagation behavior of light at this level. In a homogeneous medium, the propagation path and characteristics of light are relatively simple; its direction and intensity are mainly determined by the global parameters of the medium (such as uniform refractive index and scattering rate). At this point, the system calculates the reflection and transmission paths of light by tracing the interaction between the light and the medium interface, and makes preliminary estimates of optical parameters such as reflectivity, refractive index, and scattering rate. The calculation of these parameters depends on high-dimensional light field data obtained after light field interpolation, such as the intensity, direction, and wavelength distribution of light, which provides the initial input for the simulation. At this level, due to the assumption of a homogeneous medium, the computational complexity of the system is low, but the obtained light field propagation characteristics can only approximately reflect the actual situation in the scene.

[0040] As the layers are refined, the system begins to consider the non-uniformity and anisotropy of the medium. In real-world scenarios, most media are not perfectly homogeneous; their internal physical properties (such as refractive index and scattering rate) may vary with spatial location, and anisotropy means that the propagation behavior of light in different directions may differ significantly. To capture these characteristics, the system introduces more local optical parameters at each refinement level and locally adjusts the light field propagation model. For example, the system may define different refractive index distributions in different spatial regions or adjust the intensity and angular distribution of light scattering in specific directions. In this process, high-dimensional light field data is again used as input, and the local characteristics of the light field propagation model are corrected by combining spatial distribution and directional information. At this point, the computation of ray tracing technology becomes more complex because the propagation path of light within the medium may be nonlinearly distorted by changes in local characteristics. At each level, the system solves the light field transmission model to calculate the propagation characteristics of light at that level and represents this characteristic as a three-dimensional vector. The three elements of this three-dimensional vector are the reflection characteristic parameter, the refraction characteristic parameter, and the scattering characteristic parameter. Reflectivity parameters quantify the degree to which light is reflected on the surface of a medium in the form of reflectivity, which is closely related to the smoothness and refractive index of the medium surface material; refractive parameters describe the degree of change in the direction of light after it enters the medium in the form of refractive index, which is a key control variable for the propagation path of light; while scattering parameters reflect the diffusion behavior of light inside the medium in the form of scattering rate, which is especially important in opaque or translucent media (such as fog or frosted glass).

[0041] Because the propagation characteristics of light in different media can exhibit complex hierarchical and local variations, the system employs an iterative calculation method to progressively optimize the light field propagation model at each level. In each iteration, the system evaluates the accuracy of its propagation model by comparing the light field characteristics of the current level with those of the previous level. If the difference in the light field between adjacent levels exceeds a preset threshold, the system further refines the propagation model and updates local optical parameters to more accurately characterize the light propagation behavior within the medium. This iterative calculation process continues until the difference in the light field between all levels is less than the threshold, at which point the system considers the propagation model to have reached sufficient accuracy. The advantage of this hierarchical modeling method is that the system can progressively improve the accuracy of the light field propagation model by gradually refining it based on an initial coarse estimate. This hierarchical processing method not only avoids the high cost of complex calculations in the initial stage but also gradually introduces more local characteristics and nonlinear behaviors, enabling the final light field propagation model to realistically reflect optical phenomena in complex media environments. After the hierarchical light field propagation model is constructed, the system generates a globally consistent and locally detailed light field description, whose three-dimensional vector-based propagation characteristics provide a precise optical foundation for modeling and rendering in augmented reality systems. In augmented reality applications, the lighting interaction between virtual objects and real-world scenes is key to enhancing immersion. Layered light field propagation models can provide the lighting calculations for virtual objects with the light propagation characteristics of a real physical environment, allowing virtual objects to blend naturally into the real scene. For example, when a virtual object is superimposed on a translucent glass table, the system can accurately simulate the refraction path of light within the glass, as well as the intensity and direction of light transmitted to the virtual object, making the lighting effects of the virtual object appear more realistic and believable.

[0042] Example 6: The data analysis and calculation unit establishes an optical field gradient tensor and uses propagation characteristics to correct the optical field gradient tensor. The process of characterizing spatial light intensity changes specifically includes: constructing a fourth-order optical field gradient tensor by calculating the partial derivatives of the optical field propagation model in the spatial and angular dimensions; multiplying the fourth-order optical field gradient tensor by the three-dimensional vector corresponding to the propagation characteristics to obtain the propagation characteristic-corrected optical field gradient tensor, which is used to characterize the local variation characteristics of the optical field. The eigenvalues ​​and eigenvectors of this fourth-order optical field gradient tensor reveal the main direction and intensity of the optical field change.

[0043] Specifically, the first step in constructing the light field gradient tensor is to calculate the partial derivatives of the light field in the spatial and angular dimensions based on the light field propagation model. Light field data is essentially a high-dimensional function that describes the distribution characteristics of light rays in spatial position and direction; therefore, its gradient tensor needs to simultaneously contain information about changes in spatial dimensions. By performing partial derivative calculations on the light field propagation model, the system can obtain the rate of change of the light field in each dimension. These rates of change directly reflect the distribution gradient of light intensity at different spatial positions and angular directions. Based on this, the system constructs a fourth-order light field gradient tensor, where each tensor component corresponds to the light intensity variation relationship in different dimensions. The structure of the fourth-order tensor provides high-dimensional feature information for the fine analysis of light field data, enabling the system to comprehensively characterize the local properties of the light field.

[0044] Next, to make the light field gradient tensor more closely reflect the optical characteristics of real-world scenarios, the system corrects the gradient tensor using three-dimensional vectors (i.e., reflection, refraction, and scattering parameters) from the ray propagation model. This step is crucial. While the original gradient tensor accurately describes the geometric changes of the light field, it fails to directly consider the physical behavior of light in different media, such as directional changes caused by reflection, path shifts caused by refraction, and intensity diffusion caused by scattering. By multiplying the fourth-order gradient tensor by the three-dimensional vector of ray propagation characteristics, the system incorporates these physical behaviors into the gradient tensor calculation, enabling it to reflect not only the geometric but also the optical properties of the light field. The corrected light field gradient tensor exhibits stronger physical consistency and environmental adaptability, with its eigenvalues ​​and eigenvectors revealing the main directions and intensities of change in the light field within local regions. Eigenvalues ​​represent the magnitude of the light field gradient change in various directions, while eigenvectors indicate the main directions of change. For example, in the edge regions of a target object, due to drastic changes in light intensity, the principal eigenvalues ​​of the gradient tensor will increase significantly, and the principal eigenvectors will point towards the edge. This provides a reliable basis for the system to identify the object's contour and surface. In smooth regions, the eigenvalues ​​of the gradient tensor tend to be uniform, indicating that the changes in the light field are relatively gentle in these regions. This provides a basis for subsequent data processing and rendering optimization.

[0045] The result of this process is a fully corrected light field gradient tensor, which not only accurately describes the local variations of the light field but also reveals the optical behavior of different regions within the scene. In augmented reality 3D modeling systems, this tensor is widely used for edge detection, high-curvature region recognition, and the simulation of complex optical phenomena. For example, when identifying object boundaries, the system uses the principal eigenvalues ​​and principal eigenvectors of the gradient tensor to determine which regions exhibit significant changes, thus extracting object contours more accurately. During the rendering phase, the light field gradient tensor also helps the system calculate the scattering and attenuation behavior of light rays, making virtual objects appear more realistic under different lighting conditions. Furthermore, this process is designed with significant iterative optimization capabilities. When calculating the light field gradient tensor, the system can further optimize the correction results by adjusting the parameters of the ray propagation model. For example, when abnormal gradient tensor behavior is detected in certain regions, the system can re-evaluate the reflection, refraction, or scattering characteristics of these regions and iteratively update the gradient tensor calculation results until the correction results meet the required accuracy standards. This optimization capability allows the light field gradient tensor to adapt to more complex scenes and varying optical conditions, providing a reliable guarantee for the application of augmented reality technology in dynamic environments.

[0046] Example 7: The region segmentation and rendering unit calculates light field gradient discontinuities based on the light field gradient tensor. The process of identifying the contours and surfaces of target objects through singular value decomposition of the light field gradient tensor specifically includes: normalizing each component of the fourth-order light field gradient tensor, then setting a global threshold. If the gradient amplitude of a camera sampling point exceeds the global threshold, it is determined that there is a discontinuity in the contour or material boundary, and the camera sampling point is considered a discontinuity point. Near the detected discontinuity point, singular value decomposition is performed on the light field gradient tensor. The decomposed eigenvalues ​​and eigenvectors reveal the main direction and intensity of gradient changes. When the largest singular value is greater than other singular values, and the sum of the differences with other singular values ​​exceeds the set first judgment threshold, it indicates that there is a directional change at the camera sampling point, corresponding to features on the surface or contour. If multiple singular values ​​are similar, it indicates that the area around the camera sampling point is smooth or has isotropic characteristics.

[0047] Specifically, the system normalizes each component of the fourth-order light field gradient tensor to eliminate deviations that may arise between sampling points from different cameras due to differences in sampling conditions or data distribution. This normalization not only ensures the consistency of the light field gradient tensor in spatial and angular dimensions but also ensures that a unified threshold can be used for judgment in subsequent processing. The normalized light field gradient tensor reflects the relative amplitude and directional characteristics of light intensity changes at each sampling point, providing a standardized analytical basis for contour and surface recognition. After normalization, the system sets a global threshold to initially screen for discontinuities that may exist at contour or material boundaries. For each camera sampling point, the magnitude of its gradient tensor is calculated and compared with the global threshold. If the gradient magnitude of a sampling point exceeds the threshold, it indicates that the light intensity change at that point is relatively drastic, possibly corresponding to the boundary, discontinuity surface, or material transition region of the target object. In this case, the sampling point is identified as a discontinuity, and the system marks it as a key analysis object. In this way, the system can quickly extract key areas requiring further processing from a large amount of light field data, thereby significantly improving computational efficiency.

[0048] For detected discontinuities, the system utilizes singular value decomposition (SVD) to perform in-depth analysis of the light field gradient tensor. SVD is a powerful linear algebra tool that decomposes a high-order tensor into a set of eigenvalues ​​and eigenvectors, revealing the intensity and principal directions of the light field gradient's variation in different directions. Specifically, eigenvalues ​​represent the magnitude of the light field gradient's variation in each principal direction, while eigenvectors indicate the direction in which these variations occur. By analyzing the results of SVD, the system can further identify the specific feature types at discontinuities. Based on SVD, the system introduces two key criteria to distinguish the specific attributes of discontinuities. First, when the maximum singular value is significantly larger than other singular values, and the sum of the differences between the maximum singular value and other singular values ​​exceeds a set first threshold, it indicates that the point has a significant directional change. This situation typically corresponds to the edges of a target object, sharp turning points, or material transition regions with highly anisotropic variations. For example, on a cube with sharp edges, the light field gradient in the edge region will show significant directional changes, and the system can accurately capture these features through SVD results. Second, if multiple singular values ​​are similar in magnitude, it indicates that the light field gradient changes relatively uniformly around the sampling point, lacking obvious directional characteristics. This situation typically corresponds to smooth surfaces or isotropic material regions of the target object. In such regions, the distribution of the light field gradient is basically consistent in all directions, allowing the system to determine that the point belongs to a low-curvature or smooth region. In this way, the system can effectively distinguish between boundaries and smooth regions, providing accurate data support for subsequent region segmentation. Combining the above processes, the region segmentation and rendering unit can accurately classify the boundaries, material variations, and smooth regions of the target object based on the singular value decomposition results of the light field gradient tensor. This classification result not only provides a highly reliable geometric basis for 3D modeling in augmented reality systems but also directly affects the processing method of the rendering unit. In boundary regions, the rendering unit can preserve details through higher resolution processing, while in smooth regions, a simpler processing method can be used to improve rendering efficiency. Furthermore, this classification can also provide important basis for simulating complex optical phenomena, such as the accurate rendering of light refraction at the boundaries of transparent media or the high-gloss areas on a surface.

[0049] Example 8: Region partitioning and rendering unit. The specific process of constructing an adaptive octree structure to partition high curvature regions includes: after obtaining the overall bounding box of the target object, an octagon is built at the coarsest level to enclose the entire target object; within each octagon, a set number of camera sampling points are maintained; for each octagon, the variance or mean curvature of its internal light field gradient tensor is calculated; if the variance or mean curvature exceeds a set second judgment threshold, it indicates that there is a surface change in this region, and it needs to be further subdivided into smaller octagons; the smaller subdivided octagons continue to undergo the same judgment until the subdivision stopping condition is met. When the octagon size reaches the resolution limit or the light field gradient tensor is in a smooth state, the subdivision stops; this adaptive octagon structure can not only effectively partition the region of the target object in space, but also reduce the oversampling and calculation of smooth regions and increase the sampling and calculation of high curvature regions.

[0050] Specifically, the construction of the adaptive octree structure begins with the overall bounding box of the target object. The bounding box is the smallest spatial area that can completely cover the target object, typically represented by a regular cuboid, whose size and position are determined by the object's boundaries. At the coarsest level, the system builds an initial octagon within the bounding box, dividing the entire target object into spatial units. The octagon is the basic unit of 3D spatial partitioning, further dividing a cubic region into eight smaller cubes of equal volume, each corresponding to an octagonal region in space. This initial partitioning provides a global framework for subsequent subdivision operations. Within each octagon, the system maintains a set number of camera sampling points. These sampling points record high-dimensional light field data of the target object in the current region, including spatial location, orientation angle, wavelength, and exposure parameters. This data not only describes the spatial distribution of light but also includes the optical properties of the object's surface. During region partitioning, the data from the camera sampling points is used to calculate the variance or mean curvature of the light field gradient tensor, a core indicator for assessing whether the current region needs further subdivision. The variance or mean curvature of the light field gradient tensor reflects the drastic change in the light field within that region. If the variance or mean curvature exceeds the set second threshold, it indicates that there is significant non-uniformity or local variation in the light field distribution within that region. This typically corresponds to high curvature regions of the target object, such as edges, sharp inflection points, or complex surface features. In this case, the system further divides the current octagon into smaller octagons and re-evaluates the distribution of the light field gradient within the new octagons.

[0051] By employing this layer-by-layer refinement approach, the system can progressively increase the resolution of high-curvature regions, ensuring that the geometric and optical details of these critical areas are fully captured. The subdivided octagons continue to be iteratively processed according to the same criteria until the subdivision stopping condition is met. The system stops further subdividing the octagon when its size reaches the resolution cap, or when the variance or mean curvature of the light field gradient tensor is in a smooth state. The resolution cap is a parameter pre-set based on the system's storage capacity and computational resources to avoid over-subdivision in high-curvature regions. A smooth light field gradient typically refers to a situation where the variance or mean curvature is below a certain threshold, indicating that the light field distribution in the current region is relatively uniform, and further subdivision will not significantly improve modeling accuracy. By setting these stopping conditions, the system can effectively control the depth of spatial subdivision and balance computational costs. The advantage of the adaptive octagon structure is that it can dynamically adjust the resolution of spatial subdivision according to the actual geometric characteristics of the target object. In smooth regions, due to the small changes in light field distribution, the system stops subdivision and maintains larger octagonal units, thereby reducing unnecessary computation and data storage. In high-curvature regions, the system increases sampling density through multi-level subdivision, ensuring that the light field information of complex areas can be fully recorded and processed. This adaptive subdivision strategy not only improves adaptability to complex scenes but also significantly enhances the overall modeling and rendering efficiency. In augmented reality 3D modeling systems, the construction of adaptive octree structures provides crucial support for real-time rendering. During the rendering phase, the system can prioritize high-curvature regions based on the octree's hierarchical information, ensuring that the details in these regions are accurately represented in the final rendering. For example, when rendering the surface texture of a complex object, high-curvature regions may include sharp edges, depressions, or decorative patterns, details that are crucial to visual effects. Through adaptive octree subdivision, the system can allocate more computational resources to these regions while simplifying smooth regions, thus achieving the optimal balance between visual effects and computational efficiency. Furthermore, the adaptive octree structure provides flexibility for augmented reality applications in dynamic scenes. When the position, shape, or lighting conditions of a target object change, the system can respond quickly by locally updating the octree structure. This update only requires recalculating the changed octree, without regenerating the entire octree, thus significantly reducing computational overhead in dynamic environments.

[0052] Example 9: Local extremum analysis is performed on the light field gradient tensor contained in each smallest octagon to find the local maximum or minimum light intensity points in the spatial and angular dimensions, as well as the points where the direction of the light field gradient tensor changes. Due to the differences in the reflection or refraction properties of different materials, their light field gradient tensors will exhibit different extremum distributions at specific angles. By analyzing the extremum distribution, the criteria for determining the material boundary are obtained. For the same octagon, if two or more distinct extremum distributions of the light field gradient tensor are observed, it is determined that there is a material transition zone within it. Further, combined with the propagation characteristics of light in different media, it is identified which extrema belong to the same material region. When the covariance of the extremum distributions of the light field gradient tensors corresponding to the camera sampling points in multiple octagons is within the set third judgment threshold, they are merged into the same material region. Conversely, if the differences in the extremum distributions are large, they are determined to be adjacent different material regions, forming a gridded description of the surface material partitions of the target object.

[0053] Specifically, for the light field gradient tensor contained within each smallest octagon, the system performs local extremum analysis. The core of this step lies in the fact that the extremum locations of the light field gradient tensor reflect significant changes in light intensity at specific directions or locations. For example, when light undergoes intense reflection, refraction, or scattering in a certain direction, the component of the light field gradient tensor in that direction will exhibit a maximum or minimum value. By analyzing the distribution of these extremum points in spatial and angular dimensions, the system can identify the locations where local characteristics of an object's surface change most significantly. These locations may correspond to material interfaces, significant edges of textures, or transition regions between media with different optical properties. Due to differences in the reflection, refraction, or scattering properties of different materials, the light field gradient tensor will exhibit different extremum distributions at specific angles. For example, the light field gradient tensor of a specular reflective material will exhibit a significant maximum value in the reflection direction, while the light field gradient tensor of a diffuse reflective material may be more uniformly distributed in multiple directions without obvious maximum values. By analyzing these extremum distribution patterns, the system can determine the material properties of local regions from an optical perspective. This analysis not only helps identify the geometric features of an object's surface, but also reflects the uniqueness of the material at the optical level, providing a precise physical basis for the lighting interaction between virtual objects and real scenes in augmented reality systems.

[0054] When two or more distinctly different extremum distributions are observed within the same minimal octet, the system identifies the region as a material transition zone. This typically occurs near the interface between two or more materials. For example, when a smooth glass surface meets a rough metal boundary, its optical field gradient tensor exhibits drastically different extremum distribution characteristics in the transition region: the extrema of the optical field gradient in the glass region are concentrated in the reflection and refraction directions, while the optical field gradient in the metal region may exhibit more diffuse reflection characteristics. By accurately analyzing these differences in local extremum distributions, the system can capture the material transition region and provide a basis for subsequent material boundary delineation. Furthermore, the system combines the propagation characteristics of light in different media to classify the extremum distributions to identify which extremum points belong to the same material region. This process relies on parameters such as reflectivity, refractive index, and scattering rate in the aforementioned optical field propagation model. By correlating these optical properties with the extremum distribution patterns, the system can more accurately group extremum points within the same material region together. For example, the extreme values ​​of the light field gradient tensor of a glass surface may occur in multiple directions, but these extreme points can be uniformly attributed to the glass material after correction for propagation characteristics. This method greatly reduces misjudgments in material region segmentation and provides a more physically consistent basis for processing multi-material objects in complex scenes. Finally, the system further determines the material region segmentation by comparing the covariance values ​​of the extreme value distributions of the light field gradient tensor corresponding to camera sampling points within multiple octagons. When the covariance values ​​of the extreme value distributions of the light field gradient tensor within multiple octagons are within a set third judgment threshold, the system merges these octagons into the same material region. This indicates that the material properties within these octagons are relatively consistent in optical performance and belong to the same material. Conversely, if the covariance value exceeds the threshold, it indicates that the light field gradient tensors of these octagons exhibit significant differences, and the system will segment them into adjacent different material regions. This covariance-based analysis method can effectively capture the optical differences between material regions, thereby forming a meshed description of the surface material partitioning of the target object. Through the above process, the system finally achieves accurate partitioning of the surface material of the target object and represents these partitions in a meshed form. This partitioning result has significant application value in augmented reality 3D modeling systems. For example, during the rendering stage, meshed material partitioning can guide the renderer to adopt differentiated rendering strategies for different material areas, thereby improving the realism and detail of the visual representation. Furthermore, this material partitioning can provide a precise physical basis for the interaction between virtual objects and real-world scenes. For instance, at the boundary between a virtual object and transparent or specular materials, the system can generate more realistic optical effects based on the partitioning result.

[0055] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An augmented reality 3D modeling system based on light field distribution, characterized in that, The system includes: The high-dimensional light field data acquisition unit is used to deploy a spherical array camera system. Each camera sampling point includes a main camera and four auxiliary cameras. The main camera acquires high-resolution spatial light field information, and the auxiliary cameras acquire HDR light field information with different exposure parameters to form high-dimensional light field data. The data analysis and computation unit is used to perform optical field interpolation using anisotropic kernel functions based on high-dimensional optical field data to obtain interpolated high-dimensional optical field data; to construct a hierarchical optical field propagation model and solve the propagation characteristics of light in different media based on the interpolated high-dimensional optical field data; to establish an optical field gradient tensor and use propagation characteristics to correct the optical field gradient tensor to characterize changes in spatial light intensity. The region partitioning and rendering unit is used to calculate the light field gradient discontinuity points based on the corrected light field gradient tensor, identify the contours and surfaces of target objects through the singular value decomposition of the light field gradient tensor, construct an adaptive octree structure, partition high curvature regions, and perform real-time rendering based on the partitioning results. The data analysis and computation unit, based on high-dimensional light field data, performs light field interpolation using anisotropic kernel functions. Specifically, this process includes: spatial resampling of the high-dimensional light field data to ensure uniform data distribution in the spherical sampling space; normalization of the high-dimensional light field data at each camera sampling point to eliminate systematic errors between different camera sampling points (achieved through iterative least squares) until the high-dimensional light field data at all camera sampling points meets a preset consistency threshold, resulting in interpolated high-dimensional light field data; adaptively designing the shape and size of the anisotropic kernel function based on the local geometric features of the target object's surface; using a first anisotropic kernel function as a support domain in high-frequency detail regions corresponding to the target object's edges to preserve detail features; and using a second anisotropic kernel function as a support domain in smooth regions to enhance data continuity; the shape and size of the first and second anisotropic kernel functions are controlled by a covariance matrix, which is determined by the principal direction of the local light field gradient tensor; and simultaneously, introducing anisotropic Gaussian weights to ensure smooth transitions in both spatial and angular dimensions of the interpolation results.

2. The augmented reality 3D modeling system based on light field distribution as described in claim 1, characterized in that, The execution process of the high-dimensional light field data acquisition unit specifically includes: uniformly arranging 12 camera sampling points on a sphere centered on the target object, with the sphere radius adaptively adjusted according to the size of the target object, and the sphere radius being larger than the maximum diameter of the target object; each camera sampling point includes one 4K resolution main camera and four 2K resolution auxiliary cameras arranged along the cross direction; using structured light projection for internal calibration of the camera group to obtain the pose relationship between each camera; performing global calibration of the entire spherical array camera system through a spherical calibration plate to establish a unified spherical coordinate system; the four auxiliary cameras adopt different exposure parameters, namely -2EV, -1EV, +1EV, and +2EV; the maximum aperture value of the main camera lens is 1.4; synchronously triggering all cameras at a sampling frequency of 30fps; performing HDR synthesis on the data acquired by the four auxiliary cameras at each camera sampling point to obtain HDR data, using an adaptive weighting algorithm to reduce the influence of overexposed and underexposed areas, and achieving a linear HDR light field through camera response curve correction; and fusing the high-resolution data acquired by the main camera with the HDR data to construct high-dimensional light field data.

3. The augmented reality 3D modeling system based on light field distribution as described in claim 2, characterized in that, The high-dimensional light field data is a vector whose vector elements include: spatial position coordinates, orientation angle, wavelength, and exposure parameters; the spatial position coordinates include: X-axis coordinates, Y-axis coordinates, and Z-axis coordinates; the orientation angle includes: the angle between the camera position and the Z-axis in the spherical coordinate system, and the angle between the camera and the X-axis in the XY plane.

4. The augmented reality 3D modeling system based on light field distribution as described in claim 1, characterized in that, The data analysis and computation unit constructs a hierarchical light field propagation model to describe the propagation characteristics of light in different media. Specifically, this process includes: constructing the hierarchical light field propagation model; firstly, at the coarsest level, ray tracing technology is used to simulate the propagation characteristics of light in a homogeneous medium; then, the model is refined layer by layer, considering the non-homogeneity and anisotropy of the medium in each layer; combining the interpolated high-dimensional light field data, the propagation characteristics of light in different media are obtained by solving the light field transmission model. This is a three-dimensional vector whose elements include: reflection characteristic parameters, refraction characteristic parameters, and scattering characteristic parameters. This process requires iterative calculation until the difference in the light field between adjacent levels is less than a preset threshold; the reflection characteristic parameter is reflectivity; the refraction characteristic parameter is refractive index; and the scattering characteristic parameter is scattering rate.

5. The augmented reality 3D modeling system based on light field distribution as described in claim 4, characterized in that, The data analysis and computation unit establishes an optical field gradient tensor and uses propagation characteristics to correct the optical field gradient tensor. The process of characterizing spatial light intensity changes specifically includes: constructing a fourth-order optical field gradient tensor by calculating the partial derivatives of the optical field propagation model in the spatial and angular dimensions; multiplying the fourth-order optical field gradient tensor by the three-dimensional vector corresponding to the propagation characteristics to obtain the propagation characteristic-corrected optical field gradient tensor, which is used to characterize the local variation characteristics of the optical field; the eigenvalues ​​and eigenvectors of this fourth-order optical field gradient tensor reveal the main direction and intensity of the optical field changes.

6. The augmented reality 3D modeling system based on light field distribution as described in claim 5, characterized in that, The region segmentation and rendering unit, based on the light field gradient tensor, calculates light field gradient discontinuities. The process of identifying the contours and surfaces of target objects through singular value decomposition (SVD) of the light field gradient tensor specifically includes: normalizing each component of the fourth-order light field gradient tensor; setting a global threshold; if the gradient magnitude of a camera sampling point exceeds the global threshold, it is determined that there is a discontinuity in the contour or material boundary, and the camera sampling point is considered a discontinuity point; near the detected discontinuity point, singular value decomposition is performed on the light field gradient tensor, and the resulting eigenvalues ​​and eigenvectors reveal the main direction and intensity of gradient changes; when the largest singular value is greater than other singular values, and the sum of the differences with other singular values ​​exceeds a set first judgment threshold, it indicates that there is a directional change at the camera sampling point, corresponding to features on the surface or contour; if multiple singular values ​​are similar, it indicates that the area around the camera sampling point is smooth or has isotropic characteristics.

7. The augmented reality 3D modeling system based on light field distribution as described in claim 6, characterized in that, The region partitioning and rendering unit constructs an adaptive octree structure. The specific process for partitioning high-curvature regions includes: after obtaining the overall bounding box of the target object, an octagon is built at the coarsest level to enclose the entire target object; within each octagon, a set number of camera sampling points are maintained; for each octagon, the variance or mean curvature of its internal light field gradient tensor is calculated; if the variance or mean curvature exceeds a set second judgment threshold, it indicates that there is a surface change in this region, and it needs to be further subdivided into smaller octagons; the smaller subdivided octagons continue to undergo the same judgment until the subdivision stopping condition is met. Subdivision stops when the octagon size reaches the resolution limit or the light field gradient tensor is in a smooth state. This adaptive octagon structure can not only effectively partition the region of the target object in space, but also reduce oversampling and calculation in smooth regions and increase sampling and calculation in high-curvature regions.

8. The augmented reality 3D modeling system based on light field distribution as described in claim 7, characterized in that, Local extremum analysis is performed on the light field gradient tensor contained in each smallest octagon to find the local maximum or minimum light intensity points in the spatial and angular dimensions, as well as the points where the direction of the light field gradient tensor changes. Due to the differences in the reflection or refraction properties of different materials, their light field gradient tensors will exhibit different extremum distributions at specific angles. By analyzing the extremum distribution, the criteria for determining material boundaries are obtained. For the same octagon, if two or more distinct extremum distributions of the light field gradient tensor are observed, it is determined that there is a material transition zone within it. Further, combined with the propagation characteristics of light in different media, it is identified which extrema belong to the same material region. When the covariance of the extremum distributions of the light field gradient tensors corresponding to the camera sampling points in multiple octagons is within the set third judgment threshold, they are merged into the same material region. Conversely, if the differences in the extremum distributions are large, they are determined to be adjacent different material regions, forming a meshed description of the material partitioning of the target object surface.

Images