Neural radiation field high-fidelity representation method and system based on geometric spectrum coupling

Abstract

The invention discloses a geometric spectrum coupling-based neural radiation field high-fidelity representation method and system. The method comprises the following steps of: obtaining sparse multi-view image data and calculating internal and external parameters of a corresponding camera; generating light based on internal and external parameters of the camera, and performing point sampling on the space along the direction of the light to obtain sampling points; constructing a differentiable coding layer, predicting the density of sampling points through forward propagation of a neural network, and obtaining a principal curvature, a density gradient and an included angle between a normal vector of the sampling points and an observation direction based on a density prediction result; constructing a spectrum-geometric coupling function, and calculating to obtain a spectrum coupling coding vector; the spectrum coupling coding vector is used as an input to be transmitted into a NeRF main network, and NeRF volume rendering is carried out according to a density and color prediction result; and performing difference optimization training on the rendering result and the real image to obtain a high-fidelity composite image. According to the invention, the expression ability of the neural radiation field model in the high-frequency detail area can be enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and system for high-fidelity representation of neural radiation fields based on geometric spectrum coupling. Background Art

[0002] Neural Radiance Fields (NeRF) builds on a deep foundation in computer graphics and vision science, particularly in the areas of 3D scene representation and view synthesis. This technology is driven by the ongoing exploration of how to reconstruct high-quality, highly detailed 3D scenes from limited 2D images. Traditional 3D modeling methods rely on complex geometries and surface properties, which are time-consuming and difficult to capture the subtle variations and complex lighting effects found in natural scenes.

[0003] With the advancement of deep learning technology, especially its application in image generation tasks and neural networks, new ideas have been provided for solving the above problems. Researchers began to try to use neural networks to directly learn the representation of three-dimensional scenes, and a new scene representation method called NeRF was born. NeRF uses neural networks to encode the color and density information of each point in three-dimensional space, and uses a volume rendering method to render the corresponding image according to different viewing directions, so that realistic three-dimensional scene images from any perspective can be generated from a set of sparse input images. This method effectively combines the physical accuracy of traditional volume rendering and the powerful expressive power of deep learning, significantly improving the quality and efficiency of three-dimensional scene reconstruction. Therefore, it can be said that NeRF is the product of the joint development of computer graphics, volume rendering and modern deep learning technology.

[0004] However, due to the characteristics of neural networks for general function fitting, it is often difficult to accurately capture subtle changes and details when dealing with complex scenes. Specifically, in the application process of NeRF technology, although it can generate high-quality three-dimensional scene images, it is unable to express high-frequency details. For example, elements such as fine textures or sharp edges in the scene may not be reproduced completely and accurately. This limitation is mainly because neural networks tend to learn the overall pattern and structure of the data, while it is more difficult to accurately capture those local and subtle changes, which leads to the problem of insufficient high-frequency details in the generated image. From the principle of NeRF, the traditional encoding method uses fixed-frequency position encoding. The preset frequency band cannot adapt to the characteristics of the scene, and cannot respond to changes in curvature and material characteristics, resulting in blurred high-frequency details and amplified low-frequency noise.

[0005] In view of this, this application is hereby filed. Summary of the Invention

[0006] The technical problem to be solved by the present invention is that it is difficult to accurately capture those local and subtle changes in the existing neural radiation field, which may lead to problems such as insufficient high-frequency details in the generated image. The purpose of the present invention is to provide a high-fidelity representation method and system for neural radiation fields based on geometric spectrum coupling. The present invention proposes a spectrum-principal curvature coupling function, which aims to fuse the principal curvature, density gradient, and normal and viewing angle directional information of each position point in the scene, and dynamically adjust the energy distribution of each frequency band in the position encoding, thereby improving the perception and expression ability of the neural radiation field model for detailed areas, so as to enhance the performance of the neural radiation field model in high-frequency detail areas.

[0007] The present invention is achieved through the following technical solutions: In a first aspect, the present invention provides a high-fidelity representation method for neural radiation fields based on geometric spectrum coupling, the method comprising: Acquire sparse multi-view image data and calculate the corresponding camera internal and external parameters; Generate light based on the camera's internal and external parameters, and perform point sampling in space along the direction of the light to obtain sampling points; The position of the sampling point is input into the pre-built differentiable coding layer. The differentiable coding layer predicts the density of the sampling point through the forward propagation of the neural network, and based on the density prediction results, obtains the principal curvature, density gradient and the angle between the normal vector of the sampling point and the observation direction; According to the principal curvature, density gradient and the angle between the normal vector of the sampling point and the observation direction, a spectrum-geometry coupling function is constructed, and the spectrum coupling coding vector is calculated; The spectral coupling coding vector is passed as input into the NeRF main network, and the density and color prediction results are output; NeRF volume rendering is performed based on the density and color prediction results; the difference between the rendering result and the real image is optimized and trained to obtain a high-fidelity synthetic image.

[0008] Furthermore, sparse multi-view image data is obtained and the corresponding camera intrinsic and extrinsic parameters are calculated, including: Acquire a set of sparse multi-view two-dimensional image data, where the two-dimensional image data includes images taken by a real camera or images synthesized based on three-dimensional software; The structure-from-motion algorithm is used to extract the camera intrinsic and extrinsic parameters of each image in the two-dimensional image data.

[0009] Furthermore, the expression of the differentiable coding layer is: , that is, constructing the frequency parameters into a learnable weight matrix, , , where is the three-dimensional position coordinate of the sampling point, is a learnable frequency vector, m is the frequency cardinality, and m determines the encoding dimension.

[0010] Furthermore, based on the density prediction results, the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction are obtained, including: Based on the density prediction results, the Hessian matrix is ​​fitted to the local neighborhood of the sampling point to calculate the principal curvature; Based on the density prediction results, calculate the density gradient; Normal vector based on sampling point and viewing direction , get the angle between the normal vector of the sampling point and the observation direction.

[0011] Furthermore, based on the density prediction results, the Hessian matrix is ​​fitted to the local neighborhood of the sampling point to calculate the principal curvature, including: Based on the K nearest neighbor strategy, at the sampling point Set a fixed radius around , select all that satisfy point; Use the location and density prediction results of neighborhood points to construct a local quadratic function approximate density distribution Where, is a constant term, is the linear coefficient in the direction of the first-order derivative, which represents the density change trend in the x, y, and z directions respectively; is the square coefficient of the second-order independent variable, which represents the degree of curvature in the x, y, and z directions respectively; Indicates the coupling change in the x and y directions, Indicates the coupling change in the x and z directions, Indicates the coupling change in the y and z directions; The local quadratic function is fitted by the least squares method to approximate the density distribution and construct the matrix , , and extract its Hessian matrix H from the fitted coefficients: Where, is a constant term, is the linear coefficient in the direction of the first-order derivative, which represents the density change trend in the x, y, and z directions respectively; is the square coefficient of the second-order independent variable, which represents the degree of curvature in the x, y, and z directions respectively; Indicates the coupling change in the x and y directions, Indicates the coupling change in the x and z directions, Indicates the coupling change in the y and z directions; Find the eigenvalue of the Hessian matrix H and take the direction corresponding to the maximum eigenvalue; and estimate the principal curvature based on the direction corresponding to the maximum eigenvalue , and obtain the principal curvatures.

[0012] Furthermore, the spectrum-geometry coupling function is: , where 、 、 is a tunable hyperparameter, is the normal vector of the sampling point, For the viewing direction, is the density prediction result, is the principal curvature.

[0013] Furthermore, the spectrum coupling coding vector is calculated, including: Calculate the energy weight of each frequency component based on the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction; The sampling points are learnably position-encoded, and the amplitude of each frequency component is modulated by the energy weight of the frequency component to obtain a modulated coding vector, namely a spectrum-coupled coding vector.

[0014] Furthermore, NeRF volume rendering is performed based on the density and color prediction results. The difference between the rendering result and the real image is optimized and trained to obtain a high-fidelity synthetic image, including: According to the density and color prediction results, the NeRF volume rendering formula is used to integrate the color of each ray to obtain the rendering result; Compare the rendered result with the pixel value in the corresponding real image and calculate the loss function; The network parameters and spectral weights are jointly optimized based on the back propagation of the differentiable coding layer until the loss function is less than the preset value. The optimization is completed and a trained model and high-fidelity synthetic images of any perspective are obtained.

[0015] In a second aspect, the present invention provides a high-fidelity representation system for neural radiation fields based on geometric spectrum coupling optimization, the system comprising: An acquisition unit, used to acquire sparse multi-view image data and calculate the corresponding camera internal and external parameters; The sampling unit is used to generate light based on the internal and external parameters of the camera, and to sample the space along the direction of the light to obtain sampling points; An information acquisition unit is used to input the position of the sampling point into a pre-built differentiable coding layer. The differentiable coding layer predicts the density of the sampling point through forward propagation of a neural network, and based on the density prediction result, obtains the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction; The coupling unit is used to construct a spectrum-geometry coupling function based on the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction, and calculate the spectrum coupling coding vector; The optimization unit is used to pass the spectral coupling coding vector as input into the NeRF main network and output density and color prediction results; and perform NeRF volume rendering based on the density and color prediction results; and optimize the difference between the rendering result and the real image to obtain a high-fidelity synthetic image.

[0016] In a third aspect, the present invention further provides a computer-readable storage medium, which stores a computer program, and is characterized in that when the computer program is executed by a processor, it implements the above-mentioned high-fidelity representation method of neural radiation field based on geometric spectrum coupling optimization.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. The present invention uses a high-fidelity neural radiation field representation method and system based on geometric spectrum coupling. This method includes constructing a spectrum-geometry coupling function. This coupling function calculates the energy weight of each frequency component based on the principal curvature, density gradient, and normal and viewing angle information of the sampling point, and modulates the frequency amplitude in the position encoding to enhance the neural radiation field model's ability to express high-frequency detail areas. The key advantages of this method include: (1) Modular integration: The proposed method can be integrated into existing NeRF implementations without modifying the original sampling strategy and volume rendering framework. It only requires feeding the geometric attributes of the input points into the spectrum modulation submodule, dynamically adjusting their encoded frequency components, and then feeding them into the main network. Finally, the network parameters and modulation function coefficients are simultaneously optimized through error backpropagation. (2) Adaptability to complex geometry: In dense textures or high curvature areas, the network can automatically enhance high-frequency components to capture details; in smooth areas, it can reduce frequencies to avoid overfitting noise; (3) View-dependent optimization: For view-sensitive areas such as specular reflections or transparent materials, the learning mechanism can dynamically adjust the direction encoding frequency to improve the physical accuracy of ray tracing; (4) Balance between high-frequency detail enhancement and low-frequency stability: By adaptively enhancing high-frequency components, the model's rendering clarity of tiny structures is significantly improved; high-frequency weights are reduced in texture-free areas to avoid artifacts in the rendering results caused by high-frequency noise, making the overall scene closer to the real optical characteristics; (5) Improved interpretability and physical consistency: The trained frequency parameters can be mapped to the geometric complexity distribution of the scene, providing an intuitive basis for algorithm debugging; in the volume rendering equation, the dynamically adjusted spectrum can better meet the physical law of light energy attenuation and reduce color drift.

[0018] 2. The present invention is based on a high-fidelity representation method and system for neural radiation fields based on geometric spectrum coupling. The expected benefits and commercial value of the technical solution of the present invention after transformation are as follows: (1) Improved visual effects: This method can more accurately capture and reproduce detailed information in the scene, such as fine textures and sharp edges, thereby significantly improving the realism and quality of the generated image. This high-quality visual presentation is crucial for film and television special effects, advertising production, virtual reality (VR) and augmented reality (AR) applications, helping to attract more users and customers and increase market share. (2) Expanding application scenarios and enhancing user interaction experience: The improved NeRF technology can be applied in more fields, such as architectural design visualization, cultural heritage protection, fine enhancement of tissue boundaries in medical images, and high-frequency enhancement of crack identification in industrial inspection. (3) The technical solution of the present invention solves the problem of NeRF's difficulty in capturing high-frequency details. This method enhances the model's ability to learn features of different scales, not only improving the ability to capture high-frequency details, but also maintaining good modeling of low-frequency information, further improving the model's overall expressiveness. With the improvement of the ability to capture high-frequency details, this technology can be applied to more situations requiring highly accurate three-dimensional reconstruction, such as digital protection of cultural heritage, precision industrial design and testing, and medical image analysis, bringing new possibilities and development opportunities to these fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings: Figure 1 This is a flow chart of the method for high-fidelity representation of neural radiation fields based on geometric spectrum coupling of the present invention; Figure 2 This is a diagram of the structure of the spectrum-geometry coupling function in the present invention, showing how the principal curvature, density gradient, normal and viewing angle information work together in the frequency modulation coefficient calculation process; Figure 3 The experimental results show the significant improvement in rendering quality in high-frequency areas before and after the introduction of the method of the present invention. Figure 4 This is a structural block diagram of the high-fidelity representation system of neural radiation field based on geometric spectrum coupling of the present invention. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with examples and drawings. The exemplary embodiments of the present invention and their descriptions are only used to explain the present invention and are not intended to limit the present invention.

[0021] In order to improve the rendering quality of neural radiation fields, it is difficult to accurately capture local and subtle changes in existing neural radiation fields, which may lead to problems such as insufficient high-frequency details in the generated images. This invention couples information such as principal curvature with the spectrum encoding process.

[0022] The main design points of the present invention are: (1) Constructing a differentiable coding layer: ; (2) The differentiable coding layer calculates the loss function through the forward propagation of the neural network ; (3) Back propagation jointly optimizes network parameters and spectrum weights ; (4) Constructing spectrum-geometry coupling function; (5) The spectral coupling coding vector is passed as input to the MLP network in NeRF, while retaining the traditional position and direction coding information, and the color prediction result is obtained through the volume rendering algorithm.

[0023] Specifically, the steps to construct the differentiable coding layer are as follows: Construct the frequency parameters as a learnable weight matrix, , Where, is the three-dimensional position coordinate of the sampling point, is a learnable frequency vector, m is the frequency base, and m determines the encoding dimension. The weight matrix is ​​decomposed into , where are left singular vectors (orthogonal matrices), Is a singular value matrix, which is a diagonal matrix with elements , Right singular vectors (orthogonal matrices). V determines the rotation direction of the basis function in 3D space, and its physical meaning is the frequency space orientation; represents the energy distribution in the frequency band, The larger the value, the stronger the frequency band energy. U represents the frequency band coupling relationship, which describes the correlation between different frequency components.

[0024] Specifically, the differentiable coding layer calculates the loss function through the neural network forward propagation. The specific steps are as follows: Build a fully connected neural network that accepts the encoded position and orientation as input and outputs density and color. Given the volume density and color function of the scene, volume rendering can obtain the color C(r) of any camera ray r(t) = o + td, given by the following formula, where o is the camera position and d is the viewing direction.

[0025]

[0026] In the formula and are the volume density and color along the camera ray at point r(t), d is the line of sight, and dt is the differential distance traveled by the ray in each integration step.

[0027] T(t) is the cumulative transmittance, which represents the probability that a ray is not intercepted from t1 to t2 and is given by the following formula:

[0028] A novel view is generated by tracing a camera ray C(r) through each pixel of the image to be synthesized. The original NeRF and most subsequent methods use a non-deterministic hierarchical sampling method, where the ray is divided into N equally spaced intervals and a sample is uniformly drawn from each interval. Therefore, the formula T(t) can be approximated as:

[0029] In the formula , where is the distance between sample i and sample i+1. is the volume density and color estimated by MLP at sampling point i. is the opacity of sampling point i, as shown in the following formula.

[0030]

[0031] The expected depth of a ray can be calculated from the accumulated transmittance:

[0032] The loss function is the sum of the squared differences between the predicted color and the pixel color of the real photo, as shown in the following formula.

[0033]

[0034] Specifically, the steps for back propagation to jointly optimize network parameters and spectrum weights are as follows: The core of learnable frequency position encoding is that the weight matrix W participates in gradient optimization. The gradient of W is calculated as: .in The following expansion is possible: ,

[0035] Specifically, the steps for constructing the spectrum-geometry coupling function are as follows: Define the function relationship between the energy of the frequency band and the geometric characteristics: , where represents the energy weight of the i-th frequency band, represents the principal curvature at position x, represents the density gradient norm, Represents the dot product of the normal vector and the viewing direction. Calculate the principal curvature for each sampling point in the scene , the method can be based on local fitting from the predicted density function The second-order derivative is derived from . The energy weight of frequency band i is defined as: , where 、 、 is a tunable hyperparameter, is the normal vector of the sampling point, For the viewing direction, is the density prediction result, is the principal curvature.

[0036] Will Applied to frequency coding, we get a learnable enhanced position coding: , where is the i-th fundamental frequency vector, is the coding characteristic after modulation.

[0037] Example 1 like Figure 1 As shown in the figure, the present invention is based on a high-fidelity representation method of neural radiation fields based on geometric spectrum coupling. The core technology of the present invention is to dynamically adjust the position encoding frequency in NeRF through the spectrum modulation function guided by the principal curvature to improve the expression ability of high-frequency detail areas. The method includes: Step 1: Obtain sparse multi-view image data and calculate the corresponding camera internal and external parameters; In this embodiment, step 1 specifically includes: Acquire a set of sparse multi-view two-dimensional image data, where the two-dimensional image data includes images taken by a real camera or images synthesized based on three-dimensional software; The structure from motion (SfM) algorithm is used to extract the camera intrinsic and extrinsic parameters of each image in the two-dimensional image data.

[0038] Step 2: Generate light based on the camera's internal and external parameters, and perform point sampling in the space along the direction of the light to obtain sampling points; In this embodiment, a three-dimensional ray is generated for each pixel based on the camera's internal and external parameters: Along the direction of the light, a number of sampling points are uniformly sampled between the near plane and the far plane. .

[0039] Step 3: Input the position of the sampling point into the pre-built differentiable coding layer. The differentiable coding layer predicts the density of the sampling point through the forward propagation of the neural network, and based on the density prediction results, obtains the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction; In this embodiment, the expression of the pre-constructed differentiable coding layer is: , that is, constructing the frequency parameters into a learnable weight matrix, , , where is the three-dimensional position coordinate of the sampling point, is a learnable frequency vector, m is the frequency cardinality, and m determines the encoding dimension.

[0040] In this embodiment, the differentiable coding layer predicts the density of the sampling points through the forward propagation of the neural network, inputs the position vector of the traditional coding into the neural network, and predicts the density of the sampling points through the forward propagation. .

[0041] In this embodiment, based on the density prediction result, the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction are obtained, including: Based on the density prediction results, the Hessian matrix is ​​fitted to the local neighborhood of the sampling point to calculate the principal curvature; Based on the density prediction results, the density gradient is calculated for each sampling point; Normal vector based on sampling point and viewing direction , get the angle between the normal vector of the sampling point and the observation direction.

[0042] Specifically, for each sampling point, its principal curvature is estimated by the following method: First, based on the K nearest neighbor strategy, at the sampling point Set a fixed radius around , select all that satisfy point; Secondly, the position and density prediction results of the neighborhood points are used to construct the local quadratic function approximate density distribution Where, is a constant term, is the linear coefficient in the direction of the first-order derivative, which represents the density change trend in the x, y, and z directions respectively; is the square coefficient of the second-order independent variable, which represents the degree of curvature in the x, y, and z directions respectively; Indicates the coupling change in the x and y directions, Indicates the coupling change in the x and z directions, Indicates the coupling change in the y and z directions; Then, the local quadratic function approximate density distribution is fitted by the least squares method to construct the matrix , , and extract its Hessian matrix H from the fitting coefficients: ; where, is the constant term, is the linear coefficient in the first derivative direction, representing the density variation trend in x, y, z directions respectively; is the second-order coefficient of the square term of the independent variable, representing the bending degree in x, y, z directions respectively; represents the coupling variation between x and y directions, represents the coupling variation between x and z directions, represents the coupling variation between y and z directions; Finally, the eigenvalues of the Hessian matrix H are calculated, and the direction corresponding to the maximum eigenvalue is taken. According to the direction corresponding to the maximum eigenvalue, the principal curvature is estimated, and the principal curvature is obtained.

[0043] Step 4, according to the principal curvature, the density gradient, and the angle between the normal vector of the sampling point and the observation direction, a spectrum-geometry coupling function is constructed, and a spectrum-coupling coding vector is calculated. In this embodiment, the spectrum-geometry coupling function is: , where , , is a tunable hyperparameter, is the normal vector of the sampling point, is the observation direction, is the density prediction result, is the principal curvature.

[0044] In this embodiment, the spectrum-coupling coding vector is calculated, including: According to the principal curvature of the sampling point, the density gradient, and the angle between the normal vector of the sampling point and the observation direction, the energy weight of each frequency component is calculated. The position of the sampling point is learned, and the amplitude of each frequency component is modulated by the energy weight of the frequency component to obtain the modulated coding vector, i.e. the spectrum-coupling coding vector.

[0045] Specifically, the position of each sampling point is used to modulate the Fourier coding together with its frequency weight , , where is the i-th basis frequency vector, is the modulated coding feature.

[0046] In step 5, the spectrum coupling coding vector is passed as input into the NeRF main network, and the density and color prediction results are output. Based on the density and color prediction results, NeRF volume rendering is performed. The difference between the rendering result and the real image is optimized and trained to obtain a high-fidelity synthetic image.

[0047] In this embodiment, the spectrum coupling code vector is input into the NeRF main network, that is, the modulated code vector , viewing direction coding The neural network outputs the predicted density of each sampling point. With color .

[0048] In this embodiment, NeRF volume rendering is performed based on the density and color prediction results. The difference between the rendering result and the real image is optimized and trained to obtain a high-fidelity synthetic image, including: Based on the density and color prediction results, the NeRF volume rendering formula is used to integrate the color on each ray , and get the rendering result; where C(r) represents the color value finally rendered along the ray r, is the opacity of sampling point i, Ti represents the transmittance, which indicates the probability that light is transmitted from the starting point to the iiith point without being blocked. ci Represents the color of the i-th sampling point, usually the RGB value predicted by the neural network; Compare the rendered result with the pixel value in the corresponding real image and calculate the loss function ; R represents the set of rays used in a batch of training; C(r) represents the predicted color of the ray in the rendered image obtained by the volume rendering formula; C gt (r) represents the pixel color value corresponding to the light in the real image; L represents the mean square error loss function, which is used to measure the difference between the predicted color and the real color.

[0049] Joint optimization of network parameters and spectral weights based on back-propagation of differentiable coding layers , until the loss function is less than the preset value, the optimization ends, and a trained model and a high-fidelity synthetic image of any perspective are obtained. W represents the joint matrix of the weight matrices in the position encoding function, Φ represents the position encoding vector function, and C represents the rendering result.

[0050] The above optimization process involves performing gradient updates on both the frequency modulation weights and the neural network weights.

[0051] The present invention is based on a trained model and renders the new perspective image in the inference phase. The trained model is loaded and the new camera pose is input to generate the ray set under the new perspective.

[0052] The proposed method is embedded as a "geometry-aware spectral modulation module" throughout the NeRF training and inference process, eliminating the need to modify the main rendering pipeline and ensuring module independence. The spectral modulation module is decoupled from the main NeRF network and can be embedded as a standalone module in any volume rendering architecture, including native NeRF, Mip-NeRF, and TensoRF. Each submodule is connected via a standard interface, enabling flexible combination and replacement, and is applicable to a variety of neural graphics architectures, including native NeRF, Mip-NeRF, and TensoRF.

[0053] This paper proposes a spectrum modulation mechanism based on principal curvature guidance. By introducing geometric information (principal curvature, density gradient, and the angle between the normal and the viewing angle) into the frequency modulation function, it dynamically adjusts the spectral energy distribution of the position encoding in the neural radiation field. This mechanism has the following significant technical advantages: (1) Improved detail capture capability: Traditional NeRF has limitations in processing high-frequency details, such as difficulty accurately reproducing fine textures or sharp edges. This method can significantly improve the NeRF model's ability to express high-frequency structures (such as fine textures, sharp edges, and structural mutation areas), adaptively enhance the frequency response in high-curvature areas, and achieve more realistic 3D scene reconstruction.

[0054] (2) Enhanced model flexibility: The frequency modulation function uses the principal curvature as the core signal source, making the spectrum weight distribution strongly correlated with the geometric complexity of the scene, thereby suppressing high-frequency redundancy in flat areas and automatically improving expressiveness in complex areas, effectively avoiding overfitting or artifacts.

[0055] (3) Enhanced viewing sensitivity: By introducing the dot product component of the normal vector and the viewing direction, the system has the ability to self-adjust the frequency when processing viewing-related effects such as mirror reflection and transparent materials, thereby enhancing physical consistency.

[0056] (4) Module independence and strong versatility: The spectrum modulation submodule has a clear structure and can be directly embedded in a variety of neural graphics systems including original NeRF, Mip-NeRF, TensoRF, etc., without changing the original rendering framework, facilitating rapid deployment and migration.

[0057] (5) Efficient optimization and strong interpretability: The learnable frequency parameters can be jointly optimized through back-propagation, making the training process stable and converging faster. At the same time, the modulated frequency map can be visualized as a geometric complexity heat map, providing intuitive support for debugging and model interpretation.

[0058] (6) Applicable to a wide range of application scenarios: This method is suitable for scenarios with high requirements for high-fidelity modeling, including cultural heritage digitization, medical image analysis, industrial crack detection, digital twins and other high-precision visualization tasks, and has significant engineering value.

[0059] The method of the present invention is suitable for modeling high-frequency detail areas, including texture-rich areas, boundary-sharp areas, and perspective-sensitive areas; and the encoding process is physically interpretable, and the frequency modulation weight visualization can be mapped to a scene complexity distribution map.

[0060] Example 2 like Figure 4 As shown, the difference between this embodiment and embodiment 1 is that this embodiment provides a high-fidelity representation system for neural radiation fields based on geometric spectrum coupling optimization, which corresponds one-to-one to the high-fidelity representation method for neural radiation fields based on geometric spectrum coupling optimization in embodiment 1. The system includes: An acquisition unit, used to acquire sparse multi-view image data and calculate the corresponding camera internal and external parameters; The sampling unit is used to generate light based on the internal and external parameters of the camera, and to sample the space along the direction of the light to obtain sampling points; An information acquisition unit is used to input the position of the sampling point into a pre-built differentiable coding layer. The differentiable coding layer predicts the density of the sampling point through forward propagation of a neural network, and based on the density prediction result, obtains the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction; The coupling unit is used to construct a spectrum-geometry coupling function based on the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction, and calculate the spectrum coupling coding vector; The optimization unit is used to pass the spectral coupling coding vector as input into the NeRF main network and output density and color prediction results; and perform NeRF volume rendering based on the density and color prediction results; and optimize the difference between the rendering result and the real image to obtain a high-fidelity synthetic image.

[0061] Among them, the execution process of each unit can be performed according to the process steps of the high-fidelity representation method of neural radiation field based on geometric spectrum coupling optimization in Example 1, and will not be repeated one by one in this embodiment.

[0062] At the same time, the present invention also provides a computer-readable storage medium, which stores a computer program, characterized in that when the computer program is executed by a processor, it implements the above-mentioned high-fidelity representation method of neural radiation field based on geometric spectrum coupling optimization.

[0063] This invention aims to address the limitations of existing Neural Radiance Field (NeRF) in detail preservation, particularly the blurred high-frequency details and insufficient texture restoration. To this end, the core of the proposed system is to introduce a spectral modulation mechanism driven by principal curvature, density gradient, and normal and viewpoint information to enhance the model's ability to capture high-frequency details.

[0064] This paper designs a spectrum-geometry coupling function to dynamically adjust the energy distribution of each frequency component in Fourier encoding, enhancing the model's perception and expression of local structural changes. This mechanism has the following advantages: (a) Detail enhancement: Automatically enhances high-frequency response in texture-dense and geometrically complex areas, achieving clear reconstruction of sharp edges and microstructures; (b) Noise suppression: suppressing high-frequency amplitudes in low-texture areas to avoid overfitting and artifacts; (c) Enhanced view consistency: The angle between the normal and the viewing direction is considered to make the model more realistic in view-sensitive areas such as reflection and transparency. (d) Strong module versatility: The modulation mechanism is decoupled from the NeRF backbone structure, making it highly adaptable and easy to integrate; (e) Strong interpretability: Frequency modulation parameters can map the complexity distribution of the scene, assisting debugging and analysis.

[0065] In summary, the present invention significantly improves the modeling fidelity of NeRF in complex scenes through the geometric spectrum joint modeling mechanism, providing key support for virtual reality, medical image reconstruction, cultural heritage digitization and other fields.

[0066] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0068] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0070] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A high-fidelity representation method for neural radiation fields based on geometric spectrum coupling, characterized by: The method includes: Acquire sparse multi-view image data and calculate the corresponding camera internal and external parameters; Generate light based on the camera's internal and external parameters, and perform point sampling on the space along the direction of the light to obtain sampling points; The position of the sampling point is input into the pre-built differentiable coding layer. The differentiable coding layer predicts the density of the sampling point through the forward propagation of the neural network, and based on the density prediction results, obtains the principal curvature, density gradient and the angle between the normal vector of the sampling point and the observation direction; According to the principal curvature, density gradient and the angle between the normal vector of the sampling point and the observation direction, a spectrum-geometry coupling function is constructed, and the spectrum coupling coding vector is calculated; The spectral coupling coding vector is passed as input into the NeRF main network, and the density and color prediction results are output; NeRF volume rendering is performed based on the density and color prediction results; the difference between the rendering result and the real image is optimized and trained to obtain a high-fidelity synthetic image.

2. The high-fidelity representation method of neural radiation field based on geometric spectrum coupling according to claim 1 is characterized in that: Acquire sparse multi-view image data and calculate the corresponding camera internal and external parameters, including: Acquire a set of sparse multi-view two-dimensional image data, wherein the two-dimensional image data includes images taken by a real camera or images synthesized based on three-dimensional software; A structure-from-motion algorithm is used to extract camera intrinsic parameters and extrinsic parameters of each image in the two-dimensional image data.

3. The high-fidelity representation method of neural radiation field based on geometric spectrum coupling according to claim 1 is characterized in that: The expression of the differentiable coding layer is: , that is, constructing the frequency parameters into a learnable weight matrix, , , where is the three-dimensional position coordinate of the sampling point, is a learnable frequency vector, m is the frequency cardinality, and m determines the encoding dimension.

4. The high-fidelity representation method of neural radiation field based on geometric spectrum coupling according to claim 1 is characterized in that: Based on the density prediction results, the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction are obtained, including: Based on the density prediction results, the Hessian matrix is ​​fitted to the local neighborhood of the sampling point to calculate the principal curvature; Based on the density prediction results, calculate the density gradient; Based on the normal vector of the sampling point and the viewing direction, the angle between the normal vector of the sampling point and the viewing direction is obtained.

5. The high-fidelity representation method of neural radiation field based on geometric spectrum coupling according to claim 4 is characterized in that: Based on the density prediction results, the Hessian matrix is ​​fitted to the local neighborhood of the sampling point to calculate the principal curvature, including: Based on the K nearest neighbor strategy, at the sampling point Set a fixed radius around , select all that satisfy point; Use the location and density prediction results of neighborhood points to construct a local quadratic function approximate density distribution , where is a constant term, is the linear coefficient in the direction of the first-order derivative, which represents the density change trend in the x, y, and z directions respectively; is the square coefficient of the second-order independent variable, which represents the degree of curvature in the x, y, and z directions respectively; Indicates the coupling change in the x and y directions, Indicates the coupling change in the x and z directions, Indicates the coupling change in the y and z directions; The local quadratic function approximates the density distribution by fitting it using the least squares method, and constructs the matrix , , and extract its Hessian matrix H from the fitted coefficients: , where is a constant term, is the linear coefficient in the direction of the first-order derivative, which represents the density change trend in the x, y, and z directions respectively; is the square coefficient of the second-order independent variable, which represents the degree of curvature in the x, y, and z directions respectively; Indicates the coupling change in the x and y directions, Indicates the coupling change in the x and z directions, Indicates the coupling change in the y and z directions; The eigenvalue of the Hessian matrix H is calculated, and the direction corresponding to the maximum eigenvalue is taken; and the principal curvature is estimated according to the direction corresponding to the maximum eigenvalue to obtain the principal curvature.

6. The high-fidelity representation method of neural radiation field based on geometric spectrum coupling according to claim 1 is characterized in that: The spectrum-geometry coupling function is: , where 、 、 is a tunable hyperparameter, is the normal vector of the sampling point, For the viewing direction, is the density prediction result, is the principal curvature.

7. The high-fidelity representation method of neural radiation field based on geometric spectrum coupling according to claim 1 is characterized in that: The spectrum coupling coding vector is calculated, including: Calculate the energy weight of each frequency component based on the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction; The sampling points are learnably position-encoded, and the amplitude of each frequency component is modulated by the energy weight of the frequency component to obtain a modulated coding vector, namely a spectrum-coupled coding vector.

8. The high-fidelity representation method of neural radiation field based on geometric spectrum coupling according to claim 1 is characterized in that: Perform NeRF volume rendering based on density and color prediction results; The difference between the rendering result and the real image is optimized and trained to obtain a high-fidelity synthetic image, including: According to the density and color prediction results, the NeRF volume rendering formula is used to integrate the color of each ray to obtain the rendering result; Compare the rendered result with the pixel value in the corresponding real image and calculate the loss function; The network parameters and spectral weights are jointly optimized based on the back propagation of the differentiable coding layer until the loss function is less than the preset value. The optimization is completed and a trained model and high-fidelity synthetic images of any perspective are obtained.

9. A high-fidelity representation system for neural radiation fields based on geometric spectrum coupling optimization, characterized by: The system includes: An acquisition unit, used to acquire sparse multi-view image data and calculate the corresponding camera internal and external parameters; A sampling unit, configured to generate light based on camera internal and external parameters, and perform point sampling in space along the direction of the light to obtain sampling points; An information acquisition unit is used to input the position of the sampling point into a pre-built differentiable coding layer. The differentiable coding layer predicts the density of the sampling point through forward propagation of a neural network, and based on the density prediction result, obtains the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction; The coupling unit is used to construct a spectrum-geometry coupling function based on the principal curvature, density gradient, and the angle between the normal vector of the sampling point and the observation direction, and calculate the spectrum coupling coding vector; The optimization unit is used to pass the spectral coupling coding vector as input into the NeRF main network and output density and color prediction results; perform NeRF volume rendering based on the density and color prediction results; and optimize the difference between the rendering result and the real image to obtain a high-fidelity synthetic image.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for high-fidelity representation of neural radiation fields based on geometric spectrum coupling optimization as described in any one of claims 1 to 8 is implemented.

Images