Rendering method of neural radiance field model and electronic device

Abstract

The application discloses a rendering method of a neural radiation field model and electronic equipment, and relates to the technical field of computers, wherein the method comprises the following steps: simplifying the network layers of an original neural radiation field model to obtain a second neural radiation field model which meets a preset byte requirement; in response to the fact that a plurality of indexes of the second neural radiation field model all meet corresponding requirements, converting the second neural radiation field model into an executable format of an online calculation component to generate a second neural radiation field model file; and rendering the second neural radiation field model, wherein in the process of rendering, the rendering precision of the second neural radiation field model is adjusted according to actual information until a target rendering precision corresponding to the actual information is adapted. The method solves the problem that a three-dimensional model rendering in the prior art cannot effectively adapt to a Web end, achieves a complete solution from image acquisition, preprocessing, model training, format conversion to front-end rendering, and helps the technical effect that the NeRF technology is applied to the Web end.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a rendering method and electronic device for a neural radiation field model. Background Technology

[0002] In related technologies, online display of 3D products based on NeRF (Neural Radiance Fields) models mainly employs two techniques. One is a static 3D model rendering scheme based on Three.js (an open-source JavaScript 3D graphics library), and the other is to use the original NeRF model to render and display the 3D scene.

[0003] However, among related technologies, the static 3D model rendering scheme based on Three.js has poor realism and is prone to long loading times due to the large file size of the 3D model. It also tends to cause lag or crashes on the Web (World Wide Web) due to its high memory consumption, which seriously affects the user experience. The rendering and display scheme using the original NeRF model to realize the 3D scene cannot adapt to the single loading size of the Web. The rendering frame rate is low and cannot meet the real-time interaction of the Web. It is dependent on the deployment environment and hardware resources and it is difficult to guarantee stable operation under various device configurations of users. These issues urgently need to be addressed. Summary of the Invention

[0004] This application provides a rendering method and electronic device for a neural radiation field model, which at least solves the following problems in the related technologies: the large file size of the 3D model rendering scheme leads to a long loading time; it also causes lag or crashes on the web due to the large amount of terminal memory occupied; it cannot adapt to the single loading size of the web; the low rendering frame rate cannot meet the real-time interaction of the web; it depends on the deployment environment and hardware resources; it is difficult to guarantee stable operation under various device configurations of users; and it seriously affects the user experience.

[0005] This application provides a rendering method for a neural radiation field model, comprising the following steps: simplifying the network layer number of the original neural radiation field model to a target layer number adapted to lightweight requirements, and inserting separable convolutional layers to obtain a first neural radiation field model; performing integer quantization processing on the model parameters of the first neural radiation field model to obtain a second neural radiation field model that meets preset byte requirements; in response to multiple indicators of the second neural radiation field model meeting the corresponding requirements, converting the second neural radiation field model into an executable format of an online computing component to generate a second neural radiation field model file; and rendering the second neural radiation field model based on the online computing component and the second neural radiation field model file, wherein, during the rendering process, the actual information of the online computing component is detected to adjust the rendering precision of the second neural radiation field model according to the actual information until the rendering precision is adapted to the target rendering precision corresponding to the actual information.

[0006] This application also provides a rendering apparatus for a neural radiation field model, comprising: a lightweighting module, used to simplify the network layer number of the original neural radiation field model to a target layer number adapted to lightweight requirements, and insert separable convolutional layers to obtain a first neural radiation field model, and to perform integer quantization processing on the model parameters of the first neural radiation field model to obtain a second neural radiation field model that meets preset byte requirements; a conversion module, used to convert the second neural radiation field model into an executable format of an online computing component in response to multiple indicators of the second neural radiation field model meeting the corresponding requirements, to generate a second neural radiation field model file; and a rendering module, used to render the second neural radiation field model based on the online computing component and the second neural radiation field model file, wherein, during the rendering process, the actual information of the online computing component is detected to adjust the rendering precision of the second neural radiation field model according to the actual information until the rendering precision is adapted to the target rendering precision corresponding to the actual information.

[0007] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the rendering method of any of the above-described neural radiation field models.

[0008] This application also provides a non-volatile computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the rendering method for any of the above-described neural radiation field models.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of rendering a method for any of the above-described neural radiation field models.

[0010] This application simplifies the original neural radiation field model to the target number of layers, inserts separable convolutional layers and integer quantization, and converts the resulting second neural radiation field model into an executable format for online computing components to generate a second neural radiation field model file. This second neural radiation field model is then rendered, and the rendering precision can be adjusted according to the actual information of the online computing components to match the target rendering precision. This addresses the technical problems in related technologies, such as large file sizes leading to long loading times for 3D model rendering schemes, memory consumption causing stuttering or crashes on the web, inability to adapt to the single-load size of the web, low rendering frame rates failing to meet real-time interaction requirements, dependence on deployment environment and hardware resources, and difficulty in ensuring stable operation across various user device configurations, severely impacting user experience. This application provides a complete solution from image acquisition, preprocessing, model training, format conversion to front-end rendering, requiring no manual intervention, lowering the barrier to entry for NeRF technology applications on the web, and enabling its application in e-commerce, cultural heritage, industry, and other sectors. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a rendering method for a neural radiation field model according to an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a rendering device for a neural radiation field model provided according to an embodiment of this application.

[0013] Figure label: 10 - Rendering device for neural radiation field model; 100 - Lightweight module, 200 - Conversion module and 300 - Rendering module. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0015] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0016] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] The embodiments of this application provide a rendering method for a neural radiation field model, and the method is described in detail in conjunction with the execution flow of the rendering method for the neural radiation field model.

[0018] Specifically, Figure 1 This is a flowchart of a rendering method for a neural radiation field model provided according to an embodiment of this application.

[0019] like Figure 1 As shown, the rendering method for this neural radiation field model includes the following steps: In step S101, the number of network layers in the original neural radiation field model is simplified to the target number of layers that are suitable for lightweight requirements, and separable convolutional layers are inserted to obtain the first neural radiation field model. The model parameters of the first neural radiation field model are then subjected to integer quantization to obtain the second neural radiation field model that meets the preset byte requirements.

[0020] Understandably, the original NeRF model (hereinafter referred to as the original NeRF model) refers to the initial NeRF model without any processing. The original NeRF model typically has 50 million to 100 million parameters, and the size of the model file after training can reach 5 to 10 GB, far exceeding the reasonable upper limit of the size for a single loading on the web (usually ≤10 MB). It cannot be directly loaded and run on the web, which severely limits the application of the original NeRF model in online scenarios.

[0021] Among related technologies, one approach is a static 3D model rendering solution based on Three.js. First, a 3D mesh model of the product is constructed using professional 3D modeling software. After processing the model with texture mapping and material settings, it is exported in a web-supported model format. Then, the 3D model is loaded into the front-end webpage using the Three.js library, and WebGL (Web Graphics Library) technology is used to implement interactive operations such as model rotation, scaling, and translation, thus providing users with multi-angle product display effects. Another approach uses a raw NeRF model to render and display 3D scenes. This involves acquiring a large number of multi-angle 2D images of the object, training a NeRF model using a deep learning framework, and generating a realistic rendered image from the input viewpoint parameters. The NeRF model is then run through a local application to achieve interactive rendering of the 3D scene. However, both approaches have limitations. They cannot meet the real-time interaction requirements of the web platform, rely heavily on the deployment environment and hardware resources, and are difficult to guarantee stable operation across various user device configurations.

[0022] In some embodiments, the original NeRF model may be lightweighted first, including but not limited to simplifying the number of network layers of the original NeRF model to the target number of layers that are adapted to the lightweighting requirements, and inserting separable convolutional layers to obtain the first neural radiation field model (hereinafter referred to as the first NeRF model).

[0023] Here, the number of network layers can be understood as the number of fully connected layers in the original NeRF model. The embodiments of this application can simplify the number of fully connected layers in the original NeRF model to a target number that suits lightweight requirements.

[0024] Here, the lightweight requirement can be understood as a set of optimization objectives for the original NeRF model when deploying it to the web application, based on various constraints (such as volume constraints, model parameter constraints, rendering latency constraints, and visual fidelity constraints). These objectives include things like a size ≤ 100MB, a model parameter count less than a certain number, meeting rendering latency standards, and no degradation in visual fidelity. The target number of layers can be understood as the desired final number of fully connected layers after simplifying the original NeRF model according to the lightweight requirement.

[0025] Taking the original NeRF model with 8 fully connected layers as an example, when the lightweight requirement is to adapt to real-time rendering on mobile and web platforms, the number of model parameters needs to be minimized. Therefore, the target number of layers can be, but is not limited to, half of the number of fully connected layers in the original NeRF model, such as 3 to 4 layers. When the lightweight requirement is to adapt to high-fidelity display on desktop platforms, the number of model parameters can be appropriately reduced, and the target number of layers can be, but is not limited to, retained to 5 to 6 layers.

[0026] Then, in the embodiments of this application, the number of fully connected layers in the original NeRF model can be further simplified to a certain number of layers that are suitable for lightweight requirements, so as to obtain the first neural radiation field model.

[0027] Next, in this embodiment of the application, the model parameters of the first neural radiation field model can be quantized by integer to obtain a second neural radiation field model (hereinafter referred to as the second NeRF model) that meets the preset byte requirements.

[0028] Among them, the model parameters are the core values ​​in the NeRF model that support the neural network to fit the radiation field (density / color function) of the 3D scene. They include, but are not limited to, the weights and bias terms of the fully connected layers, as well as tensor data such as the mean, variance, and scaling factor pre-computed by the batch normalization (BN) layer.

[0029] Integer quantization here refers to a lightweight compression technique for NeRF model parameters, used to reduce the number of integer bits in the NeRF model parameters from the original NeRF model parameters to a certain number of integer bits (below the original NeRF model parameters).

[0030] The preset byte requirement here refers to the total number of bytes of all model parameters of the second NeRF model after integer quantization processing meeting the preset byte threshold for web deployment. Here, the preset byte threshold can be understood as a byte limit set in advance based on lightweight requirements (such as the upper limit of mobile browser video memory, mobile network transmission speed, and single-frame rendering memory usage constraints). Specifically, it can be set or adjusted by those skilled in the art according to actual needs. This embodiment is only an example and does not impose specific limitations.

[0031] For example, this application can perform 8-bit integer quantization on the model parameters of the original NeRF model. That is, by calculating the maximum / minimum value of the model parameters, the scaling factor is determined, and the floating-point range of the model parameters of the original NeRF model is mapped from the original 32-bit floating-point type to the 8-bit integer range of [-128, 127]. This significantly reduces the memory and video memory usage of the model parameters, thereby reducing the loading time of the model on the Web side. In addition, integer calculation is more compatible with the parallel computing efficiency of WebGPU (Web Graphics Processing Unit) than floating-point calculation, which can effectively reduce rendering latency.

[0032] The embodiments of this application can simplify the number of network layers of the original NeRF model and insert separable convolutional layers, which can not only adapt to the lightweight requirements of the Web and reduce the computational complexity, but also retain the model rendering accuracy. Furthermore, by performing integer quantization on the model parameters, the parameter volume can be significantly compressed to meet certain byte requirements, which helps the NeRF model to achieve efficient loading and smooth real-time rendering on the Web, while taking into account both rendering effects and the resource constraints of browsers / WebGPUs.

[0033] Optionally, in one embodiment of this application, the number of network layers in the original neural radiation field model is simplified to a target number of layers that is suitable for lightweight requirements, and separable convolutional layers are inserted to obtain a first neural radiation field model. This includes: simplifying the fully connected layers of the original neural radiation field model to the target number of layers to obtain a third neural radiation field model; and, based on the target number of layers and the simplification requirements of the original neural radiation field model, inserting separable convolutional layers to replace the fully connected layers in the fully connected layers of the third neural radiation field model to obtain the first neural radiation field model.

[0034] As can be understood from the descriptions of other embodiments, the number of network layers in this application refers to the number of fully connected layers in the original NeRF model.

[0035] Therefore, in actual implementation, when simplifying the network layer number of the original NeRF model to the target layer number and inserting separable convolutional layers to obtain the first NeRF model, this application mainly simplifies the fully connected layers of the original NeRF model to a certain number of layers.

[0036] The NeRF model after simplifying the number of fully connected layers is called the third NeRF model. In this embodiment, separable convolutional layers can be inserted into the fully connected layers of the third NeRF model to obtain the first neural radiation field model in this embodiment.

[0037] Specifically, embodiments of this application may, but are not limited to, insert separable convolutional layers to replace fully connected layers in the fully connected layers of the third NeRF model to obtain the first neural radiation field model, based on the simplification requirements of the target number of layers and the original NeRF model.

[0038] Here, simplification requirements can be understood as the need for lightweighting based on the original NeRF model, or the pre-set requirements for the computational cost of the original NeRF model. For example, the computational cost of the NeRF model after simplifying the number of fully connected layers (the third NeRF model) is reduced by 35% compared to the original NeRF model.

[0039] It should be noted that the first neural radiation field model, the second NeRF model, and the third NeRF model are all essentially NeRF models. In this application embodiment, they are only referred to as such to distinguish NeRF models after different processing.

[0040] The embodiments of this application can significantly reduce the computational complexity and number of parameters of the original NeRF model by simplifying the fully connected layers of the original NeRF model to a certain number of layers that meet the requirements of lightweighting. Separable convolutional layers are inserted to replace some of the fully connected layers, thereby controlling resource consumption while retaining the model's ability to fit the radiation field of 3D scene. This lays the foundation for subsequent integer quantization and web-based deployment, while also taking into account the model's rendering accuracy and browser / WebGPU compatibility.

[0041] Step S102: In response to the fact that multiple indicators of the second neural radiation field model meet the corresponding requirements, the second neural radiation field model is converted into an executable format of the online computing component to generate the second neural radiation field model file.

[0042] In some embodiments, after obtaining the lightweight second NeRF model, this application can convert the second NeRF model into an executable format of an online computing component to generate a second NeRF model file, provided that multiple indicators of the second NeRF model meet the corresponding requirements.

[0043] Here, the multiple metrics of the second NeRF model can be understood as several key evaluation metrics used to assess the actual state of the lightweight second NeRF model.

[0044] If multiple metrics of the second NeRF model meet the corresponding requirements, the embodiments of this application can convert the second NeRF model into an executable format of an online computing component in response to the fact that multiple metrics of the second NeRF model meet the corresponding requirements, thereby generating the second NeRF model.

[0045] In this context, online computing components can be understood as general-purpose graphics processors implemented on the web, such as WebGPU computing shaders. WebGPU, as a next-generation Web graphics API (Application Programming Interface), possesses powerful parallel computing capabilities and efficient graphics rendering performance. Compared to traditional WebGL, it can fully utilize the computing potential of GPUs (Graphics Processing Units), providing crucial technical support for the real-time execution of complex graphics algorithms on web pages.

[0046] However, when applying NeRF model technology to the Web, there is an inherent contradiction between the original architecture of the NeRF model and the resource limitations of the Web environment. Therefore, in other embodiments of this application, the computationally and storage-intensive NeRF model has been lightweighted to adapt to the resource limitations of the Web (e.g., model size ≤ 5MB, loading time ≤ 3 seconds, etc.).

[0047] Converting the second NeRF model into an executable format for the online computing component to generate the second NeRF model file involves converting the lightweight second NeRF model from a training-state weight file into a format that can be directly loaded and executed in the online computing component on the web, and generating the corresponding executable model file (i.e., the second NeRF model file).

[0048] This application embodiment can convert the lightweight second NeRF model into an executable format adapted to the Web-based online computing component and generate corresponding model files, based on the key indicators of the second NeRF model meeting certain requirements. This resolves the inherent contradiction between the original NeRF model and the environmental resource limitations of the Web. By relying on the parallel computing and efficient rendering capabilities of WebGPU, the NeRF model can be directly loaded and run on the Web, realizing real-time rendering of complex 3D scenes, thus balancing the adaptability of the NeRF model to the Web, running efficiency, and rendering effect.

[0049] Optionally, in one embodiment of this application, before multiple metrics of the second neural radiation field model meet the corresponding requirements, the method further includes: training the second neural radiation field model based on batch size information, learning rate, iteration decay, and target iteration number to obtain the number of parameters, model volume, and training time of the trained second neural radiation field model; and determining multiple metrics based on the number of parameters, model volume, and training time.

[0050] In some embodiments, the multiple metrics of the second NeRF model detected in this application include, but are not limited to, the number of parameters, model size, and training time. These parameters, model size, and training time can be obtained, but are not limited to, by training the second NeRF model based on batch size information, learning rate, iteration decay, and target number of iterations.

[0051] For example, this application may, but is not limited to, setting up a cloud processing subsystem in the cloud. This cloud processing subsystem includes, but is not limited to, an image acquisition module, an image preprocessing module, a lightweight NeRF model training module, and a model format conversion module. These modules can collaboratively achieve processes such as product image preprocessing, lightweighting and training of the original NeRF model, and model format conversion. Specifically, this may include, but is not limited to, the following: Image Acquisition Module: This module can be used to acquire multi-angle two-dimensional images of an object. For example, this embodiment can employ an automatic image acquisition device, including but not limited to a rotating platform, a high-definition camera (4K resolution), and a light source system (ring LED light source, color temperature 5500K). After fixing the object on the rotating platform, the platform rotates at preset angle intervals of 5° (5° is an exemplary preferred angle; the specific angle interval can be, but is not limited to, determined by those skilled in the art based on actual conditions; this embodiment is only illustrative and not specifically limited). After each rotation, the high-definition camera captures an image of the object at a fixed position, while the light source system provides uniform and stable lighting conditions to ensure that the acquired image is clear and free of shadow interference. During the acquisition process, the camera parameters need to remain consistent, for example, a focal length of 50mm, an aperture of f / 8, and ISO 100. It can acquire, but is not limited to, images of the object from 72 angles (360° / 5°=72, balancing coverage and efficiency), covering a 360° all-around view of the object, with an acquisition time ≤30 minutes per item.

[0052] Image preprocessing module: This module preprocesses the acquired multi-angle images to improve the accuracy and efficiency of NeRF model training. The preprocessing process includes, but is not limited to, image denoising, size normalization, image alignment, and illumination correction, and can be specifically represented as follows: (1) Image denoising: A 5×5 Gaussian filtering algorithm is used to remove random noise in the image and improve the signal-to-noise ratio to more than 45dB; (2) Size normalization: Adjust all images to 512×512 pixels to ensure that the image specifications of the input model are consistent; (3) Image alignment: The SIFT (Scale-Invariant Feature Transform, an algorithm for extracting and matching local image features) feature point matching algorithm is used to extract the feature points of the objects in each image (≥200 feature points are extracted from each image) to achieve spatial alignment between multiple images and ensure that the position offset error is ≤1 pixel. (4) Illumination correction: Histogram equalization algorithm is used to balance the brightness and contrast of images from different angles, ensuring that the brightness difference coefficient is ≤0.1, so as to reduce the impact of illumination difference on model training.

[0053] Lightweight NeRF Model Training Module: This module is the core of the cloud processing subsystem, mainly responsible for building and training lightweight NeRF models. Specifically, it can be represented as follows: (1) Simplify the number of network layers: The original NeRF model's 8 fully connected layers are reduced to 4 layers (4 layers is an exemplary simplification scheme. In actual applications, it can be determined and adjusted by professionals in this field according to the actual situation. This application embodiment is only an example and does not impose specific restrictions). The number of neurons in each layer is reduced from 256 to 128, and the number of basic parameters is reduced from 8.3 million to 1.6 million. While ensuring the model's expressive power, the number of model parameters is greatly reduced. The NeRF model obtained at this time can be understood as the third NeRF model. (2) Introducing depthwise separable convolution: Insert a 3×3 depthwise separable convolution layer between the 2nd and 3rd fully connected layers (1 layer is an exemplary simplified scheme. In actual application, it can be determined and adjusted by professionals in this field according to the actual situation. This application embodiment is only an exemplary illustration and does not impose specific restrictions), to replace the traditional fully connected layer, further reducing the amount of computation by 35% while maintaining the feature extraction capability; the NeRF model obtained at this time can be understood as the first neural radiation field model; (3) Using parameter quantization technology: The trained model parameters are quantized into 8-bit integers (8-bit integer quantization is an exemplary simplified scheme. In actual application, it can be determined and adjusted by professionals in this field according to the actual situation. This application embodiment is only an exemplary illustration and does not impose specific restrictions). The 32-bit floating-point parameters are converted into 8-bit integers, thereby reducing the volume compression ratio of the original NeRF model to 75% or more. The NeRF model obtained at this time can be understood as the second NeRF model. (4) The training process of the lightweight NeRF model (second NeRF model) in this application embodiment can be implemented based on the PyTorch deep learning framework (an open source deep learning framework based on Python). The training parameters can be set as follows: the batch size is 8, the initial learning rate can be set to 0.001, the decay every 5000 iterations can be set to 50%, and the Adam (Adaptive Moment Estimation) optimizer is used to optimize the parameters with optimization parameters (β1=0.9, β2=0.999). The number of training iterations can be set to 20000.

[0054] During training, multi-angle object images that have been collected and preprocessed can be used as input, but are not limited to. The model parameters of the lightweight NeRF model can be optimized by minimizing the L2 loss function between the rendered image and the real image (the target loss value can be set to ≤0.01), so that the lightweight NeRF model can learn the spatial structure, lighting characteristics and material information of the object.

[0055] Furthermore, the key metrics and corresponding requirements of the lightweight NeRF model can be set, but are not limited to: number of parameters ≤ 4 million (4%~8% of the original NeRF model), model size ≤ 5MB (0.05%~0.1% of the original NeRF model), and training time ≤ 6 hours / piece (single GPU).

[0056] It should be noted that the specific evaluation indicators and their corresponding requirements can be set or adjusted by those skilled in the art according to the actual situation. The embodiments in this application are only illustrative and do not impose specific limitations.

[0057] This application embodiment can train the lightweight second NeRF model using key training parameters such as batch size and learning rate, thereby obtaining key evaluation indicators such as the number of parameters, model size, and training time of the trained second NeRF model. Only when all indicators meet the corresponding requirements are they converted into an executable format adapted to web-based online computing components and a model file is generated. This can effectively ensure that the lightweight NeRF model can meet the resource adaptation standards of the web at the training level, avoid invalid format conversion operations, and ensure that the final generated model file can run efficiently in online computing components such as WebGPU.

[0058] Optionally, in one embodiment of this application, the second neural radiation field model is converted into an executable format of the online computing component to generate a second neural radiation field model file. This includes: extracting the network structure, integer-quantized model parameters, and operators of the second neural radiation field model; constructing a mapping relationship between the operators and the online computing component based on the network structure, integer-quantized model parameters, and operators; converting the forward inference process of the second neural radiation field model into the target language of the online computing component, and encoding the integer-quantized model parameters into binary data; combining the target language and binary data to obtain the definition code of the second neural radiation field model in the online computing component; and combining the mapping relationship and the definition code to complete the conversion of the second neural radiation field model into an executable format of the online computing component to obtain the second neural radiation field model file.

[0059] In actual implementation, when converting the second NeRF model into an executable format of an online computing component and generating the second NeRF model file, this application can first extract the network structure, integer quantized model parameters, and operators of the second NeRF model, and then construct the mapping relationship between the operators and the online computing component based on the network structure, integer quantized model parameters, and operators of the second NeRF model.

[0060] Here, the network structure can be understood as the hierarchical organization of the second NeRF model, including the simplified fully connected layers, the number of inserted separable convolutional layers, the hierarchical order, and the connection relationships between each layer, which is the computational logic framework of the second NeRF model.

[0061] The model parameters after integer quantization can be understood here as the numerical data such as weights and biases in the second NeRF model after integer quantization transformation. They are also the core numerical basis for the second NeRF model to complete radiation field fitting and ray sampling calculation.

[0062] The operator here can be understood as the core computational unit required to complete forward inference in the second NeRF model, such as the ReLU activation function, batch normalization, ray integration and other deep learning / 3D rendering related computational logic.

[0063] The mapping relationship between operators and online computing components can be understood here as a correspondence rule that is compatible with WebGPU computing shaders, established for the core operators of the NeRF model (regardless of whether it is the original / first / second / third NeRF model). This is used to solve the problem of insufficient original support of some deep learning operators in the NeRF model (regardless of whether it is the original / first / second / third NeRF model) by WebGPU, and to ensure that the operators can be executed normally in the online computing components.

[0064] Operators determine what computations the second NeRF model needs to perform; for example, ReLU takes the maximum value. The network structure determines the order and manner in which these computations can be performed. The integer-quantized model parameters determine the numerical values ​​used for these computations. By combining the network structure of the second NeRF model, the integer-quantized model parameters, and the operators, a mapping relationship between operators and online computing components can be constructed to translate the computational logic of the entire model into WGSL code that WebGPU can execute.

[0065] Furthermore, this application can convert the forward inference process of the second NeRF model into the target language of the online computing component, and encode the model parameters after integer quantization into binary data, so as to combine the target language and binary data to obtain the definition code of the second NeRF model in the online computing component.

[0066] Here, the target language can be understood as a code language that can be directly parsed and executed by the online computing component, such as WGSL (WebGPU Shading Language), a dedicated programming language for WebGPU computing shaders.

[0067] Encoding the model parameters after integer quantization into binary data means encoding the model parameters after integer quantization into binary format data, thereby ensuring that the data can be directly read by the WebGPU storage buffer. This method saves more memory and has higher reading efficiency compared to plaintext parameters.

[0068] By combining the target language and the binary data, the definition code of the second NeRF model in the online computing component is obtained. That is, the embodiments of this application can integrate the WGSL code of the forward inference process of the second NeRF model and the encoded parameter binary data to form complete code that can be executed by the online computing component, including but not limited to the computational logic and parameter storage definition required for the model to run in the WebGPU computing shader.

[0069] Finally, by combining the mapping relationship between the operators of the second NeRF model and the online computing component, as well as the definition code formed by integrating the WGSL code of the forward inference process of the second NeRF model and the encoded parameter binary data, the conversion of the second NeRF model to the executable format of the online computing component can be completed. This results in a complete second NeRF model file that can be directly loaded and run by the WebGPU computing shader, including but not limited to the definition code, mapping relationship and parameter data required for the second NeRF model to run. This is also the final executable file of the second NeRF model on the Web.

[0070] This application embodiment can extract the core structure, parameters, and operators of the second NeRF model and construct its mapping relationship with the WebGPU computing shader. Then, the model inference logic is converted into the shader target language and the parameters are encoded into binary data, finally generating an executable model file. While ensuring the compatibility of the deep learning operators of the NeRF model with the Web computing components, the loading and execution efficiency of the model on the browser side is optimized, ensuring that the lightweight NeRF model can run stably in the WebGPU environment, realizing the complete landing of the NeRF model from offline training products to real-time rendering on the Web side.

[0071] Optionally, in one embodiment of this application, based on the network structure, model parameters after integer quantization, and operators, a mapping relationship between operators and online computing components is constructed, including: obtaining the actual size of the operator; when the actual size is less than or equal to the target size, rewriting the operator's computational logic according to the operator, so as to construct a mapping relationship between the operator and online computing components based on the rewritten operator's computational logic; when the actual size is greater than the target size, calculating the mean parameter and / or variance parameter of the second neural radiation field model, so as to convert the operator's computational logic into linear operation logic based on the mean parameter and / or variance parameter, so as to construct a mapping relationship between the operator and online computing components based on the linear operation logic.

[0072] In some embodiments, during the process of constructing the mapping relationship between operators and online computing components based on network structure, model parameters after integer quantization, and operators, considering the different complexities of operators, this application can also obtain the actual size of the operators of the second NeRF model, and then construct the corresponding mapping relationship based on the comparison result between the actual size of the operators and the target size.

[0073] In this context, actual scale can be understood as the actual computational complexity of a single core operator (such as ReLU, batch normalization, ray integral operator, etc.) in the NeRF model. It can generally be reflected in quantitative indicators such as the number of computational steps, the number of parameters, and the frequency of memory accesses contained in the operator.

[0074] Here, the target size refers to a pre-set operator size threshold to accommodate the parallel computing characteristics of online computing components (such as WebGPU computing shaders) and browser resource constraints. It serves as a benchmark for determining whether an operator needs simplification or transformation of its computational logic. The specific target size can be determined or adjusted by those skilled in the art based on actual circumstances. The embodiments in this application are merely illustrative and do not impose specific limitations.

[0075] Specifically, when the actual size of the operator in the second NeRF model is less than or equal to the target size, embodiments of this application may, but are not limited to, rewrite the computational logic of the operator based on the operator, so as to construct the mapping relationship between the operator and the online computing component based on the computational logic of the rewritten operator.

[0076] For example, for a simple operator of the scale ReLU that calculates the maximum value of the input value and 0, this application can, but is not limited to, directly rewrite the operator's calculation logic using WGSL syntax, writing it as a WGSL function float relu(float x){return max(x, 0.0);}, so that WebGPU can execute this logic.

[0077] When the actual size of the operator in the second NeRF model is greater than the target size, embodiments of this application may, but are not limited to, pre-calculate the mean parameter and / or variance parameter of the second NeRF model, so as to convert the computation logic of the operator whose actual size is greater than the target size into linear operation logic based on the mean parameter and / or variance parameter, so as to construct the mapping relationship between the operator and the online computing component according to the linear operation logic.

[0078] Here, the mean parameter and variance parameter refer to the statistical characteristic parameters of the second NeRF model that have been calculated in advance. The mean parameter can reflect the central tendency of the data in the second NeRF model, and the variance can reflect the degree of dispersion of the data in the second NeRF model. They are the basis for converting complex operators into linear operation logic.

[0079] For example, for complex operators such as batch normalization, the embodiments of this application may not directly replicate the original logic of the second NeRF model, but pre-calculate fixed parameters such as the mean and variance obtained after training the second NeRF model, and then convert the originally complex batch normalization operation into a simple linear operation that WebGPU can execute efficiently, such as y=ax+b.

[0080] It should be noted that the final second NeRF model file contains not only a single mapping relationship between operators and online computing components, but also multiple mapping relationships between various operators and online computing components within the second NeRF model. Furthermore, the construction of different operators into computational logic executable by online computing components can be determined by those skilled in the art based on the actual situation. The embodiments in this application are merely illustrative and do not impose specific limitations.

[0081] For example, the model format conversion module in the cloud processing subsystem of this application can, but is not limited to, convert the trained second NeRF model into a format executable by online computing components (such as WebGPU computing shaders), including but not limited to deep learning operator mapping, accuracy compatibility, performance optimization, etc. The specific process can be, but is not limited to, represented as follows: (1) Model parsing: Parse the trained PyTorch model file (.pth format), extract the network structure, quantized parameters (8-bit integers) and computation logic (including ReLU activation function, batch normalization and other operators) of the second NeRF model. (2) Operator mapping: For the core operators of the second NeRF model, establish WebGPU shader compatible mapping relationships (such as simple ReLU → float relu(float x){return max(x,0.0);}, complex batch normalization → pre-calculate mean / variance and then convert to linear operation) to solve the problem of insufficient original support of WebGPU for some deep learning operators; (3) Code generation: The forward inference process of the second NeRF model is converted into WebGPU computing shader code (WGSL language), and the model parameters of the quantized second NeRF model are encoded into binary data and embedded into the storage buffer definition of the shader code.

[0082] At this point, the second NeRF model file can be generated. The number of model parameters is compressed to below 4 million, the size is ≤5MB, the loading time on a 4G network is ≤3 seconds, and the runtime memory usage is ≤80MB. The size is significantly compressed, the loading efficiency can be greatly improved, and it is highly adaptable to the resource limitations of the web client.

[0083] This application embodiment can construct the mapping relationship between operators and WebGPU by first comparing the actual size of the operators of the second NeRF model with a certain size, and then selectively choosing to directly rewrite the computational logic of the operators or convert it into linear operation logic. This not only adapts to the compatibility of WebGPU with operators of different sizes, but also reduces the computational complexity of complex operators, ensuring that all operators can be executed efficiently and stably in the online computing component on the Web side, thus solving the core problem of insufficient original support for deep learning operators and WebGPU.

[0084] Optionally, in one embodiment of this application, after converting the second neural radiation field model into an executable format of the online computing component to generate a second neural radiation field model file, the method further includes: executing the second neural radiation field model file to detect the compatibility between the second neural radiation field model file and the online computing component; and determining the second neural radiation field model file if the compatibility is that the second neural radiation field model file is compatible with the online computing component.

[0085] In some embodiments, after generating the second NeRF model file, this application may also use an online computing component to execute the second NeRF model file and test the compatibility between the second NeRF model file and the online computing component, so that if the second NeRF model file and the online computing component are compatible, it can be determined that the second NeRF model file can run normally and can be used as the final model file.

[0086] For example, after obtaining the second NeRF model file, this application can, but is not limited to, utilize the Subgroup function of WebGPU (a fine-grained parallel collaboration mechanism within a thread group provided by WebGPU (and the underlying graphics API)) to achieve data sharing within the thread group, reduce memory access overhead, and optimize the loop structure to improve parallel execution efficiency.

[0087] Finally, the embodiments of this application can perform compatibility checks on the second NeRF model file, for example, performing syntax verification and compatibility tests on various mainstream browsers to ensure that there are no runtime errors. At this point, it can be determined that the second NeRF model file is fine.

[0088] The generated WebGPU-compatible model file, i.e., the second NeRF model file, should have a size of ≤5MB, shader code lines of ≤2000, and operator mapping accuracy loss of ≤2% (which can be measured, but is not limited to, by PSNR (Peak Signal-to-Noise Ratio, used to evaluate image quality)).

[0089] This application embodiment can optimize memory access and improve parallel efficiency by leveraging WebGPU's Subgroup function after generating the second NeRF model file, and further test the compatibility of online computing components with the second NeRF model file for mainstream browsers. This ensures the stable operation of the second NeRF model file in the Web environment and further optimizes rendering performance, ensuring that the lightweight NeRF model is efficiently implemented on the Web and meets the needs of practical applications.

[0090] Step S103: Based on the online computing component and the second neural radiation field model file, render the second neural radiation field model. During the rendering process, detect the actual information of the online computing component to adjust the rendering accuracy of the second neural radiation field model according to the actual information until the rendering accuracy matches the target rendering accuracy corresponding to the actual information.

[0091] In some embodiments, after obtaining the second NeRF model file, this application can render the second NeRF model based on online computing components and the second NeRF model file. That is, it relies on online computing components such as WebGPU computing shaders to load the second NeRF model file and perform logic such as ray sampling, density / color prediction, and lighting parameter calculation, and finally generate a 3D scene visualization image on the browser to complete the rendering of the second NeRF model.

[0092] Furthermore, in order to achieve dynamic adaptation of the item rendering effect to the webpage display environment and solve the problem of insufficient realism in rendering under different ambient light conditions, this application can detect the actual information of the online computing component during the rendering process, and then adjust the rendering accuracy of the second NeRF model in the rendering process according to the actual information of the online computing component, until the rendering accuracy of the second NeRF model is adapted to the target rendering accuracy corresponding to the actual information of the online computing component.

[0093] Here, the actual information of the online computing component refers to key data about the display environment of the webpage where the online computing component is located, collected during the rendering process, such as the ambient light intensity of the webpage where the online computing component is located.

[0094] Rendering precision here refers to the visual effect metrics of the image rendered by the second NeRF model, including but not limited to image brightness, shadow contrast, and color temperature. Target rendering precision here refers to the optimal visual effect standard corresponding to the actual information of different online computing components (such as the ambient light intensity of the webpage). For example, in a low-light environment, the rendering precision corresponds to "high brightness, low contrast, and warm tones," while in a strong-light environment, it corresponds to "low brightness, high contrast, and cool tones."

[0095] Based on the target rendering precision corresponding to the actual information of the online computing component, this application can adjust the rendering precision of the second NeRF model during the rendering process until the rendering precision of the second NeRF model is adapted to the target rendering precision corresponding to the actual information of the online computing component.

[0096] It should be noted that the target rendering precision corresponding to the specific actual information can be set or adjusted by those skilled in the art according to the actual situation. The embodiments in this application are only illustrative and do not impose specific limitations.

[0097] This application embodiment can rely on online computing components to render a second NeRF model, and dynamically adjust the rendering accuracy of the second NeRF model in combination with the actual information of the webpage display environment where the online computing components are located. This achieves real-time adaptation of the rendering effect of the second NeRF model with display conditions such as ambient light, effectively solving the problem of insufficient rendering realism under different lighting scenarios and improving the user's 3D visualization experience in various webpage environments.

[0098] Optionally, in one embodiment of this application, rendering the second neural radiation field model based on the online computing component and the second neural radiation field model file includes: obtaining the user's target rendering viewpoint; calculating the viewpoint parameters corresponding to the target rendering viewpoint, so as to render the second neural radiation field model according to the viewpoint parameters.

[0099] In actual implementation, during the rendering of the second NeRF model, this application may, but is not limited to, render the second NeRF model according to the viewpoint parameters corresponding to the user's target rendering viewpoint. Here, the target rendering viewpoint can be understood as the viewpoint of the model that the user expects to render.

[0100] For example, this application can deploy a front-end rendering subsystem in the user's web browser to handle model loading, user interaction responses, and real-time rendering and display. The front-end rendering subsystem and the cloud processing subsystem can transmit data over a network, thereby enabling interaction between model files and viewpoint parameters.

[0101] In this embodiment, the front-end rendering subsystem may be developed based on the Vue 3.0 framework (a progressive JavaScript framework for building user interfaces) and integrate WebGPU-related APIs (Application Programming Interface, conforming to the W3C WebGPU standard (Web Graphics Processing Unit Standard)) to implement model loading, user interaction, and real-time rendering functions. This front-end rendering subsystem includes, but is not limited to, a model loading module, an interaction control module, a WebGPU rendering module, and an ambient light estimation module.

[0102] The model loading module is responsible for downloading the WebGPU-compatible lightweight NeRF model file (i.e., the second NeRF model file) from the cloud server and parsing and initializing it on the web frontend. The specific process can be, but is not limited to, the following: (1) Download strategy: Send model download requests through the axios (the most mainstream HTTP (Hypertext Transfer Protocol) client library) asynchronous request interface of Vue.js (a front-end framework). It can adopt, but is not limited to, chunked download (1MB chunk size) + breakpoint resume mechanism to improve the download efficiency of 5MB-level files. (2) Progress feedback: The system can monitor the download progress in real time while downloading, and display the download percentage (accuracy 1%) and the estimated remaining time on the web page; (3) Parsing and initialization: After downloading, the shader code is parsed through the GPU Shader Module (Graphics Processing Unit Shader Module) interface of WebGPU, and a GPUCompute Pipeline (Graphics Processing Unit Compute Pipeline) object is created; a storage buffer (size ≤10MB) is created through the GPU Buffer (Graphics Processing Unit Buffer) interface, and the model parameters of the lightweight second NeRF model are loaded into the GPU memory, and the initialization time is ≤500ms.

[0103] The interactive control module is mainly used to respond to user mouse, touch and other operations, obtain rendering view parameters specified by the user, and realize low-latency interaction. The specific process can be, but is not limited to, represented as follows: (1) Event listening: Listen for events such as mousemove, wheel, and touchmove on the webpage. The event response delay should be ≤10ms. (2) Viewpoint calculation: Calculate the viewpoint parameters (azimuth: 0°~360°, elevation: -30°~30°, scaling ratio: 0.5x~2x) based on user operation. Among them, moving the mouse 10 pixels corresponds to a change of 1° in azimuth / elevation, and scrolling the scroll wheel 1 unit corresponds to a change of 0.1x in scaling ratio. (3) Smoothing: The exponential moving average algorithm (α=0.1) is used to smooth the viewpoint parameters to avoid abrupt viewpoint switching and improve the smoothness of interaction.

[0104] This application embodiment can be based on a lightweight second NeRF model to freely render from any desired angle by the user. No professional 3D modeling is required. Model training can be completed by collecting only 72 multi-angle images. The production cost is ≤8 hours / piece. It can accurately restore the spatial structure, material details and lighting effects of the object, and can ensure that the rendering result has an SSIM (Structural Similarity Index Measure, which can be used to evaluate image similarity) value ≥0.90 and a PSNR value ≥35dB, thereby effectively ensuring the realism of the static mesh model.

[0105] Optionally, in one embodiment of this application, rendering the second neural radiation field model based on the online computing component and the second neural radiation field model file includes: obtaining the ray sampling parameters corresponding to the viewpoint parameters in the second neural radiation field model; performing parallel sampling on each ray of the target rendering viewpoint based on the ray sampling parameters to obtain multiple sampling points and their spatial positions; calculating the radiation intensity and color value of each ray based on the spatial positions of the multiple sampling points, and mapping the radiation intensity and color value of each ray to the corresponding screen pixel position in the second neural radiation field model to generate a rendered image of the target rendering viewpoint, thereby completing the rendering of the second neural radiation field model.

[0106] Those skilled in the art will understand that the rendering process of the NeRF model requires a large amount of ray sampling and neural network inference calculations. The original NeRF model has a rendering frame rate of far less than 1 FPS on ordinary consumer-grade GPUs (usually 0.3~0.8 frames / second), and a single viewpoint switch requires a wait of 1~3 seconds, which is completely unable to meet the needs of real-time interaction on the web (usually requiring ≥24 frames / second).

[0107] Based on this, in some embodiments, when rendering the second NeRF model, this application can first obtain the ray sampling parameters corresponding to the viewpoint parameters in the second NeRF model based on the viewpoint parameters corresponding to the user's target rendering viewpoint.

[0108] By using the ray sampling parameters corresponding to the viewpoint parameters of each rendering viewpoint, this application can sample each ray of each rendering viewpoint in parallel, and obtain multiple sampling points and the spatial positions (spatial coordinates) corresponding to the multiple sampling points.

[0109] Furthermore, based on the spatial coordinates of each sampling point, this application can calculate the radiation intensity and color value of each ray, and map the radiation intensity and color value of each ray to the corresponding screen pixel position in the second NeRF model, thereby generating the rendered image of the user from the target rendering perspective, which completes the rendering of the second NeRF model.

[0110] For example, the WebGPU rendering module in the front-end rendering subsystem can leverage the parallel computing capabilities of WebGPU to achieve real-time inference rendering of the second NeRF model. The specific implementation process can be represented as follows: (1) Viewpoint parameter transmission: The viewpoint parameters (such as azimuth, elevation, and scaling) of the user's target rendering viewpoint obtained by the interactive control module are converted into the ray sampling parameters (ray origin and direction vector) required by the NeRF model and transmitted to the computation shader through the GPU Buffer in WebGPU. The data transmission delay is ≤2ms. (2) Parallel ray sampling: In the WebGPU computing shader, computing tasks are allocated according to a 16×16 thread group size, and each ray under the current rendering view is sampled in parallel (32 / 64 / 128 sampling points, etc., which can be set or adjusted by professionals in this field according to the actual situation. This application embodiment is only an example and does not impose specific limitations). The sampling interval can be adaptively adjusted based on the spatial density learned by the second NeRF model. (3) Neural network inference: Based on the spatial coordinates of the sampling points, the radiation intensity and color value of the light are calculated by the second NeRF model embedded in the WebGPU computing shader. This process can utilize the parallel computing architecture of WebGPU to process the inference calculation of ≥1000 light rays at the same time. (4) Pixel color synthesis: Using the volume rendering integral formula, the calculation result of each ray is mapped to the corresponding screen pixel position in the second NeRF model (default rendering resolution 1920×1080), synthesizing and generating the object rendering image under the current viewpoint, and outputting it to the Canvas element of the webpage through the WebGPU rendering channel (such as GPU Color Target State (Graphics Processing Unit Color Target State)). The single frame rendering time is ≤33ms (corresponding to a frame rate ≥30FPS (frames per second), and the number of frames transmitted per second ≥30 frames).

[0111] This application embodiment can leverage the parallel computing capabilities of WebGPU to efficiently process the entire process from viewpoint parameter conversion to light sampling, radiation intensity and color value calculation, and pixel synthesis output, effectively ensuring the real-time performance and image accuracy of the second NeRF model rendering; and the rendering frame rate can be stably maintained at over 30 FPS (for low- to mid-range devices), reaching up to 130 FPS (for high-end devices), with a viewpoint switching latency of ≤50ms, and the interaction response speed can be improved by 37 to 100 times compared to the original model, thereby effectively meeting the real-time interaction requirements of the Web client.

[0112] Optionally, in one embodiment of this application, after rendering the second neural radiation field model, the method further includes: acquiring ambient light information of the device where the online computing component is located, or the image pixel brightness of the device; and adjusting the rendering parameters of the second neural radiation field model to meet the target rendering requirements based on the pre-built mapping relationship between ambient light intensity and rendering parameters of the second neural radiation field model.

[0113] Based on the descriptions in other embodiments, it can be understood that this application can adjust the rendering accuracy of the second NeRF model to match the target rendering accuracy corresponding to its actual information based on the actual information of the online computing component (such as the ambient light intensity of the webpage).

[0114] In actual execution, after rendering the second NeRF model, this application can also collect the ambient light information of the device where the online computing component is located, or the image pixel brightness of the device, and adjust the rendering parameters of the second NeRF model to meet the target rendering requirements based on the pre-built mapping relationship between the ambient light intensity and the rendering parameters of the second NeRF model.

[0115] In this context, the device where the online computing component is located can be understood as the terminal device of the web application when the user opens the second NeRF model. The ambient light information of the device refers to the ambient light intensity data obtained through the device's built-in sensors and some interfaces. Image pixel brightness can be understood as the average pixel brightness (range 0-255) of the background image of the webpage where the online computing component is located, captured by the Canvas element when the device does not support an ambient light sensor; this can serve as a substitute evaluation indicator for ambient light intensity.

[0116] For example, the ambient light estimation module in the front-end rendering subsystem can first obtain the device's ambient light sensor data (unit: lux) through the Web API (AmbientLight Sensor interface). If the device does not support the sensor, it can capture the background image of the webpage through Canvas and calculate the average brightness of the background pixels (0~255) as a substitute value for the ambient light intensity.

[0117] The pre-built mapping relationship between ambient light intensity and the rendering parameters of the second NeRF model can be understood here as a pre-defined correspondence rule between ambient light intensity ranges and the rendering parameters of the second NeRF model. For example, a low-light environment corresponds to high brightness, low contrast, and warm color parameters, while a strong-light environment corresponds to low brightness, high contrast, and cool color parameters, and so on. Here, the rendering parameters can be understood as the key parameters that affect the visual effect of the (second) NeRF model's rendered image, including but not limited to three core dimensions: overall image brightness, shadow contrast, and color temperature.

[0118] The target rendering requirement can be understood here as matching the optimal visual effect standard through mapping relationships based on different ambient light intensities (ambient light information / image pixel brightness). In other words, the brightness, contrast, and color temperature of the rendered image can be adapted to the lighting conditions around the device, thereby improving the realism of the rendering.

[0119] This application embodiment can dynamically adjust the rendering parameters of the second NeRF model by flexibly collecting ambient light information or image pixel brightness of the device and combining it with the preset mapping relationship between lighting and rendering parameters. This allows the rendering effect to accurately adapt to the lighting environment around the device, improve the realism of the picture, and the adjustment process is imperceptible, ensuring a smooth user visualization experience.

[0120] Optionally, in one embodiment of this application, before adjusting the rendering parameters of the second neural radiation field model to meet the target rendering requirements based on the pre-built mapping relationship between ambient light intensity and the rendering parameters of the second neural radiation field model, the method further includes: determining the mapping relationship between multiple levels of ambient light intensity and image brightness, shadow contrast, and color temperature based on the target rendering requirements; and determining the mapping relationship between the pre-built ambient light intensity and the rendering parameters of the second neural radiation field model by combining the mapping relationship between multiple levels of ambient light intensity and image brightness, shadow contrast, and color temperature.

[0121] In some embodiments, this application may, but is not limited to, determine the mapping relationship between pre-built ambient light intensity and the rendering parameters of the second NeRF model based on the mapping relationship between multiple levels of ambient light intensity and image brightness, shadow contrast, and color temperature, according to the target rendering requirements. That is, this application may determine the mapping relationship between pre-built ambient light intensity and the rendering parameters of the second NeRF model according to the mapping relationship between different levels of ambient light intensity and image brightness, shadow contrast, and color temperature.

[0122] For example, when the ambient light intensity is ≤50 lux (weak light), the mapping relationship between the ambient light intensity and the rendering parameters of the second NeRF model can be set to, but is not limited to: increase the overall brightness of the rendered image by 15%, reduce the shadow contrast by 20%, and adjust the color temperature to a warm tone (5000K). When the ambient light intensity is 50~500 lux (normal light), the mapping relationship between the ambient light intensity and the rendering parameters of the second NeRF model can be set, but is not limited to: maintaining the default lighting parameters (color temperature 5500K, shadow contrast 1.0). When the ambient light intensity is ≥500 lux (strong light), the mapping relationship between the ambient light intensity and the rendering parameters of the second NeRF model can be set to, but is not limited to: reduce the overall brightness of the rendered image by 10%, increase the shadow contrast by 30%, and adjust the color temperature to a cool tone (6000K). Furthermore, in this embodiment of the application, ambient light data (ambient light intensity) may be updated every 500ms, thereby ensuring the quality of the rendered image by adjusting the rendering parameters (frame rate fluctuation ≤2FPS) in real time.

[0123] This application embodiment can determine the mapping relationship between the pre-built ambient light intensity and the rendering parameters of the second NeRF model based on the mapping relationship between different levels of ambient light intensity and image brightness, shadow contrast, and color temperature, thereby enabling the rendering effect of the item to be dynamically adapted to the web page display environment and solving the problem of insufficient realism in rendering under different ambient light conditions.

[0124] Optionally, in one embodiment of this application, after rendering the second neural radiation field model, the method further includes: collecting the average frame rate calculation result of the device where the online computing component is located and the bandwidth information of the device; and adjusting the rendering accuracy of the second neural radiation field model to meet the target accuracy requirements based on the average frame rate calculation result and the bandwidth information.

[0125] In some embodiments, after rendering the second NeRF model, considering the differences in user devices, in order to ensure that the rendering effect can be adapted to different types of devices, this application can also collect the average frame rate calculation result of the device where the online computing component is located and the bandwidth information of the device to adjust the rendering accuracy of the second NeRF model to meet the target accuracy requirements.

[0126] Here, the target accuracy requirement can be understood as the optimal accuracy corresponding to the average frame rate calculation result of the device on which the current online computing component is located and the bandwidth information of the device.

[0127] The embodiments of this application can dynamically adjust the rendering precision after rendering the second NeRF model by combining the average frame rate and bandwidth information of the user device. This allows the model rendering effect to be adapted to terminal devices with different performance levels, ensuring smooth operation of low-performance devices while enabling high-performance devices to achieve better picture performance and improving the user experience across all devices.

[0128] Optionally, in one embodiment of this application, adjusting the rendering accuracy of the second neural radiation field model to meet the target accuracy requirements based on the average frame rate calculation result and bandwidth information includes: determining the actual performance of the device based on the average frame rate calculation result and bandwidth information; adjusting the number of light sampling points, rendering resolution, peak signal-to-noise ratio, and image frame rate of the second neural radiation field model according to the actual performance until the rendering accuracy of the second neural radiation field model meets the target accuracy requirements.

[0129] In some embodiments, when adjusting the rendering accuracy of the second NeRF model to meet certain accuracy requirements, this application may, but is not limited to, first determine the actual performance of the device based on the average frame rate calculation result of the device where the online computing component is located and the bandwidth information of the device, and then adjust the number of ray sampling points, rendering resolution, peak signal-to-noise ratio, and image frame rate of the lightweight NeRF model according to the actual performance of the device, until the rendering accuracy of the lightweight NeRF model meets certain accuracy requirements.

[0130] For example, this application may add a model caching module and a rendering precision adaptive adjustment module to the front-end rendering subsystem.

[0131] The model caching module can be implemented based on the browser's Indexed DB (Indexed Database API, an indexed database API), and the specific process can be represented as follows: (1) Cache storage: When a user visits the item display page for the first time, after the online calculation component downloads and parses the second NeRF model, it can store information such as shader code, quantization parameter data, and model version number in Indexed DB. The cache space occupied by a single model is ≤6MB. (2) Cache verification: When a user visits the item page again, the model loading module first checks whether there is a valid model in the local cache (by comparing the model version number with the cloud). If the cache is valid, it will be loaded directly from the local cache. (3) Cache update: When the second NeRF model in the cloud is updated (version number changes), the local cache will be automatically invalidated and the latest model file will be downloaded again; (4) Cache cleanup: Set the cache expiration time (usually 30 days by default). Model caches that have not been accessed after the expiration period will be automatically cleaned up; a manual cache cleanup interface is provided for users to manage themselves.

[0132] Adaptive rendering precision adjustment module: Dynamically adjusts rendering precision parameters based on the user's device performance and network conditions to ensure a smooth experience across different devices. The specific process can be, but is not limited to, the following: (1) Performance testing: Device performance evaluation: Obtain the device GPU model through the WebGPU GPU Adapter (the logical object representation of the physical graphics processor or software backend in the WebGPU API), run a 100-frame benchmark test (fixed viewpoint rendering), and calculate the average frame rate as the performance score P; Network bandwidth test: The actual transmission rate is calculated as network bandwidth B (unit: Mbps) by downloading a 1MB test file. (2) Adjustment strategy: High-performance scenes (P≥60FPS and B≥10Mbps): The number of light sampling points is adjusted to 128, the rendering resolution is increased to 2560×1440, the PSNR value is ≥38dB, and the detail performance is excellent. Balanced scene (30FPS≤P<60FPS and 5Mbps≤B<10Mbps): The number of light sampling points is kept at 64, the rendering resolution is 1920×1080, the PSNR value is ≥35dB, balancing accuracy and performance. Basic scene (P < 30 FPS or B < 5 Mbps): The number of light sampling points is adjusted to 32, and the rendering resolution is reduced to 1280×720 to ensure a frame rate of ≥ 30 FPS, with the goal of prioritizing smoothness. (3) Dynamic switching: The device performance and network bandwidth are reassessed every 2 seconds, and the rendering parameters are automatically switched according to the assessment results. The switching process adopts a gradual transition to avoid sudden changes in the screen.

[0133] This application embodiment can ensure a smooth display experience on devices with different performance and under different network environments by adaptively adjusting model caching and rendering accuracy, and limiting the loading time of secondary access to ≤300ms. Combined with the ambient light adaptive function, it can further enhance the realism of model rendering in different scenarios, ensure the stable operation of the model on different user devices, support direct access by mainstream browsers, require no additional software installation, is easy to deploy, and is compatible with all scenarios from high-end PCs (Personal Computers) to ordinary mobile phones.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0135] Embodiments of this application also provide a rendering apparatus for a neural radiation field model, such as... Figure 2 As shown, the rendering device for the neural radiation field model includes: a lightweight module 100, a conversion module 200, and a rendering module 300.

[0136] The lightweight module 100 is used to simplify the number of network layers of the original neural radiation field model to the target number of layers that are compatible with the lightweight requirements, and insert separable convolutional layers to obtain a first neural radiation field model. The model parameters of the first neural radiation field model are then subjected to integer quantization to obtain a second neural radiation field model that meets the preset byte requirements.

[0137] The conversion module 200 is used to convert the second neural radiation field model into an executable format of the online computing component in response to the fact that multiple indicators of the second neural radiation field model meet the corresponding requirements, so as to generate the second neural radiation field model file.

[0138] The rendering module 300 is used to render the second neural radiation field model based on the online computing component and the second neural radiation field model file. During the rendering process, the actual information of the online computing component is detected to adjust the rendering accuracy of the second neural radiation field model according to the actual information until the rendering accuracy matches the target rendering accuracy corresponding to the actual information.

[0139] Optionally, in one embodiment of this application, the lightweight module 100 includes: a simplification unit, used to simplify the fully connected layers of the original neural radiation field model to the target number of layers to obtain a third neural radiation field model; and an insertion unit, used to insert separable convolutional layers to replace the fully connected layers in the fully connected layers of the third neural radiation field model based on the target number of layers and the simplification requirements of the original neural radiation field model to obtain a first neural radiation field model.

[0140] Optionally, in one embodiment of this application, it further includes: a training module, configured to train the second neural radiation field model based on batch size information, learning rate, iteration decay, and target iteration number before multiple indicators of the second neural radiation field model meet the corresponding requirements, so as to obtain the number of parameters, model volume, and training time of the trained second neural radiation field model; and a first determination module, configured to determine multiple indicators based on the number of parameters, model volume, and training time.

[0141] Optionally, in one embodiment of this application, the conversion module 200 includes: an extraction unit, used to extract the network structure, integer-quantized model parameters, and operators of the second neural radiation field model, so as to construct a mapping relationship between the operators and the online computing component based on the network structure, integer-quantized model parameters, and operators; an encoding unit, used to convert the forward inference process of the second neural radiation field model into the target language of the online computing component, and to encode the integer-quantized model parameters into binary data, so as to combine the target language and binary data to obtain the definition code of the second neural radiation field model in the online computing component; and a conversion unit, used to combine the mapping relationship and the definition code to complete the conversion of the second neural radiation field model into the executable format of the online computing component to obtain the second neural radiation field model file.

[0142] Optionally, in one embodiment of this application, the extraction unit includes: an acquisition subunit for acquiring the actual size of the operator; a first construction subunit for rewriting the computational logic of the operator according to the operator when the actual size is less than or equal to the target size, so as to construct a mapping relationship between the operator and the online computing component according to the rewritten computational logic of the operator; and a second construction subunit for calculating the mean parameter and / or variance parameter of the second neural radiation field model when the actual size is greater than the target size, so as to convert the computational logic of the operator into linear operation logic based on the mean parameter and / or variance parameter, so as to construct a mapping relationship between the operator and the online computing component according to the linear operation logic.

[0143] Optionally, in one embodiment of this application, it further includes: an execution module, configured to execute the second neural radiation field model file after converting the second neural radiation field model into an executable format of the online computing component to generate the second neural radiation field model file, to detect the compatibility between the second neural radiation field model file and the online computing component; and a second determination module, configured to determine the second neural radiation field model file if the compatibility is that the second neural radiation field model file is compatible with the online computing component.

[0144] Optionally, in one embodiment of this application, the rendering module 300 includes: a first acquisition unit, used to acquire the user's target rendering viewpoint; a calculation unit, used to calculate the viewpoint parameters corresponding to the target rendering viewpoint and acquire the ray sampling parameters corresponding to the viewpoint parameters in the second neural radiation field model; a sampling unit, used to perform parallel sampling on each ray of the target rendering viewpoint based on the ray sampling parameters to obtain multiple sampling points and their spatial positions; and a mapping unit, used to calculate the radiation intensity and color value of each ray based on the spatial positions of the multiple sampling points, and map the radiation intensity and color value of each ray to the corresponding screen pixel position in the second neural radiation field model to generate a rendered image of the target rendering viewpoint and complete the rendering of the second neural radiation field model.

[0145] Optionally, in one embodiment of this application, it further includes: a third determining module, used to determine the mapping relationship between multiple levels of ambient light intensity and image brightness, shadow contrast, and color temperature based on the target rendering requirements after rendering the second neural radiation field model; a fourth determining module, used to determine the mapping relationship between ambient light intensity and rendering parameters of the second neural radiation field model by combining the mapping relationship between multiple levels of ambient light intensity and image brightness, shadow contrast, and color temperature; and a first adjusting module, used to collect ambient light information of the device where the online computing component is located, or the image pixel brightness of the device, so as to adjust the rendering parameters of the second neural radiation field model to meet the target rendering requirements based on the mapping relationship and combining the ambient light information and image pixel brightness.

[0146] Optionally, in one embodiment of this application, it further includes: a second acquisition module, used to acquire the average frame rate calculation result of the device where the online computing component is located and the bandwidth information of the device after rendering the second neural radiation field model; a fifth determination module, used to determine the actual performance of the device based on the average frame rate calculation result and the bandwidth information; and a second adjustment module, used to adjust the number of light sampling points, rendering resolution, peak signal-to-noise ratio, and image frame rate of the second neural radiation field model according to the actual performance, until the rendering accuracy of the second neural radiation field model meets the target accuracy requirements.

[0147] For a description of the features in the embodiment corresponding to the rendering device of the neural radiation field model, please refer to the relevant description of the embodiment corresponding to the rendering method of the neural radiation field model, which will not be repeated here.

[0148] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in the rendering method embodiments of any of the above-described neural radiation field models.

[0149] Embodiments of this application also provide a non-volatile computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the rendering method embodiments of any of the above-described neural radiation field models when running.

[0150] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0151] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the rendering method embodiments of any of the above-described neural radiation field models.

[0152] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the rendering method embodiments of any of the above-described neural radiation field models.

[0153] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0154] The rendering method for a neural radiation field model provided in this application has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A rendering method for a neural radiation field model, characterized in that, Includes the following steps: The number of network layers in the original neural radiation field model is simplified to the target number of layers that are suitable for lightweight requirements, and separable convolutional layers are inserted to obtain the first neural radiation field model. The model parameters of the first neural radiation field model are then subjected to integer quantization to obtain the second neural radiation field model that meets the preset byte requirements. In response to the fact that multiple indicators of the second neural radiation field model meet the corresponding requirements, the second neural radiation field model is converted into an executable format of the online computing component to generate the second neural radiation field model file; Based on the online computing component and the second neural radiation field model file, the second neural radiation field model is rendered. During the rendering process, the actual information of the online computing component is detected to adjust the rendering accuracy of the second neural radiation field model according to the actual information until the rendering accuracy matches the target rendering accuracy corresponding to the actual information.

2. The method according to claim 1, characterized in that, The process of simplifying the network layer count of the original neural radiation field model to a target number that suits lightweight requirements, and inserting separable convolutional layers to obtain the first neural radiation field model, includes: The fully connected layers of the original neural radiation field model are simplified to the target number of layers to obtain the third neural radiation field model; Based on the target number of layers and the simplification requirements of the original neural radiation field model, a separable convolutional layer is inserted into the fully connected layer of the third neural radiation field model to replace the fully connected layer, so as to obtain the first neural radiation field model.

3. The method according to claim 1, characterized in that, Before the plurality of indices of the second neural radiation field model meet the corresponding requirements, the method further includes: The second neural radiation field model is trained based on batch size information, learning rate, iteration decay, and target number of iterations to obtain the number of parameters, model volume, and training time of the trained second neural radiation field model. The multiple metrics are determined based on the number of parameters, the model volume, and the training time.

4. The method according to claim 1, characterized in that, The step of converting the second neural radiation field model into an executable format of an online computing component to generate a second neural radiation field model file includes: The network structure of the second neural radiation field model, the model parameters after integer quantization, and the operators are extracted, and a mapping relationship between the operators and the online computing component is constructed based on the network structure, the model parameters after integer quantization, and the operators. The forward inference process of the second neural radiation field model is converted into the target language of the online computing component, and the model parameters after integer quantization are encoded into binary data. The target language and the binary data are combined to obtain the definition code of the second neural radiation field model in the online computing component. By combining the mapping relationship and the defined code, the second neural radiation field model is converted into the executable format of the online computing component to obtain the second neural radiation field model file.

5. The method according to claim 4, characterized in that, The step of constructing a mapping relationship between the operator and the online computing component based on the network structure, the model parameters after integer quantization, and the operator includes: Obtain the actual size of the operator; When the actual scale is less than or equal to the target scale, the computation logic of the operator is rewritten according to the operator, so as to construct the mapping relationship between the operator and the online computing component based on the rewritten computation logic of the operator; When the actual scale is greater than the target scale, the mean parameter and / or variance parameter of the second neural radiation field model are calculated, and the computational logic of the operator is converted into linear operation logic based on the mean parameter and / or the variance parameter, so as to construct the mapping relationship between the operator and the online computing component according to the linear operation logic.

6. The method according to claim 1, characterized in that, After converting the second neural radiation field model into the executable format of the online computing component to generate the second neural radiation field model file, the method further includes: The second neural radiation field model file is executed to test its compatibility with the online computing component; If the compatibility condition is that the second neural radiation field model file is compatible with the online computing component, then the second neural radiation field model file is determined.

7. The method according to claim 1, characterized in that, The step of rendering the second neural radiation field model based on the online computing component and the second neural radiation field model file includes: Obtain the user's target rendering perspective; Calculate the view parameters corresponding to the target rendering view, and obtain the ray sampling parameters corresponding to the view parameters in the second neural radiation field model; Based on the light sampling parameters, each ray of light from the target rendering viewpoint is sampled in parallel to obtain multiple sampling points and their spatial positions; Based on the spatial location of the multiple sampling points, the radiation intensity and color value of each ray are calculated, and the radiation intensity and color value of each ray are mapped to the corresponding screen pixel position in the second neural radiation field model to generate a rendered image of the target rendering viewpoint, thus completing the rendering of the second neural radiation field model.

8. The method according to claim 1, characterized in that, After rendering the second neural radiation field model, the following steps are also included: Based on the target rendering requirements, the mapping relationship between multiple levels of ambient light intensity and image brightness, shadow contrast, and color temperature is determined respectively; By combining the mapping relationship between the multiple levels of ambient light intensity and the image brightness, the shadow contrast, and the color temperature, the mapping relationship between the ambient light intensity and the rendering parameters of the second neural radiation field model is determined; The ambient light information of the device where the online computing component is located, or the image pixel brightness of the device, is collected. Based on the mapping relationship, and in combination with the ambient light information and the image pixel brightness, the rendering parameters of the second neural radiation field model are adjusted to meet the target rendering requirements.

9. The method according to claim 1, characterized in that, After rendering the second neural radiation field model, the following steps are also included: Collect the average frame rate calculation results of the device where the online computing component is located, as well as the bandwidth information of the device. Based on the average frame rate calculation result and the bandwidth information, the actual performance of the device is determined; Adjust the number of light sampling points, rendering resolution, peak signal-to-noise ratio, and image frame rate of the second neural radiation field model according to the actual performance until the rendering accuracy of the second neural radiation field model meets the target accuracy requirements.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the rendering method for the neural radiation field model as described in any one of claims 1 to 9 when executing the computer program.

Images