Method, device and storage medium for processing explicit radiation field

Abstract

The application provides a processing method, device and storage medium of an explicit radiance field, the method comprising: acquiring a plurality of images of a plurality of perspectives of a target scene; acquiring an explicit radiance field model comprising a first voxel density grid and a first color grid trained using the plurality of images, the first voxel density grid comprising voxel density information of a plurality of spatial points in the target scene, and the first color grid comprising color information of the plurality of spatial points; and compressing the first voxel density grid and the first color grid using a first compression rate and a second compression rate respectively to obtain a second voxel density grid and a second color grid, the first compression rate being smaller than the second compression rate. The present scheme can reduce the occupation of storage space and ensure the rendering accuracy by compressing the voxel density grid and the color grid in the explicit radiance field model that can be used for a new perspective synthesis task through a non-uniform compression method, and can be used in the field of virtual reality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a method and device for processing explicit radiance field and a storage medium. BACKGROUND

[0002] At present, a neural radiance field (NeRF) algorithm is popular and is widely applied to a novel view synthesis task. The NeRF algorithm is actually an implicit radiance field algorithm. The NeRF algorithm can be simply summarized as follows: a 3D scene is implicitly modeled by using a multi-layer perceptron (MLP) neural network. Specifically, a plurality of pictures of the same object are taken at different angles in a scene. Based on the pictures taken at different angles and known camera parameters, the neural network model is trained. Then, based on the trained neural network model, a picture of the object at an arbitrary angle can be rendered.

[0003] In the NeRF algorithm, after a plurality of pictures at different angles are taken, a plurality of space points are sampled along a camera ray. A 5D vector (x, y, z, θ, φ) is formed by the coordinates of each space point and the angle direction corresponding to the camera ray, where (x, y, z) is the position coordinates of the space point, and (θ, φ) is the angle direction. The output is the color (c) and the voxel density (σ) corresponding to the space point. Then, the color and the voxel density of each space point on the ray are rendered by using a volume rendering equation.

[0004] The above implicit NeRF method needs to sample a large number of points on each camera ray, input the points into a large neural network for calculation, and then perform volume rendering based on the calculation result. Therefore, the method has problems of slow training speed and long rendering time.

[0005] To overcome the above problems, an explicit (Explicit) radiation field algorithm is provided, such as Plenoxels (Radiance Fields without Neural Network) which does not use a neural network. The explicit method improves the training speed to several minutes and the rendering time to almost real-time compared to the original implicit NeRF method which generally takes several days. However, this explicit radiation field model improves the storage space occupation in exchange for the improvement of training time and rendering time. For example, the Plenoxels algorithm represents the scene as a sparse 3D grid, and directly stores the voxel density and color information (color information is represented by spherical harmonic coefficients) of a plurality of discrete spatial points in the 3D grid. At this time, the color and voxel density of each sampling point of a camera ray are calculated by trilinear interpolation of the color and voxel density of the eight adjacent spatial points in the 3D grid obtained by training.

[0006] Obviously, the speed of directly accessing the 3D grid is much faster than the neural network inference, which is why the explicit radiation field algorithm can greatly exceed the implicit NeRF algorithm in training and rendering speed. However, this method of saving a complete 3D grid also greatly increases the storage space. The storage space occupied by the above 3D grid is much larger than the implicit NeRF network model. Therefore, if the rendering speed of the explicit radiation field algorithm is maintained while reducing the storage space occupation, it is an urgent problem to be solved. SUMMARY

[0007] Embodiments of the present application provide a kind of explicit radiation field processing method, device, equipment and storage medium, by non-uniform compression reduces the storage space occupation of explicit radiation field model.

[0008] In a first aspect, embodiments of the present application provide a kind of explicit radiation field processing method, the method comprises:

[0009] Obtain a plurality of images of a target scene shot under a plurality of perspectives;

[0010] Obtain an explicit radiation field model corresponding to the target scene trained using the plurality of images, the explicit radiation field model includes a first voxel density grid and a first color grid, the first voxel density grid includes voxel density information of a plurality of spatial points in the target scene, and the first color grid includes color information of the plurality of spatial points;

[0011] compress the first voxel density grid using a first compression rate to obtain a second voxel density grid; and compress the first color grid using a second compression rate to obtain a second color grid; wherein the first compression rate is less than the second compression rate.

[0012] In a second aspect, an embodiment of the present application provides a device for processing an explicit radiance field, the device comprising:

[0013] an image obtaining module configured to obtain a plurality of images captured from a plurality of perspectives of a target scene;

[0014] a grid obtaining module configured to obtain an explicit radiance field model corresponding to the target scene trained using the plurality of images, the explicit radiance field model comprising a first voxel density grid and a first color grid, the first voxel density grid comprising voxel density information of a plurality of spatial points in the target scene, and the first color grid comprising color information of the plurality of spatial points;

[0015] a compression module configured to compress the first voxel density grid using a first compression rate to obtain a second voxel density grid; and compress the first color grid using a second compression rate to obtain a second color grid; wherein the first compression rate is less than the second compression rate.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the processing method of the explicit radiance field according to the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a non-transitory machine readable storage medium, the non-transitory machine readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor performs the processing method of the explicit radiance field according to the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a processing method of an explicit radiance field, the method comprising:

[0019] receiving a request triggered by calling a set service by a terminal device, the request comprising a plurality of images captured from a plurality of perspectives of a target scene;

[0020] performing the following steps by using a processing resource corresponding to the set service:

[0021] acquire an explicit radiance field model corresponding to the target scene trained using the plurality of images, the explicit radiance field model including a first voxel density grid and a first color grid, the first voxel density grid including voxel density information of a plurality of spatial points in the target scene, and the first color grid including color information of the plurality of spatial points;

[0022] compress the first voxel density grid using a first compression rate to obtain a second voxel density grid, and compress the first color grid using a second compression rate to obtain a second color grid, wherein the first compression rate is less than the second compression rate;

[0023] send the second voxel density grid and the second color grid to the terminal device for storage.

[0024] In a sixth aspect, an embodiment of the present application provides a processing method of an explicit radiance field, applied to an extended reality device, and the method comprises:

[0025] display a plurality of images taken at a plurality of viewing angles of a target scene;

[0026] acquire an explicit radiance field model corresponding to the target scene trained using the plurality of images, the explicit radiance field model including a first voxel density grid and a first color grid, the first voxel density grid including voxel density information of a plurality of spatial points in the target scene, and the first color grid including color information of the plurality of spatial points;

[0027] compress the first voxel density grid using a first compression rate to obtain a second voxel density grid, and compress the first color grid using a second compression rate to obtain a second color grid, wherein the first compression rate is less than the second compression rate;

[0028] receive an input new viewing angle;

[0029] perform up-sampling processing on the second voxel density grid and the second color grid based on the first compression rate and the second compression rate respectively to obtain a third voxel density grid and a third color grid;

[0030] generate an image corresponding to the new viewing angle according to the third voxel density grid and the third color grid;

[0031] display the image corresponding to the new viewing angle.

[0032] In the compression scheme of the explicit radiation field model provided in this embodiment of the invention, the explicit radiation field model composed of a first voxel density grid and a first color grid is compressed using a non-uniform compression method to reduce the storage space occupied by the explicit radiation field model and ensure the rendering accuracy when performing new perspective compositing tasks based on the compressed explicit radiation field model. Non-uniform compression refers to using different compression ratios for the first voxel density grid and the first color grid. Specifically, the first voxel density grid is compressed using a first compression ratio to obtain a second voxel density grid, and the first color grid is compressed using a second compression ratio to obtain a second color grid. Research has shown that the first voxel density grid occupies relatively little storage space, while the voxel density information it stores is more important for voxel rendering accuracy; therefore, a smaller first compression ratio is used to compress it. The first color grid occupies more storage space and has greater redundancy; therefore, a larger second compression ratio is used to compress it. This non-uniform compression process not only reduces the storage space occupied by the explicit radiation field model but also ensures the accuracy of voxel rendering based on the compressed explicit radiation field model. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 A flowchart illustrating a method for processing an explicit radiation field, provided as an embodiment of the present invention;

[0035] Figure 2 A schematic diagram illustrating the training process of an explicit radiation field model provided in an embodiment of the present invention;

[0036] Figure 3 A schematic diagram illustrating the application of an explicit radiation field processing method provided in an embodiment of the present invention;

[0037] Figure 4 A flowchart illustrating a method for processing an explicit radiation field, provided as an embodiment of the present invention;

[0038] Figure 5 A flowchart illustrating an important spatial point determination method provided in an embodiment of the present invention;

[0039] Figure 6 This is a schematic diagram illustrating the application of an important spatial point determination method provided in an embodiment of the present invention;

[0040] Figure 7A flowchart of a high-frequency detail recovery network model training method provided by an embodiment of the present application is shown in the figure.

[0041] Figure 8 A composition schematic diagram of a high-frequency detail recovery network model provided by an embodiment of the present application is shown in the figure.

[0042] Figure 9 A flowchart of a method for processing an explicit radiation field provided by an embodiment of the present application is shown in the figure.

[0043] Figure 10 An application schematic diagram of a method for processing an explicit radiation field provided by an embodiment of the present application is shown in the figure.

[0044] Figure 11 A structural schematic diagram of a processing device for an explicit radiation field provided by an embodiment of the present application is shown in the figure.

[0045] Figure 12 A structural schematic diagram of an electronic device provided by the embodiment is shown in the figure. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0047] In addition, the step timing in each of the following method embodiments is only an example, not a strict limitation.

[0048] Figure 1 A flowchart of a method for processing an explicit radiation field provided by an embodiment of the present application is shown in the figure. Figure 1 As shown in the figure, the method comprises the following steps:

[0049] 101. Obtain multiple images corresponding to multiple viewing angles of a target scene.

[0050] 102. Obtain an explicit radiation field model corresponding to the target scene trained using the multiple images, wherein the explicit radiation field model comprises a first voxel density grid and a first color grid, the first voxel density grid comprises voxel density information of multiple spatial points in the target scene, and the first color grid comprises color information of the multiple spatial points.

[0051] 103. Compress the first voxel density grid using a first compression rate to obtain a second voxel density grid; and compress the first color grid using a second compression rate to obtain a second color grid; wherein the first compression rate is less than the second compression rate.

[0052] In the embodiments of the present application, the explicit radiance field method refers to a method of using a radiance field method to model a target scene in a display expression, and the modeling result is referred to as an explicit radiance field model. In actual applications, the display radiance field method may include, for example, DVGO (Direct Voxel Grid Optimization), plenoxels (an abbreviation of plenoptic voxels), and the like.

[0053] When the explicit radiance field method is used to model a target scene, an explicit radiance field model corresponding to the target scene is trained, and the explicit radiance field model includes a first voxel density grid and a first color grid. The first voxel density grid includes voxel density information of a plurality of spatial points in the target scene, and the first color grid includes color information of the spatial points.

[0054] In fact, the voxel density can also be referred to as opacity. For example, in the plenoxels model, the color information is represented by spherical harmonic coefficients, and the color refers to RGB color. Each color channel needs 9 spherical harmonic coefficients to represent, and there are three color channels in total. Therefore, 27 spherical harmonic coefficients are needed to represent the color of each spatial point.

[0055] Although the training process of the explicit radiance field model composed of the first color grid and the first voxel density grid is prior art, in order to facilitate understanding of the related steps in the following text, the training process of the explicit radiance field model will be briefly introduced here.

[0056] It can be understood that the explicit radiance field model is trained based on a plurality of images obtained by multi-view shooting in a target scene. The target scene is a general statement. In fact, it can be an object (such as a car, shoes, a sofa, etc.), or a room, etc. When shooting with a camera at multiple angles, not only a plurality of images can be obtained, but also corresponding camera parameters can be recorded.

[0057] In the process of training the explicit radiance field model corresponding to the target scene using the plurality of images, a voxel density grid and a color grid containing a plurality of spatial points (discrete points, i.e., points with integer coordinate positions) can be initially set. In the initially set voxel density grid and color grid, the color information and the voxel density information of the spatial points are randomly initialized. The purpose of training the explicit radiance field model is to optimize the color information and the voxel density information in the two grids. As can be seen, the explicit radiance field model is not actually a neural network trained, but two grids learned.

[0058] Then, the explicit radiance field model is trained by combining the plurality of images and the camera parameters. Figure 2An example process of how to train an explicit radiance field model using images taken from a given viewpoint is illustrated.

[0059] In Figure 2 which image F is any one of the above plurality of images, assuming that the resolution of this image F is 1000*1000, i.e. including 1000*1000 pixels, from any pixel o, a ray passing through the camera optical center, i.e. the camera ray r, is emitted, the corresponding viewpoint of which can be determined according to the camera parameters, and then a plurality of sampling points (assuming N) can be sampled on this camera ray in a certain sampling strategy, such as the plurality of circles shown in the figure, numbered 1, 2…

[0060] After that, the voxel density information and color information corresponding to these sampling points need to be calculated respectively. In fact, the position coordinates of the sampling points are often not integers, but have decimals, so they can be called continuous sampling points in contrast to the above discrete spatial points.

[0061] Therefore, the voxel density information and color information of each sampling point on this camera ray r are the voxel density information and color information of the 8 adjacent (closest) spatial points in the voxel density grid and color grid, which are then calculated by trilinear interpolation.

[0062] Specifically, for any sampling point i, after determining the 8 adjacent spatial points in the voxel density grid and color grid, the interpolation weight coefficients corresponding to the 8 adjacent spatial points can be determined based on the distances between the sampling point and the 8 adjacent spatial points. Then, taking the color information as an example, the color information of the sampling point i can be determined based on the interpolation weight coefficients corresponding to the 8 adjacent spatial points and the color information corresponding to the 8 adjacent spatial points by trilinear interpolation method (because it is a 3D grid modeling, so it is trilinear interpolation). The voxel density information is the same, and is not repeated here.

[0063] After calculating the color information and voxel density information corresponding to each sampling point on the above camera ray r, the following volume rendering equation can be used for 3D rendering:

[0064]

[0065] where α i = 1-exp(-σ i ·δ i ),

[0066] where c i represents the color information of the sampling point i, and σ idenotes the voxel density information of the sampling point i, δ i denotes the distance between the sampling point i and the sampling point i+1. i denotes the light termination probability corresponding to the sampling point i, 1-α j denotes the light transmission rate corresponding to the sampling point j, T i denotes the cumulative light transmission rate when reaching the sampling point i.

[0067] It can be understood that, based on the above-mentioned voxel rendering equation, the color prediction value of the pixel point o corresponding to the camera ray r can be obtained, and by comparing the color value corresponding to the pixel point in the image F, the loss function value corresponding to the pixel point o can be obtained, and the voxel density and color information of the spatial points in the voxel density grid and the color grid are adjusted based on the loss function value. Under the assumption that the resolution of the above-mentioned one image is 1000*1000, each pixel can emit a camera ray, so that after a large amount of training, a trained explicit radiance field model, i.e., a first voxel density grid and a first color grid, can be obtained.

[0068] The training speed of the explicit radiance field model is relatively fast, and generally it can be completed in tens of minutes, but as introduced above, the first voxel density grid and the first color grid obtained after training store the voxel density information and the color information of a plurality of spatial points, and the occupation of the storage space is relatively large, which can generally reach the level of hundreds of megabytes or even several gigabytes. This is not convenient for deploying the explicit radiance field model on terminal devices with limited storage space, such as smart phones, extended reality (XR), etc.

[0069] Therefore, while still using the explicit radiance field method to achieve high-speed rendering speed, it is necessary to compress the explicit radiance field model to reduce its occupation of the storage space, so as to provide stronger practicability.

[0070] In the embodiment of the present application, a non-uniform compression compression scheme is provided to compress the first voxel density grid and the first color grid included in the explicit radiance field model.

[0071] Taking a 512x512x512 3D grid size as an example, that is, assuming that the original first voxel density grid and the first color grid are both 512x512x512 3D grids. First, for the first voxel density grid storing the voxel density information of a plurality of spatial points, because the storage space of the first voxel density grid itself is not large, and at the same time, the stored voxel density information is very important for the accuracy of voxel rendering, a lower first compression rate (such as using a 1 / 2 compression rate) is sampled for compression processing, that is, the average of every 8 spatial points (2x2x2) is saved, and after compression in this way, the resolution of the first voxel density grid is reduced to 256x256x256, and the storage space occupied by the voxel density information is reduced to 1 / 8 of the original. The above average processing includes: 8 spatial point positions are averaged to obtain a new spatial point position, and the voxel densities of the 8 spatial points are averaged to obtain an average voxel density, and the average voxel density is assigned to the new spatial point position.

[0072] As for the first color grid, the color information stored in the color grid actually has a large redundancy, and because the storage space occupied by the color information is originally about 27 times that of the voxel density information, a larger second compression rate (such as a 1 / 4 compression rate) is used for compression processing of the first color grid, that is, every 64 spatial points are combined into one point by averaging, and after compression in this way, the resolution of the first color grid is reduced to 128x128x128, and the storage space of the color information is reduced to 1 / 64 of the original.

[0073] After the first voxel density grid and the first color grid are compressed into the second voxel density grid and the second color grid respectively, the second voxel density grid and the second color grid are stored, which will occupy a smaller storage space. And because the voxel density information has a greater impact on the ability to maintain the explicit radiation field model than the high-dimensional color information, the non-uniform compression scheme of sampling a smaller compression rate for the first voxel density grid and a larger compression rate for the first color grid reduces the occupation of the storage space while avoiding the adverse effects on the model performance.

[0074] Figure 3 An application diagram of the processing method of the explicit radiation field provided by the embodiment of the present application is shown in FIG. 1. Figure 3 As shown in FIG. 1, the method can be executed by a terminal device, and after the first voxel density grid and the first color grid are compressed into the second voxel density grid and the second color grid respectively, the second voxel density grid and the second color grid can be stored in the disk space of the terminal device.

[0075] When the terminal device needs to perform a new-view suitable task, the second voxel density grid and the second color grid are read from the disk space, and the second voxel density grid and the second color grid are respectively subjected to corresponding up-sampling processing based on the first compression rate and the second compression rate to obtain a third voxel density grid and a third color grid, and the third voxel density grid and the third color grid are deployed into the memory. Then, based on an input new view, an image corresponding to the new view is generated using the third voxel density grid and the third color grid.

[0076] In an optional embodiment, to further reduce the occupation of the storage space, a set lossless compression algorithm can also be used to compress the second voxel density grid and the second color grid, and the compression result of the lossless compression algorithm is stored. For example, the second voxel density grid and the second color grid are packaged into a file using a Huffman compression algorithm, and the file is stored in the disk.

[0077] In addition, optionally, to further reduce the occupation of the storage space, the color information and the voxel density information stored in the second voxel density grid and the second color grid can be stored in a half-precision floating-point (half) format.

[0078] Figure 4 A flowchart of a processing method of an explicit radiation field provided by an embodiment of the present application is shown in FIG. 4, and the method includes the following steps. Figure 4

[0079] 401. Obtain a plurality of images of a target scene corresponding to a plurality of views.

[0080] 402. Obtain an explicit radiation field model corresponding to the target scene trained using the plurality of images, the explicit radiation field model including a first voxel density grid and a first color grid, the first voxel density grid including voxel density information of a plurality of spatial points in the target scene, and the first color grid including color information of the plurality of spatial points.

[0081] 403. Compress the first voxel density grid using a first compression rate to obtain a second voxel density grid, and compress the first color grid using a second compression rate to obtain a second color grid, wherein the first compression rate is less than the second compression rate.

[0082] 404. Determine important spatial points in the plurality of spatial points according to usage statistical information of the plurality of spatial points in the training process of the explicit radiation field model.

[0083] 405. Store the second voxel density grid, the second color grid, and the voxel density information and the color information of the important spatial points in the first voxel density grid and the first color grid.

[0084] ​406、receive the input new view angle, up-sample the second voxel density grid and the second color grid based on the first compression rate and the second compression rate respectively to obtain a third voxel density grid and a third color grid; replace the voxel density information and the color information of the corresponding spatial points in the third voxel density grid and the third color grid with the stored voxel density information and color information of the important spatial points; and generate an image corresponding to the new view angle according to the third voxel density grid and the third color grid after the replacement processing.

[0085] In the embodiment, while ensuring to reduce the storage space occupied by the explicit radiation field model, a scheme for further improving the voxel rendering quality is provided: identifying an important spatial point from the original explicit radiation field model, and saving the color information and the voxel density information corresponding to the important spatial point in the original explicit radiation field model.

[0086] Because it is found through research that the importance of some spatial points is much higher than that of other spatial points during voxel rendering, specifically, the color and voxel density information of these spatial points is used many times (such as 50 times) more than the average number of times of use during voxel rendering. Therefore, the use statistical information of each spatial point in the original first voxel density grid and the first color grid can be counted, so as to determine the important spatial points based on the use statistical information, and store the voxel density information and the color information of the important spatial points, i.e., the corresponding voxel density information and color information in the first voxel density grid and the first color grid.

[0087] Specifically, the importance score of each spatial point can be calculated, and then the importance scores of multiple spatial points can be sorted from high to low. For example, the spatial points with scores in the top 5% can be selected as important spatial points, or the spatial points with scores greater than a set score threshold can be selected as important spatial points. Because the number of important spatial points will not be very large, the storage space will not be significantly increased.

[0088] In an optional embodiment, during the training process of the explicit radiation field model composed of the first voxel density grid and the first color grid, the number of times each spatial point is used can be counted as its importance score.

[0089] As described above, for a sampling point, the color information and the voxel density information of the sampling point need to be interpolated by its eight adjacent spatial points. The use of spatial points can be the number of times of interpolation.

[0090] Another optional method for determining important spatial points is introduced in the following embodiment. Figure 5

[0091] ​Here, it is assumed that the important spatial points have been determined, and the color information and the voxel density information of the important spatial points have been stored together with the second color grid and the second voxel density grid that have been compressed.

[0092] When a new view synthesis task needs to be performed due to the input of a new view, the second voxel density grid and the second color grid are first up-sampled based on the first compression rate and the second compression rate to obtain a third voxel density grid and a third color grid. Although the same size as the original first voxel density grid and the first color grid can be restored through up-sampling, the color information and the voxel density information of the plurality of spatial points in the up-sampled third voxel density grid and the third color grid are not necessarily exactly the same as the color information and the voxel density information of the plurality of spatial points in the original first voxel density grid and the first color grid, for example, information loss may occur. Therefore, replacing the voxel density information and the color information of the important spatial points in the third voxel density grid and the third color grid with the corresponding voxel density information and color information of the important spatial points in the first voxel density grid and the first color grid stored can ensure the accuracy of the color information and the voxel density information of the important spatial points. Then, an image corresponding to the new view is generated according to the third voxel density grid and the third color grid after the replacement processing. In this way, based on the original voxel density information and color information of the saved important spatial points, a better quality image can be rendered.

[0093] Figure 5 A flowchart of an important spatial point determination method provided by an embodiment of the present application is shown in Figure 5 The method includes the following steps:

[0094] 501. Obtain the sampling points respectively sampled on a plurality of camera rays determined based on a plurality of images in an explicit radiance field model training process; for a target spatial point in the plurality of spatial points, determine a target sampling point taking the target spatial point as a neighbor point and a target camera ray on which the target sampling point is located from the sampling points, and the target spatial point is any one of the plurality of spatial points.

[0095] 502. Determine the ray termination probability and the cumulative ray transmittance corresponding to the target sampling point under the target camera ray, and determine the interpolation weight coefficient when the color information and the voxel density information of the target sampling point are determined in an interpolation manner.

[0096] 503. Determine the importance score of the target spatial point as the use statistical information according to the interpolation weight coefficient, the ray termination probability and the cumulative ray transmittance.

[0097] 504. Determine the target spatial point whose score meets the set condition according to the importance scores of the plurality of spatial points respectively.

[0098] Referring to the explicit radiation field model training process described earlier, during training, several camera rays can be generated based on each pixel in multiple known images. Multiple sampling points can be sampled along each camera ray, and the color and voxel density information of each sampling point is obtained by trilinear interpolation based on the color and voxel density information of its eight neighboring spatial points. Therefore, for any sampling point on any camera ray, on the one hand, the interpolation weight coefficients corresponding to its eight neighboring spatial points can be determined; on the other hand, based on the aforementioned voxel rendering equation, the ray termination probability and cumulative ray transmittance corresponding to that sampling point can be calculated.

[0099] In this embodiment, for any target spatial point, the target sampling points that are its neighbors and the target camera ray where the target sampling point is located can be determined. It is understood that for a target spatial point, there may be many sampling points (i.e., multiple target sampling points) that are used as one of the eight neighboring spatial points when determining color information and voxel density information.

[0100] For ease of understanding, combined with Figure 6 Let's illustrate with an example. In Figure 6 In this context, it is assumed that the nearest 8 neighboring spatial points corresponding to sampling point s1 on camera ray r1 and sampling point s2 on camera ray r2 each include spatial point v1.

[0101] Assume that when determining the color and voxel density information of sampling point s1, the interpolation weight coefficient corresponding to spatial point v1 is w11, and when determining the color and voxel density information of sampling point s2, the interpolation weight coefficient corresponding to spatial point v1 is w12. Assume that when performing voxel rendering based on each sampling point on camera ray r1, the cumulative transmittance of the light ray corresponding to sampling point s1 is T1, and the ray termination probability corresponding to sampling point s1 is α1. Assume that when performing voxel rendering based on each sampling point on camera ray r2, the cumulative transmittance of the light ray corresponding to sampling point s2 is T2, and the ray termination probability corresponding to sampling point s2 is α2.

[0102] Based on the above assumptions, the importance score corresponding to spatial point v1 can optionally be calculated using the following formula:

[0103] Score(v1)=w11*T1*α1+w12*T2*α2.

[0104] In other words, when there are multiple target sampling points with target spatial point v1 as a neighbor (adjacent spatial point), the interpolation weight coefficient, ray termination probability and cumulative ray transmittance corresponding to each target sampling point can be calculated. The importance score of target spatial point v1 can be obtained by accumulating the product of multiple target sampling points.

[0105] After non-uniform compression, the storage space of the second color grid and the second voxel density grid has been greatly reduced. However, at the same time, when performing voxel rendering based on the upsampling results of the second color grid and the second voxel density grid, the high-frequency details of the rendered image may be lost, resulting in poor image quality. Therefore, in order to recover these high-frequency details, this invention proposes a neural network model—the High-Frequency Detail Recovery Network Model—for recovering high-frequency details. Specifically, utilizing the correlation between the color information and voxel density information of a spatial point helps in the recovery of high-frequency information.

[0106] The high-frequency detail recovery network model is a neural network model that can be built by drawing on the network architecture of synthetic networks in style generative networks (style-GAN).

[0107] Figure 7 A flowchart of a high-frequency detail recovery network model training method provided in an embodiment of the present invention is shown below. Figure 7 As shown, the method includes the following steps:

[0108] 701. Obtain multiple images of the target scene corresponding to multiple viewpoints.

[0109] 702. Obtain an explicit radiation field model corresponding to the target scene, trained using multiple images. The explicit radiation field model includes a first voxel density grid and a first color grid. The first voxel density grid includes voxel density information of multiple spatial points in the target scene, and the first color grid includes color information of the multiple spatial points.

[0110] 703. Compress the first voxel density mesh using a first compression ratio to obtain a second voxel density mesh; compress the first color mesh using a second compression ratio to obtain a second color mesh; wherein the first compression ratio is less than the second compression ratio.

[0111] 704. The second voxel density grid and the second color grid are upsampled based on the first compression ratio and the second compression ratio, respectively, to obtain the fourth voxel density grid and the fourth color grid.

[0112] 705. Using the voxel density information and color information of multiple spatial points in the fourth voxel density grid and the fourth color grid as training samples, and combining the position encoding vectors of multiple spatial points, train a high-frequency detail recovery network model. The supervision information of the high-frequency detail recovery network model is the voxel density information and color information of multiple spatial points in the first voxel density grid and the first color grid.

[0113] 706. Store the second voxel density mesh, the second color mesh, and the high-frequency detail recovery network model.

[0114] As can be seen from the above steps, in the process of training the high-frequency detail recovery network model, it is necessary to use the second voxel density grid and the second color grid obtained through non-uniform compression processing, as well as the spatial point position encoding vector.

[0115] Specifically, the second voxel density grid and the second color grid can be upsampled based on the first compression ratio and the second compression ratio to obtain the fourth voxel density grid and the fourth color grid, such that the size of the fourth voxel density grid and the fourth color grid is the same as that of the original first voxel density grid and the first color grid. That is, the fourth voxel density grid and the fourth color grid include the same number of spatial points as the first voxel density grid and the first color grid.

[0116] Furthermore, for each spatial point, its position coordinates are feature-encoded using a predetermined high-frequency function to obtain a position encoding vector for each spatial point. Optionally, this high-frequency function can be composed of a series of sine and cosine functions.

[0117] For example, the process of using the high-frequency function γ to encode the location of any spatial point v can be represented as:

[0118] γ(v)=[cos(2πa1v),sin(2πa1v),…,cos(2πa L v),sin(2πa L v)]

[0119] Where A = [a1, ..., a L ] is a value sampled from the standard normal distribution.

[0120] High-frequency functions are not limited to the examples mentioned above.

[0121] When training the high-frequency detail recovery network model, the voxel density information and color information of a spatial point can be determined from the fourth voxel density grid and the fourth color grid, and the position encoding vector of the spatial point can be determined. These three pieces of information are input into the high-frequency detail recovery network model, and the high-frequency detail recovery network model will output the predicted color information and voxel density information corresponding to the spatial point. By comparing the predicted result with the color information and voxel density information corresponding to the spatial point in the first voxel density grid and the first color grid, the loss function value corresponding to the spatial point can be determined.

[0122] Figure 8 This is a schematic diagram illustrating the composition of a high-frequency detail recovery network model provided in an embodiment of the present invention, as shown below. Figure 8As shown, the high-frequency detail recovery network model includes a backbone network, a first branch network corresponding to color information, and a second branch network corresponding to voxel density information. The first and second branch networks are connected to the output of the backbone network.

[0123] like Figure 8 As shown, the backbone network includes at least one set of fully connected (FC) layers and adaptive layer normalization (AdaLN) layers, such as the three sets illustrated in the figure.

[0124] like Figure 8 As shown, a concat layer can be included at the top of the backbone network. After determining the voxel density and color information of a spatial point from the fourth voxel density grid and the fourth color grid, the concatenated voxel density and color information of this spatial point can be input into the first fully connected (FC) layer in the backbone network. It can be understood that the voxel density and color information here are input into the model in the form of mathematical vectors.

[0125] In this embodiment, to make the high-frequency detail recovery network model more sensitive to the high-frequency encoded position information of spatial points, the AdaLN algorithm is used to further enhance the model's ability to process high-frequency information.

[0126] In one optional embodiment, both the first branch network and the second branch network include at least one set of FC layers and AdaLN layers. Figure 8 As shown, each of these two branch networks includes two FC layers and an AdaLN layer located between the two FC layers. In practice, each of these branch networks could also contain only one FC layer.

[0127] In addition, such as Figure 8 As shown, optionally, a fusion layer can also be connected to the output of the first branch network and the second branch network.

[0128] Combination Figure 8 This diagram illustrates the structure of a high-frequency detail recovery network model and briefly introduces its training process.

[0129] like Figure 8 As shown, from the fourth voxel density grid V σ1 and the fourth color grid V c1 The voxel density and color information of any selected spatial point L are input into the backbone network, and the position encoding vector P of spatial point L is generated. LEach AdaLN layer is input separately; the AdaLN layer is used to normalize the output information of the FC layer connected above it, and to encode the vector P based on the position of the input spatial point L. L A first position adjustment parameter ys and a second position adjustment parameter yb are generated to adjust the output information of the connected fC layer according to the first position adjustment parameter ys and the second position adjustment parameter yb.

[0130] The output information of the backbone network is input into the first branch network and the second branch network respectively to obtain the first color prediction information of the first branch network and the first voxel density prediction information of the second branch network.

[0131] Combine the first color prediction information with the fourth color grid V c1 The color information of point L in the mid-space is fused to obtain the second color prediction information (V). c-r The first voxel density prediction information is combined with the information in the fourth voxel density grid V. σ1 The voxel density information of the mid-space point L is fused to obtain the second voxel density prediction information (V). σ-r );

[0132] Based on the second color prediction information and the second voxel density prediction information, and compared with the first voxel density grid V σ0 and the first color grid V c0 The voxel density and color information of spatial point L are used to determine the loss function value corresponding to spatial point L, and a high-frequency detail recovery network model is trained based on the loss function value.

[0133] exist Figure 8 In the middle, V c-r and V σ-r These are used to represent the recovered color network and voxel density mesh, respectively. For example... Figure 8 As shown, the first color prediction information is compared with the fourth color grid V. c1 The color information of point L in the mid-space can be fused by performing additive calculations.

[0134] The function of the AdaLN layer described above can be represented as:

[0135]

[0136] Where u represents the output information of the FC layer, and μ(u) and v(u) represent the calculation of the mean and standard deviation, respectively. y = (ys, yb) is the position encoding vector P generated by the AdaLN layer. L The position adjustment parameters are obtained through learning and conversion.

[0137] Using the fourth voxel density grid V σ1 and the fourth color grid Vc1 The color information, voxel density information, and position encoding vector of several spatial points are used to train the high-frequency detail restoration network model until convergence, so as to obtain the trained high-frequency detail restoration network model.

[0138] Then, the non-uniformly compressed second voxel density mesh, the second color mesh, and the high-frequency detail recovery network model can be stored.

[0139] It is understandable that, in the case where important spatial points have been screened as described in the previous embodiments, it is also necessary to store the color information and voxel density information corresponding to the important spatial points in the first voxel density grid and the first color grid.

[0140] In an optional embodiment, to further reduce the storage space occupied, a lossless compression algorithm (such as the Huffman compression algorithm) can be used to package and compress all data, including the second voxel density grid, the second color grid, the high-frequency detail recovery network model, and the color information and voxel density information corresponding to important spatial points in the first voxel density grid and the first color grid, and store the compressed file.

[0141] Figure 9 A flowchart illustrating a method for processing explicit radiation fields provided in an embodiment of the present invention is shown below. Figure 9 As shown, the method includes the following steps:

[0142] 901. Obtain multiple images of the target scene corresponding to multiple viewpoints.

[0143] 902. Obtain an explicit radiation field model corresponding to the target scene, trained using multiple images. The explicit radiation field model includes a first voxel density grid and a first color grid. The first voxel density grid includes voxel density information of multiple spatial points in the target scene, and the first color grid includes color information of the multiple spatial points.

[0144] 903. Compress the first voxel density mesh using a first compression ratio to obtain a second voxel density mesh; compress the first color mesh using a second compression ratio to obtain a second color mesh; wherein the first compression ratio is less than the second compression ratio.

[0145] 904. Receive the new input perspective, and upsample the second voxel density grid and the second color grid based on the first compression ratio and the second compression ratio respectively to obtain the fourth voxel density grid and the fourth color grid. Input the voxel density information and color information of multiple spatial points in the fourth voxel density grid and the fourth color grid, as well as the position encoding vectors of multiple spatial points, into the trained high-frequency detail recovery network model to obtain the recovered fifth voxel density grid and the fifth color grid. Generate an image corresponding to the new perspective based on the fifth voxel density grid and the fifth color grid.

[0146] It should be noted that in this embodiment, the high-frequency detail recovery network model is used, that is, the high-frequency detail recovery network model is used to recover the fifth voxel density grid and the fifth color grid with high-frequency detail information, and the fifth voxel density grid and the fifth color grid are used to generate images from new perspectives.

[0147] Because the High Frequency Detail Reconstruction Network (HFDRN) model is a shallow neural network model, the number of network parameters required for training is small, resulting in a relatively low storage footprint. However, its ability to recover high-frequency information can significantly improve the quality of the rendered image.

[0148] The explicit radiation field processing method provided in this invention can be executed in the cloud. Several computing nodes (cloud servers) can be deployed in the cloud, each with computing and storage resources. In the cloud, multiple computing nodes can be organized to provide a certain service; of course, a single computing node can also provide one or more services. The cloud can provide this service by providing an external service interface, which users can call to use the corresponding service.

[0149] According to the solution provided in this embodiment of the invention, the cloud can provide a service interface with a defined service (explicit radiation field processing service). Users invoke this service interface through their terminal devices to trigger an explicit radiation field processing request to the cloud. This request includes multiple images obtained by the terminal device from multiple perspectives of the target scene. The cloud determines the computing node that responds to the request and utilizes the processing resources in that computing node to perform the following steps:

[0150] Obtain an explicit radiation field model corresponding to the target scene trained using the multiple images. The explicit radiation field model includes a first voxel density grid and a first color grid. The first voxel density grid includes voxel density information of multiple spatial points in the target scene, and the first color grid includes color information of the multiple spatial points.

[0151] The first voxel density mesh is compressed using a first compression ratio to obtain a second voxel density mesh; the first color mesh is compressed using a second compression ratio to obtain a second color mesh; wherein the first compression ratio is less than the second compression ratio;

[0152] The second voxel density grid and the second color grid are sent to the terminal device for storage.

[0153] The above execution process can be referred to the relevant descriptions in the other embodiments mentioned above, and will not be repeated here.

[0154] For ease of understanding, combined with Figure 10 To illustrate this, users can... Figure 10 The terminal device E1, as illustrated, calls the explicit radiation field processing service to upload multiple acquired images. The service interfaces for users to call this service include Software Development Kits (SDKs) and Application Programming Interfaces (APIs). Figure 10 The diagram illustrates the API interface scenario. In the cloud, as shown in the figure, assume that the explicit radiation field processing service is provided by service cluster E2, which includes at least one computing node. Upon receiving the request, service cluster E2 executes the steps described in the preceding embodiments to obtain a compressed second voxel density mesh and a second color mesh.

[0155] The following describes in detail one or more embodiments of an explicit radiation field processing apparatus of the present invention. Those skilled in the art will understand that these apparatuses can be configured using commercially available hardware components through the steps taught in this invention.

[0156] Figure 11 This is a schematic diagram of the structure of an explicit radiation field processing device provided in an embodiment of the present invention, as shown below. Figure 11 As shown, the device includes: an image acquisition module 11, a grid acquisition module 12, a compression module 13, and a storage module 14.

[0157] Image acquisition module 11 is used to acquire multiple images of the target scene corresponding to multiple viewpoints.

[0158] The mesh acquisition module 12 is used to acquire an explicit radiation field model corresponding to the target scene, which is trained using the multiple images. The explicit radiation field model includes a first voxel density grid and a first color grid. The first voxel density grid includes voxel density information of multiple spatial points in the target scene, and the first color grid includes color information of the multiple spatial points.

[0159] Compression module 13 is used to compress the first voxel density grid using a first compression ratio to obtain a second voxel density grid; and to compress the first color grid using a second compression ratio to obtain a second color grid; wherein the first compression ratio is less than the second compression ratio.

[0160] Optionally, the device further includes a storage module 14 for storing the second voxel density grid and the second color grid.

[0161] Optionally, the device further includes: an important point determination module, used to determine important spatial points among the multiple spatial points based on usage statistics of the multiple spatial points during the training of the explicit radiation field model. The storage module 14 is specifically used to: store the second voxel density grid and the second color grid, as well as the voxel density information and color information of the important spatial points in the first voxel density grid and the first color grid.

[0162] Optionally, the apparatus further includes: a new perspective rendering module, configured to receive an input new perspective; perform upsampling processing on the second voxel density grid and the second color grid based on the first compression ratio and the second compression ratio, respectively, to obtain a third voxel density grid and a third color grid; replace the voxel density information and color information of corresponding spatial points in the third voxel density grid and the third color grid with the stored voxel density information and color information of the important spatial points in the first voxel density grid and the first color grid; and generate an image corresponding to the new perspective based on the replaced third voxel density grid and the third color grid.

[0163] Optionally, the importance point determination module is specifically used for: acquiring sampling points sampled on multiple camera rays determined based on the multiple images during the explicit radiation field model training process; for a target spatial point among the multiple spatial points, determining a target sampling point with the target spatial point as a neighbor and the target camera ray where the target sampling point is located from the sampling points; the target spatial point is any one of the multiple spatial points; determining the ray termination probability and cumulative ray transmittance corresponding to the target sampling point under the target camera ray; determining the interpolation weight coefficient when the target spatial point is used to determine the color information and voxel density information of the target sampling point by interpolation; determining the importance score as the usage statistics information of the target spatial point based on the interpolation weight coefficient, the ray termination probability and the cumulative ray transmittance; and determining the target spatial point whose score meets the set conditions based on the importance scores corresponding to each of the multiple spatial points.

[0164] Optionally, the apparatus further includes: a training module, configured to upsample the second voxel density grid and the second color grid based on the first compression ratio and the second compression ratio, respectively, to obtain a fourth voxel density grid and a fourth color grid; and to train a high-frequency detail recovery network model using the voxel density information and color information of the plurality of spatial points in the fourth voxel density grid and the fourth color grid as training samples, combined with the position encoding vectors of the plurality of spatial points. The high-frequency detail recovery network model is a neural network model, and the supervision information of the high-frequency detail recovery network model is the voxel density information and color information of the plurality of spatial points in the first voxel density grid and the first color grid. Therefore, the storage module 14 is specifically configured to: store the second voxel density grid, the second color grid, and the high-frequency detail recovery network model.

[0165] Optionally, the high-frequency detail recovery network model includes a backbone network, a first branch network corresponding to color information, and a second branch network corresponding to voxel density information. The backbone network includes at least one set of fully connected layers and adaptive normalization layers. The first and second branch networks are respectively connected to the output of the backbone network. The training module is specifically used to: input the voxel density information and color information of any spatial point selected from the fourth voxel density grid and the fourth color grid into the backbone network; input the position encoding vector of any spatial point into each adaptive normalization layer; wherein the adaptive normalization layer is used to normalize the output information of the connected fully connected layers and generate a first position adjustment parameter and a second position adjustment parameter based on the input position encoding vector of any spatial point, so as to adjust the output information of the connected fully connected layers according to the first and second position adjustment parameters; input the output information of the backbone network into the first and second branch networks respectively to obtain the first... The network obtains first color prediction information of the second branch network and first voxel density prediction information of the second branch network; it then fuses the first color prediction information with the color information of any spatial point in the fourth color grid to obtain second color prediction information; it further fuses the first voxel density prediction information with the voxel density information of any spatial point in the fourth voxel density grid to obtain second voxel density prediction information; based on the second color prediction information, the second voxel density prediction information, and the voxel density information and color information of any spatial point in the first voxel density grid and the first color grid, it determines the loss function value corresponding to any spatial point, and trains the high-frequency detail recovery network model based on the loss function value.

[0166] Optionally, both the first branch network and the second branch network include at least one set of fully connected layers and an adaptive normalization layer.

[0167] Optionally, the new perspective rendering module is further configured to: receive an input new perspective; input the voxel density information and color information of the plurality of spatial points in the fourth voxel density grid and the fourth color grid, as well as the position encoding vector of the plurality of spatial points, into the trained high-frequency detail recovery network model to obtain the recovered fifth voxel density grid and fifth color grid; and generate an image corresponding to the new perspective based on the fifth voxel density grid and the fifth color grid.

[0168] Optionally, the storage module 14 is further configured to: compress the second voxel density grid and the second color grid using a set lossless compression algorithm; and store the compression result of the lossless compression algorithm.

[0169] Figure 11 The device shown can perform the steps in the foregoing embodiments. For detailed execution process and technical effects, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0170] In one possible design, the above Figure 11 The structure of the explicit radiation field processing device shown can be implemented as an electronic device. For example... Figure 12 As shown, the electronic device may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code, which, when executed by the processor 21, enables the processor 21 to at least implement the explicit radiation field processing method provided in the foregoing embodiments.

[0171] In an alternative embodiment, the electronic device described above may be an extended reality device. This extended reality device may perform the following methods:

[0172] Display multiple images of the target scene corresponding to multiple viewpoints;

[0173] Obtain an explicit radiation field model corresponding to the target scene trained using the multiple images. The explicit radiation field model includes a first voxel density grid and a first color grid. The first voxel density grid includes voxel density information of multiple spatial points in the target scene, and the first color grid includes color information of the multiple spatial points.

[0174] A first voxel density mesh is compressed using a first compression ratio to obtain a second voxel density mesh; a first color mesh is compressed using a second compression ratio to obtain a second color mesh; wherein the first compression ratio is less than the second compression ratio;

[0175] A new perspective on receiving input;

[0176] The second voxel density grid and the second color grid are upsampled based on the first compression ratio and the second compression ratio, respectively, to obtain the third voxel density grid and the third color grid.

[0177] Generate an image corresponding to the new perspective based on the third voxel density grid and the third color grid;

[0178] Display the image corresponding to the new perspective.

[0179] In addition, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the explicit radiation field processing method provided in the foregoing embodiments.

[0180] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0181] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of processing an explicit radiation field, characterized by, The method comprises: acquiring a plurality of images of a target scene, the plurality of images corresponding to different perspectives; acquiring an explicit radiance field model corresponding to the target scene trained using the plurality of images, the explicit radiance field model comprising a first voxel density grid and a first color grid, the first voxel density grid comprising voxel density information of a plurality of spatial points in the target scene, the first color grid comprising color information of the plurality of spatial points; compressing the first voxel density grid using a first compression rate to obtain a second voxel density grid; compressing the first color grid using a second compression rate to obtain a second color grid; wherein the first compression rate is less than the second compression rate.

2. The method of claim 1, wherein, The method further comprises: determining important spatial points among the plurality of spatial points according to usage statistics of the plurality of spatial points in the training process of the explicit radiance field model; storing the second voxel density grid, the second color grid, and the voxel density information and color information of the important spatial points in the first voxel density grid and the first color grid.

3. The method of claim 2, wherein, The method further comprises: receiving an input new perspective; performing upsampling processing on the second voxel density grid and the second color grid based on the first compression rate and the second compression rate respectively to obtain a third voxel density grid and a third color grid; replacing voxel density information and color information of corresponding spatial points in the third voxel density grid and the third color grid with the stored voxel density information and color information of the important spatial points in the first voxel density grid and the first color grid; generating an image corresponding to the new perspective according to the third voxel density grid and the third color grid after the replacement processing.

4. The method of claim 2, wherein, The method further comprises: acquiring a plurality of images of a target scene, the plurality of images corresponding to different perspectives; acquiring an explicit radiance field model corresponding to the target scene trained using the plurality of images, the explicit radiance field model comprising a first voxel density grid and a first color grid, the first voxel density grid comprising voxel density information of a plurality of spatial points in the target scene, the first color grid comprising color information of the plurality of spatial points; compressing the first voxel density grid using a first compression rate to obtain a second voxel density grid; compressing the first color grid using a second compression rate to obtain a second color grid; wherein the first compression rate is less than the second compression rate. The method further comprises: determining important spatial points among the plurality of spatial points according to usage statistics of the plurality of spatial points in the training process of the explicit radiance field model; 5. The method according to claim 1 or 2, characterized in that, storing the second voxel density grid, the second color grid, and the voxel density information and color information of the important spatial points in the first voxel density grid and the first color grid. The method further comprises: receiving an input new perspective; performing upsampling processing on the second voxel density grid and the second color grid based on the first compression rate and the second compression rate respectively to obtain a third voxel density grid and a third color grid; replacing voxel density information and color information of corresponding spatial points in the third voxel density grid and the third color grid with the stored voxel density information and color information of the important spatial points in the first voxel density grid and the first color grid; generating an image corresponding to the new perspective according to the third voxel density grid and the third color grid after the replacement processing. The method further comprises: acquiring a plurality of images of a target scene, the plurality of images corresponding to different perspectives; acquiring an explicit radiance field model corresponding to the target scene trained using the plurality of images, the explicit radiance field model comprising a first voxel density grid and a first color grid, the first voxel density grid comprising voxel density information of a plurality of spatial points in the target scene, the first color grid comprising color information of the plurality of spatial points; compressing the first voxel density grid using a first compression rate to obtain a second voxel density grid; compressing the first color grid using a second compression rate to obtain a second color grid; wherein the first compression rate is less than the second compression rate. The method further comprises: performing upsampling processing on the second voxel density grid and the second color grid based on the first compression rate and the second compression rate respectively to obtain a fourth voxel density grid and a fourth color grid; The voxel density information and the color information of the plurality of spatial points in the fourth voxel density grid and the fourth color grid are taken as training samples, and a position encoding vector of the plurality of spatial points is combined to train a high-frequency detail recovery network model, the high-frequency detail recovery network model being a neural network model, and supervision information of the high-frequency detail recovery network model being the voxel density information and the color information of the plurality of spatial points in the first voxel density grid and the first color grid; The second voxel density grid, the second color grid and the high-frequency detail recovery network model are stored.

6. The method of claim 5, wherein, The high-frequency detail recovery network model comprises a backbone network, a first branch network corresponding to color information and a second branch network corresponding to voxel density information, wherein the backbone network comprises at least one group of fully connected layers and adaptive layer normalization layers, and the first branch network and the second branch network are connected to an output end of the backbone network respectively; The training of the high-frequency detail recovery network model comprises: The voxel density information and the color information of any spatial point selected from the fourth voxel density grid and the fourth color grid are input into the backbone network, and a position encoding vector of the any spatial point is input into a respective adaptive layer normalization layer; wherein the adaptive layer normalization layer is used to normalize output information of a connected fully connected layer, and generate a first position adjustment parameter and a second position adjustment parameter according to the position encoding vector of the any spatial point, so as to adjust the output information of the connected fully connected layer according to the first position adjustment parameter and the second position adjustment parameter; The output information of the backbone network is input into the first branch network and the second branch network respectively, so as to obtain first color prediction information of the first branch network and first voxel density prediction information of the second branch network; The first color prediction information is fused with the color information of the any spatial point in the fourth color grid to obtain second color prediction information, and the first voxel density prediction information is fused with the voxel density information of the any spatial point in the fourth voxel density grid to obtain second voxel density prediction information; According to the second color prediction information, the second voxel density prediction information, the voxel density information and the color information of the any spatial point in the first voxel density grid and the first color grid, a loss function value corresponding to the any spatial point is determined, and the high-frequency detail recovery network model is trained based on the loss function value.

7. The method of claim 6, wherein, The first branch network and the second branch network each comprise at least one group of fully connected layers and adaptive layer normalization layers.

8. The method of claim 5, wherein, The method further comprises: receiving an input new view angle; inputting the voxel density information and the color information of the plurality of spatial points in the fourth voxel density grid and the fourth color grid, and the position encoding vector of the plurality of spatial points into the trained high-frequency detail recovery network model, so as to obtain a recovered fifth voxel density grid and a fifth color grid. According to the fifth voxel density grid and the fifth color grid, an image corresponding to the new perspective is generated.

9. The method of claim 1, wherein, The method further comprises: compressing the second voxel density grid and the second color grid using a set lossless compression algorithm; storing the compression result of the lossless compression algorithm.

10. A method of processing an explicit radiation field, characterized by, Comprise: receiving a request triggered by a terminal device calling a set service, the request including a plurality of images captured under a plurality of perspectives of a target scene; using a processing resource corresponding to the set service to perform the following steps: obtaining an explicit radiation field model corresponding to the target scene trained using the plurality of images, the explicit radiation field model including a first voxel density grid and a first color grid, the first voxel density grid including voxel density information of a plurality of spatial points in the target scene, and the first color grid including color information of the plurality of spatial points; compressing the first voxel density grid using a first compression rate to obtain a second voxel density grid, and compressing the first color grid using a second compression rate to obtain a second color grid, wherein the first compression rate is less than the second compression rate; sending the second voxel density grid and the second color grid to the terminal device for storage.

11. An electronic device, comprising: Comprise: a memory, a processor, and a communication interface, wherein the memory stores executable code, and when the executable code is executed by the processor, the processor performs the processing method of the explicit radiation field according to any one of claims 1 to 9.

12. A non-transitory machine-readable storage medium, comprising: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by the processor of the cloud server, the processor performs the processing method of the explicit radiation field according to any one of claims 1 to 9.

13. A method of processing an explicit radiation field, characterized by, Applied to an extended reality device, the method comprises: displaying a plurality of images captured under a plurality of perspectives of a target scene; obtaining an explicit radiation field model corresponding to the target scene trained using the plurality of images, the explicit radiation field model including a first voxel density grid and a first color grid, the first voxel density grid including voxel density information of a plurality of spatial points in the target scene, and the first color grid including color information of the plurality of spatial points; compressing the first voxel density grid using a first compression rate to obtain a second voxel density grid, and compressing the first color grid using a second compression rate to obtain a second color grid, wherein the first compression rate is less than the second compression rate; receiving an input new perspective; respectively performing up-sampling processing on the second voxel density grid and the second color grid based on the first compression rate and the second compression rate to obtain a third voxel density grid and a third color grid; generating an image corresponding to the new perspective according to the third voxel density grid and the third color grid; displaying the image corresponding to the new perspective.

Images