Image rendering method, system, device and readable storage medium

Abstract

The application provides an image rendering method, system, device and readable storage medium. The image rendering method comprises: acquiring a current posture of an image acquisition device; predicting a target posture based on the current posture and a current motion trend of the image acquisition device; determining a sorting rendering result corresponding to the target posture; determining a significant Gaussian based on a Gaussian under the current posture and a Gaussian under the target posture; caching the significant Gaussian based on the sorting rendering result, performing color integration on the significant Gaussian, and obtaining a target image, wherein the sorting result can be shared across frames, and the sorting rendering time of a rendering frame is greatly reduced; and through caching the significant Gaussian, color integration calculation in rasterization is reduced, and the calculation efficiency of rasterization is improved.

Description

Technical Field

[0001] This application belongs to the field of image rendering technology, and relates to an image rendering method, system, device and readable storage medium. Background Technology

[0002] In existing technologies, Neural Radiance Fields (NeRF) have revolutionized the landscape of virtual reality (VR) / augmented reality (AR), large-scale landscape modeling, virtual avatars, and novel perspective synthesis. Despite the significant achievements of NeRF, its intensive computation remains a key bottleneck for achieving real-time, high-resolution rendering.

[0003] To address this issue, 3D Gaussian Splatting (3DGS) was proposed as a rapid alternative to NeRF. 3DGS projects pre-computed Gaussian points onto the rendering screen, simplifying color integration and light transport modeling in 3D space. Despite numerous improvements, 3DGS still falls short of real-time performance in AR / VR applications. For example, on mobile Volta GPUs, 3DGS only achieves 5-21 FPS in real-world scenes, far from the real-time target. Therefore, improving the rendering efficiency of 3DGS has become a pressing technical challenge. Summary of the Invention

[0004] This application provides an image rendering method, system, device, and readable storage medium to solve the technical problem of low rendering efficiency of 3DGS in the prior art.

[0005] In a first aspect, embodiments of this application provide an image rendering method, the method comprising: acquiring the current pose of an image acquisition device; predicting a target pose based on the current pose and current motion trend of the image acquisition device; determining a sorted rendering result corresponding to the target pose; determining a significant Gaussian based on the Gaussian under the current pose and the Gaussian under the target pose; caching the significant Gaussian based on the sorted rendering result, and performing color integration on the significant Gaussian to obtain a target image.

[0006] In one implementation of the first aspect, determining the sorted rendering result corresponding to the target pose includes: determining the Gaussian spheroids to be sorted under the target pose; performing a pre-sorting operation on the Gaussian spheroids to obtain a pre-sorting result of the Gaussian spheroids; and performing sorted shared rendering on multiple rendering frames within the target pose sequentially based on the pre-sorting result to obtain the sorted rendering result.

[0007] In one implementation of the first aspect, predicting the target posture based on the current posture and current motion trend of the image acquisition device includes: determining the current position information corresponding to the current posture of the image acquisition device and the current velocity information corresponding to the current motion trend; and predicting the target posture based on the current position information and the current velocity information.

[0008] In one implementation of the first aspect, determining the Gaussian to be sorted under the target pose includes: determining the near clipping plane and the far clipping plane of the target pose projection; and determining the Gaussian located between the near clipping plane and the far clipping plane as the Gaussian to be sorted.

[0009] In one implementation of the first aspect, determining a significant Gaussian based on the Gaussian in the current pose and the Gaussian in the target pose includes: determining the current Gaussian in the current pose of the image acquisition device; determining intersecting Gaussians that intersect the current Gaussian with the Gaussian in the target pose; and determining a preset number of Gaussians whose Gaussian values ​​are greater than a preset Gaussian threshold as the significant Gaussians among the intersecting Gaussians.

[0010] In one implementation of the first aspect, the salient Gaussian includes a first salient Gaussian and a second salient Gaussian. The step of caching the salient Gaussian based on the rendering result to perform color integration on the salient Gaussian to obtain a target image includes: performing color integration on the first salient Gaussian to obtain a first pixel of the first salient Gaussian; storing the first salient Gaussian and the first pixel in a corresponding radiative buffer list; when rendering a frame to be rendered, sequentially determining the second salient Gaussian corresponding to each pixel in the frame to be rendered; if the second salient Gaussian matches the first salient Gaussian in the radiative buffer list, then determining the first pixel corresponding to the first salient Gaussian as the second pixel of the second salient Gaussian; if the second salient Gaussian does not match the first salient Gaussian in the radiative buffer list, then performing color integration on the second salient Gaussian to obtain a second pixel, and storing the second salient Gaussian and the second pixel in the radiative buffer list; and obtaining the target image based on the first pixel and the second pixel.

[0011] In one implementation of the first aspect, the pre-sorting operation on the Gaussian to obtain the pre-sorting result of the Gaussian includes: determining the distance information between the target pose and the Gaussian; and determining the pre-sorting result of the Gaussian based on the distance information.

[0012] Secondly, embodiments of this application provide a hardware-algorithm co-design system. The system includes a camera pose determination unit configured to determine the current pose of an image acquisition device and predict a target pose based on the current pose of the image acquisition device; a sorting and sharing unit configured to determine a sorted rendering result corresponding to the target pose; a salient Gaussian determination unit configured to determine a salient Gaussian based on the Gaussian under the current pose of the image acquisition device and the Gaussian under the target pose; and a radiation buffer unit configured to buffer the salient Gaussian based on the sorted rendering result, and to perform color integration on the salient Gaussian to obtain a target image.

[0013] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image rendering method described in any one of the first aspects of embodiments of this application.

[0014] Fourthly, embodiments of this application provide an electronic device, the electronic device including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the image rendering method as described in any one of the first aspects of the embodiments of this application.

[0015] As described above, the image rendering method, system, device, and readable storage medium of this application have the following beneficial effects:

[0016] This application obtains the target pose within a preset time interval and performs a pre-sorting operation on the Gaussian spheres under the target pose to obtain the pre-sorted Gaussian sphere result. This application can reuse the pre-sorted Gaussian sphere result to perform sorted shared rendering on multiple rendering frames within the target pose to obtain sorted rendering results. This application can share the sorting result across frames, greatly reducing the sorting rendering time of rendering frames.

[0017] This application determines a salient Gaussian based on the Gaussian in the current pose and the Gaussian in the target pose; it caches the salient Gaussian based on the sorted rendering results, and performs color integration on the salient Gaussian to obtain the target image. By caching the salient Gaussian, this application can reduce the color integration calculation in rasterization and improve the computational efficiency of rasterization. Attached Figure Description

[0018] Figure 1A The flowchart shown is a representation of the corresponding 3DGS technology.

[0019] Figure 1B This is a graph showing the percentage of normalized execution time across different scenarios in some technologies.

[0020] Figure 1CThis diagram illustrates the significant Gaussian-to-pixel correspondence provided by some technologies.

[0021] Figure 1D This shows a schematic diagram of the corresponding thread beam divergence during the rasterization process provided by some technologies.

[0022] Figure 2A The diagram shown is an application scenario diagram corresponding to the image rendering method provided in an embodiment of this application.

[0023] Figure 2B The flowchart shown is an embodiment of an image rendering method provided in this application.

[0024] Figure 3 The flowchart shown is a process for determining the sorting and rendering results provided in an embodiment of this application.

[0025] Figure 4 The flowchart shown is a process for predicting the pose of a target provided in an embodiment of this application.

[0026] Figure 5 The flowchart shown is a process for determining the Gaussian sigma to be sorted under the target pose, as provided in an embodiment of this application.

[0027] Figure 6 The flowchart shown is a process for determining a significant Gaussian in one embodiment of this application.

[0028] Figure 7 The diagram shown is a schematic representation of determining a significant Gaussian according to an embodiment of this application.

[0029] Figure 8 The flowchart shown is a process for determining a target image according to an embodiment of this application.

[0030] Figure 9 The diagram shown is an algorithm flowchart corresponding to the RC mechanism provided in one embodiment of this application.

[0031] Figure 10 The diagram shown illustrates the hardware support for a radiation cache provided in one embodiment of this application.

[0032] Figure 11 The flowchart shown is a process for determining the pre-sorting results of Gaussians according to an embodiment of this application.

[0033] Figure 12A This application illustrates a hardware-algorithm co-design system provided in one embodiment.

[0034] Figure 12B The diagram shown is a schematic diagram of the execution logic corresponding to the S2 algorithm provided in an embodiment of this application.

[0035] Figure 13The diagram shown is a SoC architecture diagram provided in one embodiment of this application.

[0036] Figure 14 The diagram shown is a schematic representation of the structure of an electronic device according to an embodiment of this application.

[0037] Component designation explanation

[0038] 140 Electronic devices

[0039] 141 processor

[0040] 142 Non-volatile storage media

[0041] 143 System Bus

[0042] 144 Internal Memory

[0043] 145 Network Interface Detailed Implementation

[0044] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0045] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0046] In existing technologies, 3DGS proposes a reverse operation called "splatting," which projects Gaussian points (or simply "Gaussians") directly onto the rendering screen. This method avoids the computationally intensive ray-object intersection task by reversing the workflow: instead of rays finding Gaussians, Gaussians are directly mapped onto the screen. To date, all 3DGS variants follow the same rendering flow. The main difference between different 3DGS algorithms lies in the training method, not the rendering flow.

[0047] Please see Figure 1A , Figure 1A This is displayed as a flowchart corresponding to 3DGS in some technologies. For example... Figure 1A As described, 3DGS performs rendering block by block in three steps: projection, sorting, and rasterization.

[0048] Projection: Given the camera pose, projection serves two main purposes: First, it filters out Gaussians (e.g., gray ellipsoids) that fall outside the view frustum, retaining only Gaussians (e.g., color ellipsoids) located between the near clipping plane and the far clipping plane, such as... Figure 1A As shown; second, each Gaussian (with a defined cutoff radius) is projected onto the screen to determine the blocks where they intersect.

[0049] Sorting: Once each block has collected the intersecting Gaussian IDs, the sorting step determines the rendering order of these Gaussians, ensuring that all points are rendered from near to far according to their depth (relative to the camera pose), as shown in the "Sorted SplattingTable".

[0050] Rasterization: Once all Gaussians are sorted, the rasterization step renders these Gaussians block by block. Each pixel within a block iterates over the same set of Gaussians, calculates the opacity, and integrates the colors of these Gaussians into its pixel in sorted order. For example, each pixel in block T0 integrates Gaussians in the order 4→2→1. The expression controlling the color integration of pixel p is:

[0051] in,

[0052] Among them, Γ i α represents the cumulative transmittance of pixel p from the first Gaussian to the (i-1)th Gaussian. i and c i Let represent the transparency and color of the i-th Gaussian element, respectively. If the Gaussian element's α... i If the value is less than 1 / 255, the Gaussian will be skipped to avoid numerical instability. Figure 1A As shown. When the cumulative transmittance Γ i Color integration terminates when the color temperature falls below a predefined threshold θ.

[0053] 3DGS simplifies the rendering process through pre-computed Gaussian points, but on mobile devices (such as Volta GPUs), it only achieves 5-21 FPS, far below the 90 FPS real-time requirement for AR / VR applications. The main bottleneck of 3DGS is that rendering time is primarily dominated by sorting and rasterization, accounting for 23% and 67% of the total execution time, respectively. Figure 1B As shown), Figure 1B As shown, where, Figure 1B The rendering time for projection in the image is relatively short. Figure 1BThe x-axis represents four datasets of increasing size: Synthetic Neural Radiation Field (S-NeRF), Tank and Temple (T&T), Deep Mixture (DB), and Unbounded 360 Degrees (U360). This shows that Sorting and Rasterization are the main time-consuming stages, together accounting for the majority of execution time. This phenomenon is consistent across all test scenarios (S-NeRF, DB, U360).

[0054] Sparse color integration: Since rasterization accounts for a large portion of the overall execution time, it further identifies a performance bottleneck. By default, pixels within a block are designed to iterate over the same set of Gaussians. However, the Gaussians only contribute to the color integration of the final pixel when their transparency α exceeds 1 / 255. We call these Gaussians significant Gaussians (SG).

[0055] Figure 1C This is a schematic diagram illustrating the significant Gaussian-to-pixel correspondence provided by some technologies. For example... Figure 3 As shown, Figure 1C The circles in the graph represent the average number of Gaussian iterations per pixel, and the boxes with horizontal lines represent the percentage of significant Gaussians per pixel. During rasterization, significant Gaussians accounted for only 10.3% (mean, standard deviation 2.1%) of the total Gaussians per pixel across the four datasets, indicating that each pixel was generated from only a small subset of Gaussians.

[0056] Figure 1D This diagram illustrates the thread beam divergence during the rasterization process provided by some technologies. Figure 1D As can be seen, during the rasterization process, taking one warp as an example, assuming each warp contains 4 threads. The middle box indicates that the threads are not masked and are performing meaningful computations. Between two computation cycles, all threads load data from global memory and synchronize. Figure 1D The sparse color integration displayed in the image resulted in severe warp divergence (WD).

[0057] because Figure 1D The sparsity of color integration means that different pixels may integrate different subsets of Gaussians. However, in typical GPU implementations, rasterization is parallelized across pixels, with each pixel assigned a thread. The workload differences between pixels can lead to low GPU utilization.

[0058] In existing GPUs, threads are grouped into warps, and all threads within a warp execute together in a single-instruction, multi-threaded manner. See also... Figure 1DMasked means masked / thread masking: During the execution of a GPU warp (thread bundle, typically 32 threads), some threads may be temporarily disabled due to conditional branches (such as if-else) or data dependencies; this is called "masking". The curved part represents the masked thread (does not perform computations, the hardware skips its instructions). The square part represents the unmasked thread (active thread, performing valid computations). Threadblock-sync: Explicit synchronization point for all threads within a thread block.

[0059] Figure 1D The example illustrates a simplified GPU execution model with four threads within a warp. Each row of boxes represents the activity of one thread during rendering. Color integration is interleaved with data synchronization (i.e., fetching Gaussians from global memory to shared memory). Since each thread requires a different Gaussian, the GPU will mask threads that do not need to integrate a specific Gaussian at a given time. Results show that threads are masked for an average of over 69% of the time across different scenes, with a standard deviation of 10%, indicating low GPU utilization and unsuitability for rasterization. Although existing research proposes methods to mitigate warp diffusion, such as dynamic warp formation and dynamic warp subdivision, these techniques rely on dynamic scheduling and synchronization, introducing additional runtime overhead and making them difficult to apply to 3DGS rasterization.

[0060] At least to address the aforementioned problems, embodiments of this application provide an image rendering method. This image rendering method can determine the current pose; predict the target pose based on the current pose; determine the sorted rendering result corresponding to the target pose; determine a salient Gaussian based on the Gaussian of the current pose and the Gaussian of the target pose; cache the salient Gaussian based on the sorted rendering result, and perform color integration on the salient Gaussian to obtain the target image; thus solving the technical problem of the lack of efficient image rendering methods in the prior art.

[0061] Figure 2A The diagram shown illustrates an application scenario corresponding to the image rendering method provided in an embodiment of this application. For example... Figure 2A As shown, this application scenario includes a camera and an electronic device. The camera is communicatively connected to the electronic device, and is used to move its pose and send the current pose and target pose to the electronic device. The electronic device includes a Lumina system, which performs a series of data processing steps on the current pose and target pose to obtain the target image.

[0062] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0063] Figure 2BThe flowchart shown is an embodiment of an image rendering method provided in this application. Figure 2B As shown, the image rendering method provided in this application embodiment includes the following steps S21 to S25.

[0064] S21, Obtain the current posture of the image acquisition device.

[0065] For example, the current pose is obtained based on the camera's current position and orientation.

[0066] The camera's current position and orientation can be obtained using a preset coordinate system.

[0067] It should be noted that the aforementioned preset coordinate system is pre-set, and this application does not restrict the method of setting the preset coordinate system.

[0068] S22, predict the target posture based on the current posture and current motion trend of the image acquisition device.

[0069] Among them, the target attitude can be predicted based on a preset time interval.

[0070] For example, the preset time interval can be the time interval between adjacent frames or a preset number of time intervals between frames.

[0071] It should be noted that in actual application scenarios, any suitable time interval can be selected as the preset time interval according to specific application requirements, and this application does not impose any restrictions on this.

[0072] For example, the target pose within a preset time interval is predicted based on the camera's operating trend information and motion trajectory information.

[0073] Another example is the prediction of target pose within a preset time interval based on techniques such as motion models, filters, and deep learning. The motion models include constant velocity models and constant acceleration models; the filters include Kalman filters, which recursively predict the target pose of the camera by combining the motion model and observation data.

[0074] It should be noted that the above-described techniques for predicting target pose are merely illustrative. In practical applications, other suitable methods can be selected to predict target pose according to specific application scenarios, and this application does not impose any restrictions on this.

[0075] S23, determine the sorted rendering result corresponding to the target pose.

[0076] S24, determine a significant Gaussian based on the Gaussian in the current pose and the Gaussian in the target pose.

[0077] S25, based on the sorted rendering result, cache the significant Gaussian, and perform color integration on the significant Gaussian to obtain the target image.

[0078] In an image rendering method provided in this application embodiment, the sorting and rendering time of rendering frames is greatly reduced by pre-sorting Gaussians; and the target image is obtained by color integration by caching significant Gaussians, which greatly reduces the color integration calculation in rasterization and improves the computational efficiency of the rasterization step.

[0079] Figure 3 The flowchart shown is a method for determining the sorting and rendering results provided in an embodiment of this application, such as... Figure 3 As shown, the process of determining the sorting rendering result in this embodiment includes the following steps S31 to S33.

[0080] S31, determine the Gaussian sigma to be sorted under the target pose.

[0081] For example, a filtering operation is performed on all Gaussians corresponding to the target pose to obtain Gaussians to be sorted.

[0082] S32, perform a pre-sorting operation on the Gaussian to obtain the pre-sorting result of the Gaussian.

[0083] For example, Gaussian is pre-sorted based on a preset sorting method to obtain the pre-sorted result of Gaussian.

[0084] For example, the preset sorting methods include sorting based on distance from nearest to farthest, sorting based on distance from farthest to nearest, etc.

[0085] It should be noted that the above-mentioned preset sorting rules are merely illustrative. In practical applications, other suitable sorting methods can be selected according to specific application requirements, and this application does not impose any restrictions on this.

[0086] S33, based on the pre-sorting result, perform sorted shared rendering on multiple rendering frames within the target pose in sequence to obtain sorted rendering result.

[0087] In the method for determining the sorted rendering results provided in the embodiments of this application, the pre-sorted results are reused for multiple rendering frames within the target pose. This eliminates the step in 3DGS where the sorting step determines the rendering order of Gaussians once each block has collected its intersecting Gaussian IDs. This ensures that all points are rendered from near to far according to their depth (relative to the camera pose), greatly reducing the time required to sort the Gaussians.

[0088] Figure 4 The flowchart shown is a prediction of target pose provided in an embodiment of this application, as follows: Figure 4As shown, the process of predicting the target pose in this embodiment includes the following steps S41 to S42.

[0089] S41, determine the current position information corresponding to the current posture of the image acquisition device and the current speed information corresponding to the current motion trend.

[0090] The current position information of the current attitude can be directly read through the preset coordinate system.

[0091] The expression corresponding to determining the speed information is: F j-1 F represents the complete camera pose of the previous frame. j This represents the complete camera pose of the current frame, and Δt represents the inter-frame time interval.

[0092] S42, based on the current position information and the current velocity information, predict the target attitude.

[0093] The expression for predicting the target pose is:

[0094] S k =T k +v×t r

[0095] Among them, S k T represents the target pose, N represents the number of rendering frames (i.e., shared windows) that share the same pre-sorted results, and T represents the target pose. k t represents the translation component in Fk. r Indicates the time radius (the predicted time span).

[0096] This application provides a method for predicting target pose. The method involves determining the current position and velocity information of the current pose; predicting the target pose based on the current position and velocity information, which quickly and accurately determines the target pose corresponding to the current pose. This provides a basis for subsequently determining the Gaussian plane under the target pose. The method performs sorted shared rendering on multiple rendering frames within the target pose based on the Gaussian plane to obtain a sorted rendering result. It also determines a significant Gaussian plane based on the Gaussian plane under the current pose and the Gaussian plane under the target pose. The significant Gaussian plane is cached based on the sorted rendering result, and color integration is performed on it to obtain an accurate camera pose for the target image.

[0097] Figure 5 This is a flowchart illustrating the process of determining the Gaussians to be sorted under the target pose, as provided in an embodiment of this application. For example... Figure 5 As shown, the process of adjusting the value of the force in this embodiment includes the following steps S51 to S52.

[0098] S51, determine the near clipping plane and far clipping plane of the target attitude projection.

[0099] The near and far clipping planes of the target attitude projection can be found in [reference needed]. Figure 1A The clipping plane closest to the camera is the near clipping plane, and the clipping plane farthest from the camera is the far clipping plane.

[0100] S52, the Gaussian located between the near clipping plane and the far clipping plane is determined as the Gaussian to be sorted.

[0101] This application provides a method for determining Gaussians to be sorted under a target pose. In this method, the near clipping plane and the far clipping plane of the target pose projection are determined; Gaussians located between the near clipping plane and the far clipping plane are determined as the Gaussians to be sorted. This method can quickly filter out Gaussians located outside the near clipping plane and the far clipping plane, effectively filtering the Gaussians and ensuring the accuracy of the obtained Gaussians to be sorted, thus providing accurate Gaussians for subsequent Gaussian-based color integration.

[0102] Figure 6 The flowchart shown is a process for determining a significant Gaussian in one embodiment of this application. For example... Figure 6 As shown, the process of determining significant Gaussians in this embodiment includes the following steps S61 to S63.

[0103] S61, determine the current Gaussian at the current pose of the image acquisition device.

[0104] S62, determine the intersecting Gaussian that intersects the current Gaussian with the Gaussian in the target pose.

[0105] S63, among the intersecting Gaussians, a preset number of Gaussians whose Gaussian values ​​are greater than a preset Gaussian threshold are identified as the significant Gaussians.

[0106] For example, the preset Gaussian threshold is 1 / 255, and the preset number can be 2 or 3, etc. In this embodiment, the first two Gaussian values ​​that are greater than the preset Gaussian threshold are determined as significant Gaussian values.

[0107] It should be noted that the preset Gaussian threshold and preset quantity are merely illustrative examples. In actual applications, any other suitable values ​​can be selected as the preset Gaussian threshold and preset quantity according to specific application requirements. This application does not impose any restrictions on this.

[0108] Please see Figure 7 , Figure 7 This illustration shows a schematic diagram of determining a significant Gaussian according to an embodiment of this application. Figure 7It can be seen that the Gaussians labeled 1, 3, 8, 9, 7, and 6 are the intersecting Gaussians when the posture N moves to the posture M. Among them, if the Gaussian values ​​of Gaussians labeled 8 and 6 are greater than the preset Gaussian threshold, then Gaussians labeled 8 and 6 can be selected as significant Gaussians.

[0109] The purpose of selecting a preset number of significant Gaussians includes: 1. Significant Gaussians contribute more to the final pixel value, thus serving as an effective indicator of ray similarity. 2. Using as few Gaussians as possible to index the cache avoids unnecessary color integration calculations. Although radiometric caching was proposed to accelerate 3DGS, it is not limited to 3DGS applications. The core idea of ​​radiometric caching stems from the basic concept of neural rendering: representing the scene using trainable primitives (such as Gaussians in 3DGS or voxels in NeRF) and performing color integration through the intersections of rays with these primitives. Therefore, even if new primitive forms develop in the field of neural rendering, as long as the basic rendering principle of the intersections of rays with these primitives remains valid, radiometric caching technology still has broad application value.

[0110] Please see Figure 8 , Figure 8 A flowchart illustrating the determination of a target image according to an embodiment of this application is shown. Figure 8 As shown,

[0111] The process of determining the target image in this embodiment includes the following steps S81 to S86.

[0112] S81, perform color integration on the first significant Gaussian to obtain the first pixel of the first significant Gaussian.

[0113] S82, store the first significant Gaussian and the first pixel into the corresponding radiation buffer list.

[0114] S83, when rendering the frame to be rendered, the second significant Gaussian corresponding to each pixel in the frame to be rendered is determined sequentially.

[0115] S84, if the second significant Gaussian matches the first significant Gaussian in the radiation buffer list, then the first pixel corresponding to the first significant Gaussian is determined as the second pixel of the second significant Gaussian.

[0116] For example, matching the second significant Gaussian with the first significant Gaussian in the radiation cache list includes: the second significant Gaussian is equal to the first significant Gaussian.

[0117] S85, if the second significant Gaussian does not match the first significant Gaussian in the radiation buffer list, then the second significant Gaussian is color integrated to obtain the second pixel, and the second significant Gaussian and the second pixel are stored in the radiation buffer list.

[0118] Specifically, the second significant Gaussian can be color integrated based on the above expression for color integration.

[0119] S86, Based on the first pixel and the second pixel, the target image is obtained.

[0120] For example, the first pixel and the second pixel are merged to obtain the target image.

[0121] In this embodiment, the initial intersection sequence of each ray is cached based on the Radiance Caching (RC) mechanism. The pixel value is quickly determined by matching the cached records, avoiding the color integration calculation of the entire process, optimizing the rasterization calculation process, and reducing the computational complexity of color integration.

[0122] Specifically, the RC algorithm flow corresponding to the RC mechanism can be obtained through... Figure 9 To illustrate, in the existing 3DGS algorithm, each pixel iterates through a set of Gaussians, integrating their contributions sequentially according to the color integral expression described above. In frame 0, since the radiance buffer is empty, the RC algorithm performs normal rasterization (the same precise rasterization as in existing 3DGS), calculating pixel values ​​from scratch. After calculating each pixel value, the first two intersecting significant Gaussians (i.e., 5, 16) are used as buffer tags, and the calculated pixel value (3, 32, 71) is used as the buffer value. Then, the radiance buffer list is updated, as shown below. Figure 9 As shown. After rendering the first frame and updating the radiation cache, the cache can be used to accelerate the rendering of subsequent frames. For example, when rendering frame k, the RC algorithm calculates the first five Gaussians to identify the first two salient Gaussians. Once the IDs of the salient Gaussians are identified, each pixel concatenates these two Gaussian IDs into a cache label and queries the radiation cache. If the label matches a salient Gaussian in the cache (e.g., pixels 2, 3, 4), these pixels will directly use the cached pixel value, skipping the remaining color integration calculation. For pixels that do not hit the cache (e.g., pixel 1), they still need to iterate through the remaining Gaussians to obtain the final pixel value. Once color integration is complete, the radiation cache updates the cached values ​​of these missing pixels according to its caching strategy.

[0123] Please see Figure 10 , Figure 10 The diagram shown illustrates the hardware support for a radiation cache provided in one embodiment of this application.

[0124] in, Figure 10 The detailed functional descriptions of each module in the hardware support for the radiation buffer are as follows:

[0125] Gaussian Record: Used to store key information for each Gaussian volume, including attributes such as ID, color (RGB), and transparency; and serves as the original data source for cached queries, providing Gaussian features to be matched.

[0126] Gaussian ID1 & ID2: Used to identify the top k salient Gaussian bodies currently being processed (k=2 in the diagram). The low-order bits (index) and high-order bits (label) of the corresponding Gaussian body are extracted using these IDs.

[0127] Combined Tag: Used to concatenate the high bits of Gaussian ID1 and ID2 into a composite tag. This tag is then compared with tags stored in the cache to verify a cache hit.

[0128] Index: Generated by combining the low bits of Gaussian ID1 and ID2, it directly maps to the cache address. It can quickly locate candidate data blocks in the cache, reducing the search range.

[0129] Broadcast: Distributes the generated indexes and tags to all cache units (M Sets). Multiple cache sets can be queried in parallel, improving access efficiency.

[0130] Hit? (Hit Check): Compares the composite label corresponding to the second significant Gaussian with the label in the radiation cache storage list. If they match, the cached value is returned (Cache Hit); otherwise, it needs to be recalculated (Cache Miss).

[0131] M Sets (Multi-way Cache Sets): Multi-way set-associative caches in hardware (such as 4-way / 8-way), each storing data;

[0132] Tag: High-bit identifier of the Gaussian body.

[0133] RGB: Cached Gaussian color values.

[0134] CachedRGB: The final cached pixel color results. Parallel matching of labels and indices accelerates the reuse of radiometric data.

[0135] The scope of protection for the image rendering method of this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0136] Figure 11 This is a flowchart illustrating the determination of the pre-sorting results of Gaussians according to an embodiment of this application. For example... Figure 11As shown, the process of determining the pre-sorting result of Gauss in this embodiment includes the following steps S111 to S112.

[0137] S111, determine the distance information between the target pose and the Gaussian.

[0138] Specifically, the distance information between the target pose and the Gaussian is determined sequentially.

[0139] S112, Based on the distance information, determine the pre-sorting result of the Gaussian.

[0140] For example, the pre-sorting result of Gaussians can be determined based on the distance between the target pose and each Gaussian from closest to furthest.

[0141] This application provides a method for determining the pre-sorting result of Gaussians. The method can determine the distance information between the target pose and the Gaussians, and determine the pre-sorting result of the Gaussians based on the distance information. Through pre-sorting, Gaussians that are far away from the target pose can be excluded again, thereby reducing the burden of subsequent calculations. When processing large-scale datasets, pre-sorting can improve the overall performance of the algorithm.

[0142] This application also provides a hardware-algorithm co-design system, namely the Lumina system, including a camera pose determination unit configured to determine the current pose and predict the target pose based on the current pose; a sorting and sharing unit configured to determine the sorted rendering result corresponding to the target pose; a salient Gaussian determination unit configured to determine a salient Gaussian based on the Gaussian under the current pose and the Gaussian under the target pose; and a radiation cache unit configured to cache the salient Gaussian based on the sorted rendering result, and perform color integration on the salient Gaussian to obtain the target image. The Lumina system also includes a hardware acceleration unit, wherein the hardware acceleration unit is configured with a hardware accelerator, which includes a neural rendering unit and a hardware cache.

[0143] Lumina refers to the use of computational redundancy to achieve real-time mobile neural rendering.

[0144] Figure 12A The embodiment shown in this application provides a hardware-algorithm co-design system: the Lumina system, which consists of... Figure 12A As can be seen, the Lumina system includes Sorting Sharing, Radiance Caching, and Co-training. In the Sorting Sharing step, based on the Sorting Sharing (S2) algorithm, the sorting results are shared across frames by utilizing temporal redundancy in rendering. The execution logic of the S2 algorithm can be found in [reference needed]. Figure 12BIn the radiative caching step, color integration calculations in rasterization are reduced by caching ray-Gaussian intersections (i.e., significant Gaussians); the target image is obtained in the co-training step. The Lumina system also includes a hardware accelerator (LuminCore), which includes a neural rendering unit and a hardware cache to address the inefficiency of GPUs in sparse rasterization.

[0145] The Lumina system can implement the image rendering method of this application, but the system implementing the image rendering method of this application includes, but is not limited to, the structure of the Lumina system listed in this embodiment. Any structural modifications and substitutions of the prior art made in accordance with the principles of this application are included within the protection scope of this application.

[0146] The Lumina system architecture is integrated into the mobile SoC and specifically supports cache rasterization. Please refer to [link / reference]. Figure 13 , Figure 13 This SoC architecture, comprising two main components, is showcased: the mobile GPU and the LuminCore. During rendering, the rasterization task is handled by the LuminCore, while the mobile GPU processes projection and sorting operations. The LuminCore distinguishes itself from baseline hardware by including a double-buffered feature buffer and an output buffer, used to store Gaussian features and output pixel values, respectively. Simultaneously, the LuminCore is connected to a local cache, LuminCache, specifically designed for radiation caching. LuminCache is shared by multiple Neural Render Units (NRUs) to accelerate the rasterization process.

[0147] The neural rendering unit is divided into two parts: the front end and the back end. The front end consists of a set of processing units (PEs) responsible for calculating the transparency of all Gaussians and using a comparator to filter out Gaussians with significantly high transparency (i.e., α > 1 / 255), inserting them into a FIFO queue implemented by shift registers. The back end is responsible for calculating color integrals, specifically handling significant Gaussians. The back end is designed to be shared among multiple PEs to ensure full utilization of hardware resources when processing sparse color integrals. The back end also contains a set of register files (α-records) used to cache the significant Gaussian IDs of different pixels to support fast lookup of the radiometric cache.

[0148] To improve PE utilization, NRU allows for reconfiguration, enabling all PEs to work collaboratively when rendering a single pixel. In this mode, all PEs read different Gaussians from a sorted list of Gaussians and sequentially write the intermediate results of each Gaussian to a shift register. When the Gaussian transparency reaches a threshold requiring further backend processing, the backend performs color integration in normal mode. This avoids the problem of insufficient PE utilization when some pixels terminate prematurely due to radiosity buffering, thus improving overall rendering efficiency.

[0149] LuminCore communicates with the GPU through SoC-level interconnect, supports DMA data transfer, and optimizes rendering performance.

[0150] In the embodiments provided in this application, it should be understood that the disclosed systems or methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or modules or units, and may be electrical, mechanical, or other forms.

[0151] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0152] Those skilled in the art will further recognize that the units and steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0153] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0154] This application also provides an electronic device. Figure 14 The diagram shown is a structural schematic of an electronic device 140 according to an embodiment of this application. The image rendering method provided in this embodiment can be applied to... Figure 14 The electronic device 140 shown is an example, but not a limitation thereof. For example... Figure 14 As shown, the electronic device 140 includes a processor 141, a memory, a system bus 143, and a network interface 145. The memory may include a non-volatile storage medium 142 and internal memory 144.

[0155] The non-volatile storage medium 142 can store an operating system and a computer program. The computer program includes program instructions that, when executed, cause the processor to perform any of the image rendering methods provided in the embodiments of this application.

[0156] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0157] The internal memory 144 provides an environment for the execution of a computer program in a non-volatile storage medium. When the computer program is executed by the processor, it enables the processor to execute any of the image rendering methods provided in the embodiments of this application.

[0158] This network interface 145 is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 14The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0159] It should be understood that processor 141 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, the general-purpose processor can be a microprocessor or any conventional processor.

[0160] The electronic device 140 in this application embodiment may include terminal devices such as supercomputers, and may also be applied to databases, servers, and service response systems based on terminal artificial intelligence. This application embodiment does not impose any restrictions on the specific type of electronic device.

[0161] For example, electronic devices can be handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, computers, laptops, handheld communication devices, handheld computing devices, and / or other devices for communicating over wireless systems, as well as next-generation communication systems, such as mobile terminals in 5G networks, mobile terminals in future evolved Public Land Mobile Networks (PLMNs), or mobile terminals in future evolved Non-terrestrial Networks (NTNs).

[0162] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0163] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. An image rendering method, characterized in that, The method includes: Obtain the current orientation of the image acquisition device; Predict the target posture based on the current posture and current motion trend of the image acquisition device; Determine the sorted rendering result corresponding to the target pose; Determining the sorted rendering result corresponding to the target pose includes: determining the Gaussian spheroids to be sorted under the target pose; performing a pre-sorting operation on the Gaussian spheroids to obtain the pre-sorting result of the Gaussian spheroids; and performing sorted shared rendering on multiple rendering frames within the target pose sequentially based on the pre-sorting result to obtain the sorted rendering result. A significant Gaussian is determined based on the Gaussian in the current pose and the Gaussian in the target pose. The step of determining significant Gaussians based on the Gaussians in the current pose and the Gaussians in the target pose includes: determining the current Gaussians in the current pose of the image acquisition device; determining the intersecting Gaussians that intersect the current Gaussians with the Gaussians in the target pose; and determining the significant Gaussians among the intersecting Gaussians whose Gaussian values ​​are greater than a preset Gaussian threshold. The significant Gaussian is cached based on the sorted rendering result, and color integration is performed on the significant Gaussian to obtain the target image; The salient Gaussian includes a first salient Gaussian and a second salient Gaussian. The step of caching the salient Gaussian based on the rendering result to perform color integration on the salient Gaussian to obtain the target image includes: performing color integration on the first salient Gaussian to obtain a first pixel of the first salient Gaussian; storing the first salient Gaussian and the first pixel in a corresponding radiative buffer list; when rendering a frame to be rendered, sequentially determining the second salient Gaussian corresponding to each pixel in the frame to be rendered; if the second salient Gaussian matches the first salient Gaussian in the radiative buffer list, then determining the first pixel corresponding to the first salient Gaussian as the second pixel of the second salient Gaussian; if the second salient Gaussian does not match the first salient Gaussian in the radiative buffer list, then performing color integration on the second salient Gaussian to obtain a second pixel, and storing the second salient Gaussian and the second pixel in the radiative buffer list; and obtaining the target image based on the first pixel and the second pixel.

2. The image rendering method according to claim 1, characterized in that, The prediction of the target pose based on the current pose and current motion trend of the image acquisition device includes: Determine the current position information corresponding to the current posture of the image acquisition device and the current velocity information corresponding to the current motion trend; Based on the current position information and the current velocity information, the target attitude is predicted.

3. The image rendering method according to claim 1, characterized in that, Determining the Gaussians to be sorted under the target pose includes: Determine the near clipping plane and far clipping plane of the target pose projection; The Gaussian located between the near clipping plane and the far clipping plane is determined as the Gaussian to be sorted.

4. The image rendering method according to claim 1, characterized in that, The pre-sorting operation on the Gaussians to obtain the pre-sorted Gaussians includes: Determine the distance information between the target pose and the Gaussian; Based on the distance information, the pre-sorting result of the Gaussian is determined.

5. A hardware algorithm co-design system, characterized in that: The system includes: The camera pose determination unit is configured to determine the current pose of the image acquisition device and predict the target pose based on the current pose of the image acquisition device. The sorting sharing unit is configured to determine the sorting rendering result corresponding to the target pose; the determination of the sorting rendering result corresponding to the target pose includes: determining the Gaussian spheroids to be sorted under the target pose; performing a pre-sorting operation on the Gaussian spheroids to obtain the pre-sorting result of the Gaussian spheroids; and performing sorting sharing rendering sequentially on multiple rendering frames within the target pose based on the pre-sorting result to obtain the sorting rendering result. A significant Gaussian determination unit is configured to determine a significant Gaussian based on the Gaussian in the current pose of the image acquisition device and the Gaussian in the target pose; the determination of the significant Gaussian based on the Gaussian in the current pose and the Gaussian in the target pose includes: determining the current Gaussian in the current pose of the image acquisition device; determining the intersecting Gaussians that intersect the current Gaussian with the Gaussian in the target pose; and determining a preset number of Gaussians whose Gaussian values ​​are greater than a preset Gaussian threshold as the significant Gaussians. A radiation cache unit is configured to cache the salient Gaussian based on the sorted rendering result, and to perform color integration on the salient Gaussian to obtain a target image. The salient Gaussian includes a first salient Gaussian and a second salient Gaussian. Cache the salient Gaussian based on the rendering result and perform color integration on the salient Gaussian to obtain the target image, including: performing color integration on the first salient Gaussian to obtain a first pixel of the first salient Gaussian; storing the first salient Gaussian and the first pixel in a corresponding radiation cache list; when rendering a frame to be rendered, sequentially determining the second salient Gaussian corresponding to each pixel in the frame to be rendered; if the second salient Gaussian matches the first salient Gaussian in the radiation cache list, then determining the first pixel corresponding to the first salient Gaussian as the second pixel of the second salient Gaussian; if the second salient Gaussian does not match the first salient Gaussian in the radiation cache list, then performing color integration on the second salient Gaussian to obtain a second pixel, and storing the second salient Gaussian and the second pixel in the radiation cache list; and obtaining the target image based on the first pixel and the second pixel.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

7. An electronic device, characterized in that, The electronic device includes: A memory that stores a computer program; The processor, which is communicatively connected to the memory, executes the method of any one of claims 1 to 4 when the computer program is invoked.

Images