Image rendering method and device, electronic equipment and storage medium

Abstract

The application provides an image rendering method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining an initial image; determining any view point of the initial image and at least one view angle of the any view point; generating at least one view angle image corresponding to the at least one view angle; performing image interweaving on the at least one view angle image according to a light ray interweaving algorithm to obtain an interwoven image; and performing reconstruction inference on the interwoven image by using a neural radiation field model to obtain a target image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image rendering method, apparatus, electronic device and storage medium. Background Technology

[0002] Neural Radiance Fields (NeRF) is a deep learning-based 3D reconstruction technique that generates continuous, high-quality 3D scene representations from a set of sparse viewpoint images. By modeling the light propagation process of a scene, NeRF can render new viewpoint images, making it widely popular in computer vision and graphics.

[0003] However, while NeRF can render images from any viewpoint, as resolution and user demands for detail increase, generating too many viewpoint images will drastically increase the time required to generate 3D images, resulting in low efficiency. At the same time, generating enough viewpoint images to support smooth 3D effects and interweaving these images without perceptible latency is a technical challenge. Summary of the Invention

[0004] In view of this, this application proposes an image rendering method, apparatus, electronic device, and storage medium to solve or partially solve the above-mentioned problems.

[0005] In view of the above objectives, firstly, this application provides an image rendering method, comprising:

[0006] Get the initial image;

[0007] Determine any viewpoint of the initial image, and determine at least one perspective of the any viewpoint;

[0008] Generate at least one viewpoint image corresponding to the at least one viewpoint;

[0009] The at least one viewpoint image is interlaced using a ray interlacing algorithm to obtain an interlaced image;

[0010] The target image is obtained by reconstructing and inferring from the interwoven image using the neural radiation field model.

[0011] In some exemplary embodiments, after determining at least one viewpoint of any viewpoint, the method further includes:

[0012] Determine whether the initial image contains a background;

[0013] In response to the inclusion of a background, the initial image is segmented using an image segmentation algorithm to generate a subject image and a background image, which are then processed separately.

[0014] In some exemplary embodiments, generating at least one viewpoint image corresponding to the at least one viewpoint includes:

[0015] For the subject image, generate at least one subject viewpoint image corresponding to the at least one viewpoint;

[0016] For the background image, at least one background view image corresponding to the at least one view is generated using an image transpose algorithm.

[0017] In some exemplary embodiments, generating at least one viewpoint image corresponding to the at least one viewpoint further includes:

[0018] Determine the position of the main subject in the main image;

[0019] Determine the initial center point of the at least one viewpoint;

[0020] For the subject image, the initial center point is adjusted to obtain a first center point, and the at least one subject viewpoint image is generated based on the first center point; wherein, the first center point is located on the side of the subject position that is far away from any viewpoint;

[0021] For the background image, the initial center point is adjusted to obtain a second center point, and the at least one background view image is generated based on the second center point; wherein, the second center point is located on the side of the subject position closer to any viewpoint.

[0022] In some exemplary embodiments, the step of reconstructing and inferring the interwoven image using the neural radiation field model to obtain the target image includes:

[0023] Using the neural radiation field model, the interlaced images corresponding to at least one subject viewpoint image and the interlaced images corresponding to at least one background viewpoint image are reconstructed and inferred to obtain the inferred subject image and the inferred background image.

[0024] The inferred subject image and the inferred background image are merged to generate the target image.

[0025] In some exemplary embodiments, the image merging of the inferred subject image and the inferred background image includes:

[0026] Determine the depth information and light interlacing relationship of the inferred subject image and the inferred background image;

[0027] Based on the depth information and the light interlacing relationship, the inferred subject image and the inferred background image are superimposed.

[0028] In some exemplary embodiments, the overlay process of the inferred subject image and the inferred background image includes:

[0029] During the overlay process, the inferred subject image and the inferred background image are subjected to at least one global illumination and color adjustment.

[0030] In some exemplary embodiments, the step of reconstructing and inferring the interwoven image using the neural radiation field model to obtain the target image includes:

[0031] The interlaced images corresponding to the at least one subject viewpoint image and the interlaced images corresponding to the at least one background viewpoint image are merged to obtain at least one merged interlaced image;

[0032] The target image is obtained by reconstructing and inferring from the at least one merged interwoven image using the neural radiation field model.

[0033] In some exemplary embodiments, the neural radiation field model is a neural radiation field model adjusted using the Instant-NGP module, and the adjustment process includes:

[0034] Replace the trigonometric function position encoding of the neural radiation field model with the hash encoding of the Instant-NGP module;

[0035] The multilayer perceptron of the neural radiation field model is adjusted according to the structural design of the Instant-NGP module;

[0036] The dense sampling strategy of the neural radiation field model is replaced with the hierarchical sampling strategy of the Instant-NGP module.

[0037] In some exemplary embodiments, the neural radiation field model adjusted using the Instant-NGP module incorporates the block optimization strategy of the Instant-NGP module during training, thereby accelerating the parameter convergence speed.

[0038] Based on the same concept, in a second aspect, this application also provides an image rendering apparatus, comprising:

[0039] The first module is used to acquire the initial image;

[0040] The second module is used to determine any viewpoint of the initial image and to determine at least one perspective of the any viewpoint;

[0041] The third module is used to generate at least one viewpoint image corresponding to the at least one viewpoint;

[0042] The fourth module is used to perform image interlacing on the at least one viewpoint image according to the ray interlacing algorithm to obtain an interlaced image;

[0043] The fifth module is used to reconstruct and infer the interwoven image using the neural radiation field model to obtain the target image.

[0044] Based on the same concept, in a third aspect, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect above.

[0045] Based on the same concept, in a fourth aspect, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to implement the method described in the first aspect above.

[0046] As described above, this application provides an image rendering method, apparatus, electronic device, and storage medium. By adjusting the pipeline mechanism during the 3D rendering process, this application modifies the order of reconstruction inference using the neural radiation field model and image interleaving, implementing a 3D image rendering generation method that interleaves first and then infers. This reduces the process of performing neural radiation field model reconstruction inference for each viewpoint image to only one reconstruction inference for the interleaved image, significantly shortening the processing time. In actual tests, the processing time for generating a 3D video of an observed object was reduced from more than one hour to less than 5 minutes. Thus, while maintaining image quality, this application greatly improves image rendering efficiency and significantly enhances the user experience. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the initial image provided for an embodiment of this application.

[0049] Figure 2 This is a schematic diagram illustrating a process for implementing 3D rendering, as provided in an embodiment of this application.

[0050] Figure 3 This is a schematic diagram illustrating image interleaving provided in an embodiment of this application.

[0051] Figure 4 A schematic diagram comparing the relationship between NeRF model reconstruction inference and ray interlacing before and after in different embodiments provided in this application.

[0052] Figure 5 This is a schematic diagram illustrating the effects of different center point positions provided in the embodiments of this application.

[0053] Figure 6 This is a schematic diagram illustrating the process of compositing a background image and a main image as provided in the embodiments of this application.

[0054] Figure 7 A flowchart illustrating an exemplary method provided in an embodiment of this application.

[0055] Figure 8 A schematic diagram of the structure of an exemplary device provided in an embodiment of this application.

[0056] Figure 9 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this specification clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0058] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element, object, or method step preceding the term covers the element, object, or method step listed after the term and its equivalents, without excluding other elements, objects, or method steps. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0059] As described in the background section, glasses-free 3D technology typically requires rendering scenes from multiple different perspectives to create 3D effects that can be viewed from various angles. NeRF model reconstruction, a novel perspective synthesis technique, can generate viewing images from any viewing angle, providing a foundational technology for glasses-free 3D. However, on a 3D screen, several images from different perspectives need to be generated for each viewing angle; for example, a 49-viewpoint 3D photo album requires generating 49 viewpoint images for each viewing angle (viewpoint).

[0060] In some embodiments, such as Figure 1 The image shown is an initial image for naked-eye 3D rendering. After obtaining this image, the 3D viewpoints need to be determined. For example, if we consider a 360° circle around the main object in the image, we can set 360 viewpoints. Of course, the number of viewpoints can be adjusted according to different scene requirements. For example, if each 1° corresponds to two viewpoints, then we can set 720 viewpoints; there is no specific limitation on this. Then, for each viewpoint, a corresponding observation angle, or field of view, will be set according to the specific scene. For example, 49 field of view can be set. To facilitate understanding, viewpoints and field of view can be compared. A viewpoint can be simply understood as observing the main object from different positions on a plane, while a field of view is observing the main object from different heights from a fixed viewpoint, thus providing the image basis for forming the final 3D image. Of course, viewpoints and field of view can be defined differently in different scenes. For example, in some embodiments, the concept of viewpoint remains unchanged, while the field of view can be 49 observation angles from left to right centered on the current viewpoint, etc. The specific settings can be determined according to the scene, and there is no specific limitation on this.

[0061] It should be noted that in this embodiment and subsequent embodiments, as... Figure 1 As shown, the illustrations use 360 ​​viewpoints surrounding the image and 49 perspectives at each viewpoint as examples.

[0062] In some embodiments, after obtaining an initial image, in order to generate a 3D image, viewpoints and the viewing angles of each viewpoint are determined. In this embodiment, there are 360 ​​viewpoints, each with 49 viewing angles. Then, as... Figure 2 The diagram shown illustrates the process of 3D rendering using a NeRF model in one embodiment. The process of NeRF model reconstruction for naked-eye 3D display involves 360 viewpoints around the object after NeRF model reconstruction. Each viewpoint requires 49 images from different perspectives. These 49 images are interwoven using an interlacing algorithm to create a 3D image, which can achieve a naked-eye 3D effect on the 3D screen.

[0063] Specifically, a NeRF (Neural Radiation Field) model can be used to generate viewpoint images for each viewpoint. With 49 viewpoints per viewpoint, this corresponds to 49 viewpoint images. The NeRF model is then used for reconstruction inference to obtain the inferred image for each viewpoint. Finally, these inferred images are interleaved to generate the final interleaved image, i.e., the target image. However, since each viewpoint requires rendering 49 viewpoint images, this embodiment requires generating 360*49 images. Actual testing showed that generating a 3D video around an object takes far more than an hour. This severely impacts the time required to generate 3D images, resulting in low efficiency.

[0064] To address the above issues, the applicant, through research, discovered that, firstly, based on the essence of NeRF model reconstruction, the input to the NeRF model is the position of light rays. From a fundamental perspective, the model's input is the 5D coordinates of the light rays, that is, the positions of the light ray's origin and destination in the world coordinate system. Secondly, the essence of 3D interlacing is the pixel rearrangement of images from multiple viewpoints. For example, 49-viewpoint interlacing is essentially rearranging the pixels of the rendered images from 49 viewpoints according to a certain pattern. Furthermore, based on these two fundamental principles, the pipeline mechanism of the previous embodiment can be adjusted. In other embodiments, such as... Figure 3 As shown, before reconstructing the NeRF model from a certain viewpoint, the light rays can be interwoven and rearranged to obtain an interwoven image (mainly including the interwoven light information). This interwoven image is then input into the NeRF model for rendering, directly yielding the interwoven 3D image. This significantly reduces the time consumption; in actual tests, generating a 3D video of the observed object's entire circumference takes less than 5 minutes. In other embodiments, such as... Figure 4 As shown, this is a schematic diagram comparing the relationship between NeRF model reconstruction inference and ray interweaving before and after in two embodiments. Figure 4 The left side of the image shows the solution of this embodiment, while the right side shows the solution of the previous embodiment. It can be seen that the previous embodiment's solution required 49 reconstruction inference steps before image interleaving of the 49 inferred images, due to the prior NeRF model reconstruction inference. In contrast, the solution of this embodiment performs image interleaving first, followed by reconstruction inference, thus requiring only one reconstruction inference step. Specifically, according to... Figure 4 As shown, the optimized pipeline only needs to go through one NeRF model reconstruction inference to directly output the interlaced image. This is because the 5D coordinates (x,y,z,u,v) of the light rays are interlaced and rearranged before the NeRF model reconstruction inference. In contrast, the pipeline in the previous embodiment requires 49 NeRF model inferences, which is very time-consuming.

[0065] Here, in the above embodiments, image interleaving can be performed using an image interleaving algorithm (or ray interleaving algorithm). The ray interleaving algorithm is mainly used in ray tracing to calculate the intersection of rays with objects in the scene.

[0066] It is important to emphasize that NeRF model inference relies on complete ray trajectories and their sampling points. The purpose of interlacing rays is to rearrange multi-viewpoint information to provide a structured output for subsequent display. Therefore, interlacing depends on multi-viewpoint images or depth information generated by the NeRF model. Furthermore, while the scheme in the previous embodiment is more easily extended to other scenes (such as dynamic scenes or complex textures), the scheme in the latter embodiment may require designing optimization strategies separately for each scene. Therefore, the latter embodiment itself has weaker scalability, requiring a dedicated optimization strategy for each scene adjustment. Considering this scalability, the order of NeRF model reconstruction inference and image interlacing is generally not interchanged. However, for fixed scenes, such as generating 3D advertisements with fixed scenes, if the optimization strategy is designed specifically, the processing efficiency of performing image interlacing first followed by NeRF model reconstruction inference will be significantly higher than that of performing NeRF model reconstruction inference first followed by image interlacing.

[0067] Furthermore, for the initial image, it might be as follows: Figure 1 As shown, it contains only one subject. In other embodiments, it may contain not only the subject but also the scene image of the scene in which the subject is located, i.e., the background image. For such images, not only the subject but also the background needs to be rendered to form a complete 3D image effect. Therefore, after obtaining an initial image, it can be determined whether it contains a background. If there is no background and only a subject, only the subject can be rendered directly; if there is a background, the subject and background need to be separated, processed separately, and then merged. Here, an image segmentation algorithm can be used to segment the initial image with a background into a subject image containing only the subject and a background image containing only the background. Among them, image segmentation algorithm is a key technology in the field of computer vision. It can divide an image into multiple meaningful parts or objects, specifically based on thresholds, regions, edges, etc.

[0068] Next, it should be emphasized that since the processing of the subject image after segmenting the initial image containing the background is the same as or similar to the processing of the initial image containing only the subject, the two will be combined in the following explanation.

[0069] Furthermore, for the subject image, the image corresponding to each viewpoint and perspective can be directly generated based on the NeRF model, i.e., the subject viewpoint image. For the background image, similarly, the corresponding background viewpoint image can be generated using an image transpose algorithm. This image transpose algorithm refers to interchanging the row and column coordinates of the image, that is, rearranging each row of data in the image into a new column. Specifically, it involves swapping the rows and columns of the image matrix, thereby changing the size and orientation of the image.

[0070] Furthermore, for an initial image with a background, it essentially sets a reference for the subject. In some scenarios, after conversion to a 3D image, this might make the background visually stronger than the subject, resulting in a "background overshadowing the subject" phenomenon. Therefore, to adjust the display effect of the subject and background in the initial image, making the subject more prominent, the subject and background images can be adjusted separately by adjusting the center point of the viewpoint. Specifically, for example... Figure 5 As shown, the 3D effect can be adjusted by adjusting the center points of 49 viewpoints. The center point is equivalent to the zero plane. If the center point is on the object, the 3D effect is not obvious; if the center point is behind the object, the out-of-screen effect is obvious; if the center point is in front of the object, the in-screen effect is obvious. It is important to emphasize that "in front" and "outside" here are relative to the screen or relative to the viewpoint. Being on the side of the object closer to the viewpoint means being in front of the object, and being on the side of the object farther from the viewpoint means being behind the object. Furthermore, to highlight the subject, the subject's position can be determined first, and an initial center point for the viewpoint can be established. Then, the subject image's initial center point is adjusted to be out of screen, that is, the initial center point is adjusted to the side of the subject's position farther from the viewpoint (the adjusted center point can be called the first center point). The subject image can then generate a corresponding subject viewpoint image based on the first center point. Similarly, the background image's initial center point is adjusted to be in-screen, that is, the initial center point is adjusted to the side of the subject's position closer to the viewpoint (the adjusted center point can be called the second center point). The background image can then generate a corresponding background viewpoint image based on the second center point.

[0071] It should be noted that in the process of generating the subject view image and the background view image, it is necessary not only to adjust the position of the center point, but also to ensure that the generated view is consistent with the rules of multi-viewpoint arrangement, so as to avoid pixel alignment problems during subsequent interweaving.

[0072] After obtaining the subject view image and the background view image, an image interleaving algorithm can be used to interleave the subject view image to obtain the corresponding interleaved image; similarly, the same algorithm can be used to interleave the background view image to obtain the corresponding interleaved image. Then, a neural radiation field model is used to reconstruct and infer the interleaved images corresponding to the subject view image and the background view image, respectively, to obtain the inferred subject image and the inferred background image. Finally, the inferred subject image and the inferred background image are merged to obtain the final target image. In specific applications, depth information and ray interleaving relationships can be used to merge the two images, or in other words, depth information and ray interleaving relationships can be used to superimpose the inferred subject image and the inferred background image to complete the merging. More specifically, depth mapping or disparity maps can be used for superposition processing.

[0073] Furthermore, in order to further optimize the visual effect of the target image, at least one global illumination and color adjustment can be performed on the inferred subject image and the inferred background image during the overlay process to improve the visual effect of the merged target image.

[0074] Subsequently, the steps of reconstruction inference using the NeRF model and merging the subject and background images can be performed in interchangeable orders or cross-operated, thereby improving computational efficiency and rendering quality according to specific application scenarios. That is, in some embodiments, the NeRF model can be used first to perform reconstruction inference separately, and then the inferred subject and background images can be merged. In other embodiments, the interlaced images corresponding to the subject viewpoint image and the interlaced images corresponding to the background viewpoint image can be merged first, and then the NeRF model can be used to perform reconstruction inference on the merged interlaced image.

[0075] In a more specific embodiment, such as Figure 6As shown, for 49 viewpoints, the background image undergoes an image transposition algorithm to generate 49 images from different viewpoints, i.e., background images. Simultaneously, corresponding to the subject image, images are composited to generate 49 composite images. Finally, these 49 composite images are interleaved and rearranged to generate an interleaved image, which can then be used for NeRF model reconstruction and inference. Specifically, the image transposition algorithm generates 49 background images from different viewpoints for each viewpoint, ensuring these viewpoint images cover all viewpoints from left to right. When generating object viewpoint images, each object image should be aligned with the corresponding background image viewpoint. Next, image compositing combines the background image and object image for each viewpoint. Compositing can be achieved through image overlay, transparently overlaying the object image onto the background image. During compositing, transparency handling is crucial to ensure seamless integration of the object image and background image. Finally, the interleaved image is generated: the 49 composite images are interleaved and rearranged according to viewpoint order to generate an interleaved image. The interleaving method can be designed according to the display device's requirements, typically alternating a column or row of pixels for each viewpoint. Through the above steps, the background image and the object image reconstructed by the NeRF model can be synthesized to generate a composite image from different viewpoints, and an interlaced image can be generated through interlacing and rearrangement.

[0076] In implementing the above embodiments, the applicant found that the NeRF model requires a large amount of computing resources for high-dimensional position encoding and multilayer perceptron (MLP) inference, and the global computational overhead of the NeRF model is too high. Therefore, in some embodiments, the NeRF model can be adjusted, and the fastest Instant-NGP module in the NeRF model can be reconstructed, that is, the NeRF model adjusted by using the Instant-NGP module. The Instant-NGP module mainly modifies the NeRF model, removes a part of the MLP network, and optimizes the position encoding. Specifically, the NeRF model adjusted by using the Instant-NGP module can be: (1) Replace the position encoding: replace the trigonometric function position encoding of the NeRF model with hash encoding. That is, in the NeRF model, the position information of each 3D spatial point is quickly encoded into a low-dimensional representation through a hash table. (2) Simplify the model structure: adjust the MLP network architecture of the NeRF model and adopt the lightweight design of the Instant-NGP module. The dense sampling strategy of the NeRF model is replaced with the hierarchical sampling of Instant-NGP, and high-precision calculations are only performed on key regions within the field of view. (3) Optimize the sampling strategy: Combine the multi-resolution hash grid of the Instant-NGP module to refine the sampling of high-density regions. That is, combine the compact data structure of the Instant-NGP module to optimize the use of video memory, so that the model can run efficiently on consumer-grade hardware.

[0077] In more specific scenarios, the process of adjusting the NeRF model using the Instant-NGP module can include (1) removing redundant parts. The MLP of a normal NeRF model usually consists of multiple fully connected layers, which is complex and computationally expensive. The Instant-NGP module addresses the performance bottleneck by significantly reducing the network size, retaining only a lightweight MLP for predicting color and density, and eliminating unnecessary complex layers. (2) optimizing position encoding. The normal NeRF model uses high-dimensional trigonometric functions (sin and cos) for position encoding, which is computationally expensive and tends to lead to high memory consumption. The Instant-NGP module introduces hash encoding, which maps the high-dimensional space to a compact low-dimensional representation, significantly improving computational efficiency and memory utilization. (3) improving sampling strategy. The normal NeRF model performs dense sampling of the entire field of view, which is inefficient. The Instant-NGP module uses a hierarchical hash grid to perform fine sampling only in high-density areas, thereby reducing the computational cost of irrelevant areas.

[0078] Comparing the NeRF model with the NeRF model modified by the Instant-NGP module, we can find that: (1) Efficiency is improved. The ordinary NeRF model requires a lot of computing resources for high-dimensional position encoding and MLP inference, while the hash encoding and simplified MLP of the Instant-NGP module significantly reduce the amount of computing power. At the same time, the Instant-NGP module introduces multi-resolution sampling, avoiding the global computing overhead of the ordinary NeRF model. (2) Real-time performance is enhanced. The rendering speed of the Instant-NGP module can reach tens of times that of the ordinary NeRF model, making real-time naked-eye 3D possible. (3) Balance between quality and smoothness. By optimizing position encoding and network structure, it can maintain high-fidelity image quality and achieve smooth switching of multiple perspectives. (4) Hardware threshold is reduced. The efficient implementation of the Instant-NGP module makes it suitable for devices with limited GPU resources, promoting the application of naked-eye 3D in consumer scenarios.

[0079] Finally, during the training and inference of the neural radiation field model tuned using the Instant-NGP module, the block optimization strategy of the Instant-NGP module is introduced during the training phase to accelerate parameter convergence. During the inference phase, efficient encoding and sampling mechanisms are used to achieve real-time rendering.

[0080] As can be seen, the Instant-NGP module, through its efficient 3D reconstruction and optimized image rendering technology, achieves unprecedented real-time performance and high-quality visual effects in naked-eye 3D. Utilizing a simplified MLP network and optimized positional encoding, the Instant-NGP module significantly improves computational efficiency and image quality, meeting the real-time and high-fidelity requirements of naked-eye 3D. The introduced hash encoding technology accelerates spatial information processing, making image display from different perspectives smoother and more stable. Furthermore, the flexibility and scalability of the Instant-NGP module make it suitable for various scenarios and devices, lowering the hardware barrier for naked-eye 3D technology and promoting its widespread adoption in the consumer market.

[0081] Figure 7 A flowchart illustrating an exemplary method provided in an embodiment of this application is shown.

[0082] like Figure 7 As shown in the embodiments of this application, the image rendering method proposed by example may specifically include the following steps.

[0083] Step 702: Obtain the initial image.

[0084] Step 704: Determine any viewpoint of the initial image and determine at least one perspective of the any viewpoint.

[0085] Step 706: Generate at least one viewpoint image corresponding to the at least one viewpoint.

[0086] Step 708: Perform image interleaving on the at least one viewpoint image according to the ray interleaving algorithm to obtain an interleaved image.

[0087] Step 710: Use the neural radiation field model to reconstruct and infer the interwoven image to obtain the target image.

[0088] In some exemplary embodiments, after determining at least one viewpoint of any viewpoint, the method further includes:

[0089] Determine whether the initial image contains a background;

[0090] In response to the inclusion of a background, the initial image is segmented using an image segmentation algorithm to generate a subject image and a background image, which are then processed separately.

[0091] In some exemplary embodiments, generating at least one viewpoint image corresponding to the at least one viewpoint includes:

[0092] For the subject image, generate at least one subject viewpoint image corresponding to the at least one viewpoint;

[0093] For the background image, at least one background view image corresponding to the at least one view is generated using an image transpose algorithm.

[0094] In some exemplary embodiments, generating at least one viewpoint image corresponding to the at least one viewpoint further includes:

[0095] Determine the position of the main subject in the main image;

[0096] Determine the initial center point of the at least one viewpoint;

[0097] For the subject image, the initial center point is adjusted to obtain a first center point, and the at least one subject viewpoint image is generated based on the first center point; wherein, the first center point is located on the side of the subject position that is far away from any viewpoint;

[0098] For the background image, the initial center point is adjusted to obtain a second center point, and the at least one background view image is generated based on the second center point; wherein, the second center point is located on the side of the subject position closer to any viewpoint.

[0099] In some exemplary embodiments, the step of reconstructing and inferring the interwoven image using the neural radiation field model to obtain the target image includes:

[0100] Using the neural radiation field model, the interlaced images corresponding to at least one subject viewpoint image and the interlaced images corresponding to at least one background viewpoint image are reconstructed and inferred to obtain the inferred subject image and the inferred background image.

[0101] The inferred subject image and the inferred background image are merged to generate the target image.

[0102] In some exemplary embodiments, the image merging of the inferred subject image and the inferred background image includes:

[0103] Determine the depth information and light interlacing relationship of the inferred subject image and the inferred background image;

[0104] Based on the depth information and the light interlacing relationship, the inferred subject image and the inferred background image are superimposed.

[0105] In some exemplary embodiments, the overlay process of the inferred subject image and the inferred background image includes:

[0106] During the overlay process, the inferred subject image and the inferred background image are subjected to at least one global illumination and color adjustment.

[0107] In some exemplary embodiments, the step of reconstructing and inferring the interwoven image using the neural radiation field model to obtain the target image includes:

[0108] The interlaced images corresponding to the at least one subject viewpoint image and the interlaced images corresponding to the at least one background viewpoint image are merged to obtain at least one merged interlaced image;

[0109] The target image is obtained by reconstructing and inferring from the at least one merged interwoven image using the neural radiation field model.

[0110] In some exemplary embodiments, the neural radiation field model is a neural radiation field model adjusted using the Instant-NGP module, and the adjustment process includes:

[0111] Replace the trigonometric function position encoding of the neural radiation field model with the hash encoding of the Instant-NGP module;

[0112] The multilayer perceptron of the neural radiation field model is adjusted according to the structural design of the Instant-NGP module;

[0113] The dense sampling strategy of the neural radiation field model is replaced with the hierarchical sampling strategy of the Instant-NGP module.

[0114] In some exemplary embodiments, the neural radiation field model adjusted using the Instant-NGP module incorporates the block optimization strategy of the Instant-NGP module during training, thereby accelerating the parameter convergence speed.

[0115] As can be seen from the above embodiments, this application provides an image rendering method. This application adjusts the pipeline mechanism during the 3D rendering process, changing the order of reconstruction inference using the neural radiation field model and image interleaving. It implements a 3D image rendering generation method that interleaves first and then infers, thereby reducing the process of reconstructing and inferring the neural radiation field model for each viewpoint image to only one reconstruction and inference process for the interleaved image. This significantly reduces processing time; in actual tests, the processing time for generating a 3D video of an observed object was reduced from more than one hour to less than 5 minutes. Thus, while maintaining image quality, it greatly improves image rendering efficiency and significantly enhances the user experience.

[0116] It should be noted that the method in this application embodiment can be executed by a single device, such as a computer or server. The method in this application embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the process. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this application embodiment, and the multiple devices will interact with each other to complete the method described.

[0117] It should be noted that the above description describes specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0118] Based on the same concept, corresponding to any of the above embodiments, this application also provides an image rendering apparatus.

[0119] refer to Figure 8 The image rendering apparatus includes:

[0120] The first module 810 is used to acquire the initial image.

[0121] The second module 820 is used to determine any viewpoint of the initial image and to determine at least one perspective of the any viewpoint.

[0122] The third module 830 is used to generate at least one viewpoint image corresponding to the at least one viewpoint.

[0123] The fourth module 840 is used to perform image interleaving on the at least one viewpoint image according to the ray interleaving algorithm to obtain an interleaved image.

[0124] The fifth module 850 is used to perform reconstruction reasoning on the interwoven image using the neural radiation field model to obtain the target image.

[0125] In some exemplary embodiments, the second module 820 is further configured to:

[0126] Determine whether the initial image contains a background;

[0127] In response to the inclusion of a background, the initial image is segmented using an image segmentation algorithm to generate a subject image and a background image, which are then processed separately.

[0128] In some exemplary embodiments, the third module 830 is further configured to:

[0129] For the subject image, generate at least one subject viewpoint image corresponding to the at least one viewpoint;

[0130] For the background image, at least one background view image corresponding to the at least one view is generated using an image transpose algorithm.

[0131] In some exemplary embodiments, the third module 830 is further configured to:

[0132] Determine the position of the main subject in the main image;

[0133] Determine the initial center point of the at least one viewpoint;

[0134] For the subject image, the initial center point is adjusted to obtain a first center point, and the at least one subject viewpoint image is generated based on the first center point; wherein, the first center point is located on the side of the subject position that is far away from any viewpoint;

[0135] For the background image, the initial center point is adjusted to obtain a second center point, and the at least one background view image is generated based on the second center point; wherein, the second center point is located on the side of the subject position closer to any viewpoint.

[0136] In some exemplary embodiments, the fifth module 850 is further configured to:

[0137] Using the neural radiation field model, the interlaced images corresponding to at least one subject viewpoint image and the interlaced images corresponding to at least one background viewpoint image are reconstructed and inferred to obtain the inferred subject image and the inferred background image.

[0138] The inferred subject image and the inferred background image are merged to generate the target image.

[0139] In some exemplary embodiments, the fifth module 850 is further configured to:

[0140] Determine the depth information and light interlacing relationship of the inferred subject image and the inferred background image;

[0141] Based on the depth information and the light interlacing relationship, the inferred subject image and the inferred background image are superimposed.

[0142] In some exemplary embodiments, the fifth module 850 is further configured to:

[0143] During the overlay process, the inferred subject image and the inferred background image are subjected to at least one global illumination and color adjustment.

[0144] In some exemplary embodiments, the fifth module 850 is further configured to:

[0145] The interlaced images corresponding to the at least one subject viewpoint image and the interlaced images corresponding to the at least one background viewpoint image are merged to obtain at least one merged interlaced image;

[0146] The target image is obtained by reconstructing and inferring from the at least one merged interwoven image using the neural radiation field model.

[0147] In some exemplary embodiments, the neural radiation field model is a neural radiation field model adjusted using the Instant-NGP module, and the adjustment process includes:

[0148] Replace the trigonometric function position encoding of the neural radiation field model with the hash encoding of the Instant-NGP module;

[0149] The multilayer perceptron of the neural radiation field model is adjusted according to the structural design of the Instant-NGP module;

[0150] The dense sampling strategy of the neural radiation field model is replaced with the hierarchical sampling strategy of the Instant-NGP module.

[0151] In some exemplary embodiments, the neural radiation field model adjusted using the Instant-NGP module incorporates the block optimization strategy of the Instant-NGP module during training, thereby accelerating the parameter convergence speed.

[0152] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0153] The apparatus described above is used to implement the corresponding image rendering methods in the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0154] Based on the same concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image rendering method as described in any of the above embodiments.

[0155] Figure 9 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0156] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0157] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0158] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0159] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0160] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0161] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0162] The electronic devices described above are used to implement the corresponding image rendering methods in any of the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0163] Based on the same concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the image rendering method as described in any of the above embodiments.

[0164] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0165] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the image rendering method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0166] Based on the same concept, corresponding to any of the above-described embodiments, this application also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processors to perform the image rendering method. Corresponding to the execution entity for each step in each embodiment of the image rendering method, the processor executing the corresponding step may belong to the corresponding execution entity.

[0167] The computer program products of the above embodiments are used to cause the computer and / or the processor to execute the image rendering method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0168] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0169] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0170] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0171] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. An image rendering method, characterized in that, include: Get the initial image; Determine any viewpoint of the initial image, and determine at least one perspective of the any viewpoint; Generate at least one viewpoint image corresponding to the at least one viewpoint; The at least one viewpoint image is interlaced using a ray interlacing algorithm to obtain an interlaced image; The target image is obtained by reconstructing and inferring from the interwoven image using the neural radiation field model.

2. The method according to claim 1, characterized in that, After determining at least one viewpoint from any given viewpoint, the method further includes: Determine whether the initial image contains a background; In response to the inclusion of a background, the initial image is segmented using an image segmentation algorithm to generate a subject image and a background image, which are then processed separately.

3. The method according to claim 2, characterized in that, Generating at least one viewpoint image corresponding to the at least one viewpoint includes: For the subject image, generate at least one subject viewpoint image corresponding to the at least one viewpoint; For the background image, at least one background view image corresponding to the at least one view is generated using an image transpose algorithm.

4. The method according to claim 3, characterized in that, The process of generating at least one viewpoint image corresponding to the at least one viewpoint further includes: Determine the position of the main subject in the main image; Determine the initial center point of the at least one viewpoint; For the subject image, the initial center point is adjusted to obtain a first center point, and the at least one subject viewpoint image is generated based on the first center point; wherein, the first center point is located on the side of the subject position that is far away from any viewpoint; For the background image, the initial center point is adjusted to obtain a second center point, and the at least one background view image is generated based on the second center point; wherein, the second center point is located on the side of the subject position closer to any viewpoint.

5. The method according to claim 3, characterized in that, The process of reconstructing and inferring the interwoven image using the neural radiation field model to obtain the target image includes: Using the neural radiation field model, the interlaced images corresponding to at least one subject viewpoint image and the interlaced images corresponding to at least one background viewpoint image are reconstructed and inferred to obtain the inferred subject image and the inferred background image. The inferred subject image and the inferred background image are merged to generate the target image.

6. The method according to claim 5, characterized in that, The step of merging the inferred subject image and the inferred background image includes: Determine the depth information and light interlacing relationship of the inferred subject image and the inferred background image; Based on the depth information and the light interlacing relationship, the inferred subject image and the inferred background image are superimposed.

7. The method according to claim 6, characterized in that, The overlay process of the reasoned subject image and the reasoned background image includes: During the overlay process, the inferred subject image and the inferred background image are subjected to at least one global illumination and color adjustment.

8. The method according to claim 3, characterized in that, The process of reconstructing and inferring the interwoven image using the neural radiation field model to obtain the target image includes: The interlaced images corresponding to the at least one subject viewpoint image and the interlaced images corresponding to the at least one background viewpoint image are merged to obtain at least one merged interlaced image; The target image is obtained by reconstructing and inferring from the at least one merged interwoven image using the neural radiation field model.

9. The method according to claim 1, characterized in that, The neural radiation field model is a neural radiation field model adjusted using the Instant-NGP module. The adjustment process includes: Replace the trigonometric function position encoding of the neural radiation field model with the hash encoding of the Instant-NGP module; The multilayer perceptron of the neural radiation field model is adjusted according to the structural design of the Instant-NGP module; The dense sampling strategy of the neural radiation field model is replaced with the hierarchical sampling strategy of the Instant-NGP module.

10. The method according to claim 9, characterized in that, The neural radiation field model adjusted using the Instant-NGP module incorporates a block optimization strategy during training to accelerate parameter convergence.

11. An image rendering apparatus, characterized in that, include: The first module is used to acquire the initial image; The second module is used to determine any viewpoint of the initial image and to determine at least one perspective of the any viewpoint; The third module is used to generate at least one viewpoint image corresponding to the at least one viewpoint; The fourth module is used to perform image interlacing on the at least one viewpoint image according to the ray interlacing algorithm to obtain an interlaced image; The fifth module is used to reconstruct and infer the interwoven image using the neural radiation field model to obtain the target image.

12. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as claimed in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing a computer to perform the method according to any one of claims 1 to 10.

Images